Process killed in CI job

[EpicEric]

Here’s the relevant job. All other jobs triggered by the same commit succeeded. This kind of error happens sporadically in different places, and restarting the job in question resulted in a successful build. Relevant part of the logs:

[ 90%] Building CXX object lib/Target/X86/CMakeFiles/LLVMX86CodeGen.dir/X86PadShortFunction.cpp.o
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.
lib/Target/X86/CMakeFiles/LLVMX86CodeGen.dir/build.make:278: recipe for target 'lib/Target/X86/CMakeFiles/LLVMX86CodeGen.dir/X86ISelDAGToDAG.cpp.o' failed
make[5]: *** [lib/Target/X86/CMakeFiles/LLVMX86CodeGen.dir/X86ISelDAGToDAG.cpp.o] Error 4
make[5]: *** Waiting for unfinished jobs....

Looking around for the specific C++ error seems to point to the process being killed due to out-of-memory. However, nothing in the test itself should be using too much memory, or else this failure would have been more consistent for this particular job and/or commit.

This example may be related. It has a segfault inside QEMU, while other QEMU-related jobs from the same trigger commit passed without issues.

[halfer]

I don’t have a ready answer for you, but I wonder if you could have an earlier step that installs a memory monitor. At its simplest, this would record memory usage on the box (say every 0.5s) and write it to a build artefact, which you can then view in the UI.

Also, if you are using the Docker executor, consider trying the Machine executor for your job. You get 8G of RAM instead of 4G. If you are on a paid tier you are also permitted to purchase higher memory allocations for Docker too, I think they go up to 16G.

[drazisil]

@EpicEric

It looks like your builds are green again, were you able to figure this out?

[EpicEric]

Not yet. These errors are very sporadic and seemingly random. So far we haven’t been able to accidentally duplicate this error on any jobs since I created this topic.

[drazisil]

Got it. Can you open a ticket with the URL the next time it happens so we can take a look?