Tests get killed with mysterious "Received 'killed' signal" message

ruby

#1

One of our project’s tests runs for about 40 minutes. That works fine in 1.0. However, when moving them to 2.0, test constantly gets killed after about 10 minutes of running bundle exec rake test with vague “Received ‘killed’ signal” message.

So far I’ve tried:

  • run:
    name: run tests
    command: bundle exec rake test
    no_output_timeout: 50m

Did not help.
Also, I tried SSHing to the test box and running bundle exec rake test manually – same thing: it gets killed after about 10 minutes. I’ve been monitoring “top” in the next window, and amount of free RAM never got even close to zero, so I don’t think RAM usage is the problem.

Any idea where is the elusive timeout setting? Searching in the documentation totally does not help.


#2

In your SSH environment, try dmesg after the process killing to see if there are any clues in there. If a process needs to be sacrificed for some reason, it is usually recorded here.


#3

sigh You are right, according to dmesg it’s memory issue. Which is interesting because a) CIrcleCI 1.0 did not have that issue, b) top certainly does not report the same usage values.


#4

Ah, someone else had that issue here recently - search for “Docker top” and you’ll find it. The issue, I think, is that the kernel isolation is Docker is not entirely seamless, so top inside a container will report the memory state of the Docker host. It’s not a fault so much as how containerisation works.

If you’re running using the standard Docker executor, and you have reduced your memory usage as much as possible via configuration, you have a couple of choices:

  • Switch to the Machine executor, currently free of charge, to bump your RAM from 4G to 8G, available for free and paid tiers
  • Switch to a higher class of Docker machine, for a fee, for paid tiers only

#5

halfer’s post is correct (as he often is!) – this sounds like a memory issue. If you don’t absolutely need docker, the machine executor would give some advantages RAM-wise, doubling from 4gb to 8gb, and allowing you to, for example, set more swap space to be used when no memory remains.

If you’re on a paid plan, you can send a ticket to support requesting a trial of resource_class and we’ll be happy to accommodate the request.


#6

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.