Way to determine if builds are running out of memory?


#1

We make extensive use of Selenium in our tests and they seem to fail randomly. Is there a way to monitor resources on builds, and/or get notified if they hit memory usage limits? It would be great not to have to SSH in to see this since the failures are sporadic, I’m guessing caused by the order in which our specs are run (we use rspec which runs in random order).


#2

I assume Selenium is a separate Docker image in your set-up, run as a non-primary container. What I would try here is to create a custom image that inherits from the standard Selenium image, and the custom image runs a shell script loop that periodically appends the output of free -h to a file.

The only thing here is that you’d have to work out how to push that data off the container before it is stopped and destroyed. Maybe it would be better to push every loop report to a remote server using curl?

The other thing that occurs to me is whether you can see Selenium logs coming from a non-primary container. If so, is there a verbose mode that you can use? That might show you errors related to running out of memory.


#3

Nope, not the case, I’m using the circleci/ruby:2.5-stretch-node-browsers container, so Selenium runs in the same container (via chromedriver).


#4

Oh marvellous, that’s even easier then. Write the shell script to periodically log memory measurements to a file, running in the background, and then treat that as a test artefact, so it comes up in your CircleCI web interface. You’ll be able to take a look at that even after the container has been removed.


#5

Excuse my ignorance w/ docker, but what’s the best way to do that for a docker container?


#6

As I understand it, your build server is in Docker, but Selenium is just installed from the distro, so you don’t need anything Docker-specific. It’s just GNU/Linux, AFAICT.

My shell skills are a bit ropey, but something like this in a file:

#!/bin/bash
while :
do
    free -h >> /tmp/memory.log
    sleep 10
done

This would create a new memory report in a log file every 10 seconds (adjust to suit). I got the loop stuff from here.

You can then start this at the start of your job in a run item, either using & to background it, or there is a background: true switch I think.

Then you’d need to declare /tmp/memory.log as an artefact - I don’t use artefacts, but the CircleCI manual will cover this.


#7

It actually turns out that free takes an argument to sample periodically so the while is unnecessary:

free -h -s 10 >> /tmp/memory.log

#8

Ooh, good find! :nerd_face:


#9

Unfortunately not getting anything useful out of this, at least that I can discern. It’s showing way more free memory than the container is supposed to have:

              total        used        free      shared  buff/cache   available
Mem:            58G        4.7G         20G        394M         33G         53G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        4.7G         20G        398M         33G         53G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        4.4G         21G        272M         32G         53G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        4.7G         21G        278M         33G         53G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        4.7G         20G        282M         33G         53G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        4.8G         20G        287M         33G         53G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        4.6G         20G        291M         33G         53G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        4.7G         20G        294M         33G         53G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        5.1G         20G        332M         33G         53G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        5.1G         20G        356M         33G         53G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        5.6G         19G        380M         33G         52G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        5.7G         19G        386M         33G         52G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        6.3G         18G        391M         33G         51G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        6.6G         18G        395M         33G         51G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        6.8G         18G        399M         33G         51G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        7.0G         18G        403M         33G         51G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        7.2G         18G        408M         33G         50G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        8.4G         17G        427M         32G         49G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        8.5G         17G        431M         32G         49G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        8.5G         17G        435M         32G         49G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        8.3G         18G        438M         32G         49G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        8.5G         17G        442M         32G         49G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        6.5G         19G        444M         32G         51G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        6.2G         20G        448M         32G         51G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        7.5G         18G        450M         32G         50G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        7.7G         18G        453M         32G         50G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        8.0G         18G        456M         32G         50G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        6.7G         19G        476M         33G         51G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        6.1G         19G        479M         33G         51G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        6.1G         18G        484M         34G         51G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        5.4G         17G        489M         36G         52G
Swap:            0B          0B          0B

              total        used        free      shared  buff/cache   available
Mem:            58G        4.4G         18G        490M         36G         53G
Swap:            0B          0B          0B

#10

Ooh, fair enough - I wonder if that binary effectively shows the memory on the Docker host?

Try top or ps with an appropriate grep instead? Again, I’d use a while loop by default, but either of these can sample every X seconds, that would be great. (Of course, both of these look at memory used, rather than memory free, but they should still let you see if you have any memory leaks).

More thoughts:

  • if you are a paid customer, you can increase your memory limits in Docker (there is a machine size key, see the docs)
  • if you are a free-tier customer, you could move over to a Machine executor (currently also free for the time being). This will double your RAM from 4G to 8G

Neither of these are an elegant solution, since they ignore the problem, but it might be an acceptable temporary fix to get your tests working again.


#11

Maybe @rohara has some input based on his attempts to get this info in the past? Memory limit warning in 2.0


#12

Possibly, but please do try what I have suggested, and offer feedback - volunteering on this forum does take time and effort :slightly_smiling_face:


#13

Sorry, should’ve mentioned that I don’t believe there’s any way to show free memory with ps. As for top, it seems to suffer from the same issue as free. This seems to be a common issue with containers.

GiB Mem :   58.967 total,   15.624 free

Changing to a Machine executor seemed like quite a bit of work at first glance, plus there are big impacts to build start time.

I do appreciate your help @halfer. Thank you!


#14

OK, thanks (replying to a post that seems now to be deleted).

Another idea is to run your own Docker environment (so it would be Docker in Docker). You can then run Selenium as a separate container, and use docker stats to get Docker-level memory information. You could either Dockerise everything, or just Selenium and expose the port to your build container (of course, the latter may be useful to reduce the amount of configuration work).

I do this for my integration tests, using Docker Compose (i.e. I Dockerise everything and run everything as D-in-D).


#15

That’s really weird, I didn’t intentionally delete that post or anything…

Edit: Got a PM that it got caught in a spam filter for some reason. I guess it’ll be back pending review.


#16

The best advice I can give is to look out for exit code 137 and allocate more resources to see if it helps.