'docker save' sometimes errors with "cannot allocate memory"

Build Environment Caching Dependencies
1

Example here: https://circleci.com/gh/ferrouswheel/nlp-services/29

I’m trying to save a rather large image using docker save (about 5-6GB, because it contains neural net models and CUDA libraries).

50% of the time it’s fine, the rest of the time I get:

write /caches/.docker_temp_050272932: cannot allocate memory
Exited with code 1

I have tried running this command locally and monitoring memory usage, but as far as I’ve seen, it never increases about 50-100MB resident memory.

Any ideas what I’m doing wrong?

2

Hmm, interesting. Your build container has a 4G RAM limit, and I’d guess you are bumping into that. The error does not sound like you are hitting into a disk space issue, which I’d regard as less likely anyway.

Does the same happen if you run it in a SSH session? If so, do you get a “child sacrificed” OS error in your dmesg?

I notice you’re on docker:17.05.0-ce-git, which is no longer current. Take a look on Docker Hub for the new tags - it’s on 18.xx now, I think - perhaps one of them have better internal memory usage? Use the latest one if you can.

Failing that, you could switch to:

  • A Machine executor, which is free for now, and will give you 8G of RAM
  • An larger Docker executor of 8G or 16G. I believe this is normally a paid option, but F/OSS projects might get a nicer deal - you can always ask
1 Like
3

Thanks for noticing that I’d been using an old docker tag. I’ve updated it to 18.06.1 and haven’t had the issue yet.

18.06.1 is the version on my local dev machine that I was profiling the memory of. Fingers crossed that the older versions either had a memory leak or an inefficient implementation of the docker save command.

Otherwise I’ll try to catch the issue with ssh and look for clues in dmesg.

1 Like
4

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.