Docker Error removing intermediate container

docker
known-issue

#1

When working with Docker containers on CircleCI, if you try to run docker run with the --rm flag your builds will fail with the error message below:

Error removing intermediate container 1f3a1d541241: 
Driver btrfs failed to remove root filesystem 
1f3a1d541241e757d48f34329508253e9ee139380b7b914a3b1104677eb0e8ee: 
Failed to destroy btrfs snapshot: operation not permitted

Docker gives this error because it’s not possible to remove the container image in the file system that we are using on CircleCI. We are working to fix the Docker daemon so that it does not not try to remove intermediate container image to suppress the error in the future container release.

Workaround
In order to fix this issue for now, skip this step by removing the --rm flag from the docker run command.


Error removing intermediate container during Docker build
Failing Tests using Dockers in CI
Docker gives kernel memory swappiness error without `--rm` flag
#2

Any estimate on how long that issue will take to address?

When doing all of the docker commands in your own custom build it’s not that hard to work around, but as more integrated solutions become prevalent – where the docker commands and configuration are built into the tool itself – it can be difficult to work around.

In Elixir, for example, we were using a plugin for mix called the Elixir Docker Image Packager. In order to get past this problem we needed to work off of a forked version because in the normal use case you’d want to ensure intermediate containers are deleted.

Not a total deal killer, but still, it’s one of the many straws that eventually crush the camel when trying to do full end-to-end docker builds for private repos.


#3

@goneflyin Unfortunately no ETA :frowning:

We totally get that this is a pretty hacky workaround and is not ideal for most people. Another suggested workaround from this blog post comment is to return true after the rm commant. - docker rm -f CONTAINER_NAME || true with this method even though the container is not actually removed, the tests will continue to run. This also means that in environments that support removing the Docker container it will still be removed. This way you do not have to maintain two separate docker configurations.


#4

I had similar issue while my Dockerfile was based on container image build with docker 1.6. It seems there is some incompatibility in the btrfs between 1.6 and 1.8 docker images. To resolve that issue I had to import content of the base container into my Dockerfile, but rebuilding base image with 1.8 should also help (if you have control over the base Dockerfile of course).


#5

That would be great to get this issue fixed so that docker run --rm works. I use --rm to clean up containers after they have been run locally on my development machine. It would be a shame to add special code for tests to pass on CircleCI.


#6

Not sure if this can be fixed since the problem appears to be related to docker running in an unprivileged LXC container. CircleCI will likely never use privileged LXC containers for builds. This would need to be fixed upstream on the Docker/LXC side.


#7

Does this mean heroku docker:release can’t work on CircleCI? I haven’t been able to deploy this way despite being told by one of your support engineers that this is possible, and the error I keep getting is related:

Error response from daemon: Driver btrfs failed to remove root filesystem 965c64a946e9456e7de152d44c839d189e4583108e1de95d25577e1d9ddeaa23: Failed to destroy btrfs snapshot /var/lib/docker/btrfs/subvolumes for 965c64a946e9456e7de152d44c839d189e4583108e1de95d25577e1d9ddeaa23: operation not permitted
Error: failed to remove containers: [965c64a946e9456e7de152d44c839d189e4583108e1de95d25577e1d9ddeaa23]

The heroku docker code relies on this working: https://github.com/heroku/heroku-docker/blob/master/commands/release.js#L128


#8

A Heroku employee has forked the heroku-docker plugin to make docker rm optional.

I’ve pushed a proof-of-concept showing how to use his fork. This isn’t optimized with caching, but if anyone would like to figure that out and share, that would be very welcome.


#9

Nice! This is way better than my hacky sed hot-patch of the offending line :slight_smile:


#10

Would upgrading to latest docker solve this problem? Found this issue, it appears to be fixed: https://github.com/docker/docker/issues/9939

I’m hitting this even though I’m not using any volumes at all and I’m not using docker CLI (I’m directly hitting the remote API) so it’s even harder for me to workaround this.


#11

I get this error during a docker build using docker versions 1.9.1 and 1.10 as given in docker-1-9-1-is-available and docker-1-10-support.

Error removing intermediate container d1eae450392f: rmdriverfs: Driver btrfs failed to remove root filesystem <volume id>: Failed to destroy btrfs snapshot /var/lib/docker/btrfs/subvolumes for <volume id>: operation not permitted


#12

Unfortunately, in our tests, docker-compose run does not return the correct exit code from the run command/script without the --rm flag (screenshot attached (tried with 1.6-1.7).


#13

Has there been any progress on this. I’m seeing this same error while evaluating CircleCI and I’m unable to build a simple Java HelloWorld Spring Boot app and build a container.


#14

Due to the way that Docker works inside of an unprivileged LXC container there is nothing that we can except the listed workaround.


#15

Seemingly this is not high up on the priority list. Travis CI, however has worked flawlessly with Docker, allowing it to remove containers as and when required.


#16

Also, seeing this when building an image. Am not using --rm.


#17

Thanks, I see the same thing when building an image as well. It is safe to ignore for most cases. If your image is failing to build it is very likely causes by some other issue.


#18

Just wanted to chime in. I’m seeing the same issue when building an image but it seems to build correctly.


#19

Also seeing this when trying to build images. It’s causing errors in the built images.

Getting docker: Error response from daemon: rpc error: code = 2 desc = "oci runtime error: not a directory". from the docker daemon when trying to run the images.


#20

Removing intermediate container would be to my understanding mandatory in builds where you want to smoke test your Docker images before pushing them into registry.

When I’m trying to run a container from the image I’ve built during build process I get error because the intermediate container is reserving the port that the smoke test container is also trying use.