Failing Tests using Dockers in CI

Build Environment
,
1

Our tests used to pass but now we get:
WARNING: Your kernel does not support memory swappiness capabilities, memory swappiness discarded.

and errors regarding containers like:
Error response from daemon: Cannot destroy container 0478e4cd1888fc63db7992119a815b022e8a29484ce1966de6723b71a25e762a: Driver btrfs failed to remove root filesystem 0478e4cd1888fc63db7992119a815b022e8a29484ce1966de6723b71a25e762a: Failed to destroy btrfs snapshot: operation not permitted
Error: failed to remove containers: [0478e4cd1888fc63db7992119a815b022e8a29484ce1966de6723b71a25e762a]

For the same commit, we have retried a build that passed a month ago and now fails:
https://circleci.com/gh/codacy/codacy-eslint/17
https://circleci.com/gh/codacy/codacy-eslint/49

At the moment the circle.yml is altered because we were exploring some workarounds, but you can check our usual circle.yml:
circleci.com/gh/codacy/codacy-eslint/17#config

We already tried a lot of ssh debug but we can’t find the issue. Maybe it is related with github.com/bioboxes/command-line-interface/issues/59? It is suppose to be fixed by github.com/bioboxes/command-line-interface/pull/60

2

This is a known issue, it is addressed here.

4

I have more details about the issue. The docker is being executed with all the security constraints:

docker run --net=none --privileged=false --cap-drop=ALL -t --entrypoint=bash “codacy-eslint:latest”

After some more testing, I found that --cap-drop=ALL might be the problem.
Inside the docker in the dockerfile we created a user with id 2004 and name docker. In the past all these settings were working, but now it’s failing with:

FATA[0000] setup user operation not permitted

Do you guys have any idea how to make this run again?

5

I don’t think I have any insight into this right now. Does the same operation succeed if you omit the --cap-drop=ALL part?

6

It works without --cap-drop=ALL, but the whole point to test in circleCI is to make sure it works in the same conditions of our production environment…
Can you have someone look into this?

7

The behavior that you are seeing seems to be correct for me.

When -cap-drop=ALL is specified, user creation should fail. Although it behaves slightly different, I also get an somewhat error when running the following command locally.

docker run --cap-drop=ALL ubuntu /bin/sh -c "useradd -m -p encryptedPassword foobar;"
useradd: failure while writing changes to /etc/shadow
useradd: /home/foobar was created, but could not be removed

This works perfectly without cap-drop. So, I have a feeling that the our old docker was acting wrong and now it works in the correct way.

How does the line in your Dockerfile to create a user look like? Also, what exec driver are you using in your production?

We are using LXC driver which you are probably not using, so unfortunately the environment of your production and CI are already different at driver level.

2 Likes