Configuring Docker Daemon for using containerd image store instead of docker

Build Environment
1

Hi,

We are trying to leverage a relative new feature released by docker - which is using the containerd image store instead of the default docker - which does not support storing attestations.
On the official Docker docs (titled “containerd image store with Docker Engine”)
it suggests to change the Docker Daemon settings by manually adding the daemon.json to the /etc/docker/ folder - which of course is not supported as part of a CircleCI pipeline.
We tried using SSH and manually setting this up (and later move the commands to the pipeline), but could not restart the Docker Daemon successfully.

Any suggestions here?

Thanks!

2

can you share some more information or commands or errors you are getting when restarting the daemon.

3

Sure, I tried these set of commands:

circleci@c4310c9b0b97:/etc/docker$ sudo systemctl restart docker
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
circleci@c4310c9b0b97:/etc/docker$ sudo service docker restart
/etc/init.d/docker: 61: ulimit: error setting limit (Operation not permitted)
circleci@c4310c9b0b97:/etc/docker$ sudo service  docker --full-restart
 * Docker already stopped - file /var/run/docker-ssd.pid not found.
/etc/init.d/docker: 61: ulimit: error setting limit (Operation not permitted)

And then this:

circleci@b8b9907352b6:/etc/docker$ sudo pkill dockerd
circleci@b8b9907352b6:/etc/docker$ sleep 5
circleci@b8b9907352b6:/etc/docker$ sudo dockerd &
[1] 118
circleci@b8b9907352b6:/etc/docker$ INFO[2024-07-09T08:46:13.187165223Z] Starting up                                  
failed to load listeners: can't create unix socket /var/run/docker.sock: device or resource busy
^C
[1]+  Exit 1                  sudo dockerd

Also tried to delete the docker.sock but without success:

circleci@b8b9907352b6:/etc/docker$ sudo rm -f /var/run/docker.sock
rm: cannot remove '/var/run/docker.sock': Device or resource busy
4

i thinks it’s related to user permission.

although sudo should have worked but i think there is some misconfigurations in systemd

5

Have you passed the docker sock to any container as some container require to pass docker sock of the host machine.

6

I have not touched anything other than the default config… we don’t even have setup_remote_docker as one of the steps
Any other commands I should try using?

7

Have you tried reinstalling docker

8

inside the CircleCI machine? no…
doesn’t sound like the right approach… too hacky for a pipeline

9

hmm if you are running it in Circleci machine then i think it is expected that you can’t restart the docker even if you run sudo apt-get update you will see some warnings.
I think they have some kind of security setup on there machines

10

what image you are using in the machine

11

this is what i get when running docker version:

circleci@b8b9907352b6:/etc/docker$ docker version
Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:31:44 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.0.2
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       7cef0d9
  Built:            Thu Apr 18 16:27:07 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
12

this is the docker image config:

    docker:
        - image: cimg/node:20.11
13

I hope you are not running docker inside docker because circleci does not support running docker inside docker you can check the answer here . try using this
image for your machine i hope it will fix the problem.

machine:
      image: ubuntu-2404:current
14

Amazing, this worked!!

I changed the image as you suggested and added this step:

    modify_docker_daemon_config:
        steps:
            - run:
                  name: Modify Docker Daemon Configuration
                  command: |
                    echo '{"features": {"containerd-snapshotter": true}}' | sudo tee /etc/docker/daemon.json
                    sudo systemctl restart docker

and now our build creates the attestation as we wanted!

Thanks :pray:

15

always happy to help