Docker Layer Caching on 1.0

Build Environment
1

Dear @anon30319619,

I am using circle 1.0, and I figured out that we could cache docker layers by modifying the “Docker Root Dir”.

Our circle.yml starts as follows:

machine:
  pre:
    - sudo curl -L -o /usr/bin/docker 'https://s3-external-1.amazonaws.com/circle-downloads/docker-1.9.1-circleci'
    - sudo chmod 0755 /usr/bin/docker
  environment:
  services:
    - docker
  post:
    - docker info
    # Cache docker installation - must be done after cache has been deployed
    - mkdir -p ~/.docker/var/lib/docker 
    - sudo chown --reference=/var/lib/docker /home/ubuntu/.docker/var/lib/docker && sudo chmod --reference=/var/lib/docker /home/ubuntu/.docker/var/lib/docker
    - echo 'DOCKER_OPTS="$DOCKER_OPTS -g /home/ubuntu/.docker/var/lib/docker"' | sudo tee -a /etc/default/docker && sudo service docker restart
    - docker info
dependencies:
  cache_directories:
    - "~/.docker/var/lib/docker"

This works ok when the cache is empty. The docker root dir is successfully moved. However, when the cache is restored, some of the permissions of the files are not correct. I’ve tried manually setting them, but didn’t work.

When building images with cached layers that need be rewritten, I get the following errors (see Step 8):

Step 1 : FROM poldracklab/neuroimaging-core:base-0.0.2
 ---> 4411cb4c0075
Step 2 : COPY docker/files/run_* /usr/bin/
 ---> Using cache
 ---> 4c2241745b07
Step 3 : RUN chmod +x /usr/bin/run_*
 ---> Using cache
 ---> 916d0fe9724f
Step 4 : RUN curl -sSLO https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh &&     /bin/bash Miniconda3-latest-Linux-x86_64.sh -b -p /usr/local/miniconda &&     rm Miniconda3-latest-Linux-x86_64.sh
 ---> Using cache
 ---> 815c9f3d6a0f
Step 5 : ENV PATH /usr/local/miniconda/bin:$PATH PYTHONPATH /usr/local/miniconda/lib/python3.5/site-packages PYTHONNOUSERSITE 1 LANG C.UTF-8 LC_ALL C.UTF-8
 ---> Using cache
 ---> fda9403a6cd7
Step 6 : WORKDIR /root
 ---> Using cache
 ---> a8b702a72395
Step 7 : RUN conda config --add channels conda-forge &&     conda install -y numpy>=1.12.0 scipy matplotlib &&     python -c "from matplotlib import font_manager"
 ---> Using cache
 ---> 3708c365a7fc
Step 8 : COPY . /root/src/mriqc/
chown /home/ubuntu/.docker/var/lib/docker/btrfs/subvolumes/afe98156c17ac078736639aa430657521e1062b10726cd55cf44fcd67360ac73-init: operation not permitted

Until step 8, docker successfully uses cached layers - this did not happen before, with the docker save / docker load trick.

The problem is that, after cache restoration, I get files with nobody:nogroup owners in the btrfs subvolumes. Is there any way to workaround this?

I can imagine this trick (modifying the docker root dir) has been tried before. Is this the reason why docker caching does not work on Circle 1.0?

Thanks very much

2

On the machine executorType, you’re running as the circleci user and likely need to sudo a lot of the commands you’re attempting. With that said, I have not seen anyone attempt layer caching like that on 2.0 (yet).

Can you link me to your build?

1 Like
3

Sure, this is the build: https://circleci.com/gh/oesteban/mriqc/687

Just to recall, it is circle 1.0.

Caching docker this way wouldn’t be the most natural way?

4

Ah I understand. Thanks for providing the link. No, I don’t think that method will work, and if it did it would likely cause issues trying to switch platforms.

Instead, do you want to try docker export and docker import? I haven’t personally tried it, and it wouldn’t be useful in all use cases of caching, but it has the potential to speed up your build.


5

I guess you meant docker save / load, which doesn’t quite work (in my experience). The export/import option only works for one flattened image, and here we need to store the layers, right?

Where is the appropriate place for this question about CircleCI 1.0?. I don’t mind learning the 2.0 way, but for some of my builds I will need to stick with 1.0 for a while. Should I move this question to the circle 1.0 support forum?

6

You’re right that the export is a flattened tar.gz. Definitely would not work in all use cases, but would fix some.

You can move it, or leave it here. It’s a fair question regardless.

The 2.0 way is to use executorType: docker and enable layer caching. You’d need access through your CSM, and I can only imagine this will be a premium feature in the future, but layer caching is supported properly at this time. Just not 100% yet as it’s not available for executorType: machine.

7

For what it’s worth, we found that pulling the image from Docker Hub was faster than doing the save/load trick with the cache in CircleCI. Apparently, the caching system in CircleCI is really slow or something, because we were pretty surprised that that ended up being true.

1 Like
8

Thanks, for some specificity of our project we need to build images based on some ubuntu version directly. Therefore, for us it is very much needed we can use the Docker cache transparently.

Additionally the save/load trick seemed to me pretty unreliable: our images are very large, and most of the times the docker load just didn’t work. Oftentimes, we would get very old layers cached.

I think I’m arriving to a point where we can use Docker cache in CircleCI 1.0. Changing Docker’s storage system to VFS may not produce the permission errors I was having with BTRFS.

I would really appreciate any expert on CircleCI 1.0 that could shed some lights on these permission problems I have with BTRFS. Also, I would really appreciate any explanation of how the Circle Cache deals with file ownership. They seem to change to ubuntu:ubuntu after restoring the cache.