Handling multiple docker project in a single repo

Build Environment
1

Hello -
I’m new to CircleCI so please bear with me. I have a github repo with a lot of different docker image environments in it.

  • /projectHercules/Dockerfile
  • /cureCancer/Dockerfile
  • …

Each dir can of course have lots of different code and static files that might get bundled into the image.

It looks like CircleCI expects each repo to be a discrete build environment. I don’t see how I can make a small update to just one docker dir and have CCI only build that image. There’s no functionality that I can find to interpret which files have been changed and to branch the build process accordingly (i.e., only build the appropriate docker image).

Breaking out every docker project into its own repo isn’t really an option as I have dozens of them.

2

Just kicking out some ideas here, not fully formed:

CircleCI by default does a build on push. This means that if you update anything in your repo, it builds everything. I wonder, to start with, you could add all your Docker builds into your project, and just build everything, like so:

  - run:
      name: Build project 1
      command: |
        docker build --tag project1 ./src/project1
  - run:
      name: Build project 2
      command: |
        docker build --tag project2 ./src/project2

OK, from there, you can start using the CircleCI cache system so that Docker will still do all the builds, but it will zip through them because the layers are unchanged. So for each one:

  - run:
      name: Load Docker image layer cache 1
      command: |
        set +o pipefail
        docker load -i /caches/project1.tar | true
  - run:
      name: Build project 1
      command: |
        # --cache-from is an image, not a file/directory
        docker build --cache-from=project1 --tag project1 ./src/project1
  - run:
      name: Save Docker image layer cache 1
      command: |
        mkdir -p /caches
        docker save -o /caches/project1.tar project1

If you have a very high number of images, you could write a script to generate the above, rather than writing it by hand.

Any good? Doesn’t exactly skip builds, but may be just as good in practice.

3

Thanks, I wasn’t aware of docker’s “–cache-from” capability.

It looks like CCI supports docker_layer_caching which supposedly transparently caches docker layers on their infrastructure. This should negate having to manually save/load images as part of the build process.

If this works as advertised then building all images with each commit will be doable.

1 Like
4

Ooh, nice. I’m on the free tier, so it looks like I can’t access that, but doing it manually has been no bother.