Use a docker image, built in previous job, as base image

Build Environment Running Tests
1

I have searched around quite a bit and still haven’t found an answer to this question. Hopefully I didn’t miss it.

Here it is: I want to build a docker image in one job and then use that docker image as a base image in a following job without pushing to a registry. Here’s any example of what I’m talking about. We can’t use the docker-remote-executor because we need the ip_ranges feature. And machine executor is too slow. As is pushing and pulling from a registry.

jobs:
  build:
    docker:
      - image: cimg/base:2023.06
    steps:
      - checkout
      - run: # build docker image
  test:
    docker:
      - image: # image that was just built
      - image: postgresql:current
    steps:
      - run: # tests

Thanks for all the help!

2

Each job is run in isolation so there is no shared state/file system between the 2 created environments or as the CircleCI docs describe it they are both ‘clean environments’.

The next limitation is the fact that your script depends on CircleCI’s own implementation of a docker load command to cause the image to be loaded before other images and steps are then loaded / run. This means you are limited to what is supported by their implementation.

The result is that there is no way to shortcut the use of a registry in your example.

What you could do is switch to using a machine executor and just use docker commands as show in the example found here

This would allow you to merge the build and test jobs into one job and so the generated image would be available to be loaded after being built.

By refactoring your environment you will also gain access to the full docker cli and so the -o and -i parameters that allow you to save and load a tar based version of an image. With the image as a tar file you can then try using the persist_to_workspace or save_cache option to move the file between jobs. In the past other threads have indicated that this can be a slow process due to the overheads of the cache solution. Someone commented that they found it faster to use an external S3 store rather than the cache.

There is also CircleCI’s docker layer caching that may help speed up your build step, but it will not help with the reuse of the generated image

I have left this to last as yesterday there was a forum post that indicated issues when large images are generated, but it is worth a look. One major is that there is a separate charge of 200 credits ‘per job run’.

3

You can save the image as tar, persist it in the workspace, and then load it in the next job. This is documented here:

It works pretty well for me.

Best.