Docker images between jobs in Workflows

docker
circle.yml

#1

Is it possible to persist docker images between build and push steps?
Ideally I’d like to build different docker images and push them in a separate step.
Even layer caching cannot guarantee that my image will be “cached” or “saved” or whatever.
The terminology used is quite confusing in the config.yml in 2.0 in general: caching should be something that speeds up execution time but is not completely necessary to persist(?). Instead it is used as a memory store between jobs in a workflow. (workspace is not a great abstraction either).

This is a feature we’d easily switch to premium for.

Also referencing this Docker + workflows
which doesn’t solve anything as it’s a major hack.


Caching Docker images in Workflows
#2

Agree that this is an issue that makes builds that build and publish Docker images pretty painful – would be really great to have a first-class story for this.

Workflows needs a 'previous image' directive and Unhappy with Docker build-test-deploy scenarios also seem to be tracking the same underlying issue.

As noted in those issues, I think the main request is to be able to have multiple jobs that build Docker images in parallel and then have the ability for the “deploy” step to publish those images that were built in the other containers. For non-Docker builds, the build artifacts in the parallel builds can be transferred using the file transfer mechanism, but without a corresponding mechanism for Docker it’s hard to implement this workflow.


#3

Can you use “docker save/load” to turn docker images into .tar artifacts that can be passed between jobs maybe ?

That’s my plan but I’m actually not sure of how to access artifacts between jobs, are they part of the workspace ?


#4

Yes, you can use docker save/load. And use cache to pass artifacts between jobs.

See here: Caching Docker build images


#5

Great, thanks for the tip – will try to give that a spin.

Out of curiosity, have you found this to be generally performant? The last time I tried to save and load Docker images using save and load in CircleCI, the amount of time it took to execute the save and load was pretty expensive (on the order of minutes), which made the approach less attractive. However, if this is reasonably performant and supports multiple images, this would probably solve my issue/use case.


#6

The performance is not great, but good enough for us. For our ~600MB docker image:

On the building side:

  1. docker save… takes ~12 seconds
  2. Saving cache takes ~50 seconds

On the publishing side:
3) Restoring cache takes ~12 seconds
4) Loading docker image (docker load…) takes ~3 seconds


#7

I think this is a good approach. I would like CircleCI to have a first-class support for this, however. The 2.0 switch was supposed to be Docker-centric but seems to be lacking simple use cases for Docker-based applications.


#8

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.