It might be worth being more explicit about the specific problem you are encountering here. Is it about preserving the Docker images between jobs in a workflow?
If so, how many images do you have, and what size is each one?
+1 — can you elaborate on how you see this working? You want multiple jobs to reuse the same environment you set up with remote docker? What makes it hard to use a single job to use that environment? Or, are you saying you want to build a single image than use that as the primary image of other jobs?
@halfer Yes, it’s about preserving the work done in the build stage through to other jobs.
I don’t think the size of the images, or how many, really matters. We happen to follow a single image strategy for our own image (using different commands in compose for our services), but this issue would be the same if we built an image for each of several services.
@ndintenfass I want to build one or more images during a build job and then either share that docker environment with further jobs, or have the images efficiently available for the new docker environments in subsequent jobs.
I want to avoid docker save/load or docker push/pull between jobs as it makes the build config more complex, less reliable, and slower.
I do not want the subsequent jobs to run in the built image, if that’s what you mean by primary.
What prevents me doing this in a single job is that I’d like to use workflows to do things like only run the deploy step on a build of master.
My workaround, for now, will be docker save/load to the workspace but for a “docker-centric” build service this feels like something I shouldn’t have to do.
@dowtkrob We have the same requirements for build/test/deploy and the only solution I could come up with is to use a registry, saving the image to the workspace is super slow. We use docker-hub (AWS was too slow), with machine executor and docker_layer_caching. Pushing/pulling our built image takes about a minute. We also use docker-compose to run our tests so we only save the docker-compose*.yml files to the workspace and use that in subsequent jobs.
To give you a rough idea:
Build:
checkout
build image
tag image with workflow sha
push image to registry
save docker-compose*.yml to the workspace
Test:
pull image from registry
tag image as latest
grab the `docker-compose*.yml from the workspace
docker-compose run test
Deploy:
pull image from registry
tag with branch name
push image to registry
With docker_layer_caching, the build step hits the cache most of the time, cache is not guaranteed:
However, some of the jobs may have cached layers, some may not have cached layers, and not all of the jobs will have identical caches.
And for the test step, the docker_layer_caching lets us re-use the already downloaded dependency images such as python, mysql, redis, etc. so nothing needs to be re-built or re-downloaded.
Sure it does, because of the workaround you’ve identified, which is pushing to a registry and pulling again. I have several separate build jobs, which each push to a registry, and then an integration test job, where I pull all the images (this is not a workaround for me, as the jobs are not joined in a workflow).
FWIW, I pull ~10 images from GitLab totalling around 950M, and this is completed in around 50 seconds. I think I might have had one network failure in the last six months. The test job runs once a day (used to be twice a day). It depends what you regard as slow, I suppose!
So, I understand what you want, and I don’t disagree with the proposal, but using a registry seems to be a pretty solid workaround.
@arthurio I’m surprised you need to bother with a registry at all if you’re using docker_layer_caching. But then again, that’s a guess, because I find the terms used in Circle’s documentation of setup_remote_docker and docker_layer_caching somewhat cryptic.
In a vanilla Docker architecture, a Docker registry manages a bunch of images and a Docker daemon has a cache of images retrieved from a registry or built directly. The docs say that setup_remote_docker creates a new Docker “environment”, which I assume means a new Docker daemon, which dies at the end of each job along with any images it has cached. I assume docker_layer_caching just saves that Docker daemon’s cache and restores it the next time a new Docker daemon is created when a setup_remote_docker directive is encountered, so it doesn’t start with a blank cache.
If I’m correct, using docker_layer_caching should solve @dowtkrob’s problem and avoid @arthurio having to push to registries.
@emersonf We were also expecting the docker_layer_caching to be enough but we noticed that the cache was missing fairly often and not in a predictable way, so we started relying on the registry instead and the build times have been predictable and consistent since then. Again, in our case, each pull/push takes only ~1m so having couple minutes added to the whole Job in favor of consistent build time is a no brainer.
Finally, based on the docs, it’s important to note that the docker_layer_caching works horizontally, notvertically, which is why it doesn’t really help in a build -> test -> deploy setup.