Using docker save/load caching strategy makes docker push VERY slow

Build Environment
1

We have a continuous integration pipeline on circleci that does the following:

  1. Loads repo/image:mytag1 from the cache directory to be able to use cached layers
  2. Builds a new version: docker build -t repoimage:mytag2
  3. Saves the new version to the cache directory with docker save
  4. Runs tests
  5. Pushes to docker hub: docker push repo/image:mytag2

The problem is with step 5. The push step takes 5 minutes every time. If I understand it correctly, docker hub is meant to cache layers so we don’t have to re-push things like the base image and dependencies if they are not updated.

I ran the build twice in a row, and I see a lot of crossover in the hash of the layers being pushed. Yet rather than “Image already exists” I see “Image successfully pushed”.

Here’s the output of build 1’s docker push:
https://gist.githubusercontent.com/jtmarmon/4a376cbb5e70d1ce2a7e/raw/79c7b8cb49c5981aed92b7a5e20855c56ecc385f/dockerhub_slow_build1.txt
And build 2:
https://gist.githubusercontent.com/jtmarmon/b13ee9c7496ad6b31e21/raw/262662daf19224c1da491b337b84739a0056da56/dockerhub_slow_build2.txt

If you diff those two files you’ll see that only 2 layers differ in each build:

1,4c1,4
< ca44fed88be6: Buffering to Disk
< ca44fed88be6: Image successfully pushed
< 5dbd19bfac8a: Buffering to Disk
< 5dbd19bfac8a: Image successfully pushed
---
> 9136b10cfb72: Buffering to Disk
> 9136b10cfb72: Image successfully pushed
> 0388311b6857: Buffering to Disk
> 0388311b6857: Image successfully pushed

So why is it that all the images have to re-push every time?

Thanks!

1 Like
2

Got a lot of info about how to fix this here - https://github.com/docker/docker/issues/19583 - but i’m at a loss as to how to effectively cache this directory without running docker push before the test step

3

I have similar issue - very slow docker push execution (even if it needs to push only one layer, it takes huge amount of time to buffer all other to disk). Also, cache restoring time + docker load + docker save takes more time than clean docker build without cache.

4

I dont suppose we can save the ~docker cached directory after a build? Would that help with buffering?

1 Like