Using docker save/load caching strategy makes docker push VERY slow



We have a continuous integration pipeline on circleci that does the following:

  1. Loads repo/image:mytag1 from the cache directory to be able to use cached layers
  2. Builds a new version: docker build -t repoimage:mytag2
  3. Saves the new version to the cache directory with docker save
  4. Runs tests
  5. Pushes to docker hub: docker push repo/image:mytag2

The problem is with step 5. The push step takes 5 minutes every time. If I understand it correctly, docker hub is meant to cache layers so we don’t have to re-push things like the base image and dependencies if they are not updated.

I ran the build twice in a row, and I see a lot of crossover in the hash of the layers being pushed. Yet rather than “Image already exists” I see “Image successfully pushed”.

Here’s the output of build 1’s docker push:
And build 2:

If you diff those two files you’ll see that only 2 layers differ in each build:

< ca44fed88be6: Buffering to Disk
< ca44fed88be6: Image successfully pushed
< 5dbd19bfac8a: Buffering to Disk
< 5dbd19bfac8a: Image successfully pushed
> 9136b10cfb72: Buffering to Disk
> 9136b10cfb72: Image successfully pushed
> 0388311b6857: Buffering to Disk
> 0388311b6857: Image successfully pushed

So why is it that all the images have to re-push every time?



Got a lot of info about how to fix this here - - but i’m at a loss as to how to effectively cache this directory without running docker push before the test step


I have similar issue - very slow docker push execution (even if it needs to push only one layer, it takes huge amount of time to buffer all other to disk). Also, cache restoring time + docker load + docker save takes more time than clean docker build without cache.


I dont suppose we can save the ~docker cached directory after a build? Would that help with buffering?