How are you defining the executor within your config.yml file, you will need to set the option
docker_layer_caching: true
If you expect to use your image frequently in the CI process you may find that uploading it to docker hub improves performance even if caching is enabled. This is because CircleCI has tight integration between their systems and Docker’s. In the past they have commented on the fact that both companies were operating in the same AWS region and that they directly accessed Docker’s S3 store.
I’ve also come across the following staff member reply that hints at how the caching process works.
As @timothyclarke has mentioned, in general we run builds over a large number of machines, and images will only be pulled from cache if they were previously used with that machine. The rate at which you will get a cache hit depends on how many other users are using the same image in their builds. If the image you are using is older or not used by many users, the chances of it getting pulled from cache is lower.
If they are caching based on usage you may find that your private image is not cached that frequently.
Circle is not caching any part of my custom but public Docker image