Docker cache-from behavior issues between CircleCI and local workstation

Build Environment
1

Hi there !

I am helping people at my company that are working with Pythons and R as development languages.

We have written a pretty simple Dockerfile and have taking care to have the requirements steps early in a separated RUN instruction.

FROM python:3.8-slim

# Update existing packages
RUN apt update \
  && apt upgrade -y

# R Source list
RUN apt install -y gnupg \
  && echo "deb http://cloud.r-project.org/bin/linux/debian buster-cran40/" >> /etc/apt/sources.list \
  && apt-key adv --keyserver keys.gnupg.net --recv-key 'E19F5F87128899B192B1A2C2AD5F960A256A04AF' \
  && apt update \
  && apt install -y \
    r-base r-base-dev r-base-core r-recommended \
    libcurl4-openssl-dev libssl-dev

# Install pip requirements
WORKDIR /app
COPY ./pip_requirements.txt ./
RUN pip3 install -r pip_requirements.txt

# Install pip requirements
COPY src/lib/required_libraries.R ./required_libraries.R
RUN Rscript required_libraries.R

# Copy src
COPY "./data/" "./data/"
COPY "./src/" "./src/"

The R Stuff take an awful lot of time to build and compile
The whole build takes around 45min the first time.

To speedup the build process, we have added instructions in our Makefile to pull the latest image we have already built from the registry and then run docker build --cache-from.

This works on our workstations (with Docker 20.10.x).

$ make build

===> pre-build-cache: Pull and tag xxxx 
docker pull xxxx:latest
latest: Pulling from xxxx
45b42c59be33: Pull complete 
...


===> Build xxx           
docker build --cache-from xxxx:latest -t xxxx:latest .
Sending build context to Docker daemon  469.5kB
Step 1/16 : FROM python:3.8-slim
3.8-slim: Pulling from library/python
45b42c59be33: Already exists 
f875e16ab19c: Already exists
...
 ---> Using cache
...
Successfully built 141ba19d3cb2

I have even tested with a completely new Linux laptop and it works !

Unfortunately, this does not work on CircleCI … we always end up with the full build.
The only thing I can think of right now is the version of Docker.
The default version shipped with Machine executors are 17.09-ce …

Have some of you any clue about this issue ?
Otherwise, how would I be able to have a more recent and up-to-date Docker engine version in the CI ?
Should I go with the remote_docker stuff to be able to select the version of Docker I need ?

Thanks all of you for your help :pray: