CircleCI Docker Orb "Problem executing scripts APT::Update::Post-Invoke"

[ksylvest]

I’m using the latest CircleCI Docker Orb to build a Dockerfile. Building the Dockerfile works fine locally, but on CI it gives:

Step 7/23 : RUN   apt-get update -qq &&   apt-get install --no-install-recommends -y build-essential libpq-dev npm &&   rm -rf /var/lib/apt/lists/* /var/cache/apt/archives
 ---> Running in 0ea16b3809b7
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code
The command '/bin/sh -c rm -rf /var/lib/apt/lists/*  &&   apt-get update -qq &&   apt-get install --no-install-recommends -y build-essential libpq-dev npm &&   rm -rf /var/lib/apt/lists/* /var/cache/apt/archives' returned a non-zero code: 100

Exited with code exit status 100

The Dockerfile command is:

RUN \
  apt-get update -qq && \
  apt-get install --no-install-recommends -y build-essential libpq-dev npm && \
  rm -rf /var/lib/apt/lists/* /var/cache/apt/archives

I’ve tried running this with both new and existing projects to no avail. What might be causing this? The project is OS and here is a link to the PR:

Optimize Docker (swap from Alpine to Slim)

[rit1010]

The first thing to do will be to split the 3 different commands you are executing in the single RUN command into 3 separate RUN commands so that you can tell which is throwing the error.

The fact that you have a error code, but no indication of what is causing the error means it is likely to be the “apt-get update -qq” command as “-qq” is telling it to be ‘very’ quiet. the apt-get manual states the following

-q, --quiet
    Quiet. Produces output suitable for logging, omitting progress indicators. 
More q's will produce more quiet up to a maximum of two. You can also 
use -q=# to set the quiet level, overriding the configuration file. Note that 
quiet level 2 implies -y, you should never use -qq without a no-action modifier 
such as -d, --print-uris or -s as APT may decided to do something you did not expect. 

Notice the bit about never using “-qq” without other modifiers.

[ksylvest]

Thanks, that’s helpful debugging - but I’m still very lost. I’ve simplified to a practically empty Dockerfile:

FROM ruby:3.2.2-slim
RUN apt-get update

This simple Dockerfile works locally - but doesn’t work on CircleCI.

+ docker build --file=./Dockerfile --tag=docker.io/********/personal:... .
Sending build context to Docker daemon  126.2MB
Step 1/2 : FROM ruby:3.2.2-slim
3.2.2-slim: Pulling from library/ruby
360eba32fa65: Pull complete 
f4acd8d05bbc: Pull complete 
28fab83683a6: Pull complete 
b1b52a1bc24f: Pull complete 
7253d00afd18: Pull complete 
Digest: sha256:...
Status: Downloaded newer image for ruby:3.2.2-slim
 ---> c2ac71ef3cf6
Step 2/2 : RUN apt-get update
 ---> Running in 5f9011810254
Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8906 kB]
Get:5 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [6432 B]
Get:6 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [62.1 kB]
Fetched 9226 kB in 1s (7229 kB/s)
Reading package lists...
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code
The command '/bin/sh -c apt-get update' returned a non-zero code: 100

Exited with code exit status 100

[rit1010]

As you have got the issue down to the updating of the ruby:3.2.2-slim image you can try

• using a different image as there is a range of 3.2.2 options

• post the issue you are seeing to the issue board for the image here

  Issues · docker-library/ruby · GitHub

[ksylvest]

I gave this a try with a base debian image and am seeing the exact same issue:

FROM debian:bookworm-slim
RUN apt-get update

...
Reading package lists...
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code
The command '/bin/sh -c apt-get update' returned a non-zero code: 100

Again this works perfectly with my local Docker. This appears to be an issue with CircleCI. Is that your opinion also?

[rit1010]

Yes, it seems to be an issue with the environment that CircleCI is providing to execute the docker container, but I have no real idea what it could be beyond the exact version of the docker tools being used.

What executor are you defining for your environment? At the moment all I can do is ask for info that may be helpful to staff members when they are next available.

[ksylvest]

@rit1010

What executor are you defining for your environment? At the moment all I can do is ask for info that may be helpful to staff members when they are next available.

Good call on the executor. Since I was using the orb I had not bothered looking into it. It appears to be using an image this is fairly out of date:

machine-agent version 1.0.58497-1b21ea3b (image: "ubuntu-2004:202107-02", public ip: "44.202.135.1", provider: "EC2")

This change to my .circleci/config.yml solves:

Before (Fails)

workflows:
  version: 2.1
  build:
    jobs:
      - docker/publish:
          image: $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME
          tag: << pipeline.git.revision >>

After (Works)

workflows:
  version: 2.1
  build:
    jobs:
      - docker/publish:
          executor:
            name: docker/machine
            image: ubuntu-2204:2023.02.1
          image: $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME
          tag: << pipeline.git.revision >>

I’ve opened an issue:

`RUN apt-get update` on Debian Images Fails w/ Problem executing scripts `APT::Update::Post-Invoke`

With the PR to fix:

fix: non up-to-date executor causing apt-get issues