Connection issues with apt-get from archive.ubuntu.com

Over the past couple of days we’re seeing a large number of build jobs fail with messages like:

E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/libw/libwebp/libwebp6_0.6.1-2ubuntu0.20.04.2_amd64.deb  Connection failed [IP: 185.125.190.36 80]

or:

E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/s/systemd/systemd-sysv_249.11-0ubuntu3.9_amd64.deb  Could not connect to archive.ubuntu.com:80 (91.189.91.38), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.91.39), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out

I’ve seen some posts in the past, like Could not connect to archive.ubuntu.com:80 and Can't apt-get update reliably in builds. Usually they go along with an Ubuntu outage, but I’ve checked the status page and no active or recent outages are showing.

Sometimes a retry will fix it. Sometimes multiple retries.

Images where we’ve encountered these include cimg/node:16.13.2, cimg/node:16.18.1-browsers. I can see at least the newer one is using apt 2.4.8, and I understand apt 2.3.2 and newer already use 3 retries by default.

Does anyone have experience with this? It seems to mainly affect CircleCI jobs: when we try the same commands locally, they work, but are likely hitting different mirrors, or taking different routes.

I’ve seen some references to configuration like:

echo 'Acquire::Retries "5";' > /etc/apt/apt.conf.d/80-retries

… but it seems like something is quite broken if it takes 5 retries to download a file.

2 Likes