Connection issues with apt-get from archive.ubuntu.com

Build Environment
1

Over the past couple of days we’re seeing a large number of build jobs fail with messages like:

E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/libw/libwebp/libwebp6_0.6.1-2ubuntu0.20.04.2_amd64.deb  Connection failed [IP: 185.125.190.36 80]

or:

E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/s/systemd/systemd-sysv_249.11-0ubuntu3.9_amd64.deb  Could not connect to archive.ubuntu.com:80 (91.189.91.38), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.91.39), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out

I’ve seen some posts in the past, like Could not connect to archive.ubuntu.com:80 and Can't apt-get update reliably in builds. Usually they go along with an Ubuntu outage, but I’ve checked the status page and no active or recent outages are showing.

Sometimes a retry will fix it. Sometimes multiple retries.

Images where we’ve encountered these include cimg/node:16.13.2, cimg/node:16.18.1-browsers. I can see at least the newer one is using apt 2.4.8, and I understand apt 2.3.2 and newer already use 3 retries by default.

Does anyone have experience with this? It seems to mainly affect CircleCI jobs: when we try the same commands locally, they work, but are likely hitting different mirrors, or taking different routes.

I’ve seen some references to configuration like:

echo 'Acquire::Retries "5";' > /etc/apt/apt.conf.d/80-retries

… but it seems like something is quite broken if it takes 5 retries to download a file.

3 Likes
2

We are seeing this today with urls like: http://archive.ubuntu.com/ubuntu/pool/universe/n/node-get-stream/node-get-stream_6.0.1-1_all.deb

The URLs resolve fine locally but are failing in our CircleCI builds. Archive domain on Ubuntu’s status page does not show any active issues at the time we are experiencing this. https://status.canonical.com/

It seems like a network issue? Also seeing what looks like 3 retries in our logs based on Ign prefix lines.