Over the past couple of days we’re seeing a large number of build jobs fail with messages like:
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/libw/libwebp/libwebp6_0.6.1-2ubuntu0.20.04.2_amd64.deb Connection failed [IP: 185.125.190.36 80]
or:
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/s/systemd/systemd-sysv_249.11-0ubuntu3.9_amd64.deb Could not connect to archive.ubuntu.com:80 (91.189.91.38), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.91.39), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out
I’ve seen some posts in the past, like Could not connect to archive.ubuntu.com:80 and Can't apt-get update reliably in builds. Usually they go along with an Ubuntu outage, but I’ve checked the status page and no active or recent outages are showing.
Sometimes a retry will fix it. Sometimes multiple retries.
Images where we’ve encountered these include cimg/node:16.13.2, cimg/node:16.18.1-browsers. I can see at least the newer one is using apt 2.4.8, and I understand apt 2.3.2 and newer already use 3 retries by default.
Does anyone have experience with this? It seems to mainly affect CircleCI jobs: when we try the same commands locally, they work, but are likely hitting different mirrors, or taking different routes.
I’ve seen some references to configuration like:
echo 'Acquire::Retries "5";' > /etc/apt/apt.conf.d/80-retries
… but it seems like something is quite broken if it takes 5 retries to download a file.