Over the past couple of days we’re seeing a large number of build jobs fail with messages like:
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/libw/libwebp/libwebp6_0.6.1-2ubuntu0.20.04.2_amd64.deb Connection failed [IP: 185.125.190.36 80]
or:
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/s/systemd/systemd-sysv_249.11-0ubuntu3.9_amd64.deb Could not connect to archive.ubuntu.com:80 (91.189.91.38), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.91.39), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out
Sometimes a retry will fix it. Sometimes multiple retries.
Images where we’ve encountered these include cimg/node:16.13.2, cimg/node:16.18.1-browsers. I can see at least the newer one is using apt 2.4.8, and I understand apt 2.3.2 and newer already use 3 retries by default.
Does anyone have experience with this? It seems to mainly affect CircleCI jobs: when we try the same commands locally, they work, but are likely hitting different mirrors, or taking different routes.
We are seeing this today with urls like: http://archive.ubuntu.com/ubuntu/pool/universe/n/node-get-stream/node-get-stream_6.0.1-1_all.deb
The URLs resolve fine locally but are failing in our CircleCI builds. Archive domain on Ubuntu’s status page does not show any active issues at the time we are experiencing this. https://status.canonical.com/
It seems like a network issue? Also seeing what looks like 3 retries in our logs based on Ign prefix lines.
We’re also seeing this variant (403s rather than timeouts), preceded by a warning about a legacy key used by apt-key:
W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/python-roman/python3-roman_3.3-1_all.deb 403 Forbidden [IP: 185.125.190.82 80]
...
This is out of my wheelhouse but wanted to share the additional context in case it’s helpful.
Get:1 https://download.docker.com/linux/ubuntu jammy InRelease [48.8 kB]
Get:2 https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages [60.7 kB]
Get:3 https://packagecloud.io/github/git-lfs/ubuntu jammy InRelease [29.2 kB]
Ign:4 http://security.ubuntu.com/ubuntu jammy-security InRelease
Ign:5 http://archive.ubuntu.com/ubuntu jammy InRelease
Ign:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Ign:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Get:8 https://ppa.launchpadcontent.net/git-core/ppa/ubuntu jammy InRelease [24.6 kB]
Get:9 https://packagecloud.io/github/git-lfs/ubuntu jammy/main amd64 Packages [2,213 B]
Get:10 https://ppa.launchpadcontent.net/git-core/ppa/ubuntu jammy/main amd64 Packages [4,109 B]
Ign:4 http://security.ubuntu.com/ubuntu jammy-security InRelease
Ign:5 http://archive.ubuntu.com/ubuntu jammy InRelease
Ign:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Ign:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Ign:4 http://security.ubuntu.com/ubuntu jammy-security InRelease
Ign:5 http://archive.ubuntu.com/ubuntu jammy InRelease
Ign:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Ign:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Err:4 http://security.ubuntu.com/ubuntu jammy-security InRelease
Could not connect to security.ubuntu.com:80 (185.125.190.82). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (185.125.190.83). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (185.125.190.81). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (185.125.190.36). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (185.125.190.39). - connect (111: Connection refused)
Err:5 http://archive.ubuntu.com/ubuntu jammy InRelease
Could not connect to archive.ubuntu.com:80 (185.125.190.82). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (185.125.190.81). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (185.125.190.36). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (185.125.190.83). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (185.125.190.39). - connect (111: Connection refused)
Err:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Unable to connect to archive.ubuntu.com:http:
Err:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Unable to connect to archive.ubuntu.com:http:
Fetched 170 kB in 7s (22.9 kB/s)
Reading package lists... Done
W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease Could not connect to archive.ubuntu.com:80 (185.125.190.82). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (185.125.190.81). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (185.125.190.36). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (185.125.190.83). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (185.125.190.39). - connect (111: Connection refused)
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease Unable to connect to archive.ubuntu.com:http:
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease Unable to connect to archive.ubuntu.com:http:
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease Could not connect to security.ubuntu.com:80 (185.125.190.82). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (185.125.190.83). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (185.125.190.81). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (185.125.190.36). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (185.125.190.39). - connect (111: Connection refused)
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package libjpeg-dev
E: Unable to locate package libpng-dev
E: Unable to locate package libgl-dev
E: Unable to locate package libxi-dev
Exited with code exit status 100
We still had some instability when it picked a bad mirror from that US.txt mirrors file so we also used a retry function like the following:
function retry_command() {
local retries="$1"
local command="$2"
local options="$-" # Get the current "set" options
# Disable set -e
if [[ $options == *e* ]]; then
set +e
fi
# Run the command, and save the exit code
$command
local exit_code=$?
# restore initial options
if [[ $options == *e* ]]; then
set -e
fi
# If the exit code is non-zero (i.e. command failed), and we have not
# reached the maximum number of retries, run the command again
if [[ $exit_code -ne 0 && $retries -gt 0 ]]; then
retry_command $(($retries - 1)) "$command"
else
# Return the exit code from the command
return $exit_code
fi
}
It still happens for us off and on but the retries usually catch it. Sometimes one of the mirrors will hang and cause the “context deadline exceeded” error in circleci.
