Apt-get update is really slow and eventually hang

[panthony]

Hello,

Creating a new issue because the following has been closed and I cannot comment on it anymore:

Machine executor still running apt when commands execute

For over a year now I’ve had a job that looks like the following: version: 2 jobs: my-job: working_directory: ~/foo machine: true steps: - checkout - run: command: | sudo apt update -qq sudo apt install -y <stuff I want to install> Fairly recently, these jobs have all been failing with the following error: E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable) E: Unable to lock directory /var/lib/…

For several days I had a similar issue and it got worse and worse and now it more or less doesn’t work at all anymore.

I get the error:

sudo apt-get update
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/
E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it? Exited with code 100

I tried an older classic image, I tried ubuntu-1604:201903-01, I tried a sleep, nothing works.

If I SSH on the machine:

circleci@default-8ef29195-a002-4bfe-9265-677069f25164:~$ ps aux | grep "apt-get"
circlec+  2006  0.0  0.0  11224  3212 ?        Ss   13:22   0:00 /bin/bash -l -c  echo '*               soft    nofile          65536' | sudo tee -a /etc/security/limits.conf; echo '*               hard    nofile          65536' | sudo tee -a /etc/security/limits.conf; sudo sed -i.bak -e 's|http://us-east1.gce.archive.ubuntu.com/ubuntu/|http://archive.ubuntu.com/ubuntu/|g' /etc/apt/sources.list; sudo rm -rf /etc/apt/sources.list.d/mysql.list;  # CIRCLE-15840, CIRCLE-16707 sudo apt-get -o Acquire::http::AllowRedirect=false update; sudo apt-get -o Acquire::http::AllowRedirect=false install -y apt openssh-client;  true
root      2019  0.0  0.0  68012  4160 ?        S    13:22   0:00 sudo apt-get -o Acquire::http::AllowRedirect=false update
root      2020  0.0  0.0  31208  5700 ?        S    13:22   0:00 apt-get -o Acquire::http::AllowRedirect=false update

circleci@default-8ef29195-a002-4bfe-9265-677069f25164:~$ sudo fuser /var/lib/apt/lists/lock
/var/lib/apt/lists/lock:  2020

If I kill the process and try the apt-get update myself it gets really REALLY slow here (like ~ 10Kb/s):

Get:3 http://archive.ubuntu.com trusty/universe amd64 Packages [5,859 kB]
99% [3 Packages 5,163 kB/5,859 kB 88%] [Connecting to security.ubuntu.com (91.1

Then it get stuck on Connecting to archive.ubuntu.com (91.189.88.162).

What can be done to workaround this issue?

[panthony]

Actually it eventually end:

Ign http://archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://archive.ubuntu.com trusty/universe Translation-en_US
Fetched 11.0 MB in 6min 57s (26.3 kB/s)
Reading package lists… Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packagecloud.io trusty InRelease: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY 4E6910DFCB68C9CD
W: Failed to fetch https://packagecloud.io/circleci/trusty/ubuntu/dists/trusty/InRelease
W: Some index files failed to download. They have been ignored, or old ones used instead.

[deinok]

same here. Looks like im not the only one with this problem

[Pavel]

Same here.
Also, I’ve tried to switch from circleci/classic:201710-01 to ubuntu-1604:201903-01 machine image but it still tries to fetch Trusty packages for some reason…

[FelicianoTech]

The Ubuntu repositories are currently having issues. Especially the security repo which is the one most people won’t mirror.

This will likely be an issue until they fix it upstream. In the past, usually no more than a few hours.

[halfer]

Already raised on the forum:

https://discuss.circleci.com/t/anyone-finding-build-speeds-to-be-slow-at-present/29457/3

@panthony: the issue you link to is a different problem. The Machine executor can get itself into a pickle, and not be able to update, unless you run a loop to wait for it to unlock. The current issue is much more likely to be that the Ubuntu server is barely functional at present.

[panthony]

@FelicianoTech Thanks for the quick response, I guess I’ll have to be patient then

[halfer]

Working fix on the other thread.