Apt-get outage

kj-brown · 2017-10-18 14:50:56 UTC

Seem to be getting a lot of outages when trying to change Ruby Version using RVM:

++ printf %b 'There has been error while updating '\''apt-get'\'', please give it some time and try again later.
404 errors should be fixed for rvm to proceed. Check your sources configured in:
    /etc/apt/sources.list
    /etc/apt/sources.list.d/*.list
\n'
There has been error while updating 'apt-get', please give it some time and try again later.
404 errors should be fixed for rvm to proceed. Check your sources configured in:
    /etc/apt/sources.list
    /etc/apt/sources.list.d/*.list

I expect this is a temporary outage, but just making sure this is being addressed.

johncalvinyoung · 2017-10-18 15:05:46 UTC

We’re completely blocked as well. RVM is blowing up attempting to install 2.3.5 with the error above. CircleCI 1.0.

Note: rebuilding without cache did not help. It looks like one of the apt-get sources is down for some reason.

SSHing into the build and manually sudo apt-get update yields the following errors:

W: GPG error: http://repo.mysql.com trusty InRelease: The following signatures were invalid: KEYEXPIRED <int> KEYEXPIRED <int> KEYEXPIRED <int>
W: Size of file /var/lib/apt/lists/repo.mysql.com_apt_ubuntu_dists_trusty_mysql-5.7_source_Sources.gz is not what the server reported <int> <int>
W: Size of file /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_binary-amd64_Packages.gz is not what the server reported <int> <int>
W: Size of file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_binary-amd64_Packages.gz is not what the server reported <int> <int>
W: Size of file /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiverse_binary-amd64_Packages.gz is not what the server reported <int> <int>
W: Size of file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_binary-amd64_Packages.gz is not what the server reported <int> <int>
W: Size of file /var/lib/apt/lists/ppa.launchpad.net_openjdk-r_ppa_ubuntu_dists_trusty_main_binary-amd64_Packages.gz is not what the server reported <int> <int>
W: Failed to fetch https://packagecloud.io/basho/riak/ubuntu/dists/trusty/main/source/Sources  HttpError401

W: Failed to fetch https://packagecloud.io/basho/riak/ubuntu/dists/trusty/main/binary-amd64/Packages  HttpError401

jruijgers · 2017-10-18 15:11:47 UTC

We get the same error:

Err https://packagecloud.io trusty/main Sources
  HttpError401

Err https://packagecloud.io trusty/main amd64 Packages
  HttpError401

Fetched 14.2 MB in 4s (2,986 kB/s)
W: GPG error: http://repo.mysql.com trusty InRelease: The following signatures were invalid: KEYEXPIRED 1487236823 KEYEXPIRED 1487236823 KEYEXPIRED 1487236823
W: Size of file /var/lib/apt/lists/repo.mysql.com_apt_ubuntu_dists_trusty_mysql-5.7_source_Sources.gz is not what the server reported 862 866
W: Size of file /var/lib/apt/lists/repo.mysql.com_apt_ubuntu_dists_trusty_mysql-apt-config_binary-amd64_Packages.gz is not what the server reported 563 565
W: Size of file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_binary-amd64_Packages.gz is not what the server reported 21099 21216
W: Size of file /var/lib/apt/lists/ppa.launchpad.net_openjdk-r_ppa_ubuntu_dists_trusty_main_binary-amd64_Packages.gz is not what the server reported 4396 7564
W: Size of file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_binary-amd64_Packages.gz is not what the server reported 15541 15574
W: Failed to fetch https://packagecloud.io/basho/riak/ubuntu/dists/trusty/main/source/Sources  HttpError401

W: Failed to fetch https://packagecloud.io/basho/riak/ubuntu/dists/trusty/main/binary-amd64/Packages  HttpError401

E: Some index files failed to download. They have been ignored, or old ones used instead.

sudo apt-get update returned exit code 100

alChrono · 2017-10-18 15:19:31 UTC

We are seeing a similar issue

showing last 15 lines of /opt/circleci/.rvm/log/1508339740_ruby-2.3.0/update_system.log
++ case "${TERM:-dumb}" in
++ case "$1" in
++ [[ -t 2 ]]
++ return 1
++ printf %b 'There has been error while updating '\''apt-get'\'', please give it some time and try again later.
404 errors should be fixed for rvm to proceed. Check your sources configured in:
    /etc/apt/sources.list
    /etc/apt/sources.list.d/*.list
\n'
There has been error while updating 'apt-get', please give it some time and try again later.
404 errors should be fixed for rvm to proceed. Check your sources configured in:
    /etc/apt/sources.list
    /etc/apt/sources.list.d/*.list

++ return 100
Requirements installation failed with status: 100.```

mlvn · 2017-10-18 15:24:29 UTC

As additional information, only Ubuntu 14.04 (trusty) seems affected by this issue.

sudo apt-get update on Ubuntu 12.04 (precise) works fine.

Error log:

[...]
Get:32 https://packagecloud.io trusty Release

Ign https://packagecloud.io trusty Release

Err https://packagecloud.io trusty/main Sources
  HttpError401

Err https://packagecloud.io trusty/main amd64 Packages
  HttpError401

Fetched 3,491 kB in 6s (557 kB/s)
W: GPG error: http://repo.mysql.com trusty InRelease: The following signatures were invalid: KEYEXPIRED 1487236823 KEYEXPIRED 1487236823 KEYEXPIRED 1487236823
W: Size of file /var/lib/apt/lists/repo.mysql.com_apt_ubuntu_dists_trusty_mysql-5.7_source_Sources.gz is not what the server reported 862 866
W: Size of file /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_restricted_binary-amd64_Packages.gz is not what the server reported 17782 17859
W: Size of file /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_multiverse_binary-amd64_Packages.gz is not what the server reported 3999 4018
W: Size of file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_binary-amd64_Packages.gz is not what the server reported 21099 21216
W: Size of file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_binary-amd64_Packages.gz is not what the server reported 15541 15574
W: Size of file /var/lib/apt/lists/ppa.launchpad.net_openjdk-r_ppa_ubuntu_dists_trusty_main_binary-amd64_Packages.gz is not what the server reported 4396 7564
W: Failed to fetch https://packagecloud.io/basho/riak/ubuntu/dists/trusty/main/source/Sources  HttpError401

W: Failed to fetch https://packagecloud.io/basho/riak/ubuntu/dists/trusty/main/binary-amd64/Packages  HttpError401

E: Some index files failed to download. They have been ignored, or old ones used instead.

sudo apt-get update returned exit code 100

Action failed: sudo apt-get update

jruijgers · 2017-10-18 16:03:07 UTC

I just checked our latest successful build. MySQL has been failing for quite a while. The new errors are from packagecloud.id

Err https://packagecloud.io trusty/main Sources
  HttpError401

Err https://packagecloud.io trusty/main amd64 Packages
  HttpError401

johncalvinyoung · 2017-10-18 16:06:35 UTC

I can see no evidence that the basho/riak package server still exists…

https://packagecloud.io/basho/riak returns a 404 page.

I’ll note that on one of my own servers (outside CircleCI) on 14.04 I don’t get the Size of File errors on core Ubuntu Trusty package sources–communication/caching problem?

ahf90 · 2017-10-18 16:10:31 UTC

I am getting the same error as jruijgers and johncalvinyoung

It seems to be a problem with riak.

I explored two temporary fixes, which both work for me.

I removed the sudo apt-get update command from my config files. Proceed with caution–you’ll definitely want to re-enable this later when this is hopefully fixed.
To your pre section in your config file, add sudo rm /etc/apt/sources.list.d/basho_riak.list

johncalvinyoung · 2017-10-18 16:17:34 UTC

CircleCI just got back to me on my support ticket, claiming to have fixed the problem so builds would run. Seems to be good on my repos!

https://status.circleci.com/incidents/699cb00695gx

tom · 2017-10-18 17:01:46 UTC

Thanks for reporting this. The issue is now resolved. You can rebuild any failed builds that you would like to run.

system · 2018-06-18 11:35:14 UTC