Intermittent connectivity issues to apt/snap repositories from remote docker engines and machine executors?

Over the past several days I’ve seen issues where connectivity to the apt/snap repositories has been intermittently very slow, enough so that my builds fail with no output timeouts.

I’ve only seen this when installing apt packages in a docker build step against a remote docker engine and from a machine executor when installing the microk8s snap package.

Is anyone else seeing this? Or does anyone have any ideas about how to debug it?

Hi Tim!

Is this with any particular endpoint(s)? Is this affecting archive.ubuntu.com? If so could you please try this mirror and see if you see any improvement? This mirror is in the same AWS region as the majority of our Docker infra.

http://us-east-1.ec2.archive.ubuntu.com/

You can switch to this mirror using the following before running any apt commands:

RUN sh -c "sed -i 's|http://archive|http://us-east-1.ec2.archive|g' /etc/apt/sources.list"

Please let me know how you get on there!

Cheers,
Al

Thanks for the response! I’ve added two tests to my periodic job that I have running and tracking this behavior and I’ll report back on whether they work consistently or not.

Welp we lucked out here. Just as I put this into testing we’ve gotten a spate of failures in the non-ec2 archive runs. So far the only builds that are failing are the non-edited sources. All of the ec2 archive builds both in and out of docker build are running fine and fast.

I guess for now I can move forward with this workaround although I don’t love that it’s an edit forced on me by the build context rather than a technical reason.