Network timeouts


#1

I’ve been using CircleCI 2 happily for a few months now, it’s awesome. We are seeing occasion network timeout issues, and they seem to have become more frequent over the last week.

Here are some examples:

  • From apt-get:

Err http://packages.cloudfoundry.org/debian/ stable/main cf-cli amd64 6.24.0
Operation too slow. Less than 10 bytes/sec transferred the last 120 seconds

  • From docker-compose:

Traceback (most recent call last):
File “”, line 3, in
File “compose/cli/main.py”, line 65, in main
File “compose/cli/main.py”, line 117, in perform_command
File “compose/cli/main.py”, line 622, in push
File “compose/project.py”, line 447, in push
File “compose/service.py”, line 854, in push
File “compose/progress_stream.py”, line 18, in stream_output
File “compose/utils.py”, line 57, in split_buffer
File “compose/utils.py”, line 33, in stream_as_text
File “site-packages/docker/client.py”, line 247, in _stream_helper
File “site-packages/requests/packages/urllib3/response.py”, line 324, in read
File “contextlib.py”, line 35, in exit
File “site-packages/requests/packages/urllib3/response.py”, line 237, in _error_catcher
requests.packages.urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host=‘localhost’, port=None): Read timed out.
docker-compose returned -1

Normally a retry works, but it is a little annoying.

Has anyone else seen this?

Mark


#2

That’s concerning. Can you PM me a link to the build? Slow network speeds is definitely a concern. We want everyone to have the best uplink we can possibly provide.


#3

We also started observing network issues when moving from machine to docker executor. Timeouts occur at random places, both for connections between the containers in the remote docker environment and now also with the connection from the primary container to the docker server. This just happened during docker-compose up:

ERROR: for xxxx  HTTPSConnectionPool(host='35.190.151.223', port=2376): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

#4

Timeouts also occur when downloading external resources:

Pulling etcd (quay.io/coreos/etcd:v3.1.7)...
ERROR: Get https://quay.io/v2/coreos/etcd/manifests/v3.1.7: Get https://quay.io/v2/auth?scope=repository%3Acoreos%2Fetcd%3Apull&service=quay.io: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

#5

Some more timeouts today. Even when getting content from CloudFront:

ERROR: error pulling image configuration: Get https://dseasb33srnrn.cloudfront.net/registry-v2/docker/registry/v2/blobs/sha256/04/041a8ec2a7b50256c1cde9293271f3d5609939edd1ccb652ea570c1ae39900fa/data?Expires=1495188925&Signature=Gt0ym3zd53YCXPbtmdMD0QNXXRt~TiELq53O-kmAdsdGryPIYtfP2mqvdTcNuGOIwlQ6B-dNPyL1tVvAwTSQ-BwOfNDk3Hr~zMCG~zDdVGYvg1S-Ix449O-plXVuZSox5WY5Ww7uF7tinZ00v8Y8iRvL9pUOR~jTzQfRcoC2hik_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q: net/http: TLS handshake timeout


#6

We’ve been seeing TLS handshake timeouts when trying to publish Docker images to quay.io. This is for CircleCI 1.0.

docker push quay.io/fundingcircle/finops-admin

The push refers to a repository [quay.io/fundingcircle/finops-admin] (len: 2)

Post https://quay.io/v2/fundingcircle/finops-admin/blobs/uploads/: net/http: TLS handshake timeout

This occurs only on some builds, maybe 1/10 or 1/8 times. When it does occur, however, it’s a waste of a 20+ min. build.


#7