CircleCi too long with no output

Build Environment
1

Our System tests on CircleCi have been randomly failing for a long a long time with the following error

Too long with no output (exceeded 5m0s): context deadline exceeded

We’ve tried increasing the WEB_MAX_WAIT_TIME all the way to 20 mins as well as increasing the size of our resource_class from medium to large yet the error still occurs at random. Interestingly in all cases where this flaky behavior occurs the system test will stop using any resources for a varying period of time until the test fails.

Needless to say we have yet to encounter an issue similar to that locally, the test suite passes. This is also not a consistent error some days we have no failures others we need to re-run the tests 3+ times to achieve a green pipeline.

2

I should add that

Net::ReadTimeout:
Net::ReadTimeout with #TCPSocket:(closed)

shows up before the too long with no output error

In addition we have these settings in spec_helper.rb

 if ENV['CI']
  $stdout.sync = true
  $stderr.sync = true
end

EDIT: i seem to have increased in trust level so i can post pictures now

image
image1920×933 118 KB

This is an image of one of our pipelines failing with the too long no output, as you can see the pipeline just seems to hang all of a sudden spending no resources resulting in a timeout