I have a pretty simple build process for a Ruby on Rails project that’s running rspec (bundle exec rspec --require spec_helper --format RspecJunitFormatter --out /tmp/circle-tests/rspec/rspec.xml --format progress $(circleci tests glob “spec/**/*_spec.rb” | circleci tests split --split-by=timings) and it just started failing last Friday morning with a message that rspec ‘Too long with no output (exceeded 10m0s)’.
I figured someone committed a bad test or something like, but after investigating I found that wasn’t the case and even builds for branches that ran perfectly fine the previous day or two were now failing with this same issue - even after rerunning without cache. I ssh’ed into the machine and tail’ed the test.log and found that up until the exact point when the circle env killed the rspec process our tests were running and not hung on something. I then ran the build with ssh and ran the tests using the same seed and command that the build runs and that completed without issue too and generated all the expected rspec output (.ie a green period is output to the console per completed test case).
So now I’m wondering why all of a sudden our tests are getting killed at exactly the 10 minute mark when they’e still progressing through the test “script” and appear to actually still be generating output (as the console output from the build summary has thousands of periods in it - we have around 10K tests) when prior to Friday everything worked fine. As mentioned above, this happens for every builds, even those that I re-run that previously ran to completion without error.
I’m kind of stumped where to begin here as nothing appears to be wrong, yet the builds are getting killed. I can modify the timeout on the run command as a workaround, but I’d like to eventually set it back if I can figure out the issue.