We have a Rails/PgSQL application. Running on an XL in series it can take up to 30’.
So, there is a branch that does parallel builds (4 x large). Using file size splitting.
In this branch, CI runtime ranges between 7.5’ and 28’ (which will fail due to a timeout). No code/config changes. Just continual reruns to collect more data points.
How can we make container performance more consistent?
I would suggest splitting on timing rather than file size.
You’ll need to make sure your test results are stored and include filename data
Use the CircleCI CLI to split tests - CircleCI has the basic idea.
roughly, something like:
circleci tests glob "test/**/*_test.rb" | circleci tests run --command="xargs bin/rails test --verbose" --split-by=timings
I’ve done that. It doesn’t make a difference. I flip between filesize and timing. Similar wild variance. This leads to intermittent failure on the overall CI run when one of the containers stalls out.
This feels like a noisy neighbor issue. The platform is under provisioned for the volume of containers being run.
Do different plans offer stronger isolation guarantees?