Run critical tests (subset) first and then the rest

Build Environment
1

My app is growing bigger and I have a lot of non-deterministic tests that will often fail because of unknown reasons (javascript synchronisation in cucumber fails sometimes)

I have a lot of test and I have tagged a subset of them as “critical”. I’d like to always run those first on circleci, with a high retry count (to avoid those nondeterministic failures in my report)

Then, when the critical tests all pass, I’d like to send a (custom) slack notification and continue running the rest of the tests. I am wondering how to set that up, and still make it work with Cucumber automatically generated reports.

My attempt so far

- >
        RAILS_ENV=test bundle exec cucumber
        -t @critical
        --retry 2
        --format junit
        --out $CIRCLE_TEST_REPORTS/cucumber/junit.xml
        &&
        RAILS_ENV=test bundle exec rake report_critical_test_status
        &&
        RAILS_ENV=test bundle exec cucumber
        -t ~@critical
        --format junit
        --out $CIRCLE_TEST_REPORTS/cucumber/junit.xml
2

Perhaps you can just write output into two separate junit files and then use a tool like junit merge to merge those on a last step and present to Circle as report artifact.

1 Like