Github PRs show if the PR branch has conflicts with the base branch.
CircleCI shows test results on the PR branch.
What is missing is the ability to see whether test results fail when the PR branch is merged into the base branch.
Admittedly, it’s not everyday that PRs break tests of other PRs, but it does happen - for example when there’s a change in UI text:
- A feature developer issues PR A testing for some specific text in the UI: “The status is FOO”. PR A contains a view test checking for the existence of “The status is FOO”.
- A another developer issues PR B that implements a product decision to change all uses of “FOO” to “BAR”. Github says branch is clear to merge
- PR B is merged (Github says there are no conflicts; CircleCI says tests pass)
- PR A is merged, without problem (Github says there are no conflicts; CircleCI says tests pass)
- The base branch now has a test which fails because the application produces a view with the text “The status is BAR”, but the test in PR A expects “The status is FOO”.
All of the ways of dealing with this currently seem bad:
- Constantly merge into the PR branch, making a mess of the commit history
- Rebase the PR branch onto the base and force push onto Github (a no no for obvious reasons)
- Catch the test failures later in the base branch (inconsistent with gitflow and other branching strategies)
A feature would be nice, but a work around would be great too.