Punkdata is correct: you want to make use of workspaces here. I’ve done this a fair number of times with both disparate jobs and when using job parallelism and it works fine. The only issues is communicating successfully between jobs, so things get a little ugly.
You’ll want a few things:
- A separate job to gather test results. If you’re gathering results from jobs A, B, and C, you should probably name it
gather-A-B-C-results or something similar so people understand what it’s doing. This job should require A, B, and C in your workflow configuration.
- Your test steps should never fail, but instead output a failure marker to a file when they do. I would highly recommend the file be named after the job and only contain the
CIRCLE_BUILD_URL environment variable.
- A place in your workspace to store results. Something like
combined_test_results is pretty obvious off the top of my head, and you can just put your job there.
- Remember to persist the workspace in your test jobs, and remember to load it in your gathering job. Too many times I forget to do that.
- Your gathering job now needs to check for the existence of an A, B, and C file in the test result directory.
- If it finds any of those files, it should fail.
- Any jobs that require A, B, or C should now require your
gather-A-B-C-results task instead.
The advantage of doing this:
- Your message can include a link to each of the jobs that failed. This may be important because of disadvantage 2 below
- It’s another job to manage
- You don’t get any signal from the A, B, or C test jobs themselves. You have to go to the gathering job to figure out which one(s) failed
- You don’t get instant results. If A fails in 2 minutes but C fails in 20, you have to wait for the slowest to finish
- If your workspace is large, it can take a long time to initialize the gathering job
There is a way to do this without workspaces, but it is a hell of a hack and requires you store test results on each of those jobs. Similar to the above, do not fail your A, B, or C tests, and make a gathering job like you did before. Then with the help of the job name and workflow id of the gathering job (Using Environment Variables - CircleCI), query the API to find the jobs the gathering job depends on. Load up and parse the test results from all of those jobs, and then fail the gathering job if you encounter any
error statuses. I would not recommend this path; use the workspace instead.