I’m running 3 E2E tests as parallel and my goal is to send notification to Slack when one of test is failed. I tried two approaches.
After e2e is finished, send notification → but, it’s impossible because job cannot be executed when required job(e2e test job) is failed.
When one of parallel tests is failed, send notification → duplicated notifications are sent, so each test should know whether one of tests already sent notification through environment variable.
To implement second method, I should be able to share environment variable between parallel tests(jobs). But, I couldn’t any clues in docs/discussion/support of CircleCI. How can I do? And, if this is impossible how can I do this in hacky way?
Example
Parallel Tests: A, B, C
When A is finished eariler than B and C, A set environment variable isTestFailed as true dynamically and sent Slack notification message. B and C were finished and checked isTestFailed is true. They didn’t send notification because isTestFailed is true and finish the test. Therefore, only one notification message is exist in slack.
jobs:
e2e_test: &e2e-test
parallelism: 3
steps:
...some steps...
check_whether_notification_was_sent:
1) check WAS_NOTIFICATION_SENT flag is true
2-1) If that flag is true, quit job.
2-2) If that flag is false, send notification to slack and set flag as true
The job e2e_test is parallel job and each parallel job runs check_whether_notification_was_sent step. This step checks boolean flag WAS_NOTIFICATION_SENT dynamically in runtime. This flag can be accessed by all parallel jobs. How can I do?
My apologies for the delay. Just got back from the holidays. I would suggest that you leverage the Workspaces features to move information between jobs in your worklflows. You could write the WAS_NOTIFICATION_SENT envar data to a file then attach the workspace. This should get you the results you’re seeking. Full transparency I have not tested this solution against your use case but seems like a reasonable solution. Let me know how you make out.
Punkdata is correct: you want to make use of workspaces here. I’ve done this a fair number of times with both disparate jobs and when using job parallelism and it works fine. The only issues is communicating successfully between jobs, so things get a little ugly.
You’ll want a few things:
A separate job to gather test results. If you’re gathering results from jobs A, B, and C, you should probably name it gather-A-B-C-results or something similar so people understand what it’s doing. This job should require A, B, and C in your workflow configuration.
Your test steps should never fail, but instead output a failure marker to a file when they do. I would highly recommend the file be named after the job and only contain the CIRCLE_BUILD_URL environment variable.
A place in your workspace to store results. Something like combined_test_results is pretty obvious off the top of my head, and you can just put your job there.
Remember to persist the workspace in your test jobs, and remember to load it in your gathering job. Too many times I forget to do that.
Your gathering job now needs to check for the existence of an A, B, and C file in the test result directory.
If it finds any of those files, it should fail.
Any jobs that require A, B, or C should now require your gather-A-B-C-results task instead.
The advantage of doing this:
Your message can include a link to each of the jobs that failed. This may be important because of disadvantage 2 below
The disadvantage:
It’s another job to manage
You don’t get any signal from the A, B, or C test jobs themselves. You have to go to the gathering job to figure out which one(s) failed
You don’t get instant results. If A fails in 2 minutes but C fails in 20, you have to wait for the slowest to finish
If your workspace is large, it can take a long time to initialize the gathering job
There is a way to do this without workspaces, but it is a hell of a hack and requires you store test results on each of those jobs. Similar to the above, do not fail your A, B, or C tests, and make a gathering job like you did before. Then with the help of the job name and workflow id of the gathering job (Using Environment Variables - CircleCI), query the API to find the jobs the gathering job depends on. Load up and parse the test results from all of those jobs, and then fail the gathering job if you encounter any failure or error statuses. I would not recommend this path; use the workspace instead.
