Workspace files intermittently missing from parallel runs

Build Environment
1

Hi All,

I have a set of test cases that is broken down into 10 parallel runs. Each of these is persisted to a workspace at the end of the run, and the workspace is then loaded into the last job to combine these results and report them.

As it stands, at the end of each parallel run, I do the following:

- persist_to_workspace:
          root: ~/
          paths:
            - integration-manager-data
            - cma-ui-automation/actortmp/out/screenshots
      - store_artifacts:
          path: ~/cma-ui-automation/actortmp
      - store_artifacts:
          path: ~/integration-manager-data

And when I view artifacts for each parallel run, this file exists, numbered in accordance with the parallel run that generated it:

  • ~/integration-manager-data/results/result{0-9}.json

However, I then load the workspace into the final job:

- attach_workspace:
          at: ~/

In an attempt to debug this issue, I persist the integration-manager-data directory to my build artifacts and note that result1.json is missing:

~/integration-manager-data/results/result0.json
~/integration-manager-data/results/result2.json
~/integration-manager-data/results/result3.json
~/integration-manager-data/results/result4.json
~/integration-manager-data/results/result5.json
~/integration-manager-data/results/result7.json
~/integration-manager-data/results/result8.json
~/integration-manager-data/results/result9.json

I’ve noticed that the longer a parallel run takes, the more likely it is that the result JSON does not appear in the workspace that’s transferred to the final job. It only happens for runs that take over an hour, and in the case of the most recent runs, these were the time taken for each:

  • 0: 28:35
  • 1: 1:33:06
  • 2: 54:18
  • 3: 55:37
  • 4: 59:34
  • 5: 53:58
  • 6: 1:11:27
  • 7: 48:13
  • 8: 55:10
  • 9: 47:22

The configuration for these two jobs:

- regression-test-run:
          requires:
            - regression-test-split
          filters:
            branches:
              only:
                - develop
                - testing/debug-template-failure
      - regression-test-report:
          requires:
            - regression-test-run
          filters:
            branches:
              only:
                - develop
                - testing/debug-template-failure

Any insight on why this issue is happening and how I can resolve it would be much appreciated!