This came up in a support conversation (https://support.circleci.com/hc/en-us/requests/26028), and the support agent suggested I discuss it further here.
I’m trying to set up two parallel jobs that do a build, and then a deploy job which deploys the output of one of those jobs, for commits tagged with version numbers only. This broke because of caching issues: I set up the jobs to use different working directories but the exact same steps (so I could share them), and also tried to use caching of node_modules.
This doesn’t work because caching appears to work with absolute paths, rather than paths relative to the working directory. That meant my node 6 job (working directory ~/node-6) would be cached, and the node 8 job (working directory ~/node-6) would restore it at ~/node-6, while I’d expected it to be restored inside ~/node-8, using paths relative to its working directory. This then breaks things, because I persisted files to the workspace that conflicted.
The workspace model also generally feels it doesn’t really fit what I’m trying to do, but is the only option available.
What I want is to pull output from an upstream job into a subsequent job, and it’s odd that the intermediate output from all jobs needs to live in one space where jobs can step on each others toes. I’d much prefer each of my jobs to persist their workspace without any manual namespacing games, and my deploy job to just say ‘take the final state from job X’ (and that declarative model would also let you skip persisting workspaces entirely, if you know nobody downstream will use them - e.g. in all my non-tagged builds).
That’d match Docker’s multistage build approach too, for example, where one stage uses the full container state from a previous one, and can later pull in individual files from others, rather than all of them having to push output out into a single shared (potentially conflicting) store. This is also more or less what Travis’s model does, for their simpler case: they’ve just got a skip_cleanup option for deploys, which carries the pre-deploy output directly into the deploy step.
Hope that’s useful!