Currently, our workflow has around 15 jobs in it. There are data that needs to be passed from one job to another. This works well but some of our jobs only need a file or two from the workspace. With only one shared workspace, our builds are taking a long time to complete. I am aware that CircleCI does not support multiple workspaces in one workflow so I am wondering if it’s possible to attach only specific items from the workspace and not all of it.
Or is there a require command for workflows similar to jobs so I can split them up into smaller workflows?
At the moment this is not possible as when you attach the workspace it will attach the whole workspace. There is no option to attach a specific directory (or file) from the workspace to the job.
This is a great idea though, so I would encourage you to open a feature request for this over on our ideas board. Every vote and comment counts as we take all features into consideration when we plan future features!
You may be able to work around this by using caching instead, but this will likely add complexity to your workflow as you will need to use cache keys (that subsequent jobs will need to know) along with the fact that caches are immutable once created.
1 Like
Hi,
This would be really great.
I’m testing a node monorepo on multiple platforms (windows, linux & macos), with each platform generating a coverage report that is persisted to workspace. Each platform needs node_modules to be installed in order to run tests
Then, at the end of the tests, I have a single linux job that should gather reports from each platform and merge them in a single full report. Unfortunately, for this, I need to attach workspace, and boom, node_modules conflicts for each platform.
The only solutions I see is either:
- Generating all conflicting files in a single job: in my case, I then should run
npm install a single time, then use those node_modules on all platforms.
Unreliable since different platforms might result in different node_modules, eg node-gyped bindings or postinstall scripts
- Never persisting conflicting files in workspace: Each test run ( + lint + build ) should each run
npm install.
Super long & expensive. Moreover, caching npm install cache dir on windows is a pain and simply does not work (though it is an unrelated issue). In my case, it would add no less than 1min for install on every test run, build & lint on every tested platform for each node version tested. In total, it would add no less than 10min to the whole worflow run time. So, I won’t do that.
Lacking a working solution, I just stopped collecting coverage for non-linux.
FYI, those feature requests seems to be possible solutions:
One workaround is to push the files that you wish to consolidate back to a git repo, so you just use standard git commands to take the results and place them somewhere you can access them later.
For the one time, I have needed to do this so far I used CIRCLE_BUILD_NUM as the unique ID needed to tie the file together.
As bytesguy noted the caching feature should allow much the same type of storage, but full access to the git command is far more flexible and as I use self-hosted runners the git store was far nearer and quicker for my setup.