Is there a nice way to use “persist_to_workspace” conditionally based on the result of previous steps, or even more specifically based on the existence of the file I want to persist?
I am facing the issue that in some cases (decided dynamically in a shell script) we create a file. If that file was created, I want to persist it to workspace so that next job can upload it to S3.
The issue is that “persist_to_workspace” fails the build if the file is not found.
I can run it conditionally with “when” but that does not allow for “dynamic checks resolved only during previous steps” so to say, as far as I am aware.
I have added a step that calls “circleci-agent step halt” on the condition that the file does not exist before the “persist_to_workspace” but it feels like a hacky solution to the problem. Is there a better way?
There is no ‘tidy’ way to do such a thing as the condition statement when can only be expressed at the workflow, job or step level using parameter values that are known at the time the config.yml is first started. The exact text used by CircleCI is
Logic statements are evaluated to boolean values at configuration compilation time,
There is a workaround for some situations where the continuation orb can be used. This would allow a number of parameters to be defined when config.yml is executed and then passed on to your main code which is placed into a different yml file. For the case you have outlined this does not help much as you are trying to decide as your script is run.
This leaves circleci-agent step halt as the only option to dynamically stop a persist_to_workspace command being executed if you need to make real-time checks to decide on its execution.
For my own personal interest, what is the reasoning behind using persist_to_workspace if you are only planning to then copy the resulting workspace to an S3 store in the next job?
If you are using CircleCI’s provided executors each job is run in total isolation, so persist_to_workspace is the standard way to move persistent data between each job within the workflow. To remove the isolation you could look at a self-hosted runner as this can be used to execute all the jobs within a workflow on a common system/file store, but that is a very different process.
One thing that came to mind is that you could just always execute the persist_to_workspace step, but within your shell script, you could just remove all the files from within the persisted directory depending on your condition check. This would achieve what you are aiming for without having to complicate the flow. The shell script in the next job can then just perform its actions based on the fact that there are no files in the retrieved workspace, or you could use a named status file or env file to indicate what is going on.
The thing is that we only sometimes have the file I want to persist. So I cannot just always execute persist_to_workspace.
To be less abstract, the first job is an Android executor that sometimes builds an .apk file. If that file was built, I want to do some steps in in in the next job (namely upload it to S3 and trigger some automated tests to run against it in a different pipeline).
Since persist_to_workspace fails the android build job if it has not built the .apk and the condition for whether or not it is build is not something I can check with the when clause, I am left with this problem.
What you are suggesting if I understand correctly is something I have considered but it seems more confusing than the circleci-agent step halt - creating a “fake” .apk to be persisted and say a flag file is-apk-real with contents true or false. Then persist_to_workspace would always succeed.
This way the next job (the one uploading to S3), rather than just checking if the .apk exists, could check the contents of the “flag file”. But as I said, that seems even more convoluted than halting.
If I had the option to
specify only directory and persist all files within, not the absolute to the file and it would not fail if the directory was empty or
invoke persist_to_workspace from a run command (I checked the circleci cli but did not find that option)
I could avoid halting, but it seems like it is the best option right now then.
specify only directory and persist all files within, not the absolute to the file and it would not fail if the directory was empty or
You can do, below is a full example where persist_to_workspace is working on a directory, rather than a file. The only real complication is correctly setting the root point within the file system to allow this to happen. If you remove the “touch work/file” line this example should still work.
version: 2.1
orbs:
python: circleci/python@2.1.1
jobs:
test_persist_job:
docker:
- image: cimg/base:current
resource_class: small
steps:
- run:
name: create a working directory with a file for persist_to_workspace
command: |
mkdir work
touch work/file
pwd
ls
ls work
- persist_to_workspace:
root: ./
paths:
- "work"
test_attach_job:
docker:
- image: cimg/base:current
resource_class: small
steps:
- attach_workspace:
at: ./
- run:
name: display what was attached
command: |
pwd
ls
ls work
workflows:
test:
jobs:
- test_persist_job
- test_attach_job:
requires:
- test_persist_job
One thing to note is that any pattern matching is done using the rules defined by the Glob package found in the Go programming language
Oh, I didn’t realize that I can only specify the directory.
If that is the case, it seems like a nicer solution.
It seems to be working, the directory is not 100% empty right now but the syntax works with directory so I am assuming it won’t fail if I at least ensure it is there, even if empty with mkdir -p in previous step)