My project has several expensive queries which reside in separate files within the src/sql folder of the code repo. For example:
│ ├── config.yml
│ └── update_queries.yml
│ ├── sql
│ ├── long-running-query1.sql
│ ├── long-running-query2.sql
│ ├── long-running-query3.sql
My goal is to reduce build cost and only run the query(ies) whose file(s) changed.
Below is the path-filtering excerpt from the CircleCI config.yml file:
src/sql/.* run-redeploy-queries-job true
base-revision: << pipeline.git.branch >>
Is there a way to access the exact names of the files which got matched by the mapping regex? I would ideally need to access those from the docker, which gets spun up by the matching workflow in update_queries.yml, so that I can trigger only that/those query(ies) which changed.
I would prefer a more general solution over the one where I would have a separate mapping for every file.
Any ideas or hints are welcome and thank you in advance.
You can set the mapping to specific files instead of the whole directory.
If you change your mapping to match as below it will only work if the specific file is updated
src/sql/long-running-query1.sql run-redeploy-queries-job true
src/sql/long-running-query2.sql run-redeploy-queries-job true
src/sql/long-running-query3.sql run-redeploy-queries-job true
If this does not work in the way you would like you could also move each into its own folder and define the mapping to each folder so that it will only trigger when the folder updates.
Hi Owen, I appreciate your reply. However, it is not what I was hoping for.
What I had in mind is described in this feature request in the path-filtering orb’s repo:
get exact names of files which changed.
My motivation is as described in the request (copy/pasting it here):
That is because to support such a solution would be difficult. For example, suppose a new file were added or a name change were introduced to an existing file. Then the continue config file, pointed to by
config-path, would need to be updated in either case. Furthermore, the need arises to document the update process and teach it to colleagues who would like to make a change, but have not yet seen the project. Finally, imagine you’ve dozens of files, changes in which need to be tracked. This means that the continue config file at
config-path would grow in size significantly and require more effort to maintain.