Path-filtering orb: get names of files which changed

Build Environment
1

My project has several expensive queries which reside in separate files within the src/sql folder of the code repo. For example:

.
├── .circleci
│   ├── config.yml
│   └── update_queries.yml
├── src
│   ├── sql
│         ├── long-running-query1.sql
│         ├── long-running-query2.sql
│         ├── long-running-query3.sql

My goal is to reduce build cost and only run the query(ies) whose file(s) changed.

Below is the path-filtering excerpt from the CircleCI config.yml file:

      - path-filtering/filter:
          name: check-updated-files
          mapping: |
            src/sql/.* run-redeploy-queries-job true
          base-revision: << pipeline.git.branch >>
          config-path: .circleci/update_queries.yml

Is there a way to access the exact names of the files which got matched by the mapping regex? I would ideally need to access those from the docker, which gets spun up by the matching workflow in update_queries.yml, so that I can trigger only that/those query(ies) which changed.

I would prefer a more general solution over the one where I would have a separate mapping for every file.

Any ideas or hints are welcome and thank you in advance.

1 Like
2

Hello

You can set the mapping to specific files instead of the whole directory.

If you change your mapping to match as below it will only work if the specific file is updated

          mapping: |
            src/sql/long-running-query1.sql run-redeploy-queries-job true
            src/sql/long-running-query2.sql run-redeploy-queries-job true
            src/sql/long-running-query3.sql run-redeploy-queries-job true

If this does not work in the way you would like you could also move each into its own folder and define the mapping to each folder so that it will only trigger when the folder updates.

Kind Regards
Owen Oliver

3

Hi Owen, I appreciate your reply. However, it is not what I was hoping for.

What I had in mind is described in this feature request in the path-filtering orb’s repo:
get exact names of files which changed.

My motivation is as described in the request (copy/pasting it here):

That is because to support such a solution would be difficult. For example, suppose a new file were added or a name change were introduced to an existing file. Then the continue config file, pointed to by config-path, would need to be updated in either case. Furthermore, the need arises to document the update process and teach it to colleagues who would like to make a change, but have not yet seen the project. Finally, imagine you’ve dozens of files, changes in which need to be tracked. This means that the continue config file at config-path would grow in size significantly and require more effort to maintain.

Thanks!

1 Like
4

@galen did you ever find a solution to your question? I have the same issue and would like to figure out a way to know which files changed based on a RegEx directory pattern.

5

I hope that CircleCI is somewhere working on better first-class support for this. The path filtering orb does get the job done, but it’s kind of clunky.

Even in an org that’s not a full on monorepo, it can be pretty useful to be able to be able to filter builds and do other operations depending on the files that changed.

To the OP: I wonder if using git would be helpful at all – that is, compare the current revision to head or a previous revision, and use that to get the file list?