Git tag triggers a pipeline only the first time

[fjinkis]

Hi folks! I’m having troubles to implement some kind of pipeline when a pre-existing tag is already “linked” to a commit.

First of all, this is my config (let’s imagine for now that those scripts are just an ‘echo’ saying main/staging/prod):

version: 2.1

jobs:
  main:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run: "scripts/main.sh"
  staging:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run: "scripts/staging.sh"
  prod:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run: "scripts/prod.sh"

workflows:
  pointers:
    jobs:
      - prod:
          filters:
            tags:
              only: PROD
            branches:
              ignore: /.*/

  developing:
    jobs:
      - main:
          filters:
            branches:
              only: main
      - staging:
          filters:
            branches:
              only: staging

I’ve already read about this topic, however, I think that my situation is slight different because I push the pre-existing tag to another commit.

So… enough words, let me show you what I’m saying:

1. I create a new branch

$ git branch test/circleci origin/main 
Branch 'test/circleci' set up to track remote branch 'main' from 'origin'.

$ git checkout test/circleci 
Switched to branch 'test/circleci'
Your branch is up to date with 'origin/main'.

$ git add . && git commit -m "Update docs"
[test/circleci ebac2e1] Update docs
 1 file changed, 3 insertions(+), 1 deletion(-)

$ git push origin test/circleci --set-upstream 
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 4 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 295 bytes | 295.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
remote: 
remote: Create a pull request for 'test/circleci' on GitHub by visiting:
remote:      https://github.com/fjinkis/git/pull/new/test/circleci
remote: 
To github.com:fjinkis/git.git
 * [new branch]      test/circleci -> test/circleci
Branch 'test/circleci' set up to track remote branch 'test/circleci' from 'origin'.

2. I merge that branch to staging using GitHub and that triggers the workflow ‘developing’ and executes the script ‘staging.sh’

3. After that, I merge from staging to main, the workflow ‘developing’ runs and the script pushes the PROD tag to the origin/HEAD

4. Because of the previous step, pushing the PROD tag triggers the workflow ‘pointers’ that executes ‘prod.sh’ (which is fine!)

5. I decide to move PROD tag to other commit (or tag, it’s the same) but that action doesn’t trigger the workflow

$ git fetch --all --prune
remote: Enumerating objects: 2, done.
remote: Counting objects: 100% (2/2), done.
remote: Compressing objects: 100% (2/2), done.
Unpacking objects: 100% (2/2), 1.19 KiB | 1.19 MiB/s, done.
remote: Total 2 (delta 0), reused 0 (delta 0), pack-reused 0
From github.com:fjinkis/git
   cf0f0d3..9350b43  main       -> origin/main
   8e0997d..a46d256  staging    -> origin/staging
 * [new tag]         PROD       -> PROD
 * [new tag]         v1.0.4     -> v1.0.4

$ git pull
Updating cf0f0d3..9350b43
Fast-forward
 README.md | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

$ git tag -d PROD
Deleted tag 'PROD' (was 9350b43)

$ git push --delete origin PROD
To github.com:fjinkis/git.git
 - [deleted]         PROD

$ git tag PROD v1.0.2

$ git push origin PROD
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
To github.com:fjinkis/git.git
 * [new tag]         PROD -> PROD

6. I create a new branch and merge it directly to main to check if the ‘pointers’ workflow runs

$ git branch --merged 
* main
  staging
  test/circleci

$ git branch -d test/circleci 
Deleted branch test/circleci (was ebac2e1).

$ git branch test/circleci-retry origin/main 
Branch 'test/circleci-retry' set up to track remote branch 'main' from 'origin'.

$ git checkout test/circleci-retry 
Switched to branch 'test/circleci-retry'
Your branch is up to date with 'origin/main'.

$ git add . && git commit -m "Update docs"
[test/circleci-retry 9b9dc70] Update docs
 1 file changed, 2 deletions(-)

$ git push origin test/circleci-retry --set-upstream 
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 4 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 266 bytes | 266.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
remote: 
remote: Create a pull request for 'test/circleci-retry' on GitHub by visiting:
remote:      https://github.com/fjinkis/git/pull/new/test/circleci-retry
remote: 
To github.com:fjinkis/git.git
 * [new branch]      test/circleci-retry -> test/circleci-retry
Branch 'test/circleci-retry' set up to track remote branch 'test/circleci-retry' from 'origin'.

Tadaaaa! The PROD tag triggered the workflow again! So my question is, what is the difference in this case? Because the main.sh does exactly the same than me:

TAG=$(git tag -l | tail -n1)
git tag -d $TAG && git push --delete origin $TAG
git tag -d PROD && git push --delete origin PROD
git tag $TAG origin/main && git tag PROD origin/main && git push --tags

I hope these screenshots and terminal snippets help to you all!
Let me know what you think and thanks in advance!!

[rit1010]

Welcome to ‘fun with tags’ a game to drive you mad over time.

CircleCI’s workflows fire based on the info passed to them via the webhook git server, so the issue is caused by the way that tags move around within git, rather than anything within CircleCI. You can see this in the GUI, where the Branch/Commit column shows all the tags that CircleCI can find.

What I can tell you is

• Your last section with the main.sh script does one key thing differently - it has “git push --tags”. This pushes all tags and so is not recommended as any only junk can end up in the central repo.

• tags cause references within the repo and while they can be reported as errors I’ve seen comments that indicate that such messages can be configured out so it is best to use --force. A good write-up on the steps seems to be this.

  https://stackabuse.com/git-push-tags-to-a-remote-repo/
  

The solution we currently have is to use unique tags - which for me as the one and only member of the ops team is to use a tag of <>, developers are currently using a unique version number for each deployment.

[fjinkis]

Thank you @rit1010 for giving your opinion! Regarding the --tags flag, you’re absolutely right, I modified that a couple of days ago.

After many days testing and reading, I think that my situation is not as different as the topic mentioned before. I’m pretty sure that the step (5) doesn’t work because PROD is referring again to db9bb2e. Due that, I was wondering if there is a way to force Circle CI to run the pipeline, whether the tag refers to a new or an old commit, something similar to [skip ci] but this would be like the reverse case

[rit1010]

After some more testing it seems that circleci has some basic loop detection hardcoded into its process where jobs are not considered worth tracking/running if the combination of commit ID and tag is repeated.

The workaround is to create a new commit with something like the following (depending on your setup)

git commit --allow-empty -m "create a new commit"
git push origin

<add back the tag>

This type of restriction does make sense from a CI workflow point of view as a workflow could be adding and removing tags as it executes jobs, as a loop could become rather costly, but there should be far clearer documentation about such a restriction.

[fjinkis]

Yeah! I totally understand about the infinite loop that you might create, however, if you build your pipeline properly, you have much more flexibility and power!

Besides the theory, I wasn’t able to replicate your workaround and I’m not completely sure if I understood what you said. In my attempt, it doesn’t matter if I create many (empty) commits ahead of the main branch, as long as I keep using an existing tag referring to a commit that has already referred, the pipeline won’t run.

Anyway, I’ve been testing an alternative trigger by using the API, but it makes the process a bit less natural

[rit1010]

I forgot to add that my backend is bitbucket rather than github, so that may change the result.

The aim of those 2 lines of code was to cause the repo to end up with a new commit ID without any code changes that the tag could then be added to.

[rit1010]

The attached image should show things better

So, on commit 5e4957c you can see that circleci only saw each tag being added once, regardless of how many times I made changes via the cli.

I then created a new commit f8c8cbe

I was then able to remove any reference to the tags ‘hello’ and ‘new-tag1’ from 5e4957c and then re-add them to f8c8cbe, with circleci registering the fact as a new pipeline entry.

[fjinkis]

Thank you @rit1010 for all your help! You are so kind!

CircleCI - as we discussed before - identifies that the relation tag/commit is new and that’s why the pipeline is triggered. However, if you remove new-tag1 and then try to create it again referring to 5e4957c (like the pipeline 49) Circle won’t trigger any pipeline. I only was able to trigger it using the API (unfortunately, that’s not automatic though)