Checkout fails on git-tags, when .git directory is already present

[floscher]

For one of my projects I cache the .git directory between builds, similar to https://circleci.com/docs/2.0/caching/#source-caching

Now I found out that the checkout step fails as soon as the build is triggered by pushing a git-tag to GitHub: https://circleci.com/gh/JOSM/wikipedia/37#action-103

This is caused by line 39 in the checkout.sh:

git fetch --force origin "refs/tags/${CIRCLE_TAG}"

This line fetches the commit associated with the tag, but does not create the tag in the local repository.

Because of that, later in the script the following line fails, because the tag does not exist:

git checkout -q "$CIRCLE_TAG"

You should change the line to the following:

git fetch --force origin "refs/tags/${CIRCLE_TAG}:refs/tags/${CIRCLE_TAG}"

(see also https://stackoverflow.com/a/45338695)

[halfer]

I wonder, would doing a shallow clone be easier/faster?

[floscher]

@halfer

I wonder, would doing a shallow clone be easier/faster?

In my case it’s probably not much of a speed difference (full clone is only about 15 MiB). In terms of how easy it is, i think nothing beats the checkout step, so I went with it.

However for some projects it would make a difference to no longer have a full clone in the CI, e.g. if they need the git-history to calculate their version number.

[floscher]

Any opinions on my suggested fix?

Note, this bug report is not really about caching the .git directory, that’s just what caused the bug for me.

It’s about the checkout step failing as soon as (a) the .git directory exists and (b) the build was triggered by a git tag. The line in the checkout step I referred to doesn’t do what it should do (i.e. create the remote tag locally). It only fetches the commit that is tagged.

[floscher]

This seems to describe the same issue:

Builds using git tags failing (x-post)

(x-post from 2.0 Support - not sure which category is more appropriate) I’ve been using git tags previously to build on CircleCI 2.0. Since about 2 weeks ago(?), it looks like a change to the checkout step is breaking builds. Previously, the last part of the script looked like this: if [ -n "$CIRCLE_BRANCH" ] then git reset --hard $CIRCLE_SHA1 git checkout -q -B $CIRCLE_BRANCH fi git reset --hard $CIRCLE_SHA1 The new code breaks tags: if [ -n "$CIRCLE_TAG" ] then git reset --hard "$C…

---

[lydiastepanek]

I also noticed the checkout step fails on jobs triggered by tags. I made the following change to my circle config file for the job I was having this issue for and this worked:

   deploy_job:
     steps:
-      - checkout
+      - run:
+          name: Checkout with tags
+          command: |
+            git fetch --force origin "refs/tags/${CIRCLE_TAG}:refs/tags/${CIRCLE_TAG}"
+            if [ -n "$CIRCLE_BRANCH" ]
+            then
+            git reset --hard $CIRCLE_SHA1
+            git checkout -q -B $CIRCLE_BRANCH
+            fi
+            git reset --hard $CIRCLE_SHA1

[floscher]

@lydiastepanek

I made the following change to my circle config file […]

Yes, while this works as a workaround, it is much more verbose. Fixing this in the checkout step itself would be much better. Then not every developer would need to create such a custom checkout step!

By the way: Here’s my workaround (even more verbose, because I clone with SSH and then need to put the SSH key into my config.yml): https://github.com/JOSM/wikipedia/blob/a100deeec7c0f5fd386c677e3f12f1ebe7fe9a98/.circleci/config.yml#L13-L22

[lydiastepanek]

Ah, I like your solution because it keeps the checkout step as-is (though I agree, in the long run I hope the checkout step itself is fixed). I’ve implemented something like your solution now. Thanks for the help!

[mrianmerry]

Would be nice to see some CircleCI Employee input on this one; I’ve just run into the same issue without attempting to cache .git

Perhaps you could directly submit a PR to the CircleCI repository? (Although you’d have to find the right one…)

[floscher]

@mrianmerry

Would be nice to see some CircleCI Employee input on this one; I’ve just run into the same issue without attempting to cache .git

Perhaps you could directly submit a PR to the CircleCI repository? (Although you’d have to find the right one…)

I searched for where the code is, but wasn’t successful, so probably the code doesn’t want to be found . The source code of the checkout.sh script is printed in my build log, but I couldn’t find a repo for it.
I described the bug here, suggested a oneliner that fixes the issue, and won’t go to further lengths to see this fixed. If CircleCI employees want the fix, they can find it here, otherwise I’ll assume they are not interested and I’ll either work around it or use something else.

For future projects I’m probably #movingtogitlab anyway, so that won’t be a problem anymore…

[halfer]

@floscher

For future projects I’m probably #movingtogitlab

I like GitLab (I use them for my image registry) but I’ve heard of sporadic reports that they’re struggling to cope with the new demand caused by GitHub users fleeing the Redmondster, and the service is slow as a result. That may just affect the free tier though, so YMMV, as with all things.

Of course, if you have the budget/time, you can self-host with GitLab.

[aiusepsi]

I’ve just hit this too. Thanks for the solutions!

[floscher]

@CircleCI team: Do you have any opinion on the error in checkout.sh reported here and the proposed fix?

The line git fetch --force origin "refs/tags/${CIRCLE_TAG}" clearly does not do what the author of that line intended.
I offered a simple change to git fetch --force origin "refs/tags/${CIRCLE_TAG}:refs/tags/${CIRCLE_TAG}", which would fix this issue.

Yes, we have a workaround (actually two of them). Yes, we could do a shallow clone instead. But nevertheless the default checkout job is broken in some cases and should be fixed to work out of the box.