Receiving error when checking out code

Feedback & Bug Reports
,
1

We have been having a problem since last Friday where some of our branches (including master!) will fail when checking out code from Github. This is the associated output from the checkout:

Warning: Permanently added the RSA host key for IP address '192.30.253.112' to the list of known hosts.

error: Could not read b508ee672a30dee91a7aacd8107d6efd8e557e8d
error: Could not read 79eb3be2352d871184c8888a46bd71aaa60d4aa9
error: Could not read 405f6886b3234ab02d76f1429f6396a15692a9d7
error: Could not read cfe3b1aa034c2c2f558f5ef657ad4809059d0e12
error: Could not read d01a8202bfafbd41e6b3f73cd25e273d73855668
error: Could not read f94851160b4bbb7cfa1ff741564555c9c35ccb3f
error: Could not read 16d3ef2357b0d9021a0ab2708d6f07efaf413a9a
error: Could not read 84c1169129394a4324a45a1bb4629d74cde8ac93
error: Could not read 25a67d3a685b3e99e1a8588a1390a62401a9478b
error: Could not read f43b5b530bc85afa46f2be6970505bf2d5358b4f
error: Could not read a4ca3435c3e81a79401fa625ed4a28c3793509cf
error: Could not read 0e4e433a392d07cd5fac7ed8dc777a6e19cb8e76
error: Could not read 039f0d3c0f1612154e35e1e8f069fc6452e61f61
error: Could not read 2f3233943d42c2481e02676f7a342c52417c0779
error: Could not read 3f2e8acd37c408501bcb944868b1ee4707d3812d
error: Could not read f9fb6ac7729d4be8c73dad241a04f99466c34da9
error: Could not read 46fd0394d8fa91b4f8ff7496b85766df8fed3188
remote: Enumerating objects: 165, done.        
remote: Counting objects: 100% (165/165), done.        
remote: Compressing objects: 100% (43/43), done.        
remote: Total 209 (delta 142), reused 128 (delta 122), pack-reused 44        
Receiving objects: 100% (209/209), 204.09 KiB | 0 bytes/s, done.
Resolving deltas:  99% (149/150), completed with 73 local objects.
fatal: pack has 1 unresolved delta
fatal: index-pack failed
Exited with code 128

Once this starts happening on a branch, it never stops. It happens for all the jobs in the workflow.

We’re using 2.1 and I created a command for this - which has worked great for a week or more. I don’t think the command is the problem.:

  checkout_project:
    description: 'Checks out the project.'
    steps:
      - restore_cache:
          name: Restore Source Cache
          keys:
            - promote-source-v1-{{ .Branch }}-{{ .Revision }}-{{ .Environment.CIRCLE_JOB }}-{{ .Environment.CIRCLE_NODE_INDEX }}
            - promote-source-v1-{{ .Branch }}-{{ .Revision }}-{{ .Environment.CIRCLE_JOB }}-
            - promote-source-v1-{{ .Branch }}-{{ .Revision }}-
            - promote-source-v1-{{ .Branch }}-
            - promote-source-v1-
      - checkout
      - save_cache:
          name: Save Source Cache
          key: promote-source-v1-{{ .Branch }}-{{ .Revision }}-{{ .Environment.CIRCLE_JOB }}-{{ .Environment.CIRCLE_NODE_INDEX }}
          paths:
            - .git

It’s also worth noting that we upgraded to the performance plan Thursday evening. This is right around the time that we first started having this problem.

I’ve reached out to support on Friday, but the only response I’ve received is, “Thanks for getting in touch. We have had a few tickets about this issue. I have filed a bug report escalating this to our product team. We will provide you with an update as soon as you have more information.”

This is becoming a real problem for us! Any help or information on this problem would be appreciated.

2

As a note, this is the output from restoring the source cache:

No cache is found for key: promote-source-v1-master-bd7e5be98ef3520b54d37e787193e5856784d6ef-spec_slow-0
No cache is found for key: promote-source-v1-master-bd7e5be98ef3520b54d37e787193e5856784d6ef-spec_slow-
No cache is found for key: promote-source-v1-master-bd7e5be98ef3520b54d37e787193e5856784d6ef-
Found a cache from build 55143 at promote-source-v1-master-
Size: 664 MB
Cached paths:
  * /home/circleci/AdWerx/promote/.git

Downloading cache archive...
Unarchiving cache...

I’m trying changing the source cache name to see if that resolves the issue.

3

I resolved this problem by renaming my cache. This shows that the cache itself was the problem. Perhaps it was somehow corrupted? Is there a way we can clear the cache without requiring an update to our config.yml?

4

This problem is not resolved. It came back on a new branch. I’ve tried removing the promote-source-v1- key when restoring the cache, since all of our failures were when the cache was hit at this level. Our builds are running again now, but I don’t have a lot of confidence that this really fixed the problem.

5

Have a look at the Bash script the Checkout step runs (open the step in the job run UI). Perhaps you can glean some clues about the issue from attempting to run it locally?

6

I would doubt that. Checkout does not use the cache, as far as I know, so if you get repo problems with Git, then it more likely to do with funky Git hard resets, or perhaps there is a file system issue on the build server.