We are testing out a new default method to checkout code (blobless clones) which aims to speed up checking out source code for all users who use our built-in checkout command.
If you experience issues with cloning your repo when using the built-in checkout step, please comment on this thread or email oran@circleci.com.
My team’s builds started mysteriously and consistently failing today, with no obvious code-related or config-related cause. This is the only thing in CircleCI’s changelog and the only reason I suspect it to be the cause is that the timing matches up.
@abbyhowell can you let me know your org name and project name and send me a job link of a failure as well please?
You can send to oran@circleci.com and we can investigate/make sure you’re opted out until we figure out the root cause.
I’m seeing a similar issue where jobs have started randomly failing around a step that uses git diff. I assume it’s caused by this change since parts of commit history would be missing.
Is there a way to disable this option in config.yml to verify it’s the cause?
@davidalbers can you send me an email at oran@circleci.com with a job link please and we will opt you out while we investigate
Questions - why blobless and why change the default action?
I’m going to be in the minority as I use self-hosted runners, but this allows me to persist a copy of the git repo on the runner host with the result that the full git clone performed by the checkout command is an update rather than a full checkout for each future job run. As none of my repos have blobs the change does not impact my current usage, but for the future it would seem better to be able to choose how checkout operates, rather than just having a hardcoded selection.
If I were using the standard runner environment with a large repo I would likely be asking why you have not gone with a treeless clone as the local repos produced would be deleted at the end of the job anyway.
As a side note.
It has been a long time since I have had to worry about the checkout step, but in the past when trying to understand what actions were being performed it was very easy as the executed shell script was included in the output of the checkout step shown in the GUI.
At some point, this must have been dropped, but I am not aware of any other way to see what actions the script is performing. Obscuring such information may make the output look cleaner, but it does not do anything to help understand what is going on if there is an issue.
Looking at past logs this detail was lost in the last 2 months.
We have noticed that sonar cloud steps are failing in our builds while using blobless clones. This is a huger blocker for us as sonar is a gatekeeper for getting our PRs merged across projects.
Sonar community is starting to be aware of this issue, but they do not have any permanent solutions yet.
How can we revert to old checkout methods?
If you look at the step output from any workflow that is older than about 3 months you will be able to see the script run for the checkout step and so duplicate it.
I do not know why, but the latest update to the checkout step has also removed this detail as I noted in a post a few weeks ago.
Hi @brettjohnsen-sy,
I can help out with disabling blobless checkout for you. Send the name of your organization and the project(s) you’d like us to turn this off for to oran@circleci.com. Thanks!
We are also experiencing blocking issues related to Sonar as described above by brettjohnsen-sy.
[ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.10.0.2594:sonar (default-cli) on project xxx: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: org.eclipse.jgit.errors.MissingObjectException: Missing blob 95ec47e800000000000009ab539724 -> [Help 1]
Can this be turned off for our org?
@sebastian-lerner In my organization, few repositories are checking-out using blobless clone, what would be the reason for that ? blobless clone should apply to all repositories in organization right ?
We also are experiencing the same problem. Our SonarScanner step has started failing consistently complaining about a “Missing blob”.
Please, could you provide a solution to this problem in your documentation? Or through a parameter of the checkout command?
Hi Sebastian,
I can help out with disabling blobless checkout for you. Send the name of your organization and the project(s) you’d like us to turn this off for to oran@circleci.com. Thanks!
Hi Roberto,
I can help out with disabling blobless checkout for you. Send the name of your organization and the project(s) you’d like us to turn this off for to oran@circleci.com. Thanks!
Hi,
thanks for the offer but we just want to disable it for a single job, not for a project.
In the end we fixed it by re-fetching all blobs after the checkout command.
I really thank you for offering help but my suggestion was more general. Instead of an ad-hoc solution, I would suggest to provide a parameter to the command that allows enabling/disabling that behaviour.
Kind regards,
Roberto Mosca
Ah, I understand now. Thanks for the context. We’re in the process of investigating better ways to enable/disable this feature now - we’ll update the thread when there’s something new to report.
Hey ! We experience issue using sentry/cli to detect git commit tree since you release the new checkout version. The sentry/cli is not able to parse and detect all commits from latest deployment now…
Is it possible to rollback our org to the previous version? It’s a huge blocker for us…
It could be good to have option parameters to setup this “Speeding up code checkout” mode and make the previous one as the default.
As noted in a past comment you will need to email oran@circleci.com. so that changes can be made to your account settings.
Thank you all again for your feedback! In our search for balancing the gains of blobless checkout against the challenges identified so far, we think we have a solution and would appreciate your feedback.
If you are not opted out of blobless yet, consider adding a fetch: https://circleci.com/docs/configuration-reference/#blobless-clones. If this doesn’t address your needs, let us know so we can continue iterating.
If you have been opted out and would like to opt back in to test this, let me know oran@circleci.com.