My company is now experiencing problems with the new v2 container runtime on specific projects. We are receiving “Received “killed” signal” errors regularly on multiple jobs. RAM usage is not even close to 100% according to the Resources tab but CPU hits 100% on all the jobs that fail. Is there a way to opt out of this until it’s fixed? I’m honestly surprised this wasn’t an “opt in” change considering the likelihood of problems.
Hi @sam-kemp,
Sorry that you’re having issues.
Would you be able send me a link to one or more of the problematic Jobs via DM? From that I’ll be able to opt your org out whilst we investigate the issue.
No, not this thread; I’m referring to the “nproc reports wrong number of CPUs” bug report (which was logged as an RFE) at Have `nproc` accurately reporter number of CPUs available to container | Feature Requests | CircleCI Ideas
Hi @swarajrao,
Just to let you know we have opted your org back in to the V2 runtime rollout as we believe the issue you were seeing has been resolved. If you continue to see issues then please let us know so that we can opt you back out and investigate accordingly.
Many thanks,
Dom
Hey @DominicLavery, we are also seeing a build failure related to v2 container runtime. We are using i386/debian for our build image, and based on the error, it looks like i386/debian cannot run on amd64. We are unable to use an amd based debian image since our project needs to be 32bit.
We are requesting to opt out of V2 for this pipeline.
Thanks,
Ryan
@DomParfitt @DominicLavery Could you take a look? It seems our org is still not properly opted out.
Hi @wilbur,
Sorry you’re having issues.
Would you be able to send me a link to one of your Jobs via DM and I’ll opt you out whilst we investigate? Could you also confirm if it’s just a singular project that needs to be opted out, rather than your entire org?
Many thanks,
Dom
Hi @AntoineAugusti,
Apologies for that. I’ll take a look at this today and hopefully get it sorted for you.
Dom
Hi @DomParfitt, I appreciate the fast response. This is only a problem for the one project, the rest of the org is fine. I am happy to send the job link via private DM, but cannot seem to find the option to do so.
Thanks,
Ryan
Hi @wilbur,
No problem, one of our Support Engineers has passed me a Job link from your Zendesk ticket.
I’ve just applied the opt-out to that project, so you should see it in around 10-15 minutes.
Thanks,
Dom
1 Like
Hi @AntoineAugusti,
I’ve just applied some changes that should hopefully opt your org out. Would you be able to confirm that it’s working as expected?
Thanks,
Dom
Hi, https://app.circleci.com/pipelines/github/etalab/transport-site/11945/workflows/3fede7ca-6bde-4b67-9756-432fc86842a3/jobs/45267 shows that we’re using build agent v1!
As a side note some workflow succeeded on build agent v2 even with Erlang/OTP 25.
Hi, I also believe v2 is the cause of some issues I’m having. In my jobs, the “Spin up environment” step succeeds, but then the job fails, it never moves onto the “Preparing environment variables” step and no errors are thrown, it just fails after successfully downloading and starting the container. We have another project with almost an identical config and it’s using v1 and has no issues.
Hi @Slighting4125,
Sorry you’re having issues.
Would you be able to send me a link to one of your problematic Jobs via DM and I’ll look at getting you opted out whilst we investigate.
Many thanks,
Dom
I’m not sure if this is related, but we recently had trouble with jobs running from branches with the word “comments” in the branch name. Any branch that was created that had that in it would result in the CCI error:
failed to fetch Branch ‘topic/spanky/comments’ from repository ‘org/repo’
Changing the name of the branch solved the problem. We tested this with multiple branches.
Hi @spanktar,
Thanks for reaching out.
I don’t believe the issue you’ve mentioned is likely to be related to this rollout, it sounds more likely to be a restriction of your VCS. Just to be certain, could you check whether the affected Jobs displayed the V2 runtime banner mentioned in this post and whether there were any Jobs displaying the same behaviour without that banner (i.e. before the rollout)?
Thanks,
Dom
Happy to, but I don’t see an option to DM you?
Yes, the banner was shown. I have never encountered this error before. I agree that it probably isn’t related, but you never know…
Hi @spanktar,
If you’d like to send me a Job link via DM then I can take a look to see if I can confirm or deny whether it’s related to this rollout. For what it’s worth I tried to recreate this issue internally and wasn’t able to, which is why I thought it may be VCS related.
Thanks,
Dom
Ever since our org has been opted into this upgrade, our e2e tests are constantly flaky. Often times they run for 45 minutes even though they should only run for 15 minutes, and sometimes they never complete (running forever). Can our org opt out of this upgrade?