I’m very sorry to hear about the issues this has caused you and thank you for the feedback.
Port 80 already being bound is not an expected change nor something we have seen before. I’ll use the information you have provided to attempt to recreate the problem and resolve it so that others don’t experience the same issue.
I see that you have raised a support request about this, and that you have found a work around for the time being (binding to another port). Based on this I will leave your project opt’ed in for now unless you have any objections. If you see any further issues please don’t hesitate to let us know and we can opt the project out pending investigation.
I’ve investigated the issue you reported. It turned out that the container runtime was not applying the correct kernel capabilities for the additional/background containers to allow them to bind privileged ports.
We’ve applied a fix, so you should be able to revert your workaround now if you’d like. Apologies again for the troubles the bug caused you.
Based on your feedback about communication, we’ve added a banner to the job page when the new runtime is in use to make it easier to find help:
Our CI has suddenly started failing. It seems limited to the tests which are using YugabyteDB inside a docker. Only difference from older working runs is that I see the banner about “This job used the new v2 container runtime.”
Error doesn’t tell me much. Any chance this is related to cgroup changes?
Starting yugabyted...
| Starting the YugabyteDB Processes...Failed to setup master. Exception: Traceback (most recent call last):
File "bin/yugabyted", line 5695, in setup_master
master_uuids = retry_op_with_argument(self.get_master_uuids, master_addrs)
File "bin/yugabyted", line 10480, in retry_op_with_argument
return func(argument)
File "bin/yugabyted", line 4179, in get_master_uuids
raise RuntimeError("process died unexpectedly.")
RuntimeError: process died unexpectedly.
For more information, check the logs in /root/var/logs
exited with error (exited with 137)
Please could you send the last 4 segments of a CIRCLE_BUILD_URL (found in the “Preparing Environment variables” step) e.g. gh/my-org/my-project/1234. I’ll take a closer look
I’ve just had https://app.circleci.com/pipelines/github/i2group/circleci-orb-node/461/workflows/49285575-198f-4658-9dea-a61610d71a3d fail because when it asked for a docker container of various sizes, it got 36 CPUs instead of the expected size.
Now, I recall an earlier conversation where I was told “sometimes CircleCI has an excess of CPUs so it generously doles out more than you expect”, which is why my code is expecting “between 8 and 16” instead of expected “exactly 8” but I’m not expecting to get more than double the amount I asked for.
…but this build just failed and it declared that it’s running on the new docker infrastructure, so I figure that’s why it failed.
FYI my “calc cpu count” command (the code under test) was written entirely to work around the old CircleCI docker infrastructure defect whereby it’d lie to auto-sizing tools about the available resources, which then caused build failures when tools were told “you’ve got half a terabyte of ram and over 100 CPUs to use” when it’d get killed off the moment it went over 8gigs of ram.
It may be that this “new” infrastructure is less predictable but no longer lies, which would make my detection code unnecessary (once everything is on the v2 docker).
It’s used across >1 project, yes.
A solution that handles v1 and v2 at the same time would be good
… especially if the code makes it clear what’s v1-only and what’s v2-only so that, once CircleCI is 100% v2 everywhere, we can drop support for v1 from the code to clean it up.
But, yes, right now I want something that will “just work” regardless of what CircleCI environment it’s running on.
Thank you @DominicLavery ; code received and it seems to work. Can you update the thread on that (long running) bug-report/RFE thread accordingly?
Note: In an ideal world, of course, none of that code would be necessary because auto-sizing stuff would “just work” without it.
… and/or there would be CircleCI-provided env vars telling folk the official size of the CircleCI agent they were on.
@DominicLavery looks like our e2e step started failing. The only thing we can think of at this point is the update to the v2 container runtime. What is the process like to get it disabled for my project just as a way to rule it out as the cause of the issue?
Hi @DominicLavery, we are having issues on builds that have started using v2 related to composer installation commands that previously had no issue. Are we able to opt out our project to rule this out?
