This is likely not common, however after this upgrade my job has started failing.
I was installing and configuring dnsmasq so that I could do wildcard DNS resolution. After the upgrade I’m getting an error that something is already running on port 52.
Here are a few of the log entries from the failure.
address=/.test/127.0.0.1
* Starting DNS forwarder and DHCP server dnsmasq
dnsmasq: failed to create listening socket for port 53: Address already in use
...fail!
Exited with code exit status 1
We’re having an issue where our builds are unable to reliably detect the core count in the docker executor where they’re running. We have depended on the core count to run python tests in parallel using xdist.
We used to use /sys/fs/cgroup/cpu/cpu.shares, we switched to /sys/fs/cgroup/cpu.max according to your solution here.
We run the same command in multiple jobs in different executor sizes, and sometimes cpu.max indicates 16 or 32 cores or an xlarge, where we would expect 8. We spawn too many workers, then they crash because they run out of memory.
Is there a better way to determine core count? If not, can we please go back to the older runtime for now?
I’ve just applied a temporary opt-out for your org, it may take 10-15 minutes to apply after that you should see your Jobs running on V1. Please let me know if you still see things running on V2 after that.
I’ve opted that project out, so you should see that take effect in around 10-15 minutes. I’ve only opted out that specific project at the moment, so if you need your entire org opting out then please let me know.
You can see the output of cat /sys/fs/cgroup/cpu/cpu.shares or cat /sys/fs/cgroup/cpu.max if you look at output from the longest step run...tests. In each job.
Maybe this is a factor:
This says “Due to the way we pack workloads on to the underlying compute that runs Docker jobs, very occasionally surplus CPUs exist. Instead of letting them go unused, we will grant them to workloads on a random basis.”
It seems like either we didn’t get extra memory to go along with the extra CPU, or scaling up xdist by 2x had some memory overhead that we can’t handle. We didn’t measure the amount of memory we saw in older jobs.
This worked on cgroups v2, but when we downgraded to cgroups v1, we saw this:
+ source '~/project/.circleci/helpers/pytest_xdist_params.sh'
/bin/bash: line 13: ~/project/.circleci/helpers/pytest_xdist_params.sh: No such file or directory
This was a bit unexpected, and it’s making us a little worried about switching back and forth.
When we switch back, is there any way that we can test out the change on an isolated build before we roll it out everywhere? If that’s not possible, then can we please coordinate a time? We want to help make sure users on our end aren’t impacted too long by any potential problems.
@DominicLavery We are also experiencing build failures when using the v2 runtime. Would it be possible to get our organization opted out of the upgrade?
Also, one thing that would be helpful was if there was a way to run your build against the v2 before deciding to migrate the project. That way teams could make sure their build passes ahead of time.
Our jobs read out the memory usage because our machine type constrains us by memory more than CPU. This broke today due to getting the rollout, and we had to update to support cgroupv2 suddenly:
I’ve opted your org out, it may take 10-15 minutes to apply but after that you should see your Jobs running on the V1 runtime.
From your link I can see that you’re using an older version of Pylint that doesn’t correctly calculate the number of available CPUs when using the auto-detection for parallel workers, which is likely why you are seeing issues. Are you able to upgrade to use a more recent version of Pylint?
I can make sure that we coordinate with you when we’re ready to opt you back in.
If you have a project that you can safely use as a testing ground for this rollout then we can explicitly opt just that project in whilst leaving the rest of your org opted out. I’ll DM you shortly if you want to send me a project you’d like to use for this.
Sorry you’re seeing this but thank you for reporting it to us. Do you have any other examples you could share? That would help us to identify potential causes.
