Upgrading the underlying kernel for the Docker executor

CircleCI is upgrading the kernel version of the operating system that is used to run customer containers as part of the Docker executor on CircleCI. This change is critical to ensuring the underlying infrastructure that runs your jobs continues to provide reliable and performant execution.

This upgrade should be effectively invisible to customers. The current kernel version used is 4.15 and it will be upgraded to 5.4.

CircleCI is gradually rolling out this change for all Docker executor customers beginning on December 1, 2021.

Relevant Information:

  • CircleCI does not expect this to be a breaking change for the overwhelming majority of Docker executor jobs as the jobs are isolated in their own containers.
  • There is a small chance that a job results in an Out of Memory (OOM) failure more frequently because of a change in the newer kernel’s OOM Killer. If a customer is experiencing more frequent OOM errors with the newer kernel version, CircleCI recommends upgrading to a resource class that offers more memory.
    • It is possible to view which kernel version a job uses during execution by outputting “uname -r”
  • If there are other issues observed as a result of this change, please comment on this post immediately so they can be analyzed and addressed.
  • Once the kernel version is upgraded to 5.4 for 100% of customers, CircleCI will begin the gradual roll-out process of updating the full underlying operating system that is used to run customer containers as part of the Docker executor from Ubuntu 18.04 to 20.04.
    • Similar communication will be shared at that time
  • This upgrade will not impact Remote Docker jobs

Please comment on this post or reach out to your support contact if there are any questions about this change. Thank you!

1 Like

Hi Sebastian! It seems like this upgrade hit our team recently and it caused some issues.

In particular, we’re a Rails app using ActiveStorage and we’ve started getting ActiveStorage::IntegrityError failures on many of our tests that touch ActiveStorage.

It seems there’s an issue with docker + certain versions of the linux kernel which is well described in this Github issue: Very specific set of circumstances leads to zero-byte (empty) file being created · Issue #1015 · docker/for-linux · GitHub

We’re seeing the same things that person described; when Rails tries to copy a file from /tmp/something to /tmp/something-else with IO.copy_stream, the destination file is created but is zero bytes.

For now, we’ve worked around it in our test suite by defining a custom ActiveStorage service that doesn’t use IO.copy_stream when writing from file → file, but for the sake of others on the platform it might be good to figure out why this is happening and if a different kernel version would help.