Meltdown and Spectre Vulnerabilities: What We Know and Recommended Actions

CircleCI is aware of critical vulnerabilities, termed Meltdown and Spectre, affecting modern processors in a vast range of computers, devices, and clouds, including CircleCI infrastructure. These vulnerabilities allow malicious users to read data from other programs. We are monitoring upstream vendors and providers and will apply fixes as soon as they become available.

More details, please?

Meltdown and Spectre represent a new class of side channel attacks in most modern processors. They exploit hardware CPU bugs to read kernel memory and escape various security isolation barriers between applications, containers, and virtualization hypervisors - to read secrets and data in other programs and in the operating system. This is a vulnerability that has existed for more than 20 years in modern processor architectures like Intel, AMD, and ARM across servers, desktops, and mobile devices.

Luckily, there are software patches against Meltdown. Patches for Linux, MacOS, and Windows are being released to mitigate these vulnerabilities and all are advised to upgrade ASAP. No patches are available for Spectre yet.

We are aware of some early Proofs of Concept being described in the exploit documentation; our initial analysis of how these operate indicates that they would not succeed in the CircleCI platform. However, we are taking every precaution to upgrade all systems as soon as possible.

How am I impacted as a Hosted CircleCI customer?

Hosted CircleCI supports multiple build configurations that are impacted differently:

2.0 Machine Executor jobs have no risk of customer data leakage as a result of these bugs. Machine jobs run on ephemeral per-job virtual machines where customer code has root access on the machine. Our cloud providers (AWS and Google Compute Engine) have patched their underlying cluster.

2.0 Docker Executor, 2.0 MacOS Executor, 1.0 builds are currently vulnerable to these attacks. We plan to upgrade the hosts as soon as the patched kernels are released for the Ubuntu distribution. We advise customers to make their own judgement about the risk profile they are comfortable with, and either move to machine executor or wait until hosts are patched. We will update here with our progress patching our hosts.

When will you patch and what will the impact be?

CircleCI continually monitors for CVEs. This is considered a critical level and we are standing by to patch our Linux machines as soon as patches become available for the Ubuntu distribution.

The kernel patches are expected to have some performance impact. While there is a wide range of speculated impact on CPU usage, the impact will vary significantly by the build workload. We will be monitoring for performance impact on build and reporting updates.

How am I impacted as a CircleCI Server customer?

We advise our Server customers to upgrade their CircleCI hosts along with the rest of their infrastructure. You can apply the kernel patches as they become available for your upstream Linux distribution.

Where can I find more details?

The Meltdown and Spectre site has plenty of useful resources, guides, and links.

6 Likes

Out of interest, could code running in a Docker container theoretically read what is happening in another container, using one of these vulnerabilities? Is the answer to that question different when considering full-fat virtualisation, such as VirtualBox and VMWare?

Thanks for your question.

With these vulnerabilities, one can read data from any process on the physical host whether it’s a different container and sometimes even in a different virtual machine. Google Cloud Engine and AWS have remedied the issue so customers cannot escape into other customers VMs - hence machine executor is fine for our customers.

OK, thanks for that. Every now and again one has to reset one’s mental model of computer security, and this is one of those times. In the past, I’ve tended to think of containers and VMs as unescapeable thanks to hardware limitations, and now we know the vulnerability is hardware-related in nature, we cannot even rely on that.

Urgh :pouting_cat:

We are activly rolling the build fleet to patch against Meltdown: Fleet roll for patching Meltdown

1 Like

This topic was automatically closed 166 days after the last reply. New replies are no longer allowed.