Meltdown and Spectre: Recommended Actions for operators of installed CircleCI servers

[ndintenfass]

NOTE: The following are instructions for operators running their own installations of CircleCI. For a more general discussion and actions for people using circleci.com, please see our previous post.

While we realize that many of our customers have our server software running in many different types of environments, this guide will primarily focus on Amazon Web Services installs which were installed using our Terraform script. If you have set up your infrastructure using a different method, please reach out to your account team for assistance.

This will require downtime, so please plan accordingly.

We suggest that you perform a backup before proceeding. You can find a guide to performing a backup in our documentation.

A typical hosted CircleCI setup consists of several components:

• Services Box
• 2.0 Builders
• Machine Image
• 1.0 Builders (legacy installs only)

Services Box
Requires:

• Services box IP address or domain
• CircleCI AWS Private Key
• SSH access to Services box

Steps:

• Access the management console commonly located at https://CircleCI_domain_or_IP:8800
• Press the stop button to shut down CircleCI
• SSH into the the instance and perform a system update. If you are using Ubuntu as the base image, simply run sudo apt-get update && sudo apt-get upgrade
• (Optional) You can purge unused previous kernels to free up disk space by running sudo apt-get autoremove --purge. If you encounter issues, please consult this link before contacting support.
• Once the upgrade has completed, reboot your system using sudo reboot. The patched kernel will be used only after a reboot.

2.0 Builders
Requires:

• Autoscaling Group Name (Typically circleci_nomad_asg)
• Launch Configuration Name (Can be located by selecting ASG name in AWS. Commonly terraform-)
• CircleCI AWS Private Key Name

Create a New Launch Configuration:

• In AWS, locate the launch configuration
• Right-click and select Copy Launch Configuration
• To the right of AMI Details, select Edit AMI
• Locate the appropriate AMI for your region from the table below and enter it into the search bar. Press Select followed by Next: Configure Details
• (Optional) Replace name field with something easier to recognize IE circleci-nomad-lc
• Press Skip to Review
• Press Create Launch Configuration, verify the key name is correct, and press Create Launch Configuration

Update the Autoscaling Group:

• Select the autoscaling group.
• At the bottom of the page and to the right of Launch Configuration select Edit
• Select the launch configuration we created in the previous section and press Save
• From the instances page, terminate all instances that contain nomad-client in the name. For example, circleci-nomad-client
• The Autoscaling Group should begin spinning up new instances. This can take several minutes so please be patient.
• If you have Admin access to CircleCI, you will be able to view builders becoming available via User Icon → Admin → Fleet State

1.0 Builders (Legacy Installs)
Notes:
This section only applies to those customers who are running CircleCI 1.0 or a CircleCI 2.0 hybrid fleet.
LXC 1.0 builders have been deprecated and are no longer supported. We highly suggest using the Docker 1.0 builders instead. Instructions for upgrading are provided.

Requires:

• Autoscaling Group Name (Typically circleci_builder_asg)
• Launch Configuration Name (Can be located by selecting ASG name in AWS. Commonly terraform-)
• CircleCI AWS Private Key Name

Create a New Launch Configuration:

• In AWS, locate the launch configuration and
• Right-click and select Copy Launch Configuration
• To the right of AMI Details, select Edit AMI
• Locate the appropriate AMI for your region from the table below and enter it into the search bar. Press Select followed by Next: Configure Details
• (Optional) Replace name field with something easier to recognize IE circleci-builder-lc
• Note: If using LXC builders, please see Switching to Docker Builders section below before proceeding.
• Press Skip to Review
• Press Create Launch Configuration, verify the key name is correct, and press Create Launch Configuration

Update the Autoscaling Group:

• Select the autoscaling group
• At the bottom of the page and to the right of Launch Configuration select Edit
• Select the launch configuration we created in the previous section and press Save
• From the instances page, terminate all instances that contain nomad-client in the name; for example, circleci-nomad-client
• The Autoscaling Group should begin spinning up new instances. This can take several minutes so please be patient
• If you have Admin access to CircleCI, you will be able to view builders becoming available via User Icon → Admin → Fleet State

Switching to Docker Builders (If Necessary):

Requires:

• IP of Services box
• 1.0 Builder Secret Passphrase
• Copy of Docker setup script: link
• http proxy IP (Only if applicable)
• https proxy IP (Only if applicable)
• no_proxy Option (Only if applicable)

Steps:

• Copy of contents of the Docker setup script into a text editor and replace the following with the correct details:
  • ${http_proxy} (If N/A then set to “”)
  • ${https_proxy} (If N/A then set to “”)
  • ${no_proxy} (If N/A then set to “”)
  • ${circle_secret_passphrase}
  • ${services_private_ip}
• Select the Advanced Details dropdown
• Clear the contents of the User Data text box
• Paste the edited Docker setup script
• Resume Create a New Launch Configuration section

AMI List

Region            AMI
ap-northeast-1    ami-2d69f14b
ap-northeast-2    ami-cd78d8a3
ap-southeast-1    ami-c38bf8bf
ap-southeast-2    ami-a437cac6
eu-central-1      ami-ff30a290
eu-west-1         ami-3cf36145
sa-east-1         ami-24642648
us-east-1         ami-0ce3bb76
us-east-2         ami-01664c64
us-west-1         ami-98595af8
us-west-2         ami-779a2d0f