Letsencrypt SSL root cert problems

Build Environment
1

We’re experiencing a blocking issue with at least one letsencrypt cert because its root cert seems to have expired as of today in docker images published by circleci and an apt-get update does not help.

The issue manifests itself when executing:
curl https://install.meteor.com

in a docker container using circleci/node:14.17.6-stretch

The error is:

SSL certificate problem: certificate has expired

The cert itself is fine, but after reading these articles I suspect the root cause is the switch of root certs for letsencrypt:

I also tried updating the root certs with
sudo apt-get update && sudo apt-get install -y ca-certificates
but that did not help

What can I do? Builds are now failing every time…

2

You’re using a “stretch” image, which is old.

My first suggestion is to switch to the next-gen CircleCI Node.js image. So instead of circleci/node:14.17.6-stretch you would use cimg/node:14.17.6. You’re using a legacy image which will be deprecated in a couple months.

If you must stick with the legacy image, I’d try swapping out “stretch” for something newer such as “buster” and see if that helps. It’s a newer Debian base and so may have a newer SSL/TLS package that contains the new Let’s Encrypt root cert.

3

just FYI @FelicianoTech, this issue is happening on the default VM image. I was able to fix it by editing /etc/ca-certificates.conf and removing DST_Root_CA_X3.pem. I then tried running update-ca-certificates but that didn’t remove the certs. I had to run dpkg-reconfigure ca-certificates and then choosing ‘yes’ and then ‘ok’. Hopefully this gets resolved quickly as it breaks curl interacting with Lets Encrypt endpoints

4

this also appears to be happening on macOS:

https://app.circleci.com/pipelines/github/facebook/flow/2973/workflows/d45aade4-9b98-49a9-b5b4-a240bf4eec1e/jobs/55415

opam.ocaml.org is using ISRG Root X1 (letsencrypt)

5

@MrMMorris i’m working around this by using a .curlrc with cacert /path/to/cacert.pem, where cacert.pem is an updated bundle from curl - Extract CA Certs from Mozilla

6

I am having the same problem with curls to ibm, example

curl -f https://download.clis.cloud.ibm.com/ibm-cloud-cli-metadata/info.json

curl: (60) SSL certificate problem: certificate has expired

More details here: curl - SSL CA Certificates

curl performs SSL certificate verification by default, using a “bundle”

of Certificate Authority (CA) public keys (CA certs). If the default

bundle file isn’t adequate, you can specify an alternate file

using the --cacert option.

If this HTTPS server uses a certificate signed by a CA represented in

the bundle, the certificate verification probably failed due to a

problem with the certificate (it might be expired, or the name might

not match the domain name in the URL).

If you’d like to turn off curl’s verification of the certificate, use
the -k (or --insecure) option.

7

@FelicianoTech any updates on this? This is also breaking go modules within our pipeline due to gopkg.in using lets encrypt. See this reddit thread for more information.

Any help you can provide would be greatly appreciated!

1 Like
8

I was able to fix this on macOS by bumping my XCode version up to 12.5, which I guess uses an image with more recent root certs.

9

Use an up-to-date image or run apt-get update && apt-get upgrade -y.

1 Like
10

Yes, this only affects Catalina based images and lower. Big Sur images (Xcode 12.5 and above) are unaffected by this.

I did a little write up on this with a solution for older macOS versions:

Meanwhile, we are looking at how we can fix this without end-user intervention.

2 Likes
11

I tried all suggested solutions on CircleCI macOS executors.
However none of them worked, SSL errors all over the place still.
So looked up where the CAs are loaded from effectively:

grafik
grafik1613×694 137 KB

Then added a command step to CircleCI config that downloads from curl_se and replaces the pem @ system /usr/local/etc/openssl/cert.pem .
Now it works as expected again and I can connect to my domains who use Let’s encrypt.

@bytesguy
Meanwhile, we are looking at how we can fix this without end-user intervention.

Maybe providing a proper/fixed file at /usr/local/etc/openssl/cert.pem by default is what you looking for?

Hope it is helpful to somebody.
BTW: I tried all kinds of available CircleCI MacOS versions (circleci config xcode: tag) and none resolved the issue. (As the suggestion was to “use Xcode 12.5”)