Give us a heads up! The latest macOS VM update broke our builds!

[tww0003]

Please give us a heads up before changes to the installed software on the VMs are made!!!

The latest macOS VM update broke our builds. I assume the macOS VMs were updated either last night or this morning…

Ruby version 3.1.3 was removed and replaced with 3.1.4 without warning.
Xcode 14.3.0 was removed and replaced with 14.3.1 without warning.

We had a sea of red with our CICD pipelines today and had to spend time investigating when this could have been prevented by a simple email letting us know ahead of time.

https:[slash slash]circleci[dot]com/docs/using-macos/#supported-xcode-versions
Has not been updated to let us know that 14.3.1 is supported.

https:[slash slash]circle-macos-docs.s3.amazonaws[dot]com/image-manifest/v11629/manifest.txt
The list of installed software does not show that Ruby 3.1.3 was removed or that 3.1.4 was installed.

I had to print the installed ruby versions and the contents of the /Applications folder in the VM to figure out what happened, wasting that sweet, precious, build minutes.

CircleCI behaving in an expected manner is critical to our work flow. We cannot have random updates breaking our pipelines.

[bytesguy]

Hi thanks for the feedback. This is the same issue as in the thread linked below, so I will post a link to my response here:

MacOS with Xcode 14.3.0 has the wrong Ruby version

As per our image policy, patch versions are updated in-place and can include bumps to patch versions of included software as part of the process. The 14.3.0 image was replaced with 14.3.1 recently: https://discuss.circleci.com/t/xcode-14-3-1-rc-released/48152 It is better to use the macOS orb as it contains logic to fuzzy match Ruby versions, allowing you to just specify a major.minor version, rather than working down to the patch level (patch images will retain the same major.minor versions …

We will take this feedback on board for future releases, but we do recommend using the macos orb for managing Ruby versions in the future.

We do have a docs pr for the change in manifest, but it is waiting for review. I will chase up the docs team on this. Add Xcode 14.3.1 by BytesGuy · Pull Request #8053 · circleci/circleci-docs · GitHub

[tww0003]

@bytesguy

we do recommend using the macos orb for managing Ruby versions in the future.

I would prefer not to use an orb because the fuzzy matching of versions. Dot releases break things.

We give specific version numbers in all our configs for a reason - to ensure that we have reproducible and predictable builds. Our CICD system should be able to product the exact same results as our developer machines. How can we efficiently debug a build issue if we aren’t sure of the environment it happened in?

Notification of changes in advance is critical.

@bytesguy

We do have a docs pr for the change in manifest, but it is waiting for review

I would like to add “waiting for PRs to pass review before release” to my feature request.

[vkorostelev]

Glad that someone created this topic. The same happened to our pipelines as the update also broke the official Python orb with the latest version: circleci/python@2.1.1

There is no circleci user anymore in the 4.3.1 image and restoring cache step is failing with the error:

mkdir: /home/circleci: Operation not supported

Exited with code exit status 1
CircleCI received exit code 1