The Problem
On Monday, April 26th, packages/formulas installed via Homebrew will either fail or take a long time to manually compile on CircleCI’s macOS images. These are the images you specify in your config using the xcode key. This issue will occur for a span of 6 hours.
When
The brownout will occur: April 26, 2021 13:00 - 19:00 UTC
Why
This brownout is in preparation for Homebrew’s hosting provider, Bintray, shutting down for good. Bintray, which hosts Homebrew’s bottles (binary packages), is being retired and is being replaced by GitHub Packages hosting. To point to the new host requires an update to Homebrew itself - at least v3.1. CircleCI’s macOS images don’t contain a Homebrew version this new as of today.
Workaround for Monday
During the 6-hour brownout, customers have two options:
- Run
brew update in your CI build before using any other brew commands. This can take around 5-8 minutes to run, not ideal in a CI environment but this will prevent your builds from failing.
- Wait it out. Homebrew/Bintray will return to normal after the 6-hour brownout.
Future Solution
This brownout is a lead up to Bintray support stopping completely on May 1st. We are already working to get our entire macOS fleet updated before then to prevent any lasting impact from the shutdown of Bintray.
Resources
Questions?
Please ask them below. Thanks.
1 Like
Update: May 4th
We’re passed May 1st and the Bintray shutdown. As we know, CircleCI’s macOS images contain an older version of Homebrew, which is affected by the Bintray shutdown. We’re addressing the issue in two ways.
A Patch
To address the situation immediately, a temporary fix to automatically update Homebrew has been put in place for existing macOS images to prevent failed pipelines due to this issue. Some users, depending on the age of the image, may notice slower startup times. This is temporary as we continue to roll out the permanent fix.
Permanent Fix
The macOS Team is working diligently to get our entire macOS fleet updated, prioritizing by image usage. These will ship updated versions of Homebrew.
1 Like
Update: 11 May
Here is the latest update on the changes we are working on to mitigate the impact of the Bintray shutdown.
Patch Removal
The macOS incident on 5 May (statuspage) required the temporary brew update patch to be removed to prevent slow boots of macOS VMs.
Meanwhile, we suggest adding a - run: brew update step to your jobs to mitigate any issues untill an updated image for your Xcode version of choice is ready.
Updated Xcode 12.4 Image Now Live
The first updated image has now been released and is available to use now.
This image is the Xcode 12.4 image which updates Homebrew from 2.6.2 to 3.1.5. This update replaces the previous image (xcode: 12.4.0) and does not require any config changes to use it.
If you experience any issues in your jobs which appear related to this change, please contact support or post in this thread.
Further Updated Images Coming Soon
We are working on deploying more updated Xcode images as soon as possible. We will update this thread with details in due course.
2 Likes
Update: 13 May
Xcode 12.3 has now been updated. The Homebrew version has been updated from 2.6.2 to 3.1.5
We are continuing to update all of the images we support and will update this thread in due course.
1 Like
Hi,
I’ve noticed over the last day or so, that the Xcode image for v12.1.0 was recently updated, and as a side-effect, upgraded node to v15 and npm to v7, which was definitely not the case before, and I believe not expected given the versions listed in your docs (CircleCI macOS Container Software), and can easily be a breaking change for many (including us).
Could you confirm this was intentional?
Many thanks!
Update: 19 May
Xcode 12.2 has been updated from brew 2.5.11 to 3.1.5. This is now available for use.
We anticipate Xcode 12.1.1 to be updated in the next 24 hours and will continue moving down the image stack over the next few days.
As per our Xcode image policy, we have removed 12.1.0 and redirected requests to 12.1.1.
@vicrep 
This is likely as a result of the above mentioned removal of 12.1.0 (it went live a little earlier than we anticipated). Looking at the manifests,
Between these two versions there is a jump from Node 14 to 15. Ideally we would have pinned Node to 14 given that is the current LTS, but this has not been the case. In the future we will look at either pinning to the next LTS or using NVM to make the experience better.
For now you can drop back to Node 14 with the following:
brew install node@14 && brew link --overwrite node@14
This may take a while due to having to update brew, but once the updated 12.1.1 image is live (which we will announce here shortly), you can add HOMEBREW_NO_AUTO_UPDATE=1 to the start of that command to reduce the time it takes. In my testing this will then only take 25-35 seconds.
Apologies for the problems this has caused!
1 Like
Since yesterday all my builds on CircleCI MacOs images are failing.
I get:
==> Pouring six--1.16.0_1.catalina.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink lib/python3.7/site-packages/six.py
Target /usr/local/lib/python3.7/site-packages/six.py
already exists. You may want to remove it:
rm '/usr/local/lib/python3.7/site-packages/six.py'
To force the link and overwrite all conflicting files:
brew link --overwrite six
To list all files that would be deleted:
brew link --overwrite --dry-run six
when the automatic script to install the dependencies tries to run:
brew install opencv
I tried to run brew update before the command or even brew upgrade, but nothing works.
Evreything was running smoothly before yesterday.
I’m a newbie in the MacOS environment so I’m not sure what to try.
Any idea? @bytesguy
Thank you
Update 24 May
Xcode 12.1.1 and Xcode 12.0.1 have now been updated to Homebrew 3.1 or higher.
We will continue to update Xcode 11 images over the course of this week
@mtamburrano I have tested this on both the updated images and non-updated images and the outcome is the same which leads me to believe it is a problem with the formula itself - rather than an issue with the updated images.
I would suggest opening an issue here for further assistance:
Hi folks, thanks for bearing with us on this! We can happily announce that all images now have Homebrew 3.1 or higher to mitigate the bintray shutdown.
If you have any issues with the updated images, please open a support ticket.
Thanks!
1 Like
This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.