Two of our test jobs in our main workflow just run tests for our SPM packages - AFAIK these don’t run within a test host, only our main app test target does, but they all seem to hang just the same?
Hello everyone,
Just an update from the macOS team. We are still actively working on a solution for this problem. Once we finish our investigation we will update this thread. This investigation is specifically for the problems with simulators and extended build times on Xcode 15+.
Thank you and have a good weekend!
Luis
8 Likes
Just wanted to chime in here that this problem is not limited to CircleCI. We are having exactly the same issues on GitHub Actions both with Xcode 15 and 15.0.1.
This is definitely an Apple bug (or bugs) that need to be resolved.
2 Likes
@shivam0sharma You would call that after the simulator(s) have booted, then reboot it/them. However, our continued investigation into this issue is showing that RAM/CPU usage isn’t much of a factor, so this solution is unlikely to yield any meaningful gains. We are working on a better resolution and will share it soon!
2 Likes
Hello everyone
I am pleased to announce that we have discovered, built and re-released the Xcode 15.1 Beta 2 image. This re-release addresses a lot of the Simulator performance regressions that we have seen with Xcode 15. The biggest change you will notice is that it is based on macOS 14 Sonoma instead of macOS 13 Ventura. We have major performance improvements on internal testing on both M1-Medium/Large and Gen2 Intel.
Luis
7 Likes
@luis_cci This new image is looking very promising for our workflows, thanks for the update! Will you be updating the Xcode 15.0.1 image as well?
Would it be possible for you to provide a list of the changes in addition to Sonoma?
Here are the release notes for the new image, including all changes made - Xcode 15 beta 2 re-released and bumped to macOS 14.1
Some good news - the new image does seem to resolved the hanging tests problem that made the previous images unusable.
My workflow now completes in an acceptable amount of time (around 15 minutes) although there still seem to be some general performance regressions on the M1 medium executor compared to Xcode 14.
Compile times seem to be up to 2x slower and test times can be up to 2x slower too overall. It’s possible there’s still a fixed delay when booting the sim to run the tests of a couple of minutes rather than test execution itself being slower.
This is acceptable performance, and probably unblocks us from upgrading to Xcode 15, but it’s still not great.
@JARinteractive We will not be updating the Xcode 15.0.1 image. Re-releasing images, especially when it includes a major OS upgrade, can be disruptive and we want to minimize the risk of breaking people’s builds. We’ll be building with macOS Sonoma for the remaining Xcode 15 images moving forward, but are limiting the re-release to just 15.1 image.
@lukeredpath I’m glad you’re seeing improvements and we hope the performance will return to your expectations once Apple releases a full fix for the underlying issues.
@alexazl I’ve been given the OK to try out a large executor for a month and see how that impacts performance and costs. Out of curiosity - is there any timeline to update the Apple Silicon hardware to M2?
I’ve done some more investigation and it seems like even on our Xcode 14 times there was some noticeable variation in compile times. For instance, our core SPM package job sometimes takes around 3m 30s to compile, sometimes it takes closer to 6m. It looks like this variation is what I was seeing in the Xcode 15 builds too.
I’m a bit unsure how the same code running on the same executor can have such a variance in compile times to be honest.
I’ve managed to get all of our tests finally passing but I am still seeing weirdly inconsistent built times between workflow runs, even after moving to the large M1 executor.
For example, building our main app target for testing on the last two workflow runs:
- First run - 4m 34s.
- Second run - 8m 53s.
This is the exact same code in both instances.
The test job seems to run in a consistent amount of time.
If all of the jobs ran in the fastest time our build times would be great but as it is I’m sometimes seeing slower compile times on large versus the faster times I’ve seen on medium executors.
This is the same workflow running three times against the exact same code.
It even affects our coverage job which runs after all build/test jobs, which gathers coverage data and uploads it to Coveralls - this job downloads and compiles a standalone Swift tool and even this has has significantly different compile times - sometimes it takes 30s, sometimes it takes double that.
Is there some other issue with these images?
@lukeredpath Can you file a support ticket with links to your builds so we can investigate further?
1 Like
@lukeredpath
Hello,
Please yet file a support ticket and we will be better able to investigate and determine the variance. My suspicion prior to having build links is that it is variance in network performance as you mentioned an upload step.
To answer some of your questions
- I’m glad you’re seeing improvements, that being said, we also saw the same behaviour in which our internal UI tests were performing 2x slower on Xcode 15 (re-release) compared to Xcode 14. This is again due to a performance regression on Apple’s side as we have seen this behaviour noted other CI platforms
- As @alexazl has stated, we will not be re-releasing
15.0.1due to potential breaking changes for a large customer based, unless this opinion changes due to customer demand - As far as we are aware there are no other issues with the images, there has been no other major changes to the images besides any changes associated with an upgrade from
macOS 13→macOS 14(as well as anyhomebrewdependencies that are associated with that)
Hello,
This is not totally related to the issue in this thread but related to a regression that started with Xcode 15 when using fastlane, and on topic re new images. Can you please also ensure that the new images have the latest version of fastlane installed (2.217.0)? There was a release last night which should address a lot of failures we started seeing when using Xcode 15.
Thanks!
This is definitely nothing to do with network performance - the upload is a red herring. The coverage upload job compiles some Swift tools and it’s this compilation step that is taking a varied amount of time. The main issue is all of the build-for-testing jobs that can randomly take double the amount of time, causing our overall workflow time to vary by up to 50% (12m is the quickest, 18m is the slowest) for the same code.
I have followed up to my existing support ticket about this.
Sharing some findings.
We just got done migrating to Xcode 15.1
- With Intel Gen 2 and x86_64 simulator - All of our Unit and Snapshot tests are passing.
- With M1 Medium and x86_64 simulator using Rosetta2 - Our Snapshot tests pass, but our Unit tests seem to be randomly failing due to tests not asserting within the mentioned timeouts.
- With M1 Medium and arm64 simulator - All of our Unit and Snapshot tests are passing.
Looks like running the tests on x86_64 simulator using Rosetta2 seems to be slower in general.
Anybody else notice similar?