FWIW that’s presumably the same fix included in Xcode 15.1 beta 1, which barely helped with this issue
Maybe, but the 15.0.1 RC release notes do reference FB12237092, which wasn’t referenced in the 15.1 notes, so there’s hope.
Oh good catch. I’ll definitely give this another try if/when CircleCI updates to use 15.0.1
Looks like CircleCI just added Xcode 15.0.1… and after some preliminary testing, it looks like all performance issues have been fixed!!! 
Update:
Well it seems like it’s not quite at Xcode 14 levels. When I enabled the suite that I had disabled for iOS 17, tests do go from ~8 minutes in Xcode 14.x / iOS 16 / Intel to 29 minutes in Xcode 15.0.1 / iOS 17.0.1 / Mac M1.
The difference is that this separate test suite does require a host app (they’re SKTestSession tests), whereas the other ones run without a host app.
And because of this performance issue, a bunch of tests fail due to timeouts: https://app.circleci.com/pipelines/github/RevenueCat/purchases-ios/15769/workflows/93f0d042-2a60-4b73-af15-54542ab496a6/jobs/134913
5 Likes
I don’t see that 15.0.1 is available an any instance classes (m1 large, m1 medium, or gen2 x86. Where are you see that it is available?
It seems that 15.0.0 is using that now. I noticed because it was no longer finding the iOS 17.0.0 simulators and needed to change it to 17.0.1
1 Like
I am not a CircleCI user, but also seeing Xcode 15 build time regression as well on other CI platform, following.
It looks like the jobs are not freezing anymore with 15.0.1 but performance seems to be up to 8x worse compared to Xcode 14.
Here’s a Xcode 14 vs Xcode 15 run side by side:
I’m building with xcode 15 command line under Mac M1 and get similar problem.
After some investigation, I found xcodebuild&clang may run with Intel arch (which can be found in Acitiviy Monitor).
So I force build with arch -arm64 xcodebuild and it works. Here is my full build time:
- xcode 14 (maybe also under Intel arch, I didn’t fallback for clean test) 1h
- xcode 15 (Intel arch) 2.5h
- xcode 15 (Apple arch) 20min.
ps. I also tried building with XCode GUI, and found it is in Apple arch.
I was skeptical but it does look like 15.0.1 has massively improved the issue for me.
Updating my list from above:
Xcode 14:
- Core package, build-for-testing - 3m 41s
- Core package, test-without-building (445 tests) - 1m 8s
- Release build (no tests, entire job) - 7m 28s
Xcode 15:
- Core package, build-for-testing - 5m 32s
- Core package, test-without-building (445 tests) - 42m 17s (!!!)
- Release build (no tests, entire job) - 9m 2s
Xcode 15.1 b1:
- Core package, build-for-testing - 3m 50s
- Core package, test-without-building (445 tests) - 19m 11s
- Release build (no tests, entire job) - 9m 21s
Xcode 15.0.1:
- Core package, build-for-testing - 3m 24s
- Core package, test-without-building (445 tests) - 2m 38s (!!!)
- Release build (no tests, entire job) - 7m 30s
This brings compile times back down to what they were on Xcode 14, or close. Test run times still seem to be about 2-3x slower than Xcode 14 but are still greatly improved.
Overall workflow time is important too, as we run most of these jobs in parallel. With 15.0.1 its about 2 minutes slower than Xcode 14. However, I still have some failing snapshot tests which run a lot slower when they fail (due to image diffing) so comparing to a passing build isn’t a completely fair comparison. Its possible that once I’ve got all tests passing that my overall workflow time might be fast enough that I can make the switch to Xcode 15.
1 Like
I was unable to actually select 15.0.1 as an option. What does the iOS sim have to be set to as well?
Yes, you need to use 17.0.1 instead of 17.0 for the simulator.
1 Like
I always get failures trying to preboot 15.0.1 on a macos.m1.large.gen1. This causes my entire build to fail. Are you also prebooting?
No, prebooting did not help me at all and just made things slower.
Got it working with iOS 17.0.1 and Xcode 15.0.1 and saw a large performance boost on m1 larges. Going to try with m1 medium and report back!
I may have spoken too soon - things seem better generally, but whilst trying to get my build all green I have seen one of the jobs hang on “Running Tests…” again for 7 minutes and counting, before I cancelled it.
1 Like
Yes I’m experiencing the same thing. It runs great on a m1 large (very costly), but times out on running tests on medium instances
Yeah I’ve been seeing a lot more flakiness and performance issues as well. I wonder if the medium instances just don’t have enough RAM.
If you look at the resource usage between the medium and large, its interesting that the CPU usage spiked and then tapered off on medium but large consistently spike per test runs it seems.
Also, RAM hit 75% and stayed there whereas RAM on the large grew kind of linearly.
Not sure if any CircleCI employees have any ideas on this. Maybe @alexazl
I just realised I was still using the large executor, which unfortunately invalidates my previous results. But the hang I saw was also on large.
Running again now with medium.