We recently upgraded our CI from Xcode 15.3 to Xcode 16 and now use the iOS 18 simulator to run our unit tests. We are seeing a large performance regression in our test times (takes twice the time as before).
I noticed a lot of CPU consumption related to the iOS simulator image that I didn’t observe with Xcode 15.3.
I checked also if the issue is specific to the iOS 18 simulator but it also happens when running with iOS 17.5 under Xcode 16.
Any ideas how to work around this?
We’re also seeing performance regressions in Xcode 16.0. Build and test times take considerably longer on CI, almost twice the time Xcode 15.4 takes.
Once, earlier when the Xcode 16.0 image was released from CircleCI, the same workflow took almost 12x the time of Xcode 15.4. That has since been reduced to 2x, but it’s still enough to force us to stay on Xcode 15.4 for now.
We didn’t try Xcode 16.1 yet to see if that improves the performance
We are seeing that too. First I tough it could be detox, but after running the tests locally in my machine I have the same performance as before running on iOS 17, 18.1, 18.2, so I believe the issue in in CircleCI xcode image config.
iOS Build (ios_build_detox job)
| Version Xcode image CircleCI | iOS version | Test Durations | Average Duration | % increase compared with current |
|---|---|---|---|---|
| 15.3.0 (current) | 17.4 | 21m 03s, 19m 30s, 21m 23s, 21m 52s, 23m 39s | 21m 26s | - |
| 16.0.0 | 17.5 | 24m 45s, 26m 5s | 25m 25s | +19% |
| 16.0.0 | 18.1 | 27m 2s | 27m 2s | |
| 16.1.0 | 18.1 | 27m 44s, 25m 31s, 30m 30s, 24m 40s, 25m 46s | 26m 50s | +25% |
iOS Detox test (ios_detox job)
| Version Xcode image CircleCI | iOS version | Test Durations | Average Duration | % increase compared with current |
|---|---|---|---|---|
| 15.3.0 (current) | 17.4 | 29m 15s, 28m 49s, 33m 09s, 28m 3s, 29m 58s | 29m 50s | - |
| 16.0.0 | 17.5 | 39m 29s, 35m 27s | 37m 27s | +26% |
| 16.0.0 | 18.1 | 56m 21s Some tests failed | 56m 21s | |
| 16.1.0 | 18.1 | 1h 15m 12s, 1h 2m 53s Some tests fail because of slow actions on this iOS version | 1h 9m 3s | +131% |
I’ve tested using xcode 16.1.0 image and m2pro.medium machine class and the issue persists. Waiting for xcode 16.2 to check if this would fix this performance regression or not.