Is anyone else having performance issues with Xcode 15? Our
test-without-building steps have taken a noticeable hit and it’s unclear if this is due to an Xcode issue or a problem with the Circle CI images/environment.
For comparison, here’s the build and test times for our job that builds and tests our features package on the previous iOS version, so for Xcode 14 that is iOS 15:
And the same job but run against the iOS 16 simulator:
build-for-testing: 3m 53s
test-without-building: 1m 36s
Tests are a bit slower as they include snapshot tests which we disable for the previous iOS version test suite.
Now for Xcode 15, here’s the job for the same package using the iOS 16 simulator (snapshot tests disabled):
build-for-testing: 6m 34s
test-without-building: 32m 30s
That is not a typo - that really is 60x slower (and twice as slow to compile).
The tests against iOS 17 don’t currently pass but the overall run is still slow:
build-for-testing: 10m 36s
test-without-building: 37m 46s
This is really not acceptable - if its an issue with Xcode then fingers crossed it gets fixed in the final release candidate, but if this problem is with Circle CI and cannot be resolved, this likely be enough motivation for us to terminate and move to a competitor service. We have 6 core jobs in our workflow and the whole thing takes on average around 11-14 minutes on Xcode 14. This has regressed to anywhere from 30-60 minutes on Xcode 15.
I have been unable to reproduce this problem locally - I downloaded the build and test report and inspected it in Xcode and it looks like random tests that would normally take < 0.01s to run are taking over a minute, which obviously adds up over the course of an entire suite. I do not see this problem when running the tests on my machine. I have opened a support ticket.
Yup same problem, our tests went from taking 10 minutes to 70 minutes. I haven’t found a solution yet
Another symptom I’m seeing is tests or builds simply timing out with:
Too long with no output (exceeded 10m0s): context deadline exceeded
@aboatti are you seeing this only on Circle? Is everything working locally for you?
Hey, I’m not getting that timeout issue, but yes, locally all the tests run quickly as ever.
Hi, we’re seeing the same issue. Build times take at least twice as much with Xcode 15 (48 minutes compared to around 23), with the build timing out in the web interface. The builds do however complete on the runner (verified by checking logs when SSH’d into the runner).
I didn’t think to try this using SSH - I re-ran a failed job so I could SSH in and can confirm that I can reproduce the issue of
test-without-building just hanging after it prints “Testing started” by running it from there. This does seem to indicate a problem with the CircleCI image. I’ve still not had a reply to my support ticket.
Also compared to the 14.x enviroments, in the 15.x image I can see a new process “update_dyld_shared_cache” taking up more than 2400M+ of memory. Does this also happen for you? It seems to be using more CPU time than the XCBBuildService process.
Yes, I also saw this. If I tried to kill it, it would spawn a new one. If I ran top, it also indicated that both that process and
diskimagesiod were “stuck”.
I see the exact same issue. I can’t post links, but I have an example where after 30 minutes they just failed with no output after 15 minutes.
This does not happen on any other Xcode, and we’ve used Xcode 13,14, and run tests on iOS 12 through 16.
We are working on the Xcode 15 RC image and are hopeful that it resolves some of the issues we’re seeing with the beta version. This appears to be an upstream issue, which gives us more confidence that it will be resolved with the RC release (and subsequent official release). I will be sure to post an update when the RC image is available. Thank you for your patience!
@lukeredpath you mentioned that “The tests against iOS 17 don’t currently pass”, curious if you’re seeing issues where tests pass locally but not on CI? We’ve definitely had a few and UI tests seem to be impacted the most and are now extremely flakey, but pass locally every time.
Unfortunately until these performance issues are resolved it’s hard for me to know if we have any genuine issues with our tests on CI. I think I’ve got everything passing locally now so as soon as I can run a successful build on CI I should know more. We don’t currently run any UI test suites.
Will CircleCI make RC1 available today ? Assuming based on the pattern of releases with the Xcode 15 betas.
Would be great if someone from CircleCI can confirm. Our workflows been severely affected since we switched to Xcode 15 beta 8 because of flakiness in tests. As others mentioned all tests works fine locally.
We are working on the Xcode 15 RC image, but ran into a couple issues. Our aim is that it will be available by early next week, if not today. We apologize for the delay and appreciate your patience.
Is there any update on the new image?
@lukeredpath The image has been built last week, we are waiting on deployment the be finalised on the infra side, which I would hope will be today (@alexazl might know more)
The Xcode 15 RC image is now available. For visionOS users, it’s important to note that Xcode 15 RC did not include the visionOS SDK. If you are building with visionOS, please use the
15.0.8 image tag to point your project to Xcode 15 Beta 8, which includes the latest release of visionSDK Beta.
More details about the RC image can be found here. Thank you for your patience!
on my first runthrough, this does not seem to have solved the slow tests problem
@aboatti Can you please submit a support ticket with links to your builds so we can investigate further? Thank you.