Is anyone else having performance issues with Xcode 15? Our test-without-building steps have taken a noticeable hit and it’s unclear if this is due to an Xcode issue or a problem with the Circle CI images/environment.
For comparison, here’s the build and test times for our job that builds and tests our features package on the previous iOS version, so for Xcode 14 that is iOS 15:
build-for-testing: 3m
test-without-building: 32s
And the same job but run against the iOS 16 simulator:
This is really not acceptable - if its an issue with Xcode then fingers crossed it gets fixed in the final release candidate, but if this problem is with Circle CI and cannot be resolved, this likely be enough motivation for us to terminate and move to a competitor service. We have 6 core jobs in our workflow and the whole thing takes on average around 11-14 minutes on Xcode 14. This has regressed to anywhere from 30-60 minutes on Xcode 15.
I have been unable to reproduce this problem locally - I downloaded the build and test report and inspected it in Xcode and it looks like random tests that would normally take < 0.01s to run are taking over a minute, which obviously adds up over the course of an entire suite. I do not see this problem when running the tests on my machine. I have opened a support ticket.
Hi, we’re seeing the same issue. Build times take at least twice as much with Xcode 15 (48 minutes compared to around 23), with the build timing out in the web interface. The builds do however complete on the runner (verified by checking logs when SSH’d into the runner).
I didn’t think to try this using SSH - I re-ran a failed job so I could SSH in and can confirm that I can reproduce the issue of test-without-building just hanging after it prints “Testing started” by running it from there. This does seem to indicate a problem with the CircleCI image. I’ve still not had a reply to my support ticket.
Also compared to the 14.x enviroments, in the 15.x image I can see a new process “update_dyld_shared_cache” taking up more than 2400M+ of memory. Does this also happen for you? It seems to be using more CPU time than the XCBBuildService process.
Yes, I also saw this. If I tried to kill it, it would spawn a new one. If I ran top, it also indicated that both that process and diskimagesiod were “stuck”.
I see the exact same issue. I can’t post links, but I have an example where after 30 minutes they just failed with no output after 15 minutes.
This does not happen on any other Xcode, and we’ve used Xcode 13,14, and run tests on iOS 12 through 16.
We are working on the Xcode 15 RC image and are hopeful that it resolves some of the issues we’re seeing with the beta version. This appears to be an upstream issue, which gives us more confidence that it will be resolved with the RC release (and subsequent official release). I will be sure to post an update when the RC image is available. Thank you for your patience!
@lukeredpath you mentioned that “The tests against iOS 17 don’t currently pass”, curious if you’re seeing issues where tests pass locally but not on CI? We’ve definitely had a few and UI tests seem to be impacted the most and are now extremely flakey, but pass locally every time.
Unfortunately until these performance issues are resolved it’s hard for me to know if we have any genuine issues with our tests on CI. I think I’ve got everything passing locally now so as soon as I can run a successful build on CI I should know more. We don’t currently run any UI test suites.
Will CircleCI make RC1 available today ? Assuming based on the pattern of releases with the Xcode 15 betas.
Would be great if someone from CircleCI can confirm. Our workflows been severely affected since we switched to Xcode 15 beta 8 because of flakiness in tests. As others mentioned all tests works fine locally.
We are working on the Xcode 15 RC image, but ran into a couple issues. Our aim is that it will be available by early next week, if not today. We apologize for the delay and appreciate your patience.
@lukeredpath The image has been built last week, we are waiting on deployment the be finalised on the infra side, which I would hope will be today (@alexazl might know more)
The Xcode 15 RC image is now available. For visionOS users, it’s important to note that Xcode 15 RC did not include the visionOS SDK. If you are building with visionOS, please use the 15.0.8 image tag to point your project to Xcode 15 Beta 8, which includes the latest release of visionSDK Beta.
More details about the RC image can be found here. Thank you for your patience!