Still no improvement here. I already have a ticket open. Build time seems worse (almost 3x as slow) and it still seems to just be hanging waiting for the tests to start.
One observation: I removed the step that pre-boots the simulator before building and that seems to have made a noticeable improvement to build time performance, almost back to Xcode 14 times in most cases.
However it remains the case that the tests simply do not run once it reaches the test step, it just hangs there after “Testing Started”. There seems to be a serious problem with the simulator in a virtualised environment.
Edit - some of my test jobs did eventually start running but were still very slow, it looks like intermittent tests that should take fractions of a second still just pause and take seconds (or up to a minute sometimes), resulting in the whole thing taking nearly 20 minutes (it should take 30s).
I’m going to submit a support ticket with an example build.
iOS 16 tests on Xcode 14.3: 1926 tests run in 6min 11s on macos.x86.medium.gen2
iOS 17 tests on Xcode 15 RC1: 1746 tests (fewer because one subset of tests just gets stuck so we’ve disabled those for now 
) run in 11min on macos.m1.medium.gen1.
That’s basically twice as slow and in what’s supposed to be a much faster (and expensive) image. Something is seriously wrong.
2 Likes
I also submitted a ticket with a few links, this is causing the Too long with no output (exceeded 10m0s): context deadline exceeded fails, together with failures in tests that use polling or async expectations and timeouts (as the timeouts fire).
I’m guessing that the image does not have the dyld cache in it and the creation process is stalling the tests.
Thanks!
After some time investigating our builds using the macos.m1.medium.gen1 resource class and also inspecting local builds I found that even for a trivial Xcode project the test suite never launches.
While attempting to debug via SSH any operations through xcrun simctl spawn take at least several seconds to return, if more than one simulator is booted this can be as long as a minute.
The workaround for now seems to be to use macos.m1.large.gen1, however as noted by others runtime here is about double what I observed on medium resources with Xcode 14.
The Xcode 15 release notes mention at least the slow simulator launch performance as a known issue, however it is unclear to me which resource is the limiting factor here since neither CPU or RAM consumption seemed unusual.
- Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776) (FB12237092)
I’ve also submitted a ticket regarding this issue with a link to the build.
Thanks
@flufff42 Yes this looks like a known issue as per the release notes 
There is also a thread on the Apple Developer Forum:
https://developer.apple.com/forums/thread/735177
Unfortunately there does not appear to be a solid workaround as this is a direct problem with the simulator runtimes. Throwing resources at the problem will likely help, but it is not a true workaround - plus like you said, it is not clear why this helps when there does not seem to be a problem with CPU or memory pressure.
We can only hope that Apple releases a patch that resolves this, or fixes it in Xcode 15.1. As they (still) do not provide public bug tracking it is impossible to know if there is a timeline for this.
I will add this information to the release notes for future reference.
The stalls seem to be unrelated with the this, right? One thing is that the simulator takes an extended period of time to launch, and another one that the job freezes in the middle and times out.
I haven’t noticed the simulator issue locally, but the Xcode 15 image is completely unusable for us at this time.
I agree that there could be something wrong with Xcode, but I think there needs to be something else that could be improved in the image. Maybe the way Apple is now distributing the SDKs (independently) is interacting with the way that CircleCI creates the images.
One user mentions that they see the process that updates the dyld cache running, this should be avoided by making sure it’s cached into the image CircleCI generates.
As for the simulator warmup time, not sure if it could also be mitigated by running the simulators and a sample test suite before generating the image.
Thanks!
@0xpablo we are investigating this but are not coming up with any definitive answer. Personally I have been unable to trigger the dyld cache issue. As for how we build the images, this has been unchanged since beta 1 but it seems the performance issues crept in around beta 8 which was the last beta before RC. The image build process is practically the same as how we built the Xcode 14 images, other than adding support for installing visionOS SDK where necessary (this is not included in the RC release). As a side note, we are hoping to open source the majority of the image build pipeline in the near future, so hopefully transparency around how the images are built, and and changes between versions, can assist in debugging these kinds of issues.
1 Like
Thanks for the details, it’s great news that you are open sourcing part of the image creation pipeline.
I will try to gather some diagnostics tomorrow when reproducing the issue in case it helps.
1 Like
Hello folks!
The macOS Infra team at CircleCI has been investigating the reports of poor performance when using Apple’s iOS simulators in the virtualized macOS environment. Thus far we haven’t been able to identify any viable workarounds. We will continue to experiment with the environment and if we identify any potential solutions, we’ll share our findings.
As noted in the thread above, the Xcode 15 release notes indicate a couple of known issues with Xcode 15 and the simulators:
- Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776) (FB12237092)
- The simulator may crash when opening Settings or Action Button settings on iPhone 15 Pro devices. (115388496)
Also, please keep in mind that the Apple Silicon executors are a virtualized environment, and aren’t running on bare metal Apple Silicon. This virtualization technology (the Apple Virtualization Framework) may ultimately be the root cause of the problems we’re seeing here with the simulators, in which case we encourage everyone to submit bug reports to Apple regarding simulator startup and runtime issues.
3 Likes
I noticed on the forum post that you linked to, it mentioned that the Sonoma RC may fix the simulator issue. I wanted to try this out and see but I couldn’t find how to specify in the config file to use Sonoma. Is this not possible or am I just overlooking some configuration setup? Do we have to wait until Friday, for the official release, before we can try our builds out using Sonoma on CircleCI?
Thanks. 