Still no improvement here. I already have a ticket open. Build time seems worse (almost 3x as slow) and it still seems to just be hanging waiting for the tests to start.
One observation: I removed the step that pre-boots the simulator before building and that seems to have made a noticeable improvement to build time performance, almost back to Xcode 14 times in most cases.
However it remains the case that the tests simply do not run once it reaches the test step, it just hangs there after “Testing Started”. There seems to be a serious problem with the simulator in a virtualised environment.
Edit - some of my test jobs did eventually start running but were still very slow, it looks like intermittent tests that should take fractions of a second still just pause and take seconds (or up to a minute sometimes), resulting in the whole thing taking nearly 20 minutes (it should take 30s).
I’m going to submit a support ticket with an example build.
iOS 16 tests on Xcode 14.3: 1926 tests run in 6min 11s on macos.x86.medium.gen2
iOS 17 tests on Xcode 15 RC1: 1746 tests (fewer because one subset of tests just gets stuck so we’ve disabled those for now 
) run in 11min on macos.m1.medium.gen1.
That’s basically twice as slow and in what’s supposed to be a much faster (and expensive) image. Something is seriously wrong.
2 Likes
I also submitted a ticket with a few links, this is causing the Too long with no output (exceeded 10m0s): context deadline exceeded fails, together with failures in tests that use polling or async expectations and timeouts (as the timeouts fire).
I’m guessing that the image does not have the dyld cache in it and the creation process is stalling the tests.
Thanks!
After some time investigating our builds using the macos.m1.medium.gen1 resource class and also inspecting local builds I found that even for a trivial Xcode project the test suite never launches.
While attempting to debug via SSH any operations through xcrun simctl spawn take at least several seconds to return, if more than one simulator is booted this can be as long as a minute.
The workaround for now seems to be to use macos.m1.large.gen1, however as noted by others runtime here is about double what I observed on medium resources with Xcode 14.
The Xcode 15 release notes mention at least the slow simulator launch performance as a known issue, however it is unclear to me which resource is the limiting factor here since neither CPU or RAM consumption seemed unusual.
- Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776) (FB12237092)
I’ve also submitted a ticket regarding this issue with a link to the build.
Thanks
@flufff42 Yes this looks like a known issue as per the release notes 
There is also a thread on the Apple Developer Forum:
https://developer.apple.com/forums/thread/735177
Unfortunately there does not appear to be a solid workaround as this is a direct problem with the simulator runtimes. Throwing resources at the problem will likely help, but it is not a true workaround - plus like you said, it is not clear why this helps when there does not seem to be a problem with CPU or memory pressure.
We can only hope that Apple releases a patch that resolves this, or fixes it in Xcode 15.1. As they (still) do not provide public bug tracking it is impossible to know if there is a timeline for this.
I will add this information to the release notes for future reference.
The stalls seem to be unrelated with the this, right? One thing is that the simulator takes an extended period of time to launch, and another one that the job freezes in the middle and times out.
I haven’t noticed the simulator issue locally, but the Xcode 15 image is completely unusable for us at this time.
I agree that there could be something wrong with Xcode, but I think there needs to be something else that could be improved in the image. Maybe the way Apple is now distributing the SDKs (independently) is interacting with the way that CircleCI creates the images.
One user mentions that they see the process that updates the dyld cache running, this should be avoided by making sure it’s cached into the image CircleCI generates.
As for the simulator warmup time, not sure if it could also be mitigated by running the simulators and a sample test suite before generating the image.
Thanks!
@0xpablo we are investigating this but are not coming up with any definitive answer. Personally I have been unable to trigger the dyld cache issue. As for how we build the images, this has been unchanged since beta 1 but it seems the performance issues crept in around beta 8 which was the last beta before RC. The image build process is practically the same as how we built the Xcode 14 images, other than adding support for installing visionOS SDK where necessary (this is not included in the RC release). As a side note, we are hoping to open source the majority of the image build pipeline in the near future, so hopefully transparency around how the images are built, and and changes between versions, can assist in debugging these kinds of issues.
1 Like
Thanks for the details, it’s great news that you are open sourcing part of the image creation pipeline.
I will try to gather some diagnostics tomorrow when reproducing the issue in case it helps.
1 Like
Hello folks!
The macOS Infra team at CircleCI has been investigating the reports of poor performance when using Apple’s iOS simulators in the virtualized macOS environment. Thus far we haven’t been able to identify any viable workarounds. We will continue to experiment with the environment and if we identify any potential solutions, we’ll share our findings.
As noted in the thread above, the Xcode 15 release notes indicate a couple of known issues with Xcode 15 and the simulators:
- Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776) (FB12237092)
- The simulator may crash when opening Settings or Action Button settings on iPhone 15 Pro devices. (115388496)
Also, please keep in mind that the Apple Silicon executors are a virtualized environment, and aren’t running on bare metal Apple Silicon. This virtualization technology (the Apple Virtualization Framework) may ultimately be the root cause of the problems we’re seeing here with the simulators, in which case we encourage everyone to submit bug reports to Apple regarding simulator startup and runtime issues.
3 Likes
I noticed on the forum post that you linked to, it mentioned that the Sonoma RC may fix the simulator issue. I wanted to try this out and see but I couldn’t find how to specify in the config file to use Sonoma. Is this not possible or am I just overlooking some configuration setup? Do we have to wait until Friday, for the official release, before we can try our builds out using Sonoma on CircleCI?
Thanks. 
2 Likes
We are also experiance huge performance issues with Xcode 15 but not with unit test but XIB complication:
[11:29:53]: ▸ Compiling <snip>.xib
Too long with no output (exceeded 10m0s): context deadline exceeded
Locally this does not take long at all, is there some setting in the XIB we missed?
Using Xcode 15 on M1 resource classes completely disrupts our workflow (no output for 20m in a test that used to complete in 1m), but we’ve observed that Xcode 15 seems to cause less degradation (~2x) on an Intel executor (macos.x86.medium.gen2), which at least keeps it running.
I know it’s hard to tell when Apple will fix it or any workaround will be found. If using Intel machines can avoid or mitigate the performance issue, is there a possibility that CircleCI might change the timeline and/or brownout schedule for the deprecation?
Has anyone given this a try? GitHub - biscuitehh/yeetd: yeetd is a lil' daemon that watches for specific CPU-intensive Simulator processes and stops them in their tracks!
3 Likes
Any updates on this? We are looking to update but blocked, we are also experiencing the timeout with not output when running tests, locally works fine and Xcode 14.3 works fine. Can’t attach links but you can see the pr here: github/artsy/eigen/pull/9323
We are exploring a couple possible workarounds and seeing if there are ways to minimize the performance degradation on our end, while we wait for Apple to release an official fix. One workaround that we’ve found that may help is shutting down the simulator and rebooting it. The command to use for this is xcrun simctl shutdown all. We can’t guarantee that this will improve performance for everyone, but our internal testing has been promising. If we discover any other potential workarounds, we will post them here.
1 Like
@alexazl Is there a timeline for when Xcode 15 jobs will be run on Sonoma? As @donnydavis noted above, one of the posts on the Apple forum thread (https://developer.apple.com/forums/thread/735177) suggests that the issue might be solved by Sonoma.
@jpeckner We typically wait until Xcode XX.2 to update the OS version, but we are looking into doing it sooner given it could potentially resolve this issue. I will post an update once I have confirmation and a timeline.
5 Likes
Turning the Yeetd approach into a simple shell script has looked promising, as a temporary workaround.
while true
do
pkill AegirPoster
pkill InfographPoster
pkill CollectionsPoster
pkill ExtragalacticPoster
pkill KaleidoscopePoster
pkill EmojiPosterExtension
pkill AmbientPhotoFramePosterProvider
pkill PhotosPosterProvider
pkill AvatarPosterExtension
pkill GradientPosterExtension
pkill MonogramPosterExtension
pkill apsd
sleep 5
done
1 Like