Severe performance problems with Xcode 15

Yes, I was seeing the same thing. Medium memory usage hits 83% for me and stays there. Re-testing 15.0.1 on medium now. I suspect 6GB just isn’t enough memory.

I spoke too soon indeed. 15.0.1 makes no difference at all when running on the medium executors. It just hangs and times out after 10 minutes of no output.

Yeah this is exactly what I experienced. Large executors are about 4x faster than on 15.0.0 but medium still hangs

I am facing the same issue. I tried both Xcode15.0.0 and Xcode15.0.1 both are having timeout issue.

Xcode15.0.0 - x86.medium.gen2 - Time out after 30min
Xcode5.0.1 - medium.gen1 : Time out after 30min.

Exact error : Too long with no output (exceeded 30m0s): context deadline exceeded

the new xcode hasn’t helped at all

Got past the hang but 15.0.1 still crashes when running tests.

So I’ve only had it successful work on an m1 large instance. Medium instances still time out for me.

Xcode version: Xcode 15.0.1
iOS version: iOS 17.0.1
Machine: macos.m1.large.gen1

My first tests were successful (I wonder if it’s because I was one of the first to use these new images?), but now I’m completely unable to get a successful run using Xcode 15.0.1 without timing out like before.

I had the same issues with medium resource class. I was able to get to run tests with below setup

Xcode 15.0.1 / iOS 17.0.1 / m1.large.gen1

However, for my other project Im getting Test runner never began executing tests after launching error. Pre-launching the simulator solves this issue but takes 5x the time and always CPU and RAM usage peaks during the whole time.

1 Like

One thing I can’t understand is when I SSH into the same machine and run the test - all seems fine and no hangs or whatsoever.

@alexazl My team is also seeing our test suite timeout with Xcode 15.0.1, both on macos.x86.medium.gen2 and macos.m1.medium.gen1 (whereas it ran fine before with 15.0.0 on macos.x86.medium.gen2). As a workaround for now, could CircleCI give us the option to choose between 15.0.0 vs 15.0.1? I know that’s not the usual patch policy, but this is a very unusual situation.

Just to add my pain story to the discussion, we are having the same issues on a different CI platform. Xcode 15 and 15.0.1 all time out when running tests. It can build and archive perfectly, but the run test freezes at the beginning and the CI times out at 30 min.
I’ve tried different solutions like pre-launching simulator, forcing arm64 arc for xcodebuild, even disabling most of the test suite and just leaving a few enabled, but nothing helps.
One run the tests actually started after about 15 minutes, but each test was taking a long time to run, with some taking over 12s when they should have taken milliseconds to complete.
Following this thread and hoping for new Xcode updates soon.

I’m also facing this issue with build that take double the time on Xcode 15.0.1 macos orb 2.4.0.

I tried investigating our pipeline to no avail… However during my testing, I tried running our pipeline through ssh and catching the simulator diagnostics through xcrun simctl diagnose -l

In my CoreSimulator.log I found that line that repeats for ~10 minutes :

Oct 25 09:19:16 distillers-Virtual-Machine CoreSimulatorService[6301] <Error>: Failed to get a device identity for simulator device 7601BC78-84E1-40A1-BA43-8A5E74559837, error: Error Code=-4 "UCRT is unavailable." UserInfo={NSLocalizedDescription=UCRT is unavailable., NSUnderlyingError=0x60000033bc90 {Error Code=-4 "UCRT is unavailable." UserInfo={NSLocalizedDescription=UCRT is unavailable.}}}

But I don’t know what UCRT is :man_shrugging:

The simctl_list.log in the simulator diagnostics has a line that reads iPhone 15 Pro (7601BC78-84E1-40A1-BA43-8A5E74559837) (Booted) which confirms that the simulator is booted

I hope this helps

Just voicing that this is also an issue for us using a combination of Github actions + Fastlane. Total build and test time went from ~30 minutes to over an hour with most tests simply failing or the entire process hanging.
Xcode 15.0.1, iOS 17.0.1

Generally in this context, UCRT stands for Universal C Runtime.

1 Like

@jpeckner Unfortunately, we’re unable to offer both 15.0.0 and 15.0.1 at this time but we’d love to look more into the issues you’re experiencing on macos.x86.medium.gen2. Can you open a support ticket and provide links to the builds that are timing out?

Facing same issue for our org as well. Unable to move to Xcode 15 on the CI.

Created support ticket with links to builds.

This is also blocking upgrade for one of our projects. We do have another project which upgraded successfully, but its test suite is much smaller and even then it took a 40% hit in build times after the upgrade.

@alexazl Sorry for the delay replying, but good news- we’ve figured out a workaround for this! It’s not a true fix for the underlying issue, but it’s at least got our unit test targets running again on CircleCI, both on macos.x86.medium.gen2 and macos.m1.medium.gen1.

Before all this, our unit test targets had our app set as the Host Application. It turns out the workaround is to set that value to None:

I’ve actually always favored setting this value to None (because unit tests should be focused on individual units, rather than the app as a whole), but shifting to that took a good amount of effort. Having no host application means that you can’t use @testable import MyApp imports; instead, for each file needed by the test targets, you have to check them off in the Target Membership section in the right-hand pane in Xcode:

Screenshot 2023-10-27 at 3.10.39 PM

(Fortunately, you can highlight multiple files in the left-hand Project Navigator, and then bulk-check them on the right-hand pane, but it still took a lot of clicks to get there :joy:)

A couple of additional caveats:

  • While this approach is working for our two unit test targets, it very likely won’t work for UI test targets, since IIRC those need a host application to work
  • With macos.x86.medium.gen2, there’s still a gap of slightly over seven minutes between when the code finishes compiling and when the tests actually run. (With macos.m1.medium.gen1, it’s 15 minutes :scream:)

So there’s a still a ways to go before this issue is fully fixed- most notably eliminating the idle gaps which burn through our CircleCI credits- but in the meantime I hope this helps someone.

1 Like

Yes I pointed out this mainly affects test targets with a host app

Some tests require host apps though (UI, SKTestSession, etc).

1 Like