Severe performance problems with Xcode 15

Build Environment Mobile
42

15.1b is now available

Claims too fix what we think might be related to this issue:

Fixed: Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776)
3 Likes
43

Wonderful!! :rocket:

@alexazl do you know how long it should take for the CircleCI team to make the new Xcode 15.1b available to us? :bowing_man: and any way we can help expedite this process?

44

We would still have wait to for Apple to accept submission with Xcode 15.1
But if you are not use CircleCI to submit your apps this might really help.

1 Like
45

We have built the 15.1 image and are getting it distributed to our fleet. We will post an announcement with all the details of the image once it’s available (hopefully tomorrow). We also did some internal testing and it looks like simulator performance has improved with this new release!

6 Likes
46

The Xcode 15.1 beta 1 image is available!! You can find more details here.

47

Unfortunately I don’t think this made a difference.

Edit:
“Good news”: enabling all our tests now actually finishes! https://app.circleci.com/pipelines/github/RevenueCat/purchases-ios/15628/workflows/f3498857-569c-450b-a1f4-11c4067ba4bb/jobs/132370
Bad news: it takes 30+ minutes total (compared to 6 minutes), and because of the slow performance there’s lots of timeout failures.

2 Likes
48

We continue to see significant regression in duration on the 15.1 image. Our pr workflow takes about 20 minutes on 14.3 and 55 minutes on 15.1, both running 3597 tests.

49

I’m seeing similar results. Xcode 15.1 offers a slight improvement over 15.0 on macos.m1.medium.gen1 but is still too slow to be valuable or reliable for our testing.

2 Likes
50

The same result as mentioned above. I am observing timeout failure when running iOS tests with Xcode Xcode 15.0.0 and 15.1.0. tvOS tests are fine.

1 Like
51

Sadly I was unable to test the performance with the new image because all of our jobs now fail to build at all.

Our release build just spammed the console with this and then failed:

Function: createItemModels(for:itemModelSource:)
Thread:   <_NSMainThread: 0x600003460240>{number = 1, name = main}
Please file a bug at https://feedbackassistant.apple.com with this warning message and any useful information you can provide.
2023-10-10 12:57:59.710 xcodebuild[3629:12215] [MT] DVTAssertions: Warning in /AppleInternal/Library/BuildRoots/2b0aeecf-5b7c-11ee-840a-5239adce7ca0/Library/Caches/com.apple.xbs/Sources/IDEFrameworks/IDEFoundation/Provisioning/Capabilities Infrastructure/IDECapabilityQuerySelection.swift:103
Details:  createItemModels creation requirements should not create capability item model for a capability item model that already exists.

For my other jobs I’m now getting dependency errors that didn’t exist before:

error: The package product 'Dependencies' requires minimum platform version 10.15 for the macOS platform, but this target supports 10.13 (in target 'WebSocketClient' from project 'CommunityCore')

(I’m getting this for a lot of package targets)

It looks like its building for macOS instead of iOS?

52

Right, seems like the issues it that you need to specify an OS of 17.0.1 instead of 17.0, so I solved that problem.

So the results - TLDR; compile times seem to be improved, test times are improved but still unacceptably slow.

A few comparisons - I am only comparing successful jobs:

Xcode 14:

  • Core package, build-for-testing - 3m 41s
  • Core package, test-without-building (445 tests) - 1m 8s
  • Release build (no tests, entire job) - 7m 28s

Xcode 15:

  • Core package, build-for-testing - 5m 32s
  • Core package, test-without-building (445 tests) - 42m 17s (!!!)
  • Release build (no tests, entire job) - 9m 2s

Xcode 15.1 b1:

  • Core package, build-for-testing - 3m 50s
  • Core package, test-without-building (445 tests) - 19m 11s
  • Release build (no tests, entire job) - 9m 21s

So as you can see, compile times when building without testing are almost back to what they were with Xcode 14, which is good. The release build is still a bit slower but that could be due to changes in Swift 5.9 or the compiler itself.

Test time has halved from 15.0 but it’s still 19x slower than Xcode 14 and not an acceptable amount of time. I note that the job is still hitting a ceiling of 98% CPU usage and 83% memory usage once the tests start.

I will retest with the yeetd script above.

3 Likes
53

The yeetd shell script above just made things worse for me - the core package job timed out with no output and everything else was slower.

54

Hello :wave:

I just wanted to provide some details on a possible workaround for the performance regressions we’re seeing in the IOS 17+ Simulators that have shipped with Xcode 15:

While testing iOS 17 and 17.0.1 simulator performance, I found that on first boot the simulator started up multiple processes that would eat up almost 100% of all of the vCPU cores available to the VM. I did not observe this behaviour stopping until I forced a simulator restart by shutting it down:

# Shutdown the simulator - you'll need to identify the device ID of the simulator.
xcrun simctl shutdown $DEVICE_ID

# This command will start the simulator (-b) and block until it finishes the boot process
xcrun simctl bootstatus -b $DEVICE_ID

Following one (but sometimes two) successful reboots, I’ve observed the iOS simulator using significantly less resources, with CPU usage on all cores hovering somewhere in the 10-20% range, which is what I’ve seen in similar tests of iOS 16 simulators using Xcode 14 simulator runtimes.

These workarounds are certainly not ideal, but could help smooth things out in the short term. I’m curious to hear if they have a positive impact on the UI testing times we’re seeing above.

55

Unfortunately I’m not seeing any improvement with this - all it has done is add an extra 7m step to an already slow build. I’m running:

xcrun simctl bootstatus D2AB0F53-7E2B-4533-979B-50E8392760F7 -b
xcrun simctl shutdown D2AB0F53-7E2B-4533-979B-50E8392760F7
xcrun simctl bootstatus D2AB0F53-7E2B-4533-979B-50E8392760F7 -b

I have to boot it once and then shut it down before booting again as there’s nothing to shut down when the workflow first starts.

It has made no material difference to the way tests run - they still freeze. I think memory usage is the bigger problem here.

56

Now the workflow has finished, I saw a further 4 minute reduction in the test run but that’s still 15x slower than Xcode 14. I also see the CPU is no longer hitting a ceiling throughout the build. But the above script added 7 minutes to the workflow time itself and strangely, the compile step took 4x longer. I’m still seeing memory usage hitting a ceiling of 83% and staying there.

57

We’re continuing to look into this and possible workarounds, but our internal testing does not show that using Sonoma resolves the issue. It looks like we’re still waiting for Apple to fix the bug in their virtualization framework, but we’ll keep you updated if we find any other solutions.

58

Hello @alexazl! Could you please share more info about the “bug in their virtualization framework”? Facing the same issue while running the Xcode 15 within Tart VM - would like to know a bit more about this :pray: Thanks a lot for your support and updates!

59

@gleb-gusev Unfortunately, the Apple Virtualization Framework is a bit of a black box, so I don’t have many details around what exactly the issue is. In the the Xcode 15 release notes, Apple noted that there are a couple of known issues with Xcode 15 and the simulators:

  • Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776) (FB12237092)
  • The simulator may crash when opening Settings or Action Button settings on iPhone 15 Pro devices. (115388496)

As @cade noted above, he found that the first boot of the simulator started up multiple processes that would consume nearly all of the vCPU cores available to the VM. We’ve seen limited success in forcing a simulator restart by shutting it down:

# Shutdown the simulator - you'll need to identify the device ID of the simulator.
xcrun simctl shutdown $DEVICE_ID

# This command will start the simulator (-b) and block until it finishes the boot process
xcrun simctl bootstatus -b $DEVICE_ID

There was some hope that the first bug was fixed as part of the Xcode 15.1 beta release, but we have not seen evidence of it resolving the issue. I recommend following along with this Apple forum thread for more updates and encourage you to file a bug report with Apple. Our hope is that they will release a complete fix to the bugs soon and a large number of bug reports could increase their urgency around it. I will continue to provide updates in this thread, as well.

I know this is a painful experience and appreciate everyone’s patience, as well as willingness to share possible solutions and workarounds.

2 Likes
60

I think CPU usage is a red herring. I’m pretty sure that whatever the issue is, it’s causing memory to be exhausted. Once tests start running in our workflow it hits 83% usage and stays there.

Anecdotally I’ve heard people have had more luck with more memory, at least 12GB, maybe even 16GB.

Booting and killing the simulator did nothing to help.

Apple have just released 15.0.1 with the same simulator first launch fix that’s in 15.1 but this also does not fix the issue.

I think perhaps it might be time to consider increasing the amount of memory available on macOS executors, at least temporarily until this issue is fixed by Apple.

2 Likes
61

Apple announced 15.0.1 RC, with a potential fix:

Will CircleCI be adding that version to their system?