Run iOS unit tests in parallel with Xcode 10

ios

#1

I enabled Xcode 10’s new “Execute in parallel on Simulator” feature for our unit tests.

It works locally but I ran it multiple times on CircleCI and it never worked. Interestingly enough, it failed in different ways. Sometimes the clones of the simulator successfully booted, sometimes they did not. Usually I see an error message like this in the CircleCI log (even after they were successfully booted):

2018-10-02 18:48:11.556 xcodebuild[5745:20490] Error Domain=IDETestOperationsObserverErrorDomain Code=6 “Early unexpected exit, operation never finished bootstrapping - no restart will be attempted” UserInfo={NSLocalizedDescription=Early unexpected exit, operation never finished bootstrapping - no restart will be attempted, NSUnderlyingError=0x7fd584fbd8e0 {Error Domain=IDETestOperationsObserverErrorDomain Code=5 “Test runner exited before starting test execution.” UserInfo={NSLocalizedDescription=Test runner exited before starting test execution., NSLocalizedRecoverySuggestion=If you believe this error represents a bug, please attach the result bundle at /Users/distiller/Library/Developer/Xcode/DerivedData/MyApp-dpjkgwlmrmmihlcazfewmkuoufsf/Logs/Test/Test-MyApp-2018.10.02_18-42-32–0700.xcresult}}}

Testing failed:
MyApp.app (7896) encountered an error (Early unexpected exit, operation never finished bootstrapping - no restart will be attempted. Underlying error: Test runner exited before starting test execution.)
** TEST FAILED **

However, if I look at Xcode’s raw log, it sometimes contains logs of successfully executed tests (that don’t show up in the CircleCI log) but it never contains the message quoted above.

Is there anything we can do to get this feature working on CircleCI?


#2

Actually, I realized that these errors occur because I also turned on Xcode’s new “Randomize execution order” feature. Once I turn that off again, the parallel execution runs successfully!

However, CircleCI does not seem to be able to parse Xcode’s log output correctly, some of our UI screenshot tests fail with a random flake, and the test runs are only marginally faster than serial execution. :frowning:

Does anyone else have a different experience?