Hey, wondering if anyone else has run into test falkiness issues when it comes to Xcode 15 beta on CircleCI? I’ve noticed one or two test failures running locally, but running on CI always seems to produce flaky tests, which was not an issue with Xcode 14 on Circle. I’m running M1 on both CI and on local, so no differences there. Curious to see other people’s experiences.
We are facing similar issues. Have to retry tests several times. @mapierce have you found any workaround ?
Aside from the performance problems I know there have been some changes to the concurrency runtime that could cause flakiness - are your tests using swift concurrency?
Not to the best of my knowledge. I’m mostly talking about UI tests. Strange things happening like taps and swipes not working. Do you have any docs I could look up around changes to Swift concurrency runtime to rule it out?
I don’t, but if you’re talking about UI tests then it won’t be that.
@lukeredpath We are using Swift Concurrency in our tests. But flakiness is mostly from expectations where in most cases we don’t have any usage of Swift concurrency.
Beta 8 indicates there was some changes to expectations, but locally things works perfectly for us.
Our issues around flaky tests has been resolved with the new Xcode 15 RC image.