Xcode Exit Code 65 + UI Tests

Hey everyone :wave:t2:,

There have been a lot of issues lately related to iOS UI tests and new kinds of xcodebuild exit code 65. Some of them are related to you switching to Xcode 8 some are probably unrelated.

I do not have a sane explanation at this point for any of your issues, nor do I have a solution for you but I have a couple of ideas and the more data we gather the faster I can resolve this for you.

I will update this post tomorrow with further information for you and maybe some stuff for you to try.

Thank you for your patience, I want to resolve this as fast as I possibly can for you.

Thanks,
Constantin

4 Likes

Hey everyone,

Sorry for not updating in this in the last two days but I was in the process of setting up a controlled area within CircleCI in which we can test. Iā€™m almost done preparing everything and will update this again as soon as I have more information and things to try for you.

Best,
Constantin

@Tzeejay any update on this?

1 Like

No not yet. Sorry for asking for more patience. Iā€™m working full time on this one issue.

Constantin

Hey everyone,

I have been really sick all week so Iā€™m a bit behind schedule, please excuse that. I have great colleagues though and they helped out getting a sandbox ready for us to test and play in and get all you back to green builds.

If you can contact me directly via a support ticket, please use the DM function in discuss or @ me here.

Thanks for your patience,
Constantin

Hi,

Iā€™ve started testing some of your projects but ran into issues, those are not related to you at all and I think weā€™ll get them resolved today. If you run into issues with your builds please let me know!

Thanks,
Constantin

Hey,

Sorry for the missed update yesterday but I was finally able to get to work. Our system is not setup to serve the tasks I ask it to do so I have to hop through all kinds of weird hoops.

As a heads up I will have to create an API token with for your account. It will be called something along the lines of CIRCLECI_EMPLOYEE_TESTS.

I will gather information all day today and update this thread again tomorrow morning with the things Iā€™ve found. My current hunch is that you are running into issues with code signing dependencies. If youā€™d like to share it either here or through your support tickets, please let me know what kind of dependencies you have and how you get them (CocoaPods, Carthage, direct install into the project).

Thanks,
Constantin

Hey Constantin,

Thanks for the update. Our dependencies are React Native (downloaded via npm and then built directly from Xcode), a big bundled JS file, and a custom Go library also incorporated directly into the Xcode project.

Cheers,
Chris

Hi Constantin, I registered a support ticket too, but I would very much like to get this looked at with our builds too. Our dependencies are gym/deliver/match for deployment/provisioning, bitbucket for the match certificate repository, TTTAttributedLabel via Carthage and the following pods:

	pod 'BlocksKit', '~> 2.2.5'
	pod 'IGHTMLQuery', '~> 0.8'
	pod 'AFNetworking', '~> 3.0'
        pod 'AFNetworkActivityLogger', git: 'https://github.com/AFNetworking/AFNetworkActivityLogger.git', branch: '3_0_0'
	pod 'OHHTTPStubs'
	pod 'BKPasscodeView', '~> 0.1.2'
	pod 'UICKeyChainStore'
	pod 'UIColor-HexRGB', '~> 1.0.1'
	pod 'Masonry', '~> 0.6.2'
	pod 'Stencils'
	pod 'ReactiveCocoa', '~> 2'
	pod 'RMStore', '~> 0.7'
	pod 'RMStore/AppReceiptVerificator'
	pod 'RMStore/TransactionReceiptVerificator'
	pod 'RMStore/KeychainPersistence'
	pod 'RMStore/NSUserDefaultsPersistence'
        pod 'Underscore.m', '~> 0.3'

        pod 'OCMock', '~> 3.1.2'
            pod 'Specta', '~> 1.0.3'
            pod 'Expecta', '~> 1.0.2'

We only use CocoaPods in our project.

We are using only using Carthage along with the following dependencies:

github ā€œbignerdranch/Freddyā€ ~> 2.1
github ā€œjdg/MBProgressHUDā€ ~> 1.0.0
github ā€œQuick/Nimbleā€ "v4.1.0"
github ā€œQuick/Quickā€ ā€œv0.9.3ā€

Additionally, we have manually added the GoogleFirebase API to our codebase.

Hello ,

we have the same error sometimes with Xcode 8 on circle with those dependecies:

target :'XXX Tests' do
  pod 'Specta', :git => 'https://github.com/specta/specta.git'
  pod 'Expecta',     '~> 1.0.5'   # expecta matchers
  pod 'OCMock',      '~> 3.3'   # OCMock
end

target :'XXXUITests' do
  pod 'Specta', :git => 'https://github.com/specta/specta.git'
  pod 'Expecta',     '~> 1.0.5'   # expecta matchers
  pod 'OCMock',      '~> 3.3'   # OCMock
  pod 'Bugsnag'
  pod 'FMDB'
end

Hey everyone,

Thanks a lot for the little insight into your stacks.

Iā€™m in the process of preparing our OS X image that has Xcode 8.1 beta 3 in it and Iā€™d like to ask everyone to set aside a branch named e65-tests which tests again Xcode 8.1. In order to do so please just change your circle.yml to look like this:

machine:
  xcode:
    version: "8.1"

If you have the branch I will find it and take a look at your builds. I will look for it :slight_smile:

Thanks for helping me helping you,

Constantin

1 Like

Branch created and build launched :wink:

But the error is thrown:

Xcode version ā€˜8.1ā€™ is not available. Available versions are: 7.0, 7.2, 7.3, 8.0

@ben-j69 of course. @anon47539524 will be running it on a private machine with beta Xcode8.1, not on the Circle public machine pool. Cheers, Pavel

Hey,

Sorry for not being as explicate as I should have been. As @pavel-zdenek correctly stated we do not have this version Xcode deployed to any of our production servers. My private machine incorporates into our system but everything is more of a hack than anything else at this point since we never tried to run such a test before. I will turn on the Xcode 8.1 feature setting on shortly and will update you on further progress :slight_smile:

Thanks for your patience, Iā€™m working as fast as I can.

Best,
Constantin

Branch created

https://circleci.com/gh/kitt-browser/adblockplus-ios/1174

Best regards,
Pavel

We would love to take a look at the root cause(s) here. Can you file radars with sysdiagnose, ~/Library/Logs/CoreSimulator, and ~/Library/Developer/Xcode/DerivedData/[project-subfolder]/Logs

Even if they are duplicates extra logs help a lot.

sysdiagnose requires root so I assume end users canā€™t run it themselves but it might be worth having a setting to collect this data on failure so people can more easily file bugs in the future.

Hey everyone,

I think everyone has seen the issues we were facing yesterday so Iā€™m sorry for more delay on this issue.

I will update this once I know more.

Best,
Constantin

1 Like

@russbishop ~/Library/Developer/Xcode/DerivedData/ does not contain subfolder ā€œLogsā€ and ~/Library/Logs/CoreSimulator contains an EMPTY subfolder structure ā€œsome-uuid/CrashReporter/DiagnosticLogsā€. Somewhat logically empty, as the simulator did not even start due to failed framework loading (insisting on having a signed one - for a simulator run?). Does it still make sense to fill a radar with sysdiagnose?

1 Like