Tests with "waitForExpectations" randomly failing after switching to Xcode 9

Build Environment Mobile
1

Am I the only one getting such fails? The tests run smoothly on my machine, and sometimes on the CI…often I have to rebuild several times in order to get them to pass.
The expectation timeouts aren’t unreasonable, and I’m speaking about tests that don’t do any network communication and that require roughly less then a second to be run.

2

I am running into the same problem when running a test like this:

    let _FILE_NAME = #file.components(separatedBy: "/").last ?? ""

    func testServiceCompletedUpdatesDataSourceToValueAndErrorToNilWhenServiceSucceeded() {
        let expectedResponse = CAIResponse()

        waitForServiceSuccess(response: expectedResponse)

        let parameters = mockDataSource.parameters(forFunction: MockResultsTableViewDataSource.InvocationKeys.updateFor)
        XCTAssertEqual(parameters[0] as? CAIResponse, expectedResponse)
        XCTAssertNil(parameters[1]) // Error

        print(_FILE_NAME, #function, " <- END .")
    }

    private func waitForServiceSuccess(response: CAIResponse?) {
        print(_FILE_NAME, #function, " <- START")
        let expectation = XCTestExpectation(description: "service completed - success")
        expectation.expectedFulfillmentCount = 1

        subject.serviceCompleted(response: response, completion: {
            expectation.fulfill()
            print(_FILE_NAME, #function, " - expectation: '\(expectation)' - fullfiled")
        })
        wait(for: [expectation], timeout: CompletionTimeout)
        print(_FILE_NAME, #function, " <- END")
    }

… which invokes this class function:

   func serviceCompleted(response: CAIResponse?, completion: (() -> Void)? = nil) {
        DispatchQueue.main.async {
            print(_FILE_NAME, #function, " <- START")
            print(_FILE_NAME, #function, "  - updating data source with response: \(String(describing: response))")
            self.resultsTableViewDataSource.updateFor(response: response, error: nil)
            print(_FILE_NAME, #function, "  - reloading data for table")
            self.tableView.reloadData()
            print(_FILE_NAME, #function, "  - reading speakable")
            self.readSpeakable()
            print(_FILE_NAME, #function, "  - calling copmletion")
            completion?()
            print(_FILE_NAME, #function, " <- END")
        }
    }

Runs just fine every time locally. But fails in circleci almost every time…

Instrumenting the test and production code tells me that the expectation actually does get fulfilled as shown in the following test output log despite the fact that the test is failing…

Test Case ‘-[SOCVoiceTests.ResultsViewControllerTests testServiceCompletedUpdatesDataSourceToValueAndErrorToNilWhenServiceSucceeded]’ started.
ResultsViewControllerTests.swift waitForServiceSuccess(response:) ← START
ResultsViewController.swift serviceCompleted(response:completion:) ← START
ResultsViewController.swift serviceCompleted(response:completion:) - updating data source with response: Optional(SOCVoice.CAIResponse(utterance: “”, speakable: “”, followUpQuestions: , plots: ))
ResultsViewController.swift serviceCompleted(response:completion:) - reloading data for table
ResultsViewController.swift serviceCompleted(response:completion:) - reading speakable
ResultsViewController.swift serviceCompleted(response:completion:) - calling copmletion
ResultsViewControllerTests.swift waitForServiceSuccess(response:) - expectation: ‘service completed - success’ - fullfiled
ResultsViewController.swift serviceCompletd(response:completion:.) ← END
<unknown>:0: error: -[ResultsViewControllerTests testServiceCompletedUpdatesDataSourceToValueAndErrorToNilWhenServiceSucceeded] : Asynchronous wait failed: Exceeded timeout of 2 seconds, with unfulfilled expectations: “service completed - success”.
ResultsViewControllerTests.swift waitForServiceSuccess(response:) ← END
ResultsViewControllerTests.swift testServiceCompletedUpdatesDataSourceToValueAndErrorToNilWhenServiceSucceeded() ← END .

I am using fastlane scan to run tests and the macos: xcode: “9.2.0” prebuilt machine on circleci.

3

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.