Hi,
When running iOS UI test locally all tests are passing but when running those tests in circleci(using fastlane scan command) most of the time same tests fails randomly. Current Xcode version is 11.7 but same issue was present in older Xcode versions as well.
See error log below -
** TEST FAILED **
ERROR [2020-09-30 06:19:09.36]: Exit status: 65
±-------------------±—+
| Test Results |
±-------------------±—+
| Number of tests | 90 |
| Number of failures | 1 |
±-------------------±—+
WARN [2020-09-30 06:19:09.61]: Lane Context:
INFO [2020-09-30 06:19:09.61]: {:DEFAULT_PLATFORM=>:ios, :PLATFORM_NAME=>:ios, :LANE_NAME=>“ios ui_tests”, :SCAN_DERIVED_DATA_PATH=>"/Users/distiller/Library/Developer/Xcode/DerivedData/PattyMelt-govyngjkjnchnebcjtcmvxpmprwm", :SCAN_GENERATED_PLIST_FILES=>, :SCAN_GENERATED_PLIST_FILE=>nil}
ERROR [2020-09-30 06:19:09.61]: Tests have failed
INFO [2020-09-30 06:19:09.61]: Successfully generated documentation at path ‘/Users/distiller/project/fastlane/README.md’
±-----±-----------------±------------+
| fastlane summary |
±-----±-----------------±------------+
| Step | Action | Time (in s) |
±-----±-----------------±------------+
| 1 | default_platform | 0 |
| 
| scan | 1089 |
±-----±-----------------±------------+
DEBUG [2020-09-30 06:19:09.67]: All plugins are up to date
ERROR [2020-09-30 06:19:09.67]: fastlane finished with errors
DEBUG [2020-09-30 06:19:09.67]: All plugins are up to date
bundler: failed to load command: fastlane (/Users/distiller/project/.bundle/ruby/2.7.0/bin/fastlane)
FastlaneCore::Interface::FastlaneTestFailure: [!] Tests have failed
/Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane_core/lib/fastlane_core/ui/interface.rb:172:in test_failure!' /Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane_core/lib/fastlane_core/ui/ui.rb:17:in method_missing’
/Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/scan/lib/scan/runner.rb:109:in handle_results' /Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/scan/lib/scan/runner.rb:22:in run’
/Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/scan/lib/scan/manager.rb:23:in work' /Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/actions/run_tests.rb:16:in run’
/Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/runner.rb:263:in block (2 levels) in execute_action' /Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/actions/actions_helper.rb:69:in execute_action’
/Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/runner.rb:255:in block in execute_action' /Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/runner.rb:229:in chdir’
/Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/runner.rb:229:in execute_action' /Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/runner.rb:157:in trigger_action_by_name’
/Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/fast_file.rb:159:in method_missing' Fastfile:37:in block (2 levels) in parsing_binding’
/Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/lane.rb:33:in call' /Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/runner.rb:49:in block in execute’
/Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/runner.rb:45:in chdir' /Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/runner.rb:45:in execute’
/Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/lane_manager.rb:47:in cruise_lane' /Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/command_line_handler.rb:36:in handle’
/Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/commands_generator.rb:108:in block (2 levels) in run' /Users/distiller/project/.bundle/ruby/2.7.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in call’
/Users/distiller/project/.bundle/ruby/2.7.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in run' /Users/distiller/project/.bundle/ruby/2.7.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in run_active_command’
/Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in run!' /Users/distiller/project/.bundle/ruby/2.7.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in run!’
/Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/commands_generator.rb:352:in run' /Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/commands_generator.rb:41:in start’
/Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/fastlane/lib/fastlane/cli_tools_distributor.rb:119:in take_off' /Users/distiller/project/.bundle/ruby/2.7.0/gems/fastlane-2.161.0/bin/fastlane:23:in <top (required)>’
/Users/distiller/project/.bundle/ruby/2.7.0/bin/fastlane:23:in load' /Users/distiller/project/.bundle/ruby/2.7.0/bin/fastlane:23:in <top (required)>’
Exited with code exit status 1
CircleCI received exit code 1