We’ve been having an issue with CircleCI executing tests in which tests compile but never get executed and we end up with the following message: Too long with no output (exceeded 30m0s): context deadline exceeded
We’ve tried different things such as increasing the timeout timer to 30 min to no avail. Note that we are able to execute the tests locally via fastlane or Xcode without an issue.
Here is the fastlane action we use to execute tests:
esc 'Run tests'
lane :test do |options|
no_focus_tests(test_dir: './ProjectTests')
if is_ci?
UI.message "Running on CI, CocoaPods have been installed already"
else
pods
end
workspace = "project.xcworkspace"
scheme = "project"
configuration = "Unit Test"
device = "iPad (10th generation)"
if is_ci?
configuration = "Unit Test"
UI.message "Running on CI, use Unit Test configuration"
else
configuration = "Debug"
UI.message "Not running on CI, use Debug configuration"
end
output_directory = options[:output_directory]
scan(
workspace: workspace,
scheme: scheme,
configuration: configuration,
devices: [device],
clean: false,
skip_build: true,
result_bundle: true,
include_simulator_logs: false,
run_rosetta_simulator: true
)
end
And this is what our config.yml file looks like:
jobs:
build-and-test:
macos:
xcode: 15.0.0
environment:
RUBY_VERSION: 3.1.4
FL_OUTPUT_DIR: output
shell: /bin/bash --login -o pipefail
steps:
- macos/preboot-simulator:
version: "17.0"
platform: "iOS"
device: "iPad (10th generation)"
- checkout
- run: xcodebuild -version
- run: brew update && brew install swiftlint
- run: git submodule update --init
- run:
name: Set Ruby Version
command: echo "ruby-3.1.4" > ~/.ruby-version
- run: sudo gem install bundler:2.3.26
- run: bundle install
# The following two lines were added to address a bug with cocoapods 1.13.0
# Remove once bug is fixed.
- run: yes | sudo gem uninstall --force activesupport -v 7.1.1 || result=$?
- run: yes | sudo gem install activesupport -v 7.0.8 || result=$?
- run:
name: Install CocoaPods
command: pod install --verbose
.....
- run:
name: Build and Run Tests
no_output_timeout: 30m
command: bundle exec fastlane test
....
The first thing that comes to mind is to set the timeout to a much higher number to see just how long this process will take within a CircleCi environment.
When you run the same task locally how long does it take and what are the specs of the system in terms of cores and RAM?
I increased the timeout to 90 min and still got the same result:
Too long with no output (exceeded 1h30m0s): context deadline exceeded
I have an M1 Macbook Pro with 32GB of memory.
It’s worth noting that in both instances (30 min and 90 min timeout), the last lines of code executing as part of the tests are the same. So somehow it’s hanging when compiling the following Airship (3rd party pod) files:
@rit1010 Forgot to mention, my laptop is a 10-core CPU with 8 performance cores and 2 efficiency cores.
I removed Airship from running on CircleCI and I’m seeing the tests hang when compiling a storyboard file again. It seems to be having trouble compiling some storyboard files for some reason.
I do not know much about Mac environments or the applications, but I’m pretty sure that your 10-core, 32GB laptop is going to handle certain tasks better than the 4 core 6GB instance that is being used as the default CircleCI system.
As you have a working local process it would be worth checking the resources it uses as it completes the tests.
Well, a near-constant 100% CPU load from 14m to 47m indicates that you are totally CPU bound. I’m guessing if you were to collect stats on your local system with its 10 cores you would see much the same but it would be doing more work over the same time period.
RAM may also be a limitation as CPU time may be allocated to memory management such as page compression/decompression tasks.
Which CircleCI pricing plan are you using? If you are on a paid plan you could upgrade to the larger M1 instance and see what happens.
Hello,
As @lukeredpath has stated, this is most likely due to the performance regression in Apple’s latest Xcode (Xcode 15). We have re-released our Xcode 15.1.0 (Xcode 15.1 Beta 2) image, now based on macOS 14 Sonoma, and it has greatly improved the performance of the executors, albeit still not to the level of performance from Xcode 14.
Due to the large number of breaking changes (with a base OS upgrade), we will not be re-releasing Xcode 15.0.1 but rather keep building on macOS 14 going forward.
Please feel free to try out 15.1.0 and you should seem the timeouts decrease dramatically.
Thank you,
Luis
