I’m attempting to upgrade our fleet of self-hosted EC2 macos runners to version 3.0 of the CircleCI runner and am unable to get the LaunchAgent to load correctly. Has anybody else run into this issue with EC2 MacOS instances?
When following the instructions that output after running brew install circleci-runner I get an Input/output error:
ec2-user@REDACTED ~ % sudo xattr -r -d com.apple.quarantine “$(brew --prefix)/bin/circleci-runner”
ec2-user@REDACTED ~ % nano /Users/ec2-user/Library/Preferences/com.circleci.runner/config.yaml
ec2-user@REDACTED~ % launchctl load /Users/ec2-user/Library/LaunchAgents/com.circleci.runner.plist
Load failed: 5: Input/output error
Try running launchctl bootstrap as root for richer errors.
When following the instructions from the documentation that say to use launchctl bootstrap I also get an error:
ec2-user@REDACTED ~ % launchctl bootstrap gui/stat -f %u $HOME/Library/LaunchAgents/com.circleci.runner.plist
Bootstrap failed: 125: Domain does not support specified action
This issue seems particular to the LaunchAgent, since I am able to successfully start the runner by running the circleci-runner command:
ec2-user@REDACTED ~ % circleci-runner machine --config /Users/ec2-user/Library/Preferences/com.circleci.runner/config.yaml
15:13:46 98b75 0.003ms starting agent app.date=2024-04-17T11:56:53Z app.version=3.0.20-5523-1c8a9cb mode=agent service.name=circleci-runner service_name=circleci-runner
So long as I leave the SSH session open, I can successfully run CircleCI jobs on this instance while circleci-runner is running, so it doesn’t seem like anything is wrong with the runner config, just the LaunchAgent.
When I follow the installation and startup instructions on my local development macbook I do not receive the above errors and am able to load the LaunchAgent successfully, so this issue does seem particular to the EC2 MacOS instances. Are there known workarounds for these types of runners?
Update: I ditched the homebrew install method and instead tried the manual installation method, and when I did this, I emulated the deprecated 1.1 runner install method in the following ways:
-
The runner config was placed in /Library/Preferences/com.circleci.runner/config.yaml with permissions set to 600
-
The launchd plist was placed in /Library/LaunchDaemons/com.circleci.runner.plist with root ownership and 644 permissions.
-
The command_prefix : ["sudo", "-niHu", "ec2-user", "--"] config option was retained since the launchd service would be running as root.
-
The service was loaded as root via sudo launchctl load /Library/LaunchDaemons/com.circleci.runner.plist
This was successful insofar as the 3.0 runner did successfully start running as a launchd service.
However, actual CI jobs failed:
03:19:19 aa958 22.224ms httpclient: runner-agent /api/v2/step/end http.attempt=1 http.base_url=https://runner.circleci.com http.client_name=runner-agent http.host=runner.circleci.com http.method=POST http.request_content_length=0 http.response_content_length=0 http.retry=false http.route=/api/v2/step/end http.scheme=https http.status_code=409 http.target=/api/v2/step/end http.url=https://runner.circleci.com/api/v2/step/end http.user_agent= mode=agent result=success service.name=circleci-runner service_name=circleci-runner span.kind=Client warning=the response from POST /api/v2/step/end was 409 (Conflict) (1 attempts)
03:19:19 aa958 22.224ms httpclient: runner-agent /api/v2/step/end http.attempt=1 http.base_url=https://runner.circleci.com http.client_name=runner-agent http.host=runner.circleci.com http.method=POST http.request_content_length=0 http.response_content_length=0 http.retry=false http.route=/api/v2/step/end http.scheme=https http.status_code=409 http.target=/api/v2/step/end http.url=https://runner.circleci.com/api/v2/step/end http.user_agent= mode=agent result=success service.name=circleci-runner service_name=circleci-runner span.kind=Client warning=the response from POST /api/v2/step/end was 409 (Conflict) (1 attempts)
03:19:19 71e34 0.057ms step end error=the response from POST /api/v2/step/end was 409 (Conflict) (1 attempts) mode=agent result=error service.name=circleci-runner service_name=circleci-runner
03:19:19 71e34 0.057ms step end error=the response from POST /api/v2/step/end was 409 (Conflict) (1 attempts) mode=agent result=error service.name=circleci-runner service_name=circleci-runner
03:19:19 d34e0 26.800ms httpclient: runner-agent /api/v2/task/event/cleanup http.attempt=1 http.base_url=https://runner.circleci.com http.client_name=runner-agent http.host=runner.circleci.com http.method=POST http.request_content_length=0 http.response_content_length=14 http.response_content_type=application/json; charset=utf-8 http.retry=false http.route=/api/v2/task/event/cleanup http.scheme=https http.status_code=200 http.target=/api/v2/task/event/cleanup http.url=https://runner.circleci.com/api/v2/task/event/cleanup http.user_agent= mode=agent result=success service.name=circleci-runner service_name=circleci-runner span.kind=Client
03:19:19 d34e0 26.800ms httpclient: runner-agent /api/v2/task/event/cleanup http.attempt=1 http.base_url=https://runner.circleci.com http.client_name=runner-agent http.host=runner.circleci.com http.method=POST http.request_content_length=0 http.response_content_length=14 http.response_content_type=application/json; charset=utf-8 http.retry=false http.route=/api/v2/task/event/cleanup http.scheme=https http.status_code=200 http.target=/api/v2/task/event/cleanup http.url=https://runner.circleci.com/api/v2/task/event/cleanup http.user_agent= mode=agent result=success service.name=circleci-runner service_name=circleci-runner span.kind=Client
03:19:18 d34e0 949.826ms service-work app.resource_class=runner.doximity/macos-selfhost-runner-3-test app.task_id=6629cbb57dd5df270426f37c-0-build error=could not run task: exit status 126 mode=agent result=error service.name=circleci-runner service_name=circleci-runner
03:19:18 d34e0 949.826ms service-work app.resource_class=runner.doximity/macos-selfhost-runner-3-test app.task_id=6629cbb57dd5df270426f37c-0-build error=could not run task: exit status 126 mode=agent result=error service.name=circleci-runner service_name=circleci-runner
03:19:19 6ac6b 0.016ms Error while running task, refer to https://circleci.com/docs/troubleshoot-self-hosted-runner for common runner troubleshooting app.task= error=could not run task: exit status 126 mode=agent result=error service.name=circleci-runner service_name=circleci-runner
03:19:19 6ac6b 0.016ms Error while running task, refer to https://circleci.com/docs/troubleshoot-self-hosted-runner for common runner troubleshooting app.task= error=could not run task: exit status 126 mode=agent result=error service.name=circleci-runner service_name=circleci-runner
So here’s my dilemma:
-
When installing via homebrew and configuring the service to run as the ec2-user, it just flat out does not work as a LaunchAgent. The Bootstrap failed: 125: Domain does not support specified action error when running launchctl bootstrap gui/$(stat -f %u) $HOME/Library/LaunchAgents/com.circleci.runner.plist suggests that recommended setup cannot work for an ec2 MacOS instance.
-
When installing manually and configuring the service to run as root, the service starts but builds fail.
Surely I’m not the only person who’s attempted to install runner 3.0 on Mac EC2 instances? Any guidance would be appreciated!