Today, we observed significant delays among all our CI jobs running on self hosted runners despite having remaining concurrent self hosted runner quota: our limit is 20, and we only observed up to 4 self hosted runners running.
When trying to debug this, I observed that the CircleCI container agent appeared to be “slacking”. It could not get any unclaimed pending jobs.
So I poked around the self hosted runner API myself. To my surprise, the following API call gives me a response of 0 unclaimed task:
curl -X GET https://runner.circleci.com/api/v3/runner/tasks\?resource-class\=<our resource class> \ -H "Circle-Token: $CIRCLE_CI_PAT"
In the meantime, I swear we have tens of jobs that’s
Not Running and have the
Queued timer counting.
What could be the cause of this?