Unclaimed task number being zero while numerous CI jobs are queued

Build Environment
1

Today, we observed significant delays among all our CI jobs running on self hosted runners despite having remaining concurrent self hosted runner quota: our limit is 20, and we only observed up to 4 self hosted runners running.

When trying to debug this, I observed that the CircleCI container agent appeared to be “slacking”. It could not get any unclaimed pending jobs.

So I poked around the self hosted runner API myself. To my surprise, the following API call gives me a response of 0 unclaimed task:

 curl -X GET https://runner.circleci.com/api/v3/runner/tasks\?resource-class\=<our resource class> \
    -H "Circle-Token: $CIRCLE_CI_PAT"

Response:

{"unclaimed_task_count":0}%

In the meantime, I swear we have tens of jobs that’s Not Running and have the Queued timer counting.

What could be the cause of this?

2

It would be best to raise a ticket directly to support as they have direct access to your account and system details.

The forums are operated more as a self-help area with volunteers (such as myself) and CircleCI staff when they have time.

1 Like
3

Curious to know this. I might have also come across this in our pipelines before. Could you also post the solution / diagnosis for future reference here.