Containers stuck in "Not Running"


#1

Despite not having any running builds no new builds are starting since about half an hour or so.
image


Builds are stuck in SCHEDULED & NOT RUNNING states
#2

We still have this problem despite Circle CI status page reporting no problems. Is the status page not actually displaying current conditions?


#3

I’m seeing similar issues, though anecdotally only with workflow builds right now.

FWIW, I’ve managed to unstick my workflow builds by triggering an unrelated non-workflow build, and then cancelling it.


#4

I can confirm that this triggering works with our currently stucked workflows as well.


#5

Hi everyone,

We’ve recently discovered a bug where jobs can queue indefinitely. The process to reproduce this is:

  1. Start up multiple workflows such that jobs queued need more containers than your plan has. At least one job should be queued by this.
  2. Cancel the workflows owning the running jobs.
  3. The queued jobs will remain in a “not running” state, when they should switch to running.

If you can confirm that this matches what you saw, I can add your cases to our ticket. Additionally, you can work around this bug by cancelling indefinitely “not running” jobs and re-run their workflows, if necessary.

The additional workaround you’ve provided about triggering a workflow and cancelling it is helpful. Thank you for sharing that!

If your issue is different, please provide reproduction steps if possible.


#6

Hi @Eric - this doesn’t seem to quite be the scenario in my case. I have no other builds running or queued, and didn’t cancel anything during the run of my workflow. Nevertheless one job is currently stuck “not running” indefinitely:

FWIW I only have one container on my plan.


#7

Hello,

Thank you for letting us know about this. Our engineers located and corrected an issue, and all your builds should be running or ran now.

If you still have a stuck build, or see this again, please fill our a ticket at https://circleci.zendesk.com/hc/en-us/requests/new with the workflow or build URL so we can investigate.


#8

Hi, I appear to have the same issue with my workflow today, and I haven’t cancelled anything. It appears to simply be stuck.


#9

Same issue here - is this related to the outage yesterday Oct 4?


#10

Hi all,

Thanks for checking your jobs again and reporting back. Our engineers deployed another fix. Please let us know if you’re still seeing this issue.


#12

#13

Happening to our builds as well, but only the ones with a workflow. If there is another build running, the entire workflow seems to get queued even if there are available containers. Cancelling any job in the queue - in the ‘not running’ workflow or not - seems to trigger them to run, or a rebuild. Some workflows have more containers that we have available, but not all.


#14