All builds consistently failing "Setup a remote Docker engine"


#1

Started happening about 20 minutes ago, and is consistently failing after trying to setup for ~8 minutes. No updates on the status page yet (https://status.circleci.com/)


EDIT (to show the exact error output)

Setup a remote Docker engine

Allocating a remote Docker Engine
....................................................................................................
Got error while creating host: Maximum number of retries (100) exceeded

#2

14 minutes and still stuck :-/ I also see some UI weirdness, some of the steps that ran before, for example, the Checkout Code step which took 3 mniutes, now states it took 22 minutes, anyone else see that issue?


#3

I’m not seeing the elapsed time weirdness, but I’ve certainly seen that in the past on builds that failed with these similar scenarios.


#4

I just hit rebuild, fingers crossed… Was hoping issues like these would have been automatically detected, but at least acknowledged by the team. Not having a good CircleCI experience so far… :-/ Availability/performance has been quite spotty… :frowning:


#5

None of our rebuilds have succeeded since it started happening 40 minutes ago. @CircleCI-Employees, can we get an acknowledgement of whether you’re aware of the issue? Your status page indicates that you aren’t.


#6

Again dont working (((


#7

Same issue here. Status page shows 2.0 is operational…


#8

CI is one of the most critical components we use for software delivery. When there is no availability, there is no deployments happening. When there is no acknowledgement/support, this is a really bad sign.


#9

I agree but personally I’m trying not to freak out because 2.0 is still technically “beta”


#10

True, but we’ve been experiencing issues with 1.0 also that we’re hoping 2.0 solves before we look for an alternative :frowning: (or perhaps run our own CI on AWS, but that takes some effort, would like to stick with what works if possible and delegate it to a service like circle)

If this was some other service where we can tolerate downtime, I wouldn’t pushing as hard for faster acknowledgement/support


#11

I will say it’s been reliable… Clearly a monitoring miss. Since we all can agree that this isn’t a true statement:


#12

Yeah, I’d agree that we’ve seen overall good reliability, and issues have always been promptly resolved.


#13

Hope to see that here too, I just joined an org that use CircleCI, so it’s new to me, hoping to have a good impression of it


#14

https://status.circleci.com/incidents/k51rp3s01r22


#15

They’re reporting that it’s now fixed, and our retried builds have proceeded past the previously failing step.


#16

Apologize for the inconvenience.

We had an incident on Docker / VM creation on CircleCI 2.0.

We have identified the issue and implemented a fix and are currently monitoring the situation.

Please refer to https://status.circleci.com/ for latest updates.

Your feedback is valid and we will be conducting a post mortem to better improve our process.

Thank you for supporting CircleCi 2.0 Beta.


#17

Thanks for the update and the quick resolution.

Can we get a promise of improved monitoring for this type of issue on the circleci end? I think we all understand that things often break in software, but visibility into those failures is the most important thing for us as consumers of the platform. Your team is generally very good about communicating service degradation, so I think this was probably just a gap in your monitoring and alerting.


#18

Absolutely.

We will continue to make improvements and add proactive monitoring/alerts to our 2.0 platform.

We take such incidents very seriously and have an internal post mortem process to analyze, improve and avoid such incidents in the future.

You will continue to see Platform improvements as we progress in this Beta milestone.

Thank you for supporting us.


#19

Is this perhaps happening again?

I’m seeing Got error while creating host: VM creation failed


#20

I hope not. Can you link me to that?