The new update to our 14.04 build image has now been rolled out.
Notable changes:
Added
Removed
Updated
Bug fix
Once again, our builds are broken because of these updates. There is a bug in chrome 53 + chrome driver 2.23 that causes it to ignore tab commands (see https://bugs.chromium.org/p/chromedriver/issues/detail?id=1484 ).
When are we going to be able to lock down to specific circle releases? We absolutely need to be able to keep our build system stable until we are ready to update. We’ve requested this before and still have no response: Ubuntu 14.04 Build Image Update 201606-01 and Feature: Container Versioning
Circle is great, but we keep being burned by this again and again. We may have to switch off if this keeps happening.
Thanks for the feedback Kevin. I understand the frustration with this.
I’ll look into getting ChromeDriver 2.24 added as soon as possible.
We are working on a solution that will allow you to do exactly what you say (and much more too). It’s not on the immediate horizon but is definitely coming.
@kevinmook We’ve shipped the fix for ChromeDiver issue in Ubuntu 14.04 Build Image Update 201610-02
Thanks for the update, we appreciate the quick response.
The main problem is this is a frequent issue that has burned us at least four times before. The circle platform is amazing in how customizable it is, and how easy it is to install or override the defaults. Yesterday we were able to quickly write a script that downloaded 2.24 and overwrote the default ChromeDriver, but I wasted a lot of time trying to figure out why several of our accessibility tests randomly started failing.
The problem is Circle platform updates are pushed with no advance warning, and there’s no way to opt-out of an update, or to push it off until there are engineering resources available to make sure the update doesn’t break something unexpected. If something does break, it’s usually not too hard to fix, but we have to drop everything to do it and that’s unacceptable and expensive for our business.
I love your platform. You’ve made so much so easy. This is my only gripe, but it’s a huge one and it’s one that undermines a lot of your value. I know we could switch to using docker containers, but if we were to do that we’d lose out on a lot of the benefits of your platform.
I ask you to please make your solution to this a priority. I have to imagine we’re not the only ones with this issue.
The new system that will allow you to be in control of individual software versions is being worked on as a top priority. It’s a big project so we’re not able to give timelines for it.
We are now announcing updates before they are released - so although it doesn’t address all your concerns, it at least gives some warning. This update was announced here 9 days before the release.
Please subscribe to our Announcements category to be notified of future updates.
Thanks again for your feedback.
@kevinmook Would love to help folks in your situation. Like Tom mentioned, we’re working on a solution to stop breaking builds (in regards to you mentioning “opt-out” or “hold off”)
In the meantime, I want to ensure that we hit your other issue ("no advance warning). If the image change will break your build - it’s not a surprise.
Now that you know the Announcments section exists (and how to subscribe to it), do you feel like this will solve this portion of the problem?
Are there other ways we can get you this information? Any other suggestions for what we can do to help you on this front?
Thanks for the response. Unfortunately that kind of announcement doesn’t really help us determine if the image change will break our build. It is helpful after the fact to give us guidance in what may have gone wrong.
The reason it doesn’t really help us determine if our builds will break is because usually the problem is with some subtle interaction. For example, in this case there is a bug with how ChromeDriver 2.23 interacts with Chrome 53. When you post an announcement that you’re going to update the Circle environment, we simply don’t have the engineering resources to go through that list, find all the things that are updated, update our local environment to match, and see if everything works. All we can reasonably do is say, “Well, if builds start breaking in 3 days, it’s probably because Circle pushed out their update.”
A better stopgap solution might be to enable us to start a build on a circle release candidate image. That way we can click a button, see if the build passes, and if it doesn’t we can know something critical broke and can prepare to deal with it. But even this is a poor solution because it requires our direct interaction and vigilance, and if something is broken we still have to go fix it before the change goes live.
I realize this is a hard thing to get right, and I appreciate you and your team’s work. I’m glad to hear this is something that’s actively being thought about and worked on.
This would be fantastic. It would allow us to test and fix our builds more on our schedule.
With the current situation, we basically have to deal with anything that breaks when the new image comes out (we’ve purposely made it difficult to deploy outside of CI/CD). Given that we sometimes tie our releases to time-limited marketing pushes, the potential business impact of a delayed release is non-trivial.
Could you add a way to be informed when a new release is done?
Because currently it seams that the process is like:
A newsletter based on the Ubuntu image we’re using, a twitter account with only these notifications… I’m open.
Thank you!