The slowest part of our build is now the Install Chrome step, which takes between 30-120s.
I think the most relevant parts of the config are:
- image: cimg/ruby:3.1.2-browsers
PAGER: cat # prevent psql commands using less
- image: cimg/postgres:13.6
I would expect that since we’re using
cimg/ruby:3.1.2-browsers rather than
cimg/ruby:3.1.2 that the
browser-tools/install-chrome step would be very quick, or even unnecessary.
Instead I find it necessary and it appears no faster than on the
Am I doing something wrong or misinterpreting the intention of the
-browsers suffixed images?
Hey there! So per the convenience image documentation, the browser-tools orb is designed to work in conjunction with the “-browser” variants. You can see that described here:
So the browser isn’t actually INSTALLED in the image, because you can then pick which one you want. So yes, you specify the image with the “-browser” variant, and then you use the orb to install the browsers you require. (could be Firefox, or Chrome, or both, etc.)
Assume you’re running CircleCI Cloud, not an on premise service? If on prem, there could be other network or resourcing considerations within your environment.
(to be continued)
Assuming Cloud, for comparison, my team has a similar concept as you describe here for a few of our pipelines. In one example, we’re using one of the cimg/openjdk “-browser” variants, and the Google Chrome install takes ~20 seconds while the Google Chromedriver takes < 0 seconds. We’re also using the default “large” executor, as you show.
Other things that come to mind:
- You’re using a Ruby Docker image with a sidecar for your Postgres image, and then running parallelism 4. I wonder if some how that’s slowing it down? Does that end up running four Ruby images + four Postgres databases? (I’ll be honest and say I’m not exactly sure how that works when combining the sidecar w/ parallel )
- Wonder if you try back to no parallelism as an experiment to see if the installs are faster? Might help eliminate a complexity to look for root cause.
- Depending on your support tier (free, higher, etc?), maybe you could check with support to see if you’re hitting any concurrency limits? (i doubt that’d be at play here but you never know; esp if all the Docker workload is trying to say get squeezed into a single VM behind the scenes)
- Another idea to experiment with might be using a Machine executor RATHER than Docker. Looks like it may include Chrome pre-installed AND Ruby preinstalled (https://raw.githubusercontent.com/circleci/image-builder/picard-vm-image/provision.sh). Sometimes those dedicated Linux VMs might be faster than getting a partial resource class in a Docker image. Then you’d have to own/design the mechanics of firing up your database, and running it in the VM rather than as a sidecar. Depends on how much maintenance that might be on you vs. accepting a wait delay on startup for the example you already have working. (give/take there) Using the Linux VM execution environment - CircleCI
Best of luck Ian!!
Hi @adamdmharvey — thanks for your feedback and thoughts.
Parallelism didn’t seem to be the issue for us — in fact we added the parallelism in response to adding slower integration tests that required Chrome. I didn’t track that carefully, so I could be mistaken, but that’s my memory of last week.
For the moment I’ve just built a custom Docker container for our project that includes the browser baked in. I’m not sure it’s the right solution, but it trimmed off 30s or so (the 20ish seconds for Chrome plus more for our other dependencies).
# Start with Circle's convenience image
# Bake in Circle's browser tools
# The scripts come from their repository: https://github.com/CircleCI-Public/browser-tools-orb
# Commit SHA 2d1428225909f4224c7c1b8e13e5da24869bd4bc
RUN mkdir -p /browser-tools
COPY browser-tools/*.sh /browser-tools
RUN chmod +x /browser-tools/*.sh
# Back in Dockerize to wait for dependencies
RUN wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && sudo tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && rm dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz
# Add our own additional dependencies
RUN sudo apt-get install ...
Our builds are only around 60-90s, so the slowdown seemed like a lot proportionally. And I’m just an impatient person.
Is there a way to background the browser install so it’s non-blocking? This didn’t work, sadly: