Build times increasing over time?

Build Environment
1

Hello,
We’ve noticed that some of our job build times have increased a lot over the last year, even though much of the code that’s running is nearly identical. Here’s an example:

A build of our generate_env job from today: https://circleci.com/gh/Ginger-Labs/Twobird/78883
A build of our generate_env job from a bit over a year ago (5/2/19): https://circleci.com/gh/Ginger-Labs/Twobird/29998

If you look at the code that’s running for individual steps, we’re not doing very much, often just echoing text into a file that we use to share environment variables throughout different jobs in the workflow. Many of the individual steps ran very quickly a year ago, taking less than a second, but now take 13 seconds. This is happening to steps in a number of different jobs in our workflow, and is cumulatively adding several minutes to our build times. What could be causing the difference in timing here?

2

Hi @christhegrand,

You can try Caching Dependencies to speed up build times.

Your checkout step appears to be running relatively fast. If this step was taking long you might consider Source Caching.

You can also save your environment variables in Contexts
and use Logic Statements to define context based on which branch triggered the build.

Let me know how it goes!

3

Thanks, but those look like general tips as opposed to anything that helps explain why these steps have gotten slower when they’re doing very little.

4

We had a bunch of unused environment variables set up in CCI. I deleted all the unused ones and the time overheads for the steps have gone from about 15 seconds to 0 seconds, like in the older job I linked to in my first post. What is happening in CCI’s setup code at the beginning of each step that could have caused a slowdown like this?

5

Hi @christhegrand - are you seeing this slowdown consistently?

I did a little spot check and the latest runs seem to be taking the expected (short) amount of time.

I suspect the slowdown in the job you linked was caused by a bottleneck in the system which records the output of steps, rather than the step itself.

1 Like
6

Yes, this slowdown was consistently happening for all of our builds for several months. The builds performed since I removed the unused env vars we had set up on Monday afternoon have all been much faster.