Start multi-container builds as soon as individual containers become available

paralellism
2.0

#1

Some of our build steps use all our containers, other steps only use 1. If the next queued step uses all containers, then all previous steps that only use 1 have to finish before the big step can start. From what I observed, it seems that containers that finished in multi-container builds are immediately released so that 1-container steps can run immediately before all containers finished up. However, this does not happen at the start of a step yet, i.e. if there’s only 1 container available for the all-containers step, it’s still waiting instead of using what is already available.

Pretty much the same question as Start build even if not all parallel containers are available, but for 2.0 instead of 1.0. It makes sense to me that the builds would wait for the required amount of containers before starting in 1.0, since you could e.g. SSH between containers, which requires all other containers to have gotten to the same step in some cases. But in 2.0, since all containers are completely independent, I don’t see why they would wait anymore.

This feature would give us drastically better container utilization since it would completely eliminate idle waiting.


#2

We noticed this too, this is not ideal and is basically making parallelism useless for larger organizations.

Looks like copypasting the jobs and using custom splitting solution provides the requested behavior but making code quite messy.

Since this problem is mostly related to customers paying for a lot of containers it would be nice if this feature got some love.


#3