Avoiding idle containers with a Fan-Out/Fan-In Workflow


#1

I have a three step workflow:

  • build
  • test
  • deploy

I want to run my tests in parallel, so I have set parallelism: 2 on the test job. The other jobs don’t need to be parallelized.

However, I’ve found this approach means I’m not able to take full advantage of my containers.

The test job won’t start until there’s two containers available. This means if I start a second workflow when the first is in the test job, the second workflow will start the build job when the first reaches the deploy job (as expected), but will then wait until the deploy job has finished before starting the second test job.

What’s more, if I start a third workflow, its build job doesn’t start, even though there’s a container available for it. Presumably this is because the second workflow’s test job has already reserved the resources.

Because of this design, I don’t see a way of ensuring I’m using all my containers when doing a Fan-Out/Fan-In Workflow. Am I missing something?


#2

I had a similar problem. Fanning out to run tests after a build seemed like a good idea but is actually causing tests to take longer in some cases!


#3

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.