Parallel jobs with different base images

build-image
docker
config

#1

Hi,

it would be nice to have build jobs with different base Docker images. As best, they could build in parallel and the artifacts can then be merged together in a deploy job afterwards. That way, it’s easier to use the official Docker images for different frameworks/languages.

As an example, we have a React website which communicates with a Go server behind. See the famous go-starter-kit.
So we have Node.js as a framework and Go as a language (framework). Both have official Docker images. Maintaining them as one integrated, own build image would be hard and not necessary as they are already there. Using them in separate CircleCI 2.0 build steps would make it easier and more concise.

A config.yml could then look like this:

version: 2
jobs:
  build-go:
    working_directory: /root/circleci-test
    docker:
      - image: golang:alpine

    steps:
      - checkout
      - run:
          name: Go build
          command: go build .

  build-node:
    docker:
      - image: node:alpine

    steps:
      - checkout
      - run:
          name: NPM build
          command: |
             npm install
             $(npm bin)/webpack
  
  deploy:
    steps:
      - run:
          name: Docker build

Trigger different jobs from GitHub
#2

We already have support for that.


#3

Oh, yes. You’re right: https://circleci.com/docs/2.0/defining-multiple-jobs/ Sorry about that.

  • It seems that created artifacts are not shared between different jobs, are they?
  • Do you plan to run >= 2 jobs in one go, i.e. it gets one build number and will be listed as one build in the web interface?

#4

No, the builds would have different IDs. Artifacts are tied to a given ID.

No, but I opened a feature request internally.


#5

Is there a way to trigger jobs other than build from a pull request? (Critically, I would need the PR to be informed if the build failed.)

EDIT:

I think I might be after the same thing. I have a repo with artifacts in six different languages and I would like a separate build for each to run concurrently, but any one of them failing would fail the build generally.

I do not need them to be part of the same build though. It would be entirely sufficient for my use case if they were separate builds as long as the builds were properly integrated with the GitHub PR in question.

Separate builds might even be better since I can trigger them conditionally.

So I guess my question is: If I trigger a build, how can I have the “parent” build not report a final status until any “child” builds it triggered are complete?


#6

Not at this time but I’ll pass on all this feedback.


Trigger different jobs from GitHub
#7

Thanks. The concept looks great, but without being able to tie them to a PR, the feature is worthless for me. :frowning:


#9

@lukesneeringer I haven’t tried it yet, but the GitHub screenshot in this post seems to be what you are after?


#10

I’ve found a way to achieve this misusing caches for it:

https://github.com/Dominik-K/circleci-test/blob/bd9e90844e1b039fff08121a6713044bd6e236d1/.circleci/config.yml

https://circleci.com/gh/Dominik-K/circleci-test/20


#11

Thanks. That feature would be really nice. The build logs wouldn’t be too cluttered with many sub-builds anymore.


#12

+1 good request.

In previous CircleCI it was possible to SSH between the containers via ssh node0, ssh node1 etc, - it was very useful. That’s pretty flexible and that’s how it allowed us to share artifacts between the parallel jobs.

Hopefully to have some similar kind of functionality in CircleCI 2.0.


#13

I am trying to get this working.

At the moment, it does not seem to work correctly (no build was triggered). I am going to look at it more tomorrow.

https://circleci.com/gh/googleapis/api-client-staging/3?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link


#14

Try this URL instead of the one you’re currently curling:

https://circleci.com/api/v1.1/project/github/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME/tree/$CIRCLE_BRANCH


#15

Yeah, okay, I get a dunce cap for that one. Thanks.


#16

I did get this working today: https://circleci.com/gh/googleapis/api-client-staging/17
(Ignore that the triggered builds fail; they should.)

Thank you for the help.

One unfortunate side effect of this workaround is that it can not work on forks, because exposing my API token for CircleCI would be a security risk. That is actually a pretty severe penalty for something that should really just be baked in.

I would note that, if there was a blessed way to run other build groups besides build in this way, it would actually be a much more powerful concurrency story than the one you currently expose with node index and node total.

This will work for now, but it would be really, really helpful if this became an official mechanism and “just worked”, and especially if it could work safely on fork PRs. As it stands, this mechanism will work on the repo where I just set it up (because it really only has internal contributors even though it is public), but is a non-starter on anything that accepts external contributions.


#17

We are enabling first class support for running multiple jobs for CircleCI 2.0. Right now we are adding only limited number of customers. If you are interested in trying out, please reach out at beta+workflows@circleci.com with your VCS organization and repository name.
Thanks!


#18

This is now possible with Workflows. Please check out this doc for more details: https://circleci.com/docs/2.0/workflows/


#19