Recommended way to use multiple languages in 2.0?


#1

Our 1.0 circle.yml has the following in its “machine” section:

java:
    version: oraclejdk8
python:
    version: 3.4.3
node:
    version: v6.1.0

and this sets our environment up with runtimes for those three languages, which we can use freely in our test scripts.

I know there are multiple ways to do it, but what’s the recommended approach in 2.0? Docker-compose? Custom Docker image? Machine executor? Something else?


#2

Very interested in this as well. Am I supposed to build my own image containing the build environment now?


#3

For languages/commands you need to run directly, a custom Docker image.


#4

Yes. Our base images are per language now. There are some exceptions, such as Ruby contains node (since its a pre req for rails). What is your stack?


#5

We have Java on the back end, a Javascript front end that is built/tested using Node-based tools, and a suite of Python scripts (admin tools, etc.). There are test suites for all three parts of the system, and a single commit can sometimes include changes in all three languages. The script that runs our tests under CircleCI 1.0 looks at the list of files changed in the current commit to determine which test suites to run.

Sadly, this will probably cause us to delay our move to CircleCI 2.0; we don’t use Docker and nobody on our development team has much experience with it, so replacing those six lines of YAML would probably require a bit more of a time commitment (both initial learning curve and ongoing maintenance) than we can justify at the moment.


#6

That makes sense.

Do you have any specific goals for moving to 2.0? Your use case is not that uncommon and we are still trying to find the right balance with the build environment.


#7

Workflows seem like they will potentially help us. 1.0 doesn’t really have a notion of a “build” step, which is fine for our Python code but kind of awkward for Java. Being able to define a set of steps with dependencies will let us get rid of some less-than-beautiful test runner scripting.

Related to that, the ability to more precisely control caching behavior seems useful. Because we kind of have to simulate a multi-stage build/test workflow in scripts right now, we end up throwing away some dependencies/artifacts that could be cached between commits.

The other big one is manual approvals. I don’t know enough about the feature to know if it’ll do this, but my hope is that we’ll be able to use it to do our production releases from CircleCI: tag a release, wait for it to hit our staging environment, then once any manual testing is finished, hit the “approve” button and it goes to the “deploy to production” workflow step.

Less important in the short term, but possibly still something we’d play with, is parallel workflow steps. 1.0’s parallel test features don’t help much for us because we end up having to build the code on each of the test systems, but if we can separate the build step from the test-running step using workflows, there’ll be a bigger performance gain from parallelizing the tests.


#8

With workflows, this may not be an issue since each step can use a different container.

This way you can run your python tests in the python container, java in the java one, etc.


#9

It didn’t look like there was a way to filter workflow steps by path, just by branch. We’d end up launching a Python container every time we changed the Java code and vice versa, correct?


#10

We’d end up launching a Python container every time we changed the Java code and vice versa, correct?

For now, yes this is true. But we are working on making the workflows more flexible in the future. So I think we will support a wide range of use cases.

Sorry if there is any confusion. What i was referring to is from the doc:

version: 2
jobs:
  build:
    working_directory: ~/<project root directory>
    docker:
      - image: circleci/<language>:<version TAG>
    steps:
      - checkout
  test1:
    working_directory: ~/<project root directory>
    docker:
      - image: circleci/<language>:<version TAG>
    steps:
      - run: <command>
  test2:
    working_directory: ~/<project root directory>
    docker:
      - image: circleci/<language>:<version TAG>
    steps:
      - run: <command>

From what I understand, you have some steps that need python and others that need java. As you can see you can use a different image for each step. If you need results from one step to persist to another then you would just need to save them to workspace.


#11

Thanks for the tip about workflows, it’s actually amazing. I threw out the custom primary image I had and can now use a language-specific image for each build (client and server).


#12

Using multiple containers makes sense. I suppose my concern about launching unnecessary containers is a bit overblown; the test script can look at the current git commit and quickly exit when it detects there’s nothing to do. Thanks!


#13

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.