Possible to parallelize tests running via docker-compose?

#1

I’d like to parallelize our Rails tests to speed up our build times, but am unsure how to go about doing it given that we run our tests in what I’ll call a double Docker setup - in their own container, inside a container created via the machine executor.

I’m unsure if either of the parallelization options listed here - either via the CLI or environment variables - will work given that my Docker container will not have full access to the connections and resources of the outer CircleCI container, and if not, what I can do to get it going.

---
version: 2
jobs:
  build_and_test:
    machine:
      image: circleci/classic:latest

    working_directory: ~/company

    steps:
      - checkout

      - run: docker-compose up -d && sleep 60

      # Run the rails tests
      - run:
          name: Run Rspec tests
          command: |
            docker exec company_app_1 bash -c 'RUBYOPT="-W0" bin/rspec \
              --exclude 'features/**/*' \
              --deprecation-out /tmp/rspec/deprecations.txt \
              --format documentation \
              --format RspecJunitFormatter \
              --out /tmp/rspec/rspec.xml'

workflows:
  version: 2
  build_and_test:
    jobs:
      - build_and_test
#2

I have an idea I’ve been toying with in my head, for PHPUnit tests, and I think it might work. I imagine your test runner would have the facility to run tests by label/folder in some fashion, which is probably enough. I have not tried this, here be dragons. :dragon:

  1. Decide how to partition your tests, such that each piece takes roughly the same amount of time to run. I will assume you’re using two blocks of tests, but you could do more than this if you wish.

  2. Work out the docker exec command to run each piece. Add each one to your YAML config, and make it a background step. Pipe the output to a log file so you can view it later.

  3. After each backgrounded exec, you will need to write a wait loop to wait for all piped logs to be written (which will signify that each piece is finished).

  4. Write a script to merge the two (or more) test outputs. For example, if your usual output lists number of tests and number of assertions, you may wish to add these up, so you can provide a total. Run this script to produce the correct test output in a final step.

I think this would be worth a try. However, the main issue may be that each docker exec is running on the same container, and if that is CPU-bound already, you may not achieve any speed-up.

#3

@halfer Thanks so much!

Yes, rspec is capable of running tests by folder.

Will try this and report back.

#4

No worries. If you are on a paid plan and have more than one machine, you could do the above but use the parallelism feature. That would be much likely to achieve some speed-up, but of course at a greater financial spend.

#5

We are on a paid plan, and can use that feature - which is really what my original question was trying to get at. That’s perfect - thanks so much for the idea!

Edit: This should work even though we’re using the machine executor, right?

#6

Ah, cool. I tend to think of one-container solutions first, since that’s what I’m on.

Hmm, good question - I don’t know. It’d be pretty quick to give it a go and see. Do you specifically need a Machine? Using Docker does have some advantages.

#7

We went with the machine executor because of this line in the docs:

That is, if you have a docker-compose file that shares local directories with a container, this will work as expected.

We do have two containers linked with a shared volume in our docker-compose. We’ll also be hooking up the test-running containers to share a volume with the CircleCI container so we can export test result files.

#8

Yes, on-host volume mounts won’t work with CircleCI’s Docker configuration. However, there are several ways around that in order to still use the Docker executor - including using docker cp to copy non-image files into a running container.

#9

@halfer Not sure I’ve done this right. Currently each run step is running in a parallel container, so I’m essentially duplicating all of my steps across 3 containers rather than splitting them. Any obvious red flags?

---
version: 2
jobs:
  build_and_test:
    machine:
      image: circleci/classic:latest

    working_directory: ~/company

    parallelism: 3

    steps:
      - checkout

      - run: docker-compose up -d && sleep 60

      - run:
          name: Install JS deps and run Ember tests
          background: true
          command: |
            docker exec company_app_1 bash -c "bundle exec rake db:create db:migrate elastic:create RAILS_ENV=test && \
              cd /company/frontend && \
              npm install && \
              /company/frontend/node_modules/bower/bin/bower install --allow-root && \
              cd /company && \
              /company/script/post-setup.sh && \
              mkdir /tmp/ember && \
              touch /tmp/ember/ember.xml && \
              RAILS_ENV=test bin/rails server -d && \
              cd /company/frontend && \
              ./node_modules/.bin/ember test --reporter=xunit > /tmp/ember/ember.xml"

      - run:
          name: Run Rspec tests part 1
          background: true
          command: |
            docker exec company_app_1 bash -c 'RUBYOPT="-W0" bin/rspec \
              --pattern "spec/api/{*,**/*}_spec.rb,\
              spec/controllers/{*,**/*}_spec.rb,\
              spec/i18n/{*,**/*}_spec.rb,\
              spec/initializers/{*,**/*}_spec.rb,\
              spec/interactors/{*,**/*}_spec.rb,\
              spec/jobs/{*,**/*}_spec.rb,\
              spec/lib/{*,**/*}_spec.rb" \
              --deprecation-out /tmp/rspec/deprecations.txt \
              --format documentation \
              --format RspecJunitFormatter \
              --out /tmp/rspec/rspec_1.xml'

      - run:
          name: Run Rspec tests part 2
          background: true
          command: |
            docker exec company_app_1 bash -c 'RUBYOPT="-W0" bin/rspec \
              --pattern "spec/mailers/{*,**/*}_spec.rb,\
              spec/models/{*,**/*}_spec.rb,\
              spec/requests/{*,**/*}_spec.rb,\
              spec/routing/{*,**/*}_spec.rb,\
              spec/spec_helpers/{*,**/*}_spec.rb,\
              spec/validators/{*,**/*}_spec.rb,\
              spec/views{*,**/*}_spec.rb" \
              --deprecation-out /tmp/rspec/deprecations.txt \
              --format documentation \
              --format RspecJunitFormatter \
              --out /tmp/rspec/rspec_2.xml'

      # Save results and artifacts from Ember test run
      - store_test_results:
          path: /tmp/ember
      - store_artifacts:
          path: /tmp/ember
          destination: test-results

      # Save results and artifacts from Rails test run
      - store_test_results:
          path: /tmp/rspec
      - store_artifacts:
          path: /tmp/rspec
          destination: test-results


workflows:
  version: 2
  build_and_test:
    jobs:
      - build_and_test
#10

Yes, sorry, I inadvertently sent you on a wild goose chase. The use of background is just if you have one build server. Since you are looking to use three parallel containers, I’d remove that, and use $CIRCLE_NODE_INDEX to determine which set of tests to run. You only need one copy.

I may not be able to help more than that, since I have only ever had one container. The link to the docs I provided shows how a command from CircleCI can be used to split the tests up, but I don’t understand that, and I don’t see how any CI system could introspect tests sufficiently to split them up. Thus, I think that is a task to be left to the integrator (you in this case). That said, perhaps you could make use of this.

#11

@halfer Just a happy update for you. I wasn’t able to get the CLI to split the tests right, so as you mentioned, I found this nice little script that reads $CIRCLE_NODE_INDEX to split up tests.

From there I set parallelism: $NUM in my .circleci/config.yml, and did my Dockerized setup steps in each container. Then I customized the script. to pass the test files in the appropriate blocks to docker exec app and it works like a charm.

Thanks!

#12

Good work! :ok_hand: :tada: