Circle on projects with sub-projects (specifically a chef-repo)

Build Environment
1

Hey all,

Firstly I’m going to target this question at CircleCI 2.0 as our long term plans are to target the VM for testing.

We’re attempting to develop a testing strategy for our chef-repo, which contains many different cookbooks. Testing chef cookbooks takes some time, which means for each PR branch we only want to test the cookbooks that should be affected by the change.

chef-repo
├── config
├── cookbooks
│   ├── users
│   |   └── test
│   |       └── recipes
│   |           └── users
│   ├── compute
│   └── etc
├── data_bags
├── environments
└── scripts

This is the sort of directory setup we’re looking at. Our cookbooks are inside cookbooks directory, and each cookbook has its own test setup and various commands that should be run on changes.

We have a script that, given the source repository and diff under test, can compute the cookbooks that require testing. We could run this as part of a test ‘step’ in our chef-repo/circle.yml file, but this has several disadvantages:

  1. Testing occurs in one command, and therefore under a single section of your UI, obscuring which cookbook failed and resulting in a huge chunk of STDOUT
  2. We are required to aggregate failures, producing the appropriate exit code so that Circle can detect a failure, even if some of the cookbooks have passed
  3. It is not possible to make use of any parallelism features provided by Circle

Do you know of any ways we can supply configuration like this? Our initial thoughts are if we could dynamically generate a circle.yml (or run embedded ruby) then we’d be able to configure each build before it starts. An alternative would be somehow triggering a circle process from configuration inside the cookbook directories.

Any ideas are welcome, we’re keen to use Circle for this!

Lawrence

We’re quite keen on keeping test commands local to the cookbook- this means configuration for the tests appears inside the cookbook/users/circle.yml file. Separating the test commands from the cookbook that they will run against seems a poor UX when working in the codebase.

2

How are you running the tests exactly?

You mentioned there is One Command To Rule Them All, what does it look like?

Perhaps there is a way to keep this setup but provide better reporting / output for debugging purposes.

3

Hey zzak,

Sorry for the late response, I wanted to put together a working solution before coming back to you guys so I could better illustrate the problems we’re having.

version: 2
jobs:
  build:
    machine: true
    parallelism: 8
    environment:
      KNIFE_ENV: test
    working_directory: ~/chef-repo
    steps:
      - checkout
      - run:
          name: Converge all kitchen cookbooks
          command: |
            for test_instance in $(.circleci/bin/kitchen-tests-for-build | circleci tests split);
            do
              .circleci/bin/run-in-cookbook "$(dirname $test_instance)" bundle exec kitchen converge "$(basename $test_instance)"
            done
      - run:
          name: Verify all kitchen cookbooks
          command: |
            for test_instance in $(.circleci/bin/kitchen-tests-for-build | circleci tests split);
            do
              .circleci/bin/run-in-cookbook "$(dirname $test_instance)" bundle exec kitchen verify "$(basename $test_instance)"
            done

This is minimal copy of the circle config we’re using on this project. Each kitchen converge can take anywhere up-to 10m to run, which means we’re keen to use the test splitting capability.

[1] Difficulty parsing converge output

Each converge can be very verbose- so much so that the circle UI will cutoff the output, leaving an option to download the text snippet to see what happened. That’s a large amount of output to parse, and given a node can be running several converges it means failures can get lost in the middle.

We’re solving this for the minute by outputting something like:

################################################################################
# [cookbooks/backup] bundle exec kitchen converge backup-ubuntu-1404
################################################################################

-----> Starting Kitchen (v1.16.0)
-----> Creating <backup-ubuntu-1404>...

       Step 1/18 : FROM ubuntu:14.04
       14.04: Pulling from library/ubuntu
       30d541b48fc0: Pulling fs layer
       8ecd7f80d390: Pulling fs layer
       46ec9927bb81: Pulling fs layer

Which provides a visual cue for when each suite starts converging, but is still difficult to navigate. It would be awesome if the circleci tool could be used to signal a new UI test section, so that you could expand and close each converge output when you’re debugging.

[2] Parallelism is a fixed value for every branch

If you look at the config, we compute what tests need to be run by using a .circleci/bin/kitchen-tests-for-build script. This analyses what files have changed and outputs the relevant tests for each cookbook, enabling pull request builds to run in minutes vs >20m of a full master build.

This is working ok for the moment, but we’re spinning up 8 vms for each test run, despite some branches having only a couple of suites to test. We can’t figure out any way of parameterising the parallelism for the volume of tests to be run on each build, which is a shame as there is no point hammering you guys for vms when we don’t need them.

[3] Unclear how to use timing data

You guys are in beta for the 2.0 product so I don’t blame you at all for slightly incomplete docs, but we’re super keen to use historic test timings to split tests among nodes. As mentioned, converge duration is highly variable and splitting the tests alphabetically can lead to one sad node receiving 2 of the >10m converges, while most of other nodes finish within minutes.

Have you got any good references of how to start with the timings? I have no issue writing some timing diagnostics into run-in-cookbook, which could output to a JSON file how long each suite took to converge, which should allow the circleci tool to split tests via timing.

I’m unsure where I should be placing these files, and what format they should be in- should I use JUnit, and is it possible to hack this to work for neither class or file keys? (the actual timings would be per suite, which is currently represented by cookbooks/<cookbook_name>/<suite>, or as a concrete example cookbooks/gc_backup/barman-ubuntu-1404).

To be clear, I’m not expecting you guys to solve all my problems here! I’ve been quite detailed so hopefully you can see what I’m trying to achieve, which might give some insight into a slightly more custom usage of your product.

Any advice about how to fix some of those issues would be great :slight_smile: