Stricter Validation for 2.1 configuration

Announcements
1

On July 7, 2020 we will roll out the first of several changes to
strengthen validation for 2.1 configuration, specifically step
validation in jobs.

Because this update contains stricter rules around validation, it will
catch some configurations that were previously considered valid,
causing them to fail. Any failing builds caused by the change on July
7 are likely the result of pre-existing syntax errors in
configuration.

The specific changes we are making are:

  • the run step now requires either:
    • a string with the command
    • a map with the command key (the name key stays optional)
  • the when/unless steps now require:
    • a condition key with a logic statement
    • a steps key with zero or more valid steps

The most common causes for failure we see in user configurations
include misspellings of keys as well as indentation errors.

For example, an incorrect configuration might read:

steps:
  - run:
    command: echo hi

parses to JSON as

{
  "steps": [
    {
      "run": null,
      "command": "echo hi"
    }
  ]
}

while the correct version would be:

steps:
  - run:
      command: echo hi

which parses to JSON as

{
  "steps": [
    {
      "run": {
        "command": "echo hi"
      }
    }
  ]
}

Please note that we have also identified several orbs in current usage
which may cause syntax errors once this change is released. If you
receive an error message indicating the error is in a command, you may
want to check your orbs are updated. You can update to the latest orb
version and/or contact the orb author and let them know about the
issue. If you believe that your configuration is failing to compile
even though it is semantically valid, please leave a comment here, or
contact our support team.

1 Like
2

Hi Robin

Thanks for the warning.
Does the circleci CLI catch these with circleci config validate?

Cheers!
Michael

3

Hi Michael,

it will after July 7, but not before, as the CLI uses our production
services to do validation remotely.

4

@sulami I am a bit concerned with the timeline and implications of this change. Can you please confirm how I can verify my configs will continue to work after July 7th?

From what I can tell, the only proposed solution is to wait until after July 7th and install the new CLI. Then I can use the new CLI to validate my configs. Of course this option is meaningless because at that point any affected builds will already be failing on CircleCI. Manual inspection is possible, but prone to error.

Is there a web based config validator I can use? If not I predict July 7th will be a rough day for the CircleCI Support and Eng teams.

Best.

5

Hi Evan,

Sadly we currently don’t have the infrastructure in place to provide the
new validation ahead of time, so your conclusions are correct.

We have run a regression test against a full week’s worth of pipelines
though (first week of June), and directly contacted any affected users
who have built during that time. Based on that data, we expect at most
about 0.1% of our users to be affected by this change.

Of course this isn’t a 100% exact science, but if you have built during
the first week of June and have not heard from us directly, chances are
you won’t be impacted.

We are also considering an architectural change in our validation
mechanism, which would allow us to provide test validation ahead of time
(e.g. via the CLI or a web UI), but that won’t be ready in time for this
change.

6

That’s good to hear and I would add a strong “yes please” / +1 to request that this be done.

7

This change has been deployed as of a couple of minutes ago (17:24 UTC to be precise).

8

Looks like my builds are now broken as well, so my concerns were valid as you said.

As an 0.1%-er I’m very disappointed with how this rollout was handled, although I do understand the low impact calc tradeoff you made. I did get an email, which is how I ended up here, however with no way to validate the configs other then a line-by-line check against the docs that does not guarantee I will find all validation issues, I could not take any action that did not feel like needle in a haystack time sink.

Please consider this feedback next time you make this kind of breaking change and also consider the option to simply delay the update until you have a workaround. Even if the workaround is a simple temporarily validation API you can expose to help out your 0.1%.

Best.

1 Like
9

Thank you for the feedback. I’ll pass this along to the team.