Rollout of pipelines functionality

Updated - March 9, 2020

Upcoming changes

On March 9, 2020, CircleCI will begin to enable pipelines for all free projects. A pipeline contains all of the workflows (and the jobs inside those workflows) that CircleCI runs after a trigger on your project. A pipeline can be triggered by a pull request, a commit, or even an API call.

To mitigate any unforeseen issues, we suggest that you enable pipelines ahead of March 9th by navigating to Advanced Settings under Project Settings.

Note

  • Enabling pipelines does not require using 2.1 config syntax. 2.0 config works with pipelines enabled.
  • There are no known issues of projects moving to pipelines.

Benefits of using Pipelines

  • New CircleCI interface
  • V2 API
  • Access to version 2.1 config, which provides:
    • Use pipeline parameters to trigger conditional workflows .
    • Reusable config elements, including executors, commands, and jobs.
    • Over 115 pre-configured 3rd-party tool integrations available using orbs.
    • Improved config validation error messages.
  • You can now enable auto-cancel, within Advanced Settings, to abort workflows when new builds are triggered on non-default branches.

If you have any questions or concerns, please feel free to post it here.

Happy building!

3 Likes

My project has been using pipelines for awhile, to good effect, so: sounds good to me!

One question: I just started seeing a bunch of pipelines that display in the UI with just the string “No workflows to execute for this pipeline”.

What’s up with that?

Hi @avi thanks for raising. That should only show when the pipeline runs without a workflow. This can happen if you have a config that might be targeting a different branch that was pushed or on a tag push. Could either of those be the source?

Hi @Kate_Catlin thanks for the rapid reply! That makes sense; my project has a job that automatically creates a pre-release entity, with a corresponding tag (in the Git sense) in the GitHub repo, if all the other jobs succeed. This is equivalent to pushing a new tag.

However:

  • My workflows haven’t changed significantly for awhile. Why have these no-op pipelines only recently started showing up in the list?
  • If this was a recent intentional change on your part, it’s a little disconcerting to have these pipelines show up with no indication as to why they were created and why there were no workfows to execute. If/when these pipelines will show up in the list, I suggest you enhance them to make that information available and accessible.
  • How can I hide or suppress these pipelines going forward?
1 Like

Great specific feedback, thanks @avi.

They were always showing, simply as blank lines before. Even less information about what was going on showing! Perhaps you noticed the blank rows in the past?

We are creating a card to address this specifically because of your post yesterday so yes we’re on it :slight_smile: If you were designing this, what information specifically would have been most helpful?

This is an interesting idea that, pending further data points, we likely can’t prioritize on our roadmap.

1 Like

My pleasure!

Excellent!

Off the top of my head, I think I’d want to be able to see, clearly, quickly, and at a glance (i.e. while scanning) what event precipitated the creation of the pipeline (i.e. a tag was pushed). And I’d want to be able to find/learn why that event did not trigger any workflows to run. In other words, I think I’d want to be able to click on it to expand it a bit to dig in and see: if it’s a tag, or a branch, what’s the name and the sha. And maybe some way to understand (even if it’s just a link to the docs) why that event didn’t trigger a workflow. i.e. if it’s a branch that didn’t match some pattern in the workflow config, it’d be great to show that.

I hope that makes sense, it’s really just off the top of my head. I’m sure you’ve already come up with an approach that’s way more coherent :wink:

Makes sense, no worries!

Will the CLI now be able to run 2.1 configs (locally) or are we still stuck with the 2.0?

Hi

Enabling pipelines – is this a potentially destructive action?

ie, will any existing configuration break? or is it backwards-compatible?

I’ve been looking for information on the impact this will have, and haven’t been able to find any.

many thanks

Matt

2 Likes

@Kate_Catlin, @KunalJain… it would be good to get a reply on this one.

1 Like

Hi Matt,

Enabling pipelines – is this a potentially destructive action?
ie, will any existing configuration break? or is it backwards-compatible?

Enabling pipelines should not break your project. You can try out pipelines by adding pipelines: true in your config, before enabling it for your project.

Please feel free to ping this thread if you have any issues.

Hi Blair - We are actively working on enabling config to build as part of 2.1. I will ping this thread once the PR is merged. Thank you so much for your patience.

Thanks @KunalJain.

BTW, I found a breaking change in the schema: a job name that worked fine in version 2 produced an error in version 2.1.

I eventually tracked it down to the presence of a colon (:) in the job name. When I replaced this with an underscore (_), the errors disappeared.

This is a breaking change, so if you’re following semver, the new version should really have had a major version change.

1 Like

@KunalJain were you aware of this breaking change?

Hi Matt,

I’m sorry to hear that you have had issues upgrading from config version 2 to 2.1. I’m on the team that implemented the 2.1 config, and I’d like to share some of the context around the breaking changes that we made.

One thing to note is that config version 2.1 is opt-in, and is not required for pipelines.

We arrived at what is now “config version 2” after a bunch of iteration, where we built the 2.0 platform using early versions of the 2.0 config ourselves, and later in a closed beta with some partners. All the time we were iterating on the config format. When we finally called CircleCI 2.0 “released”, the config file format had a sizeable amount of backwards compatibility shims in it under the hood, from where we had iterated on the config schema. This lead to us having some really poor error messages for invalid config, because we had some hand-written schema checks which allowed a lot of questionable yaml to build. The 2.0 release was iterative, where we on-boarded more and more customers over time, with a closed beta, and open beta, etc, so we never had a single “release” where we would have been able to define the “finished” config schema (and break the builds of customers who had adopted 2.0 early).

When we started the project to add the features that we added in config version 2.1 (commands, parameters, orbs, etc), we found that we couldn’t add these new features into config 2.0 in a manner which was backwards compatible without compromising the design, or in a way that produced useful error messages. We decided to make a clean break, and bump the config version to 2.1.

Config version 2.1 is a lot more strict about what it accepts than 2.0 was, especially in regards to additional keys that are not expected. In particular, in config 2.0 you could add extra keys nearly anywhere in the YAML document, and our parser would ignore them. This lead to cases where a typo would be silently ignored. When we looked at the data, we found that the resource_class was the most commonly misspelled key. The pitfall here is that if you add resource_class: large to your config file but you misspell resource, our config validation for 2.0 would ignore the key, and your job would run on the medium resource class.

The other area that we tightened up was the names that we allow – we restricted the character set and length of a bunch of fields in the config file. The one that has affected you here was job names. We added a lot of restrictions on the contents or orbs, since they are stored in a registry, passed through an API, processed in the command line tool, etc. Since jobs can be contained within orbs, we restricted what a job can be called, and the restriction is global, and affects all jobs in the config file.

So that’s the context :slight_smile:

In your case, do you know what the error message produced was? I’ve added a job with a : in the name, and when I run circleci config validate in the CLI tool I get the following:

marc@blaster ~/dev/circleci/circle $ circleci config validate
Error: ERROR IN CONFIG FILE:
[#/jobs/job-with-a-:-in-the-name] string [job-with-a-:-in-the-name] does not match pattern ^[A-Za-z][A-Za-z\s\d_-]*$

Did you get a similar error message? Echoing the regex back to the user as part of the error message is not the most clear way for us to convey the issue here. Do you have any thoughts on what error message that we could show to make it clear to you what the problem is?

Thanks,

Marc

1 Like

Hi @marc

Thank you for the detailed reply, and apologies my late response.

I was trying to highlight that a breaking change should be reflected in the version number – config that worked in version 2 broke in version 2.1, so following semver, the new version should have been 3.

Or is the major part of the config version tied to the overall CircleCI version?

In response to your questions…

Did you get a similar error message?

Yes, that was like the error message I received. It was fixed by replacing the : with _.

Do you have any thoughts on what error message that we could show to make it clear to you what the problem is?

It would be more useful if the line number where the error occurs is displayed – as it stands, I have no idea which job has the invalid name.

True, I can (and did) look at the regex, but that’s only part of the solution: I need to be able to go straight to the source of the error.

Thanks again for your reply.

kind regards

Matt

Roll out of this previously mentioned pipelines feature is now complete. See announcement post for more details.