Bug: Test Workflow for Developing Orbs Reports Warnings in Extension

Ecosystem VS Code Extension
1

This is a bit of an edge case, but when using the VS Code CircleCI extension while developing an orb, the validation repos that my orb references are invalid because I’m referring a dynamic.

For example,

# filename: .circleci/test-deploy.yml
version: 2.1

orbs:
  custom-orb: myorg/custom-orb@dev:<<pipeline.git.revision>>
  orb-tools: circleci/orb-tools@11.6

filters: &filters
  tags:
    only: /.*/

jobs:
  # Create a job to test the commands of your orbs.
  # You may want to add additional validation steps to ensure the commands are working as expected.
  # if we add commands back to the orb, uncomment this again
  command-tests:
    docker:
      - image: cimg/python:3.11-node
    steps:
      - checkout
      # Run your orb's commands to validate them.
      - custom-orb/do-stuff:
          path: ".circleci/testfiles"

This always warns me that my “custom-orb/do-stuff” command isn’t valid. This is an interesting case in that I can appreciate why the extension can’t find it. I wonder if it can’t determine the version of the orb in such a case, but where it thinks maybe I’m trying to develop an orb using the guidelines from the Orb Tools procedures, that maybe it shouldn’t warn me? Or maybe it could do JUST enough to figure out if the orb name has ever been published, for any version, and if so, just then not warn me on all of the commands, but maybe warn near the top to say “hey we know you have an orb but we can’t determine the version, so are unable to validate commands in the rest of the file” or something.

This is all in reference to the guidelines here:

Not 100% sure what the best method is here, but perhaps a collaboration between the orb team and the VSCode Extension team to come up with a happy medium. :slight_smile:

2

Hello @adamdmharvey, thanks for flagging this!
We have noticed this too, and have a ticket open for addressing it.

Our thinking is that, as a first step, we will NOT display errors if we are not able to download the orb and therefore to perform validation. We’d rather show no validation at all, than a false negative.

We did also consider adding a warning on the orb import, as you suggested, but we worry it might become annoying since there is no action you can take to make it disappear. However, if you believe it would be important to show this information somehow, we can reconsider this.

Benny

3

I agree with your first line of thinking; less noise to the developer if we can’t confirm there really IS an issue is likely the right default. Helps me focus on when there are, and will make sure I don’t start to ignore warnings when they are there.

Thanks @Benny !

1 Like