Trying to use the AWS cli (or any) CircleCI orb for the first time, plus how to run shell scripts...help me understand

We use CircleCI to run node.js integration tests, and it works great for us. We can run them manually or by checking in code to the corresponding github branch. It’s perfect.

However, we’re now trying to gain access to aws in one of the tests, and I’m a bit confused on how to do this. Here’s our current (somewhat simplified) config.ym file:

  jobs:
  build:
    docker:
    - image: circleci/node:12.9
    working_directory: ~/test-runner
    steps:
    - checkout
    - restore_cache:
        keys:
        - v1-dependencies-{{ checksum "package.json" }}
        - v1-dependencies-
    - run: echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > ~/.npmrc
    - run: npm install
    - save_cache:
        paths:
        - node_modules
        key: v1-dependencies-{{ checksum "package.json" }}
    - run: npm run test
    - run: cd ~/test-runner
    - run: echo "[default]" > ~/.aws/credentials
    - run: echo "aws_access_key_id = ${AWS_ACCESS_KEY}" >> ~/.aws/credentials
    - run: echo "aws_secret_access_key = ${AWS_SECRET_ACCESS_KEY}" >> ~/.aws/credentials
    - run: echo "[default]" > ~/.aws/config
    - run: echo "region = us-east-1" >> ~/.aws/config
workflows:
  version: 2
  workflow:
    jobs:
    - build

As you can see, we’re using your circleci/node image currently, and I’m in the process of getting our aws credentials lined up in the executor. That said, I’m not quite sure I follow orb usage…would the aws-cli orb replace our node image? Or augment it somehow, in the orbs section of the config?

Also, I had a question about running shell scripts and how best to do it in CircleCI. As the config above shows I’m doing some pretty simple stuff, single-line commands and simple run statements. What we’ll need to do with aws is run a script like this one to grant some whitelisting in our aws instance using the aws cli and a shell script.

How do I run or call a script like that in CircleCI, is it done from my config file, and if so do I echo it out somehow or some other way?

Sorry for the basic nature of these questions, it’s just that I’ve never done these things in CircleCI.

Bump. Still trying to figure out how / if we can run shell commands in a CCI script. Particularly interested in using the aws cli in it.

1 Like

Maybe @KyleTryon can help can help?

1 Like

I can certainly try.

Hello @thekatertot :wave:

The example you are showing above does not seem to contain orbs, though orbs are not at all required to do anything. Orbs can however make a lot of what you have here easier (potentially).

You can absolutely use more than a single orb and intermix them. Each orb will provide a set of useful components that can be put together for instance to create a job for the task you are attempting.

Here is a simple example a manual nodejs testing job using the node orb’s “install-packages” command and the executor.

version: 2.1
orbs:
  node: 'circleci/node:x.y' #importing the node orb here. The version referrs to the version of the orb.
jobs:
  test:
    executor:
      name: node/default # From the node orb, provides a nodejs docker image
      tag: '13'
    steps:
      - checkout #standard CircleCI command to checkout code
      - node/install-packages # from the node orb. Installs and caches our npm modules
      - run:
          command: npm run test
workflows:
  test_my_app:
    jobs:
      - test

Registry page for the node orb: https://circleci.com/orbs/registry/orb/circleci/node#quick-start

Now, if we wanted to use the AWS CLI, we have an orb for that as well.
This useage example demonstrates how we can install the AWS CLI and set up our credentials with a single command “aws/setup”
https://circleci.com/orbs/registry/orb/circleci/aws-cli#usage-install-aws-cli

steps:
      - checkout
      - aws-cli/setup:
          profile-name: example
      - run: echo "Run your code here"

To clear up the question about what executor to use, that is entirely up to you. You dont have to use the executor from either of these orbs, you could simply just use the commands provided. If you attempted to use the “install-packages” command from the node orb in an environment that did not have node install though, there would obviously be an error, so for that reason, the Node orb provides this executor for your convenience, but it is not required.

The executor in the AWS CLI orb provides by default, a docker image with both Python and Node installed, which is typical for some serverless applications deployed to AWS for instance.
https://circleci.com/orbs/registry/orb/circleci/aws-cli#executors-default

So lets look at our config example again, a little closer to what you asked for.

version: 2.1
orbs:
  node: 'circleci/node:x.y' #importing the node orb here. The version referrs to the version of the orb.
  aws-cli: circleci/aws-cli@x.y
jobs:
  test:
    executor:
      name: node/default # From the node orb, provides a nodejs docker image
      tag: '13'
    steps:
      - checkout #standard CircleCI command to checkout code
      - aws-cli/setup
      - node/install-packages # from the node orb. Installs and caches our npm modules
      - run:
          name: my custom script
          command: |
              echo "my script"
      - run:
          command: npm run test
workflows:
  test_my_app:
    jobs:
      - test

To get an idea of how we use shell scrips in commands and such, you can take a look at the source code of the node orb itself, as all orbs are actually just config.

Hope that answered the majority of your questions. Let me know if that clears everything up :+1:

2 Likes

Thanks Kyle, this is very helpful info. I’ve been going over the docs regarding orbs and wasn’t sure quite how to slot them into the mix-- as I mentioned in my original post, we use CircleCI for running integration tests and have been happy with that, but never stretched beyond it.

I’ll review your tips and directions and do another read through on the docs. Much appreciated.

One question, if I may-- in the sample yaml files you attached some of the keys have greater than signs and pipe symbols after them ('command: |" “description: >”.) I’m not entirely sure what they mean and if there’s a doc out there that explains it I’d be grateful to review it.

1 Like

Hey @larryq :wave:

These pipes and such are actually just a regular part of YAML.
https://yaml-multiline.info/

The | indicates that everything indented below should be treated as a string literal, preserving those new lines, which you can imagine, would be crucial to executing our code properly.

If we had only a single line of code to run, we wouldnt need any symbol at all.

The > symbol takes all of the content indented and collapses it to a single line string.

I actually personally like to only use the |, but orbs are open source and we haven’t been that strict about style.

2 Likes

Ah, ok, thanks Kyle. BTW, your suggestions above regarding the aws orb worked great, I’ve got dynamic whitelisting working now, so many thanks there as well.

For others to look at, in case they need it, here is the relevant info in our config.yaml for our whitelisting:

- aws-cli/setup:
    aws-access-key-id: AWS_ACCESS_KEY
    aws-region: AWS_REGION
- run:
    name: open IP
    command: |
      public_ip_address=$(wget -qO- http://checkip.amazonaws.com)
      echo "this computers public ip address is $public_ip_address"
      aws ec2 authorize-security-group-ingress --region $AWS_REGION --group-id <our group id> --ip-permissions "[{\"IpProtocol\": \"tcp\", \"FromPort\": <our ports>, \"ToPort\": <our ports>, \"IpRanges\": [{\"CidrIp\": \"${public_ip_address}/32\"}]}]"

afterwards you can unwhitelist by using the aws ec2 revoke-security-group-ingress call with the same parameters. One thing I need to do is figure out how to run that command no matter what, in a separate job or whatnot, because right now if our testing step fails all subsequent steps stop as well, so the unwhitelisting doesn’t happen.

2 Likes

How cool! You could even roll this into your own orb if you want! We’ll have to find a good way to share that or maybe even consider adding it to the orb. Very cool idea :tada:

To run the revoke command, you can in the same job run a step which will run no matter what.
https://circleci.com/docs/2.0/configuration-reference/#the-when-attribute


- aws-cli/setup:
    aws-access-key-id: AWS_ACCESS_KEY
    aws-region: AWS_REGION
- run:
    name: open IP
    command: |
      public_ip_address=$(wget -qO- http://checkip.amazonaws.com)
      echo "this computers public ip address is $public_ip_address"
      aws ec2 authorize-security-group-ingress --region $AWS_REGION --group-id <our group id> --ip-permissions "[{\"IpProtocol\": \"tcp\", \"FromPort\": <our ports>, \"ToPort\": <our ports>, \"IpRanges\": [{\"CidrIp\": \"${public_ip_address}/32\"}]}]"
- run:
    name: Do stuff
    command: echo "foo" && exit 1 # cause an error, which will halt the job.
- run:
    name: Clear Allow List
    command: aws ec2 revoke-security-group-ingress ...
    when: always # Will run no matter if a previous step has failed

1 Like