Builds started failing around PATH/BASH_ENV usage

[lightpriest]

Our builds have started to fail, specifically those the depends on populating and re-using $PATH.
Was there any change done recently that prevents this usage or something?

The agent version that runs for us is: Build-agent version 1.0.38841-5271c1f7 (2020-08-28T15:52:24+0000).
The failure is in one of the commands of the https://circleci.com/orbs/registry/orb/circleci/aws-sam-serverless orb, though we didn’t change a version or configuration that might trigger this.

Eventually the error that we see is:

/bin/bash: line 2: sam: command not found

Although this command worked as expected in previous commits (older agent versions).

When running with SSH, we needed to manually source $BASH_ENV for it to work. Was that always the case? Or was $BASH_ENV supposed to be sourced automatically in a bash session?

[annapamela]

Hi @lightpriest and welcome to the forum!

When running with SSH, we needed to manually source $BASH_ENV for it to work. Was that always the case? Or was $BASH_ENV supposed to be sourced automatically in a bash session?

Yes, you do usually need to manually source $BASH_ENV in the SSH container. The shells in your SSH container and your job container are slightly different.

The default shell is a non-login shell, while in SSH, we prepend - to the shell when not specified to force it to run a login shell. This article walks through which files the shell sources: https://medium.com/@rajsek/zsh-bash-startup-files-loading-order-bashrc-zshrc-etc-e30045652f2e.

Since the job is non-interactive, it follows the “Interactive? -> No. -> --login? -> No. -> $BASH_ENV” path. The SSH job is interactive and therefore follows a different path.

So for example, in a config like this:

version: 2.1

jobs:
  my-test-job:
    docker:
      - image: circleci/ruby:2.6.0-stretch
    steps:
      - checkout
      - run: |
          echo "export FOO='BAR'" >> $BASH_ENV
          cat $BASH_ENV
      - run: echo $FOO

workflows:
  tests:
    jobs:
      - my-test-job

BAR will print in downstream steps as expected. However, if you SSH into the job, you will have to source $BASH_ENV to have access to $FOO:

circleci@3be2ac6111ff:~$ echo $FOO

circleci@3be2ac6111ff:~$ source $BASH_ENV
circleci@3be2ac6111ff:~$ echo $FOO
BAR

Finally, if you are continuing to encounter issues with the orb, please don’t hesitate to submit an issue on the orb source Github page: https://github.com/CircleCI-Public/aws-sam-serverless-orb/issues. Our Community & Partner Engineering team reviews these issues regularly!

I hope that clarifies things! Please do let me know if you have additional questions!

[lightpriest]

Thanks a lot for the response. Eventually, yeah, we figured it was an issue with sam’s brew tap, and that simply calling brew install .. twice works around it, for the time being.