However, I’m finding that when this is done, it prevents environment variables from being passed along to subshells in shell scripts in accordance with the documented order of precedence. For example, things like the following won’t work (the highest-precedence example in Circle CI’s list):
FOO=bar make install
This is a problem because a repository can have lots of shell scripts that use this method to override environment variables when calling other scripts, and it will cause all of them not to work (or to break subtly).
I did find this Circle CI page (“Unable to Override PATH or NVM_DIR Environment Variables with Ubuntu 20.04 Machine Images”) from July 2022, but it only mentions problems with using Circle CI’s environment parameter – not general overriding of environment variables within shell scripts.
You can reproduce this by adding a script test.sh to your repository with the following contents:
- run: FOO=200 ; ./test.sh # Will display 100 not 200.
rather than
- run: FOO=200 ./test.sh # Will display 100 not 200.
Without the semicolon, you are not asking bash to execute a sequence of commands. I am no expert in terms of the inner workings of BASH so can not say exactly what is going on, but the above change results in 200 being displayed rather than 100.
Using the command
run: echo $FOO; FOO=200 ; echo $FOO
Shows the flow better, with 100 and then 200 being shown as the output. So the value from $BASH_ENV is loaded and available, but then correctly overwritten with the new value that is set.
I tried with the semicolon, and it still shows 100. If you look at the CircleCI section on order-of-precedence I linked to, you will see that CircleCI also doesn’t include a semicolon in their example:
Environment variables declared inside a shell command in a run step, for example FOO=bar make install.
You will see that it does work, but what do you have in your test.sh as anything that causes a new shell to be created will also cause $BASH_ENV to be run for the new shell.
The example that does not contain the semicolon seems to be a circleci ‘feature’/‘complication’ as a full statement a few lines below the one you have quoted, it states
Environment variables declared inside a shell command run step, for example FOO=bar make install, will override environment variables declared with the environment and contexts keys.
So they are changing the values provided by CircleCI at the time that the shell is executed. The shell is still going to load any values defined in $BASH_ENV as part of its start-up steps.
@rit1010 Your replies aren’t really helping with the issue I’m raising.
I have a large set up with lots of shell scripts that works fine with Travis CI. But I’m having trouble finding a way to get it to work in Circle CI because Circle CI’s suggestion to use BASH_ENV to persist dynamic environment variables breaks the ability to override environment variables when calling other scripts – anywhere in any script.
Thus, I’m looking for possible ways / workarounds to help convert a working setup from another provider without having to make major changes across many shell scripts.
Well, that is the first time you have raised Travis CI as a ‘base line’ for what you are trying to do.
So far we have discussed the example script you have provided with the 3 following things being clear
$BASH_ENV is not connected to the order of precedence used by the CircleCI agent as it is a bash shell based solution.
$BASH_ENV will always override any variable values passed to a new shell as it is executed after the shell is created.
The setting of a variable as detailed in the docs you have linked to is a CircleCI ‘feature’ and so overwritten by $BASH_ENV.
As such $BASH_ENV is not a solution that can work as you wish to push down environment variables to different shell’s which are also expected to inherit environment variables from the calling shell.
In terms of the wider issue to allow you to operate in the same way that you have done within a Travis CI environment. Your example has 3 independent run commands and so 3 independent shell’s. Do you need this in real life or do you just run a single script (in a single shell) that then runs different scripts?
It’s not just that it works fine on Travis. That’s also how it works during a terminal session: You can set some environment variables at the beginning, and you can recursively override those variables on a per-script basis, in nested fashion. With Circle CI and their suggestion to use BASH_ENV to mimic variable persistence, you lose that ability: every script invocation starts things over with BASH_ENV, even if you’re further down in your scripts.
Yes. In real life, I have multiple (top-level) run steps that call scripts, interspersed between Circle CI primitives like cache operations and orb commands. And those top-level scripts recursively call other scripts.
I guess that means that Travis CI allows you to set environment variables within the shell environment that it itself executes in.
CircleCI is very different as it does not allow this root/top shell to have its environment changed, instead, all run commands are executed in new shells, hence the use of the $BASH_ENV to move variables between different shell instances. The one thing this solution/workaround fails to take into account is the use extensive use of environment variables and multi-level use of shells as the shell will keep reloading the values held in $BASH_ENV, which is the problem you are now facing.
The only idea that I can put forward is that you operate an additional file where you place variables that you want to persist between run commands and you execute it only at the start of each run step. This would allow you to control its execution, rather than having BASH run it at all times.
Yes, that’s the same conclusion I came to. It will require a small amount of boilerplate at the beginning of each run step, but at least it’s not much.
It would have been nice if Circle CI supported something like this out of the box. For example, using the same approach as BASH_ENV, there could be a CIRCLE_ENV variable you could write to that Circle CI would automatically source at the beginning of every run step. But unlike BASH_ENV, it wouldn’t interfere with bash’s behavior and get reloaded every time bash is used within a step.
hi @cjerdonek - I can see you are using one of our provided machine images. We do have something similar to a CIRCLE_ENV: the ~/.circlerc file gets sourced on every run step through ~/.bashrc.
I think this would be a good place to apply what you’re suggesting