Unable to set a docker image envar with another envar

Hello,

This is my very first experience building a pipeline from scratch on CircleCI. Currently I am having a hard time configuring a postgres image on my project because of wrong assignments of their required environment variables. Here is how my config.yml looks like now:


version: 2.1
orbs:
  ruby: circleci/ruby@0.1.2

jobs:
  build:
    docker:
      - image: circleci/ruby:2.6.3-stretch-node
        environment:
          RAILS_ENV: test
          RACK_ENV: test
      - image: postgres:12.3
        environment:
          POSTGRES_USER: TEST_DB_USERNAME
          POSTGRES_PASSWORD: TEST_DB_PASSWORD
          POSTGRES_DB: TEST_DB_NAME
    executor: ruby/default
    steps:
      ...

I tried many approaches already: using parameters, interpolation, nothing worked. Can anyone more experienced with this help me out? To be clear, I have set on the project level the said envars: TEST_DB_USERNAME, TEST_DB_PASSWORD, TEST_DB_NAME.

EDIT:
The exact step that fails on my build is a plain and simple db setup, like so:

- run:
    name: Prepare database
    command: bundle exec rake db:setup

Here is the execution outcome:

Stack trace
#!/bin/bash -eo pipefail
bundle exec rake db:setup
FATAL:  role "*********" does not exist
Couldn't create '**************' database. Please check your configuration.
rake aborted!
PG::ConnectionBad: FATAL:  role "*********" does not exist
/usr/local/bundle/ruby/2.6.0/gems/pg-1.2.3/lib/pg.rb:58:in `initialize'
/usr/local/bundle/ruby/2.6.0/gems/pg-1.2.3/lib/pg.rb:58:in `new'
/usr/local/bundle/ruby/2.6.0/gems/pg-1.2.3/lib/pg.rb:58:in `connect'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/postgresql_adapter.rb:46:in `postgresql_connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:887:in `new_connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:931:in `checkout_new_connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:910:in `try_to_checkout_new_connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:871:in `acquire_connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:593:in `checkout'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:437:in `connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:1119:in `retrieve_connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_handling.rb:221:in `retrieve_connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_handling.rb:189:in `connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/postgresql_database_tasks.rb:12:in `connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/postgresql_database_tasks.rb:21:in `create'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:126:in `create'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:185:in `block in create_current'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:479:in `block (2 levels) in each_current_configuration'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:476:in `each'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:476:in `block in each_current_configuration'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:475:in `each'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:475:in `each_current_configuration'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:184:in `create_current'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/railties/databases.rake:39:in `block (2 levels) in <main>'
/usr/local/bundle/ruby/2.6.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `load'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `kernel_load'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:28:in `run'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/cli.rb:476:in `exec'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/cli.rb:30:in `dispatch'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/cli.rb:24:in `start'
/usr/local/bundle/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/usr/local/bundle/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/usr/local/bundle/bin/bundle:23:in `load'
/usr/local/bundle/bin/bundle:23:in `<main>'
Tasks: TOP => db:setup => db:schema:load_if_ruby => db:create
(See full trace by running task with --trace)

Exited with code exit status 1
CircleCI received exit code 1

Hi Daniel,

Welcome to CircleCI Discuss! For my own clarification, are you defining the POSTGRES_USER environment variable on the project level, or are you trying to have POSTGRES_USER be equal to what TEST_DB_USERNAME is equal to within the project level environment variables?

I am trying to have POSTGRES_USER equal to TEST_DB_USERNAME; POSTGRES_PASSWORD equal to TEST_DB_PASSWORD; and POSTGRES_DB equal to TEST_DB_NAME.

In other words, I have configured the TEST_* envars in the project level.

So as you may have already noticed, you cannot reference a project level environment variable from the job environment parameter. I would initially recommend storing the POSTGRES_* environment variables within the project level, but I take it that you are not looking to do that.

Here’s some playing around I did trying to figure out a good solution for you: https://app.circleci.com/pipelines/github/Jaryt/Nested-EV-test/9/workflows/09205d31-2a95-45bd-bb32-166d76b1b2c8/jobs/10 (The output is masked because I have environment variable masking enabled.)

I eventually found that the best course of action for you would be along the lines of this configuration example. Assuming that you’re doing this for environment variable security purposes, I would recommend cloning the file at $BASH_ENV as a temp, and then reset it to that after your Postgres steps.

Thank you for your quick reply, Jaryt.

Something is puzzling me with this approach you proposed. The appending that takes place on BASH_ENV would occur only after the spin up of the postgres image which requires the configuration. It fails right away.

Am I missing something here? Is there a way to execute this command before the docker part comes into play? Here is my config.yml now:

config.yml
version: 2.1
orbs:
  ruby: circleci/ruby@0.1.2

jobs:
  build:
    docker:
      - image: circleci/ruby:2.6.3-stretch-node
        environment:
          RAILS_ENV: test
          RACK_ENV: test
      - image: postgres:12.3
    executor: ruby/default
    steps:
      - run:
          name: Define Environment Variables at Runtime
          command: |
            echo 'export POSTGRES_USER=$TEST_DB_USERNAME' >> $BASH_ENV
            echo 'export POSTGRES_PASSWORD=$TEST_DB_PASSWORD' >> $BASH_ENV
            echo 'export POSTGRES_DB=$TEST_DB_NAME' >> $BASH_ENV
            source $BASH_ENV
      ...

Good point! My recommendation to get around this would be to create another job specifically for setting up those environment variables. The changes made to BASH_ENV will remain, and should then be accessible from the postgres image.

Thanks for getting back to me again, Jaryt. I tried following your advice, but I am still not there yet. I have a pipeline now with a build job configuring BASH_ENV, but unfortunately my additions there do not hold on consecutive jobs.

Here is my config file:

.circleci/config.yml
version: 2.1
orbs:
  ruby: circleci/ruby@0.1.2

jobs:
  build:
    docker:
      - image: circleci/ruby:2.6.3-stretch-node
    working_directory: ~/exploreit-space
    steps:
      - checkout
      - attach_workspace:
          at: ~/exploreit-space
      - run:
          name: Define Environment Variables at Runtime
          command: |
            echo "export POSTGRES_USER=$TEST_DB_USERNAME" >> $BASH_ENV
            echo "export POSTGRES_PASSWORD=$TEST_DB_PASSWORD" >> $BASH_ENV
            echo "export POSTGRES_DB=$TEST_DB_NAME" >> $BASH_ENV
            source $BASH_ENV
      - run: env | grep POSTGRES
  tests:
    docker:
      - image: circleci/ruby:2.6.3-stretch-node
#      - image: postgres:12.3
    working_directory: ~/exploreit-space
    steps:
      - run: env | grep POSTGRES
#      - run:
#          name: Install dependencies
#          command: |
#            gem install bundler
#            bundle install
#            yarn install --check-files
#      - run:
#          name: Prepare database
#          command: bundle exec rake db:setup
#      - run:
#          name: Linter
#          command: bundle exec rake standard
#      - run:
#          name: Specs
#          command: bundle exec rspec spec

workflows:
  version: 2

  exploreit:
    jobs:
      - build
      - tests:
          requires:
            - build

In order to debug this process, I execute $ env on both jobs. The result of the build one seems to get new vars right… but I can’t say the same for tests:

$env | grep POSTGRES on `build`

$env | grep POSTGRES on `tests`

Any idea of what I could be missing?

Cheers,
Daniel