Unable to set a docker image envar with another envar

danielfsbarreto · May 29, 2020, 3:32pm

Hello,

This is my very first experience building a pipeline from scratch on CircleCI. Currently I am having a hard time configuring a postgres image on my project because of wrong assignments of their required environment variables. Here is how my config.yml looks like now:


version: 2.1
orbs:
  ruby: circleci/ruby@0.1.2

jobs:
  build:
    docker:
      - image: circleci/ruby:2.6.3-stretch-node
        environment:
          RAILS_ENV: test
          RACK_ENV: test
      - image: postgres:12.3
        environment:
          POSTGRES_USER: TEST_DB_USERNAME
          POSTGRES_PASSWORD: TEST_DB_PASSWORD
          POSTGRES_DB: TEST_DB_NAME
    executor: ruby/default
    steps:
      ...

I tried many approaches already: using parameters, interpolation, nothing worked. Can anyone more experienced with this help me out? To be clear, I have set on the project level the said envars: TEST_DB_USERNAME, TEST_DB_PASSWORD, TEST_DB_NAME.

EDIT:
The exact step that fails on my build is a plain and simple db setup, like so:

- run:
    name: Prepare database
    command: bundle exec rake db:setup

Here is the execution outcome:

Stack trace

#!/bin/bash -eo pipefail
bundle exec rake db:setup
FATAL:  role "*********" does not exist
Couldn't create '**************' database. Please check your configuration.
rake aborted!
PG::ConnectionBad: FATAL:  role "*********" does not exist
/usr/local/bundle/ruby/2.6.0/gems/pg-1.2.3/lib/pg.rb:58:in `initialize'
/usr/local/bundle/ruby/2.6.0/gems/pg-1.2.3/lib/pg.rb:58:in `new'
/usr/local/bundle/ruby/2.6.0/gems/pg-1.2.3/lib/pg.rb:58:in `connect'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/postgresql_adapter.rb:46:in `postgresql_connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:887:in `new_connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:931:in `checkout_new_connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:910:in `try_to_checkout_new_connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:871:in `acquire_connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:593:in `checkout'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:437:in `connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:1119:in `retrieve_connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_handling.rb:221:in `retrieve_connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/connection_handling.rb:189:in `connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/postgresql_database_tasks.rb:12:in `connection'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/postgresql_database_tasks.rb:21:in `create'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:126:in `create'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:185:in `block in create_current'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:479:in `block (2 levels) in each_current_configuration'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:476:in `each'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:476:in `block in each_current_configuration'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:475:in `each'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:475:in `each_current_configuration'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/tasks/database_tasks.rb:184:in `create_current'
/usr/local/bundle/ruby/2.6.0/gems/activerecord-6.0.3.1/lib/active_record/railties/databases.rake:39:in `block (2 levels) in <main>'
/usr/local/bundle/ruby/2.6.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `load'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `kernel_load'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:28:in `run'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/cli.rb:476:in `exec'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/cli.rb:30:in `dispatch'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/cli.rb:24:in `start'
/usr/local/bundle/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/usr/local/bundle/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/usr/local/bundle/bin/bundle:23:in `load'
/usr/local/bundle/bin/bundle:23:in `<main>'
Tasks: TOP => db:setup => db:schema:load_if_ruby => db:create
(See full trace by running task with --trace)

Exited with code exit status 1
CircleCI received exit code 1

Jaryt · May 29, 2020, 3:43pm

Hi Daniel,

Welcome to CircleCI Discuss! For my own clarification, are you defining the POSTGRES_USER environment variable on the project level, or are you trying to have POSTGRES_USER be equal to what TEST_DB_USERNAME is equal to within the project level environment variables?

danielfsbarreto · May 29, 2020, 3:51pm

I am trying to have POSTGRES_USER equal to TEST_DB_USERNAME; POSTGRES_PASSWORD equal to TEST_DB_PASSWORD; and POSTGRES_DB equal to TEST_DB_NAME.

In other words, I have configured the TEST_* envars in the project level.

Jaryt · May 29, 2020, 4:22pm

So as you may have already noticed, you cannot reference a project level environment variable from the job environment parameter. I would initially recommend storing the POSTGRES_* environment variables within the project level, but I take it that you are not looking to do that.

Here’s some playing around I did trying to figure out a good solution for you: https://app.circleci.com/pipelines/github/Jaryt/Nested-EV-test/9/workflows/09205d31-2a95-45bd-bb32-166d76b1b2c8/jobs/10 (The output is masked because I have environment variable masking enabled.)

I eventually found that the best course of action for you would be along the lines of this configuration example. Assuming that you’re doing this for environment variable security purposes, I would recommend cloning the file at $BASH_ENV as a temp, and then reset it to that after your Postgres steps.

danielfsbarreto · May 29, 2020, 6:07pm

Thank you for your quick reply, Jaryt.

Something is puzzling me with this approach you proposed. The appending that takes place on BASH_ENV would occur only after the spin up of the postgres image which requires the configuration. It fails right away.

Am I missing something here? Is there a way to execute this command before the docker part comes into play? Here is my config.yml now:

config.yml

version: 2.1
orbs:
  ruby: circleci/ruby@0.1.2

jobs:
  build:
    docker:
      - image: circleci/ruby:2.6.3-stretch-node
        environment:
          RAILS_ENV: test
          RACK_ENV: test
      - image: postgres:12.3
    executor: ruby/default
    steps:
      - run:
          name: Define Environment Variables at Runtime
          command: |
            echo 'export POSTGRES_USER=$TEST_DB_USERNAME' >> $BASH_ENV
            echo 'export POSTGRES_PASSWORD=$TEST_DB_PASSWORD' >> $BASH_ENV
            echo 'export POSTGRES_DB=$TEST_DB_NAME' >> $BASH_ENV
            source $BASH_ENV
      ...

Jaryt · June 3, 2020, 5:51pm

Good point! My recommendation to get around this would be to create another job specifically for setting up those environment variables. The changes made to BASH_ENV will remain, and should then be accessible from the postgres image.

danielfsbarreto · June 5, 2020, 12:47am

Thanks for getting back to me again, Jaryt. I tried following your advice, but I am still not there yet. I have a pipeline now with a build job configuring BASH_ENV, but unfortunately my additions there do not hold on consecutive jobs.

Here is my config file:

.circleci/config.yml

version: 2.1
orbs:
  ruby: circleci/ruby@0.1.2

jobs:
  build:
    docker:
      - image: circleci/ruby:2.6.3-stretch-node
    working_directory: ~/exploreit-space
    steps:
      - checkout
      - attach_workspace:
          at: ~/exploreit-space
      - run:
          name: Define Environment Variables at Runtime
          command: |
            echo "export POSTGRES_USER=$TEST_DB_USERNAME" >> $BASH_ENV
            echo "export POSTGRES_PASSWORD=$TEST_DB_PASSWORD" >> $BASH_ENV
            echo "export POSTGRES_DB=$TEST_DB_NAME" >> $BASH_ENV
            source $BASH_ENV
      - run: env | grep POSTGRES
  tests:
    docker:
      - image: circleci/ruby:2.6.3-stretch-node
#      - image: postgres:12.3
    working_directory: ~/exploreit-space
    steps:
      - run: env | grep POSTGRES
#      - run:
#          name: Install dependencies
#          command: |
#            gem install bundler
#            bundle install
#            yarn install --check-files
#      - run:
#          name: Prepare database
#          command: bundle exec rake db:setup
#      - run:
#          name: Linter
#          command: bundle exec rake standard
#      - run:
#          name: Specs
#          command: bundle exec rspec spec

workflows:
  version: 2

  exploreit:
    jobs:
      - build
      - tests:
          requires:
            - build

In order to debug this process, I execute $ env on both jobs. The result of the build one seems to get new vars right… but I can’t say the same for tests:

$env | grep POSTGRES on `build`

$env | grep POSTGRES on `tests`

Any idea of what I could be missing?

Cheers,
Daniel

Topic		Replies	Views
How to securely set environment variables for Postgres Image? Build Environment	3	5354	July 24, 2020
DATABASE_URL being ignored Build Environment	6	1949	November 26, 2018
Mystery DATABASE_USER=ubuntu Setting Build Environment postgresql	5	2925	December 18, 2018
Local CLI fails to set environment variables Build Environment	0	552	March 17, 2020
Using the circleci/mysql and circleci/postgres images simultaneously in the same job Databases	3	2870	February 14, 2020

Unable to set a docker image envar with another envar

Related topics