Postgres failing with upgrading from 11.6 to 11.14

Hi, upgrading from Ruby 2.6.6 to 2.6.9, that seems to be working, but now when updating from postgres 11.6-alpine to postgres 11.14 things get stuck (I’ve also tried 11.13 from cimg). It look like the Postgres server gets setup but then is forced to shut down, so the DB setup doesn’t complete:

From the postgres setup:

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

waiting for server to start....2022-02-14 09:08:37.421 UTC [205] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-02-14 09:08:37.430 UTC [210] LOG:  database system was shut down at 2022-02-14 09:08:37 UTC
2022-02-14 09:08:37.433 UTC [205] LOG:  database system is ready to accept connections
 done
server started
CREATE DATABASE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

2022-02-14 09:08:37.801 UTC [205] LOG:  received fast shutdown request
waiting for server to shut down...2022-02-14 09:08:37.801 UTC [205] LOG:  aborting any active transactions
.2022-02-14 09:08:37.802 UTC [205] LOG:  background worker "logical replication launcher" (PID 216) exited with exit code 1
2022-02-14 09:08:37.802 UTC [211] LOG:  shutting down
2022-02-14 09:08:37.807 UTC [205] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2022-02-14 09:08:37.909 UTC [82] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2022-02-14 09:08:37.909 UTC [82] LOG:  listening on IPv6 address "::", port 5432
2022-02-14 09:08:37.909 UTC [82] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-02-14 09:08:37.919 UTC [240] LOG:  database system was shut down at 2022-02-14 09:08:37 UTC
2022-02-14 09:08:37.922 UTC [82] LOG:  database system is ready to accept connections
2022-02-14 09:09:45.766 UTC [3250] LOG:  incomplete startup packet


Build was canceled

From the DB Setup:

#!/bin/bash -eo pipefail
bin/rails db:structure:load --trace
** Invoke db:structure:load (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
DEPRECATION: Skylight support was moved into the `skylight` gem.
** Execute db:load_config
** Invoke db:check_protected_environments (first_time)
** Invoke db:load_config 
** Execute db:check_protected_environments
** Execute db:structure:load
WARNING: terminal is not fully functional
-  (press RETURN)a set_config 
------------
 
(1 row)

Config.yml:

- image: circleci/postgres:11.14 # database image
        environment: # environment variables for database
          POSTGRES_USER: postgres
          POSTGRES_HOST_AUTH_METHOD: trust
          POSTGRES_DB: v12_test
          POSTGRES_PASSWORD: ""

Anyone seen this or have a fix?

I’ve also seen this, but it seems to be related to moving to the cimg images so that I could use Ruby 3.1 in my case. I’ve tried reverting to the old-style postgres image and that doesn’t fix it, so it seems to be an issue with the ruby runner. Here’s the offending commit:

Step output:

#!/bin/bash -eo pipefail
bin/rake db:setup --trace

Calling `DidYouMean::SPELL_CHECKERS.merge!(error_name => spell_checker)' has been deprecated. Please call `DidYouMean.correct_error(error_name, spell_checker)' instead.
** Invoke db:setup (first_time)
** Invoke db:create (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:load_config
** Execute db:create
Created database 'lifework_test'
** Invoke environment 
** Invoke db:schema:load (first_time)
** Invoke db:load_config 
** Invoke db:check_protected_environments (first_time)
** Invoke db:load_config 
** Execute db:check_protected_environments
** Execute db:schema:load
WARNING: terminal is not fully functional
-  (press RETURN)a set_config 
------------
 
(1 row)

(END)a(END)rake aborted!
Interrupt: 
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/activerecord-6.1.4.6/lib/active_record/tasks/postgresql_database_tasks.rb:111:in `system'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/activerecord-6.1.4.6/lib/active_record/tasks/postgresql_database_tasks.rb:111:in `run_cmd'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/activerecord-6.1.4.6/lib/active_record/tasks/postgresql_database_tasks.rb:86:in `structure_load'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/activerecord-6.1.4.6/lib/active_record/tasks/database_tasks.rb:314:in `structure_load'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/activerecord-6.1.4.6/lib/active_record/tasks/database_tasks.rb:328:in `load_schema'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/activerecord-6.1.4.6/lib/active_record/tasks/database_tasks.rb:435:in `block in load_schema_current'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/activerecord-6.1.4.6/lib/active_record/tasks/database_tasks.rb:506:in `block (2 levels) in each_current_configuration'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/activerecord-6.1.4.6/lib/active_record/tasks/database_tasks.rb:503:in `each'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/activerecord-6.1.4.6/lib/active_record/tasks/database_tasks.rb:503:in `block in each_current_configuration'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/activerecord-6.1.4.6/lib/active_record/tasks/database_tasks.rb:502:in `each'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/activerecord-6.1.4.6/lib/active_record/tasks/database_tasks.rb:502:in `each_current_configuration'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/activerecord-6.1.4.6/lib/active_record/tasks/database_tasks.rb:434:in `load_schema_current'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/activerecord-6.1.4.6/lib/active_record/railties/databases.rake:462:in `block (3 levels) in <top (required)>'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/bugsnag-6.24.2/lib/bugsnag/integrations/rake.rb:20:in `execute'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:243:in `block in invoke_prerequisites'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `each'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `invoke_prerequisites'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/home/circleci/lw-web/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
bin/rake:4:in `<main>'
Tasks: TOP => db:setup => db:schema:load

Too long with no output (exceeded 10m0s): context deadline exceeded

I also tried following this suggestion to add TERM=xterm but that only gets rid of “WARNING: terminal is not fully functional” - not the timeout

I was able to fix this following the solution in this PR to rails that unfortunately was never merged. It seems that sometimes this rake task triggers psql interactive mode, which causes the process to hang. The solution is to add PSQL_PAGER: '' to your environment block for the ruby image, ex:

version: 2.1
jobs:
  build:
    docker:
      - image: cimg/ruby:3.1.0-browsers
        environment:
          NODE_ENV: test
          RAILS_ENV: test
          PSQL_PAGER: ''
1 Like

THANK YOU! That worked and the tests were able to run! :slight_smile: I’m not sure how you figured that out, but I’m thankful you did.

Is it worth commenting on that closed PR or submitting a new bug to Rails at this point? It’s over my head to submit the technical nature of why, but happy to help however I can.

Thank you again.

Is this even Rails’ problem to fix? I only upgraded my base image version, I don’t think I changed the Postgres version at all. What actually changed between the two images that triggered this bug? Without more details, it’s unclear whether this is even something that Rails needs to fix or just another Circle mistake.

Either way, huge thank you for posting this fix!

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.