Postgres image breaking changes (role postgres not found)

Yesterday all our builds suddenly started failing on a project which has postgres:10-alpine-ram set up.

I see new tags were built and published yesterday (https://hub.docker.com/r/circleci/postgres/tags/) aligning with when builds started failing.

I don’t see any changes to the postgres Dockerfile, so I’m assuming the breaking change is coming from upstream (https://github.com/CircleCI-Public/circleci-dockerfiles/commit/54985a27f8eb93725645801983414be1beea792d).

The failure I’m seeing is:

2018-09-11 09:07:01.912 UTC [910] FATAL:  role "postgres" does not exist

which is an odd one, since by default postgres user & db should be set up.

connecting to the db as user root fixes it, but I don’t think introducing breaking changes like this to the images is a good practice. I know that the tag 10 is rolling, but I wouldn’t expect such breaking changes.

When there need to be breaking changes, please announce them. Maybe under different tags.

Hopefully someone else will find this solution helpful.

1 Like

How are you starting this container? I believe there are subkeys to specify a username, and I wonder if you could use this to specify a user you wish to connect through.

Having a similar issue, but connecting as root did not solve the issue. Taking a look at the logs between yesterday’s and today’s environments, there is a difference in the “Container circleci/postgres:9.5” step.

Yesterday:

...
waiting for server to start....LOG:  database system was shut down at 2018-09-10 11:56:48 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  autovacuum launcher started
LOG:  database system is ready to accept connections
 done
server started
CREATE DATABASE

CREATE ROLE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
...

and today:

...
waiting for server to start....LOG:  database system was shut down at 2018-09-11 09:07:42 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
 done
server started
CREATE DATABASE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
...

Clearly, the part of this step that is creating the role has been changed or omitted.

We automatically build images as they’re pushed from upstream, so the following change to the postgres image, where the username ‘postgres’ was being hardcoded:

eval "initdb --username=postgres $POSTGRES_INITDB_ARGS"

To respecting the environment variable:

eval 'initdb --username="$POSTGRES_USER" --pwfile=<(echo "$POSTGRES_PASSWORD") '"$POSTGRES_INITDB_ARGS"

Username is now populated with the value of POSTGRES_USER, which we set in the CircleCI dockerfile as root:

ENV POSTGRES_USER=root \
    POSTGRES_DB=circle_test

For now, connecting as the user root should solve the issue, or defining any other username using POSTGRES_USER.

The exact changes to the postgres image can be viewed in this github diff. No changes have been made to the CircleCI dockerfile. Apologies for any inconvenience caused, but sometimes upstream changes do cause issues we can’t predict due to how we currently handle building convenience images.

That’s odd, could you link your build? If it’s closed source it’ll only be visible to CircleCI employees.

Sure thing: https://circleci.com/gh/LumenResearch/viewpoints/890

Thanks for looking into it!

The change is not in line with their (upstream Docker) docs:

POSTGRES_USER
This optional environment variable is used in conjunction with POSTGRES_PASSWORD to set a user and its password. This variable will create the specified user with superuser power and a database with the same name. If it is not specified, then the default user of postgres will be used.

The only difference is that we set POSTGRES_USER in our dockerfile for the convenience image:

For this reason it won’t default to postgres regardless of whether the user sets the environment variable.

I should mention that this is the 9.5 image, but if that’s the only difference, can you think of another reason why the build is failing? Thank you for responding so promptly.

Sorry @ShaunGillies, was replying to @anzecesar – do you have any postgres config in his django settings, and if so could you update this as well? To use root for username and circle_test for the db, although I also can’t see a reason there shouldn’t be a postgres database for django to connect to.

It did before yesterday though. Or for whatever reason, before the build yesterday, postgres user was available.

You could still set it to postgres by defining the environment variable POSTGRES_USER = postgres, but the dockerfile I linked sets it to root. The environment variable wasn’t previously respected – it was set to postgres regardless. Now it is respected, so it’s set to root.

1 Like

Ah, yes, it was still being set to another user name in the settings file. Thank you so much for your help!

1 Like

It looks like the cause and possible solution was already uncovered here, just linking to my response here just in case: CircleCI Postgres Image suddenly causing problems? "role postgres not found"

1 Like

We are pushing changes to mitigate this issue—please see https://discuss.circleci.com/t/convenience-images-update-removing-hardcoded-postgres-user :pray:

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.