Postgres with in memory DB

circle.yml

#1

I’ve tried the following approaches (note: not using the circle postgres as they don’t support the DB version actually in use for us).

All items below are used under Circle 2.0 with workflows (original build):

version: 2
jobs:
  rspec:
      working_directory: /home/circle/repository_name
  docker:
    - image: <privateimage>   # main test environment    
    - image: postgres:9.6.3   # this section keeps getting adjusted in the examples below.
      environment:
        POSTGRES_USER: postgres
        POSTGRES_DB: repository_name_test
        POSTGRES_PASSWORD: postgres

Variations I’ve tried to integration shm and tmpfs:

tmpfs directly:

     - image: postgres:9.6.3
       tmpfs:
         - /var/lib/postgresql/data

Add DB backing to /dev/shm directly:

  - image: postgres:9.6.3
    volumes:
      - "/dev/shm/pg_${PG_PORT}:/var/lib/postgresql/data"

shm via environment variable:

  - image: postgres:9.6.3
    environment:
      - PGDATA: /dev/shm/pgdata/data

Circle documentation optimization section:

# Dockerfile
FROM circleci/postgres
ENV PGDATA /dev/shm/pgdata/data

None seems to have any effect on performance of bundle exec rake db:schema:load or a DB intensive test run (under rspec).

What’s the best way (outside of performance, where I see no difference) to verify that my configuration is being properly applied?

Some links I’ve found on the forum and taken ideas from:


Documentation section (note: I saw the optimization section - look up):
https://circleci.com/docs/2.0/postgres-config


#2

Tagging some users who seem to have something working?

@eseliger - @kernow - @uhlenbrock - @rohara - @gionn - @voidlily


#3

Hi CJ,

What version do you need that isn’t offered here?

https://hub.docker.com/r/circleci/postgres/tags?

Any of our -ram images will have the /dev/shm trick implemented.


#4

Try out this image and let me know how it goes:
circleci/postgres:9.6-alpine-ram


#5

I think tmpfs is not accessible on CircleCI, you can’t use in-memory images (postgres, mongo…).

I already submitted a feature request here: Tmpfs support for docker images


#6

We use MariaDB.


#7

Hi @rohara and @rose - thanks for the updates.

We use 9.6.3. I expect the 9.6.5 image will end up working OK for our purposes, but it’s great if we can exactly match the production environment we’re running against, and we are OK with managing the docker images ourselves as needed. I notice that Circle does not make their Dockerfiles public - is that intentional?

I went ahead with the different version and saw a minor performance boost on a couple runs, but did not do enough to reach any sort of statistical significance.

As with all things, DB is one of many bottlenecks!


#8

Not really, it’s just hard to document. They’re artifacts in the generate_dockerfiles job:
https://circleci.com/gh/circleci/circleci-images/2192#artifacts/containers/0

They should work to generate an image for 9.6.3 :+1:


#9

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.