When running CI for one of our rails projects, we sometimes observe that the postgres container (postgres-9.6-alpine) takes an unusually long time to start, on the order of 6-9 minutes. When this happens, the container usually emits its last log traffic at about the 2m mark. The rest of the time it appears to be hung on something. Has anyone else seen this happen? It doesn’t affect the outcome of our tests as we seem to be properly waiting for the container to come ready, but slow CI is a productivity killer.
I’m not familiar with this image, but there are a few things you could try:
- Play around with the
entrypointfor this image to make start-up more verbose (I am not sure if this is visible in build output though, would be worth investigating)
- Install Postgres into your primary container, and drop the secondary container
- Build a custom primary image containing the things you need, including Postgres
- Build a custom secondary image containing Postgres
Thanks for the debugging ideas! I think I figured out that several of my assumptions were wrong. Specifically:
- We do not appear to be waiting on the postgres container setup task to complete.
- The postgres container setup task does not complete in a way that is recognized by CircleCI, but it does start up a usable postgresql server.
- Step 2 happens quickly enough that we don’t notice.
The end result is that this is less of a problem than I thought it was, since the time displayed for the DB setup step is really “time from start of DB setup step to end of CI run”.