Out of space in Remote Docker Environment?


#1

We’re using CircleCI 2.0 and have enabled Docker Layer Caching and have begun see build failures today because we cannot start a Postgres instance instance in our remote Docker environment (docker run -d --name postgres postgres:9.6.2). Digging in a bit further, we enabled SSH after a failed build, then entered our remote Docker environment.

ssh -p 32xxx 107.22.xxx.xxx docker run postgres:9.6.2
initdb: could not create directory “/var/lib/postgresql/data/pg_xlog”: No space left on device
initdb: removing contents of data directory “/var/lib/postgresql/data”
docker run --name postgres postgres:9.6.2 sleep 3600 docker exec -it postgres /bin/bash
root$ df
Filesystem…1K-blocks…Used…Available…Use%…Mounted on
none…103178528…100124696…0…100%…/
tmpfs…3829692…0…3829692…0%…/dev
tmpfs…3829692…0…3829692…0%…/sys/fs/cgroup
/dev/sda1…103178528…100124696…0…100%…/etc/hosts
shm…65536…0…65536…0%…/dev/shm
tmpfs…3829692…0…3829692…0%…/sys/firmware

It seems like we’ve used nearly 100GB of storage. How can we clear out this space so that our builds stop failing? And what commands can we run to prevent it from building up in the future?


#2

It turns out we needed to clear out our Docker images. After we SSH’d into a build we ran docker images, which listed nearly 400 images.

To delete most of the images, but not recent ones that could be used for caching, or older ones that are the root references, we added the following to our build tasks:

docker images -f "dangling=true" --format "{{.ID}} {{.CreatedSince}}" \
  | grep "weeks ago" | cut -f 1 -d " " \
  | xargs --no-run-if-empty docker rmi


#3

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.