Docker-compose doesn't mount volumes with host files with Circle-Ci


#1

Problem

Hi Circle-CI teams !

I’m using a CI (Circle CI V2) and docker compose run ... command doesn’t mount volumes from host. How could i fix this issue ? Thanks a lot for your help ! :blush: :rotating_light:

NOTE 1 : Everything is working fine on my computer (MacOSX) and on my EC2 server for production (Ubuntu 16.04).
NOTE 2 : i checked #4585 but didn’t found a way to fix my problem

Details of the problem

command

docker-compose run --rm api-server

docker-compose.yaml

version: '2'
services:

  maildev:
    image: djfarrelly/maildev
    ports:
      - "${CLOVIS_MAILDEV_PORT}:80"


  api-server:
    environment:
      CLOVIS_CLIENT_URL: "http://localhost:${CLOVIS_CLIENT_PORT}"
      AWS_ACCESS_KEY_ID: "${CLOVIS_AWS_ID}"
      AWS_SECRET_ACCESS_KEY: "${CLOVIS_AWS_SECRET}"
      CLOVIS_RUN_SLOW_TESTS: "${CLOVIS_RUN_SLOW_TESTS}"
      CLOVIS_SMTP_HOST: "maildev"
      CLOVIS_SMTP_PORT: "25"
      GOOGLEMAP_KEY: "${CLOVIS_GOOGLEMAP_KEY}"
    build:
      context: .
      dockerfile: api-server/Dockerfile.dev

    volumes:
      - ./api-server/src:/usr/src/api-server/src:ro
      - ./api-server/resources:/usr/src/api-server/resources:ro
      - ./email/src:/usr/src/email/src:ro
      - ./sms/src:/usr/src/sms/src:ro
      - ./api-server/scripts:/usr/src/api-server/scripts:ro

    command: yarn run watch

    ports:
      - "${CLOVIS_API_PORT}:3000"

    links:
      - mongo:mongo
      - maildev:maildev

    tty: true


  mongo:
    image: mongo
    logging:
      driver: "none"

:white_check_mark: Local (MacOSX) - Everything works !

Docker version

$ docker --version
Docker version 17.09.1-ce, build 19e2cf6
$ docker-compose --version
docker-compose version 1.17.1, build 6d101fb

Listing the files

$ ls -ln
-rw-r--r--   1 501  20  1897 Dec 26 17:25 Dockerfile.dev
drwxr-xr-x  14 501  20   476 Dec 28 15:42 api-server
-rwxr-xr-x   1 501  20  2650 Dec 29 18:03 dev.sh
-rw-r--r--   1 501  20  3098 Dec 29 17:43 docker-compose.dev.yml
drwxr-xr-x  11 501  20   374 Dec 26 17:25 email
drwxr-xr-x  20 501  20   680 Dec 29 18:10 scripts
drwxr-xr-x   9 501  20   306 Dec 26 17:25 sms


$ docker-compose exec api-server pwd
/usr/src/api-server

$ docker-compose exec p2 ls -ln src
-rw-r--r--  1 0 0  1363 Dec 26 16:25 app.js
-rw-r--r--  1 0 0   797 Dec 26 16:25 auth.js
-rw-r--r--  1 0 0  1122 Dec 26 16:25 avatar.js
-rw-r--r--  1 0 0    99 Feb 26  2017 aws.js
-rw-r--r--  1 0 0  1678 Dec 26 16:25 config.js
drwxr-xr-x 21 0 0   714 Dec 26 16:25 controllers
drwxr-xr-x  8 0 0   272 Dec 28 13:05 cron
-rw-r--r--  1 0 0   989 Dec 26 16:25 db.js
etc... 

# ok all my files and folders are copied from host ./api-server/src 
# to mounted container /usr/src/api-server/src ! perfect !

$ docker-compose exec p2 ls -ln ..
drwxr-xr-x 1 0 0 4096 Dec 28 15:48 api-server
drwxr-xr-x 1 0 0 4096 Dec 28 15:48 email
drwxr-xr-x 1 0 0 4096 Dec 28 15:48 sms


:x: Remote (Circle-CI, Ubuntu 14.04) - Volume mount doesn’t work

Docker version

$ docker --version
Docker version 17.12.0-ce, build c97c6d6

$ docker-compose --version
docker-compose version 1.11.2, build dfed245

Listing the files

$ ls -ln
-rw-r--r-- 1 3434 3434 1897 Dec 29 20:16 Dockerfile.dev
drwxr-xr-x 5 3434 3434 4096 Dec 29 20:16 api-server
-rwxr-xr-x 1 3434 3434 2650 Dec 29 20:16 dev.sh
-rw-r--r-- 1 3434 3434 3098 Dec 29 20:16 docker-compose.dev.yml
drwxr-xr-x 3 3434 3434 4096 Dec 29 20:16 email
drwxr-xr-x 3 3434 3434 4096 Dec 29 20:16 scripts
drwxr-xr-x 3 3434 3434 4096 Dec 29 20:16 sms

$ docker-compose exec -T api-server pwd
/usr/src/api-server

$ docker-compose exec -T api-server ls -ln src

# ?!!! there's no file here, the volume isn't mounted with the host files :/
# (cf docker-compose.yaml file '-./api-server/src:/usr/src/api-server/src:ro' )

$ docker-compose exec -T api-server ls -ln ..
drwxr-xr-x 6 0 0 4096 Dec 29 20:23 api-server
drwxr-xr-x 4 0 0 4096 Dec 29 20:23 email
drwxr-xr-x 4 0 0 4096 Dec 29 20:23 sms


Do you have any idea what happened on the remote Circle Ci and how to fix that ?


Mount project folder using docker-compose and circle ci
Running a script using docker compose
.circleci/config.yml with more than one Docker image
#2

Ok that looks like Circle CI doesn’t allow mounted volumes with host files : https://circleci.com/docs/2.0/building-docker-images/#mounting-folders

Why ? I spent 2 days to find the origin of this problem …


#3

I bumped into this, it’s come up several times around the web. Here are some links…

The essence of the problem is that in the “docker” executor, the volumes are local and the containers are remote, and Docker volumes don’t work non-locally. However if you use the “machine” executor, it will work, since everything is local (but it will be slower to start up and may have an additional cost in the future).


#4

Thanks halfer !

The inception to make circle-ci work with mounted volumes becomes crazy…
I don’t understand why Circle-CI teams didn’t made a full tutorial to explain how to set the .config.yml of circle-ci to manage mounted volumes, and not changing our docker-compose.yaml file and diving into advanced docker cancer…

Where could i find operational help to write my circle-ci config.ymlfile with custom options for mouting volumes ? We want to leave Travis Ci because of overpricing, but we are losing a lot of time to pass to Circle CI…


#5

Same reason as any product development team - there is always more work that can be done on everything. If you’ve ever worked damn hard in sprints and then still wound up with one or two unhappy customers, you’ll know what it’s like on the other side. You have to work on the best priorities as your product manager sees them, so you can satisfy most people, most of the time.

I think there is a Feature Request category in this forum, so post a suggestion there if you wish.

In the meantime, why not switch to the machine executor? That will get you up and running again with a one-line change. I would expect CircleCI would give good advance notice of future extra fees for that executor type, so you could change again if you found it too expensive.

The other approach would be to run Docker Compose inside Docker, so that all containers are local. DID (Docker In Docker) configurations do work in general, though it’d be interesting to find out how Circle would interact with it (e.g. saving/loading the Docker cache, storing artefacts, etc).

Yep. CI is complex, and moving CI is even more so :smiley_cat:


#6

Ok everythings works fine for me. I created a specific docker-compose.test.yaml file for tests, with no volumes and specific sub dockerfiles with many COPY commands to put the files on containers.

Here is the full post on github : https://github.com/docker/compose/issues/5519#issuecomment-354502164

Yep. CI is complex, and moving CI is even more so :smiley_cat:

Docker and CI should be very easy to install and deploy (tech UX). It’s the future of those kind of tech business. I know that’s hard, but Circle Ci was funded 16M$ in 2016, so i think they can invest more money in those kind of tasks, and hire more Tech UX to make things absolutely easy. (and they can also just display a big warning when their users use volumes in their docker-compose files…)

Otherwise this forum is cool, thanks for your support :wink:


#7

Great. Don’t forget that you can use several docker-compose files, with subsequent files overriding values in the first one. This makes it really easy to set up per-environment changes without having to copy+paste the whole thing.


#8