Testing docker built apps is harder in 2.0?


#1

I currently build, test, and deploy a dockerized webapp through circle 1.0 and it works great (although it’s slow due to lack of layer caching). From what I understand, to continue this workflow under 2.0 I have two choices:

  • use the machine executor to run basically the same flow i have in 1.0
  • have a service like quay.io automatically build my container in response to a github push, and then use the docker executor to test and deploy it.

questions:

  • are there any other options i’m missing here?
  • if i use the machine executor is there any way to benefit from layer caching?

i love the idea of the docker executor in 2.0 but it seems crippled since it’s impossible to build dockerized apps in it. so it seems 2.0 is great for everything except dockerized apps - definitely room for improvement. but i’m also hoping i’m wrong on the above.


#2

Same problem here, we have an internal registry running and can’t execute the frontend build without the dockerized backend running during the build.

We achieved that in 1.0 by using docker-compose.


#3

You can build a Docker container from the docker executor. You just can not connect directly from where your commands are running to the Docker container; i.e., you can not curl the container. You can, however, use docker-compose up (or however you have your workflow setup) to spin up an entire fleet of containers to do everything you need.

Not yet.


#4

i see - but once you do setup_docker_engine you lose out on all of the docker caching correct? if so circle 2.0 has zero benefit over 1.0 for this workflow.


#5

No; using setup_docker_engine is the only way to enable layer caching, but it’s currently only on docker.


#6

but does layer caching help with anything inside the remote docker or will images i build in there have to be rebuilt from scratch every time?


#7

Yes, layer caching would preserve your images from setup_docker_engine.


#8

ok thank you for the help and quick response. i will keep testing!


#9

@rohara how about volume mounts? Is there a workaround for that? Compose helps when you need multiple docker containers for your tests, we need to mount configuration files and keys into our application container to run our integration tests. Is there a way to do that with the docker executor or does that have to be the machine executor?


#10

@mitom you can do it with docker executor + setup_remote_docker step. You’ll need to create a dummy container that will hold a volume for config:
docker create -v /cfg --name configs alpine:3.4 /bin/true
Copy your configs there:
docker cp app_config.yml configs:/cfg
After that you can start container with your application mounting volume from this dummy container:
docker run --volumes-from configs app-image:1.2.3

There’s some more information about this in docs: https://circleci.com/docs/2.0/building-docker-images/#separation-of-environments


#11

Thank you, I’ll give that a try!


#12

I’m not seeing any cached layers being used inside remote docker. Is there something I have to do to enable this?


#13

Also not seeing the cache being used when pulling my “job space” containers either.


#14

Yes. Contact your CSM for information and whitelisting.

Images are automatically cached. The cache isn’t used, however, when you end up on a host without your cache present.


#15

Thanks - got it enabled by my CSM.

so are you saying that it’s the luck of the draw whether i’m going to have a fast build (with layer cache) or a slow build (no layer cache)?


#16

I’m saying if your build ends up on a new host, your base images are not cached.


#17

Wouldn’t supporting --cache-from help with this? Then you could pull the existing docker layers into the build?


#18

An example of how to copy configuration files and artifacts between containers on jobs using docker executor and setup_remote_docker using docker cp is now in the documentation: https://circleci.com/docs/2.0/building-docker-images/#mounting-folders


#19

Thanks for the update @tom. I would like to ask if you have a doc on how to achieve this with docker-compose?


#20

Essentially I want to be able to create a volume with docker compose without switching to Machine Executor…