Running circleci locally should allow to ignore some files


#1

I have some large files in my checkout which should not be copied over during local testing because they are not available on CI anyway, and because otherwise Checkout code step takes forever. But it seems there is no way to do something like that? Maybe .gitignore file could be used and files in .gitignore should not be copied over? (If we simulate git checkout this might be reasonable.)


#2

The correct approach here is not to keep your large files checked into the same repository. Either create a sub module that requires explicit initialization or manually load the files from elsewhere. Perhaps different sets of files, a much smaller one for testing, makes sense.


#3

No, they are not checked in the repository. They are just in the same local directory during development. This is why running it on CircleCI is fast, but running local circleci client is very slow.


#4

I apologize, I did not read carefully enough the first time.
To confirm, you’re using this:
https://circleci.com/docs/2.0/local-jobs/#running-a-build
And running circleci build?

I guess I’m unclear then - if we look here at the sample:
https://circleci.com/docs/2.0/sample-config/

We see that you can “run:” any bash command you want. So you could start by moving the files out of the way temporarily, and then the last build step is to put them back.

i.e.
jobs:
build:
steps:
run:
name: Move large files away
command: mv ./path/to/large_files/ /tmp
…other build commands…
test:
steps:
…all testing commands…
run:
name: Bring large files back
command: mv /tmp/large_files ./path/to/

Perhaps I’m missing some other constraints?


#5

So I am running local Circle CI on Mac OS X. The main issue is that copying my repository (10 GB+) into a Docker container take a long long time. So having a step which would move files away does not help, because it would be run inside the Docker container, after all files were copied there, which would take a lot of time. This is hw I understand at least how things work.

Ideally, I could configure this, maybe through .circleci/ignore or something similar. Or .dockerignore file could be respected?


#6

I recently migrated from 1.0 to 2.0 and am now trying out local builds. I face the same issue, code checkout takes a looong time due to all files being copied to the docker container.

I’d love it if circleci build didn’t copy over files in .gitignore to the container. It’s frustrating to have to clean various caches to boost up local CI testing setup.

Would it be possible to have circleci build do a git clone --depth=1 or similar strategy to boost up the checkout time considerably?


#7

This happens to me too using CircleCI’s CLI. There is a file that I don’t want it to be present when I run a job. I have tried creating a .dockerignore file that contains it’s path but it does not work.


#8

I’m having the same issue, this is so frustrating. Is there no way to prevent the local cli from copying everything? Anyone that has “.node-modules” in their repo will peg the CPU while copying things over. This makes me not want to use circleci as it’s such an obvious and frustrating first user experience issue, and it’s sat here stagnant for months. Why not look for .gitignore and .dockerignore files in the root and adhere to them?


#9