Not copied hidden files on local build by circleci cli

It has not been copied .* files of the project by cp command in checkout phase of local build.
I need copying hidden files like .babelrc to build successfully.

Suspicious log:
====>> Checkout code
#!/bin/sh
mkdir -p /home/circleci/workspace && cp -r /tmp/_circleci_local_build_repo/* /home/circleci/workspace

Also I hope to not copy local caches generated without using docker.
In my project, yarn is used and install binary files each os.
Therefore, the build fails to copy local caches like node_modules.

1 Like

If you don’t specify the files (i.e. replace /tmp/_circleci_local_build_repo/* with /tmp/_circleci_local_build_repo this should copy all of your hidden files.

Thank you for your quick reply.

The processing is executed in checkout phase of local build.
I think the script is in docker image of circleci, so I can not replace them.
I am sorry if it is a wrong idea.

Im not sure what you mean, can you give me a link to the build please?

I’m sorry my bad English.
I will explain from the beginning again.

I use circleci cli of https://circleci.com/docs/2.0/local-jobs/, so I updated it recently by $ circleci update.
Then, my project build fails after the updates.

I think that the updates include to change the way of deploying local project files to a docker container.
In building using docker in local, the circleci cli or its docker image mounts current project directory to /tmp/_circleci_local_build_repo in docker container, and copy files to work_directory and build and test by steps not to influence local files outside the container. I expected it to be happening from the next log.

The part of logs by $ circleci build :

====>> Checkout code
#!/bin/sh
mkdir -p /home/circleci/workspace && cp -r /tmp/_circleci_local_build_repo/* /home/circleci/workspace

The part of .circleci/config.yml :

    steps:
      - checkout

I tried to search where it executed, but I could not find in /usr/local/bin/circleci as script. I think the script is in circleci/picard of docker image. Maybe I can not modify the script, so please tell me the resoltion to copy hidden files.

Hi @inoutch

I just pushed the fix. Can you run circleci update and try again? As you are probably aware, we changed the way to handle local checkout. It used to be directly mounting the current directory to working_directory but now we first mount to /tmp/_circleci_local_build_repo and copy file contents to working_directory. The problem was * wasn’t expanded to hidden files. The change was made in order to handle when ~ is specified in working_directory. (~ didn’t work because Docker refuses ~ in bind mount dest).

If you still see the problem, please let us know.

ご迷惑をおかけいたしました!!! :bow:

2 Likes

Its happening again.

#!/bin/sh
mkdir -p /home/circleci/rippling-webapp && cp -r /tmp/_circleci_local_build_repo/. /home/circleci/rippling-webapp
cp: cannot open ‘/tmp/_circleci_local_build_repo/./node_modules/arr-diff/LICENSE’ for reading: Permission denied
cp: cannot open ‘/tmp/_circleci_local_build_repo/./node_modules/arr-flatten/README.md’ for reading: Permission denied

CircleCI version info:
circleci version: 0.0.4579-9c6473a
Build Agent version: 0.0.4580-9c6473a
built: 2017-11-30T03:28:56+0000

1 Like

Yup. Same here.

====>> Checkout code
  #!/bin/sh
mkdir -p /home/circleci/workspace && cp -r /tmp/_circleci_local_build_repo/. /home/circleci/workspace
cp: cannot open '/tmp/_circleci_local_build_repo/./.bundle/gems/axiom-types-0.1.1/.gitignore' for reading: Permission denied
...

circleci version: 0.0.4623-f5a3344
Build Agent version: 0.0.4622-f5a3344
built: 2017-12-28T15:02:57+0000

Thx

1 Like

Regarding the rash of "Permission denied"s cited by others, it seems we need to ensure all files up and down the directory structure from where we type circleci build are world readable.

1 Like

This issue happens again

====>> Checkout code
  #!/bin/sh
mkdir -p /root/ferry-app && cp -r /tmp/_circleci_local_build_repo/. /home/circleci/workspace
cp: cannot stat '/tmp/_circleci_local_build_repo/.': No such file or directory
Error: Exited with code 1
Step failed
Task failed
/bin/sh: pkill: command not found
/bin/sh: pgrep: command not found

circleci version: 0.0.5609-e7533b7
OS: mac OS High Sierra

1 Like

It is happening now on my PC. Ubuntu 18.04

====>> Checkout code
  #!/bin/sh
mkdir -p /root/project && cp -r /tmp/_circleci_local_build_repo/. /root/project
cp: cannot stat `/tmp/_circleci_local_build_repo/.': No such file or directory
Error: Exited with code 1
Step failed
Task failed

I just upgraded and receiving the following:

====>> Spin up Environment
Build-agent version 0.0.5706-58f0e71 (2018-05-14T22:35:09+0000)
Starting container circleci/python:3.6-node-browsers
  using image circleci/python@sha256:9fb4c0703cf9d3ed0ed476217826f85bbd94bd2bb72097c91744b389f51ded1d

Using build environment variables:
  BASH_ENV=/tmp/.bash_env-localbuild-1526363651
  CI=true
  CIRCLECI=true
  CIRCLE_BRANCH=master
  CIRCLE_BUILD_NUM=
  CIRCLE_JOB=build
  CIRCLE_NODE_INDEX=0
  CIRCLE_NODE_TOTAL=1
  CIRCLE_REPOSITORY_URL=***
  CIRCLE_SHA1=****
  CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1526363651
  CIRCLE_WORKING_DIRECTORY=~/project

====>> Checkout code
  #!/bin/sh
mkdir -p /home/circleci/project && cp -r /tmp/_circleci_local_build_repo/. /home/circleci/project
cp: cannot stat '/tmp/_circleci_local_build_repo/.': No such file or directory
Error: Exited with code 1
Step failed
Task failed

Work Around
circleci --tag 0.0.5427-f011aef build

Same here…
====>> Checkout code
#!/bin/sh
mkdir -p /home/circleci/app && cp -r /tmp/_circleci_local_build_repo/. /home/circleci/app
cp: cannot stat ‘/tmp/_circleci_local_build_repo/./.git/info/exclude’: No such file or directory
cp: cannot stat ‘/tmp/_circleci_local_build_repo/./.git/description’: No such file or directory
cp: cannot stat ‘/tmp/_circleci_local_build_repo/./.git/hooks’: No such file or directory

Any permanent solution?

There was an unrelated regression introduced ~11 days ago, but fixed the following day. Anyone seeing issues should circleci update again. If you are seeing issues on 0.0.5763-f6bc418 (currently latest), please include version details and error output.

Still seeing this issue

circleci version: 0.0.5894-01cdb92
Build Agent version: 0.0.5895-01cdb92
built: 2018-05-29T20:07:10+0000

Can you please include the error output that you get?

Same issue here.
====>> Spin up Environment
Build-agent version 0.0.5895-01cdb92 (2018-05-29T20:07:10+0000)
Starting container circleci/buildpack-deps:latest-dind
using image circleci/buildpack-deps@sha256:f9dea38e6583c663bbe2d950b8c9abdb771f9dd587fdb18460da2f4253c3a70a

Using build environment variables:
BASH_ENV=/tmp/.bash_env-localbuild-1531283476
CI=true
CIRCLECI=true
CIRCLE_BRANCH=feature/circle_ci
CIRCLE_BUILD_NUM=
CIRCLE_JOB=build
CIRCLE_NODE_INDEX=0
CIRCLE_NODE_TOTAL=1
CIRCLE_REPOSITORY_URL=git@github.com:whatever
CIRCLE_SHA1=12345
CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1531283476
CIRCLE_WORKING_DIRECTORY=~/code

====>> Checkout code
#!/bin/sh
mkdir -p /home/circleci/code && cp -r /tmp/_circleci_local_build_repo/. /home/circleci/code
cp: cannot stat ‘/tmp/_circleci_local_build_repo/.’: No such file or directory
Error: Exited with code 1
Step failed
====>> Uploading artifacts
Uploading /home/circleci/code/app/build/reports to reports
No artifact files found at /home/circleci/code/app/build/reports
Task failed

Hitting the same issue as well.

Here’s the version info for my host:

circleci version: 0.0.5894-01cdb92
Build Agent version: 0.0.5895-01cdb92
built: 2018-05-29T20:07:10+0000

Seems like the ‘Checkout’ command is barfing. I spliced in a sanity check command to just list whether a file existed in my repo, and it fails outright. Digging a bit further, a command ran ‘ls’ and the output seems to indicate perhaps the working_directory field is being ignored?

Sure enough, running the build remotely displays the correct folder contents.

Is there a bug already filed for this?

To answer my own question - its been fixed. There’s a new build with the following version details:

circleci version: 0.1.0
Build Agent version: 0.0.7725-9681bc36
built: 2018-08-09T15:10:50+0000

Where this is no longer an issue. Vetted on OSX and Linux.

Still happening on local Checkout on version:

circleci version: 0.1.1640
Build Agent version: 0.0.7725-9681bc36
built: 2018-08-09T15:10:50+0000

Issue occurs when running this locally on a Mac 10.13.6. Problem doesn’t occur when running remotely.