Docker build fails with nonsensical EPERM: operation not permitted, copyfile

Build Environment
1

I have a circleci build that’s failing when it tries to build a Docker image. Specifically it fails when it tries to execute an npm build script. The script fails immediately because the root user apparently doesn’t have permission to copy files within the working directory that it owns.

This is the output from the script:

#!/bin/bash -eo pipefail
.circleci/build.sh
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
Attempting to pull Docker image 472490737436.dkr.ecr.us-east-1.amazonaws.com/viewer-frontend:9d4881bac9edb04ba7395e67f5ce1bebe174b732 to check if it already exists
Building Docker image 472490737436.dkr.ecr.us-east-1.amazonaws.com/viewer-frontend:9d4881bac9edb04ba7395e67f5ce1bebe174b732 in path . with args --build-arg GITHUB_OAUTH_TOKEN=****************************************
Sending build context to Docker daemon  2.209MB
Step 1/23 : FROM ubuntu:18.04 as intermediate
 ---> 6526a1858e5d
Step 2/23 : ARG GITHUB_OAUTH_TOKEN
 ---> Using cache
 ---> 729e0176df2a
Step 3/23 : RUN if [ -z "$GITHUB_OAUTH_TOKEN" ]; then echo "ERROR: You must set GITHUB_OAUTH_TOKEN as a Docker build arg."; exit 1; fi
 ---> Using cache
 ---> 9c1e76fd9f31
Step 4/23 : ENV GRUNTWORK_INSTALLER_VERSION v0.0.22
 ---> Using cache
 ---> 5be8a12bf0e7
Step 5/23 : ENV GRUNTKMS_VERSION v0.0.8
 ---> Using cache
 ---> 9c63f9293dab
Step 6/23 : RUN apt-get update &&     apt-get install -y curl sudo
 ---> Using cache
 ---> 82905c5c5225
Step 7/23 : RUN curl -Ls https://raw.githubusercontent.com/gruntwork-io/gruntwork-installer/master/bootstrap-gruntwork-installer.sh | bash /dev/stdin --version "$GRUNTWORK_INSTALLER_VERSION" &&     gruntwork-install --repo "https://github.com/gruntwork-io/gruntkms" --binary-name "gruntkms" --tag "$GRUNTKMS_VERSION"
 ---> Using cache
 ---> 3d7c7a813451
Step 8/23 : FROM node:14
 ---> 40ce906a3734
Step 9/23 : COPY --from=intermediate /usr/local/bin/gruntkms /usr/local/bin/gruntkms
 ---> Using cache
 ---> 722669b4aef6
Step 10/23 : RUN npm install --verbose -g nodemon
 ---> Using cache
 ---> 26b365fb9c36
Step 11/23 : WORKDIR /app
 ---> Using cache
 ---> 70e28906a6fc
Step 12/23 : COPY package*.json ./
 ---> Using cache
 ---> 0c1a57856ce2
Step 13/23 : RUN npm install --verbose
 ---> Using cache
 ---> ba1140507066
Step 14/23 : COPY . .
 ---> 9e37f7329a72
Step 15/23 : RUN chown -R $USER:$USER /app
 ---> Running in 135fbbe7be09
 ---> f731eb0b9069
Removing intermediate container 135fbbe7be09
Step 16/23 : RUN chmod -R 777 /app
 ---> Running in 67a201716d10
 ---> 626afd0d553f
Removing intermediate container 67a201716d10
Step 17/23 : RUN ls -lah /app/
 ---> Running in a41a7bc9d85d
total 1.1M
drwxrwxrwx   23 root root 4.0K Sep 10 01:01 .
drwxr-xr-x   70 root root 4.0K Sep 10 01:01 ..
drwxrwxrwx    2 root root 4.0K Sep 10 01:01 .circleci
-rwxrwxrwx    1 root root   31 Sep 10 01:00 .dockerignore
-rwxrwxrwx    1 root root  145 Sep 10 01:00 .env
-rwxrwxrwx    1 root root   49 Sep 10 01:00 .eslintignore
-rwxrwxrwx    1 root root  280 Sep 10 01:00 .gitignore
-rwxrwxrwx    1 root root 1.8K Sep 10 01:00 Dockerfile
-rwxrwxrwx    1 root root 5.5K Sep 10 01:00 README.md
drwxrwxrwx    2 root root 4.0K Sep 10 01:01 bin
drwxrwxrwx    2 root root 4.0K Sep 10 01:01 config
-rwxrwxrwx    1 root root  419 Sep 10 01:00 docker-compose.yml
drwxrwxrwx 3503 root root  36K Sep 10 01:01 node_modules
-rwxrwxrwx    1 root root 925K Sep 10 01:00 package-lock.json
-rwxrwxrwx    1 root root 3.5K Sep 10 01:00 package.json
drwxrwxrwx    5 root root 4.0K Sep 10 01:01 public
-rwxrwxrwx    1 root root 3.2K Sep 10 01:00 server.js
drwxrwxrwx   23 root root 4.0K Sep 10 01:01 src
drwxrwxrwx    2 root root 4.0K Sep 10 01:01 tls
-rwxrwxrwx    1 root root  636 Sep 10 01:00 tsconfig.json
 ---> 2d7aa40e3015
Removing intermediate container a41a7bc9d85d
Step 18/23 : RUN pwd
 ---> Running in bbc89db60d6c
/app
 ---> babae612e9c4
Removing intermediate container bbc89db60d6c
Step 19/23 : RUN whoami
 ---> Running in 2931d31574c0
root
 ---> 26e10bac5681
Removing intermediate container 2931d31574c0
Step 20/23 : RUN npm run build
 ---> Running in 3351dd1a1e5a

> viewer-frontend@0.2.0 build /app
> react-scripts build

Failed to compile.

EPERM: operation not permitted, copyfile '/app/public/favicon.ico' -> '/app/build/favicon.ico'
2

Hi @jtiscione,

Welcome! - I came across a similar permissions issue earlier and I was able to resolve it by doing the following:

  • Add RUN apt-get update && apt-get install -y --no-install-recommends apt-utils to your .Dockerfile

  • unchecking read-only for your node_modules

In other cases, the issue has been resolved by specifying a tag to pin the node base image to an earlier version (before upstream changes were made).

Let me know how it goes!

1 Like
3

It unfortunately didn’t have an effect. I should have mentioned that it does run locally with no problems using docker-compose build --pull even with the same image.

4

Hi @Henna, I’m also debugging this with @jtiscione. The part we forgot to mention was that this builds fine locally with a build --pull. That’s why there’s some confusion, it seems to be something in CircleCI.

5

Just noticed we’ve got docker_layer_caching set to true, going to test removing that.

6

nope, still seems to be an issue.

7

@jtiscione @penglish Thanks for trying that. Since I do not have access to your local or remote docker environment, I am not able to investigate those. Can you please share a build link and package.json (either here or by opening a support ticket).

Alternatively, You can ssh into the remote docker environment to investigate. You might want to keep an eye out for the base node image being used.

I’ve seen some customers resolve the issue by using a machine executor.

8

Hi-
Here is the build link (latest failed): https://app.circleci.com/pipelines/github/pathologywatch/viewer-frontend/198/workflows/f2f31674-5abb-4168-9af6-ba79a0246208/jobs/509

This is the content of package.json:
{
“name”: “viewer-frontend”,
“version”: “0.2.0”,
“private”: true,
“dependencies”: {
“@datadog/browser-rum”: “^1.13.0”,
“@fortawesome/fontawesome”: “^1.1.8”,
“@fortawesome/fontawesome-free-solid”: “^5.0.13”,
“@fortawesome/fontawesome-svg-core”: “^1.2.29”,
“@fortawesome/free-solid-svg-icons”: “^5.13.1”,
“@fortawesome/react-fontawesome”: “^0.1.11”,
“@sentry/browser”: “^5.19.1”,
“@sentry/node”: “^5.19.2”,
“@sweetalert2/theme-bootstrap-4”: “^3.2.0”,
“@testing-library/user-event”: “^12.0.11”,
“@types/jest”: “^26.0.4”,
“@types/lodash.throttle”: “^4.1.6”,
“@types/openseadragon”: “^2.4.5”,
“@types/react”: “^16.9.41”,
“@types/react-dom”: “^16.9.8”,
“@types/react-helmet”: “^6.0.0”,
“@types/react-router-dom”: “^5.1.5”,
“@types/uuid”: “^8.0.1”,
“@zeit/next-sass”: “^1.0.1”,
“axios”: “^0.19.2”,
“babel-plugin-transform-remove-console”: “^6.9.4”,
“bootstrap”: “^4.5.0”,
“bootstrap-material-design”: “^4.1.3”,
“coreapi”: “^0.1.1”,
“dotenv”: “^8.2.0”,
“express”: “^4.17.1”,
“formik”: “^2.1.4”,
“http-proxy-middleware”: “^1.0.4”,
“isomorphic-unfetch”: “^3.0.0”,
“js-cookie”: “^2.2.1”,
“jwt-decode”: “^2.2.0”,
“lodash.throttle”: “^4.1.1”,
“moment”: “^2.27.0”,
“nodemon”: “^2.0.4”,
“openseadragon”: “^2.4.2”,
“react”: “^16.13.1”,
“react-data-table-component”: “^6.9.6”,
“react-dom”: “^16.13.1”,
“react-feather”: “^2.0.8”,
“react-helmet”: “^6.1.0”,
“react-hotkeys”: “^2.0.0”,
“react-idle-timer”: “^4.2.12”,
“react-redux”: “^7.2.0”,
“react-router”: “^5.2.0”,
“react-router-dom”: “^5.2.0”,
“react-scripts”: “^3.4.0”,
“reactstrap”: “^8.5.1”,
“redux”: “^4.0.5”,
“redux-api-middleware”: “^3.2.1”,
“redux-devtools-extension”: “^2.13.8”,
“redux-persist”: “^6.0.0”,
“redux-thunk”: “^2.3.0”,
“sass”: “^1.26.10”,
“styled-components”: “^5.1.1”,
“sweetalert2”: “^9.17.1”,
“typescript”: “^3.7.5”,
“uuid”: “^8.3.0”,
“validate.js”: “^0.13.1”,
“yup”: “^0.29.1”
},
“devDependencies”: {
“@testing-library/jest-dom”: “^5.11.0”,
“@testing-library/react”: “^10.4.5”,
“@types/js-cookie”: “^2.2.5”,
“@types/jwt-decode”: “^2.2.1”,
“@types/node”: “^14.0.20”,
“@types/react”: “16.9.42”,
“@types/react-dom”: “^16.9.8”,
“@types/react-redux”: “^7.1.7”,
“@types/reactstrap”: “^8.4.2”,
“@types/redux-api-middleware”: “^3.0.7”,
“@types/redux-mock-store”: “^1.0.2”,
“@types/styled-components”: “^5.1.2”,
“@types/yup”: “^0.29.3”,
“babel-jest”: “^24.9.0”,
“babel-plugin-styled-components”: “^1.10.0”,
“eslint”: “^6.6.0”,
“eslint-config-airbnb”: “^18.0.1”,
“eslint-plugin-import”: “^2.20.1”,
“eslint-plugin-jsx-a11y”: “^6.2.3”,
“eslint-plugin-react”: “^7.18.3”,
“eslint-plugin-react-hooks”: “^4.0.6”,
“jest”: “^24.9.0”,
“react-test-renderer”: “^16.12.0”,
“redux-mock-store”: “^1.5.4”,
“typescript”: “^3.7.5”
},
“scripts”: {
“start”: “nodemon server.js”,
“dev”: “react-scripts start”,
“build”: “react-scripts build”,
“test”: “react-scripts test”,
“eject”: “react-scripts eject”
},
“eslintConfig”: {
“extends”: “react-app”,
“rules”: {
“react-hooks/rules-of-hooks”: “off”,
“no-plusplus”: 0
}
},
“browserslist”: {
“production”: [
“>0.2%”,
“not dead”,
“not op_mini all”
],
“development”: [
“last 1 chrome version”,
“last 1 firefox version”,
“last 1 safari version”
]
}
}

9

@jtiscione can you try rerunning your last passing build from within the UI. In the meantime, I will continue to investigate. Thanks!

1 Like
10

Rerunning ones that succeeded always generates brief output like this:
Login Succeeded Attempting to pull Docker image 472490737436.dkr.ecr.us-east-1.amazonaws.com/viewer-frontend:46d4b18594981cd5ac7c3c0c2bdd9ff4c3dd950a to check if it already exists Docker image 472490737436.dkr.ecr.us-east-1.amazonaws.com/viewer-frontend:46d4b18594981cd5ac7c3c0c2bdd9ff4c3dd950a already exists in the Docker registry. Will not rebuild. Test command is empty. Not running any tests. Docker image 472490737436.dkr.ecr.us-east-1.amazonaws.com/viewer-frontend:46d4b18594981cd5ac7c3c0c2bdd9ff4c3dd950a already exists in the registry. Will not push. The --output-properties-file option is not set, so not writing tag to properties file

CircleCI received exit code 0

11

Hi-

Is there a way to force it to ignore any cached docker image? I can’t get it to perform a rebuild of the old ones, because it finds the cached docker image and exits immediately.

12

OK- changing the Docker base image from node:14 to node:14.8 works. Apparently this weird permission problem started with 14.9 and still exists in 14.10.

13

I’m noticing this with node 14.11 also.

14

Had this issue with 14.10, reverted to 14.8 as jtiscione suggested and issue was gone.

14.11 has 2 critical CVE fixes though.

1 Like
15

Can confirm this is an ongoing issue with the node:14.11.0-alpine images – reverting to the node:14.8.0-alpine image was the only solution I could find to get our yarn install back on track.

16

I opened an issue with CircleCI about this. They’re claiming it’s an upstream problem with yarn.

17

Signed up just to say I’m getting that too. Any idea what’s the fix? Can’t deploy now.

18

Haven’t yet found the root cause but upgrading the remote Docker engine in setup_remote_docker to anything above the default 17.x fixed it for us (Node.js 12.19.0).

1 Like
19

Cheers @mikkopiu, saved my ass there. Kept thinking it was my fault with Dockerfile, and I’m hours sleep deprived. Thanks CircleCI!

1 Like
20

We also had the same issue this morning since the https://deb.nodesource.com/setup_12.x script started installing 12.19.0

We resolved the issue by forcing installation of nodejs 12.18.4

3 Likes