We use Circle CI to package and deploy our serverless API to AWS and have noticed an increasing number of builds with the following error. Has anyone experienced this error and how were you able to resolve it?
#!/bin/bash -eo pipefail
npm run package -- --stage $STAGE --branch $CIRCLE_BRANCH
> example-api@1.0.0 package /home/circleci/project
> serverless package "--stage" "tst" "--branch" "example-branch"
sh: 1: serverless: Permission denied
npm ERR! code ELIFECYCLE
npm ERR! errno 126
npm ERR! example-api@1.0.0 package: `serverless package "--stage" "tst" "--branch" "example-branch"`
npm ERR! Exit status 126
npm ERR!
npm ERR! Failed at the cb-olo-acl@1.0.0 package script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /home/circleci/.npm/_logs/2019-05-17T05_58_25_306Z-debug.log
Exited with code 126
We too have been experiencing this recently. We use Circle CI to deploy to AWS via the Serverless framework and we have been finding that a significant number (~15-25%) of deployment workflows fail with this specific error. Re-running the job with no changes usually fixes the issue, which makes this seem like a Circle CI reliability issue.
@brannondorsey: can you supply in this thread the logs that I asked Triqi for? I’m just a fellow CircleCI user, but whoever looks at it will need a reproducible case.
Hey there @halfer, thanks for your response. I’ve pasted the npm log from a failed deploy today below.
0 info it worked if it ends with ok
1 verbose cli [ '/usr/local/bin/node',
1 verbose cli '/usr/local/bin/npm',
1 verbose cli 'run',
1 verbose cli 'deploy-stage' ]
2 info using npm@5.6.0
3 info using node@v8.10.0
4 verbose run-script [ 'predeploy-stage', 'deploy-stage', 'postdeploy-stage' ]
5 info lifecycle REDACTED@1.0.0~predeploy-stage: REDACTED@1.0.0
6 info lifecycle REDACTED@1.0.0~deploy-stage: REDACTED@1.0.0
7 verbose lifecycle REDACTED@1.0.0~deploy-stage: unsafe-perm in lifecycle true
8 verbose lifecycle REDACTED@1.0.0~deploy-stage: PATH: /usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/tmp/workspace/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
9 verbose lifecycle REDACTED@1.0.0~deploy-stage: CWD: /tmp/workspace
10 silly lifecycle REDACTED@1.0.0~deploy-stage: Args: [ '-c', 'serverless deploy --stage stage --env stage -v' ]
11 silly lifecycle REDACTED@1.0.0~deploy-stage: Returned: code: 126 signal: null
12 info lifecycle REDACTED@1.0.0~deploy-stage: Failed to exec deploy-stage script
13 verbose stack Error: REDACTED@1.0.0 deploy-stage: `serverless deploy --stage stage --env stage -v`
13 verbose stack Exit status 126
13 verbose stack at EventEmitter.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:285:16)
13 verbose stack at emitTwo (events.js:126:13)
13 verbose stack at EventEmitter.emit (events.js:214:7)
13 verbose stack at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack at emitTwo (events.js:126:13)
13 verbose stack at ChildProcess.emit (events.js:214:7)
13 verbose stack at maybeClose (internal/child_process.js:925:16)
13 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)
14 verbose pkgid REDACTED@1.0.0
15 verbose cwd /tmp/workspace
16 verbose Linux 4.15.0-1035-aws
17 verbose argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "deploy-stage"
18 verbose node v8.10.0
19 verbose npm v5.6.0
20 error code ELIFECYCLE
21 error errno 126
22 error REDACTED@1.0.0 deploy-stage: `serverless deploy --stage stage --env stage -v`
22 error Exit status 126
23 error Failed at the REDACTED@1.0.0 deploy-stage script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 126, true ]
The only thing that I can think of that is notable in our workflow is that we npm install dependencies (including serverless) in one job and then save these dependencies to a workspace that gets mounted to another job container later in the workflow during the deployment. Perhaps there is some sort of race condition where the serverless binary sometimes doesn’t have the correct executable permissions once mounted in the deploy job?
From some research, it looks like you might have mixed sudo and npm - there seems to be some suggestions in this thread not to do that.
Either way, would you provide your config.yml?
Also, I wonder if you could judiciously apply some ls -l to see the user/group/perms of the binary this error might apply to? It looks to me (bearing in mind I don’t do Node) that there is a binary called “serverless” and it is being run by (or in) the “sh” shell. Would this be in a node_modules folder somewhere?
yea – we are also starting to see this issue using circleci/node:8.10.
same behavior as what @brannondorsey mentioned – it happens intermittently and re-running the workflow gets it to work, so its random and ephemeral issue.