Hi!
We use HeadlessChrome to run a variety of javascript tests with great success for a long time. On around February 27th, we started to see failures like this occasionally across various branches in this project:
Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted
The issue appears to be with how the kernel handles namespacing. In particular, Chrome tries to sandbox itself when setting up and on some of the machines that our jobs are getting scheduled onto it can, and some machines it cannot.
In searching around we found a few different workarounds:
- Enable namespaces
- Run it with a custom seccomp
- Run with
--no-sandbox
We went with --no-sandbox for now, but we’re wondering how we could be seeing this intermittently? Is this a bug with certain machines having different kernel settings? Is Circle A/B testing different VMS? We’ve noticed the frequency increase pretty significantly in the past few days and is now starting to impact all of our ChromeHeadless tests.
Example Output:
#!/bin/bash -eo pipefail
yarn ci:react
yarn run v1.6.0
$ karma start config/karma.config.react.ci.js --browsers ChromeHeadless --single-run
Happy[babel]: Version: 5.0.0. Threads: 4 (shared pool)
Happy[babel]: All set; signaling webpack to proceed.
Happy[coffee]: Version: 5.0.0. Threads: 4 (shared pool)
Happy[coffee]: All set; signaling webpack to proceed.
Happy[eco]: Version: 5.0.0. Threads: 4 (shared pool)
Happy[eco]: All set; signaling webpack to proceed.
Happy[sass]: Version: 5.0.0. Threads: 4 (shared pool)
Happy[sass]: All set; signaling webpack to proceed.
Happy[css]: Version: 5.0.0. Threads: 4 (shared pool)
Happy[css]: All set; signaling webpack to proceed.
Happy[new-css]: Version: 5.0.0. Threads: 4 (shared pool)
Happy[new-css]: All set; signaling webpack to proceed.
Happy[cssModules]: Version: 5.0.0. Threads: 4 (shared pool)
Happy[cssModules]: All set; signaling webpack to proceed.
01 03 2019 00:28:04.797:INFO [karma]: Front-end scripts not present. Compiling...
01 03 2019 00:28:05.837:INFO [karma]: Karma v2.0.0 server started at http://0.0.0.0:9876/
01 03 2019 00:28:05.837:INFO [launcher]: Launching browser ChromeHeadless with unlimited concurrency
01 03 2019 00:28:05.840:INFO [launcher]: Starting browser ChromeHeadless
01 03 2019 00:28:06.162:ERROR [launcher]: Cannot start ChromeHeadless
/home/app/.local/share/applications/mimeapps.list Stats {
dev: 106,
mode: 33188,
nlink: 1,
uid: 1000,
gid: 1000,
rdev: 0,
blksize: 4096,
ino: 73953,
size: 0,
blocks: 0,
atimeMs: 1551400085000,
mtimeMs: 1551400085000,
ctimeMs: 1551400085979.8123,
birthtimeMs: 1551400085979.8123,
atime: 2019-03-01T00:28:05.000Z,
mtime: 2019-03-01T00:28:05.000Z,
ctime: 2019-03-01T00:28:05.980Z,
birthtime: 2019-03-01T00:28:05.980Z }
Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted
Failed to generate minidump.
01 03 2019 00:28:06.162:ERROR [launcher]: ChromeHeadless stdout:
01 03 2019 00:28:06.162:ERROR [launcher]: ChromeHeadless stderr: /home/app/.local/share/applications/mimeapps.list Stats {
dev: 106,
mode: 33188,
nlink: 1,
uid: 1000,
gid: 1000,
rdev: 0,
blksize: 4096,
ino: 73953,
size: 0,
blocks: 0,
atimeMs: 1551400085000,
mtimeMs: 1551400085000,
ctimeMs: 1551400085979.8123,
birthtimeMs: 1551400085979.8123,
atime: 2019-03-01T00:28:05.000Z,
mtime: 2019-03-01T00:28:05.000Z,
ctime: 2019-03-01T00:28:05.980Z,
birthtime: 2019-03-01T00:28:05.980Z }
Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted
Failed to generate minidump.
01 03 2019 00:28:06.166:INFO [launcher]: Trying to start ChromeHeadless again (1/2).
01 03 2019 00:28:06.307:ERROR [launcher]: Cannot start ChromeHeadless
Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted
Failed to generate minidump.
01 03 2019 00:28:06.308:ERROR [launcher]: ChromeHeadless stdout:
01 03 2019 00:28:06.308:ERROR [launcher]: ChromeHeadless stderr: Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted
Failed to generate minidump.
01 03 2019 00:28:06.308:INFO [launcher]: Trying to start ChromeHeadless again (2/2).
01 03 2019 00:28:06.459:ERROR [launcher]: Cannot start ChromeHeadless
Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted
Failed to generate minidump.
01 03 2019 00:28:06.459:ERROR [launcher]: ChromeHeadless stdout:
01 03 2019 00:28:06.459:ERROR [launcher]: ChromeHeadless stderr: Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted
Failed to generate minidump.
01 03 2019 00:28:06.460:ERROR [launcher]: ChromeHeadless failed 2 times (cannot start). Giving up.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Exited with code 1
Further Reading:
- karma-runner/karma-chrome-launcher#125
- jessfraz/dockerfiles#65
- https://forums.docker.com/t/docker-user-namespaces/16885/4
- https://stackoverflow.com/questions/43665276/how-to-run-google-chrome-headless-in-docker
Example Builds: (internal only, requires login)
https://circleci.com/gh/mavenlink/mavenlink/439510
https://circleci.com/gh/mavenlink/mavenlink/443993