There are some patterns that cause random issues which have never happened in your local machine. The random issue means the job fails sometimes without any changes. In this article, I’ll share the most common cause of the random issue and how you can debug it.
OOM (Out of memory) error
OOM error is the most common cause of the random issue in Node.js project. This happens when the process has been SIGKILL ed by the OOM killer. It means when your process uses almost all the memory that the executor has, the OOM killer kills your process.
If you use Node.js on your project and run a job on Docker executor, you might see this error more often. This is because the host VM for Docker executor has more CPUs and memory resources than the docker executor has been allocated, and os.cpus() which is the native Node.js function counts the number of CPUs on the host VM. Many of Node.js packages use this function to makes workers/threads for speeding up its process. The current host VM of the docker executor has 32 cores, but the default Docker executor has only 2 vCPUs, so there is a huge gap.
What should I check at first?
You can check max memory usage and the existence of the child process. If there are many child processes, you should better configure your script to run serially in the current process.
The following article shares the way to see the max memory usage and see the existence of the child process.
This is an example output and you can see child processes of the react-scripts build command. If you see similar outputs and your job is failed, it might be failed with OOM error.
circleci 91 0.0 0.0 19740 3088 pts/1 Ss+ 06:59 0:00 \_ /bin/bash -eo pipefail -c while true; do sleep 5 ps auxwwf echo "======" done
circleci 350 0.0 0.0 38540 3180 pts/1 R+ 07:00 0:00 | \_ ps auxwwf
circleci 93 0.0 0.0 4280 736 pts/2 Ss+ 06:59 0:00 \_ sh ./build.sh
circleci 137 1.2 0.0 734681 43540 pts/2 Sl+ 07:00 0:00 \_ npm
circleci 148 0.0 0.0 4292 760 pts/2 S+ 07:00 0:00 \_ sh -c react-scripts build --max-old-space-size=3072
circleci 149 0.2 0.0 111111 31264 pts/2 Sl+ 07:00 0:00 \_ node /home/circleci/project/node_modules/.bin/react-scripts build --max-old-space-size=3072
circleci 156 139 0.5 1222871 413948 pts/2 Dl+ 07:00 0:29 \_ node /home/circleci/project/node_modules/react-scripts/scripts/build.js --max-old-space-size=3072
circleci 175 9.0 0.0 561533 30992 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci 182 8.0 0.0 561533 30964 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci 189 9.0 0.0 561533 31008 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci 196 8.0 0.0 561533 30976 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci 203 9.0 0.0 561533 31040 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci 210 6.0 0.0 561533 30992 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci 217 9.0 0.0 561533 31008 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci 224 8.0 0.0 561533 30944 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci 231 7.0 0.0 561533 30960 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci 238 6.0 0.0 561533 31012 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci 245 9.0 0.0 561533 30940 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci 252 6.0 0.0 561533 30884 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci 259 6.0 0.0 561533 31064 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci 266 9.0 0.0 561533 30988 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci 273 8.0 0.0 561533 30988 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci 280 6.0 0.0 561020 30888 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci 287 8.0 0.0 561533 30856 pts/2 Sl+ 07:00 0:00 \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
How to fix it?
You can check an option to serially run your Node.js package on a documentation. If you use popular packages such as Jest and Webpack, they have an option to disable the parallelism option.
Jest
In single run mode, Jest uses the number of the cores available on a machine minus one for the main thread as the the maximum number of workers. So, you need to use --maxWorkers or --runInBand option to specify the maximum number of workers.
Webpack
Webpack4 starts using terser-webpack-plugin to minify your JavaScript as default. The default parallel option in terser-webpack-plugin is set to the number of CPUs minus one (os.cpus().length - 1). So, if you use webpack4, you need to intentionally set the parallel option in the minimizer config.
Karma
concurrency option set Infinity as default, which is for how many browsers Karma launches in parallel.
https://github.com/karma-runner/karma/blob/master/docs/config/01-configuration-file.md#concurrency
fork-ts-checker-webpack-plugin
Until v3, this plugin has multi-process mode and spawn multiple workers. However this feature has dropped since v4.
--max-old-space-size option doesn’t restrict the max memory usage?
Yes and No. If your Node script creates a single process, it will work. However, this option affects only the current process, so when the process are forked, child processes can use memory up to the limit that specified in this option, and it causes to use too much memory.
A script doesn’t fork but it spends all the memory
If a script needs to buffer data into the memory, there is a case that the default resource_class memory size isn’t enough. For example, ts-loader is buffering every file in memory as default. In that case, you need to consider using resouce_class feature to increse vCPU and memory resources.
https://circleci.com/docs/2.0/configuration-reference/#resource_class