Debugging random Node.js build failures

There are some patterns that cause random issues which have never happened in your local machine. The random issue means the job fails sometimes without any changes. In this article, I’ll share the most common cause of the random issue and how you can debug it.

OOM (Out of memory) error

OOM error is the most common cause of the random issue in Node.js project. This happens when the process has been SIGKILL ed by the OOM killer. It means when your process uses almost all the memory that the executor has, the OOM killer kills your process.

If you use Node.js on your project and run a job on Docker executor, you might see this error more often. This is because the host VM for Docker executor has more CPUs and memory resources than the docker executor has been allocated, and os.cpus() which is the native Node.js function counts the number of CPUs on the host VM. Many of Node.js packages use this function to makes workers/threads for speeding up its process. The current host VM of the docker executor has 32 cores, but the default Docker executor has only 2 vCPUs, so there is a huge gap.

What should I check at first?

You can check max memory usage and the existence of the child process. If there are many child processes, you should better configure your script to run serially in the current process.

The following article shares the way to see the max memory usage and see the existence of the child process.

This is an example output and you can see child processes of the react-scripts build command. If you see similar outputs and your job is failed, it might be failed with OOM error.

circleci    91  0.0  0.0  19740  3088 pts/1    Ss+  06:59   0:00  \_ /bin/bash -eo pipefail -c while true; do sleep 5 ps auxwwf echo "======" done 
circleci   350  0.0  0.0  38540  3180 pts/1    R+   07:00   0:00  |   \_ ps auxwwf
circleci    93  0.0  0.0   4280   736 pts/2    Ss+  06:59   0:00  \_ sh ./build.sh
circleci   137  1.2  0.0 734681 43540 pts/2    Sl+  07:00   0:00      \_ npm
circleci   148  0.0  0.0   4292   760 pts/2    S+   07:00   0:00          \_ sh -c react-scripts build --max-old-space-size=3072 
circleci   149  0.2  0.0 111111 31264 pts/2    Sl+  07:00   0:00              \_ node /home/circleci/project/node_modules/.bin/react-scripts build --max-old-space-size=3072 
circleci   156  139  0.5 1222871 413948 pts/2  Dl+  07:00   0:29                  \_ node /home/circleci/project/node_modules/react-scripts/scripts/build.js --max-old-space-size=3072 
circleci   175  9.0  0.0 561533 30992 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci   182  8.0  0.0 561533 30964 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci   189  9.0  0.0 561533 31008 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci   196  8.0  0.0 561533 30976 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci   203  9.0  0.0 561533 31040 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci   210  6.0  0.0 561533 30992 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci   217  9.0  0.0 561533 31008 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci   224  8.0  0.0 561533 30944 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci   231  7.0  0.0 561533 30960 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci   238  6.0  0.0 561533 31012 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci   245  9.0  0.0 561533 30940 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci   252  6.0  0.0 561533 30884 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci   259  6.0  0.0 561533 31064 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci   266  9.0  0.0 561533 30988 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci   273  8.0  0.0 561533 30988 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci   280  6.0  0.0 561020 30888 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js
circleci   287  8.0  0.0 561533 30856 pts/2    Sl+  07:00   0:00                      \_ /usr/local/bin/node /home/circleci/project/node_modules/terser-webpack-plugin/node_modules/jest-worker/build/workers/processChild.js

How to fix it?

You can check an option to serially run your Node.js package on a documentation. If you use popular packages such as Jest and Webpack, they have an option to disable the parallelism option.

Jest

In single run mode, Jest uses the number of the cores available on a machine minus one for the main thread as the the maximum number of workers. So, you need to use --maxWorkers or --runInBand option to specify the maximum number of workers.

Webpack

Webpack4 starts using terser-webpack-plugin to minify your JavaScript as default. The default parallel option in terser-webpack-plugin is set to the number of CPUs minus one (os.cpus().length - 1). So, if you use webpack4, you need to intentionally set the parallel option in the minimizer config.

Karma

concurrency option set Infinity as default, which is for how many browsers Karma launches in parallel.

karma/01-configuration-file.md at master · karma-runner/karma · GitHub

fork-ts-checker-webpack-plugin

Until v3, this plugin has multi-process mode and spawn multiple workers. However this feature has dropped since v4.

--max-old-space-size option doesn’t restrict the max memory usage?

Yes and No. If your Node script creates a single process, it will work. However, this option affects only the current process, so when the process are forked, child processes can use memory up to the limit that specified in this option, and it causes to use too much memory.

A script doesn’t fork but it spends all the memory

If a script needs to buffer data into the memory, there is a case that the default resource_class memory size isn’t enough. For example, ts-loader is buffering every file in memory as default. In that case, you need to consider using resouce_class feature to increse vCPU and memory resources.

https://circleci.com/docs/2.0/configuration-reference/#resource_class

2 Likes