Unexpected parallelism makes webpack build explode

Feedback & Bug Reports
1

During a webpack step in our builds, a large number of unexpected parallel node processes are started, leading to the build hanging and finally bailing out due to exceeding the 4 GB memory quota.

We don’t see the parallel node processes when running the exact same steps locally, in the exact same environment (including env variables set by Circle and by ourselves in the build config). So locally the build takes no more than 1.5 GB of memory.

Here’s a link to the minimal build configuration where this behaviour can be observed:. I’ve also collected a memory profile by polling all node processes in ps aux every second.

local results (expected)

profile
https://gist.githubusercontent.com/tomekwi/d9ec4d1301bb13bb47f56624648cc436/raw/56466ab2142025344e09e7f71dbd9b99b5571dc0/local

running time
59 s

peak memory consumption
< 1.5 GB

CircleCI results (:boom:)

profile
https://gist.githubusercontent.com/tomekwi/d9ec4d1301bb13bb47f56624648cc436/raw/56466ab2142025344e09e7f71dbd9b99b5571dc0/circle

running time

  • output freezes after 3 min 38 s
  • SSH session terminates after 4 min 18 s
  • Circle reports build step duration of 8 min 40 s

peak memory consumption
> 4.3 GB

other idiosyncracies

  • note that the profile only reports 1 min 2 s in the TIME column of ps aux just before the freeze (just
  • also notice how in the profile the TIME column doesn’t increment by 1 on each actual second as it should – it runs a lot slower

profiling method

Build analysed https://circleci.com/gh/kalohq/frontend/24882#tests/containers/1

Locally, all steps run 1:1 as in the build, with environment variables 1:1 copied over from the steps Start container (1) and echo "AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID" echo "AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY" echo "CODECOV_TOKEN=$CODECOV_TOKEN" echo "DOCKER_EMAIL=$DOCKER_EMAIL" echo "DOCKER_PASS=$DOCKER_PASS" echo "DOCKER_USER=$DOCKER_USER" echo "GITHUB_TOKEN=$GITHUB_TOKEN" echo "HIPCHAT_V1_KEY=$HIPCHAT_V1_KEY" echo "LYSTACOP_SECRET=$LYSTACOP_SECRET" echo "RELEASE_ROCKET_SLACK_TOKEN=$RELEASE_ROCKET_SLACK_TOKEN" echo "SALT_KEY=$SALT_KEY" (1)

Profile run during step case $CIRCLE_NODE_INDEX in 0) true;; 1) ./bin/build.sh -e docker;; esac (1).

Profiling loop:

while sleep 1; do
  echo -e '\n\n\n\n\n'
  ps aux | sort -k6,6n | grep node | grep -v Applications | grep -v 'grep node'
done