PID 1 In `circleci/rust:1.34.1-browsers` not reaping zombies

levlaz · May 4, 2019, 6:27pm

I am adding this here on behalf of a user who opened up a ticket.

I’m looking at the build at , and looking at the process list I’ve noticed it looks like we have /bin/sh as PID 1, but we also have a lot of zombies (as below).

circleci@561cb2c1ca61:~$ ps auxwf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
circleci 23 11.9 0.0 125128 45396 ? Ssl 07:33 0:05 /bin/circleci-agent --config /.circleci-runner-config.json --task-data /.circleci-task-data --outerServerUrl https://circleci-internal-outer-build-agent:5500 _in
circleci 495 0.0 0.0 19984 3584 pts/2 Ss 07:34 0:00 \_ /bin/bash
circleci 8908 0.0 0.0 38708 3528 pts/2 R+ 07:34 0:00 \_ ps auxwf
circleci 1 0.0 0.0 4280 724 pts/0 Ss+ 07:33 0:00 /bin/sh
circleci 6 0.4 0.0 321204 18424 pts/0 Sl+ 07:33 0:00 Xvfb :99 -screen 0 1280x1024x24
circleci 532 0.0 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 564 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 565 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 638 0.0 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 646 0.1 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 648 0.1 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 649 0.0 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 650 0.1 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 651 0.2 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 652 0.1 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 653 0.1 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 654 0.1 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 655 0.1 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 656 0.1 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 660 0.1 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 661 0.1 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 663 0.1 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 664 0.1 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 665 0.1 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 666 0.2 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 667 0.2 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 684 0.1 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 685 0.1 0.0 0 0 ? Z 07:34 0:00 [chromedriver] <defunct>
circleci 785 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 806 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 809 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 815 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 816 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 827 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 828 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 830 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 833 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 848 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 849 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 862 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 864 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 866 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 869 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 871 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 872 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 880 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 881 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 884 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 885 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 887 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 888 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 890 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 891 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 892 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 893 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 894 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 895 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 896 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 897 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 898 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 899 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 900 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 901 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 902 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 903 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 904 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 905 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 906 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 907 0.0 0.0 0 0 ? Z 07:34 0:00 [cat] <defunct>
circleci 909 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 911 0.1 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 922 0.1 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 925 0.1 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 926 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 931 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 932 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 933 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 937 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 938 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 939 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 940 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 941 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 947 0.1 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 948 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 949 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 950 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 951 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 954 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 955 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1126 0.4 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1127 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1128 0.4 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1129 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1130 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1137 0.4 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1140 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1148 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1149 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1153 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1162 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1163 0.2 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1205 0.4 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1217 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1242 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1256 0.4 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1258 0.4 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1261 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1262 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1269 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1272 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1281 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1291 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1305 0.4 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1309 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1310 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1335 0.4 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1364 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1366 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1375 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1403 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1432 0.4 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1437 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1445 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1451 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1452 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1474 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1505 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1512 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1515 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1517 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1611 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1618 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1628 0.4 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1689 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1717 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1751 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1767 0.9 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1808 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1812 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1883 0.4 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1929 0.6 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1948 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1949 0.3 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1963 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1970 0.8 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 1983 0.4 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 2007 0.9 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 2015 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 2047 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 2054 0.4 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 2055 0.5 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 2099 0.7 0.0 0 0 ? Z 07:34 0:00 [chrome] <defunct>
circleci 2169 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2285 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2288 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2292 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2293 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2294 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2303 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2306 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2307 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2311 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2312 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2314 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2315 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2323 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2332 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2333 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2334 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2339 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2346 0.1 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2348 0.1 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 2349 0.0 0.0 0 0 ? Z 07:34 0:00 [geckodriver] <defunct>
circleci 5408 1.9 0.0 0 0 ? Z 07:34 0:00 [WebExtensions] <defunct>
circleci 5785 1.5 0.0 0 0 ? Z 07:34 0:00 [Web Content] <defunct>
circleci 7167 1.7 0.0 0 0 ? Z 07:34 0:00 [Web Content] <defunct>
circleci 7900 1.1 0.0 0 0 ? Z 07:34 0:00 [Web Content] <defunct>
circleci@561cb2c1ca61:~$

levlaz · May 4, 2019, 6:29pm

I am not sure if this is specific to our image, the upstream image, or something wrong with the configuration itself, but I’d love to gather more data here.

If you are reading this, and are seeing a similar issue please post your details.

forozco · May 8, 2019, 9:45pm

We have ran into a similar issue in github/palantir/sls-packaging. There we have tests that rely on zombie processed being reaped and have observed builds that previously succeeded start to fail because of zombie processes not being reaped (successful: https://circleci.com/gh/palantir/sls-packaging/2406, failed: https://circleci.com/gh/palantir/sls-packaging/2479). Both runs are using the exact same underlying image (circleci/openjdk@sha256:12a55407eae538da192ccb6f1ef380bc5cc97d710ca400c12205b1ed4fb079c3) but between runs the docker engine (successful: 17.05.0-ce, failed: 18.09.1) and build agent (successful: 1.0.10758-83ab9dd3, failed: 1.0.11043-90328079). This leads me to believe that the issue is not necessarily related to the image