Hello,
we are currently trying to run our acceptance tests in CircleCI. But all of our Akka based services are killed by the Linux OOM Killer
[72436.342058] java invoked oom-killer: gfp_mask=0x50, order=0, oom_score_adj=0 [72436.342063] java cpuset=4f188f0cf68997499ad2b412869a847183ec6402d43ad436a1cbffc7f5a37545 mems_allowed=0 [72436.342067] CPU: 1 PID: 114094 Comm: java Not tainted 3.13.0-91-generic #138-Ubuntu [72436.342069] Hardware name: Xen HVM domU, BIOS 4.2.amazon 05/12/2016 [72436.342070] 0000000000000000 ffff880104b83c50 ffffffff81729906 ffff8801e6f9c800 [72436.342077] ffff880111a02c00 ffff880104b83cd8 ffffffff81723eca ffffea0003328a00 [72436.342081] ffff880055d313d0 ffff880055d31280 ffff880104b83ca8 ffffffff81155567 [72436.342085] Call Trace: [72436.342094] [<ffffffff81729906>] dump_stack+0x64/0x82 [72436.342098] [<ffffffff81723eca>] dump_header+0x7f/0x1f1 [72436.342103] [<ffffffff81155567>] ? find_lock_task_mm+0x27/0x70 [72436.342106] [<ffffffff8115599e>] oom_kill_process+0x1ce/0x330 [72436.342110] [<ffffffff811b3c33>] ? mem_cgroup_iter+0x153/0x340 [72436.342116] [<ffffffff812dc655>] ? security_capable_noaudit+0x15/0x20 [72436.342120] [<ffffffff811b7c4c>] mem_cgroup_oom_synchronize+0x51c/0x560 [72436.342124] [<ffffffff811b7180>] ? mem_cgroup_charge_common+0xa0/0xa0 [72436.342127] [<ffffffff81156114>] pagefault_out_of_memory+0x14/0x80 [72436.342130] [<ffffffff817224d5>] mm_fault_error+0x67/0x140 [72436.342135] [<ffffffff81735da2>] __do_page_fault+0x4a2/0x560 [72436.342140] [<ffffffff810de16a>] ? do_futex+0x10a/0x760 [72436.342144] [<ffffffff810a2ba5>] ? set_next_entity+0x95/0xb0 [72436.342149] [<ffffffff810135db>] ? __switch_to+0x16b/0x4f0 [72436.342153] [<ffffffff81735e7a>] do_page_fault+0x1a/0x70 [72436.342157] [<ffffffff817321a8>] page_fault+0x28/0x30 [72436.342161] Task in /user/1001.user/5141.session/lxc/box862/lxc/4f188f0cf68997499ad2b412869a847183ec6402d43ad436a1cbffc7f5a37545 killed as a result of limit of /user/1001.user/5141.session/lxc/box862/lxc/4f188f0cf68997499ad2b412869a847183ec6402d43ad436a1cbffc7f5a37545 [72436.342164] memory: usage 262144kB, limit 262144kB, failcnt 14076 [72436.342166] memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0 [72436.342167] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0 [72436.342169] Memory cgroup stats for /user/1001.user/5141.session/lxc/box862/lxc/4f188f0cf68997499ad2b412869a847183ec6402d43ad436a1cbffc7f5a37545: cache:1968KB rss:260176KB rss_huge:67584KB mapped_file:808KB writeback:0KB inactive_anon:0KB active_anon:260176KB inactive_file:1032KB active_file:808KB unevictable:0KB [72436.342186] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [72436.342503] [110239] 165536 110239 377 63 6 0 0 sh [72436.342514] [113885] 165536 113885 4666526 60635 615 0 0 java [72436.342605] Memory cgroup out of memory: Kill process 41681 (java) score 934 or sacrifice child [72436.347067] Killed process 41681 (java) total-vm:18666104kB, anon-rss:233996kB, file-rss:8544kB
From this I can see that the Java VM has tried to allocate more than 18GB of memory. But actually we are limit the Heap size to 128MB in tests and have also limited the Metaspace to 128MB of RAM.
Does anybody has an idea why our Java process is trying to allocate so much memory, despite not needing it and having size limits on its heap?
In more constraints environments (e.g. local Docker machine with 1GB of RAM) the services run fine and aren’t killed.
What is different in CircleCI or what can we do to debug this further?
Thanks in advance