I can’t run dmesg nor find system logs inside of a CircleCI-proviced Docker image (circleci/openjdk:11.0.4-jdk-stretch).
When I try running dmesg, I get
dmesg: read kernel buffer failed: Operation not permitted
… even though I already did
sysctl kernel.dmesg_restrict=0 in the host (Docker for Mac’s Hyperkit), and it is shown as 0 in the container.
And /var/log/ doesn’t contain anything interesting:
# ls -la
drwxr-xr-x 1 root root 4096 Dec 17 03:49 .
drwxr-xr-x 1 root root 4096 Sep 10 00:00 ..
-rw-r--r-- 1 root root 17125 Oct 16 19:39 alternatives.log
drwxr-xr-x 1 root root 4096 Oct 16 19:39 apt
-rw-rw---- 1 root utmp 0 Sep 10 00:00 btmp
-rw-r--r-- 1 root root 204721 Oct 16 19:39 dpkg.log
drwxr-s--- 2 Debian-exim adm 4096 Oct 16 19:36 exim4
-rw-r--r-- 1 root root 109920 Oct 16 19:37 faillog
-rw-r--r-- 1 root root 1177 Oct 16 19:36 fontconfig.log
-rw-rw-r-- 1 root utmp 1003020 Oct 16 19:37 lastlog
drwxr-xr-x 2 root root 4096 May 25 2017 sysstat
-rw-rw-r-- 1 root utmp 0 Sep 10 00:00 wtmp
Is this a bug? How can we access the system logs?
For context: the problem I am trying to fix is that I suspect that my gradle test tasks are receiving OOM kills, and I was hoping to check the system logs inside the container to find out if this is actually happening.
I have found that if I do
docker run -it --privileged circleci/openjdk:11.0.4-jdk-stretch dmesg
it does work. Is there a way to make circleCI use that
--privileged flag when it runs a docker executor? The documentation doesn’t mention how to do that.
Giving up. I’ve seen a number of other posts asking for a way to pass flags to the Docker runner, and they never even got an answer. Would be nice if there was at least a clear “not gonna happen” answer.
In any case, looks like the easiest way to get kernel logs without CircleCI’s help is by doing something like
$ docker run --rm -it --privileged --pid=host walkerlee/nsenter -t 1 -m -u -i -n bash
# tail -f /var/log/kmsg.log | grep oom
At a guess, this is a Docker problem, not a CircleCI problem. Build machines using the Docker executor are already using Docker, so if you are using Docker on top of that, you are using two layers of Docker. It works for most things, but there are sometimes privilege issues with edge cases.
If you are stuck memory-wise, consider switching to the Machine executor. This is a traditional VM, from where you can run Docker with more permissions. You also get 8G of RAM as standard, instead of 4G with the Docker executor, so you may be less likely to run into memory issues anyway.
I’m not sure I’m following you. I am NOT using Docker-inside-of-Docker. What I was wishing for was a way to modify the Docker calls that CircleCI does.
docker ... nsenter line is to be used in parallel to whatever CircleCI runs, not inside a container.
The problem is that I couldn’t even know whether my problem was lack of memory or misuse of memory. Thanks to the kernel logs I finally could diagnose that the problem was misuse, and after fixing it I could even more-than-half the size of my container, both in memory and CPU.
It’s kinda ironic that the CircleCI blog talks about OOM problems with Java, but never mentions the kernel log, which AFAIK is the only place where one can actually diagnose this.