I can’t run dmesg nor find system logs inside of a CircleCI-proviced Docker image (circleci/openjdk:11.0.4-jdk-stretch).
When I try running dmesg, I get dmesg: read kernel buffer failed: Operation not permitted
… even though I already did sysctl kernel.dmesg_restrict=0 in the host (Docker for Mac’s Hyperkit), and it is shown as 0 in the container.
And /var/log/ doesn’t contain anything interesting:
For context: the problem I am trying to fix is that I suspect that my gradle test tasks are receiving OOM kills, and I was hoping to check the system logs inside the container to find out if this is actually happening.
I have found that if I do docker run -it --privileged circleci/openjdk:11.0.4-jdk-stretch dmesg
it does work. Is there a way to make circleCI use that --privileged flag when it runs a docker executor? The documentation doesn’t mention how to do that.
Giving up. I’ve seen a number of other posts asking for a way to pass flags to the Docker runner, and they never even got an answer. Would be nice if there was at least a clear “not gonna happen” answer.
In any case, looks like the easiest way to get kernel logs without CircleCI’s help is by doing something like
At a guess, this is a Docker problem, not a CircleCI problem. Build machines using the Docker executor are already using Docker, so if you are using Docker on top of that, you are using two layers of Docker. It works for most things, but there are sometimes privilege issues with edge cases.
If you are stuck memory-wise, consider switching to the Machine executor. This is a traditional VM, from where you can run Docker with more permissions. You also get 8G of RAM as standard, instead of 4G with the Docker executor, so you may be less likely to run into memory issues anyway.
I’m not sure I’m following you. I am NOT using Docker-inside-of-Docker. What I was wishing for was a way to modify the Docker calls that CircleCI does.
The docker ... nsenter line is to be used in parallel to whatever CircleCI runs, not inside a container.
The problem is that I couldn’t even know whether my problem was lack of memory or misuse of memory. Thanks to the kernel logs I finally could diagnose that the problem was misuse, and after fixing it I could even more-than-half the size of my container, both in memory and CPU.
It’s kinda ironic that the CircleCI blog talks about OOM problems with Java, but never mentions the kernel log, which AFAIK is the only place where one can actually diagnose this.