Scenario:
场景:
I have a JVM running in a docker container. I did some memory analysis using two tools: 1) top 2) Java Native Memory Tracking. The numbers look confusing and I am trying to find whats causing the differences.
我有一个运行在docker容器中的JVM。我用两个工具做了一些内存分析:1)top 2) Java本机内存跟踪。这些数字看起来很令人困惑，我试图找出造成这些差异的原因。
Question:
问题:
The RSS is reported as 1272MB for the Java process and the Total Java Memory is reported as 790.55 MB. How can I explain where did the rest of the memory 1272 - 790.55 = 481.44 MB go?
对于Java进程，RSS被报告为1272MB，总的Java内存被报告为790.55 MB。我如何解释其余的内存1272 - 790.55 = 481.44 MB ?
Why I want to keep this issue open even after looking at this question on SO:
为什么我要在看到这个问题之后继续讨论这个问题?
I did see the answer and the explanation makes sense. However, after getting output from Java NMT and pmap -x , I am still not able to concretely map which java memory addresses are actually resident and physically mapped. I need some concrete explanation (with detailed steps) to find whats causing this difference between RSS and Java Total committed memory.
我确实看到了答案，这个解释说得通。但是，在从Java NMT和pmap -x中获得输出之后，我仍然不能具体地映射哪些Java内存地址是驻留的和物理映射的。我需要一些具体的解释(包括详细的步骤)来发现RSS和Java总提交内存之间的差异。
Top Output
最高产量
Java NMT
Java NMT
Docker memory stats
码头工人内存统计数据
Graphs
图
I have a docker container running for most than 48 hours. Now, when I see a graph which contains:
我有一个docker容器可以运行超过48小时。现在，当我看到一个包含:
- Total memory given to the docker container = 2 GB
- 给docker容器的总内存= 2gb
- Java Max Heap = 1 GB
- Java Max Heap = 1gb
- Total committed (JVM) = always less than 800 MB
- 总提交(JVM) =总小于800mb。
- Heap Used (JVM) = always less than 200 MB
- 堆Used (JVM) =总是小于200mb
- Non Heap Used (JVM) = always less than 100 MB.
- 非堆使用(JVM) =总是小于100 MB。
- RSS = around 1.1 GB.
- RSS =大约1.1 GB。
So, whats eating the memory between 1.1 GB (RSS) and 800 MB (Java Total committed memory)?
那么，是什么在消耗1.1 GB (RSS)和800 MB (Java总提交内存)之间的内存呢?
1 个解决方案
#1
19
You have some clue in " Analyzing java memory usage in a Docker container" from Mikhail Krestjaninoff:
关于“分析Docker容器中的java内存使用”，您有一些线索，请参见Mikhail Krestjaninoff的文章:
Resident Set Size is the amount of physical memory currently allocated and used by a process (without swapped out pages). It includes the code, data and shared libraries (which are counted in every process which uses them)
驻留集大小是一个进程当前分配和使用的物理内存的数量(不交换页面)。它包括代码、数据和共享库(在使用它们的每个进程中进行计数)
Why does docker stats info differ from the ps data?
为什么docker stats信息与ps数据不同?
Answer for the first question is very simple - Docker has a bug (or a feature - depends on your mood): it includes file caches into the total memory usage info. So, we can just avoid this metric and use
psinfo about RSS.第一个问题的答案很简单——Docker有一个bug(或特性)，这取决于你的心情。因此，我们可以避免这种度量，并使用关于RSS的ps信息。
Well, ok - but why is RSS higher than Xmx?
好的，但是为什么RSS比Xmx高呢?
Theoretically, in case of a java application
理论上，对于java应用程序
RSS = Heap size + MetaSpace + OffHeap size
where OffHeap consists of thread stacks, direct buffers, mapped files (libraries and jars) and JVM code itse
在什么地方有线程堆栈、直接缓冲区、映射文件(库和jar)和JVM代码itse ?
Since JDK 1.8.40 we have Native Memory Tracker!
从JDK 1.8.40开始，我们就有了本机内存跟踪器!
As you can see, I’ve already added
-XX:NativeMemoryTracking=summaryproperty to the JVM, so we can just invoke it from the command line:如您所见，我已经向JVM添加了-XX:NativeMemoryTracking=summary属性，因此我们可以从命令行调用它:
docker exec my-app jcmd 1 VM.native_memory summary
(This is what the OP did)
(这就是OP所做的)
Don’t worry about the “Unknown” section - seems that NMT is an immature tool and can’t deal with CMS GC (this section disappears when you use an another GC).
不要担心“未知”部分——看起来NMT是一个不成熟的工具，不能处理CMS GC(当您使用另一个GC时，这个部分将消失)。
Keep in mind, that NMT displays “committed” memory, not "resident" (which you get through the ps command). In other words, a memory page can be committed without considering as a resident (until it directly accessed).
请记住，NMT显示的是“提交”内存，而不是“驻留”内存(您可以通过ps命令获得)。换句话说，一个内存页可以在不考虑作为驻留(直到直接访问)的情况下提交。
That means that NMT results for non-heap areas (heap is always preinitialized) might be bigger than RSS values.
这意味着对于非堆区域(堆总是预先初始化)的NMT结果可能大于RSS值。
(that is where "Why does a JVM report more committed memory than the linux process resident set size?" comes in)
(这就是“为什么JVM报告的提交内存比linux进程驻留的内存大?”)
As a result, despite the fact that we set the jvm heap limit to 256m, our application consumes 367M. The “other” 164M are mostly used for storing class metadata, compiled code, threads and GC data.
因此，尽管我们将jvm堆限制设置为256m，但是我们的应用程序消耗了367M。“其他”164M主要用于存储类元数据、编译代码、线程和GC数据。
First three points are often constants for an application, so the only thing which increases with the heap size is GC data.
This dependency is linear, but the “k” coefficient (y = kx + b) is much less then 1.前三点通常是应用程序的常量，因此随着堆大小的增加，惟一增加的就是GC数据。这种依赖是线性的，但是k系数(y = kx + b)比1小得多。
More generally, this seems to be followed by issue 15020 which reports a similar issue since docker 1.7
更一般地说，这之后是第15020期，它报告了docker 1.7以来的类似问题
I'm running a simple Scala (JVM) application which loads a lot of data into and out of memory.
I set the JVM to 8G heap (-Xmx8G). I have a machine with 132G memory, and it can't handle more than 7-8 containers because they grow well past the 8G limit I imposed on the JVM.我正在运行一个简单的Scala (JVM)应用程序，它将大量数据加载到内存中或内存中。我将JVM设置为8G堆(-Xmx8G)。我有一台拥有132G内存的机器，它不能处理超过7-8个容器，因为它们远远超过了我在JVM上施加的8G限制。
(docker stat was reported as misleading before, as it apparently includes file caches into the total memory usage info)
(docker stat之前被报告为具有误导性，因为它显然在总内存使用信息中包含了文件缓存)
docker statshows that each container itself is using much more memory than the JVM is supposed to be using. For instance:docker stat表明，每个容器本身使用的内存比JVM应该使用的内存要多得多。例如:
CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O
dave-1 3.55% 10.61 GB/135.3 GB 7.85% 7.132 MB/959.9 MB
perf-1 3.63% 16.51 GB/135.3 GB 12.21% 30.71 MB/5.115 GB
It almost seems that the JVM is asking the OS for memory, which is allocated within the container, and the JVM is freeing memory as its GC runs, but the container doesn't release the memory back to the main OS. So... memory leak.
看起来JVM向操作系统请求内存(在容器中分配)，JVM在GC运行时释放内存，但是容器没有将内存释放回主操作系统。所以…内存泄漏。
#1
19
You have some clue in " Analyzing java memory usage in a Docker container" from Mikhail Krestjaninoff:
关于“分析Docker容器中的java内存使用”，您有一些线索，请参见Mikhail Krestjaninoff的文章:
Resident Set Size is the amount of physical memory currently allocated and used by a process (without swapped out pages). It includes the code, data and shared libraries (which are counted in every process which uses them)
驻留集大小是一个进程当前分配和使用的物理内存的数量(不交换页面)。它包括代码、数据和共享库(在使用它们的每个进程中进行计数)
Why does docker stats info differ from the ps data?
为什么docker stats信息与ps数据不同?
Answer for the first question is very simple - Docker has a bug (or a feature - depends on your mood): it includes file caches into the total memory usage info. So, we can just avoid this metric and use
psinfo about RSS.第一个问题的答案很简单——Docker有一个bug(或特性)，这取决于你的心情。因此，我们可以避免这种度量，并使用关于RSS的ps信息。
Well, ok - but why is RSS higher than Xmx?
好的，但是为什么RSS比Xmx高呢?
Theoretically, in case of a java application
理论上，对于java应用程序
RSS = Heap size + MetaSpace + OffHeap size
where OffHeap consists of thread stacks, direct buffers, mapped files (libraries and jars) and JVM code itse
在什么地方有线程堆栈、直接缓冲区、映射文件(库和jar)和JVM代码itse ?
Since JDK 1.8.40 we have Native Memory Tracker!
从JDK 1.8.40开始，我们就有了本机内存跟踪器!
As you can see, I’ve already added
-XX:NativeMemoryTracking=summaryproperty to the JVM, so we can just invoke it from the command line:如您所见，我已经向JVM添加了-XX:NativeMemoryTracking=summary属性，因此我们可以从命令行调用它:
docker exec my-app jcmd 1 VM.native_memory summary
(This is what the OP did)
(这就是OP所做的)
Don’t worry about the “Unknown” section - seems that NMT is an immature tool and can’t deal with CMS GC (this section disappears when you use an another GC).
不要担心“未知”部分——看起来NMT是一个不成熟的工具，不能处理CMS GC(当您使用另一个GC时，这个部分将消失)。
Keep in mind, that NMT displays “committed” memory, not "resident" (which you get through the ps command). In other words, a memory page can be committed without considering as a resident (until it directly accessed).
请记住，NMT显示的是“提交”内存，而不是“驻留”内存(您可以通过ps命令获得)。换句话说，一个内存页可以在不考虑作为驻留(直到直接访问)的情况下提交。
That means that NMT results for non-heap areas (heap is always preinitialized) might be bigger than RSS values.
这意味着对于非堆区域(堆总是预先初始化)的NMT结果可能大于RSS值。
(that is where "Why does a JVM report more committed memory than the linux process resident set size?" comes in)
(这就是“为什么JVM报告的提交内存比linux进程驻留的内存大?”)
As a result, despite the fact that we set the jvm heap limit to 256m, our application consumes 367M. The “other” 164M are mostly used for storing class metadata, compiled code, threads and GC data.
因此，尽管我们将jvm堆限制设置为256m，但是我们的应用程序消耗了367M。“其他”164M主要用于存储类元数据、编译代码、线程和GC数据。
First three points are often constants for an application, so the only thing which increases with the heap size is GC data.
This dependency is linear, but the “k” coefficient (y = kx + b) is much less then 1.前三点通常是应用程序的常量，因此随着堆大小的增加，惟一增加的就是GC数据。这种依赖是线性的，但是k系数(y = kx + b)比1小得多。
More generally, this seems to be followed by issue 15020 which reports a similar issue since docker 1.7
更一般地说，这之后是第15020期，它报告了docker 1.7以来的类似问题
I'm running a simple Scala (JVM) application which loads a lot of data into and out of memory.
I set the JVM to 8G heap (-Xmx8G). I have a machine with 132G memory, and it can't handle more than 7-8 containers because they grow well past the 8G limit I imposed on the JVM.我正在运行一个简单的Scala (JVM)应用程序，它将大量数据加载到内存中或内存中。我将JVM设置为8G堆(-Xmx8G)。我有一台拥有132G内存的机器，它不能处理超过7-8个容器，因为它们远远超过了我在JVM上施加的8G限制。
(docker stat was reported as misleading before, as it apparently includes file caches into the total memory usage info)
(docker stat之前被报告为具有误导性，因为它显然在总内存使用信息中包含了文件缓存)
docker statshows that each container itself is using much more memory than the JVM is supposed to be using. For instance:docker stat表明，每个容器本身使用的内存比JVM应该使用的内存要多得多。例如:
CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O
dave-1 3.55% 10.61 GB/135.3 GB 7.85% 7.132 MB/959.9 MB
perf-1 3.63% 16.51 GB/135.3 GB 12.21% 30.71 MB/5.115 GB
It almost seems that the JVM is asking the OS for memory, which is allocated within the container, and the JVM is freeing memory as its GC runs, but the container doesn't release the memory back to the main OS. So... memory leak.
看起来JVM向操作系统请求内存(在容器中分配)，JVM在GC运行时释放内存，但是容器没有将内存释放回主操作系统。所以…内存泄漏。