This code snippet will allocate 2Gb every time it reads the letter 'u' from stdin, and will initialize all the allocated chars once it reads 'a'.
每次从stdin读取字母'u'时,此代码段将分配2Gb,并在读取'a'时初始化所有已分配的字符。
#include <iostream>
#include <stdlib.h>
#include <stdio.h>
#include <vector>
#define bytes 2147483648
using namespace std;
int main()
{
char input [1];
vector<char *> activate;
while(input[0] != 'q')
{
gets (input);
if(input[0] == 'u')
{
char *m = (char*)malloc(bytes);
if(m == NULL) cout << "cant allocate mem" << endl;
else cout << "ok" << endl;
activate.push_back(m);
}
else if(input[0] == 'a')
{
for(int x = 0; x < activate.size(); x++)
{
char *m;
m = activate[x];
for(unsigned x = 0; x < bytes; x++)
{
m[x] = 'a';
}
}
}
}
return 0;
}
I am running this code on a linux virtual machine that has 3Gb of ram. While monitoring the system resource usage using the htop tool, I have realized that the malloc operation is not reflected on the resources.
我在具有3Gb内存的Linux虚拟机上运行此代码。在使用htop工具监视系统资源使用情况时,我意识到malloc操作没有反映在资源上。
For example when I input 'u' only once(i.e. allocate 2GB of heap memory), I don't see the memory usage increasing by 2GB in htop. It is only when I input 'a'(i.e. initialize), I see the memory usage increasing.
例如,当我只输入'u'一次(即分配2GB的堆内存)时,我看不到内存使用量在htop中增加了2GB。只有当我输入“a”(即初始化)时,才会看到内存使用量的增加。
As a consequence, I am able to "malloc" more heap memory than there exists. For example, I can malloc 6GB(which is more than my ram and swap memory) and malloc would allow it(i.e. NULL is not returned by malloc). But when I try to initialize the allocated memory, I can see the memory and swap memory filling up till the process is killed.
因此,我能够“malloc”存在比存在更多的堆内存。例如,我可以使用malloc 6GB(这比我的ram和swap内存更多)和malloc允许它(即malloc不返回NULL)。但是当我尝试初始化分配的内存时,我可以看到内存和交换内存填满,直到进程被终止。
-My questions:
1.Is this a kernel bug?
这是一个内核错误吗?
2.Can someone explain to me why this behavior is allowed?
有人可以向我解释为什么允许这种行为?
6 个解决方案
#1
11
It is called memory overcommit. You can disable it by running as root:
它被称为内存过量使用。您可以通过以root身份运行来禁用它:
echo 2 > /proc/sys/vm/overcommit_memory
and it is not a kernel feature that I like (so I always disable it). See malloc(3) and mmap(2) and proc(5)
它不是我喜欢的内核功能(所以我总是禁用它)。参见malloc(3)和mmap(2)以及proc(5)
NB: echo 0 instead of echo 2 often -but not always- works also. Read the docs (in particular proc man page that I just linked to).
注意:经常回声0而不是回声2但是也不总是有效。阅读文档(特别是我刚刚链接到的proc手册页)。
#2
8
from man malloc (online here):
来自man malloc(在线):
By default, Linux follows an optimistic memory allocation strategy. This means that when malloc() returns non-NULL there is no guarantee that the memory really is available.
默认情况下,Linux遵循乐观的内存分配策略。这意味着当malloc()返回非NULL时,无法保证内存确实可用。
So when you just want to allocate too much, it "lies" to you, when you want to use the allocated memory, it will try to find enough memory for you and it might crash if it can't find enough memory.
因此,当你只想分配太多时,它“谎言”给你,当你想使用分配的内存时,它会尝试为你找到足够的内存,如果它找不到足够的内存,它可能会崩溃。
#3
5
No, this is not a kernel bug. You have discovered something known as late paging (or overcommit).
不,这不是内核错误。您发现了一种称为延迟分页(或过度使用)的内容。
Until you write a byte to the address allocated with malloc (...) the kernel does little more than "reserve" the address range. This really depends on the implementation of your memory allocator and operating system of course, but most good ones do not incur the majority of kernel overhead until the memory is first used.
直到你用malloc(...)分配的地址写一个字节,内核只会“保留”地址范围。这当然取决于内存分配器和操作系统的实现,但是在首次使用内存之前,大多数好的内核都不会产生大部分内核开销。
The hoard allocator is one big offender that comes to mind immediately, through extensive testing I have found it almost never takes advantage of a kernel that supports late paging. You can always mitigate the effects of late paging in any allocator if you zero-fill the entire memory range immediately after allocation.
囤积分配器是一个立即想到的大罪犯,通过广泛的测试,我发现它几乎从不利用支持延迟分页的内核。如果在分配后立即零填充整个内存范围,则可以始终减轻任何分配器中延迟分页的影响。
Real-time operating systems like VxWorks will never allow this behavior because late paging introduces serious latency. Technically, all it does is put the latency off until a later indeterminate time.
像VxWorks这样的实时操作系统永远不会允许这种行为,因为后期分页会引入严重的延迟。从技术上讲,它所做的就是延迟延迟,直到不确定的时间。
For a more detailed discussion, you may be interested to see how IBM's AIX operating system handles page allocation and overcommitment.
有关更详细的讨论,您可能有兴趣了解IBM的AIX操作系统如何处理页面分配和过度使用。
#4
3
This is a result of what Basile mentioned, over commit memory. However, the explanation kind of interesting.
这是Basile提到的结果,而不是提交内存。但是,解释有点有趣。
Basically when you attempt to map additional memory in Linux (POSIX?), the kernel will just reserve it, and will only actually end up using it if your application accesses one of the reserved pages. This allows multiple applications to reserve more than the actual total amount of ram / swap.
基本上,当您尝试在Linux(POSIX?)中映射其他内存时,内核将保留它,并且只有在您的应用程序访问其中一个保留页面时才会实际使用它。这允许多个应用程序保留超过实际的ram / swap总量。
This is desirable behavior on most Linux environments unless you've got a real-time OS or something where you know exactly who will need what resources, when and why.
这在大多数Linux环境中都是理想的行为,除非你有一个实时操作系统或者你确切知道谁将需要什么资源,何时以及为什么。
Otherwise somebody could come along, malloc up all the ram (without actually doing anything with it) and OOM your apps.
否则有人可能会来,所有的ram(没有实际做任何事情)和OOM你的应用程序。
Another example of this lazy allocation is mmap(), where you have a virtual map that the file you're mapping can fit inside - but you only have a small amount of real memory dedicated to the effort. This allows you to mmap() huge files (larger than your available RAM), and use them like normal file handles which is nifty)
这个延迟分配的另一个例子是mmap(),你有一个虚拟地图,你正在映射的文件可以放在里面 - 但你只有少量的实际内存专用于工作。这允许你mmap()巨大的文件(大于你的可用内存),并像普通的文件句柄一样使用它们很漂亮)
-n
#5
3
Initializing / working with the memory should work:
初始化/使用内存应该工作:
memset(m, 0, bytes);
Also you could use calloc that not only allocates memory but also fills it with zeros for you:
你也可以使用calloc,它不仅可以分配内存,还可以为你填充零:
char* m = (char*) calloc(1, bytes);
#6
1
1.Is this a kernel bug?
这是一个内核错误吗?
No.
2.Can someone explain to me why this behavior is allowed?
有人可以向我解释为什么允许这种行为?
There are a few reasons:
有几个原因:
-
Mitigate need to know eventual memory requirement - it's often convenient to have an application be able to an amount of memory that it considers an upper limit on the need it might actually have. For example, if it's preparing some kind of report either of an initial pass just to calculate the eventual size of the report or a realloc() of successively larger areas (with the risk of having to copy) may significantly complicate the code and hurt performance, where-as multiplying some maximum length of each entry by the number of entries could be very quick and easy. If you know virtual memory is relatively plentiful as far as your application's needs are concerned, then making a larger allocation of virtual address space is very cheap.
缓解需要知道最终的内存需求 - 让应用程序能够使用一定量的内存通常很方便,它认为它可能实际需要的上限。例如,如果它正在准备某种类型的报告,或者只是计算报告的最终大小的初始通过,或者是连续更大区域的realloc()(具有必须复制的风险)可能会使代码复杂化并损害性能,其中 - 将每个条目的一些最大长度乘以条目数可以非常快速和容易。如果您知道虚拟内存相对丰富,就您的应用程序的需求而言,那么进行更大的虚拟地址空间分配非常便宜。
-
Sparse data - if you have the virtual address space spare, being able to have a sparse array and use direct indexing, or allocate a hash table with generous capacity() to size() ratio, can lead to a very high performance system. Both work best (in the sense of having low overheads/waste and efficient use of memory caches) when the data element size is a multiple of the memory paging size, or failing that much larger or a small integral fraction thereof.
稀疏数据 - 如果您有虚拟地址空间备用,能够拥有稀疏数组并使用直接索引,或者分配具有大容量()到大小()比率的哈希表,则可以产生非常高性能的系统。当数据元素大小是存储器分页大小的倍数时,两者都工作得最好(在具有低开销/浪费和有效使用存储器高速缓存的意义上),或者使其失败大得多或小的整数部分。
-
Resource sharing - consider an ISP offering a "1 giga-bit per second" connection to 1000 consumers in a building - they know that if all the consumers use it simultaneously they'll get about 1 mega-bit, but rely on their real-world experience that, though people ask for 1 giga-bit and want a good fraction of it at specific times, there's inevitably some lower maximum and much lower average for concurrent usage. The same insight applied to memory allows operating systems to support more applications than they otherwise would, with reasonable average success at satisfying expectations. Much as the shared Internet connection degrades in speed as more users make simultaneous demands, paging from swap memory on disk may kick in and reduce performance. But unlike an internet connection, there's a limit to the swap memory, and if all the apps really do try to use the memory concurrently such that that limit's exceeded, some will start getting signals/interrupts/traps reporting memory exhaustion. Summarily, with this memory overcommit behaviour enabled, simply checking
malloc()/newreturned a non-NULL pointer is not sufficient to guarantee the physical memory is actually available, and the program may still receive a signal later as it attempts to use the memory.资源共享 - 考虑一个ISP向一个建筑物中的1000个消费者提供“每秒1千兆位”的连接 - 他们知道,如果所有消费者同时使用它,他们将获得大约1兆位,但依赖于他们的真实 - 世界经验虽然人们要求1千兆比特并且在特定时间想要它的很大一部分,但不可避免地会有一些较低的最大值和低得多的并发使用率。应用于内存的相同见解允许操作系统支持比其他方式更多的应用程序,并且在满足期望的情况下获得合理的平均成功。由于共享的Internet连接速度随着更多用户同时需求而降低,因此从磁盘上的交换内存进行分页可能会启动并降低性能。但与互联网连接不同的是,交换内存存在限制,并且如果所有应用程序确实尝试同时使用内存以超出限制,则某些应用程序将开始获取信号/中断/陷阱报告内存耗尽。总而言之,启用此内存过量使用行为,只需检查malloc()/ new返回的非NULL指针不足以保证物理内存实际可用,并且程序稍后在尝试使用内存时仍可能会收到信号。
#1
11
It is called memory overcommit. You can disable it by running as root:
它被称为内存过量使用。您可以通过以root身份运行来禁用它:
echo 2 > /proc/sys/vm/overcommit_memory
and it is not a kernel feature that I like (so I always disable it). See malloc(3) and mmap(2) and proc(5)
它不是我喜欢的内核功能(所以我总是禁用它)。参见malloc(3)和mmap(2)以及proc(5)
NB: echo 0 instead of echo 2 often -but not always- works also. Read the docs (in particular proc man page that I just linked to).
注意:经常回声0而不是回声2但是也不总是有效。阅读文档(特别是我刚刚链接到的proc手册页)。
#2
8
from man malloc (online here):
来自man malloc(在线):
By default, Linux follows an optimistic memory allocation strategy. This means that when malloc() returns non-NULL there is no guarantee that the memory really is available.
默认情况下,Linux遵循乐观的内存分配策略。这意味着当malloc()返回非NULL时,无法保证内存确实可用。
So when you just want to allocate too much, it "lies" to you, when you want to use the allocated memory, it will try to find enough memory for you and it might crash if it can't find enough memory.
因此,当你只想分配太多时,它“谎言”给你,当你想使用分配的内存时,它会尝试为你找到足够的内存,如果它找不到足够的内存,它可能会崩溃。
#3
5
No, this is not a kernel bug. You have discovered something known as late paging (or overcommit).
不,这不是内核错误。您发现了一种称为延迟分页(或过度使用)的内容。
Until you write a byte to the address allocated with malloc (...) the kernel does little more than "reserve" the address range. This really depends on the implementation of your memory allocator and operating system of course, but most good ones do not incur the majority of kernel overhead until the memory is first used.
直到你用malloc(...)分配的地址写一个字节,内核只会“保留”地址范围。这当然取决于内存分配器和操作系统的实现,但是在首次使用内存之前,大多数好的内核都不会产生大部分内核开销。
The hoard allocator is one big offender that comes to mind immediately, through extensive testing I have found it almost never takes advantage of a kernel that supports late paging. You can always mitigate the effects of late paging in any allocator if you zero-fill the entire memory range immediately after allocation.
囤积分配器是一个立即想到的大罪犯,通过广泛的测试,我发现它几乎从不利用支持延迟分页的内核。如果在分配后立即零填充整个内存范围,则可以始终减轻任何分配器中延迟分页的影响。
Real-time operating systems like VxWorks will never allow this behavior because late paging introduces serious latency. Technically, all it does is put the latency off until a later indeterminate time.
像VxWorks这样的实时操作系统永远不会允许这种行为,因为后期分页会引入严重的延迟。从技术上讲,它所做的就是延迟延迟,直到不确定的时间。
For a more detailed discussion, you may be interested to see how IBM's AIX operating system handles page allocation and overcommitment.
有关更详细的讨论,您可能有兴趣了解IBM的AIX操作系统如何处理页面分配和过度使用。
#4
3
This is a result of what Basile mentioned, over commit memory. However, the explanation kind of interesting.
这是Basile提到的结果,而不是提交内存。但是,解释有点有趣。
Basically when you attempt to map additional memory in Linux (POSIX?), the kernel will just reserve it, and will only actually end up using it if your application accesses one of the reserved pages. This allows multiple applications to reserve more than the actual total amount of ram / swap.
基本上,当您尝试在Linux(POSIX?)中映射其他内存时,内核将保留它,并且只有在您的应用程序访问其中一个保留页面时才会实际使用它。这允许多个应用程序保留超过实际的ram / swap总量。
This is desirable behavior on most Linux environments unless you've got a real-time OS or something where you know exactly who will need what resources, when and why.
这在大多数Linux环境中都是理想的行为,除非你有一个实时操作系统或者你确切知道谁将需要什么资源,何时以及为什么。
Otherwise somebody could come along, malloc up all the ram (without actually doing anything with it) and OOM your apps.
否则有人可能会来,所有的ram(没有实际做任何事情)和OOM你的应用程序。
Another example of this lazy allocation is mmap(), where you have a virtual map that the file you're mapping can fit inside - but you only have a small amount of real memory dedicated to the effort. This allows you to mmap() huge files (larger than your available RAM), and use them like normal file handles which is nifty)
这个延迟分配的另一个例子是mmap(),你有一个虚拟地图,你正在映射的文件可以放在里面 - 但你只有少量的实际内存专用于工作。这允许你mmap()巨大的文件(大于你的可用内存),并像普通的文件句柄一样使用它们很漂亮)
-n
#5
3
Initializing / working with the memory should work:
初始化/使用内存应该工作:
memset(m, 0, bytes);
Also you could use calloc that not only allocates memory but also fills it with zeros for you:
你也可以使用calloc,它不仅可以分配内存,还可以为你填充零:
char* m = (char*) calloc(1, bytes);
#6
1
1.Is this a kernel bug?
这是一个内核错误吗?
No.
2.Can someone explain to me why this behavior is allowed?
有人可以向我解释为什么允许这种行为?
There are a few reasons:
有几个原因:
-
Mitigate need to know eventual memory requirement - it's often convenient to have an application be able to an amount of memory that it considers an upper limit on the need it might actually have. For example, if it's preparing some kind of report either of an initial pass just to calculate the eventual size of the report or a realloc() of successively larger areas (with the risk of having to copy) may significantly complicate the code and hurt performance, where-as multiplying some maximum length of each entry by the number of entries could be very quick and easy. If you know virtual memory is relatively plentiful as far as your application's needs are concerned, then making a larger allocation of virtual address space is very cheap.
缓解需要知道最终的内存需求 - 让应用程序能够使用一定量的内存通常很方便,它认为它可能实际需要的上限。例如,如果它正在准备某种类型的报告,或者只是计算报告的最终大小的初始通过,或者是连续更大区域的realloc()(具有必须复制的风险)可能会使代码复杂化并损害性能,其中 - 将每个条目的一些最大长度乘以条目数可以非常快速和容易。如果您知道虚拟内存相对丰富,就您的应用程序的需求而言,那么进行更大的虚拟地址空间分配非常便宜。
-
Sparse data - if you have the virtual address space spare, being able to have a sparse array and use direct indexing, or allocate a hash table with generous capacity() to size() ratio, can lead to a very high performance system. Both work best (in the sense of having low overheads/waste and efficient use of memory caches) when the data element size is a multiple of the memory paging size, or failing that much larger or a small integral fraction thereof.
稀疏数据 - 如果您有虚拟地址空间备用,能够拥有稀疏数组并使用直接索引,或者分配具有大容量()到大小()比率的哈希表,则可以产生非常高性能的系统。当数据元素大小是存储器分页大小的倍数时,两者都工作得最好(在具有低开销/浪费和有效使用存储器高速缓存的意义上),或者使其失败大得多或小的整数部分。
-
Resource sharing - consider an ISP offering a "1 giga-bit per second" connection to 1000 consumers in a building - they know that if all the consumers use it simultaneously they'll get about 1 mega-bit, but rely on their real-world experience that, though people ask for 1 giga-bit and want a good fraction of it at specific times, there's inevitably some lower maximum and much lower average for concurrent usage. The same insight applied to memory allows operating systems to support more applications than they otherwise would, with reasonable average success at satisfying expectations. Much as the shared Internet connection degrades in speed as more users make simultaneous demands, paging from swap memory on disk may kick in and reduce performance. But unlike an internet connection, there's a limit to the swap memory, and if all the apps really do try to use the memory concurrently such that that limit's exceeded, some will start getting signals/interrupts/traps reporting memory exhaustion. Summarily, with this memory overcommit behaviour enabled, simply checking
malloc()/newreturned a non-NULL pointer is not sufficient to guarantee the physical memory is actually available, and the program may still receive a signal later as it attempts to use the memory.资源共享 - 考虑一个ISP向一个建筑物中的1000个消费者提供“每秒1千兆位”的连接 - 他们知道,如果所有消费者同时使用它,他们将获得大约1兆位,但依赖于他们的真实 - 世界经验虽然人们要求1千兆比特并且在特定时间想要它的很大一部分,但不可避免地会有一些较低的最大值和低得多的并发使用率。应用于内存的相同见解允许操作系统支持比其他方式更多的应用程序,并且在满足期望的情况下获得合理的平均成功。由于共享的Internet连接速度随着更多用户同时需求而降低,因此从磁盘上的交换内存进行分页可能会启动并降低性能。但与互联网连接不同的是,交换内存存在限制,并且如果所有应用程序确实尝试同时使用内存以超出限制,则某些应用程序将开始获取信号/中断/陷阱报告内存耗尽。总而言之,启用此内存过量使用行为,只需检查malloc()/ new返回的非NULL指针不足以保证物理内存实际可用,并且程序稍后在尝试使用内存时仍可能会收到信号。