Linux:由于虚拟内存限制，不能在一个进程中分配超过32 GB/64 GB的内存

I have a computer with 128 GB of RAM, running Linux (3.19.5-200.fc21.x86_64). However, I cannot allocate more than ~30 GB of RAM in a single process. Beyond this, malloc fails:

我有一台128gb内存的电脑，运行Linux (3.19.5-200.fc21.x86_64)。但是，我不能在一个进程中分配超过30 GB的RAM。除此之外,malloc失败:

#include <stdlib.h>
#include <iostream>

int main()
{
   size_t gb_in_bytes = size_t(1)<<size_t(30); // 1 GB in bytes (2^30).
   // try to allocate 1 block of 'i' GB.
   for (size_t i = 25; i < 35; ++ i) {
      size_t n = i * gb_in_bytes;
      void *p = ::malloc(n);
      std::cout << "allocation of 1 x " << (n/double(gb_in_bytes)) << " GB of data. Ok? " << ((p==0)? "nope" : "yes") << std::endl;
      ::free(p);
   }
}

This produces the following output:

这将产生以下输出:

/tmp> c++ mem_alloc.cpp && a.out

allocation of 1 x 25 GB of data. Ok? yes
allocation of 1 x 26 GB of data. Ok? yes
allocation of 1 x 27 GB of data. Ok? yes
allocation of 1 x 28 GB of data. Ok? yes
allocation of 1 x 29 GB of data. Ok? yes
allocation of 1 x 30 GB of data. Ok? yes
allocation of 1 x 31 GB of data. Ok? nope
allocation of 1 x 32 GB of data. Ok? nope
allocation of 1 x 33 GB of data. Ok? nope
allocation of 1 x 34 GB of data. Ok? nope

I searched for quite some time, and found that this is related to the maximum virtual memory size:

我搜索了很长时间，发现这与最大的虚拟内存大小有关:

~> ulimit -all
[...]
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
virtual memory          (kbytes, -v) 32505856
[...]

I can increase this limit to ~64 GB via ulimit -v 64000000, but not further. Beyond this, I get operation not permitted errors:

我可以通过ulimit - v64000000将这个限制增加到~ 64gb，但不能进一步。除此之外，我得到操作不允许的错误:

~> ulimit -v 64000000
~> ulimit -v 65000000                                                                                                                                  
bash: ulimit: virtual memory: cannot modify limit: Operation not permitted                                                                              
~> ulimit -v unlimited
bash: ulimit: virtual memory: cannot modify limit: Operation not permitted 

Some more searching revealed that in principle it should be possible to set these limits via the "as" (address space) entry in /etc/security/limits.conf. However, by doing this, I could only reduce the maximum amount of virtual memory, not increase it.

更多的搜索表明，原则上可以通过/etc/ security/limitf .conf中的“as”(地址空间)条目设置这些限制。然而，通过这样做，我只能减少虚拟内存的最大数量，而不是增加它。

Is there any way to either lift this limit of virtual memory per process completely, or to increase it beyond 64 GB? I would like to use all of the physical memory in a single application.

有什么方法可以完全取消每个进程的虚拟内存限制，或者将其增加到超过64 GB吗?我希望在一个应用程序中使用所有的物理内存。

EDIT:

编辑:

• Following Ingo Leonhardt, I tried ulimits -v unlimited after logging in as root, not as standard user. Doing this solves the problem for root (the program can then allocate all the physical memory while logged in as root). But this works only for root, not for other users. However, at the least this means that in principle the kernel can handle this just fine, and that there is only a configuration problem.

  继Ingo Leonhardt之后，我以root用户身份登录，而不是作为标准用户。这样做可以解决root的问题(程序可以在以root身份登录时分配所有物理内存)。但这只适用于root用户，不适用于其他用户。但是，至少这意味着原则上内核可以很好地处理这个问题，并且只有一个配置问题。

• Regarding limits.conf: I tried explicitly adding

  关于限制。conf:我尝试了显式加法

  • hard as unlimited
  • 无限的努力
  • soft as unlimited
  • 软是无限的

  to /etc/security/limits.conf, and rebooting. This had no effect. After login as standard user, ulimit -v still returns about 32 GB, and ulimit -v 65000000 still says permission denied (while ulimit -v 64000000 works). The rest of limits.conf is commented out, and in /etc/security/limits.d there is only one other, unrelated entry (limiting nproc to 4096 for non-root users). That is, the virtual memory limit must be coming from elsewhere than limits.conf. Any ideas what else could lead to ulimits -v not being "unlimited"?

  对/etc/security/limits.配置,重新启动。这没有影响。作为标准用户登录后，ulimit -v仍然返回约32gb, ulimit -v 65000000仍然表示拒绝访问权限(ulimit -v 64000000有效)。其余的限制。conf被注释掉，在/etc/security/limits中d只有另一个不相关的条目(非根用户将nproc限制为4096)。也就是说，虚拟内存限制必须来自其他地方，而不是限制。有什么想法会导致ulimits -v不是“无限的”吗?

EDIT/RESOLUTION:

编辑/决议:

• It was caused by my own stupidity. I had a (long forgotten) program in my user setup which used setrlimit to restrict the amount of memory per process to prevent Linux from swapping to death. It was unintentionally copied from a 32 GB machine to the 128 GB machine. Thanks to Paul and Andrew Janke and everyone else for helping to track it down. Sorry everyone :/.

  这是我自己的愚蠢造成的。我的用户设置中有一个(早就被遗忘的)程序，它使用setrlimit限制每个进程的内存数量，以防止Linux交换到死。它无意中从32gb的机器复制到128gb的机器。感谢Paul和Andrew Janke以及其他所有人帮助我们找到了它。对不起每个人:/。

• If anyone else encounters this: Search for ulimit/setrlimit in the bash and profile settings, and programs potentially calling those (both your own and the system-wide /etc settings) and make sure that /security/limits.conf does not include this limit... (or at least try creating a new user, to see if this happens in your user or the system setup)

  如果其他人遇到这种情况:在bash和概要设置中搜索ulimit/setrlimit，以及可能调用这些设置的程序(包括您自己的和系统范围内的/etc设置)，并确保/security/limit。conf不包括这个限制…(或者至少尝试创建一个新用户，看看是否在用户或系统设置中发生这种情况)

1 个解决方案

#1

This is a ulimit and system setup problem, not a c++ problem.

这是一个ulimit和系统设置问题，而不是c++问题。

I can run your appropriately modified code on an Amazon EC2 instance type r3.4xlarge with no problem. These cost less than $0.20/hour on the spot market, and so I suggest you rent one, and perhaps take a look around in /etc and compare to your own setup... or maybe you need to recompile a Linux kernel to use that much memory... but it is not a C++ or gcc problem.

我可以在Amazon EC2实例类型r3.4xlarge上运行您适当修改的代码，没有问题。这些东西在现货市场上每小时的价格还不到0.20美元，所以我建议你租一个，或者看看周围的环境，和你自己的设置做个比较……或者你需要重新编译一个Linux内核来使用这么多的内存……但这不是c++或gcc的问题。

Ubuntu on the EC2 machine was already set up for unlimited process memory.

EC2机器上的Ubuntu已经设置为无限制的进程内存。

$ sudo su
# ulimit -u
--> unlimited

This one has 125GB of ram

这个有125GB的内存

# free
             total       used       free     shared    buffers     cached
Mem:     125903992    1371828  124532164        344      22156     502248
-/+ buffers/cache:     847424  125056568
Swap:            0          0          0

I modified the limits on your program to go up to 149GB.

我将您的程序的限制修改为149GB。

Here's the output. Looks good up to 118GB.

这是输出。看起来好到118GB。

root@ip-10-203-193-204:/home/ubuntu# ./memtest
allocation of 1 x 25 GB of data. Ok? yes
allocation of 1 x 26 GB of data. Ok? yes
allocation of 1 x 27 GB of data. Ok? yes
allocation of 1 x 28 GB of data. Ok? yes
allocation of 1 x 29 GB of data. Ok? yes
allocation of 1 x 30 GB of data. Ok? yes
allocation of 1 x 31 GB of data. Ok? yes
allocation of 1 x 32 GB of data. Ok? yes
allocation of 1 x 33 GB of data. Ok? yes
allocation of 1 x 34 GB of data. Ok? yes
allocation of 1 x 35 GB of data. Ok? yes
allocation of 1 x 36 GB of data. Ok? yes
allocation of 1 x 37 GB of data. Ok? yes
allocation of 1 x 38 GB of data. Ok? yes
allocation of 1 x 39 GB of data. Ok? yes
allocation of 1 x 40 GB of data. Ok? yes
allocation of 1 x 41 GB of data. Ok? yes
allocation of 1 x 42 GB of data. Ok? yes
allocation of 1 x 43 GB of data. Ok? yes
allocation of 1 x 44 GB of data. Ok? yes
allocation of 1 x 45 GB of data. Ok? yes
allocation of 1 x 46 GB of data. Ok? yes
allocation of 1 x 47 GB of data. Ok? yes
allocation of 1 x 48 GB of data. Ok? yes
allocation of 1 x 49 GB of data. Ok? yes
allocation of 1 x 50 GB of data. Ok? yes
allocation of 1 x 51 GB of data. Ok? yes
allocation of 1 x 52 GB of data. Ok? yes
allocation of 1 x 53 GB of data. Ok? yes
allocation of 1 x 54 GB of data. Ok? yes
allocation of 1 x 55 GB of data. Ok? yes
allocation of 1 x 56 GB of data. Ok? yes
allocation of 1 x 57 GB of data. Ok? yes
allocation of 1 x 58 GB of data. Ok? yes
allocation of 1 x 59 GB of data. Ok? yes
allocation of 1 x 60 GB of data. Ok? yes
allocation of 1 x 61 GB of data. Ok? yes
allocation of 1 x 62 GB of data. Ok? yes
allocation of 1 x 63 GB of data. Ok? yes
allocation of 1 x 64 GB of data. Ok? yes
allocation of 1 x 65 GB of data. Ok? yes
allocation of 1 x 66 GB of data. Ok? yes
allocation of 1 x 67 GB of data. Ok? yes
allocation of 1 x 68 GB of data. Ok? yes
allocation of 1 x 69 GB of data. Ok? yes
allocation of 1 x 70 GB of data. Ok? yes
allocation of 1 x 71 GB of data. Ok? yes
allocation of 1 x 72 GB of data. Ok? yes
allocation of 1 x 73 GB of data. Ok? yes
allocation of 1 x 74 GB of data. Ok? yes
allocation of 1 x 75 GB of data. Ok? yes
allocation of 1 x 76 GB of data. Ok? yes
allocation of 1 x 77 GB of data. Ok? yes
allocation of 1 x 78 GB of data. Ok? yes
allocation of 1 x 79 GB of data. Ok? yes
allocation of 1 x 80 GB of data. Ok? yes
allocation of 1 x 81 GB of data. Ok? yes
allocation of 1 x 82 GB of data. Ok? yes
allocation of 1 x 83 GB of data. Ok? yes
allocation of 1 x 84 GB of data. Ok? yes
allocation of 1 x 85 GB of data. Ok? yes
allocation of 1 x 86 GB of data. Ok? yes
allocation of 1 x 87 GB of data. Ok? yes
allocation of 1 x 88 GB of data. Ok? yes
allocation of 1 x 89 GB of data. Ok? yes
allocation of 1 x 90 GB of data. Ok? yes
allocation of 1 x 91 GB of data. Ok? yes
allocation of 1 x 92 GB of data. Ok? yes
allocation of 1 x 93 GB of data. Ok? yes
allocation of 1 x 94 GB of data. Ok? yes
allocation of 1 x 95 GB of data. Ok? yes
allocation of 1 x 96 GB of data. Ok? yes
allocation of 1 x 97 GB of data. Ok? yes
allocation of 1 x 98 GB of data. Ok? yes
allocation of 1 x 99 GB of data. Ok? yes
allocation of 1 x 100 GB of data. Ok? yes
allocation of 1 x 101 GB of data. Ok? yes
allocation of 1 x 102 GB of data. Ok? yes
allocation of 1 x 103 GB of data. Ok? yes
allocation of 1 x 104 GB of data. Ok? yes
allocation of 1 x 105 GB of data. Ok? yes
allocation of 1 x 106 GB of data. Ok? yes
allocation of 1 x 107 GB of data. Ok? yes
allocation of 1 x 108 GB of data. Ok? yes
allocation of 1 x 109 GB of data. Ok? yes
allocation of 1 x 110 GB of data. Ok? yes
allocation of 1 x 111 GB of data. Ok? yes
allocation of 1 x 112 GB of data. Ok? yes
allocation of 1 x 113 GB of data. Ok? yes
allocation of 1 x 114 GB of data. Ok? yes
allocation of 1 x 115 GB of data. Ok? yes
allocation of 1 x 116 GB of data. Ok? yes
allocation of 1 x 117 GB of data. Ok? yes
allocation of 1 x 118 GB of data. Ok? yes
allocation of 1 x 119 GB of data. Ok? nope
allocation of 1 x 120 GB of data. Ok? nope
allocation of 1 x 121 GB of data. Ok? nope
allocation of 1 x 122 GB of data. Ok? nope
allocation of 1 x 123 GB of data. Ok? nope
allocation of 1 x 124 GB of data. Ok? nope
allocation of 1 x 125 GB of data. Ok? nope
allocation of 1 x 126 GB of data. Ok? nope
allocation of 1 x 127 GB of data. Ok? nope
allocation of 1 x 128 GB of data. Ok? nope
allocation of 1 x 129 GB of data. Ok? nope
allocation of 1 x 130 GB of data. Ok? nope
allocation of 1 x 131 GB of data. Ok? nope
allocation of 1 x 132 GB of data. Ok? nope
allocation of 1 x 133 GB of data. Ok? nope
allocation of 1 x 134 GB of data. Ok? nope
allocation of 1 x 135 GB of data. Ok? nope
allocation of 1 x 136 GB of data. Ok? nope
allocation of 1 x 137 GB of data. Ok? nope
allocation of 1 x 138 GB of data. Ok? nope
allocation of 1 x 139 GB of data. Ok? nope
allocation of 1 x 140 GB of data. Ok? nope
allocation of 1 x 141 GB of data. Ok? nope
allocation of 1 x 142 GB of data. Ok? nope
allocation of 1 x 143 GB of data. Ok? nope
allocation of 1 x 144 GB of data. Ok? nope
allocation of 1 x 145 GB of data. Ok? nope
allocation of 1 x 146 GB of data. Ok? nope
allocation of 1 x 147 GB of data. Ok? nope
allocation of 1 x 148 GB of data. Ok? nope
allocation of 1 x 149 GB of data. Ok? nope

Now, about that US$0.17 I spent on this...

我花了0。17美元