I know there are a ton of numpy memory error topics, so I hope I haven't duplicated anything. I'm trying to create a np array using np.zeros((500000,10000)). This works fine on my Mac with 16G of memory, but on a Linux server with 28G of RAM it fails instantly with Memory Error. I've verified that I'm running the 64 bit version of Ubuntu and Python, and I'm on Numpy 1.9.3. The only difference I noticed between systems (apart from the obvious) is that when running ulimit -a I get:
我知道有大量的numpy内存错误主题，所以我希望我没有重复任何内容。我正在尝试使用np. 0(500000,10000)创建一个np数组。这在内存为16G的Mac上运行得很好，但是在内存为28G的Linux服务器上，它会因为内存错误而立即失败。我已经验证了我运行的是64位版本的Ubuntu和Python，而我的代码是Numpy 1.9.3。我注意到系统之间唯一的区别(除了明显的区别)是当运行ulimit -a时，我得到:
Linux: max locked memory (kbytes, -l) 64
最大锁定内存(kbytes， -l) 64
Mac: max locked memory (kbytes, -l) unlimited
最大锁定内存(kbytes， -l)无限
Could this be the reason I can't run this command? If not, is there some other configuration option I'm missing?
这就是我不能运行这个命令的原因吗?如果没有，是否还缺少其他配置选项?
1 个解决方案
#1
3
My best guess are:
我最好的猜测是:
- The Mac has a swap that allows more allocated memory than the RAM you see.
- Mac有一个交换，允许比您看到的RAM更多的分配内存。
- The Mac does not realise that the array does not fit in memory until the memory is actually used. So the array actually does not fit in memory but you will not know it until you use that memory.
- Mac直到实际使用内存时才意识到数组并不适合内存。所以这个数组实际上不适合内存，但是在你使用那个内存之前你不会知道它。
I base my first guess in the fact that in 64 bit your array will take 500000*10000*8= 40GB of RAM 20GB in 32 bit, and therefore the array does not fit in the memory you have. There may be a swap to account for the missing memory.
我的第一个猜测是，在64位中，您的数组将在32位中占用500000*10000*8= 40GB的RAM 20GB，因此该数组不适合您的内存。可能会有一个交换来解释丢失的内存。
I base my second guess in this link, where it is explained that np.zeros will not allocate actually in memory the zeros until that memory is accessed for the first time. I have tested in my linux (Ubuntu) computer that np.zeros works with increasing arrays until I reach my RAM limit. Then I get a memory error even if it does not actually allocate the memory.
我的第二个猜想是基于这个链接，在这里它被解释为np。0将不会在内存中分配，直到第一次访问内存。我在我的linux (Ubuntu)电脑上测试过这个np。0在增加数组时起作用，直到达到RAM的极限。然后我得到一个内存错误，即使它实际上没有分配内存。
Once you create the matrix (increase the size enough to make it clear the memory usage):
一旦您创建了矩阵(增加足够的大小以使其清晰地显示内存使用):
a = np.zeros((50,10))
You can check the actual memory required by storing a zero in each cell of the matrix:
您可以通过在矩阵的每个单元中存储一个0来检查实际所需的内存:
a[:,:] = 0.0
Or forcing an operation so the memory is accessed and therefore allocated:
或强制执行操作以访问内存并因此分配内存:
a = a + a
Keep track of the memory usage of the computer while performing this check to understand when the memory is allocated.
在执行此检查以了解何时分配内存时，请跟踪计算机的内存使用情况。
#1
3
My best guess are:
我最好的猜测是:
- The Mac has a swap that allows more allocated memory than the RAM you see.
- Mac有一个交换，允许比您看到的RAM更多的分配内存。
- The Mac does not realise that the array does not fit in memory until the memory is actually used. So the array actually does not fit in memory but you will not know it until you use that memory.
- Mac直到实际使用内存时才意识到数组并不适合内存。所以这个数组实际上不适合内存，但是在你使用那个内存之前你不会知道它。
I base my first guess in the fact that in 64 bit your array will take 500000*10000*8= 40GB of RAM 20GB in 32 bit, and therefore the array does not fit in the memory you have. There may be a swap to account for the missing memory.
我的第一个猜测是，在64位中，您的数组将在32位中占用500000*10000*8= 40GB的RAM 20GB，因此该数组不适合您的内存。可能会有一个交换来解释丢失的内存。
I base my second guess in this link, where it is explained that np.zeros will not allocate actually in memory the zeros until that memory is accessed for the first time. I have tested in my linux (Ubuntu) computer that np.zeros works with increasing arrays until I reach my RAM limit. Then I get a memory error even if it does not actually allocate the memory.
我的第二个猜想是基于这个链接，在这里它被解释为np。0将不会在内存中分配，直到第一次访问内存。我在我的linux (Ubuntu)电脑上测试过这个np。0在增加数组时起作用，直到达到RAM的极限。然后我得到一个内存错误，即使它实际上没有分配内存。
Once you create the matrix (increase the size enough to make it clear the memory usage):
一旦您创建了矩阵(增加足够的大小以使其清晰地显示内存使用):
a = np.zeros((50,10))
You can check the actual memory required by storing a zero in each cell of the matrix:
您可以通过在矩阵的每个单元中存储一个0来检查实际所需的内存:
a[:,:] = 0.0
Or forcing an operation so the memory is accessed and therefore allocated:
或强制执行操作以访问内存并因此分配内存:
a = a + a
Keep track of the memory usage of the computer while performing this check to understand when the memory is allocated.
在执行此检查以了解何时分配内存时，请跟踪计算机的内存使用情况。