I have a malloc in C that is 26901^2*sizeof(double)
我有malloc C 26901 ^ 2 * sizeof(双)
This got me thinking what the largest value can be here?
这让我想到这里最大的价值是什么?
Also, would I have any problems defining a macro to access this 2D array?
另外，如果定义一个宏来访问这个2D数组，我还有什么问题吗?
#define DN(i,j) ((int)i * ny + (int)j)
Because this seems to not be working for me - or I am at least unsure it is. I can't figure out how to make totalview dive on a macro to tell me what A[DN(indx,jndx)] is actually looking at.
因为这似乎对我不起作用——或者至少我不确定。我不知道如何让totalview深入到宏上，告诉我a [DN(indx,jndx)]实际上在看什么。
6 个解决方案
#1
36
Observations
Assuming a typical allocator, such as the one glibc uses, there are some observations:
假设有一个典型的分配器，比如glibc使用的那个，这里有一些观察:
- Whether or not the memory is actually used, the region must be reserved contiguously in virtual memory.
- 无论是否实际使用该内存，该区域必须在虚拟内存中连续地保留。
- The largest free contiguous regions depends on the memory usage of existing memory regions, and the availability of those regions to
malloc. - 最大的空闲连续区域取决于现有内存区域的使用情况，以及这些区域对malloc的可用性。
- The mapping practices depend on the architecture and OS. Furthermore underlying system calls to obtain memory regions are affected by these practices (such as
malloccalling through tommapto acquire pages). - 映射实践依赖于体系结构和操作系统。此外，获取内存区域的底层系统调用也受到这些实践的影响(例如通过mmap调用malloc来获取页面)。
Experiment
Here's a simple program to allocate the largest possible block (compile with gcc largest_malloc_size.c -Wall -O2:
这里有一个简单的程序来分配最大可能的块(用gcc largest_malloc_size编译)。c - wall - 02:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
static void *malloc_wrap(size_t size)
{
void *p = malloc(size);
if (p) {
printf("Allocated %zu bytes from %p to %p\n", size, p, p + size);
}
else {
printf("Failed to allocated %zu bytes\n", size);
}
return p;
}
int main()
{
size_t step = 0x1000000;
size_t size = step;
size_t best = 0;
while (step > 0)
{
void *p = malloc_wrap(size);
if (p) {
free(p);
best = size;
}
else {
step /= 0x10;
}
size += step;
}
void *p = malloc_wrap(best);
if (p) {
pause();
return 0;
}
else {
return 1;
}
}
Running the above program (./a.out) on my Linux stanley 2.6.32-24-generic-pae #39-Ubuntu SMP Wed Jul 28 07:39:26 UTC 2010 i686 GNU/Linux machine obtains this result:
在我的Linux stanley 2.6.32-24-通用-pae #39-Ubuntu SMP上运行上述程序(./a.out)，得到以下结果:
<snip>
Allocated 2919235584 bytes from 0x9763008 to 0xb7763008
Allocated 2936012800 bytes from 0x8763008 to 0xb7763008
Failed to allocated 2952790016 bytes
Failed to allocated 2953838592 bytes
Failed to allocated 2953904128 bytes
Failed to allocated 2953908224 bytes
Allocated 2936012800 bytes from 0x85ff008 to 0xb75ff008
This is an allocation of exactly 2800MiB. Observing the relevant mapping from /proc/[number]/maps:
这是精确的2800MiB的分配。观察/proc/[number]/maps的相关映射:
<snip>
0804a000-0804b000 rw-p 00001000 08:07 3413394 /home/matt/anacrolix/public/*/a.out
085ff000-b7600000 rw-p 00000000 00:00 0 [heap]
b7600000-b7621000 rw-p 00000000 00:00 0
b7621000-b7700000 ---p 00000000 00:00 0
b7764000-b7765000 rw-p 00000000 00:00 0
b7765000-b78b8000 r-xp 00000000 08:08 916041 /lib/tls/i686/cmov/libc-2.11.1.so
<snip>
bfc07000-bfc1c000 rw-p 00000000 00:00 0 [stack]
Conclusion
It appears the heap has been expanded in the area between the program data and code, and the shared library mappings, which sit snug against the user/kernel memory space boundary (obviously 3G/1G on this system).
看来堆已经扩展到程序数据和代码之间的区域，以及共享库映射，它们与用户/内核内存空间边界保持一致(显然是这个系统的3G/1G)。
This result suggests that the maximum allocatable space using malloc is roughly equal to:
这一结果表明，使用malloc的最大可分配空间大致等于:
- The user space region (3GB in the example)
- 用户空间区域(本例中为3GB)
- Less the offset to the start of the heap (program code and data)
- 减少堆开始时的偏移量(程序代码和数据)
- Less the space reserved for the main thread stack
- 减少主线程堆栈的空间
- Less the space occupied by all the mapped in shared libraries
- 减少共享库中所有映射所占用的空间
- Finally, the largest contiguous region that can be found by the underlying system call within the region available to the heap (which may be fragmented by other mappings)
- 最后，可以通过底层系统调用在堆可用区域内找到的最大连续区域(可能被其他映射分割)
Notes
With respect to glibc and Linux implementations, the following manual snippets are of great interest:
关于glibc和Linux实现，以下手动代码片段非常有趣:
malloc
Normally, malloc() allocates memory from the heap, and adjusts the size
of the heap as required, using sbrk(2). When allocating blocks of mem‐
ory larger than MMAP_THRESHOLD bytes, the glibc malloc() implementation
allocates the memory as a private anonymous mapping using mmap(2).
MMAP_THRESHOLD is 128 kB by default, but is adjustable using mal‐
lopt(3).
mmap
MAP_ANONYMOUS
The mapping is not backed by any file; its contents are initial‐
ized to zero.
Afterword
This test was done on a x86 kernel. I'd expect similar results from a x86_64 kernel, albeit with vastly larger memory regions returned. Other operating systems may vary in their placement of mappings, and the handling of large mallocs, so results could be quite considerably different.
这个测试是在x86内核上完成的。我预计x86_64内核也会产生类似的结果，尽管返回的内存区域要大得多。其他操作系统在映射的位置和对大型mallocs的处理上可能会有所不同，因此结果可能会大不相同。
#2
9
That depends on your malloc implementation!
这取决于您的malloc实现!
According to Wikipedia, "Since the v2.3 release, the GNU C library (glibc) uses a modified ptmalloc2, which itself is based on dlmalloc v2.7.0." dlmalloc refers to Doug Lea's malloc implementation. The important thing to note in this implementation is that large mallocs are accomplished through the operating system's memory mapped file functionality, so these blocks can be quite large indeed without many problems of finding a contiguous block.
根据*，“自从v2.3发布以来，GNU C库(glibc)使用了一个修改过的ptmalloc2，它本身基于dlmalloc v2.7.0。”在此实现中需要注意的重要一点是，大型的mallocs是通过操作系统的内存映射文件功能实现的，因此这些块可以非常大，而不存在查找连续块的许多问题。
#3
7
The malloc question is answered (depends on OS, which you don't specify), so about that define:
malloc问题得到了回答(取决于OS，您没有具体说明)，所以关于这个定义:
#define DN(i,j) ((int)i * ny + (int)j)
is not quite safe, for someone might do DN(a+b,c) which expands to
不是很安全，因为有些人可能会做DN(a+b,c)扩展到
((int)a+b * ny + (int)c)
which is probably not what you wanted. So put a lot of parentheses in there:
这可能不是你想要的。在这里放很多括号
#define DN(i,j) ((int)(i) * ny + (int)(j))
to see what DN(indx,jndx) points to, just printf("%d\n",DN(indx,jndx));
要查看DN(indx,jndx)指向什么，只需printf(“%d\n”，DN(indx,jndx);
#4
1
The size parameter in a call to malloc is of type size_t, which varies by implementation. See this question for more.
对malloc的调用中的size参数是size_t类型，它随实现而变化。请更多地了解这个问题。
#5
1
This got me thinking what the largest value can be here?
这让我想到这里最大的价值是什么?
26'901^2 = 723'663'801. If your double is 8 bytes, then it is less than 8GB. I see totally no problem allocating that much of memory and my apps routinely allocate (on 64 bit systems) much more. (Biggest memory consumption I have ever seen was 420GB (on Solaris 10 numa system with 640GB RAM) with largest continuous block of ~24GB.)
26 ' 901 ^ 2 = 723 ' 663 ' 801。如果双字节是8字节，那么它小于8GB。我完全没有发现分配内存的问题，而且我的应用程序经常(在64位系统上)分配更多的内存。(我所见过的最大内存消耗是420GB(在Solaris 10 numa系统中，内存为640GB)，最大的连续块为~24GB。)
Largest value is hard to identify since it is platform dependent: similar to the 32bit systems it depends on user-space / kernel-space split. As things stand at the moment, I think one would first come to the limit of the actual physical RAM - before reaching the limit of what libc can allocate. (And kernel doesn't care, it just expands virtual memory often without even considering whether there is sufficient RAM to pin it to.)
最大的值很难识别，因为它依赖于平台:类似于它依赖于用户空间/内核空间分割的32位系统。就目前的情况而言，我认为一个人首先会达到实际物理RAM的极限——在达到libc所能分配的极限之前。(内核并不介意，它只是经常扩展虚拟内存，甚至不考虑是否有足够的RAM来固定它。)
#6
1
The largest memory block you can ask malloc() for is the largest size_t value - this is SIZE_MAX from <limits.h>. The largest amount you can sucessfully request is obviously dependent on the operating system and the configuration of the individual machine.
您可以向malloc()请求的最大内存块是最大的size_t值——这是
Your macro is not safe. It performs the index calculation with an int variable, which is only required to have a range up to 32767. Any value higher than this can cause signed overflow, which results in undefined behaviour. You are probably best off doing the calculation as a size_t, since that type must be able to hold any valid array index:
你的宏不安全。它使用int变量执行索引计算，该变量的范围仅需要达到32767。任何高于此值的值都可能导致有符号溢出，从而导致未定义的行为。您最好使用size_t进行计算，因为该类型必须能够保存任何有效的数组索引:
#define DN(i, j) ((size_t)(i) * ny + (size_t)(j))
(Although note that if you supply negative values for i or j, you'll get an index far out of bounds).
(尽管要注意，如果为i或j提供负值，则索引将远远超出界限)。
#1
36
Observations
Assuming a typical allocator, such as the one glibc uses, there are some observations:
假设有一个典型的分配器，比如glibc使用的那个，这里有一些观察:
- Whether or not the memory is actually used, the region must be reserved contiguously in virtual memory.
- 无论是否实际使用该内存，该区域必须在虚拟内存中连续地保留。
- The largest free contiguous regions depends on the memory usage of existing memory regions, and the availability of those regions to
malloc. - 最大的空闲连续区域取决于现有内存区域的使用情况，以及这些区域对malloc的可用性。
- The mapping practices depend on the architecture and OS. Furthermore underlying system calls to obtain memory regions are affected by these practices (such as
malloccalling through tommapto acquire pages). - 映射实践依赖于体系结构和操作系统。此外，获取内存区域的底层系统调用也受到这些实践的影响(例如通过mmap调用malloc来获取页面)。
Experiment
Here's a simple program to allocate the largest possible block (compile with gcc largest_malloc_size.c -Wall -O2:
这里有一个简单的程序来分配最大可能的块(用gcc largest_malloc_size编译)。c - wall - 02:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
static void *malloc_wrap(size_t size)
{
void *p = malloc(size);
if (p) {
printf("Allocated %zu bytes from %p to %p\n", size, p, p + size);
}
else {
printf("Failed to allocated %zu bytes\n", size);
}
return p;
}
int main()
{
size_t step = 0x1000000;
size_t size = step;
size_t best = 0;
while (step > 0)
{
void *p = malloc_wrap(size);
if (p) {
free(p);
best = size;
}
else {
step /= 0x10;
}
size += step;
}
void *p = malloc_wrap(best);
if (p) {
pause();
return 0;
}
else {
return 1;
}
}
Running the above program (./a.out) on my Linux stanley 2.6.32-24-generic-pae #39-Ubuntu SMP Wed Jul 28 07:39:26 UTC 2010 i686 GNU/Linux machine obtains this result:
在我的Linux stanley 2.6.32-24-通用-pae #39-Ubuntu SMP上运行上述程序(./a.out)，得到以下结果:
<snip>
Allocated 2919235584 bytes from 0x9763008 to 0xb7763008
Allocated 2936012800 bytes from 0x8763008 to 0xb7763008
Failed to allocated 2952790016 bytes
Failed to allocated 2953838592 bytes
Failed to allocated 2953904128 bytes
Failed to allocated 2953908224 bytes
Allocated 2936012800 bytes from 0x85ff008 to 0xb75ff008
This is an allocation of exactly 2800MiB. Observing the relevant mapping from /proc/[number]/maps:
这是精确的2800MiB的分配。观察/proc/[number]/maps的相关映射:
<snip>
0804a000-0804b000 rw-p 00001000 08:07 3413394 /home/matt/anacrolix/public/*/a.out
085ff000-b7600000 rw-p 00000000 00:00 0 [heap]
b7600000-b7621000 rw-p 00000000 00:00 0
b7621000-b7700000 ---p 00000000 00:00 0
b7764000-b7765000 rw-p 00000000 00:00 0
b7765000-b78b8000 r-xp 00000000 08:08 916041 /lib/tls/i686/cmov/libc-2.11.1.so
<snip>
bfc07000-bfc1c000 rw-p 00000000 00:00 0 [stack]
Conclusion
It appears the heap has been expanded in the area between the program data and code, and the shared library mappings, which sit snug against the user/kernel memory space boundary (obviously 3G/1G on this system).
看来堆已经扩展到程序数据和代码之间的区域，以及共享库映射，它们与用户/内核内存空间边界保持一致(显然是这个系统的3G/1G)。
This result suggests that the maximum allocatable space using malloc is roughly equal to:
这一结果表明，使用malloc的最大可分配空间大致等于:
- The user space region (3GB in the example)
- 用户空间区域(本例中为3GB)
- Less the offset to the start of the heap (program code and data)
- 减少堆开始时的偏移量(程序代码和数据)
- Less the space reserved for the main thread stack
- 减少主线程堆栈的空间
- Less the space occupied by all the mapped in shared libraries
- 减少共享库中所有映射所占用的空间
- Finally, the largest contiguous region that can be found by the underlying system call within the region available to the heap (which may be fragmented by other mappings)
- 最后，可以通过底层系统调用在堆可用区域内找到的最大连续区域(可能被其他映射分割)
Notes
With respect to glibc and Linux implementations, the following manual snippets are of great interest:
关于glibc和Linux实现，以下手动代码片段非常有趣:
malloc
Normally, malloc() allocates memory from the heap, and adjusts the size
of the heap as required, using sbrk(2). When allocating blocks of mem‐
ory larger than MMAP_THRESHOLD bytes, the glibc malloc() implementation
allocates the memory as a private anonymous mapping using mmap(2).
MMAP_THRESHOLD is 128 kB by default, but is adjustable using mal‐
lopt(3).
mmap
MAP_ANONYMOUS
The mapping is not backed by any file; its contents are initial‐
ized to zero.
Afterword
This test was done on a x86 kernel. I'd expect similar results from a x86_64 kernel, albeit with vastly larger memory regions returned. Other operating systems may vary in their placement of mappings, and the handling of large mallocs, so results could be quite considerably different.
这个测试是在x86内核上完成的。我预计x86_64内核也会产生类似的结果，尽管返回的内存区域要大得多。其他操作系统在映射的位置和对大型mallocs的处理上可能会有所不同，因此结果可能会大不相同。
#2
9
That depends on your malloc implementation!
这取决于您的malloc实现!
According to Wikipedia, "Since the v2.3 release, the GNU C library (glibc) uses a modified ptmalloc2, which itself is based on dlmalloc v2.7.0." dlmalloc refers to Doug Lea's malloc implementation. The important thing to note in this implementation is that large mallocs are accomplished through the operating system's memory mapped file functionality, so these blocks can be quite large indeed without many problems of finding a contiguous block.
根据*，“自从v2.3发布以来，GNU C库(glibc)使用了一个修改过的ptmalloc2，它本身基于dlmalloc v2.7.0。”在此实现中需要注意的重要一点是，大型的mallocs是通过操作系统的内存映射文件功能实现的，因此这些块可以非常大，而不存在查找连续块的许多问题。
#3
7
The malloc question is answered (depends on OS, which you don't specify), so about that define:
malloc问题得到了回答(取决于OS，您没有具体说明)，所以关于这个定义:
#define DN(i,j) ((int)i * ny + (int)j)
is not quite safe, for someone might do DN(a+b,c) which expands to
不是很安全，因为有些人可能会做DN(a+b,c)扩展到
((int)a+b * ny + (int)c)
which is probably not what you wanted. So put a lot of parentheses in there:
这可能不是你想要的。在这里放很多括号
#define DN(i,j) ((int)(i) * ny + (int)(j))
to see what DN(indx,jndx) points to, just printf("%d\n",DN(indx,jndx));
要查看DN(indx,jndx)指向什么，只需printf(“%d\n”，DN(indx,jndx);
#4
1
The size parameter in a call to malloc is of type size_t, which varies by implementation. See this question for more.
对malloc的调用中的size参数是size_t类型，它随实现而变化。请更多地了解这个问题。
#5
1
This got me thinking what the largest value can be here?
这让我想到这里最大的价值是什么?
26'901^2 = 723'663'801. If your double is 8 bytes, then it is less than 8GB. I see totally no problem allocating that much of memory and my apps routinely allocate (on 64 bit systems) much more. (Biggest memory consumption I have ever seen was 420GB (on Solaris 10 numa system with 640GB RAM) with largest continuous block of ~24GB.)
26 ' 901 ^ 2 = 723 ' 663 ' 801。如果双字节是8字节，那么它小于8GB。我完全没有发现分配内存的问题，而且我的应用程序经常(在64位系统上)分配更多的内存。(我所见过的最大内存消耗是420GB(在Solaris 10 numa系统中，内存为640GB)，最大的连续块为~24GB。)
Largest value is hard to identify since it is platform dependent: similar to the 32bit systems it depends on user-space / kernel-space split. As things stand at the moment, I think one would first come to the limit of the actual physical RAM - before reaching the limit of what libc can allocate. (And kernel doesn't care, it just expands virtual memory often without even considering whether there is sufficient RAM to pin it to.)
最大的值很难识别，因为它依赖于平台:类似于它依赖于用户空间/内核空间分割的32位系统。就目前的情况而言，我认为一个人首先会达到实际物理RAM的极限——在达到libc所能分配的极限之前。(内核并不介意，它只是经常扩展虚拟内存，甚至不考虑是否有足够的RAM来固定它。)
#6
1
The largest memory block you can ask malloc() for is the largest size_t value - this is SIZE_MAX from <limits.h>. The largest amount you can sucessfully request is obviously dependent on the operating system and the configuration of the individual machine.
您可以向malloc()请求的最大内存块是最大的size_t值——这是
Your macro is not safe. It performs the index calculation with an int variable, which is only required to have a range up to 32767. Any value higher than this can cause signed overflow, which results in undefined behaviour. You are probably best off doing the calculation as a size_t, since that type must be able to hold any valid array index:
你的宏不安全。它使用int变量执行索引计算，该变量的范围仅需要达到32767。任何高于此值的值都可能导致有符号溢出，从而导致未定义的行为。您最好使用size_t进行计算，因为该类型必须能够保存任何有效的数组索引:
#define DN(i, j) ((size_t)(i) * ny + (size_t)(j))
(Although note that if you supply negative values for i or j, you'll get an index far out of bounds).
(尽管要注意，如果为i或j提供负值，则索引将远远超出界限)。