如何在大型双维数组中存储数据

时间:2021-03-01 15:56:47

I want to allocate memory of 10^9*10^9 in a double dimension array but this is not possible.is their any way out?

我想在双维数组中分配10 ^ 9 * 10 ^ 9的内存,但这是不可能的。他们的出路是什么?

I think vector could be solution to this but i dont know how to do it.

我认为矢量可以解决这个问题,但我不知道该怎么做。

3 个解决方案

#1


7  

You cannot allocate 1018 bytes of memory in any computer today (that's roughly a million terabytes). However, if your data is mostly zeros (ie. is a sparse matrix), then you can use a different kind of data structure to store your data. It all depends on what kind of data you are storing and whether it has any redundant characteristics.

你今天不能在任何计算机上分配1018字节的内存(大约是一百万兆字节)。但是,如果您的数据大多为零(即,是稀疏矩阵),那么您可以使用不同类型的数据结构来存储数据。这完全取决于您存储的数据类型以及是否具有任何冗余特征。

#2


3  

Assuming that the number of non-zero elements is much less than 10^18, you'll want to read up on sparse arrays. In fact, it's not even a requirement that most of the elements in a sparse array be zero -- they just need to be the same. The essential idea is to keep the non-default values in a structure like a list; any values not found in the list are assumed to be the default value.

假设非零元素的数量远小于10 ^ 18,您将需要读取稀疏数组。实际上,甚至不要求稀疏数组中的大多数元素为零 - 它们只需要相同。基本思想是将非默认值保存在类似列表的结构中;列表中未找到的任何值都被假定为默认值。

#3


0  

I want to allocate memory of 10^9*10^9 in a double dimension array but this is not possible.is their any way out?

我想在双维数组中分配10 ^ 9 * 10 ^ 9的内存,但这是不可能的。他们的出路是什么?

That's way beyond current hardware capabilities, and array this big is unsuitable for any practical purpose (you're free to calculate how many thousands of years it would take to walk through every element).

这超出了当前的硬件功能,而且这个大的数组不适用于任何实际目的(你可以*计算走过每个元素需要几千年的时间)。

You need to create "sparse" array. Store only non-zero elements in memory, provide array-like interface to access them, but internally store them in something like std::map<std::pair<xcoord, ycoord>, value>, return zero for all elements not in map. As long as you don't do something reckless like trying to set every element to non-zero value, this should be sufficient array replacement.

您需要创建“稀疏”数组。只在内存中存储非零元素,提供类似数组的接口来访问它们,但在内部存储它们的内容类似于std :: map ,value>,对于所有不在的元素都返回0地图。只要你不做任何鲁莽的事情,比如试图将每个元素设置为非零值,这应该是足够的数组替换。

so....

所以....

What do you need that much memory for?

你需要那么多记忆?

#1


7  

You cannot allocate 1018 bytes of memory in any computer today (that's roughly a million terabytes). However, if your data is mostly zeros (ie. is a sparse matrix), then you can use a different kind of data structure to store your data. It all depends on what kind of data you are storing and whether it has any redundant characteristics.

你今天不能在任何计算机上分配1018字节的内存(大约是一百万兆字节)。但是,如果您的数据大多为零(即,是稀疏矩阵),那么您可以使用不同类型的数据结构来存储数据。这完全取决于您存储的数据类型以及是否具有任何冗余特征。

#2


3  

Assuming that the number of non-zero elements is much less than 10^18, you'll want to read up on sparse arrays. In fact, it's not even a requirement that most of the elements in a sparse array be zero -- they just need to be the same. The essential idea is to keep the non-default values in a structure like a list; any values not found in the list are assumed to be the default value.

假设非零元素的数量远小于10 ^ 18,您将需要读取稀疏数组。实际上,甚至不要求稀疏数组中的大多数元素为零 - 它们只需要相同。基本思想是将非默认值保存在类似列表的结构中;列表中未找到的任何值都被假定为默认值。

#3


0  

I want to allocate memory of 10^9*10^9 in a double dimension array but this is not possible.is their any way out?

我想在双维数组中分配10 ^ 9 * 10 ^ 9的内存,但这是不可能的。他们的出路是什么?

That's way beyond current hardware capabilities, and array this big is unsuitable for any practical purpose (you're free to calculate how many thousands of years it would take to walk through every element).

这超出了当前的硬件功能,而且这个大的数组不适用于任何实际目的(你可以*计算走过每个元素需要几千年的时间)。

You need to create "sparse" array. Store only non-zero elements in memory, provide array-like interface to access them, but internally store them in something like std::map<std::pair<xcoord, ycoord>, value>, return zero for all elements not in map. As long as you don't do something reckless like trying to set every element to non-zero value, this should be sufficient array replacement.

您需要创建“稀疏”数组。只在内存中存储非零元素,提供类似数组的接口来访问它们,但在内部存储它们的内容类似于std :: map ,value>,对于所有不在的元素都返回0地图。只要你不做任何鲁莽的事情,比如试图将每个元素设置为非零值,这应该是足够的数组替换。

so....

所以....

What do you need that much memory for?

你需要那么多记忆?