Numpy - 从1d数组高效构建2d数组

I am building 2d array from 1d arrays in numpy (Python 2.7). I am looking for most efficient way to do it. Up to date I came up with:

我正在numpy(Python 2.7)中从1d数组构建2d数组。我正在寻找最有效的方法来做到这一点。到目前为止,我提出了:

a=np.ones(100000)

#   SUBSCRIPTING

n_dim=3
x=0

for i in xrange(0,1000):
    x=np.zeros(shape=(100000,n_dim))
    for j in xrange(0,n_dim):
        x[:,j]=a*j

# ~1.574 s/1000 loops - joinind 3 1d arrays
# ~9.162 s/1000 loops - joinind 10 1d arrays



#   STACKING

for i in xrange(0,1000):
    x=a*0.
    for j in xrange(1,n_dim):
        x=np.vstack((x,a*j))
    x=x.T

# ~1.786 s/1000 loops - joinind 3 1d arrays
# ~16.603 s/1000 loops - joinind 10 1d arrays

First method (subscripting) is the fastest I came up with and performance gain over the second method (stacking) grows with number of 1d arrays I am joining. As I need to repeat this step quite a bit, I wonder if there is something faster? I am willing to go with solution that loses in clarity if it offers significant performance boost.

第一种方法(下标)是我提出的最快的方法,第二种方法(堆叠)的性能提升随着我加入的1d数组的数量而增长。由于我需要重复这一步骤,我想知道是否有更快的东西?如果它提供显着的性能提升,我愿意采用明显失去的解决方案。

Maybe I can try stacking arrays in a way that limits number of stacking operations (e.g. case of joining 4 1d arrays: first stack arrays 1 and 2, then arrays 3 and 4, and resulting arrays at the end).

也许我可以尝试以限制堆叠操作次数的方式堆叠阵列(例如,加入4个1d阵列的情况:第一个堆栈阵列1和2,然后是阵列3和4,最后是结果阵列)。

My question is about efficiently building 2d array from 1d arrays. Values in the arrays I am using here are dummy. In the real application most of the values in 1d arrays I am joining will likely differ.

我的问题是从1d数组有效地构建2d数组。我在这里使用的数组中的值是虚拟的。在实际应用中,我加入的1d数组中的大多数值可能会有所不同。

2 个解决方案

#1

Because numpy stores (by default) arrays in row-major order it is more efficient to set the values by rows. Therefore, I would use:

因为numpy存储(默认情况下)以行主顺序排列的数组,所以按行设置值会更有效。因此,我会用:

x=np.zeros(shape=(n_dim, 100000))
for j in range(0,n_dim):
    x[j,:]=a*j

Alternatively, you can define x to be column-major, and then, this is as fast as the previous code:

或者,您可以将x定义为column-major,然后,这与前面的代码一样快:

x=np.zeros(shape=(100000,n_dim), order='F')
for j in range(0,n_dim):
    x[:,j]=a*j

You could also create x with the numpy outer product:

您还可以使用numpy外部产品创建x:

v = np.arange(n_dim)
x = np.outer(v, a)

#2

This is poor way of using vstack; you are calling it repeatedly, creating a new x for each j

这是使用vstack的糟糕方式;你反复调用它,为每个j创建一个新的x

x=a*0.
for j in xrange(1,n_dim):
    x=np.vstack((x,a*j))
x=x.T

The correct way is to build a list of the arrays, and use vstack only once.

正确的方法是构建数组列表,并仅使用vstack一次。

xlist=[]
for j in xrange(1,n_dim):
    xlist.append(a*j)
x = np.array(xlist).T

In this context append works just as well as vstack, and may be faster. There is also a column_stack function. The key difference is that I am taking advantage of the fast list append, and the ability of array (and vstack) to take many items in its argument list.

在这种情况下,append和vstack一样工作,并且可能更快。还有一个column_stack函数。关键的区别在于我正在利用快速列表追加,以及数组(和vstack)在其参数列表中包含许多项的能力。

It's even better if you can write the loop as a list comprehension

如果你可以把循环写成列表理解,那就更好了

x = np.array([a*j for j in xrange(1,n_dim)])

Insertion in a preallocated array is often the fastest choice. But you should be familiar with this build-from-a-list method.

插入预分配的阵列通常是最快的选择。但是你应该熟悉这种从列表构建的方法。

The basic np.array expression

基本的np.array表达式

np.array([[1,2,3],[4,5,6]])

is just this, building 2d from a list of 1d arrays (or in this case lists).

就是这样,从1d数组列表中构建2d(或者在这种情况下列表)。

np.array([a*0,a*1,a*2])

jakub noted that np.array is slow. For n_dim=10:

jakub注意到np.array很慢。对于n_dim = 10:

In [257]: timeit x=np.array([(a*j) for j in range(n_dim)]).T
1 loops, best of 3: 228 ms per loop

In [258]: timeit x=np.array([(a*j).tolist() for j in range(n_dim)]).T
1 loops, best of 3: 228 ms per loop

Apparently np.array is converting the input arrays to lists, and then doing its usual construction from nested lists (or something equivalent).

显然np.array正在将输入数组转换为列表,然后从嵌套列表(或类似的东西)中进行常规构造。

In [259]: timeit x=np.vstack([(a*j) for j in range(n_dim)]).T
10 loops, best of 3: 24.9 ms per loop

vstack on the list of arrays is considerably faster. Faster than the iterative vstack (which I expected). And basically the same as Ramon's row insertion (and insertion into order='F')

数组列表上的vstack要快得多。比迭代vstack更快(我预期)。和Ramon的行插入基本相同(插入顺序='F')

In [272]: %%timeit
x=np.zeros((n_dim,a.shape[0]))
for j in range(n_dim):
   x[j,:]=a*j
   .....: x=x.T
   .....: 
10 loops, best of 3: 23.3 ms per loop

While concatenate (used by vstack) is compiled, I suspect it does something similar to the iterative insertion. It is common in the source C code to create an empty target array, and then fill it with the appropriate values.

虽然连接(由vstack使用)被编译,但我怀疑它与迭代插入类似。源C代码中常见的是创建一个空目标数组,然后用适当的值填充它。

#1