I'm working on a project where I have a set of samples for gesture recognition (ChaLearn dataset).
我正在做一个项目，我有一组手势识别的样本(ChaLearn dataset)。
The input is a (10000,6,32,64,64) array. I have 10000 examples, 6 featuremaps (body, l-hand, r-hand in depth and rgb images), 32 frames of 64 by 64.
输入是一个(10000、6、32、64、64)数组。我有10000个例子，6个特点(身体，l-hand, r-hand in depth和rgb images)， 32帧64×64。
I have to select only the depth images (which are indices 1, 3 and 5) and 4 frames out of 32 (indices 8,13,18,25)
我只需要选择深度图像(索引1、3和5)和4帧32(索引8、13、18、25)
I'm not that familiar with python so I was messing around and I think my solution is fine but it is really slow. Mostly the two selections take up alot of time (70-80 seconds for this bit).
我对python不是很熟悉，所以我在搞混，我认为我的解决方案很好，但它真的很慢。大多数情况下，这两个选择占用了大量的时间(这段时间是70-80秒)。
samples.shape gives me (10000,6,32,64,64) at the start
样本。形状一开始给我(10000,6,32,64,64
samples = samples[0:samples.shape[0],0:samples.shape[1],(7,13,18,25)]
samples = samples[0:samples.shape[0],(1,3,5)]
samples = floatX(samples.reshape((samples.shape[0],12,64,64)))/255.
samples.shape gives me (10000,12,64,64) at the end.
样本。形状给我(10000,12,64,64)最后。
Is there a more efficient way of accomplishing my goal? Or am I completely wrong here?
有没有更有效的方法来实现我的目标?还是我完全错了?
ignore the floatx part for now :)
暂时忽略floatx部分:)
3 个解决方案
#1
2
As mentioned in the other answer, there are some discussions in numpy about making this easier, but in your case, since the two axes you want to index along are adjacent in the shape, things aren't so bad. This should get you what you are after avoiding that intermediate copy altogether:
正如在另一个答案中提到的，numpy中有一些关于简化的讨论，但是在你的例子中，由于你想要索引的两个轴在形状上是相邻的，所以事情并没有那么糟糕。这应该会让你在避免了中间拷贝之后得到你想要的东西:
depth_idx = np.array([1, 3, 5], dtype=np.intp)
frame_idx = np.array([8, 13, 18, 25], dtype=np.intp)
samples = samples[:, depth_idx[:, None], frame_idx].reshape(-1, 12, 64, 64)
Actually, since your depth_idx happens to be writeable as a slice, for this particular case you will actually be better off doing:
实际上，因为你的depth_idx恰好是可写的，对于这个特殊的情况，你最好这样做:
samples = samples[:, 1::2, frame_idx].reshape(-1, 12, 64, 64)
#2
1
First, you can just do samples[:,:,(7,13,18,25)]. However, the reason this is slow is because you are doing you are doing 2 copy operations. You can't currently get around this, although you can simplify stuff somewhat using take:
首先，你可以做样本[:，:，(7,13,18,25)]。但是，这是慢的原因是因为你在做2个复制操作。你现在不能回避这个问题，尽管你可以用take来简化一些东西:
samples = np.take(np.take(samples, [7,13,18,25], axis=2), [1,3,4], axis=1)
There is actually work going on to make the operation you are trying to do much easier, but it isn't ready yet.
实际上，您正在进行的操作要容易得多，但还没有准备好。
If you aren't using numpy 1.9, you should definitely upgrade to it, since the "fancy" indexing you are doing is much faster. The take version will be faster in earlier versions.
如果您不使用numpy 1.9，那么您肯定应该升级到它，因为您正在进行的“花哨”索引要快得多。在早期版本中，take版本将会更快。
#3
1
Is there a more efficient way of accomplishing my goal?
有没有更有效的方法来实现我的目标?
If you are willing to look into another layer of abstraction, you might want to consider a format that is designed for larger data. Personally, I use HDF which has a few nice python modules like h5py and pytables.
如果您愿意研究另一个抽象层，您可能希望考虑为更大的数据设计的格式。就我个人而言，我使用HDF，它有一些不错的python模块，如h5py和pytables。
Two main advantages I see, you'll be able to store much larger datasets at once (since you don't have to load the entire set into memory) and you'll be able to apply meta-data to the data (with .attrs). When you build the hdf data set you can organize it so your "query" of something like "indices 1, 3 and 5" can be much more logical.
我看到的两个主要优点是，您可以同时存储更大的数据集(因为您不必将整个数据集加载到内存中)，并且您可以将元数据应用到数据中(使用.attrs)。当您构建hdf数据集时，您可以对其进行组织，以便您对“索引1、3和5”之类的内容的“查询”可以更符合逻辑。
#1
2
As mentioned in the other answer, there are some discussions in numpy about making this easier, but in your case, since the two axes you want to index along are adjacent in the shape, things aren't so bad. This should get you what you are after avoiding that intermediate copy altogether:
正如在另一个答案中提到的，numpy中有一些关于简化的讨论，但是在你的例子中，由于你想要索引的两个轴在形状上是相邻的，所以事情并没有那么糟糕。这应该会让你在避免了中间拷贝之后得到你想要的东西:
depth_idx = np.array([1, 3, 5], dtype=np.intp)
frame_idx = np.array([8, 13, 18, 25], dtype=np.intp)
samples = samples[:, depth_idx[:, None], frame_idx].reshape(-1, 12, 64, 64)
Actually, since your depth_idx happens to be writeable as a slice, for this particular case you will actually be better off doing:
实际上，因为你的depth_idx恰好是可写的，对于这个特殊的情况，你最好这样做:
samples = samples[:, 1::2, frame_idx].reshape(-1, 12, 64, 64)
#2
1
First, you can just do samples[:,:,(7,13,18,25)]. However, the reason this is slow is because you are doing you are doing 2 copy operations. You can't currently get around this, although you can simplify stuff somewhat using take:
首先，你可以做样本[:，:，(7,13,18,25)]。但是，这是慢的原因是因为你在做2个复制操作。你现在不能回避这个问题，尽管你可以用take来简化一些东西:
samples = np.take(np.take(samples, [7,13,18,25], axis=2), [1,3,4], axis=1)
There is actually work going on to make the operation you are trying to do much easier, but it isn't ready yet.
实际上，您正在进行的操作要容易得多，但还没有准备好。
If you aren't using numpy 1.9, you should definitely upgrade to it, since the "fancy" indexing you are doing is much faster. The take version will be faster in earlier versions.
如果您不使用numpy 1.9，那么您肯定应该升级到它，因为您正在进行的“花哨”索引要快得多。在早期版本中，take版本将会更快。
#3
1
Is there a more efficient way of accomplishing my goal?
有没有更有效的方法来实现我的目标?
If you are willing to look into another layer of abstraction, you might want to consider a format that is designed for larger data. Personally, I use HDF which has a few nice python modules like h5py and pytables.
如果您愿意研究另一个抽象层，您可能希望考虑为更大的数据设计的格式。就我个人而言，我使用HDF，它有一些不错的python模块，如h5py和pytables。
Two main advantages I see, you'll be able to store much larger datasets at once (since you don't have to load the entire set into memory) and you'll be able to apply meta-data to the data (with .attrs). When you build the hdf data set you can organize it so your "query" of something like "indices 1, 3 and 5" can be much more logical.
我看到的两个主要优点是，您可以同时存储更大的数据集(因为您不必将整个数据集加载到内存中)，并且您可以将元数据应用到数据中(使用.attrs)。当您构建hdf数据集时，您可以对其进行组织，以便您对“索引1、3和5”之类的内容的“查询”可以更符合逻辑。