提高性能 - 应用于numpy数组的每一行的符号函数

I have a list of input vectors that I need to use as inputs to a list of functions that are generated using SymPy. In the real application, the number of input vectors is ~100k, and there are ~5M sets of symbolic functions. This is currently the bottleneck in my code, so I'm trying to speed things up.

我有一个输入向量列表，我需要将其用作使用SymPy生成的函数列表的输入。在实际应用中，输入向量的数量约为~100k，并且有~5M组的符号函数。这是我的代码中的瓶颈，所以我试图加快速度。

I've already made big improvements by using Sympy's lambdify to create lambda functions based on numpy, but I can't help but think there's a way to vectorize this and get the for loops into numpy/C instead of python.

通过使用Sympy的lambdify来创建基于numpy的lambda函数，我已经做了很大的改进，但我不禁想到有一种方法可以对此进行向量化并将for循环变为numpy / C而不是python。

I initially thought numpy.apply_along_axis() would help, but it still does the looping in python.

我最初认为numpy.apply_along_axis（）会有所帮助，但它仍然在python中进行循环。

This is a simplified version of what I'm doing now:

这是我现在正在做的简化版本：

import time
import sympy as sp
import numpy as np

#Input for performance testing
# sampleSize = 200000
# inputVector = [1.2, -0.33]

# inputArray = np.array(inputVector*np.ones((sampleSize,1)))


#This array would have ~100k rows in actual data set
inputArray = [[-.333, -.558],
              [-.454, -.367],
              [-.568, -.678]]                    


start = time.time()


#These are the equations of motion of a mechanical system. Each row represents 
#a unique arrangement of components. There may be a better way to handle this, 
#but I haven't understood the system well enough to do so yet.

#This array would have ~5M rows in actual data set
symEqns = [['(R_1 - 1)/(R_0 - 1)', '0', '-R_1 + 1',     '(R_1 - 1)/(R_0*R_1 - 1)','1'],
           ['R_1/R_0',             '0', '-1/(R_0 - 1)', '(R_1 - 1)/(R_0 - 1)',    '1']]

for eqnSet in symEqns:
  #Create lambda functions 
  lambdaFuncs = []
  for eqn in eqnSet:
    func = sp.lambdify(['R_0', 'R_1'], eqn, 'numpy')

    #This is ~5x slower, due to use of pure python vs. numpy ??
    # func = lambda R_0, R_1: eval(eqn)  

    lambdaFuncs.append(func)

  #Evaluate each lambda func for each input set

  # in my actual code, this is a parameter of an object. forgot to store it in my example code
  outputList = []   
  for row in inputArray:
    results = []
    for func in lambdaFuncs:
      results.append(func(*row))
    outputList.append(results)

end = time.time()
print "\nTotal Time Elapsed: {:d}:{:0>5.2f}".format(int((end-start)/60), (end-start)%60)

If it helps, I could also structure the evaluation to calculate each function independently, creating a column of results for each function. Here's an example of the evaluation block in that case (using for loops for illustration, I'd like to get to vectorized evaluation using numpy):

如果它有帮助，我还可以构建评估以独立计算每个函数，为每个函数创建一列结果。以下是该案例中评估块的示例（使用for循环进行说明，我想使用numpy进行矢量化评估）：

  #Evaluate each lambda func for each input set
  outputList = []
  for func in lambdaFuncs:
    results = []
    for row in inputArray:
      results.append(func(*row))
    outputList.append(results)       

[EDIT] For future reference, here's my improved working sample code for this problem. I've tweaked a few things from Oliver's response, mainly allowing for variable length input vectors:

[编辑]为了将来参考，这是我改进的这个问题的工作示例代码。我已经从Oliver的响应中调整了一些内容，主要是允许可变长度的输入向量：

import time
import sympy as sp
import numpy as np

# This array would have ~100k rows in actual data set
input_array = np.array([[-.333, -.558],
              [-.454, -.367],
              [-.568, -.678]])



#This array would have ~5M rows in actual data set (generated via Sympy linear algebraic solns)
sym_eqns = [['(R_1 - 1)/(R_0 - 1)', '0', '-R_1 + 1',     '(R_1 - 1)/(R_0*R_1 - 1)','1'],
           ['R_1/R_0',             '0', '-1/(R_0 - 1)', '(R_1 - 1)/(R_0 - 1)',    '1']]

for eqn_set in sym_eqns:
  output_list = []
  for eqn in eqn_set:
    func = sp.lambdify(['R_0', 'R_1'], eqn, 'numpy')
    results = func(*[input_array[:,n] for n in range(input_array.shape[1])])  
    output_list.append(results)     

1 个解决方案

#1

It's hard to make any concrete timings without the actual equations, but there are a few recommendations regarding your code.

没有实际的方程式很难做出任何具体的时间，但是有一些关于你的代码的建议。

First, let's talk about the equations:

首先，我们来谈谈方程式：

• If there's always a column of zeros and ones, why bother evaluating?
• 如果总是有一列零和一列，为什么还要评估呢？
• There seems to be symmetry in the equations: your symEqns[0][0] == symEqns[1][3]. Again, why evaluate?
• 方程似乎是对称的：你的symEqns [0] [0] == symEqns [1] [3]。再次，为什么评估？

What is the origin of these equations? I see that R_1 - 1 is a rather common factor. Perhaps your original problem is much easier to solve.

这些方程的起源是什么？我看到R_1 - 1是一个相当普遍的因素。也许你原来的问题要容易解决。

Second, let's talk about the looping. You can get rid of one looping construct already, out of 4:

其次，我们来谈谈循环。你可以从4中删除一个循环结构：

This:

这个：

for eqnSet in symEqns:
  lambdaFuncs = []
  for eqn in eqnSet:
    func = sp.lambdify(['R_0', 'R_1'], eqn, 'numpy')
    lambdaFuncs.append(func)

  # This seems out of place in your example code: whenever a
  # new row of functions is being processed, you lose all the 
  # data from outputList, because you're not storing it anywhere.
  outputList = [] 
  for row in inputArray:
    results = []
    for func in lambdaFuncs:
      results.append(func(*row))
    outputList.append(results)

could be changed into this:

可以改成这个：

outputlist = [] # Better position for outputList
for eqnSet in symEqns:
  for eqn in eqnSet:
    func = sp.lambdify(['R_0', 'R_1'], eqn, 'numpy')
    for row in inputArray:
        results = []
        results.append(func(*row))
    outputList.append(results)

Unless you actually need to store all the lambdified numpy functions, which I highly doubt.

除非你真的需要存储所有lambdified numpy函数，我非常怀疑。

You can get rid of yet another looping construct, by realizing your lambdified functions work exactly like numpy functions: they're vectorized as well.

你可以通过实现lambdified函数与numpy函数完全相同来摆脱另一个循环结构：它们也是矢量化的。

>>> for row in inputArray:
...     print(f(*row)),
1.16879219805 0.940165061898 1.07015306122

>>> arr = np.array(inputArray)
>>> f(arr[:,0], arr[:,1])
array([ 1.1687922 ,  0.94016506,  1.07015306])

Same output, no for-loop. That would bring your quadruple for-loop down to:

相同的输出，没有for循环。这将使您的四重换环降至：

input_data = np.array(inputArray)
outputlist = [] # Better position for outputList
for eqnSet in symEqns:
    for eqn in eqnSet:
        func = sp.lambdify(['R_0', 'R_1'], eqn, 'numpy')
        outputList.append(func(input_data[:,0], input_data[:,1]))

This'll be much faster, because now basically you're only looping over the list of sympy functions, not any longer over the data (which is now contiguous, so has cache benefits) or the list of lambdified sympy functions. It would be very nice if you could add some timing results in the comments, once you've applied these techniques.

这会快得多，因为现在基本上你只是循环遍历sympy函数列表，而不是遍历数据（现在是连续的，因此具有缓存优势）或lambdified sympy函数列表。如果您在应用这些技术后可以在注释中添加一些时序结果，那将是非常好的。

Also, a gentle hint: in the python programming language, most programmers follow the PEP8 coding style, which means variables are all lowercase, with underscores separating words.

另外，一个温和的提示：在python编程语言中，大多数程序员遵循PEP8编码风格，这意味着变量都是小写的，下划线分隔单词。