I am doing some performance test on a variant of the prime numbers generator from http://docs.cython.org/src/tutorial/numpy.html. The below performance measures are with kmax=1000
我正在从http://docs.cython.org/src/tutorial/numpy.html对质数生成器的一个变体进行性能测试。下面的性能指标是kmax=1000。
Pure Python implementation, running in CPython: 0.15s
纯Python实现,运行于CPython: 0.15s。
Pure Python implementation, running in Cython: 0.07s
纯Python实现,运行在Cython: 0.07s。
def primes(kmax):
p = []
k = 0
n = 2
while k < kmax:
i = 0
while i < k and n % p[i] != 0:
i = i + 1
if i == k:
p.append(n)
k = k + 1
n = n + 1
return p
Pure Python+Numpy implementation, running in CPython: 1.25s
纯Python+Numpy实现,运行于CPython: 1.25s。
import numpy
def primes(kmax):
p = numpy.empty(kmax, dtype=int)
k = 0
n = 2
while k < kmax:
i = 0
while i < k and n % p[i] != 0:
i = i + 1
if i == k:
p[k] = n
k = k + 1
n = n + 1
return p
Cython implementation using int*: 0.003s
Cython实现使用int*: 0.003。
from libc.stdlib cimport malloc, free
def primes(int kmax):
cdef int n, k, i
cdef int *p = <int *>malloc(kmax * sizeof(int))
result = []
k = 0
n = 2
while k < kmax:
i = 0
while i < k and n % p[i] != 0:
i = i + 1
if i == k:
p[k] = n
k = k + 1
result.append(n)
n = n + 1
free(p)
return result
The above performs great but looks horrible, as it holds two copies of the data... so I tried reimplementing it:
上面的表现很好,但是看起来很糟糕,因为它包含了两个数据的副本……所以我尝试重新实施它:
Cython + Numpy: 1.01s
Cython + Numpy:1.01 s
import numpy as np
cimport numpy as np
cimport cython
DTYPE = np.int
ctypedef np.int_t DTYPE_t
@cython.boundscheck(False)
def primes(DTYPE_t kmax):
cdef DTYPE_t n, k, i
cdef np.ndarray p = np.empty(kmax, dtype=DTYPE)
k = 0
n = 2
while k < kmax:
i = 0
while i < k and n % p[i] != 0:
i = i + 1
if i == k:
p[k] = n
k = k + 1
n = n + 1
return p
Questions:
问题:
- why is the numpy array so incredibly slower than a python list, when running on CPython?
- 当在CPython上运行时,为什么numpy数组要比python列表慢得令人难以置信呢?
- what did I do wrong in the Cython+Numpy implementation? cython is obviously NOT treating the numpy array as an int[] as it should.
- 我在Cython+Numpy实现中做错了什么?显然,cython不将numpy数组视为int[]。
-
how do I cast a numpy array to a int*? The below doesn't work
如何将numpy数组转换为int*?以下是行不通的
cdef numpy.nparray a = numpy.zeros(100, dtype=int) cdef int * p = <int *>a.data
3 个解决方案
#1
8
cdef DTYPE_t [:] p_view = p
Using this instead of p in the calculations. reduced the runtime from 580 ms down to 2.8 ms for me. About the exact same runtime as the implementation using *int. And that's about the max you can expect from this.
用这个代替p。将运行时从580 ms降低到2.8 ms。与使用*int的实现完全相同的运行时。这就是你所能期望的最大值。
DTYPE = np.int
ctypedef np.int_t DTYPE_t
@cython.boundscheck(False)
def primes(DTYPE_t kmax):
cdef DTYPE_t n, k, i
cdef np.ndarray p = np.empty(kmax, dtype=DTYPE)
cdef DTYPE_t [:] p_view = p
k = 0
n = 2
while k < kmax:
i = 0
while i < k and n % p_view[i] != 0:
i = i + 1
if i == k:
p_view[k] = n
k = k + 1
n = n + 1
return p
#2
5
why is the numpy array so incredibly slower than a python list, when running on CPython?
当在CPython上运行时,为什么numpy数组要比python列表慢得令人难以置信呢?
Because you didn't fully type it. Use
因为你没有完全输入。使用
cdef np.ndarray[dtype=np.int, ndim=1] p = np.empty(kmax, dtype=DTYPE)
how do I cast a numpy array to a int*?
如何将numpy数组转换为int*?
By using np.intc
as the dtype, not np.int
(which is a C long
). That's
使用np.intc作为dtype,而不是np.int(这是一个C长)。这是
cdef np.ndarray[dtype=int, ndim=1] p = np.empty(kmax, dtype=np.intc)
(But really, use a memoryview, they're much cleaner and the Cython folks want to get rid of the NumPy array syntax in the long run.)
(但实际上,使用memoryview,它们更干净,而Cython人员希望从长期运行中去掉NumPy数组语法。)
#3
0
Best syntax I found so far:
迄今为止我发现的最佳语法:
import numpy
cimport numpy
cimport cython
@cython.boundscheck(False)
@cython.wraparound(False)
def primes(int kmax):
cdef int n, k, i
cdef numpy.ndarray[int] p = numpy.empty(kmax, dtype=numpy.int32)
k = 0
n = 2
while k < kmax:
i = 0
while i < k and n % p[i] != 0:
i = i + 1
if i == k:
p[k] = n
k = k + 1
n = n + 1
return p
Note where I used numpy.int32 instead of int. Anything on the left side of a cdef is a C type (thus int = int32 and float = float32), while anything on the RIGHT side of it (or outside of a cdef) is a python type (int = int64 and float = float64)
注意,我使用numpy.int32代替int。任何cdef的左边是一个C型(因此int = int32和浮动= float32),而任何的右边(或cdef之外)是python类型(int = int64和浮动= float64)
#1
8
cdef DTYPE_t [:] p_view = p
Using this instead of p in the calculations. reduced the runtime from 580 ms down to 2.8 ms for me. About the exact same runtime as the implementation using *int. And that's about the max you can expect from this.
用这个代替p。将运行时从580 ms降低到2.8 ms。与使用*int的实现完全相同的运行时。这就是你所能期望的最大值。
DTYPE = np.int
ctypedef np.int_t DTYPE_t
@cython.boundscheck(False)
def primes(DTYPE_t kmax):
cdef DTYPE_t n, k, i
cdef np.ndarray p = np.empty(kmax, dtype=DTYPE)
cdef DTYPE_t [:] p_view = p
k = 0
n = 2
while k < kmax:
i = 0
while i < k and n % p_view[i] != 0:
i = i + 1
if i == k:
p_view[k] = n
k = k + 1
n = n + 1
return p
#2
5
why is the numpy array so incredibly slower than a python list, when running on CPython?
当在CPython上运行时,为什么numpy数组要比python列表慢得令人难以置信呢?
Because you didn't fully type it. Use
因为你没有完全输入。使用
cdef np.ndarray[dtype=np.int, ndim=1] p = np.empty(kmax, dtype=DTYPE)
how do I cast a numpy array to a int*?
如何将numpy数组转换为int*?
By using np.intc
as the dtype, not np.int
(which is a C long
). That's
使用np.intc作为dtype,而不是np.int(这是一个C长)。这是
cdef np.ndarray[dtype=int, ndim=1] p = np.empty(kmax, dtype=np.intc)
(But really, use a memoryview, they're much cleaner and the Cython folks want to get rid of the NumPy array syntax in the long run.)
(但实际上,使用memoryview,它们更干净,而Cython人员希望从长期运行中去掉NumPy数组语法。)
#3
0
Best syntax I found so far:
迄今为止我发现的最佳语法:
import numpy
cimport numpy
cimport cython
@cython.boundscheck(False)
@cython.wraparound(False)
def primes(int kmax):
cdef int n, k, i
cdef numpy.ndarray[int] p = numpy.empty(kmax, dtype=numpy.int32)
k = 0
n = 2
while k < kmax:
i = 0
while i < k and n % p[i] != 0:
i = i + 1
if i == k:
p[k] = n
k = k + 1
n = n + 1
return p
Note where I used numpy.int32 instead of int. Anything on the left side of a cdef is a C type (thus int = int32 and float = float32), while anything on the RIGHT side of it (or outside of a cdef) is a python type (int = int64 and float = float64)
注意,我使用numpy.int32代替int。任何cdef的左边是一个C型(因此int = int32和浮动= float32),而任何的右边(或cdef之外)是python类型(int = int64和浮动= float64)