多处理。池:何时使用apply, apply_async或map?

I have not seen clear examples with use-cases for Pool.apply, Pool.apply_async and Pool.map. I am mainly using Pool.map; what are the advantages of others?

我还没有看到关于池的用例的清晰示例。运用,池。apply_async Pool.map。我主要使用Pool.map;别人的优点是什么?

2 个解决方案

#1

293

Back in the old days of Python, to call a function with arbitrary arguments, you would use apply:

在过去的Python时代，要调用具有任意参数的函数，可以使用apply:

apply(f,args,kwargs)

apply still exists in Python2.7 though not in Python3, and is generally not used anymore. Nowadays,

apply在Python2.7中仍然存在，但是在Python3中不存在，并且通常不再被使用。如今,

f(*args,**kwargs)

is preferred. The multiprocessing.Pool modules tries to provide a similar interface.

者优先。多处理。池模块尝试提供类似的接口。

Pool.apply is like Python apply, except that the function call is performed in a separate process. Pool.apply blocks until the function is completed.

池。apply类似于Python apply，但是函数调用是在一个单独的进程中执行的。池。应用块直到函数完成。

Pool.apply_async is also like Python's built-in apply, except that the call returns immediately instead of waiting for the result. An ApplyResult object is returned. You call its get() method to retrieve the result of the function call. The get() method blocks until the function is completed. Thus, pool.apply(func, args, kwargs) is equivalent to pool.apply_async(func, args, kwargs).get().

池。apply_async也类似于Python的内置应用程序，只是调用立即返回而不是等待结果。返回一个ApplyResult对象。您调用它的get()方法来检索函数调用的结果。get()方法阻塞，直到函数完成。因此,池。apply(func, args, kwargs)相当于pool。apply_async(kwargs func,args). get()。

In contrast to Pool.apply, the Pool.apply_async method also has a callback which, if supplied, is called when the function is complete. This can be used instead of calling get().

与池。运用,池中。apply_async方法也有一个回调，如果提供的话，在函数完成时调用这个回调。可以使用它而不是调用get()。

For example:

例如:

import multiprocessing as mp
import time

def foo_pool(x):
    time.sleep(2)
    return x*x

result_list = []
def log_result(result):
    # This is called whenever foo_pool(i) returns a result.
    # result_list is modified only by the main process, not the pool workers.
    result_list.append(result)

def apply_async_with_callback():
    pool = mp.Pool()
    for i in range(10):
        pool.apply_async(foo_pool, args = (i, ), callback = log_result)
    pool.close()
    pool.join()
    print(result_list)

if __name__ == '__main__':
    apply_async_with_callback()

may yield a result such as

可能产生这样的结果

[1, 0, 4, 9, 25, 16, 49, 36, 81, 64]

Notice, unlike pool.map, the order of the results may not correspond to the order in which the pool.apply_async calls were made.

注意,与池。映射，结果的顺序可能与池的顺序不一致。apply_async调用。

So, if you need to run a function in a separate process, but want the current process to block until that function returns, use Pool.apply. Like Pool.apply, Pool.map blocks until the complete result is returned.

因此，如果您需要在一个单独的进程中运行一个函数，但是要让当前进程阻塞，直到函数返回，使用Pool.apply。像池。运用,池。映射块直到返回完整的结果。

If you want the Pool of worker processes to perform many function calls asynchronously, use Pool.apply_async. The order of the results is not guaranteed to be the same as the order of the calls to Pool.apply_async.

如果希望工作进程池异步执行许多函数调用，请使用Pool.apply_async。结果的顺序不能保证与调用Pool.apply_async的顺序相同。

Notice also that you could call a number of different functions with Pool.apply_async (not all calls need to use the same function).

还请注意，您可以使用Pool调用许多不同的函数。apply_async(不是所有的调用都需要使用相同的函数)。

In contrast, Pool.map applies the same function to many arguments. However, unlike Pool.apply_async, the results are returned in an order corresponding to the order of the arguments.

相比之下,池。map对许多参数应用相同的函数。但是,与池。apply_async，返回结果的顺序与参数的顺序相对应。

#2

Regarding apply vs map:

关于申请vs地图:

pool.apply(f, args): f is only executed in ONE of the workers of the pool. So ONE of the processes in the pool will run f(args).

池。应用(f, args): f只在池的一个工作人员中执行。所以池中的一个进程会运行f(args)

pool.map(f, iterable): This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks. So you take advantage of all the processes in the pool.

池。map(f, iterable):该方法将迭代分解成多个块，并将其作为单独的任务提交到流程池中。利用池中的所有进程。

#1

293