不能从pool.map中pickle对象模块

I'm trying to parallelize loop with pair of numbers and some other fixed data. That is, I'm iterating pair of numbers like (1,2), (6,4), (4,3)... with other fixed dataset.

我试图用一对数字和一些其他固定数据并行化循环。也就是说,我正在迭代一对数字,如(1,2),(6,4),(4,3)...与其他固定数据集。

def myfunc(list of iterates, list of various datasets)
    doing stuff

if __name__ == '__main__':
    read_data(a lot)
    iterates = [[1,2], [3,8], [7,9], [12,5]] # means nothing

    pool = multiprocessing.Pool(processes=4)
    partial_myfunc = partial(myfunc, list of various datasets=some_data)
    results = pool.map(partial_myfunc, iterates)

myfunc is doing stuff with two inputs, list of iterates, list of various datasets.

myfunc正在处理两个输入,迭代列表,各种数据集列表。

'list of iterates' consists of like this [[1,2],[3,4],[5,6]...],
'list of various datasets' contains various types of dataset like [dates, float, dataframe,... etc] but it's fixed.

'迭代列表'由这样[[1,2],[3,4],[5,6] ...]组成,'各种数据集列表'包含各种类型的数据集,如[date,float,dataframe ,...等等,但它是固定的。

And when I run the main program mentioned above, python says

当我运行上面提到的主程序时,python说

Traceback (most recent call last):
File "path/main.py", line 117, in <module>
results = pool.map(partial_myfunc, iterates)
File "lib\multiprocessing\pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "lib\multiprocessing\pool.py", line 644, in get
raise self._value
File "lib\multiprocessing\pool.py", line 424, in _handle_tasks
put(task)
File "lib\multiprocessing\connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "lib\multiprocessing\reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
TypeError: can't pickle module objects

Is there anybody can tell what's going on? I really need some help guys.

有人能说出发生了什么吗?我真的需要帮助一些人。

1 个解决方案

#1

could you adjust your code for the threading to be the following please:

你可以调整你的线程代码如下:

...
max_threads = multiprocessing.cpu_count()

...
#create pool with max_threads - feel free to do max_threads-1 so your device is still usable
p = multiprocessing.Pool(max_threads)

results = p.map(partial_myfunc, iterates)

also are you using python 2 or 3? The more information you proivde the easier it is to figure out :) Let me know if it makes any difference!

你也在使用python 2或3吗?你提供的信息越多,弄清楚就越容易:)让我知道它是否有任何区别!

#1