多处理：使用tqdm显示进度条

To make my code more "pythonic" and faster, I use "multiprocessing" and a map function to send it a) the function and b) the range of iterations.

为了使我的代码更“pythonic”和更快,我使用“多处理”和一个map函数发送它a)函数和b)迭代范围。

The implanted solution (i.e., call tqdm directly on the range tqdm.tqdm(range(0, 30)) does not work with multiprocessing (as formulated in the code below).

植入的解决方案(即直接在范围tqdm.tqdm(范围(0,30))上调用tqdm不适用于多处理(如下面的代码所示)。

The progress bar is displayed from 0 to 100% (when python reads the code?) but it does not indicate the actual progress of the map function.

进度条显示从0到100%(当python读取代码时?)但它并不表示地图功能的实际进度。

How to display a progress bar that indicates at which step the 'map' function is ?

如何显示一个进度条,指示“map”功能在哪一步?

from multiprocessing import Pool
import tqdm
import time

def _foo(my_number):
   square = my_number * my_number
   time.sleep(1)
   return square 

if __name__ == '__main__':
   p = Pool(2)
   r = p.map(_foo, tqdm.tqdm(range(0, 30)))
   p.close()
   p.join()

Any help or suggestions are welcome...

欢迎任何帮助或建议......

3 个解决方案

#1

Use imap instead of map, which returns an iterator of processed values.

使用imap而不是map,它返回已处理值的迭代器。

from multiprocessing import Pool
import tqdm
import time

def _foo(my_number):
   square = my_number * my_number
   time.sleep(1)
   return square 

if __name__ == '__main__':
   with Pool(2) as p:
      r = list(tqdm.tqdm(p.imap(_foo, range(30)), total=30))

#2

Solution Found : Be careful! Due to multiprocessing, estimation time (iteration per loop, total time, etc... could be unstable) but the progress bar works perfectly.

找到解决方案:小心!由于多处理,估计时间(每个循环的迭代,总时间等......可能不稳定)但进度条完美地工作。

Note: Context manager for Pool is only available from Python version 3.3

注意:Pool的上下文管理器仅适用于Python 3.3版

from multiprocessing import Pool
import time
from tqdm import *

def _foo(my_number):
   square = my_number * my_number
   time.sleep(1)
   return square 

if __name__ == '__main__':
    with Pool(processes=2) as p:
        max_ = 30
        with tqdm(total=max_) as pbar:
            for i, _ in tqdm(enumerate(p.imap_unordered(_foo, range(0, max_)))):
                pbar.update()

#3

based on the answer of Xavi Martínez I wrote the function imap_unordered_bar. It can be used in the same way as imap_unordered with the only difference that a processing bar is shown.

根据XaviMartínez的答案,我编写了函数imap_unordered_bar。它可以与imap_unordered一样使用,唯一的区别是显示处理栏。

from multiprocessing import Pool
import time
from tqdm import *

def imap_unordered_bar(func, args, n_processes = 2):
    p = Pool(n_processes)
    res_list = []
    with tqdm(total = len(args)) as pbar:
        for i, res in tqdm(enumerate(p.imap_unordered(func, args))):
            pbar.update()
            res_list.append(res)
    pbar.close()
    p.close()
    p.join()
    return res_list

def _foo(my_number):
    square = my_number * my_number
    time.sleep(1)
    return square 

if __name__ == '__main__':
    result = imap_unordered_bar(_foo, range(5))

#1