Python(多进程multiprocessing模块)

时间:2022-05-23 04:07:53

day31

http://www.cnblogs.com/yuanchenqi/articles/5745958.html

由于GIL的存在,python中的多线程其实并不是真正的多线程,如果想要充分地使用多核CPU的资源,在python中大部分情况需要使用多进程。Python提供了非常好用的多进程包multiprocessing,只需要定义一个函数,Python会完成其他所有事情。借助这个包,可以轻松完成从单进程到并发执行的转换。

多进程

 1 from multiprocessing import Process#进程
 2 import time
 3 def f(name):
 4     time.sleep(1)
 5     print('hello', name,time.ctime())
 6 
 7 if __name__ == '__main__':
 8     p_list=[]
 9     for i in range(3):#和threading,差不多
10         p = Process(target=f, args=('alvin',))
11         p_list.append(p)
12         p.start()
13     for i in p_list:
14         p.join()
15     print('end')
16 
17 #延迟一秒后全部输出

延迟1面。输出。

执行结果:

/home/nizhipeng/PycharmProjects/learnPython/venv/bin/python /home/nizhipeng/PycharmProjects/learnPython/week8/多进程.py
hello alvin Mon Nov  5 22:25:45 2018
hello alvin Mon Nov  5 22:25:45 2018
hello alvin Mon Nov  5 22:25:45 2018
end

Process finished with exit code 0

 

类式调用多线程

 

 1 from multiprocessing import Process
 2 import time
 3 
 4 class MyProcess(Process):#继承
 5     def __init__(self):
 6         super(MyProcess, self).__init__()
 7         #self.name = name 可以从新赋值
 8 
 9     def run(self):
10         time.sleep(1)
11         print ('hello', self.name,time.ctime())#hello MyProcess-3 Mon Nov  5 21:20:11 2018
12 
13 
14 if __name__ == '__main__':
15     p_list=[]
16     for i in range(3):
17         p = MyProcess()
18         p.start()
19         p_list.append(p)
20 
21     for p in p_list:
22         p.join()
23 
24     print('end')

 

使用方式和threading多线程一样。

执行结果:

hello MyProcess-1 Mon Nov  5 22:27:51 2018
hello MyProcess-2 Mon Nov  5 22:27:51 2018
hello MyProcess-3 Mon Nov  5 22:27:51 2018
end

Process finished with exit code 0

 

进程关系

 1 from multiprocessing import Process
 2 import os
 3 import time
 4 def info(title):
 5     print(title)#传入信息
 6     print('module name:', __name__)#main
 7     print('parent process:', os.getppid())#父进程号
 8     print('process id:', os.getpid())#本进程号
 9 #每一个进程都有根进程
10 
11 def f(name):
12     info('\033[31;1mfunction f\033[0m')
13     print('hello', name)
14 
15 if __name__ == '__main__':
16     info('\033[32;1mmain process line\033[0m')
17     time.sleep(3)
18     p = Process(target=info, args=('bob',))#子进程 ,其父进程是主进程
19     p.start()
20     p.join()

 

执行结果:

main process line
module name: __main__
parent process: 4319
process id: 1896
bob
module name: __main__
parent process: 1896
process id: 1929

Process finished with exit code 0

从输出结果可以看出,子进程p的父进程号1896是主进程号1896。主进程的父进程号是4319。

每一个进程都有根进程

主进程与子进程通信Pipe

 1 from multiprocessing import Process, Pipe
 2 
 3 
 4 def f(conn):
 5     conn.send([42, None, 'hello'])#其中conn,为其父进程的child_conn
 6     conn.close()
 7 
 8 
 9 if __name__ == '__main__':
10     parent_conn, child_conn = Pipe()
11     p = Process(target=f, args=(child_conn,))
12     p.start()
13     print(parent_conn.recv())  # prints "[42, None, 'hello']"#子进程发送,主进程接收
14     p.join()

执行结果:

[42, None, 'hello']

Process finished with exit code 0

 

manager实现数据共享

 1 from multiprocessing import Process, Manager
 2 import threading
 3 def f(d, l,n):
 4     d[n] = '1'#n 为0~1个线程,全赋为1
 5     d['2'] = 2
 6     d[0.25] = None
 7     l.append(n)
 8     #print('l:', l)
 9 
10 if __name__ == '__main__':
11     with Manager() as manager: #manager = manager()
12         d = manager.dict()#字典
13 
14         l = manager.list(range(5))#初始值为零到四的列表
15 
16         p_list = []
17         for i in range(10):
18             p = Process(target=f, args=(d, l, i))
19             #p = threading.Thread(target=f, args=(d, l, i)) #多线程
20             p.start()
21             p_list.append(p)
22 
23         for res in p_list:
24             res.join()
25 
26         print(d)
27         print(l)

执行结果:

{0.25: None, 1: '1', 2: '1', 3: '1', '2': 2, 5: '1', 6: '1', 7: '1', 8: '1', 9: '1', 4: '1', 0: '1'}
[0, 1, 2, 3, 4, 1, 0, 6, 2, 3, 7, 4, 8, 9, 5]

Process finished with exit code 0

d字典中,n 0~9,都赋值为1,其余按要求赋值。

列表l中,前五个初始为0~4,后十个值由于是多进程,排列顺序随机。