Skip to content
This repository has been archived by the owner on May 30, 2022. It is now read-only.

需求:启用多进程计算 #13

Open
jiajunsu opened this issue Nov 24, 2018 · 2 comments
Open

需求:启用多进程计算 #13

jiajunsu opened this issue Nov 24, 2018 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@jiajunsu
Copy link
Owner

jiajunsu commented Nov 24, 2018

0.选择启用多进程的核心数(不能大于CPU总核数)
1.用imap_unordered进行多进程计算
2.各进程计算结果分别保存至result-*.xls,单文件限制数量与当前保持一致
3.全部计算完成后,将result-*.xls合并为result.xls,对超出总数限制的场景提示错误,并取消文件合并

参考实现
https://stackoverflow.com/questions/38839170/python-itertools-with-multiprocessing-huge-list-vs-inefficient-cpus-usage-with

@jiajunsu jiajunsu added the enhancement New feature or request label Nov 24, 2018
@jiajunsu
Copy link
Owner Author

先在result_combination上实现多进程并行,顺便用tqdm来做进度条

from tqdm import tqdm
import multiprocessing
from itertools import combinations


def f(i):
    pass

cores = multiprocessing.cpu_count()
p = multiprocessing.Pool(processes=cores)

for i in tqdm(p.imap_unordered(f, combinations(range(0,50), 4)), desc='Calculating', total=230300, unit='comb'):
    pass

p.close()
p.join()

@jiajunsu jiajunsu self-assigned this Nov 30, 2018
jiajunsu added a commit that referenced this issue Dec 3, 2018
@jiajunsu
Copy link
Owner Author

修改多进程实现方案,使用multiprocessing.Manager的Queue实现进程间数据共享,一个进程负责将结果保存到文件,其他进程负责计算,主进程负责进度条显示

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant