preprocess_step1 takes too long time

[qiaopr]

Hi,How fast are you preprocess_step1.py files running?Mine has take several hours to process only 150 piece of data. It's unbeliviable!

[RichardHGL]

Did you modify any parameters? Such as graph size. I think it won't be so slow.
When I processed the data, I just keep it run in the backend and get the data the next day.
I guess any dataset can be processed in one day.

[qiaopr]

I exactly followed the preprocess >Freebase>README.md.I didn't modify any parameters. The data I'm processed is CWQ.

[LLLiaomeng]

I also faced this problem.

[JasonCen-sweetdreams]

Also ran into this problem. It seems that the preprocess_step1 takes a lot of time to output only a few, while my CPU, GPU and memory usage are in a healthy state. sad :(

[RichardHGL]

Okay, I'll check this problem next month. You can also try to look into ppr_util.py, I think the majority of time is spent on the calculation of ppr for every graph.

[RichardHGL]

The preprocessed datasets can be found in Readme of this repo. If you find any possible ways to improve the efficiency of preprocessing, please kindly let me know.