Optimize BlockLocators Iterator with Parallel Processing #3430
Conversation
This PR enhances the BlockLocators iterator by introducing parallel processing with rayon. The checkpoints and recents maps are combined using into_par_iter, leveraging multi-threading to potentially improve performance. Signed-off-by: Hwangjae Lee <meetrick@gmail.com>
This patch introduces a performance test for the BlockLocators iterator, outputting the processing times.
|
hey guys! I expected that processing speed would increase with parallel processing. However, after testing, I observed that up to a certain number of items, parallel processing introduced overhead, which caused it to take longer. Based on my test code, sequential processing performed better up to approximately 100,000 items. The performance is also affected by the machine’s capabilities. I would appreciate reviewers’ opinions on this. It seems that additional changes may be needed in the backend to fully accommodate these updates. |
| self.checkpoints | ||
| .into_par_iter() | ||
| .chain(self.recents.into_par_iter()) | ||
| .collect::<Vec<_>>(), |
There was a problem hiding this comment.
can't this be collected directly into a BTreeMap via from_par_iter?
|
@meetrick in general, I'm not convinced that parallelizing the iteration itself is likely to be a performance improvement at any scale, unless we can utilize |
|
@ljedrz It seems I misunderstood the TODO. I’ll work on it again. Thanks for your input. |
|
In addition, it is possible that sorting is not always necessary; I don't recall the related logic very well, but it seems to me that |
Motivation
This PR aims to optimize the BlockLocators into_iter method by introducing parallel processing. The existing sequential processing approach can be time-consuming when handling large datasets of block data. By leveraging rayon for parallel processing, this change seeks to improve performance in merging checkpoints and recents maps, providing a more efficient solution for large-scale data handling.
Test Plan
Generate a dataset of 1000 items to compare processing speeds.
Related PRs
(Link any related PRs here)