Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

state sync: randomize order in which parts are requested #12498

Open
Tracked by #101
saketh-are opened this issue Nov 21, 2024 · 0 comments
Open
Tracked by #101

state sync: randomize order in which parts are requested #12498

saketh-are opened this issue Nov 21, 2024 · 0 comments
Assignees
Labels
T-core Team: issues relevant to the core team

Comments

@saketh-are
Copy link
Collaborator

saketh-are commented Nov 21, 2024

In the current state sync implementation, every node in the network will attempt to obtain state parts in roughly the same order (modulo any variation due to parallelism of buffer_unordered):

tokio_stream::iter(0..num_parts)
.map(|part_id| {
let future = downloader.ensure_shard_part_downloaded(
shard_id,
sync_hash,
part_id,
header.clone(),
cancel.clone(),
);
respawn_for_parallelism(&*future_spawner, "state sync download part", future)
})
.buffer_unordered(MAX_PARALLELISM_PER_SHARD_FOR_FAIRNESS)
.try_collect::<Vec<_>>()
.await?;

The peer selection mechanism is designed such that requests for the same state parts will be routed to the same hosts. The motivation is that we can benefit from caching of parts on the host side.

We should shuffle the order of requests on the client side to avoid generating spikes of requests to the same hosts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T-core Team: issues relevant to the core team
Projects
None yet
Development

No branches or pull requests

1 participant