[Bug] Cannot sync a new validator / client node, error at block 719600 - 719650

[zklimaleo]

🐛 Bug Report

Node stop running and having error at syncing block 719600 - 719650 using run-validator.sh file:

2024-10-17T01:21:34.902750Z  WARN snarkos_node_cdn::blocks: Failed to deserialize blocks 719600 to 719650 - io error:  - retrying (1 attempt(s) so far)
2024-10-17T01:21:34.902750Z  WARN snarkos_node_cdn::blocks: Failed to deserialize blocks 719600 to 719650 - io error:  - retrying (2 attempt(s) so far)
…
2024-10-17T01:24:45.369990Z  WARN snarkos_node_cdn::blocks: Failed to deserialize blocks 719600 to 719650 - io error: retrying (10 attempt(s) so far)
2024-10-17T01:24:45.568959Z  WARN snarkos_node_cdn::blocks: Maximum number of requests to https://blocks.aleo.org/mainnet/v0/719600.719650.blocks reached - shutting down...
2024-10-17T01:24:46.033593Z  DEBUGsnarkos_node_cdn::blocks: Finished network requests to the CDN
2024-10-17T01:24:46.040834Z  INFO snarkos_node_cdn::blocks: Stopping block sync at 719599 - shutting down

and run-client.sh file:

2024-10-17T08:46:33.016390Z DEBUG snarkos_node_cdn::blocks: Waiting for the first relevant blocks (1550 pending)
2024-10-17T08:46:33.739239Z  WARN snarkos_node_cdn::blocks: Failed to deserialize blocks 719600 to 719650 - io error:  - retrying (10 attempt(s) so far)
2024-10-17T08:46:34.017227Z DEBUG snarkos_node_cdn::blocks: Waiting for the first relevant blocks (1550 pending)
2024-10-17T08:46:34.290588Z  WARN snarkos_node_cdn::blocks: Maximum number of requests to https://blocks.aleo.org/mainnet/v0/719600.719650.blocks reached - shutting down...
2024-10-17T08:46:34.982415Z  DEBUGsnarkos_node_cdn::blocks: Finished network requests to the CDN
2024-10-17T08:46:35.018702Z  INFO snarkos_node_cdn::blocks: Stopping block sync at 719599 - shutting down

Steps to Reproduce

Run either run-client.sh file or run-validator.sh file

Expected Behavior

Sync node to latest block

Your Environment

snarkOS Version: mainnet-v1.0.0
Rust Version: 1.76.0
Computer OS: Ubuntu 22.04

[OuIChien]

I have the same issue

[vicsn]

Just tried it with some extra logging. Most block ranges are ~2.7 MB. Whatever we get back for block range 719600 - 719650 is only 243 bytes. The file might be corrupted or accidentally overwritten.

[sweet-tao]

I have the same problem, does anyone have a solution?

[raychu86]

@zosorock @alzger This looks like a possible routing issue in blocks.aleo.org.

Notice that - https://blocks.aleo.org/mainnet/v0/719600.719650.blocks returns an Access Denied response. Other block ranges are working fine.