Acceleration of the balance bootstrapping process

[samuel-rufi]

Hi! is there a way (e.g. some rosetta-cli config option) to speed up the balance boostrapping process? Unfortunately, to bootstrap 85,000 balances it takes more than 10 hours (on a quite performant server).

To Reproduce

1. generate a big bootsrap_balances.json file with 85,000 accounts for example
2. start the rosetta-cli data check

Expected behavior
I assume that the bootstrap process is faster. IOTA nodes for example need to be boostrapped with the same data and it takes (when optimized) less than a minute.

Additional context
IOTA full-nodes are able to start up from a more recent block instead of having to synchronize from genesis. This can be achieved by booting the IOTA full-node with a so called snapshot. A snapshot represents the ledger state at a certain block.
Starting from that loaded state, the IOTA node then syncs with it's peers.
To test our implementation we need to boot rosetta-cli with the same snapshot data (therefore we do have such a big bootstrap_balances.json file).

[shrimalmadhur]

Hi @rufsam,

Is this happening even though you have a memory limit disabled setting turned off?

[samuel-rufi]

Hi @shrimalmadhur, I have tested with following configuration values:

{
  "network": {
    "blockchain": "iota",
    "network": "chrysalis-mainnet"
  },
  "online_url": "http://localhost:3030",
  "data_directory": "rosetta-cli-db",
  "http_timeout": 10,
  "max_retries": 5,
  "max_sync_concurrency": 5,
  "tip_delay": 5,
  "memory_limit_disabled": true,
  "data": {
    "historical_balance_enabled": false,
    "reconciliation_disabled": false,
    "active_reconciliation_concurrency": 5,
    "inactive_reconciliation_concurrency": 5,
    "pruning_disabled": false,
    "bootstrap_balances": "bootstrap_balances.json",
    "end_conditions": {
      "reconciliation_coverage": {
        "coverage": 0.95,
        "from_tip": true
      }
    },
    "start_index": 1823971
  }
}

Also with "memory_limit_disabled": false the balance bootstrapping process is slow, however I can't fully compare it because after loading around 35,000 accounts (#245) rosetta-cli breaks. But it definitely takes multiple hours to bootstrap these 35,000 accounts.

[shrimalmadhur]

I see. thanks for the info. Is your rosetta implementation public? If yes, I would like to repro this with your bootstrap file. It would also help me to check where we can it more efficient.

[samuel-rufi]

This would be really great. Many thanks @shrimalmadhur!
I have just produced a release with all the latest changes, you can find the repository here: https://github.com/iotaledger/rosetta-iota

If there is anything I can help you with, just tell me!