Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1545 without dos and stats cleanups #2046

Open
wants to merge 299 commits into
base: v4.0.0
Choose a base branch
from

Conversation

kladkogex
Copy link
Collaborator

@kladkogex kladkogex commented Nov 25, 2024

State locking/ performance improvements for SKALEd.

  1. Use LevedlDB read-only snaps for all read-only JSON-RPC calls

In the past we had to constantly lock/unlock LevelDB state because it was used both during block processing
(write access) and JSON-RPC calls (read access)

Now we use the actual LevelDB only during block processing, so we do not need to lock it, since LevelDB will always be used by a single thread.

For read-only calls through JSON-RPC we use snapshots created at the end of each block.
This significantly improved performance and made calls much simpler, because we do not do state locking anymore.

image

Added SnapshotManager class that manages snapshots and closes them once they are not used anymore.

Since we already have a feature to reopen LevelDB during execution, added mechanics to close anbd reopen snapshot objects during reopen of the database (if you want to close database handle, the snapshot objects created for this handle need to be closed first)

  1. Simplified pending queue/broadcast and removed unnecessary locks.

Now when a new transaction comes from JSON-RPC it is immediately broadcast to other nodes and added to the pending queue. Previously a transaction would not be added to the blockproposal until it was broadcast, which was making things unnecessary complex and delaying transaction inclusion in the block.

  1. Increased default ZMQ outgoing queue size to 1024 transactions.

It was only 16 transactions before, which caused the queue to overfill and lose transactions during broadcast.

  1. Removed unnecessary transaction cache on transaction arrival. The cash did not improve performance but caused high memory consumption and memory leaks because it was never cleaned.

  2. Fixed a bug in partial catchup receipts storage, which caused large blocks processing to stuck for seconds and also could process the same block twice. Now for each transaction once it is processed the receipt is immediately written to LevelDB. If there is a crash and restart, only the unprocessed transactions in the block are processed. Added a test for partial catchup (previously there was no test)

  3. Added a set of parallel performance tests to test_eth that test ERC-20 token tranfers, native token transfers and all of major lots of JSON-RPC calls

  4. Changed m_rawData field in Transaction to shared pointer to speed up Transaction copying and save memory. Skaled copies Transaction objects all the time, shared pointer guarantees that copies do not consume additional memory.

  5. Did some small fixes to JSON-RPC API error handling to make it similar to Geth.

@kladkogex kladkogex self-assigned this Nov 25, 2024
@kladkogex kladkogex added this to the SKALE 4.0 milestone Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cleanup state locking and make JSON RPC calls like eth_call faster
1 participant