This repository has been archived by the owner on Oct 28, 2021. It is now read-only.
Create database performance tests #5300
Assignees
Labels
database
Database-related work
Comments
|
Some tooling notes:
I'd like to gain more insight into what client functionality and resources this test stresses...I don't see any comments in the file, @winsvega : Can you please provide me with some insight when you get a chance or point me to some documentation? |
|
Some additional notes:
|
|
I want to go with |
|
Blocks 5000000 to 5100000 is 2.2 GB. And I'm not sure will would be able to verify them. |
|
Also import replays all transactions and saves the state after each block into StateDB. So it's not so pure measurement of DB write performance (as there's also EVM execution). At the same time snapshot import mode writes 30000 recent blocks into database without replaying them (just checking PoW) and not requiring previous blocks. Maybe it can better represent DB write performance. |
|
Also a more general note: although the bottleneck of the sync performance is DB writes, I don't expect switching the DB engine and fine-tuning it it to solve the sync issues, the performance gains for the "sync with the main net" use case probably will be marginal. From my perspective there are two really significant issues with the sync:
With this in mind, my preference would be to go for the moment with the simplest approach with this performance testing framework, if we need it at all. |
|
I would not recommend to use BlockchainLoader. There is no blockchain tests filler documentation I could not provide you with it. Nobody really asced about the documentation thats why. So if you want I could guide you through blockchain test filler feautures and perhaps we write the documentation with this. |
|
could be related: |
|
Thank you for the information @winsvega ! No need for blockchain tests filler documentation yet, after chatting with chfast and gumb0 I think we're going to go in a different direction for the tests. I'll let you know if this changes though 😄 @chfast / @gumb0 : Following up on our chat this morning, I had some time so I gave snapshot download a go following the instructions here: https://github.com/ethereum/aleth/blob/master/doc/snapshot_sync.rst Unfortunately I don't think snapshot sync is working. The node listed in the docs refused connection attempts...console output: As a result, I chose the first node from the foundation.json file linked in the documentation and tried to connect to it - I think I established a successful connection but I don't think that a snapshot was downloaded. The snapshot directory that I supplied on the command line wasn't created and I didn't see any indication in the console output of a snapshot being downloaded. Here's an output snippet: It looks like the node that I connected to has a snapshot available:
However, I'm not sure why it wasn't downloaded. Perhaps the protocol has changed...@gumb0 : Any ideas? |
|
I debugged it a bit and couldn't yet understand why exactly it's happening, but this is something with boost.fiber on our side not working as expected on Windows. The Parity node you connect to has the snapshot and we get its manifest and then get the DAO fork block header successfully, but then the fiber that was expected to wake up when we give it the manifest, doesn't wake up. Anyway for now I'll share the snapshot I have, uploading it now, it's 2.76 GB |
|
Thanks, I'll file a new issue to track this in case we want to look into it later. I'll also try importing the snapshot tonight. |
Great, thanks! I downloaded and imported part of the snapshot, it contained 862 chunks and I imported the first 100. It took between 5 and 15 seconds to import each chunk (on my Lenovo X1 carbon gen 6 with Core i7, 16 GB ram, and SSD) which means that importing all 862 chunks would take (using an average import time of 10 seconds per chunk) of ~144 minutes or over 2 hours. I think it would be good to limit test execution time to a few minutes per iteration (perhaps something like <= 5 minutes?), is it possible to add functionality to only import the first N chunks of a snapshot? Or is snapshot import an all-or-nothing process, in which case maybe we should explore using aleth's chain export functionality to generate the test data? I know it's not ideal due to the inability to import just recent blocks, but it might be good enough to start with. |
|
There are two stages in snapshot import and 862 chunks I think is only the first one (much larger) part - importing the state. And yes, to be able to verify the following blocks you need to import the whole state. My original suggestion we discussed in the call was to import this snapshot once, so that we have the node (synced to block ~5m) on top of which we can then |
|
Let's use this as the first db peft test. |
|
@gumb0 : Right, I didn't realize though that importing a snapshot would take over 2 hours...I was thinking that we'd have tests which could be setup and ready for execution within a few minutes, with test execution itself also taking a few minutes. That makes running the tests more flexible and robust, i.e. we don't have to care about preserving any existing state...for example, in the case of snapshots, since importing one takes a long time we'll only want to do it once (unless we want to move to a newer snapshot e.g. new blocks on mainnet have significantly different usage patterns so we want to test them). I think we can do this by having these tests run against an existing DB i.e. they don't import a snapshot as a part of their setup and check the latest block in that DB...if the block number is < the latest block in the snapshot then we clear the database and reimport the snapshot. Regarding this, do you know if it's possible to programmatically check the number of the most recent block in a snapshot? No worries if not, I can just hard-code 5M in the tests otherwise. Another concern with importing an entire snapshot is memory usage - your uncompressed snapshot is over 3 GB and presumably the corresponding state trie and blocks consume more space after they've been imported...depending on memory usage and amount of available system memory, one might not be able to test the MemoryDB backend. Regarding rewinding the blockchain after each test iteration - I see that Lines 845 to 866 in 0a997db I'll take a look at it / give it a spin and see if we can use it in tests. I'll also do a full snapshot import when I get back from my trip to verify that there aren't any bugs we need to investigate. |
|
there is a testRPC to use rewind |
|
I changed my mind on this. Let's add rocksdb support first, this might be not trivial and take some time. Even switching leveldb version / build commands causes big issues: #5293. Focusing on performance tests is premature. |
|
Quick update on snapshot import - it worked correctly, the snapshot (https://drive.google.com/open?id=158LiPhDiNa-qfwdbjNMCP-Fuwr85iyMU) was successfully imported and my node is syncing with the blockchain (I'm killing the sync now before I run out of disk space but it's currently at block 5145635). |
|
No longer needed as we're now installing LevelDB from Hunter so we don't need support for other databases. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Since we're working on making Aleth compatible with different databases (e.g. RocksDB i.e. #4037, potentially SQLite), it would be helpful to have database performance tests so we can have quantitative benchmarks which measure how Aleth performs with each database backend. Some initial thoughts:
The approach should rather be to have tests which one can compile and run locally. The tests should load blocks from disk and not from the network (so they aren't impacted by network latency) and should be run against each database implementation - this can be achieved easily enough by iterating through the
DatabaseKindvalues defined indb.hand creating a newDBFactoryinstance for each one:aleth/libdevcore/DBFactory.h
Lines 30 to 34 in f838dd4
If the tests use
TestEththey should be marked such that they can easily be excluded from CircleCI runs (e.g. via a command-line switch - @chfast mentioned that TestEth already has a performance test option which we can leverage to achieve this).Finally, the tests should output a table summarizing the results for each database type so one can easily compare the results among the different databases.
The text was updated successfully, but these errors were encountered: