Version 1.2.12 slow syncing

[Crypto2]

Describe the bug
We just updated to 1.2.12 yesterday from 1.1.11 and installed the new blockchain as per directions from https://docs.waves.tech/en/waves-node/options-for-getting-actual-blockchain/state-downloading-and-applying

Since then syncing has been crawling, taking an hour to do 100-200 blocks. It is the only thing running on the system other than just the core Linux system (Debian 10.5 64-bit) and only about 50% of RAM is in use and practically no swap in use.

No sure what is going on. Maybe are there any changes to waves.conf from 1.1.x to 1.2.x that need to be made? Not seeing any errors on the log except the occasional:

2020-09-09 20:02:23,050 WARN [ecution-context-global-32] kamon.metric.Timer - Failed to record value [18984192277738] on [span.processing-time,{operation=block-appender,error=true}] because the value is outside of the configured range. The recorded value was adjusted to the highest trackable value [3600000000000]. You might need to change your dynamic range configuration for this metric

Expected behavior
Syncing at a much faster speed.

[msmolyakov]

Has your import process on 1.2.12 finished yet? How long did it take in total?

The kamon.metric.Timer errors are not related to the problem and are not the problem themselves. They will no longer be in the next release.

[Crypto2]

It's only up to 2245276 right now.

[msmolyakov]

So your node is on the current blockchain height

[Crypto2]

No it's still way behind.

For example I'm only on block 2248073 now and current on wavesexplorer is 2249822 so still over 24 hours behind even after running 11 days.

[msmolyakov]

Compare block headers /blocks/headers/at/2248073 of your node and official node, please. The id field values should be equal on the same height. Otherwise your node is most likely forked.

And could you send me log file /var/log/waves/waves.log of your node?

[Crypto2]

It's on the right chain, it's just syncing extremely slowly. Normally I keep the log off, I turned it on now and will post one later on.

[msmolyakov]

How is it going?

[Crypto2]

Not well, I've also tried it on a dedicated server with a lot more power as well and it's a bit faster but still crawling. I don't see any errors in the log or anything. Right now I'm just watching http://blockchain.wavesnodes.com/ daily and hope if I can load the next blockchain snapshot fast enough it might be able to catch up.

[Crypto2]

It has caught up now. 25 days to sync is pretty crazy :(

[msmolyakov]

So did you turn logs on? Could you send it?

[DrBlast]

Closed due to inactivity. Please feel free to reopen the issue if this problem is still actual.

[Crypto2]

It is definitely still slow, I updated to the latest about 24 hours ago since I was having rollback issues and saw there was an update. It was only 2000 blocks behind at that time and now it is 700 blocks behind still.

[Crypto2]

Def still an issue in version 1.3.6. I had to move Waves to a new server and downloaded the latest snapshot and it's as slow as ever syncing up from there :(

See the timestamps from the log:

2021-07-28 20:10:37,639 INFO  [appender-33] c.w.s.BlockchainUpdaterImpl - New height: 2697655
2021-07-28 20:11:05,236 INFO  [appender-33] c.w.s.BlockchainUpdaterImpl - New height: 2697656
2021-07-28 20:11:33,526 INFO  [appender-33] c.w.s.BlockchainUpdaterImpl - New height: 2697657
2021-07-28 20:12:01,877 INFO  [appender-33] c.w.s.BlockchainUpdaterImpl - New height: 2697658
2021-07-28 20:12:29,658 INFO  [appender-33] c.w.s.BlockchainUpdaterImpl - New height: 2697659
2021-07-28 20:12:57,790 INFO  [appender-33] c.w.s.BlockchainUpdaterImpl - New height: 2697660
2021-07-28 20:13:25,649 INFO  [appender-33] c.w.s.BlockchainUpdaterImpl - New height: 2697661
2021-07-28 20:13:52,638 INFO  [appender-33] c.w.s.BlockchainUpdaterImpl - New height: 2697662
2021-07-28 20:14:20,332 INFO  [appender-33] c.w.s.BlockchainUpdaterImpl - New height: 2697663

[msmolyakov]

hi, @Crypto2
As we asked earlier, could you send TRACE logs?
How to enable trace logging: https://docs.waves.tech/en/waves-node/logging-configuration