Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prevent an Application CONSENSUS_FAILURE #186

Open
ggarri opened this issue Jun 25, 2019 · 7 comments
Open

Prevent an Application CONSENSUS_FAILURE #186

ggarri opened this issue Jun 25, 2019 · 7 comments
Assignees
@ggarri
Labels
bug Something isn't working research

Comments

@ggarri
Copy link
Contributor

ggarri commented Jun 25, 2019 •
edited
Loading

Lightchain uses Ethereum as the blockchain storage and due to that our application need to compliant with Ethereum restrictions. One of them is timestamp equals parent's, two consecutive blocks cannot be done within the same second otherwise it fells in an error which causes the following log output and causes a consensus failure which cannot recover

Log output

########## BAD BLOCK #########
Chain config: {ChainID: 161 Homestead: <nil> DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: <nil>  ConstantinopleFix: <nil> Engine: unknown}

Number: 3791
Hash: 0x333aee86bedcf6199db4731d84ee9095ba519833fcf2ffa91c8c7dd45fbe6cfb


Error: timestamp equals parent's
##############################

How to reproduce it

To reproduce this issue in a safe manner we are going to use Standalone network and apply the consensus config values:

timeout_propose = "200ms"
timeout_propose_delta = "100ms"
timeout_prevote = "200ms"
timeout_prevote_delta = "100ms"
timeout_precommit = "200ms"
timeout_precommit_delta = "100ms"
timeout_commit = "200ms"

blocktime_iota = "100ms"

Change the genesis or the consensus params to use less than 1000ms

"consensus_params": {
    "block": {
      ...
      "time_iota_ms": 500"
    }
    ...

After that we will run the workload test as follow. Maybe it requires more than one try-out

$> cd truffle
$> npx truffle test ./test/04_workload.js --network=sirius

Sample wal logs of failure

ENDHEIGHT 448298
{"time":"2019-06-21T11:59:41.406323507Z","msg":{"type":"tendermint/wal/EventDataRoundState","value":{"height":"448299","round":"0","step":"RoundStepNewHeight"}}}
{"time":"2019-06-21T11:59:41.406484538Z","msg":{"type":"tendermint/wal/EventDataRoundState","value":{"height":"448299","round":"0","step":"RoundStepPropose"}}}
{"time":"2019-06-21T11:59:41.444441764Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448298","round":"0","block_id":{"hash":"E15AB95DC5A544E21598DD6C69EA6308ABA320026F373E186FF07439F1663D18","parts":{"total":"1","hash":"6734D65D34904CDEE71C1CC59A7262C4910701F5671FA5D763B06514D8495C6B"}},"timestamp":"2019-06-21T11:59:41.078920915Z","validator_address":"9A171D8706D6E09BE57326402D231B969DF5D872","validator_index":"2","signature":"+AJJuIGHAn9lz6brE587uFvwc82prtCKr8JsikJmRpmdYXX+FWSI28h5BXuxVw90a0SvKfYVhnGQuyx0DDrcAg=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:41.444472213Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448298","round":"0","block_id":{"hash":"E15AB95DC5A544E21598DD6C69EA6308ABA320026F373E186FF07439F1663D18","parts":{"total":"1","hash":"6734D65D34904CDEE71C1CC59A7262C4910701F5671FA5D763B06514D8495C6B"}},"timestamp":"2019-06-21T11:59:41.074198765Z","validator_address":"B7DFDC424F64EC89C42A502F89EB29F811144CA1","validator_index":"3","signature":"hh9lujKYoT9c8Qop1Dpo3vWOTfBdqGcGVQB9Nb4CsKMn28Mhw++31PB71jNeara+ZQt17hx23A5PC/egsJmxBA=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:41.641748682Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Proposal","value":{"Proposal":{"Type":32,"height":"448299","round":"0","pol_round":"-1","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.34699815Z","signature":"AiWXyGgx/6nmI88I7Ejzx9GXzZtbVoeq0br3650ED9LiL5wEv06aE8pcOTlyA6U76ZUOt00OTAurJzRPNuaAAw=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:41.643268279Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/BlockPart","value":{"Height":"448299","Round":"0","Part":{"index":"0","bytes":"EF090A86030A020809121B6C6967687473747265616D732D7369726975732D746573746E657418ABAE1B220B08AD8DB3E80510EDDDB023280230FB673A480A20E15AB95DC5A544E21598DD6C69EA6308ABA320026F373E186FF07439F1663D181224080112206734D65D34904CDEE71C1CC59A7262C4910701F5671FA5D763B06514D8495C6B42206C4B72A2B811AE3227BD8FF74E6B218CE43D0BA222AD7A42DBF18E47C289EF0F4A204A2E431B117BF2493DE893A089E94A97991909C81BAAC1244AA6469096947F825220C3E72F3070D899832F659459A2EDBCFD036AD716C0D143D689DFDA3C0E0E4CF25A20C3E72F3070D899832F659459A2EDBCFD036AD716C0D143D689DFDA3C0E0E4CF26220048091BC7DDC283F77BFBF91D73C44DA58C3DF8A9CBC867405D8B7F3DAADA22F6A2022F282C2B23F3093DFA843F158A76DA28DEBC63B5AFFEFAD7E5DBC3B7E3930027220F857106116E31EA0951D99C76A1F0E19C5ADEA0A92791B38670D8D1B2D8201508201149A171D8706D6E09BE57326402D231B969DF5D87212E8010A72F8708227D685746A528800825208943263766A8C3B1A3FEA3937EE4B0CA01242E715B988016345785D8A000080820167A0E75E35D9BD575391F85310E0F16AE9415DF7DCF51B59E3683D16BD89EB3E216EA07903E9272C3C889C7DDCDEE28E083A67BAED2CEAA2F1EDE83915F43BA609DAB80A72F8708227D785746A528800825208943263766A8C3B1A3FEA3937EE4B0CA01242E715B988016345785D8A000080820168A0DD42CD272F6C7C5097D3D2DB827D8A1FD3A4D23ADD85410BB2B07A65B83091D5A0313941B14764307A1CBD7DFC4B5DF0AE392BE6812CAC10C3DDC484071BE155E222F8040A480A20E15AB95DC5A544E21598DD6C69EA6308ABA320026F373E186FF07439F1663D181224080112206734D65D34904CDEE71C1CC59A7262C4910701F5671FA5D763B06514D8495C6B12B501080210AAAE1B22480A20E15AB95DC5A544E21598DD6C69EA6308ABA320026F373E186FF07439F1663D181224080112206734D65D34904CDEE71C1CC59A7262C4910701F5671FA5D763B06514D8495C6B2A0B08AD8DB3E80510F49DF022321421B62C962F60618DB4DA34CD623C39FC7CC35CE642403ADB90D9D278A8DD8AAB98E29D81107A9E7AB221E89F33CD4EF7F191460CBB89ECDEE72B9B49BAA56F114A2D67FB2AB339F8C26BBA9EEA7C04D9EADA647EDA03120012B701080210AAAE1B22480A20E15AB95DC5A544E21598DD6C69EA6308ABA320026F373E186FF07439F1663D181224080112206734D65D34904CDEE71C1CC59A7262C4910701F5671FA5D763B06514D8495C6B2A0B08AD8DB3E80510D3F9D02532149A171D8706D6E09BE57326402D231B969DF5D87238024240F80249B88187027F65CFA6EB139F3BB85BF073CDA9AED08AAFC26C8A426646999D6175FE156488DBC879057BB1570F746B44AF29F615867190BB2C740C3ADC0212B701080210AAAE1B22480A20E15AB95DC5A544E21598DD6C69EA6308ABA320026F373E186FF07439F1663D181224080112206734D65D34904CDEE71C1CC59A7262C4910701F5671FA5D763B06514D8495C6B2A0B08AD8DB3E80510EDDDB0233214B7DFDC424F64EC89C42A502F89EB29F811144CA138034240861F65BA3298A13F5CF10A29D43A68DEF58E4DF05DA8670655007D35BE02B0A327DBC321C3EFB7D4F07BD6335E6AB6BE650B75EE1C76DC0E4F0BF7A0B099B104","proof":{"total":"1","index":"0","leaf_hash":"uO9gMB4RX8LA2zfc105fndtGvUsGDHBnJsq71AMItDo=","aunts":null}}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:41.644737157Z","msg":{"type":"tendermint/wal/EventDataRoundState","value":{"height":"448299","round":"0","step":"RoundStepPrevote"}}}
{"time":"2019-06-21T11:59:41.64478298Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":1,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.385770115Z","validator_address":"9A171D8706D6E09BE57326402D231B969DF5D872","validator_index":"2","signature":"b+dkEayqYjuSHy/J0manKxQpD3ThMYtwaI/Mk+aP6ngWl3mbCMK2a+vsgdwubAMI1e96ajk6JMdS2soN1Y3BCw=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:41.668486684Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Proposal","value":{"Proposal":{"Type":32,"height":"448299","round":"0","pol_round":"-1","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.34699815Z","signature":"AiWXyGgx/6nmI88I7Ejzx9GXzZtbVoeq0br3650ED9LiL5wEv06aE8pcOTlyA6U76ZUOt00OTAurJzRPNuaAAw=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:41.669107042Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/BlockPart","value":{"Height":"448299","Round":"0","Part":{"index":"0","bytes":"EF090A86030A020809121B6C6967687473747265616D732D7369726975732D746573746E657418ABAE1B220B08AD8DB3E80510EDDDB023280230FB673A480A20E15AB95DC5A544E21598DD6C69EA6308ABA320026F373E186FF07439F1663D181224080112206734D65D34904CDEE71C1CC59A7262C4910701F5671FA5D763B06514D8495C6B42206C4B72A2B811AE3227BD8FF74E6B218CE43D0BA222AD7A42DBF18E47C289EF0F4A204A2E431B117BF2493DE893A089E94A97991909C81BAAC1244AA6469096947F825220C3E72F3070D899832F659459A2EDBCFD036AD716C0D143D689DFDA3C0E0E4CF25A20C3E72F3070D899832F659459A2EDBCFD036AD716C0D143D689DFDA3C0E0E4CF26220048091BC7DDC283F77BFBF91D73C44DA58C3DF8A9CBC867405D8B7F3DAADA22F6A2022F282C2B23F3093DFA843F158A76DA28DEBC63B5AFFEFAD7E5DBC3B7E3930027220F857106116E31EA0951D99C76A1F0E19C5ADEA0A92791B38670D8D1B2D8201508201149A171D8706D6E09BE57326402D231B969DF5D87212E8010A72F8708227D685746A528800825208943263766A8C3B1A3FEA3937EE4B0CA01242E715B988016345785D8A000080820167A0E75E35D9BD575391F85310E0F16AE9415DF7DCF51B59E3683D16BD89EB3E216EA07903E9272C3C889C7DDCDEE28E083A67BAED2CEAA2F1EDE83915F43BA609DAB80A72F8708227D785746A528800825208943263766A8C3B1A3FEA3937EE4B0CA01242E715B988016345785D8A000080820168A0DD42CD272F6C7C5097D3D2DB827D8A1FD3A4D23ADD85410BB2B07A65B83091D5A0313941B14764307A1CBD7DFC4B5DF0AE392BE6812CAC10C3DDC484071BE155E222F8040A480A20E15AB95DC5A544E21598DD6C69EA6308ABA320026F373E186FF07439F1663D181224080112206734D65D34904CDEE71C1CC59A7262C4910701F5671FA5D763B06514D8495C6B12B501080210AAAE1B22480A20E15AB95DC5A544E21598DD6C69EA6308ABA320026F373E186FF07439F1663D181224080112206734D65D34904CDEE71C1CC59A7262C4910701F5671FA5D763B06514D8495C6B2A0B08AD8DB3E80510F49DF022321421B62C962F60618DB4DA34CD623C39FC7CC35CE642403ADB90D9D278A8DD8AAB98E29D81107A9E7AB221E89F33CD4EF7F191460CBB89ECDEE72B9B49BAA56F114A2D67FB2AB339F8C26BBA9EEA7C04D9EADA647EDA03120012B701080210AAAE1B22480A20E15AB95DC5A544E21598DD6C69EA6308ABA320026F373E186FF07439F1663D181224080112206734D65D34904CDEE71C1CC59A7262C4910701F5671FA5D763B06514D8495C6B2A0B08AD8DB3E80510D3F9D02532149A171D8706D6E09BE57326402D231B969DF5D87238024240F80249B88187027F65CFA6EB139F3BB85BF073CDA9AED08AAFC26C8A426646999D6175FE156488DBC879057BB1570F746B44AF29F615867190BB2C740C3ADC0212B701080210AAAE1B22480A20E15AB95DC5A544E21598DD6C69EA6308ABA320026F373E186FF07439F1663D181224080112206734D65D34904CDEE71C1CC59A7262C4910701F5671FA5D763B06514D8495C6B2A0B08AD8DB3E80510EDDDB0233214B7DFDC424F64EC89C42A502F89EB29F811144CA138034240861F65BA3298A13F5CF10A29D43A68DEF58E4DF05DA8670655007D35BE02B0A327DBC321C3EFB7D4F07BD6335E6AB6BE650B75EE1C76DC0E4F0BF7A0B099B104","proof":{"total":"1","index":"0","leaf_hash":"uO9gMB4RX8LA2zfc105fndtGvUsGDHBnJsq71AMItDo=","aunts":null}}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:41.669136647Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":1,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.385770115Z","validator_address":"9A171D8706D6E09BE57326402D231B969DF5D872","validator_index":"2","signature":"b+dkEayqYjuSHy/J0manKxQpD3ThMYtwaI/Mk+aP6ngWl3mbCMK2a+vsgdwubAMI1e96ajk6JMdS2soN1Y3BCw=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:41.74305026Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":1,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.446630258Z","validator_address":"2DE3B810E4EAC51A10A3740D15AE92142B01DC7B","validator_index":"1","signature":"874aeWS9acK93V38v/KNiyyTJgHdcUCPX596L+MxG5Zce5crRPrTHPUN8LhmpNqBDe1sZWY2iuyD00GpljcuDw=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:41.743504856Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":1,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.541968992Z","validator_address":"B7DFDC424F64EC89C42A502F89EB29F811144CA1","validator_index":"3","signature":"536TxvR2lZ2pnVPtioEgvhEew/cZAtCPxDmo6y6IygvUWk54NqxZrftGHSqQCCbKPKd2v9Ahxu1ENe9645QGAA=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:41.744813425Z","msg":{"type":"tendermint/wal/EventDataRoundState","value":{"height":"448299","round":"0","step":"RoundStepPrecommit"}}}
{"time":"2019-06-21T11:59:41.776091151Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":1,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.541968992Z","validator_address":"B7DFDC424F64EC89C42A502F89EB29F811144CA1","validator_index":"3","signature":"536TxvR2lZ2pnVPtioEgvhEew/cZAtCPxDmo6y6IygvUWk54NqxZrftGHSqQCCbKPKd2v9Ahxu1ENe9645QGAA=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:41.776139098Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":1,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.446630258Z","validator_address":"2DE3B810E4EAC51A10A3740D15AE92142B01DC7B","validator_index":"1","signature":"874aeWS9acK93V38v/KNiyyTJgHdcUCPX596L+MxG5Zce5crRPrTHPUN8LhmpNqBDe1sZWY2iuyD00GpljcuDw=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:41.776173233Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":1,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.477346702Z","validator_address":"21B62C962F60618DB4DA34CD623C39FC7CC35CE6","validator_index":"0","signature":"AKXzbungRmppwVnX6+0KVukk4IYNk4lBIpvFzbH4sH3OyGD1FWk9TOwN9xS+504zKjmVw5KAfa9HftyO++fJAg=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:41.77663795Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.621270427Z","validator_address":"9A171D8706D6E09BE57326402D231B969DF5D872","validator_index":"2","signature":"mHir9bufnXo+q/AL31FgD1O7xxotKC1DphOTxIRPFD5SfA2/R/1g9jP3hUVzDaYX0NSnyffLjAcmKQQOC6r3CA=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:41.849440883Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":1,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.477346702Z","validator_address":"21B62C962F60618DB4DA34CD623C39FC7CC35CE6","validator_index":"0","signature":"AKXzbungRmppwVnX6+0KVukk4IYNk4lBIpvFzbH4sH3OyGD1FWk9TOwN9xS+504zKjmVw5KAfa9HftyO++fJAg=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:41.849490956Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.667929937Z","validator_address":"B7DFDC424F64EC89C42A502F89EB29F811144CA1","validator_index":"3","signature":"KCrpIpoEAp9rxfrhgnbNEG8NTttucwEObh9WxaBKau3CcZhk3YA6uQWpaDMJjAM2YgzjXg8NmwqDPevKoj7xBg=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:41.947753927Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.667929937Z","validator_address":"B7DFDC424F64EC89C42A502F89EB29F811144CA1","validator_index":"3","signature":"KCrpIpoEAp9rxfrhgnbNEG8NTttucwEObh9WxaBKau3CcZhk3YA6uQWpaDMJjAM2YgzjXg8NmwqDPevKoj7xBg=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:41.947809644Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.666971376Z","validator_address":"21B62C962F60618DB4DA34CD623C39FC7CC35CE6","validator_index":"0","signature":"X0B2xHaFnM9uQmXaafa7KaDtdhRn0+lfA1TB7DqO2YtQlqWlS3eMLunWPnG5MZxqtDxPaaE2Wefv5RDCkS46BQ=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:41.948452822Z","msg":{"type":"tendermint/wal/EventDataRoundState","value":{"height":"448299","round":"0","step":"RoundStepCommit"}}}
{"time":"2019-06-21T11:59:41.959873958Z","msg":{"type":"tendermint/wal/EndHeightMessage","value":{"height":"448299"}}}
ENDHEIGHT 448299
{"time":"2019-06-21T11:59:41.989125025Z","msg":{"type":"tendermint/wal/EventDataRoundState","value":{"height":"448300","round":"0","step":"RoundStepNewHeight"}}}
{"time":"2019-06-21T11:59:41.989308207Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.666971376Z","validator_address":"21B62C962F60618DB4DA34CD623C39FC7CC35CE6","validator_index":"0","signature":"X0B2xHaFnM9uQmXaafa7KaDtdhRn0+lfA1TB7DqO2YtQlqWlS3eMLunWPnG5MZxqtDxPaaE2Wefv5RDCkS46BQ=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:41.989416487Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.621270427Z","validator_address":"9A171D8706D6E09BE57326402D231B969DF5D872","validator_index":"2","signature":"mHir9bufnXo+q/AL31FgD1O7xxotKC1DphOTxIRPFD5SfA2/R/1g9jP3hUVzDaYX0NSnyffLjAcmKQQOC6r3CA=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:42.047295403Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.661372736Z","validator_address":"2DE3B810E4EAC51A10A3740D15AE92142B01DC7B","validator_index":"1","signature":"4ORVRXi4mnLWWQjBl66bCKiLWjl6lVWPWSxOrCUzp5na25IdHffhztULMkwghvz7SwIA5wzd5Bq0okEFWJy7Cw=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:42.068075388Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.661372736Z","validator_address":"2DE3B810E4EAC51A10A3740D15AE92142B01DC7B","validator_index":"1","signature":"4ORVRXi4mnLWWQjBl66bCKiLWjl6lVWPWSxOrCUzp5na25IdHffhztULMkwghvz7SwIA5wzd5Bq0okEFWJy7Cw=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:42.147387944Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.666971376Z","validator_address":"21B62C962F60618DB4DA34CD623C39FC7CC35CE6","validator_index":"0","signature":"X0B2xHaFnM9uQmXaafa7KaDtdhRn0+lfA1TB7DqO2YtQlqWlS3eMLunWPnG5MZxqtDxPaaE2Wefv5RDCkS46BQ=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:42.147447619Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.667929937Z","validator_address":"B7DFDC424F64EC89C42A502F89EB29F811144CA1","validator_index":"3","signature":"KCrpIpoEAp9rxfrhgnbNEG8NTttucwEObh9WxaBKau3CcZhk3YA6uQWpaDMJjAM2YgzjXg8NmwqDPevKoj7xBg=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:42.147481373Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.661372736Z","validator_address":"2DE3B810E4EAC51A10A3740D15AE92142B01DC7B","validator_index":"1","signature":"4ORVRXi4mnLWWQjBl66bCKiLWjl6lVWPWSxOrCUzp5na25IdHffhztULMkwghvz7SwIA5wzd5Bq0okEFWJy7Cw=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:42.182560809Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.666971376Z","validator_address":"21B62C962F60618DB4DA34CD623C39FC7CC35CE6","validator_index":"0","signature":"X0B2xHaFnM9uQmXaafa7KaDtdhRn0+lfA1TB7DqO2YtQlqWlS3eMLunWPnG5MZxqtDxPaaE2Wefv5RDCkS46BQ=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:42.182624221Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.621270427Z","validator_address":"9A171D8706D6E09BE57326402D231B969DF5D872","validator_index":"2","signature":"mHir9bufnXo+q/AL31FgD1O7xxotKC1DphOTxIRPFD5SfA2/R/1g9jP3hUVzDaYX0NSnyffLjAcmKQQOC6r3CA=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:42.182657127Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448299","round":"0","block_id":{"hash":"7D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7","parts":{"total":"1","hash":"B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A"}},"timestamp":"2019-06-21T11:59:41.661372736Z","validator_address":"2DE3B810E4EAC51A10A3740D15AE92142B01DC7B","validator_index":"1","signature":"4ORVRXi4mnLWWQjBl66bCKiLWjl6lVWPWSxOrCUzp5na25IdHffhztULMkwghvz7SwIA5wzd5Bq0okEFWJy7Cw=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:44.053397784Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Proposal","value":{"Proposal":{"Type":32,"height":"448300","round":"0","pol_round":"-1","block_id":{"hash":"01169C113CA88525730A423BE536BB12077DAC325C7658ECAF20137EC67B86D9","parts":{"total":"1","hash":"B43AAEADEB16E22C32B00D75AB2381B51B3D1D647FB92C8EEFB431D5955C2755"}},"timestamp":"2019-06-21T11:59:43.815315009Z","signature":"Z23GPmTa7fF0OJJx/7kWoqDBTidSYKo8DR97E+IYgbCImju0zsUJFDRiUulj8qKrauCtrFsLeiwNn+OCQuqFBw=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:44.054428422Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/BlockPart","value":{"Height":"448300","Round":"0","Part":{"index":"0","bytes":"9D090AE3020A020809121B6C6967687473747265616D732D7369726975732D746573746E657418ACAE1B220C08AD8DB3E80510C0FEAEBB0230FB673A480A207D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7122408011220B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A42205EAE6E7D6DD8760C2C0CB89E45C5861F700A2B06F4F22740FD828B37F8C4AFCA5220C3E72F3070D899832F659459A2EDBCFD036AD716C0D143D689DFDA3C0E0E4CF25A20C3E72F3070D899832F659459A2EDBCFD036AD716C0D143D689DFDA3C0E0E4CF26220048091BC7DDC283F77BFBF91D73C44DA58C3DF8A9CBC867405D8B7F3DAADA22F6A200ED81ACAAD13F7777977857F24B80CD0B9232CE00833ED85CE7DFFDC9974D03E7220FE43D66AFA4A9A5C4F9C9DA89F4FFB52635C8F342E7FFB731D68E36C5982072A820114B7DFDC424F64EC89C42A502F89EB29F811144CA122B4060A480A207D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7122408011220B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A12B601080210ABAE1B22480A207D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7122408011220B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A2A0C08AD8DB3E80510F0D984BE02321421B62C962F60618DB4DA34CD623C39FC7CC35CE642405F4076C476859CCF6E4265DA69F6BB29A0ED761467D3E95F0354C1EC3A8ED98B5096A5A54B778C2EE9D63E71B9319C6AB43C4F69A13659E7EFE510C2912E3A0512B801080210ABAE1B22480A207D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7122408011220B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A2A0C08AD8DB3E80510C0FEAEBB0232142DE3B810E4EAC51A10A3740D15AE92142B01DC7B38014240E0E4554578B89A72D65908C197AE9B08A88B5A397A95558F592C4EAC2533A799DADB921D1DF7E1CED50B324C2086FCFB4B0200E70CDDE41AB4A24105589CBB0B12B801080210ABAE1B22480A207D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7122408011220B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A2A0C08AD8DB3E805109BAB9FA80232149A171D8706D6E09BE57326402D231B969DF5D872380242409878ABF5BB9F9D7A3EABF00BDF51600F53BBC71A2D282D43A61393C4844F143E527C0DBF47FD60F633F78545730DA617D0D4A7C9F7CB8C072629040E0BAAF70812B801080210ABAE1B22480A207D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7122408011220B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A2A0C08AD8DB3E80510D19ABFBE023214B7DFDC424F64EC89C42A502F89EB29F811144CA138034240282AE9229A04029F6BC5FAE18276CD106F0D4EDB6E73010E6E1F56C5A04A6AEDC2719864DD803AB905A96833098C0336620CE35E0F0D9B0A833DEBCAA23EF106","proof":{"total":"1","index":"0","leaf_hash":"tDquresW4iwysA11qyOBtRs9HWR/uSyO77Qx1ZVcJ1U=","aunts":null}}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:44.057520696Z","msg":{"type":"tendermint/wal/EventDataRoundState","value":{"height":"448300","round":"0","step":"RoundStepPrevote"}}}
{"time":"2019-06-21T11:59:44.057602793Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":1,"height":"448300","round":"0","block_id":{"hash":"01169C113CA88525730A423BE536BB12077DAC325C7658ECAF20137EC67B86D9","parts":{"total":"1","hash":"B43AAEADEB16E22C32B00D75AB2381B51B3D1D647FB92C8EEFB431D5955C2755"}},"timestamp":"2019-06-21T11:59:43.847405194Z","validator_address":"B7DFDC424F64EC89C42A502F89EB29F811144CA1","validator_index":"3","signature":"NMyg7ndoegNj6VAwoJnRRvQDAQEtNDiD23fJOPbU1b1SJulSPU6WPaEl3xizgaS0y19leKrSxFs616jQ1VBECA=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:44.087915814Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":1,"height":"448300","round":"0","block_id":{"hash":"01169C113CA88525730A423BE536BB12077DAC325C7658ECAF20137EC67B86D9","parts":{"total":"1","hash":"B43AAEADEB16E22C32B00D75AB2381B51B3D1D647FB92C8EEFB431D5955C2755"}},"timestamp":"2019-06-21T11:59:43.847405194Z","validator_address":"B7DFDC424F64EC89C42A502F89EB29F811144CA1","validator_index":"3","signature":"NMyg7ndoegNj6VAwoJnRRvQDAQEtNDiD23fJOPbU1b1SJulSPU6WPaEl3xizgaS0y19leKrSxFs616jQ1VBECA=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:44.087953756Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Proposal","value":{"Proposal":{"Type":32,"height":"448300","round":"0","pol_round":"-1","block_id":{"hash":"01169C113CA88525730A423BE536BB12077DAC325C7658ECAF20137EC67B86D9","parts":{"total":"1","hash":"B43AAEADEB16E22C32B00D75AB2381B51B3D1D647FB92C8EEFB431D5955C2755"}},"timestamp":"2019-06-21T11:59:43.815315009Z","signature":"Z23GPmTa7fF0OJJx/7kWoqDBTidSYKo8DR97E+IYgbCImju0zsUJFDRiUulj8qKrauCtrFsLeiwNn+OCQuqFBw=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:44.088604104Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/BlockPart","value":{"Height":"448300","Round":"0","Part":{"index":"0","bytes":"9D090AE3020A020809121B6C6967687473747265616D732D7369726975732D746573746E657418ACAE1B220C08AD8DB3E80510C0FEAEBB0230FB673A480A207D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7122408011220B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A42205EAE6E7D6DD8760C2C0CB89E45C5861F700A2B06F4F22740FD828B37F8C4AFCA5220C3E72F3070D899832F659459A2EDBCFD036AD716C0D143D689DFDA3C0E0E4CF25A20C3E72F3070D899832F659459A2EDBCFD036AD716C0D143D689DFDA3C0E0E4CF26220048091BC7DDC283F77BFBF91D73C44DA58C3DF8A9CBC867405D8B7F3DAADA22F6A200ED81ACAAD13F7777977857F24B80CD0B9232CE00833ED85CE7DFFDC9974D03E7220FE43D66AFA4A9A5C4F9C9DA89F4FFB52635C8F342E7FFB731D68E36C5982072A820114B7DFDC424F64EC89C42A502F89EB29F811144CA122B4060A480A207D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7122408011220B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A12B601080210ABAE1B22480A207D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7122408011220B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A2A0C08AD8DB3E80510F0D984BE02321421B62C962F60618DB4DA34CD623C39FC7CC35CE642405F4076C476859CCF6E4265DA69F6BB29A0ED761467D3E95F0354C1EC3A8ED98B5096A5A54B778C2EE9D63E71B9319C6AB43C4F69A13659E7EFE510C2912E3A0512B801080210ABAE1B22480A207D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7122408011220B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A2A0C08AD8DB3E80510C0FEAEBB0232142DE3B810E4EAC51A10A3740D15AE92142B01DC7B38014240E0E4554578B89A72D65908C197AE9B08A88B5A397A95558F592C4EAC2533A799DADB921D1DF7E1CED50B324C2086FCFB4B0200E70CDDE41AB4A24105589CBB0B12B801080210ABAE1B22480A207D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7122408011220B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A2A0C08AD8DB3E805109BAB9FA80232149A171D8706D6E09BE57326402D231B969DF5D872380242409878ABF5BB9F9D7A3EABF00BDF51600F53BBC71A2D282D43A61393C4844F143E527C0DBF47FD60F633F78545730DA617D0D4A7C9F7CB8C072629040E0BAAF70812B801080210ABAE1B22480A207D58F95FA33ECD9646BE29500351495651FBB686A99DB3A81FAAB43B979359A7122408011220B8EF60301E115FC2C0DB37DCD74E5F9DDB46BD4B060C706726CABBD40308B43A2A0C08AD8DB3E80510D19ABFBE023214B7DFDC424F64EC89C42A502F89EB29F811144CA138034240282AE9229A04029F6BC5FAE18276CD106F0D4EDB6E73010E6E1F56C5A04A6AEDC2719864DD803AB905A96833098C0336620CE35E0F0D9B0A833DEBCAA23EF106","proof":{"total":"1","index":"0","leaf_hash":"tDquresW4iwysA11qyOBtRs9HWR/uSyO77Qx1ZVcJ1U=","aunts":null}}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:44.160507242Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":1,"height":"448300","round":"0","block_id":{"hash":"01169C113CA88525730A423BE536BB12077DAC325C7658ECAF20137EC67B86D9","parts":{"total":"1","hash":"B43AAEADEB16E22C32B00D75AB2381B51B3D1D647FB92C8EEFB431D5955C2755"}},"timestamp":"2019-06-21T11:59:43.937531454Z","validator_address":"2DE3B810E4EAC51A10A3740D15AE92142B01DC7B","validator_index":"1","signature":"NPwp+IYzcqE+y33yFTt5/RybFGjIR22NBSwm69Z2hu3ZjbkpyXS+SrwZgJ2tzbPkY+MHZuZDbi6TYi00plzAAw=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:44.187233166Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":1,"height":"448300","round":"0","block_id":{"hash":"01169C113CA88525730A423BE536BB12077DAC325C7658ECAF20137EC67B86D9","parts":{"total":"1","hash":"B43AAEADEB16E22C32B00D75AB2381B51B3D1D647FB92C8EEFB431D5955C2755"}},"timestamp":"2019-06-21T11:59:43.937531454Z","validator_address":"2DE3B810E4EAC51A10A3740D15AE92142B01DC7B","validator_index":"1","signature":"NPwp+IYzcqE+y33yFTt5/RybFGjIR22NBSwm69Z2hu3ZjbkpyXS+SrwZgJ2tzbPkY+MHZuZDbi6TYi00plzAAw=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:44.240545631Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":1,"height":"448300","round":"0","block_id":{"hash":"01169C113CA88525730A423BE536BB12077DAC325C7658ECAF20137EC67B86D9","parts":{"total":"1","hash":"B43AAEADEB16E22C32B00D75AB2381B51B3D1D647FB92C8EEFB431D5955C2755"}},"timestamp":"2019-06-21T11:59:43.945692557Z","validator_address":"9A171D8706D6E09BE57326402D231B969DF5D872","validator_index":"2","signature":"QTFyGqXafUEniEPqbF8jAQke0jv+mhfAyjuk2V95Wzj0ZLfg7L1eS5vMzqdAEjitBbLmj8GAQFBZl6f4zOTEDg=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:44.244584355Z","msg":{"type":"tendermint/wal/EventDataRoundState","value":{"height":"448300","round":"0","step":"RoundStepPrecommit"}}}
{"time":"2019-06-21T11:59:44.244691265Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":1,"height":"448300","round":"0","block_id":{"hash":"01169C113CA88525730A423BE536BB12077DAC325C7658ECAF20137EC67B86D9","parts":{"total":"1","hash":"B43AAEADEB16E22C32B00D75AB2381B51B3D1D647FB92C8EEFB431D5955C2755"}},"timestamp":"2019-06-21T11:59:43.940383434Z","validator_address":"21B62C962F60618DB4DA34CD623C39FC7CC35CE6","validator_index":"0","signature":"ulIQCERW1swVj3xUUFSAWFDq1HbfgCxMgs8fVg9OH1ALC8Ad1qO1+33jQIgp4V7VnPJu/+j4k3JbvlEPXP75BQ=="}}},"peer_key":"0f04a32ffd95529c277e92aa9f006cc613595e54"}}}
{"time":"2019-06-21T11:59:44.260532289Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":1,"height":"448300","round":"0","block_id":{"hash":"01169C113CA88525730A423BE536BB12077DAC325C7658ECAF20137EC67B86D9","parts":{"total":"1","hash":"B43AAEADEB16E22C32B00D75AB2381B51B3D1D647FB92C8EEFB431D5955C2755"}},"timestamp":"2019-06-21T11:59:43.940383434Z","validator_address":"21B62C962F60618DB4DA34CD623C39FC7CC35CE6","validator_index":"0","signature":"ulIQCERW1swVj3xUUFSAWFDq1HbfgCxMgs8fVg9OH1ALC8Ad1qO1+33jQIgp4V7VnPJu/+j4k3JbvlEPXP75BQ=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:44.260653732Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":1,"height":"448300","round":"0","block_id":{"hash":"01169C113CA88525730A423BE536BB12077DAC325C7658ECAF20137EC67B86D9","parts":{"total":"1","hash":"B43AAEADEB16E22C32B00D75AB2381B51B3D1D647FB92C8EEFB431D5955C2755"}},"timestamp":"2019-06-21T11:59:43.945692557Z","validator_address":"9A171D8706D6E09BE57326402D231B969DF5D872","validator_index":"2","signature":"QTFyGqXafUEniEPqbF8jAQke0jv+mhfAyjuk2V95Wzj0ZLfg7L1eS5vMzqdAEjitBbLmj8GAQFBZl6f4zOTEDg=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:44.375952813Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448300","round":"0","block_id":{"hash":"01169C113CA88525730A423BE536BB12077DAC325C7658ECAF20137EC67B86D9","parts":{"total":"1","hash":"B43AAEADEB16E22C32B00D75AB2381B51B3D1D647FB92C8EEFB431D5955C2755"}},"timestamp":"2019-06-21T11:59:44.046891738Z","validator_address":"21B62C962F60618DB4DA34CD623C39FC7CC35CE6","validator_index":"0","signature":"uXspb/h0SAmCM5MF77v90e7uJcoDvbb5z/yygy+BqMzlhxepTvUPl22pobGZJnpDbnJ4hFrpqzxdkQK6AqbTCA=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:44.377366476Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448300","round":"0","block_id":{"hash":"01169C113CA88525730A423BE536BB12077DAC325C7658ECAF20137EC67B86D9","parts":{"total":"1","hash":"B43AAEADEB16E22C32B00D75AB2381B51B3D1D647FB92C8EEFB431D5955C2755"}},"timestamp":"2019-06-21T11:59:44.145278903Z","validator_address":"2DE3B810E4EAC51A10A3740D15AE92142B01DC7B","validator_index":"1","signature":"IZpIojTxVsDEOwZyXDip4QLjvGR82yMHStdT6iIAHWJiLxMVun+J4AYYw0DSBceNF1LVF4LS5gznRUKNXJdfCw=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:44.378404538Z","msg":{"type":"tendermint/wal/MsgInfo","value":{"msg":{"type":"tendermint/Vote","value":{"Vote":{"type":2,"height":"448300","round":"0","block_id":{"hash":"01169C113CA88525730A423BE536BB12077DAC325C7658ECAF20137EC67B86D9","parts":{"total":"1","hash":"B43AAEADEB16E22C32B00D75AB2381B51B3D1D647FB92C8EEFB431D5955C2755"}},"timestamp":"2019-06-21T11:59:44.158962352Z","validator_address":"B7DFDC424F64EC89C42A502F89EB29F811144CA1","validator_index":"3","signature":"MjVgfbUmSc/VNVDh4R0yqwEeg9FW3RY5fZsJ3a6dIZeLEHtedZZSVdz/JwhrJxzavGyckuNVkIPPYVzo7Dl8Bw=="}}},"peer_key":"4172cede3e68412577965d0655597b9a7f235820"}}}
{"time":"2019-06-21T11:59:44.379671073Z","msg":{"type":"tendermint/wal/EventDataRoundState","value":{"height":"448300","round":"0","step":"RoundStepCommit"}}}
{"time":"2019-06-21T11:59:44.395199287Z","msg":{"type":"tendermint/wal/EndHeightMessage","value":{"height":"448300"}}}
ENDHEIGHT 448300
ggarri added a commit that referenced this issue Jun 26, 2019
@ggarri
#186 Logs copy of standalone error
78b3a3b
ggarri added a commit that referenced this issue Jun 26, 2019
@ggarri
#186 Fix
da8980b
@ggarri
Copy link
Contributor Author

ggarri commented Jun 27, 2019

The population of Tendermint Headers happen in the following part of Tendermint code.

github.com/tendermint/tendermint/state/state.go:MakeBlock()

if height == 1 {
    timestamp = state.LastBlockTime // genesis time
} else {
    timestamp = MedianTime(commit, state.LastValidators)
}

// Fill rest of header with state data.
block.Header.Populate(
	state.Version.Consensus, state.ChainID,
	timestamp, state.LastBlockID, state.LastBlockTotalTx+block.NumTxs,
	state.Validators.Hash(), state.NextValidators.Hash(),
	state.ConsensusParams.Hash(), state.AppHash, state.LastResultsHash,
	proposerAddress,
)

@ggarri
Copy link
Contributor Author

ggarri commented Jun 27, 2019

Due to the implementation of Tendermint ABCI Application, it is not possible to reject/deny a block for consensus when the application layer does not consider a block valid.

IMO that is a clear limitation on Tendermint code I created an issue to request that feature tendermint/tendermint#3755

ggarri added a commit that referenced this issue Jun 27, 2019
@ggarri
#186 Added force prevention measure | Force block creation to 1s
795b4cf
ggarri added a commit that referenced this issue Jun 27, 2019
@ggarri
#186 Implement invalid ethereum block to prevent corrupted state
69c33c7
ggarri added a commit that referenced this issue Jun 27, 2019
@ggarri
#186 Implement invalid ethereum block to prevent corrupted state
5334036
@ggarri
Copy link
Contributor Author

ggarri commented Jun 28, 2019

The block time is being populated by the MedianTIme of every validator preCommit using as weight the voting power of each of the validators, as we see in the following impl:

func MedianTime(commit *types.Commit, validators *types.ValidatorSet) time.Time {

	weightedTimes := make([]*tmtime.WeightedTime, len(commit.Precommits))
	totalVotingPower := int64(0)

	for i, vote := range commit.Precommits {
		if vote != nil {
			_, validator := validators.GetByIndex(vote.ValidatorIndex)
			totalVotingPower += validator.VotingPower
			weightedTimes[i] = tmtime.NewWeightedTime(vote.Timestamp, validator.VotingPower)
		}
	}

	return tmtime.WeightedMedian(weightedTimes, totalVotingPower)
}

@ggarri
Copy link
Contributor Author

ggarri commented Jun 28, 2019

According to the answer given by Tendermint team at created issue tendermint/tendermint#3755 IT IS IMPOSSIBLE to generate two blocks within the same second if the time_iota_ms >= 1000 BUT we were capable to reproduce it even with the "right setup".

The time_iota_ms is used in the _voteTime() methos as an addition to the proposal block time as we see from the following code:

func (cs *ConsensusState) voteTime() time.Time {
	now := tmtime.Now()
	minVoteTime := now
	// TODO: We should remove next line in case we don't vote for v in case cs.ProposalBlock == nil,
	// even if cs.LockedBlock != nil. See https://github.com/tendermint/spec.
	timeIotaMs := time.Duration(cs.state.ConsensusParams.Block.TimeIotaMs) * time.Millisecond
	if cs.LockedBlock != nil {
		// See the BFT time spec https://tendermint.com/docs/spec/consensus/bft-time.html
		minVoteTime = cs.LockedBlock.Time.Add(timeIotaMs)
	} else if cs.ProposalBlock != nil {
		minVoteTime = cs.ProposalBlock.Time.Add(timeIotaMs)
	}

	if now.After(minVoteTime) {
		return now
	}
	return minVoteTime
}

ggarri added a commit that referenced this issue Jun 28, 2019
@ggarri
#186 Implement invalid ethereum block to prevent corrupted state
a65c8c3
@ggarri
Copy link
Contributor Author

ggarri commented Jun 28, 2019

At the moment the invalid block was created the consensus setup was the one as follow:

timeout_propose = "2s"
timeout_propose_delta = "500ms"
timeout_prevote = "2s"
timeout_prevote_delta = "500ms"
timeout_precommit = "2s"
timeout_precommit_delta = "500ms"
timeout_commit = "2s"

blocktime_iota = "1s"

Therefore it was possible that a timeout situation happened overwritting the "default behavour" mentioned above abouve time_iota_ms = 1000, BUT according to the Tendermint documentation those values are only considered once the +2/3 votes are achieved. Verified on the following code:

github.com/tendermint/tendermint/config

// Prevote returns the amount of time to wait for straggler votes after receiving any +2/3 prevotes
// Propose returns the amount of time to wait for a proposal
func (cfg *ConsensusConfig) Propose(round int) time.Duration {
	return time.Duration(
		cfg.TimeoutPropose.Nanoseconds()+cfg.TimeoutProposeDelta.Nanoseconds()*int64(round),
	) * time.Nanosecond
}

// Prevote returns the amount of time to wait for straggler votes after receiving any +2/3 prevotes
func (cfg *ConsensusConfig) Prevote(round int) time.Duration {
	return time.Duration(
		cfg.TimeoutPrevote.Nanoseconds()+cfg.TimeoutPrevoteDelta.Nanoseconds()*int64(round),
	) * time.Nanosecond
}

// Precommit returns the amount of time to wait for straggler votes after receiving any +2/3 precommits
func (cfg *ConsensusConfig) Precommit(round int) time.Duration {
	return time.Duration(
		cfg.TimeoutPrecommit.Nanoseconds()+cfg.TimeoutPrecommitDelta.Nanoseconds()*int64(round),
	) * time.Nanosecond
}

// Commit returns the amount of time to wait for straggler votes after receiving +2/3 precommits for a single block (ie. a commit).
func (cfg *ConsensusConfig) Commit(t time.Time) time.Time {
	return t.Add(cfg.TimeoutCommit)
}

Therefore we could discard that the issue was caused due to low values on the timeout_*

@ggarri ggarri self-assigned this Jun 28, 2019
ggarri added a commit that referenced this issue Jun 28, 2019
@ggarri
#186 Prevent invalid block time from Tendermint
713a124
ggarri added a commit that referenced this issue Jun 28, 2019
@ggarri
#186 Prevent invalid block time from Tendermint
c92d4b8
ggarri added a commit that referenced this issue Jun 28, 2019
@ggarri
#186 Prevent invalid block time from Tendermint
f16717d
@ggarri
Copy link
Contributor Author

ggarri commented Jun 28, 2019

After a team discussion around how to solve this issue we achieved the following statements:

  • We cannot guarantee that Tendermint consensus will produce a block header which is not compliant by ethereum code base
  • Tendermint is our source of truth and we must rely on block headers approved in consensus
  • Ethereum verify header action might change in the future making possible to have two blocks within the same second
  • Ethereum validate header contains many more check which are still relevant and valid for our application logic such a max_gas, block_size ....
  • Every client node should write the same block header on ethereum db

According to what was written on above points, the solution cannot involve a complex or hacky code, and it has to rely on Tendermint protocol. We cannot either remove the entire ethereum verify header.

The proposed solution is:

  • Before block headers are written verify the time is correct, otherwise replace it by the minimum time diff with the parent block, so 1 second
  • Create a metric to track the amount of times that situation occur
  • Alarm URGENTLY to the team every time it happens

ggarri added a commit that referenced this issue Jun 28, 2019
@ggarri
#186 Prevent invalid block time from Tendermint
785d335
@ggarri ggarri added the bug Something isn't working label Jun 28, 2019
@ggarri
Copy link
Contributor Author

ggarri commented Jun 28, 2019

After the fix was applied we have identified another correction of the time on ethereum code which enforce nodes to sleep when blocks are persisted in the future, that improve the proposed solution as it will slow down the node and adjust block times without letting the block time goes to far in future, max of 4 seconds

/github.com/ethereum/go-ethereum/miner/worker.go

func (w *worker) commitNewWork(interrupt *int32, noempty bool, timestamp int64) {
    w.mu.RLock()
    defer w.mu.RUnlock()

    tstart := time.Now()
    parent := w.chain.CurrentBlock()

    if parent.Time() >= uint64(timestamp) {
        timestamp = int64(parent.Time() + 1)
    }
        // this will ensure we're not going off too far in the future
    if now := time.Now().Unix(); timestamp > now+1 {
        wait := time.Duration(timestamp-now) * time.Second
        log.Info("Mining too far in the future", "wait", common.PrettyDuration(wait))
        time.Sleep(wait)
    }

EnchanterIO added a commit that referenced this issue Jul 1, 2019
@EnchanterIO
#186 Improves logging in ABCI::BeginBlock.
df8ca57
EnchanterIO added a commit that referenced this issue Jul 1, 2019
@EnchanterIO
#186 Improves additional logging formatting in ABCI::DeliverTx.
c393470
EnchanterIO added a commit that referenced this issue Jul 1, 2019
@EnchanterIO
#186 Renames nextBlockTime with validEthBlockTime to avoid confus…
c86dea3 
…ion.
ggarri added a commit that referenced this issue Jul 1, 2019
@ggarri
#186 Prevent invalid block time from Tendermint
1fa9261
ggarri pushed a commit that referenced this issue Jul 1, 2019
@EnchanterIO @ggarri
#186 Improves logging in ABCI::BeginBlock.
2b45f71
ggarri pushed a commit that referenced this issue Jul 1, 2019
@EnchanterIO @ggarri
#186 Improves additional logging formatting in ABCI::DeliverTx.
04a01c2
ggarri pushed a commit that referenced this issue Jul 1, 2019
@EnchanterIO @ggarri
#186 Renames nextBlockTime with validEthBlockTime to avoid confus…
cfd3aba 
…ion.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ggarri ggarri

Labels
bug Something isn't working research
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ggarri