Fast syncing on a digitalocean droplet with a block storage volume results in a rocksdb exception #591

e-nikolov · 2020-03-25T22:55:32Z

When Fast syncing the ethereum mainnet using a digitalocean block storage volume for storing the data, at some point I always start repeatedly getting this error:

2020-03-25 22:49:22.359+00:00 | EthScheduler-Services-1 (importBlock) | ERROR | PipelineChainDownloader | Chain download failed. Restarting after short delay.
java.util.concurrent.CompletionException: org.hyperledger.besu.plugin.services.exception.StorageException: org.rocksdb.RocksDBException: block checksum mismatch: expected 826185092, got 2532181494  in /opt/besu/data/database/030185.sst offset 63492071 size 7574
        at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1019) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
        at org.hyperledger.besu.services.pipeline.Pipeline.abort(Pipeline.java:152) ~[besu-pipeline-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:134) ~[besu-pipeline-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.hyperledger.besu.plugin.services.exception.StorageException: org.rocksdb.RocksDBException: block checksum mismatch: expected 826185092, got 2532181494  in /opt/besu/data/database/030185.sst offset 63492071 size 7574
        at org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBColumnarKeyValueStorage$RocksDbTransaction.commit(RocksDBColumnarKeyValueStorage.java:272) ~[besu-plugin-rocksdb-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.services.kvstore.SegmentedKeyValueStorageTransactionTransitionValidatorDecorator.commit(SegmentedKeyValueStorageTransactionTransitionValidatorDecorator.java:49) ~[besu-kvstore-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.services.kvstore.SegmentedKeyValueStorageAdapter$1.commit(SegmentedKeyValueStorageAdapter.java:85) ~[besu-kvstore-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.ethereum.storage.keyvalue.KeyValueStoragePrefixedKeyBlockchainStorage$Updater.commit(KeyValueStoragePrefixedKeyBlockchainStorage.java:189) ~[besu-core-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.ethereum.chain.DefaultBlockchain.appendBlockHelper(DefaultBlockchain.java:250) ~[besu-core-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.ethereum.chain.DefaultBlockchain.appendBlock(DefaultBlockchain.java:229) ~[besu-core-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.ethereum.mainnet.MainnetBlockImporter.fastImportBlock(MainnetBlockImporter.java:73) ~[besu-core-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.ethereum.eth.sync.fastsync.FastImportBlocksStep.importBlock(FastImportBlocksStep.java:66) ~[besu-eth-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.ethereum.eth.sync.fastsync.FastImportBlocksStep.accept(FastImportBlocksStep.java:51) ~[besu-eth-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.ethereum.eth.sync.fastsync.FastImportBlocksStep.accept(FastImportBlocksStep.java:30) ~[besu-eth-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.services.pipeline.CompleterStage.run(CompleterStage.java:37) ~[besu-pipeline-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:130) ~[besu-pipeline-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        ... 5 more
Caused by: org.rocksdb.RocksDBException: block checksum mismatch: expected 826185092, got 2532181494  in /opt/besu/data/database/030185.sst offset 63492071 size 7574
        at org.rocksdb.Transaction.commit(Native Method) ~[rocksdbjni-6.4.6.jar:?]
        at org.rocksdb.Transaction.commit(Transaction.java:206) ~[rocksdbjni-6.4.6.jar:?]
        at org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBColumnarKeyValueStorage$RocksDbTransaction.commit(RocksDBColumnarKeyValueStorage.java:270) ~[besu-plugin-rocksdb-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.services.kvstore.SegmentedKeyValueStorageTransactionTransitionValidatorDecorator.commit(SegmentedKeyValueStorageTransactionTransitionValidatorDecorator.java:49) ~[besu-kvstore-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.services.kvstore.SegmentedKeyValueStorageAdapter$1.commit(SegmentedKeyValueStorageAdapter.java:85) ~[besu-kvstore-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.ethereum.storage.keyvalue.KeyValueStoragePrefixedKeyBlockchainStorage$Updater.commit(KeyValueStoragePrefixedKeyBlockchainStorage.java:189) ~[besu-core-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.ethereum.chain.DefaultBlockchain.appendBlockHelper(DefaultBlockchain.java:250) ~[besu-core-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.ethereum.chain.DefaultBlockchain.appendBlock(DefaultBlockchain.java:229) ~[besu-core-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.ethereum.mainnet.MainnetBlockImporter.fastImportBlock(MainnetBlockImporter.java:73) ~[besu-core-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.ethereum.eth.sync.fastsync.FastImportBlocksStep.importBlock(FastImportBlocksStep.java:66) ~[besu-eth-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.ethereum.eth.sync.fastsync.FastImportBlocksStep.accept(FastImportBlocksStep.java:51) ~[besu-eth-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.ethereum.eth.sync.fastsync.FastImportBlocksStep.accept(FastImportBlocksStep.java:30) ~[besu-eth-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.services.pipeline.CompleterStage.run(CompleterStage.java:37) ~[besu-pipeline-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:130) ~[besu-pipeline-1.4.1-SNAPSHOT.jar:1.4.1-dev-83ee5fa3]
        ... 5 more

I tried with both pruning enabled/disabled with no difference. I haven't seen this error when full syncing nor when fast syncing with the digital ocean machine's local ssd instead of a block storage volume.

Versions (Add all that apply)

Software version: [besu --version]:
besu/v1.4.1-dev-83ee5fa3/linux-x86_64/oracle_openjdk-java-11
Java version: [java -version]:
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment (build 11.0.2+9-Debian-3bpo91)
OpenJDK 64-Bit Server VM (build 11.0.2+9-Debian-3bpo91, mixed mode, sharing)
OS Name & Version: [cat /etc/*release]
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
Kernel Version: [uname -a]
Linux besu-mainnet-fast-prune-0 4.19.0-0.bpo.6-amd64 Update Jenkins Build for Pantheon → Besu considerations #1 SMP Debian 4.19.67-2+deb10u2~bpo9+1 (2019-11-12) x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

timbeiko · 2020-03-26T15:23:30Z

Hi @e-nikolov,

Unfortunately, this is a known issue with RocksDB (see https://github.com/facebook/rocksdb/search?q=block+checksum+mismatch&type=Issues). The only workaround we can offer is for you to delete your database and start the sync again. If you did not delete your DB between sync attempts, that would explain why you hit the failure multiple times.

e-nikolov · 2020-03-26T16:22:11Z

I do clear my database between attempts. I've done the syncing from scratch about 10 times and the strange thing is that I consistently get this issue only when using fast sync on a block storage.

I haven't had it happen with full sync on block storage nor with fast sync on local ssd.

timbeiko · 2020-03-26T17:36:36Z

Ok, after checking with some of our engineers, it seems there are a few things happening here:

This looks like the RocksDB compaction issue described here
The reason it happens during fast and not full sync is that fast sync causes more compactions more which increases the likelihood of the issue above. It looks like there are too many compactions during fast sync for Digital Ocean droplets to effectively handle I/O-wise.
The cause of this is probably Digital Ocean's drivers, which means you can maybe "fix" the issue with a full restart. Similar cases on AWS have been fixed that way.

timbeiko · 2020-04-06T20:20:07Z

@e-nikolov update on this issue: we are currently looking into better tuning RocksDB and/or making Besu compatible with other DBs as medium-term fixes for this issue. Will keep updating here as we have more data.

nysxah · 2020-11-03T15:04:46Z

hello, we are interested in introducing Besu as a replacement for Parity 2.5.13, on DigitalOcean w/ block storage. is fast sync (pruning, non-archive) on DO still an issue?

timbeiko · 2020-11-10T14:51:24Z

@nysxah we believe it still is, but are looking into reproducing and fixing the issue shortly. I will tag you when we have a potential fix.

nysxah · 2021-06-01T15:30:08Z

hi, would you please comment if this issue has been fixed/resolved on Besu if running on AWS or DigitalOcean?

danbartlett · 2022-09-16T21:13:51Z

Still an issue on DigitalOcean as of last week

timbeiko added the bug Something isn't working label Mar 26, 2020

timbeiko added this to the Chupacabra Sprint 61 milestone Mar 30, 2020

timbeiko mentioned this issue Apr 6, 2020

Added AWS and Digital Ocean droplet issue hyperledger/besu-docs#288

Merged

timbeiko removed this from the Chupacabra Sprint 61 milestone Apr 8, 2020

timbeiko added the P2 High (ex: Degrading performance issues, unexpected behavior of core features (DevP2P, syncing, etc)) label May 13, 2020

timbeiko self-assigned this May 20, 2020

vmichalik unassigned timbeiko May 26, 2021

sajz closed this as completed Jun 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fast syncing on a digitalocean droplet with a block storage volume results in a rocksdb exception #591

Fast syncing on a digitalocean droplet with a block storage volume results in a rocksdb exception #591

e-nikolov commented Mar 25, 2020

timbeiko commented Mar 26, 2020

e-nikolov commented Mar 26, 2020

timbeiko commented Mar 26, 2020

timbeiko commented Apr 6, 2020

nysxah commented Nov 3, 2020

timbeiko commented Nov 10, 2020

nysxah commented Jun 1, 2021

danbartlett commented Sep 16, 2022

Fast syncing on a digitalocean droplet with a block storage volume results in a rocksdb exception #591

Fast syncing on a digitalocean droplet with a block storage volume results in a rocksdb exception #591

Comments

e-nikolov commented Mar 25, 2020

Versions (Add all that apply)

timbeiko commented Mar 26, 2020

e-nikolov commented Mar 26, 2020

timbeiko commented Mar 26, 2020

timbeiko commented Apr 6, 2020

nysxah commented Nov 3, 2020

timbeiko commented Nov 10, 2020

nysxah commented Jun 1, 2021

danbartlett commented Sep 16, 2022