Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf(tree): integrate parallel state root #7161

Merged
merged 5 commits into from
Mar 21, 2024
Merged

perf(tree): integrate parallel state root #7161

merged 5 commits into from
Mar 21, 2024

Conversation

rkrasiuk
Copy link
Member

Description

Integrate parallel state root computation (pre-computing storage roots in parallel) introduced in #6903 into the blockchain tree.

@rkrasiuk rkrasiuk added C-perf A change motivated by improving speed, memory usage or disk footprint A-blockchain-tree Related to sidechains, reorgs and pending blocks labels Mar 15, 2024
@rkrasiuk
Copy link
Member Author

Performance Improvements on tip

main branch

ubuntu@reth3:~/reth-beta$ RUST_LOG=info,blockchain_tree::chain=trace ./reth-main debug replay-engine --engine-api-store ./eng
ine-store/
2024-03-15T21:41:50.424584Z  INFO net::peers: Loading saved peers file=/home/ubuntu/.local/share/reth/mainnet/known-peers.json
2024-03-15T21:41:50.427750Z  INFO reth::cli: Connected to P2P network peer_id=0xf3343fcc43811b66c7aadf67c8399a87ca69b5278ac9ed9a00086852316eeca95b5cdbf2a17ecebd3bca2b78c4c0469884db29636355a982f4afeacda2d0df2e local_addr=0.0.0.0:30303
2024-03-15T21:41:50.427875Z  INFO reth::cli: Consensus engine initialized
2024-03-15T21:41:50.427881Z  INFO reth::cli: Starting consensus engine
2024-03-15T21:41:50.704767Z DEBUG blockchain_tree::chain: Validated state root number=19443061 hash=0xee38e0eb246c0bd7871ce964d8acb914ad99f6f5114294fd253067c99c4d99b3 elapsed=251.100854ms
2024-03-15T21:41:53.170699Z DEBUG blockchain_tree::chain: Validated state root number=19443062 hash=0xc83b6d6d75510b682bcdef4829cc8573502ed7dd7b415370bc944e019a13d205 elapsed=168.068762ms
2024-03-15T21:41:55.416434Z DEBUG blockchain_tree::chain: Validated state root number=19443063 hash=0xe40d417224c401975055b36ef2cb4be643e695c708fa616383a96e33b60b2da1 elapsed=78.426627ms
2024-03-15T21:41:57.825854Z DEBUG blockchain_tree::chain: Validated state root number=19443064 hash=0x41dcb60cbe8f12c155d849d5bbdc88f9d456649dd1e9b68260b225eea52bc299 elapsed=293.203093ms
2024-03-15T21:42:00.214564Z DEBUG blockchain_tree::chain: Validated state root number=19443065 hash=0x255ab3f07868348191e58f9904184ff22fd189c4b1aec76b341cbe725984f53c elapsed=102.214605ms
2024-03-15T21:42:02.555004Z DEBUG blockchain_tree::chain: Validated state root number=19443066 hash=0x2500f92ff7c3ff1f0e2579641608d9daa8cbe2ac3f2c74e9a7c1e061b9a4cad9 elapsed=202.597941ms
2024-03-15T21:42:04.883019Z DEBUG blockchain_tree::chain: Validated state root number=19443067 hash=0x42862f8a0a7c5197afd61372d9b261cf4d12b76d2f0b68c142e2d6654cbac520 elapsed=89.038742ms
2024-03-15T21:42:07.177940Z DEBUG blockchain_tree::chain: Validated state root number=19443068 hash=0x4d9557d1933a02fb780063b4c35c0cd9fb590014f851f50641befa034dbdc7e4 elapsed=144.592695ms
2024-03-15T21:42:09.392441Z DEBUG blockchain_tree::chain: Validated state root number=19443069 hash=0x347bc8144c5765a842a689dede001fb28b78ec14b1cef1cbb96b6e00391d5cf1 elapsed=78.425076ms
2024-03-15T21:42:11.555783Z DEBUG blockchain_tree::chain: Validated state root number=19443070 hash=0xdccb394534f5d6fbd05d99df93550850bb5d2c338369df431c3bee7435d3113b elapsed=91.280258ms
2024-03-15T21:42:13.787587Z DEBUG blockchain_tree::chain: Validated state root number=19443071 hash=0x73d6324fdfae69fe1edb6f87f52f5279b7841e591394b27c4e08fe212aed4d00 elapsed=114.554667ms
2024-03-15T21:42:16.225936Z DEBUG blockchain_tree::chain: Validated state root number=19443072 hash=0x57455edaca89d85238f0b328cf3b409f7385b2557af493fa4d5cc6bb2ddb4756 elapsed=290.905236ms
2024-03-15T21:42:18.570596Z DEBUG blockchain_tree::chain: Validated state root number=19443073 hash=0x1323242df21063cb8117f1bbc72abceb387e9588c6a73e9a71d7aad22ac41587 elapsed=39.142205ms
2024-03-15T21:42:20.928289Z DEBUG blockchain_tree::chain: Validated state root number=19443074 hash=0xc8830538a05b68935fab9c4f8ffb7d9aeffbff2016b34d052f614ad0be9d58b2 elapsed=286.450305ms
2024-03-15T21:42:23.355957Z DEBUG blockchain_tree::chain: Validated state root number=19443075 hash=0xd39c1d60996475e65d1ab5b4e755f510ca466564a8155d35db6667988d6c0e44 elapsed=123.590833ms
2024-03-15T21:42:25.491985Z  INFO reth::cli: Finished replaying engine API messages

rkrasiuk/tree-integrate-parallel-state-root

2024-03-15T21:42:45.055047Z  INFO net::peers: Loading saved peers file=/home/ubuntu/.local/share/reth/mainnet/known-peers.json
2024-03-15T21:42:45.058907Z  INFO reth::cli: Connected to P2P network peer_id=0xf3343fcc43811b66c7aadf67c8399a87ca69b5278ac9ed9a00086852316eeca95b5cdbf2a17ecebd3bca2b78c4c0469884db29636355a982f4afeacda2d0df2e local_addr=0.0.0.0:30303
2024-03-15T21:42:45.059011Z  INFO reth::cli: Consensus engine initialized
2024-03-15T21:42:45.059018Z  INFO reth::cli: Starting consensus engine
2024-03-15T21:42:45.223135Z DEBUG blockchain_tree::chain: Validated state root number=19443061 hash=0xee38e0eb246c0bd7871ce964d8acb914ad99f6f5114294fd253067c99c4d99b3 elapsed=135.417165ms
2024-03-15T21:42:47.600318Z DEBUG blockchain_tree::chain: Validated state root number=19443062 hash=0xc83b6d6d75510b682bcdef4829cc8573502ed7dd7b415370bc944e019a13d205 elapsed=84.578201ms
2024-03-15T21:42:49.820100Z DEBUG blockchain_tree::chain: Validated state root number=19443063 hash=0xe40d417224c401975055b36ef2cb4be643e695c708fa616383a96e33b60b2da1 elapsed=48.608217ms
2024-03-15T21:42:52.095177Z DEBUG blockchain_tree::chain: Validated state root number=19443064 hash=0x41dcb60cbe8f12c155d849d5bbdc88f9d456649dd1e9b68260b225eea52bc299 elapsed=152.347111ms
2024-03-15T21:42:54.445691Z DEBUG blockchain_tree::chain: Validated state root number=19443065 hash=0x255ab3f07868348191e58f9904184ff22fd189c4b1aec76b341cbe725984f53c elapsed=58.468744ms
2024-03-15T21:42:56.693296Z DEBUG blockchain_tree::chain: Validated state root number=19443066 hash=0x2500f92ff7c3ff1f0e2579641608d9daa8cbe2ac3f2c74e9a7c1e061b9a4cad9 elapsed=100.102435ms
2024-03-15T21:42:58.990079Z DEBUG blockchain_tree::chain: Validated state root number=19443067 hash=0x42862f8a0a7c5197afd61372d9b261cf4d12b76d2f0b68c142e2d6654cbac520 elapsed=55.19599ms
2024-03-15T21:43:01.157377Z DEBUG blockchain_tree::chain: Validated state root number=19443068 hash=0x4d9557d1933a02fb780063b4c35c0cd9fb590014f851f50641befa034dbdc7e4 elapsed=80.405192ms
2024-03-15T21:43:03.308961Z DEBUG blockchain_tree::chain: Validated state root number=19443069 hash=0x347bc8144c5765a842a689dede001fb28b78ec14b1cef1cbb96b6e00391d5cf1 elapsed=44.166268ms
2024-03-15T21:43:05.440653Z DEBUG blockchain_tree::chain: Validated state root number=19443070 hash=0xdccb394534f5d6fbd05d99df93550850bb5d2c338369df431c3bee7435d3113b elapsed=50.422767ms
2024-03-15T21:43:07.646484Z DEBUG blockchain_tree::chain: Validated state root number=19443071 hash=0x73d6324fdfae69fe1edb6f87f52f5279b7841e591394b27c4e08fe212aed4d00 elapsed=62.172106ms
2024-03-15T21:43:09.962348Z DEBUG blockchain_tree::chain: Validated state root number=19443072 hash=0x57455edaca89d85238f0b328cf3b409f7385b2557af493fa4d5cc6bb2ddb4756 elapsed=154.168851ms
2024-03-15T21:43:12.305681Z DEBUG blockchain_tree::chain: Validated state root number=19443073 hash=0x1323242df21063cb8117f1bbc72abceb387e9588c6a73e9a71d7aad22ac41587 elapsed=28.52023ms
2024-03-15T21:43:14.603558Z DEBUG blockchain_tree::chain: Validated state root number=19443074 hash=0xc8830538a05b68935fab9c4f8ffb7d9aeffbff2016b34d052f614ad0be9d58b2 elapsed=198.894801ms
2024-03-15T21:43:16.987283Z DEBUG blockchain_tree::chain: Validated state root number=19443075 hash=0xd39c1d60996475e65d1ab5b4e755f510ca466564a8155d35db6667988d6c0e44 elapsed=60.45892ms
2024-03-15T21:43:19.133952Z  INFO reth::cli: Finished replaying engine API messages

@rkrasiuk rkrasiuk marked this pull request as ready for review March 15, 2024 22:32
gakonst
gakonst approved these changes Mar 16, 2024
View reviewed changes
crates/blockchain-tree/src/chain.rs
Comment on lines +214 to +220
let mut state = provider.bundle_state_data_provider.state().clone();
state.extend(bundle_state.clone());
let hashed_state = state.hash_state_slow();
ParallelStateRoot::new(consistent_view, hashed_state)
.incremental_root_with_updates()
.map(|(root, updates)| (root, Some(updates)))
.map_err(ProviderError::from)?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

guessing these clones are fine

onbjerg
onbjerg approved these changes Mar 20, 2024
View reviewed changes
Copy link
Member

@onbjerg onbjerg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

crates/storage/provider/src/traits/trie.rs Outdated
Comment on lines 12 to 16
/// NOTE: It is recommended to provide a different implementation from
/// `state_root_with_updates` since it affects the memory usage during state root
/// computation.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

smol doc nit

Suggested change
/// NOTE: It is recommended to provide a different implementation from
/// `state_root_with_updates` since it affects the memory usage during state root
/// computation.
/// # Note
///
/// It is recommended to provide a different implementation from
/// `state_root_with_updates` since it affects the memory usage during state root
/// computation.

@rkrasiuk rkrasiuk enabled auto-merge March 21, 2024 06:19
@mattsse
Copy link
Collaborator

mattsse commented Mar 21, 2024

@rkrasiuk some conflicts left

@rkrasiuk rkrasiuk force-pushed the rkrasiuk/tree-integrate-parallel-state-root branch from ffde69e to 81259bd Compare March 21, 2024 15:59
@rkrasiuk rkrasiuk added this pull request to the merge queue Mar 21, 2024
Merged via the queue into main with commit 56b63ad Mar 21, 2024
28 checks passed
@rkrasiuk rkrasiuk deleted the rkrasiuk/tree-integrate-parallel-state-root branch March 21, 2024 16:17
Ruteri pushed a commit to Ruteri/reth that referenced this pull request Apr 17, 2024
@rkrasiuk @Ruteri
perf(tree): integrate parallel state root (paradigmxyz#7161)
f8ceaec
Ruteri pushed a commit to Ruteri/reth that referenced this pull request Apr 17, 2024
@rkrasiuk @Ruteri
perf(tree): integrate parallel state root (paradigmxyz#7161)
1f12dac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@onbjerg onbjerg onbjerg approved these changes

@gakonst gakonst gakonst approved these changes

@mattsse mattsse mattsse approved these changes

@rakita rakita Awaiting requested review from rakita rakita is a code owner

@joshieDo joshieDo Awaiting requested review from joshieDo joshieDo is a code owner

@shekhirin shekhirin Awaiting requested review from shekhirin shekhirin is a code owner

@Rjected Rjected Awaiting requested review from Rjected Rjected is a code owner

Assignees
No one assigned
Labels
A-blockchain-tree Related to sidechains, reorgs and pending blocks C-perf A change motivated by improving speed, memory usage or disk footprint
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@rkrasiuk @mattsse @onbjerg @gakonst