Reth panicked at crates/storage/provider/src/bundle_state/bundle_state_with_receipts.rs:254:61

[tobernguyen]

Describe the bug

Reth crashed. These are the logs before it exited.

2024-01-22T01:41:21.916263Z  INFO reth::cli: Status connected_peers=9 freelist=460657 latest_block=19059064
2024-01-22T01:41:23.777256Z  INFO reth::commands::node::events: Canonical chain committed number=19059065 hash=0xe6bd0e879b0c4748a03b50b2739f6227a4ac1e440bc525ea35c274ad7f9f917f elapsed=5.138737399s
2024-01-22T01:41:23.777289Z  INFO reth::commands::node::events: Forkchoice updated head_block_hash=0xe6bd0e879b0c4748a03b50b2739f6227a4ac1e440bc525ea35c274ad7f9f917f safe_block_hash=0xeab964a803079c8107abe2bc2796b62d4b7a
2024-01-22T01:41:30.152816Z  INFO reth::commands::node::events: Block added to canonical chain number=19059067 hash=0x1737cd9c16b0a68d36f3fc233692e644f332c0360eda28db60ab98831f3cd7ef
2024-01-22T01:41:36.562725Z  INFO reth::commands::node::events: Block added to canonical chain number=19059068 hash=0x0e54f4085893602bf337b8fe459255b3128398af43907cbd6653600e6e97b3d5
thread 'tokio-runtime-worker' panicked at crates/storage/provider/src/bundle_state/bundle_state_with_receipts.rs:254:61:
called `Option::unwrap()` on a `None` value
stack backtrace:
   0: rust_begin_unwind
             at ./rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at ./rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14
   2: core::panicking::panic
             at ./rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:127:5
   3: reth_blockchain_tree::blockchain_tree::BlockchainTree<DB,EF>::split_chain
   4: reth_blockchain_tree::blockchain_tree::BlockchainTree<DB,EF>::make_canonical
   5: <reth_provider::providers::BlockchainProvider<DB,Tree> as reth_interfaces::blockchain_tree::BlockchainTreeEngine>::make_canonical
   6: reth_beacon_consensus::engine::BeaconConsensusEngine<DB,BT,Client,EngineT>::forkchoice_updated
   7: <reth_beacon_consensus::engine::BeaconConsensusEngine<DB,BT,Client,EngineT> as core::future::future::Future>::poll
   8: tokio::runtime::task::raw::poll
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2024-01-22T01:41:37.038099Z ERROR reth_tasks: Critical task `consensus engine` panicked: `called `Option::unwrap()` on a `None` value`
2024-01-22T01:41:37.050501Z  INFO reth::cli: Wrote network peers to file peers_file="/reth/known-peers.json"
Error: channel closed
Location:
    /project/bin/reth/src/builder/mod.rs:1375:9

Steps to reproduce

I don't know how to reproduce it.

Node logs

reth-crash-2023-01-22.log.zip

Platform(s)

No response

What version/commit are you on?

0.1.0-alpha.15

What database version are you on?

No response

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[Rjected]

hmm, this is panicking here:

reth/crates/storage/provider/src/bundle_state/bundle_state_with_receipts.rs

Lines 248 to 254 in 7e982ef

	let (mut lower_state, mut higher_state) = (self.clone(), self);
	// Revert lower state to [..at].
	lower_state.revert_to(at.checked_sub(1).unwrap());
	// Truncate higher state to [at..].
	let at_idx = higher_state.block_number_to_index(at).unwrap();

because this method is returning None:

reth/crates/storage/provider/src/bundle_state/bundle_state_with_receipts.rs

Lines 142 to 148 in 7e982ef

	if self.first_block > block_number {
	return None
	}
	let index = block_number - self.first_block;
	if index >= self.receipts.len() as u64 {
	return None
	}

[tobernguyen]

I'm seeing this in alpha17. This is the debug log.
reth-2024-02-08.log.zip

[tobernguyen]

I'm seeing this in alpha18.
Uploading reth-2024-02-20.log.zip…

[Rjected]

I just ran into this:

2024-02-23T18:41:38.367287Z  INFO reth::commands::node::events: Block added to canonical chain number=1005564 hash=0x9d1e5a85b118de3e9509f6748101419a3ffc87b97531700ec8c0d84380303608
thread 'tokio-runtime-worker' panicked at crates/storage/provider/src/bundle_state/bundle_state_with_receipts.rs:254:61:
called `Option::unwrap()` on a `None` value
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic
   3: core::option::Option<T>::unwrap
   4: reth_provider::bundle_state::bundle_state_with_receipts::BundleStateWithReceipts::split_at
   5: reth_provider::chain::Chain::split
   6: reth_blockchain_tree::blockchain_tree::BlockchainTree<DB,EF>::split_chain
   7: reth_blockchain_tree::blockchain_tree::BlockchainTree<DB,EF>::make_canonical
   8: <reth_blockchain_tree::shareable::ShareableBlockchainTree<DB,EF> as reth_interfaces::blockchain_tree::BlockchainTreeEngine>::make_canonical
   9: <reth_provider::providers::BlockchainProvider<DB,Tree> as reth_interfaces::blockchain_tree::BlockchainTreeEngine>::make_canonical
  10: reth_beacon_consensus::engine::BeaconConsensusEngine<DB,BT,Client,EngineT>::forkchoice_updated
  11: reth_beacon_consensus::engine::BeaconConsensusEngine<DB,BT,Client,EngineT>::on_forkchoice_updated
  12: <reth_beacon_consensus::engine::BeaconConsensusEngine<DB,BT,Client,EngineT> as core::future::future::Future>::poll
  13: reth::builder::NodeBuilderWithDatabase<DB>::launch::{{closure}}::{{closure}}
  14: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::future::future::Future>::poll
  15: <futures_util::future::future::catch_unwind::CatchUnwind<Fut> as core::future::future::Future>::poll::{{closure}}
  16: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  17: std::panicking::try::do_call
  18: ___rust_try
  19: std::panicking::try
  20: std::panic::catch_unwind
  21: <futures_util::future::future::catch_unwind::CatchUnwind<Fut> as core::future::future::Future>::poll
  22: <F as futures_core::future::TryFuture>::try_poll
  23: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
  24: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  25: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
  26: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
  27: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  28: <core::pin::Pin<P> as core::future::future::Future>::poll
  29: futures_util::future::future::FutureExt::poll_unpin
  30: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  31: reth_tasks::TaskExecutor::spawn_critical_as::{{closure}}
  32: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
  33: tokio::runtime::park::CachedParkThread::block_on
  34: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
  35: tokio::runtime::handle::Handle::block_on::{{closure}}
  36: tokio::runtime::context::runtime::enter_runtime
  37: tokio::runtime::handle::Handle::block_on
  38: reth_tasks::TaskExecutor::spawn_on_rt::{{closure}}
  39: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
  40: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  41: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
  42: tokio::runtime::task::core::Core<T,S>::poll
  43: tokio::runtime::task::harness::poll_future::{{closure}}
  44: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  45: std::panicking::try::do_call
  46: ___rust_try
  47: std::panicking::try
  48: std::panic::catch_unwind
  49: tokio::runtime::task::harness::poll_future
  50: tokio::runtime::task::harness::Harness<T,S>::poll_inner
  51: tokio::runtime::task::harness::Harness<T,S>::poll
  52: tokio::runtime::task::raw::poll
  53: tokio::runtime::task::raw::RawTask::poll
  54: tokio::runtime::task::UnownedTask<S>::run
  55: tokio::runtime::blocking::pool::Task::run
  56: tokio::runtime::blocking::pool::Inner::run
  57: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2024-02-23T18:41:39.135484Z ERROR reth_tasks: Critical task `consensus engine` panicked: `called `Option::unwrap()` on a `None` value`
2024-02-23T18:41:39.136005Z ERROR reth::cli: shutting down due to error
Error: channel closed

Location:
    /Users/dan/projects/reth/bin/reth/src/builder.rs:502:9