Make BlockPayload::from_transactions async and pass ValidatedState

[imabdulbasit]

The BlockPayload::from_transactions function takes an Instance parameter, which contains ChainConfig on the sequencer side. To support updates to ChainConfig, we need to add it to a mutable ValidatedState. This requires the function to accept an additional ValidatedState parameter. When the chainconfig commitment does not match the instance chainconfig, this function on the sequencer side will do catchup from the peers (GET request) to get the updated ChainConfig so it needs to be async.

This PR:

• adds ValidatedState parameter to BlockPayload::from_transactions
• makes the from_transactions fn async

EspressoSystems/espresso-sequencer#1475
EspressoSystems/espresso-sequencer#1492 (comment)

[jparr721]

This seems fine, though the testing changes should be reviewed by at least one other person. Your justification seems good though for adding this. Just check over my requests and let me know if any of them are unclear. Feel free to disregard if something isn't relevant.

[bfish713]

I think we are introducing a problem where this will block the consensus task for potentially a very long time. It only happens when there is an upgrade in flight so it would probably lay dormant until we need to do an upgrade and some nodes might stall.

I think we need to do this in the spawned proposal task now to avoid this issue

[imabdulbasit]

done in 2d78d1e

[imabdulbasit]

reverted

[bfish713]

Same issue here. I think the fix is straightforward. Just move this logic till the broadcast of BlockRecv into a ansyc task (async_spawn it)

[imabdulbasit]

done in 2d78d1e

[imabdulbasit]

where do I keep the handle of this task if we want to do it? I see some tasks handles kept in TaskRegistry

[jparr721]

You could do something like the consensus task does:

    pub spawned_tasks: BTreeMap<TYPES::Time, Vec<JoinHandle<()>>>,

So we can keep a map around and key on the view or whatever relevant key we need, then we can cancel the task from there. The key here cancels old tasks if we propose for a newer view, so in this case we can probably cancel when we get a new view change? We could cancel with something reminiscent of the following: https://github.com/EspressoSystems/HotShot/blob/main/crates/task-impls/src/consensus/mod.rs#L151 but we might be able to just cancel all tasks older than view. Thoughts @bfish713?

[imabdulbasit]

I think this shouldn't be canceled on a view change since we should be able to complete catch up even if there is a view change to keep the updated chain config in ValidatedState.

[imabdulbasit]

could be canceled after a certain time maybe

[ss-es]

I think we shouldn't drop the handle, though I don't think TaskRegistry is the right place to put this (it will likely be impossible to put it there very soon). It should definitely be stored in TransactionTaskState somehow, probably either in a Vec<JoinHandle<()>> or a BTreeMap like Jarred suggested.

I guess the question is: what is the expected behavior if consensus is on view (say) 100 and a calculation for view 50 is still going? do we just keep going indefinitely and risk accumulating these spawned subtasks? or is there a point where we can just give up? (I don't know enough to know what the answer is)

[imabdulbasit]

reverted this change as builder_fee function is synchronous now 047cf5c

[jparr721]

Looks good, but I'm going to let Brendon approve since he had more specific requests.

Feedback addressed

[bfish713]

Looks good I like that we change to add empty to make this synchronous in consensus.

[bfish713]

@jparr721 I'm holding off approval because this would be considered a breaking change during the RC process. Can you accept this we're ok merging it?

[jparr721]

Phew, glad we got the dependency task tests included here.