[DEPENDENCY_TASK] Fix CDN Crash Tests #3390
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bfish713
requested review from
jparr721,
lukaszrzasik,
ss-es and
shenkeyao
and removed request for
elliedavidson
rob-maron
previously approved these changes
Jun 28, 2024
| @@ -686,6 +686,7 @@ pub(crate) async fn fetch_proposal<TYPES: NodeType>( | |||
| quorum_membership: Arc<TYPES::Membership>, | |||
| consensus: Arc<RwLock<Consensus<TYPES>>>, | |||
| ) -> Result<Leaf<TYPES>> { | |||
| tracing::info!("Fetchign proposal for view {:?}", view); | |||
There was a problem hiding this comment.
nit: maybe we should consider logs like these being debug
There was a problem hiding this comment.
My thinking was that this should be a relatively rare occurrence. We should only get this if we miss a proposal and are catching back up but I'm not 100%
jparr721
previously approved these changes
Jun 28, 2024
jparr721
previously approved these changes
Jun 28, 2024
ss-es
previously approved these changes
Jun 28, 2024
| @@ -253,6 +254,8 @@ struct DelayedRequester<TYPES: NodeType, I: NodeImplementation<TYPES>> { | |||
| pub network: Arc<I::Network>, | |||
| /// Shared state to check if the data go populated | |||
| state: Arc<RwLock<Consensus<TYPES>>>, | |||
There was a problem hiding this comment.
Nit: not really related to the PR, but could this be called consensus rather than state for consistency with the other tasks?
There was a problem hiding this comment.
Yeah it should be I want to follow up on this pr with one that handles the VID stuff a little smarter
Co-authored-by: ss-es <[email protected]>
Co-authored-by: ss-es <[email protected]>
Co-authored-by: ss-es <[email protected]>
ss-es
approved these changes
Jun 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #3368
This PR:
Solves the final tests failures for the dependency task refactor. The failure was caused because for some code paths we can calculate the VID Shares, store them in the vid share map but not emit a
VidShareRecvevent. This only happens when running with the combined network. There are 2 places we could calculate the VID shares from a DAProposal. The first is in the DA task if the primary is down. The second is if we get a request from another peer for a VID share we'll try to calculate it if we don't have it but do have a DA Proposal. This led to some nodes not voting when they should be able to because the dependency task was waiting for a VIDShareRecv event that wouldn't come.Typically what I was seeing in CI was that 6/10 nodes would vote and 4 would not. I think the reasoning for this is that the VID calculations were triggered by the requests for VID. What happens was the first nodes to request VID get back a share but the later nodes to request it are already calculating the VID due to the earlier requesters and don't try to request until the calculation finishes and when it does the request task cancels itself.
The sequence that could cause the bug, from the perspective of one were:
In this example the VID calculation could also be triggered if primary network was known down by the node.
The fix for this is to just broadcast that we have the share if we end up cancelling the VID Task because we have the share. This means there is no way for the vote dependency to not get the event if we got the share or calculated it.
This PR does not:
Key places to review: