Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[1 / 5] Optimize logic for gossiping assignments #4848

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

alexggh
Copy link
Contributor

@alexggh alexggh commented Jun 20, 2024

This part of the work to further optimize the approval subsystems, if you want to understand the full context start with reading #4849 (comment), however that's not necessary, as this change is self-contained and nodes would benefit from it regardless of subsequent changes landing or not.

While testing with 1000 validators found out that the logic for determining the validators and assignment should be gossiped to is taking a lot of time, because it always iterated through all the peers(1000), to determine which are X and Y neighbours and to which we should randomly gossip(4 samples).

This could be actually optimised, so we don't have to iterate through all 1000 peers for each new assignment, by fetching the list of X and Y peer ids from the topology first and then stopping the loop once we took the 4 random samples.

With this improvements we reduce the total CPU time spent in approval-distribution with 15% on networks with 500 validators and 20% on networks with 1000 validators.

Test coverage:

propagates_assignments_along_unshared_dimension and propagates_locally_generated_assignment_to_both_dimensions cover already logic and they passed, confirm that there is no breaking change.

Additionally, the approval voting benchmark measure the traffic sent to other peers, so I confirmed that for various network size there is no difference in the size of the traffic sent to other peers.

@alexggh alexggh changed the title Optimize logic for gossiping assignments [4 / 5] Optimize logic for gossiping assignments Jun 20, 2024
@alexggh alexggh changed the base branch from alexaggh/approval-voting-parallel-3-5 to master July 2, 2024 11:39
@alexggh alexggh force-pushed the alexaggh/approval-voting-parallel-4-5 branch from cb57906 to 4b3f489 Compare July 2, 2024 11:40
@alexggh alexggh changed the title [4 / 5] Optimize logic for gossiping assignments [1 / 5] Optimize logic for gossiping assignments Jul 2, 2024
@alexggh alexggh force-pushed the alexaggh/approval-voting-parallel-4-5 branch 2 times, most recently from 7add070 to 4b3f489 Compare July 2, 2024 12:01
@alexggh alexggh marked this pull request as ready for review July 2, 2024 12:24
@alexggh alexggh added the T0-node This PR/Issue is related to the topic “node”. label Jul 2, 2024
@alexggh alexggh force-pushed the alexaggh/approval-voting-parallel-4-5 branch from 4b3f489 to 713aef4 Compare July 2, 2024 12:29
@paritytech-cicd-pr
Copy link

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable 2/3
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6605079

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T0-node This PR/Issue is related to the topic “node”.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants