Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

server: Add suspending sessions that did not pass the pHash verification #2103

Merged
merged 7 commits into from
Nov 22, 2021

Conversation

leszko
Copy link
Contributor

@leszko leszko commented Nov 12, 2021

What does this pull request do? Explain your changes. (required)
Suspend untrusted session when the following conditions are correct:

  • The untrusted session returned incorrect p-hash
  • Another untrusted session returned correct p-hash

It would be nice to add some automated tests for this change, however, it's hard to do it with the current design. I think we should tackle it separately while refactoring broadcast.go.

Specific updates (required)

  • Suspend the untrusted session that returned incorrect p-hash

    How did you test each of these updates (required)

  1. Create a fake orchestrator that returns incorrect p-hash
  2. Run the setup with 1 trusted orchestrator, 2 correct untrusted orchestrators, and 1 incorrect untrusted orchestrator
  3. Check that from time to time the incorrect untrusted orchestrator is suspended for one session selection

Does this pull request close any open issues?

Checklist:

Copy link
Contributor

@darkdarkdragon darkdarkdragon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@jailuthra jailuthra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Made an unrelated sidecomment

return untrustedResult.Session, untrustedResult.TranscodeResult, untrustedResult.Err
} else if monitor.Enabled {
monitor.FastVerificationFailed()
} else {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not related to this exact issue or very high priority, so maybe we should open a new issue/PR for this:

Maybe we should reset our verifiedSession to nil if the hash it sends doesn't match?

Suggested change
} else {
} else {
if (bsm.verifiedSession == untrustedResult.Session) {
bsm.verifiedSession = nil
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see one issue with adding this condition. If we add it, then a false-negative will cause bsm.verifiedSession = nil and in a result, we'll start looking for a new session. That can in turn cause bouncing between sessions if we encounter many false-negative scenarios.

We decided that we'll suspend sessions only if it failed p-hash verification but at the same time the other untrusted sessions passed the verification. We could do the same and maybe set verifiedSession to nil when we suspend the session, but technically it does not matter, because this condition takes into consideration both bsm.verifiedSession = nil and excluding suspended sessions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point about false negatives causing swaps, and yeah we'd anyway suspend a verified session and look for a new one if the other untrusted session passes. LGTM feel free to merge 👍

Copy link
Member

@yondonfu yondonfu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few nits, but looks good!

server/push_test.go Outdated Show resolved Hide resolved
server/push_test.go Outdated Show resolved Hide resolved
@leszko leszko requested a review from yondonfu November 19, 2021 15:07
Copy link
Contributor Author

@leszko leszko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comments @yondonfu . I addressed them. PTAL.

Copy link
Member

@yondonfu yondonfu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Let's just cleanup the last few commits either via squashing and/or a rebase + push prior to merging.

@leszko
Copy link
Contributor Author

leszko commented Nov 19, 2021

LGTM! Let's just cleanup the last few commits either via squashing and/or a rebase + push prior to merging.

Yeah, sure I'll clean it up

@leszko leszko merged commit 02460b9 into livepeer:master Nov 22, 2021
@leszko leszko deleted the 2097-suspend-untrusted-session branch November 22, 2021 09:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants