Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core: Fix standalone orchestrator not crashing under UnrecoverableError #2352

Merged
merged 2 commits into from
Apr 7, 2022

Conversation

leszko
Copy link
Contributor

@leszko leszko commented Apr 5, 2022

What does this pull request do? Explain your changes. (required)

Fix the issue that O does not fail after the unrecoverable error from lpms (ffmpeg).

The issue is described in this Discord thread. In short, when GPU crashes, NVIDIA recommends restarting the whole process, since it's not possible to recover. It worked that way until the Gracefully notify orchestrator in case of a panic in transcoder PR. In the mentioned PR, the following logic was implemented:

  1. Recover from LPMS (ffmpeg) panic
  2. Send a message T->O
  3. Panic

The logic is fine for the split O/T topology, however for the combined OT, there is a separate code which also needs to implement Point 3. Panic.

Specific updates (required)

How did you test each of these updates (required)

Artificially introduce panic() and check that a standalone O crashes.

Does this pull request close any open issues?

Checklist:

Copy link
Member

@victorges victorges left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Just a few questions/discussions on the crashing logic

core/orchestrator.go Show resolved Hide resolved
core/orchestrator.go Show resolved Hide resolved
Copy link
Member

@victorges victorges left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@leszko leszko merged commit 77d9e0f into livepeer:master Apr 7, 2022
@leszko leszko deleted the fix-handling-unrecoverable-error branch April 7, 2022 08:26
ad-astra-video pushed a commit to ad-astra-video/go-livepeer that referenced this pull request May 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants