Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

server: fix OT error handling #1864

Merged
merged 2 commits into from
May 12, 2021

Conversation

reubenr0d
Copy link
Contributor

@reubenr0d reubenr0d commented May 1, 2021

What does this pull request do? Explain your changes. (required)

Specific updates (required)

  • Moved check for transcode errors on O before decodingPixels
  • Added nil check for chanData.TranscodeData

How did you test each of these updates (required)

  • Tested with errored out video segments, one without a keyframe, and another without Video
  • Observed that proper error messages now show on the O
  • Observed that EOF is not triggered in case of these errors

Does this pull request close any open issues?

Fixes #1843
Fixes #1762

Checklist:

@reubenr0d reubenr0d marked this pull request as ready for review May 1, 2021 17:41
@kyriediculous
Copy link
Contributor

kyriediculous commented May 4, 2021

I like the solution, it clearly returns the error that should be returned.

Let's also make sure the header is always a sane value. Currently when we encounter a transcoding error from the T

tData, err := n.Transcoder.Transcode(md)

We will not set the "Pixels" header

if tData != nil {

We should always be setting the "Pixels" header, even if tData is nil and an error is encountered.

If this header is not set we will run into an error converting the string to an integer.

At L190 in ot_rpc.go let's make this change

pixels := int64(0)
    if tData != nil {
        pixels = tData.Pixels
    }
    req.Header.Set("Pixels", strconv.FormatInt(pixels, 10))

I'm also wondering if we can add unit tests for this, we can create erroneous responses like this:

		body.Write([]byte(err.Error()))
   	contentType = transcodingErrorMimeType

See

body.Write([]byte(err.Error()))

@kyriediculous
Copy link
Contributor

Nevermind on the unit tests, this unit tests should already take care of those cases:

func TestRemoteTranscoderError(t *testing.T) {

@kyriediculous kyriediculous self-requested a review May 4, 2021 14:28
Copy link
Contributor

@kyriediculous kyriediculous left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@yondonfu any remarks ? We have tested this both seperately and came to the same conclusions.

No source:

E0504 11:50:45.678492   78423 orchestrator.go:553] Error transcoding manifestID=test sessionID=53e9ccc9 segNo=1 segName=https://127.0.0.1:8935/stream/53e9ccc9/1.tempfile - TranscoderInvalidVideo

No keyframe

E0504 11:48:54.873341   78423 orchestrator.go:553] Error transcoding manifestID=test sessionID=0e05dfc6 segNo=1 segName=https://127.0.0.1:8935/stream/0e05dfc6/1.tempfile - No keyframes in input

@kyriediculous
Copy link
Contributor

I know we didn't touch this in this PR but

https://github.com/livepeer/go-livepeer/blob/master/core/orchestrator.go#L553

This log statement still uses a dash to seperate the error from the rest of the message. We currently use label=..

do you think we can change the end of this statement to err=%v to match this idiom ?

@reubenr0d
Copy link
Contributor Author

reubenr0d commented May 4, 2021

do you think we can change the end of this statement to err=%v to match this idiom ?

Yea, sure. Done.

@kyriediculous
Copy link
Contributor

Looks good to me, let's rebase (squash the fixup)

https://thoughtbot.com/blog/autosquashing-git-commits

@yondonfu
Copy link
Member

yondonfu commented May 4, 2021

Looks good to me, but I'm still looking into enabling CI runs for PRs based off of forks (thought I got it working, but looks like some additional tweaks are needed). Will approve & merge once we get CI running & passing.

@reubenr0d
Copy link
Contributor Author

reubenr0d commented May 4, 2021

Looks good to me, but I'm still looking into enabling CI runs for PRs based off of forks (thought I got it working, but looks like some additional tweaks are needed). Will approve & merge once we get CI running & passing.

Ahh, seems to me like the dockerhub creds aren't getting picked up from the Github Secrets over here. Maybe it would be a good idea to not do the docker push for MRs, but only on master merge and only do the build/tests locally for PRs? Let me see what I can do tomorrow.

@kyriediculous kyriediculous merged commit b933f4b into livepeer:master May 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve RemoteTranscoder error handling Remote Transcoder hangs after EOF
3 participants