Unified install_ffmpeg.sh for go-livepeer and lpms with HEVC and VP8/9 support

[cyberj0g]

What does this pull request do? Explain your changes. (required)

• support for HEVC, VP8/9 PR on go-livepeer side
• use a single install_ffmpeg.sh for both lpms and go-livepeer, which will reside in go-livepeer repo
• Support additional container formats for VP8/9 #2134

Specific updates (required)

All changes are in install_ffmpeg.sh:

• add libx265 and libvpx
• add debug-video mode for development and testing of lpms
• add libtensorflow installation

How did you test each of these updates (required)

Does this pull request close any open issues?

#2134

Checklist:

• Read the contribution guide
• make runs successfully
• All tests in ./test.sh pass
• [n/a] README and other documentation updated
• [n/a] Pending changelog updated

[iameli]

Concerned about multi-platform support - this script should work in as many environments as possible. I can live with no Tensorflow on Windows if we don't think it's feasible, but we should be able to get x265 and libvpx working on Mac and Windows.

[cyberj0g]

Concerned about multi-platform support - this script should work in as many environments as possible. I can live with no Tensorflow on Windows if we don't think it's feasible, but we should be able to get x265 and libvpx working on Mac and Windows.

Why support any software codec on any platform in production, and why spend efforts to support unusual codec-platform combinations for development build, which only required for running some tests in lpms?

[darkdarkdragon]

Concerned about multi-platform support - this script should work in as many environments as possible. I can live with no Tensorflow on Windows if we don't think it's feasible, but we should be able to get x265 and libvpx working on Mac and Windows.

Why support any software codec on any platform in production, and why spend efforts to support unusual codec-platform combinations for development build, which only required for running some tests in lpms?

Not just tests in lpms - I'm running go-livepeer node on Mac in software mode for development and would like to be able to continue to do so 😄
Also, many people use go-livepeer node in production on Windows, so them should have software versions of codecs too.

[cyberj0g]

I'm running go-livepeer node on Mac in software mode for development and would like to be able to continue to do so

In this case I invite you to add support for MacOS, because I don't have access to one and won't be able to test it 😁

many people use go-livepeer node in production on Windows

Does someone really running a transcoder on Windows with software codecs? If it's remotely possible with H.264, HEVC and Vp8/9 unlikely to work real-time. Nvidia codecs are working just fine without libx265 and libvpx.

[iameli]

Does someone really running a transcoder on Windows with software codecs? If it's remotely possible with H.264, HEVC and Vp8/9 unlikely to work real-time.

.\ffmpeg.exe -i "2021-12-15 07-17-32.mkv" -c:v libx265 -c:a copy -y "2021-12-15 07-17-32.h265.mkv" on my mid-range Windows PC, no tuning of that command whatsoever, seems to produce just under 3x realtime transcoding. Why would non-Linux platforms be slower? It's all just C.

Why support any software codec on any platform in production

Because we're a weird, decentralized transcoding platform. It's very hard for us to predict ahead of time the unusual environments where our software will be deployed. If odd platform combinations end up causing bugs, that's one thing, and we could have a conversation about the effort required to fix those bugs. But so far, once we've gotten things compiling successfully, the actual transcoding has worked flawlessly on all three platforms. Let's not give up on that just because it makes the build script a bit more complex!

and why spend efforts to support unusual codec-platform combinations for development build, which only required for running some tests in lpms?

I don't follow — this script isn't just for the development build, it's also what we use to produce binaries for distribution. Surely those tests exist in lpms because we want to eventually support these codecs in go-livepeer?

In this case I invite you to add support for MacOS, because I don't have access to one and won't be able to test it 😁

That's fair! I'd ask that you start out by writing a version of this that works the same on all platforms and let the CI process attempt to build it. It might just work out of the box. If you run into platform-specific compile problems, or you want @darkdarkdragon or I to test the resulting binaries on Mac/Windows, give us a ping.

[cyberj0g]

seems to produce just under 3x realtime transcoding.

That's faster than I expected, however, I suggest running the test using our test segment:

wget https://storage.googleapis.com/lp_testharness_assets/bbb_1080p_30fps_1min_2sec_hls.tar.gz

And ffmpeg command closer resembling transcoding logic:

ffmpeg -y -i bbb/source.m3u8 -filter_complex '[0:v]split=3[out1][out2][out3]' \
                  -map '[out1]' -s 1280x720 -acodec copy -vcodec libx265 output720p.mp4 \
                  -map '[out2]' -s 640x360  -acodec copy -vcodec libx265 output360p.mp4 \
                  -map '[out3]' -s 320x240  -acodec copy -vcodec libx265 output240p.mp4

Also try libvpx and libvpx-vp9 with .mkv outputs. According to last GPU benchmarks (with H.264, though), GTX 1080 was able to run about 20 such sessions real-time in parallel with mining or CUDA computations. You can calculate LPT payout per watt for CPU, and I believe it will be much lower than electricity cost. Livepeer was never intended to run on CPU and existence of Windows software transcoding nodes is news to me. If we must support that - I'll fix the script, no problem.

I don't follow — this script isn't just for the development build

As I see it:

• all production deployments (where users supposed to be incentivized by payouts) are using hardware acceleration and Nvidia codecs (hevc_cuvid, h264_cuvid, vp8_cuvid, vp9_cuvid, hevc_nvenc, h264_nvenc), which don't require software codecs libx265 or libvpx - so they are not built by this version of script for any platform
• developers may want to use software codecs if they don't have Nvidia GPU in their environment
• tests in lpms are using software codecs only for two things: 1. to run on CircleCI without GPU 2. to validate encoding quality of hardware codecs compared to reference software decoded segments.

For me, it makes perfect sense to not support libvpx and libx265 at all in this codec-related PR. We can add support of these for developers separately, or even just add install instructions to contributing guide.

[iameli]

all production deployments (where users supposed to be incentivized by payouts) are using hardware acceleration and Nvidia codecs

I'm not confident of that at all - I think people run all sorts of different hardware on the network, though I hear you that these software encoders are much slower than others.

Started hacking on a cross-platform approach here, but it'll probably take some iteration to get right: #2136. I'm cool with this landing in the meantime if it unblocks our support for hardware HEVC and whatnot.

[cyberj0g]

Added comment on static build of new codecs: livepeer/lpms#279 (comment)

[yondonfu]

Chiming in on the discussion of supporting software based transcoding for the new codecs:

I see the following prioritization right now for new codec support:

1. HW transcoding on Linux
   • Relevant for production usage, since its the most common [1] setup on the network
2. HW transcoding on Windows
   • Relevant for production usage, since its the second most common [2] setup on the network
3. SW transcoding on Linux
4. SW transcoding on Mac
   • I don't expect this to ever be used in production, but since Mac/Linux are common development environments I rank this above SW transcoding on Windows
5. SW transcoding on Windows

Given the above prioritization, I think it is reasonable to focus on 1/2 and to look into 3/4/5 separately if a) it is challenging to support everything at once and b) we believe there is a reasonable way for someone without access to a GPU to do manual testing with go-livepeer for new codecs (not to test the actual decoding/encoding, but for the code paths around selecting the codec). If we do this, we can just make it clear that initially there is limited support for new codecs.

At the same time, while for new codecs we can opt to support HW transcoding first and look into supporting SW transcoding separately, I do think it is important not to take away support for SW transcoding for existing codecs (i.e. H264) just because I think we should aim to avoid removing features unless we absolutely need to (i.e. security reasons, we are confident that the pain of supporting it outweighs any benefit certain users might get from it, etc.).

[1] This is anecdotal, but I feel like its right...
[2] This is also anecdotal, but I also feel like its right...

resolved in following conversation

[yondonfu]

Hm looks like the Linux Tensorflow Build is failing. Any idea why?

[cyberj0g]

Hm looks like the Linux Tensorflow Build is failing

Fixed, I initially thought it's CircleCI caching issue, but it was permission-related.

[yondonfu]

LGTM! Let's squash all the commits prior to merging.