Use ffmpeg to limit the read rate instead of setTimeout

[longnguyen2004]

Currently, setTimeout is used in AudioStream and VideoStream to control the read rate of the media. FFmpeg has a built-in flag -re that does the same thing, and is potentially better? Need some more experimenting to see whether there's an improvement, and whether it impacts actual live streams (RTSP, SRT,...)

Some info about the flag: https://ffmpeg.org/ffmpeg.html#:~:text=as%20live%20streaming.-,%2Dre%20(input),-Read%20input%20at

[dank074]

We previously used the -re flag but it resulted in desynchronized audio/video for certain hls streams:

#31 (comment)

[longnguyen2004]

Interesting, I'll see if the related -readrate_initial_burst can help out here. Do you still keep the hls stream?

[dank074]

This is the stream I was testing with which was bugged:

https://rbmn-live.akamaized.net/hls/live/590964/BoRB-AT/master_1660.m3u8

[longnguyen2004]

Did some more testing with the -re flags, and during the process I discovered a bug with the setTimeout approach.

It assumes that the entire frame will arrive at once, which is not the case. At least on my Node.js installation, the incoming buffer size maxes out at 64KB, which is definitely not enough to contain the whole video frame, which can reach up to 1MB. This results in an effective frame rate of 1/3 to 1/4 actual. Using the -re flag fixes the issue, since ffmpeg knows where the frame boundaries are and will limit the read rate accordingly. We can do the same thing in our code by moving the setTimeout to the packetizer, but that's fragile.

About the -re flag itself, it works really well for local files, but fails on HLS streams, with stuttering audio and video. Removing the -re flag makes it even worse, with frame rates jumping all over the place. We'll probably have to find another solution for HLS streams, since neither approach works well with HLS livestreams (haven't tested HLS VOD, but I assume it'll work the same as local files)

[dank074]

Did some more testing with the -re flags, and during the process I discovered a bug with the setTimeout approach.

It assumes that the entire frame will arrive at once, which is not the case. At least on my Node.js installation, the incoming buffer size maxes out at 64KB, which is definitely not enough to contain the whole video frame, which can reach up to 1MB. This results in an effective frame rate of 1/3 to 1/4 actual. Using the -re flag fixes the issue, since ffmpeg knows where the frame boundaries are and will limit the read rate accordingly. We can do the same thing in our code by moving the setTimeout to the packetizer, but that's fragile.

About the -re flag itself, it works really well for local files, but fails on HLS streams, with stuttering audio and video. Removing the -re flag makes it even worse, with frame rates jumping all over the place. We'll probably have to find another solution for HLS streams, since neither approach works well with HLS livestreams (haven't tested HLS VOD, but I assume it'll work the same as local files)

Unfortunately it wasn't just hls streams that had this weird behavior. Author of issue also reported that some local mkv video files also had delayed audio when the -re flag was used #31 (comment)

[longnguyen2004]

About ffplay: It has an internal packet queue, which allows it to buffer a few hundred KBs of video to smooth out the playback. Not sure if we can add some kind of buffer in the library.