Add support for L16 codec (uncompressed audio) #3116
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lminiero
changed the title
|
can you please file an issue in https://github.com/w3c/webrtc-nv-use-cases requesting access to raw audio data? |
|
Merging. |
|
Very nice, a project wz_mini_hacks, adds RTSP server to wyze cameras. It sends v2/v3 audio as
At the moment, it causes the stream not to play in chrome until the audio stream is removed. Looking forward to support for more sampling rates. Thank you for your work! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
One of the hidden "secrets" in Chrome is that you can tell it, by munging the SDP, to send RTP packets with uncompressed audio data, instead of encoding them. This means that, instead of using Opus, G.711 or whatever else, the samples are sent exactly as they're captured. This audio format is called L16, and uses two bytes per sample.
In general, you can enable it in the SDP by using the
L16/<samplerate>/<channels>rtpmap in the SDP, e.g.:Chrome supports a few variants of those (not sure how many, I haven't tried them all), so to start using it with Janus I added support for two different versions:
L16/16000(uncompressed wideband) andL16/48000(uncompressed fullband), in both cases mono only and not stereo (there will be time to expand on this).In case you're wondering why support this at all (as uncompressed audio is very inefficient to send around), the main reason is facilitating the experimentation with WASM audio codecs. You may have heard about Lyra, a new audio codec with very low bandwidth: a WASM version of it has already surfaced, and so it would be cool to see how it works in a WebRTC context, even if it's not available as a codec in browsers themselves. A way to do that is basically use a combination of WASM codecs, L16 samples, and Insertable Frames: you basically ask the browser for uncompressed audio samples, and then you encode/decode them yourself, using Insertable Streams to "replace" the uncompressed audio data in the original RTP packets with the bytes that have been encoded with WASM. In order to test all this with Janus, you need support for L16 negotiation, which is what we're adding here.
I tested this briefly with plain L16, and it seems to be working as expected. The patch also supports postprocessing L16 recordings (which of course only works if the RTP packets actually contain L16 data, and not Insertable Streams replacements). In case you do plan to experiment with WASM codecs and Janus as well, please do let me know, so that we can share some results.