Initial support for VP9 and AV1 simulcast (and fix for broken AV1/SVC)

[lminiero]

I recently read Chrome had started adding experimental support for VP9 and AV1 simulcast. Janus plugins are currently hardcoded to only accept simulcast for VP8 and H.264, so I wanted to check how much work it would take to remove that constraint and accept it for VP9 and AV1 as well.

As you can see in this patch (which currently only targets the EchoTest plugin), it took just a few minimal changes, namely:

1. making sure that when we create the simulcast envelope in janus.js we provide the right scalability mode (scalabilityMode: 'L1T2' in this case)
2. removing any VP8/H.264 constraint in the EchoTest when detecting simulcast
3. modifying the janus_rtp_simulcasting_context_process_rtp function to have it be aware of keyframe detection for other codecs as well.

This was enough for me to test the EchoTest demo using simulcast for both VP9 and AV1 on a recent Chrome version:

http://localhost:8000/echotest.html?simulcast=true&vcodec=vp9
http://localhost:8000/echotest.html?simulcast=true&vcodec=av1

In the next few days I'll patch other plugins to support the same functionality, so that it can be tested in more interesting scenarios as well (e.g., VideoRoom or RTP forwarders).

What I haven't figured out yet is how to deal with temporal layers (which would be two in the L1T2 case, for instance). For VP8, we parse the payload descriptor of the VP8 header to extract the temporal layer; for H.264 we don't do anything, instead, since temporal layers are not enabled when doing H.264 simulcasting. I don't know what should be done for VP9 and AV1 instead: should we parse the beginning of the payload the same way as we do for VP8? Or is a specific RTP extension used for the purpose instead? As such, at the moment, we don't do anything with those: I'll add support for that too as soon as I know more.

Feedback welcome!

Edit: blog post on the effort, for those that may be interested:
https://www.meetecho.com/blog/vp9-av1-simulcast-svc/

[lminiero]

I updated the patch so that temporal layers now work for both VP9 and AV1, with some caveats:

• for VP9, I went the easy way, and just used the SVC parsing code we had already, which means we parse the payload descriptor as we do for VP8;
• for AV1, I used the code I had written at the time for parsing the Dependency Descriptor, and so use that for extracting info on temporal layers if a DD is available.

In both cases it seems to be working as expected, in the simple tests I made.

Of course for AV1 this means actively negotiating the Dependency Descriptor extension, which Chrome doesn't do by default. Manually adding it to the SDP before setLocalDescription seemed to do the trick for me, so in the EchoTest as an example I just added this to the customizeSdp call:

jsep.sdp = jsep.sdp.replace("a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id",
	"a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\r\na=extmap:9 https://aomediacodec.github.io/av1-rtp-spec/#dependency-descriptor-rtp-header-extension");

Very ugly but it worked, and got Chrome to negotiate the DD extension and fill it with info on temporal layers (since I had used L1T2). I code that decides whether to relay or not depending on the temporal information extracted via the function is at the moment hardcoded in the EchoTest plugin code as a POC: I'll definitely clean this up so that it's, maybe, a separate *_process_rtp kind of function like we have for simulcast and SVC. I think I'll keep it separate, rather than bundling it in one of the two above, since otherwise I'd have to add a ton of extra arguments to their signatures which would be quite ugly; besides, it makes sense to leave DD-based processing on its own, even if this means doing simulcast + DD in a row for AV1 in this case.

Feedback welcome!

[lminiero]

Eureka! Finally got AV1/SVC to work too 🥳

In this case I did extend the existing janus_rtp_svc_context structure to envisage support for Dependency Descriptor parsing too, so the signature for janus_rtp_svc_context_process_rtp changed. I guess I'll probably do the same for janus_rtp_simulcasting_context_process_rtp, since at the moment, as explained above, using DD just for temporal layers when doing simulcast is a bit awkward for plugins.

I'll address this next week, and then move to extending support for all these new features to other plugins too (mainly the VideoRoom).

[lminiero]

I've made support for Dependency Descriptor a native part of the simulcast struct, as I already did for the SVC one: this means that the signature for their _process_rtp functions now just optionally asks for the DD data. At the moment, DD is only used if the codec is AV1 in both cases, but in the future we may extend it to other codecs too (where other mechanisms exist).

I also modified the other plugins to support this new flavours of simulcast, the VideoRoom in particular. The only one where support is still partial is the Streaming plugin, but the cause there is that packets arrive not from the Janus core (where we can provide a pointer to the DD data) but from the network, and so as generic packets: considering there may be heterogeneous sources, and we don't know if DD exists or what ID it has, it means we cannot get access to that info. I'll try to come up with a solution in a future PR, especially considering it may be useful also for remote publishers in the VideoRoom (which work in a similar way).

[lminiero]

Merging.