Streaming Plugin: RTSP does not support "threads" configuration like RTP

[jreich888]

Since RTSP ultimately uses RTP for stream reception, the RTSP sources should support configuration for "threads" to add helper threads as is supported for RTP. When I try this for RTP I can see the created helper threads using ps -T, but no such additional threads exist for the RTSP streams.

When I look at the code (streaming plugin v0.0.8, Janus v0.10.5) I can see that the thread creation occurs only in the janus_streaming_create_rtp_source() function, and not in the _create_rtsp_source() method.
https://github.com/meetecho/janus-gateway/blob/v0.10.5/plugins/janus_streaming.c#L6060
Additionally, the threads parameter is not even parsed for the RTSP type case.

To reproduce, add a streaming source with a configuration as below. See that the sources is created and that when reviewing threads using ps -T that a main processor thread mp 1 exists but no helper threads for that source.

thread_test: {
        type = "rtsp"
        id = 1
        description = "Thread Test RTSP"
        audio = false
        video = true
        url = "rtsp://192.168.1.123/0"
        rtsp_user = "username"
        rtsp_pwd = "password"
        threads = 2
}

[atoppi]

HI @jreich888 ,
I've been going through your post on the group.

• Do you have any evidence that using helper threads on RTSP mountpoints will avoid the issue you described on huge KF ?
• Did you get to find the cause of the drop ?
• Have you tried increasing the kernel UDP buffers ?

[atoppi]

Another question: do you have a packet capture filtered on the RTSP negotiated ports ? (e.g. input capture (not with Admin API) on the ports that Janus is using to get media from the RasperryPi).

[jreich888]

Hi Alessandro, Thanks for the additional input and questions. Answering your four questions (over two messages): - Do you have any evidence that using *helper threads* on RTSP mountpoints will avoid the issue you described on huge KF ? - *[JR] No, just trying to confirm if they are supposed to be there. I'm guessing the packets could not be handled by the reading thread, but still looking for confirmation, and assume that helper threads would give more time for the receiver thread, by offloading the sending. Do you believe they would not help with higher packet rates?* - Did you get to find the cause of the drop ? - *[JR] Just inferring and looking for assistance from the group.* - Have you tried increasing the kernel UDP buffers ? - *[JR] Will try that; does Janus try to set anything while running? Or does it just use the system defaults?* - Another question: do you have a packet capture filtered on the RTSP negotiated ports ? (e.g. *input* capture (not with Admin API) on the ports that Janus is using to get media from the RasperryPi). - *[JR] Yes, I have packet captures of the packets received, the admin capture, and the transmitted packets from a single test run. The several "lost" packets are only present in the receive capture.* thanks for any advice and insight you can provide, Jason.

…

On Tue, Sep 15, 2020 at 2:05 AM Alessandro Toppi ***@***.***> wrote: Another question: do you have a packet capture filtered on the RTSP negotiated ports ? (e.g. *input* capture (not with Admin API) on the ports that Janus is using to get media from the RasperryPi). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2359 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB26YPQSM7YGWFEDFB2U243SF4U4HANCNFSM4RL7MRXA> .

-- ---- Jason Reich | [email protected].

[lminiero]

@jreich888 see the commit above, that you can find in PR #2361. I haven't tested yet, so please do that when you have some time and provide some feedback (on the PR, not here) on whether it works as expected.

[atoppi]

@jreich888 thanks for the response.
If the packets are present in the pcap at the mountpoint input then:

• I suspect there's some point where Janus is simply dropping them
• Changing kernel buffers should be useless at this moment (the OS got them)

Configuring the helper threads is definitely worth trying but I'm interested in tracking down the root issue that lead to the drop.
Very likely the same issue might be present in RTP mountpoints too.
Helpers threads should be used as an enhancement and not a fix.

[atoppi]

@jreich888 on second thought, setting a higher value to UDP recv buffer size is still worth trying.

To set ~1MB of buffer:
sysctl -w net.core.rmem_default=1064960
sysctl -w net.core.rmem_max=1064960

Janus is just using the system defaults, it does not set a specifc skt option in that regard.

[atoppi]

To check if UDP datagrams are being discarded the following command works on my machine (Ubuntu 20.04)

atoppi@atoppi:~$ sudo netstat -suna
IcmpMsg:
    InType3: 41
    OutType3: 830
Udp:
    227106 packets received
    1205 packets to unknown port received
    0 packet receive errors
    194589 packets sent
    0 receive buffer errors
    0 send buffer errors
    IgnoredMulti: 1970
UdpLite:
IpExt:
    InMcastPkts: 648
    OutMcastPkts: 637
    InBcastPkts: 1970
    OutBcastPkts: 5
    InOctets: 504174491
    OutOctets: 151779263
    InMcastOctets: 94376
    OutMcastOctets: 93495
    InBcastOctets: 450618
    OutBcastOctets: 377
    InNoECTPkts: 507411

packet receive errors and receive buffer errors are the parameters to monitor in this case

[jreich888]

Hi Alessandro, The UDP receive buffers are per socket, right? So I'm thinking they're likely the best fix for basic usage. (I'm thinking the threads will help, but will be most valuable when there are multiple viewers to send to) In my case, I'm suspecting that the OS was getting the packets (confirmed by packet capture), but the dropped as the rx buffer was full (with the Raspbian default buffer size of 80kB). The keyframes I originally saw in my sniffer were about 125 kB (in 1K packets), and I saw about 115 of them passed through and the rest missing. I'm running 1MB buffer size now. As for the threads PR, it may be another day or two before I can integrate / exercise it, but will let you know. thanks Jason.

…

On Wed, Sep 16, 2020 at 3:18 AM Alessandro Toppi ***@***.***> wrote: To check if UDP datagrams are being discarded the following command works on my machine (Ubuntu 20.04) ***@***.***:~$ sudo netstat -suna IcmpMsg: InType3: 41 OutType3: 830 Udp: 227106 packets received 1205 packets to unknown port received 0 packet receive errors 194589 packets sent 0 receive buffer errors 0 send buffer errors IgnoredMulti: 1970 UdpLite: IpExt: InMcastPkts: 648 OutMcastPkts: 637 InBcastPkts: 1970 OutBcastPkts: 5 InOctets: 504174491 OutOctets: 151779263 InMcastOctets: 94376 OutMcastOctets: 93495 InBcastOctets: 450618 OutBcastOctets: 377 InNoECTPkts: 507411 packet receive errors and receive buffer errors are the parameters to monitor in this case — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2359 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB26YPUZCIBG5HKUWDQQN3LSGCGGXANCNFSM4RL7MRXA> .

-- ---- Jason Reich | [email protected].

[atoppi]

The UDP receive buffers are per socket, right?

Yes, that setting is per socket.

In my case, I'm suspecting that the OS was getting the packets (confirmed by
packet capture), but the dropped as the rx buffer was full (with the
Raspbian default buffer size of 80kB).

I agree with your analysis and you could confirm it with netstat -suna.

I'm running 1MB buffer size now

Did it solve the issue ?