[1.0.1] Multiple Janus side candidates when NAT 1-1 is being used

[RaniRaven]

The phenomenon we get is the we receive multiple candidates from Janus due to some reason, no idea what ahs changed, when we use the NAT 1-1 settings.

To my understanding this is a BUG as NAT 1-1 means Janus "knows" the public endpoint, an certainly there is no reason whatsoever to give 2 candidates which are the very same ones, with 2 different ports. This can also cause issues as the range of ports can be exhausted fast in that way (for 1 subscriber and 1 publisher we will get 4 ports being used for connectivity check). It also doesn't make much sense, as the assumption is that all the port range is opened in the same way.

I've marked the issue

[218008902809326] There are 0 audio, 1 video and 0 data m-lines

Anonymized

---

[218008902809326] We have 2 candidates for Stream #1, Component #1
[218008902809326] Public IP specified and 1:1 NAT mapping enabled (198.208.11.78), using that as host address in the candidates
[218008902809326] Address: 10.126.180.30:33045
[218008902809326] Priority: 2015363327
[218008902809326] Foundation: 1
[218008902809326] 1 1 udp 2015363327 198.208.11.78 33045 typ host
[218008902809326] Public IP specified and 1:1 NAT mapping enabled (198.208.11.78), using that as host address in the candidates
[218008902809326] Address: 10.173.0.1:34176
[218008902809326] Priority: 2015363583
[218008902809326] Foundation: 2
[218008902809326] 2 1 udp 2015363583 198.208.11.78 34176 typ host

Merged (1054 bytes)

---

v=0
o=- 4803130522175137613 1654756009 IN IP4 198.208.11.78
s=VideoRoom 6
t=0 0
a=group:BUNDLE 0
a=ice-options:trickle
a=fingerprint:sha-256 A2:7A:01:A8:D1:6D:73:98:85:5C:DC:00:B5:AD:09:37:2E:47:53:ED:B7:DC:4D:59:66:F8:70:99:4E:EB:46:EC
a=extmap-allow-mixed
a=msid-semantic: WMS janus
m=video 9 UDP/TLS/RTP/SAVPF 102 121
c=IN IP4 198.208.11.78
a=recvonly
a=mid:0
a=rtcp-mux
a=ice-ufrag:0/tg
a=ice-pwd:4wqyZGCB+qBw9zQmqo8vVe
a=ice-options:trickle
a=setup:active
a=rtpmap:102 H264/90000
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack
a=rtcp-fb:102 nack pli
a=rtcp-fb:102 goog-remb
a=rtcp-fb:102 transport-cc
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=extmap:1 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=rtpmap:121 rtx/90000
a=fmtp:121 apt=102
a=msid:janus janus0
a=ssrc:1746293785 cname:janus
a=ssrc:0 cname:janus
a=candidate:1 1 udp 2015363327 198.208.11.78 33045 typ host
a=candidate:2 1 udp 2015363583 198.208.11.78 34176 typ host
a=end-of-candidates
Ad you can see in the pasted text from Janus log.
I can supply the full log, of course.

[lminiero]

There's no bug. You clearly have at least two interfaces active in Janus, each of them resulting in a different candidate (and so port), both of which Janus will put the nat-1-1 address you specified on, since it obviously doesn't know which one it should stick to unless you tell it. If you just want one, disable the other in janus.jcfg. Closing.

[RaniRaven]

I bag to differ, please read carefully the issues here.
I'm not sure about the need for using different port in order to understand the different interface, but maybe this is a part of ICE.
Nevertheless, I'm certain there is an issue here.
Since there is a NAT-1-1 used, always the candidates will have single host address, so if there are 2 interfaces between the NAT-1-1 value and the actual private location of Janus-Gateway, giving 2 different ports will only quadruple the problem, since each of the interfaces will get the ICE connection trial through each of the ports. It will not help at all, only complicate things. And the use of the ports, even for the negotiation part is useless, its just taking too much ports. Anyway, all the NATs in the way will try to allocate those ports inside, in vain.
I wonder whether this would have effect on the streaming as well.
Anyhow, it seems like in the specific case of NAT-1-1 it is very wrong to do that.

What is the configuration of janus.jcfg that will cause Janus-Gateway not to give 2 candidates in this case ? Cause this looks like a good way to overcome this problem.

[lminiero]

Since there is a NAT-1-1 used, always the candidates will have single host address

No, that only means that Janus will advertise a single address, but the actual addresses behind that are more than one. This is evident from the snippets you shared, where two different candidates have been gathered, one with address 10.126.180.30 and one with address 198.208.11.78, each with their own port. Since nat-1-1 is in use, Janus will replace both of those addresses with the public IP you chose, but since two candidates were gathered, and candidates with different ports, that's why you end up with two candidates with the same address and different ports. As such, clearly not a bug, but the intended behaviour.

Again, if you just want one candidate, then you have to programmatically disable one of the two interfaces, which means you have to tell Janus if it only needs to gather addresses on the interface with address 10.126.180.30, or the one with interface 198.208.11.78. You can exclude the interfaces or addresses using ice_ignore_list in janus.jcfg.

[RaniRaven]

o.k., now I'm sure there is a real issue.
Like you say, Janus is keeping inside the different interfaces, eventhough the NAT-1-1 flag is used.
It seems like Janus assumes that it will be able to understand the interface from the port. Anyway, it makes no much sense to give 2 candidates, which will be the very same ones , due to the NAT-1-1 option flag being used.
The thing is that what will happen is that all the interfaces from the other peer will try to send to all the ports packets, which will come from all the interfaces through all the ports (in the ICE negotiation part, till the establishment)
Now the main question is what will be the successful route after all, cause if we have 2 interfaces between the public IP and Janus-Gateway there will be now 4 options, but probably Janus-Gateway will not deal correctly with 50% (2) of them, cause the port will be different from the intended interface.

Actually, I guess this is what happens to us in the current architecture, we have a load balancer between the public IP and internal Janus-Gateway instance.

Now, the ice-ignore-list is working only from command line, the configuration file ignores it due to some reason, but it doesn't help since I tried to ignore each one of the interfaces and now ICE failure.
I am guessing this problem will happen to anyone that uses the NAT-1-1 flag, in any situation which is closed to reality than running inside a flat network in docker. If I am guessing correctly this might be the reason for the tons of complaints there are on google groups about ICE failed issue.

It doesn't make sense to me, the Janus-Gateway will try to "map" the interfaces by ports, in the case of NAT-1-1. Some different handling must be provided there.

[lminiero]

Will you PLEASE stop using this gigantic font to write your messages?

I can't explain why it's not an issue better than I already did. If you don't want to make an effort to read and understand my explainations, then I'm afraid there's nothing more I have to add.

[atoppi]

The thing is that what will happen is that all the interfaces from the other peer will try to send to all the ports packets, which will come from all the interfaces through all the ports (in the ICE negotiation part, till the establishment)
Now the main question is what will be the successful route after all

What will happen is that ICE protocol should do its job, that is finding working candidate pairs, either notified or discovered.

If I am guessing correctly this might be the reason for the tons of complaints there are on google groups about ICE failed issue.

I strongly disagree, the reason for most ICE failures is the misconfiguration of network in WebRTC deployments (setting up firewalls, available ports, TURN server in clients etc.).

It doesn't make sense to me, the Janus-Gateway will try to "map" the interfaces by ports, in the case of NAT-1-1. Some different handling must be provided there.

I don't understand what you mean by "mapping interfaces by ports". As Lorenzo explained, setting NAT-1-1 will just overwrite the address of any gathered local candidate with the one specified in the NAT-1-1.
There is no mapping or any other special network operation going on, just a bare overwriting in the candidate being sent to the peer.

[RaniRaven]

You miss my meaning, media servers don't need different ports for different candidates. This is not a requirement of the ICE. In the logs we can clearly see that each of the candidate is assigned, by Janus-Gateway with different port. Now, the question is what does Janus do with it :
[1] Ignores this "mapping' of candidate network interface to port
[2] Keep this inside, and do "problems" in case we get UDP packet, through an interface with different port.
If the 2nd is true, we have a problem, big one.
It still doesn't make any sense to just replace the candidate with the NAT-1-1, cause this way you'll get just as many ports in use for the negotiation phase, as the number of candidates, and that's make no sense.

Maybe what is missing here in the support for all ICE plan, but it seems like something is wrong.
It doesn't make sense that I remove the virtual interface of the docker which is not used by the final ICE completed check, and now I get an immediate failure, something is wrong and it doesn't have to do with ICE, but with the iimplementation of Janus.
I've already tracked some ICE debug logs, and saw that something is causing Janus specifically to fail, when there is a pair of successful candidates.

[lminiero]

We're done here.