Skip to content

Commit

Permalink
jvb: remove TCP support
Browse files Browse the repository at this point in the history 
It's not tested at all and has problems. The recommendation is to use a
TURN server for TCP.
  • Loading branch information
@saghul
saghul committed Feb 15, 2022
1 parent c37706c commit 7a93978
Show file tree
Hide file tree
Showing 3 changed files with 0 additions and 18 deletions.
4 changes: 0 additions & 4 deletions docker-compose.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -263,7 +263,6 @@ services:
restart: ${RESTART_POLICY}
ports:
- '${JVB_PORT}:${JVB_PORT}/udp'
- '${JVB_TCP_PORT}:${JVB_TCP_PORT}'
volumes:
- ${CONFIG}/jvb:/config:Z
environment:
Expand All @@ -275,9 +274,6 @@ services:
- JVB_BREWERY_MUC
- JVB_PORT
- JVB_MUC_NICKNAME
- JVB_TCP_HARVESTER_DISABLED
- JVB_TCP_PORT
- JVB_TCP_MAPPED_PORT
- JVB_STUN_SERVERS
- JVB_OCTO_BIND_ADDRESS
- JVB_OCTO_PUBLIC_ADDRESS
Expand Down
5 changes: 0 additions & 5 deletions env.example
View file
Original file line number Diff line number Diff line change
Expand Up @@ -257,11 +257,6 @@ JVB_STUN_SERVERS=meet-jit-si-turnrelay.jitsi.net:443
# Media port for the Jitsi Videobridge
JVB_PORT=10000

# TCP Fallback for Jitsi Videobridge for when UDP isn't available
JVB_TCP_HARVESTER_DISABLED=true
JVB_TCP_PORT=4443
JVB_TCP_MAPPED_PORT=4443

# XMPP user for Jicofo client connections.
# NOTE: this option doesn't currently work due to a bug
JICOFO_AUTH_USER=focus
Expand Down
9 changes: 0 additions & 9 deletions jvb/rootfs/defaults/jvb.conf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
{{ $ENABLE_COLIBRI_WEBSOCKET := .Env.ENABLE_COLIBRI_WEBSOCKET | default "1" | toBool }}
{{ $ENABLE_OCTO := .Env.ENABLE_OCTO | default "0" | toBool }}
{{ $JVB_MUC_NICKNAME := .Env.JVB_MUC_NICKNAME | default .Env.HOSTNAME -}}
{{ $JVB_TCP_PORT := .Env.JVB_TCP_PORT | default "4443" }}
{{ $JVB_TCP_MAPPED_PORT := .Env.JVB_TCP_MAPPED_PORT | default $JVB_TCP_PORT }}
{{ $PUBLIC_URL_DOMAIN := .Env.PUBLIC_URL | default "https://localhost:8443" | trimPrefix "https://" | trimSuffix "/" -}}
{{ $SHUTDOWN_REST_ENABLED := .Env.SHUTDOWN_REST_ENABLED | default "false" | toBool }}
{{ $WS_DOMAIN := .Env.JVB_WS_DOMAIN | default $PUBLIC_URL_DOMAIN -}}
Expand All @@ -14,13 +12,6 @@ videobridge {
udp {
port = {{ .Env.JVB_PORT }}
}
tcp {
enabled = {{ not (.Env.JVB_TCP_HARVESTER_DISABLED | default "true" | toBool) }}
port = {{ .Env.JVB_TCP_PORT }}
{{ if not (eq $JVB_TCP_PORT $JVB_TCP_MAPPED_PORT) }}
mapped-port = {{ $JVB_TCP_MAPPED_PORT }}
{{ end }}
}
}
apis {
xmpp-client {
Expand Down

32 comments on commit 7a93978

@bernhardreiter
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What problems does it have in particular?

@saghul
Copy link
Member Author

@saghul saghul commented on 7a93978 Feb 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not tested nor used by us and we've seen weird deadlocks IIRC. Our recommendation is to use a TURN server instead.

@peb-adr
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have this feature in use productively and heard only good feedback about it.
We were very happy to have it running like this without the overhead/added complexity of a turn server.
Seeing it getting removed means we'll have to rework our setup again :(

Is the functionality in the jvb itself even planned to be removed?
I think, as long as that's not the case, it seems strange to have the support for an existing feature removed from the docker setup.

@saghul
Copy link
Member Author

@saghul saghul commented on 7a93978 Feb 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will eventually get removed from the JVB yeah.

@bernhardreiter
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have this feature in use productively and heard only good feedback about it.

Same here. What I hear from operations, it is even more robust than a TURN server in settings, where communication partners have aggressive firewalls and we set the fallback port to TCP 443. That is because TCP 443 is allowed through quite often with firewalls.

@saghul
Copy link
Member Author

@saghul saghul commented on 7a93978 Feb 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can run the TURN server yourself, then the problem disappears.

While you might not have run into issues yet, they exist, believe me.

@bernhardreiter
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@saghul thanks for the info, can you be more specific what the root or the cause of the issues may be?

@saghul
Copy link
Member Author

@saghul saghul commented on 7a93978 Feb 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I said, deadlocks IIRC.

I guess that if we knew precisely how to fix it we’d have done that ;-) After some analysis we concluded using an external TURN server was a better option going forward.

We have been using coturn successfully in production for a long time now.

@DanielMcAssey
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got to agree with @saghul we had a few issues with JVB TCP, we now run a few TURN servers on port 443 for aggressive firewalls.

@bernhardreiter
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@saghul @DanielMcAssey thanks for sharing your experiences!

@bgrozev
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another reason we use TURN/TLS instead of ICE/TCP is that ICE/TCP uses a fake SSL handshake that can be detected and blocked by firewalls.

@cedricroijakkers
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could someone elaborate a bit on how to set up a TURN server and configure it in the Docker environment variables? The documentation in the handbook is, depending how you look at it, either not applicable or partially missing. We use the TCP mode of JVB almost exclusively, and while it has its quirks, it works flawlessly. It might be buggy, but it still works better than other WebRTC products that use UDP.

@saghul
Copy link
Member Author

@saghul saghul commented on 7a93978 Mar 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can configure an external TURN server as follows:

docker-jitsi-meet/env.example

Line 370 in 1b51c77

# Configure an external TURN server

@mghadam
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was using JVB TCP as well in production and it was working fine.

Now I have to switch to TURN. Are you going to implement a TURN server docker in the docker-jitsi-meet ? I see there is a TURN server. line available in TODO section of the github page.

Any recommendations on which docker image / open source TURN server product I shall use for Jitsi?

@damencho
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The debian packages by default use coturn with this config: https://github.com/jitsi/jitsi-meet/blob/master/doc/debian/jitsi-meet-turn/turnserver.conf
You need to take care of using valid certificates for the domain used to access the turn server.

@mghadam
Copy link

@mghadam mghadam commented on 7a93978 Mar 11, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like there is an official docker image for coturn: https://github.com/coturn/coturn/tree/master/docker/coturn

Going to check whether I can configure docker-jitsi-meet to run with this docker image over TCP correctly

@mghadam
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am trying to force jitsi to use TCP with no UDP and no P2P (config.webrtcIceUdpDisable=true&config.webrtcIceTcpDisable=false&config.p2p.enabled=false).

Can an external TURN server such as coturn work in this case?

I have tried many configs and still cant get jitsi running in TCP with coturn

@mghadam
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I managed to run coturn docker image successfully and validated the TURN server by using Trickle ICE

Jitsi meet docker does not still work when UDP is disabled with the TURN server.

Has anybody here been able to use jitsi / TCP with coturn ?

@damencho
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jitsi meet docker does not still work when UDP is disabled with the TURN server.

What do you mean? The turn server should be able to use udp, as it needs it to connect to jvb.

@bgrozev
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am trying to force jitsi to use TCP with no UDP and no P2P (config.webrtcIceUdpDisable=true&config.webrtcIceTcpDisable=false&config.p2p.enabled=false).

These config flags probably don't work as you intend with TURN. When allocating through TURN/TLS the candidate's protocol will be 'UDP', so webrtcIceUdpDisable=true will effectively disable TURN. Also, these only supress signaling certain candidates, while they may still be used. I suggest not using webrtcIceUdpDisable and webrtcIceTcpDisable and blocking the network instead.

@cedricroijakkers
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've managed to get it working by setting these environment variables in the deployment:

TURN_CREDENTIALS: "very_secret"
TURNS_HOST: "turnserver.mydomain.com"
TURNS_PORT: "443"

And then blocking udp port 10000 traffic with:

sudo iptables -A OUTPUT -p udp --dport 10000 -j DROP

And then starting the Jitsi meeting. With a tcpdump you will then see all traffic going to the TURN server on port 443, instead of the Jitsi server on port 10000.

Don't forget you need to at least have 3 participants in the meeting otherwise you will have a peer-to-peer connection.

@mghadam
Copy link

@mghadam mghadam commented on 7a93978 Mar 15, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I eventually managed to run a coturn-docker on the same server as the jitsi-docker server and force TLS/port 4443 for all clients.

First, p2p can be disabled by adding config.p2p.enabled=false to the URL of the jitsi meet room. This will make all clients to use JVB.

Second, You may disable direct UDP connections to JVB if you wish your clients to always use TLS/TCP. I commented out the DOCKER_HOST_ADDRESS= line in the .env file of docker-jitsi-meet for this purpose. This way, the Videobridge will advertise the internal IP address of the Docker which is inaccessible by clients.

Third, Configure the turn server in jitsi-docker .env file:

# Configure an external TURN server
TURN_CREDENTIALS=SECRETKEY
# TURN_HOST=turnserver.example.com
# TURN_PORT=4443
TURNS_HOST=JITSI.MYDOMAIN.COM
TURNS_PORT=4443

Since coturn-docker is hosted on the same server as jitsi-docker, TURNS_HOST is set to the same domain that we have used for jitsi and we have got a SSL for it using LETSENCRYPT_DOMAIN section of the .env file.

Fourth, Add the following entries in the docker-compose.yml file of docker-jitsi-meet-stable-7001, to set up a coturn container alongside other containers:

    coturn_server:
        image: coturn/coturn:latest
        restart: ${RESTART_POLICY}
        ports:
            - "4443:4443/tcp"
        volumes:
            - /usr/src/docker-jitsi-meet-stable-7001/turnserver.conf:/etc/coturn/turnserver.conf
            - /usr/src/jitsi-meet-cfg/web/acme.sh/JITSI.MYDOMAIN.COM/fullchain.cer:/cert.pem
            - /usr/src/jitsi-meet-cfg/web/acme.sh/JITSI.MYDOMAIN.COM/JITSI.MYDOMAIN.COM.key:/cert.key
        networks:
            meet.jitsi:
  • Make sure that the above certificate and key files exist in the path provided.

Finally the content of /usr/src/docker-jitsi-meet-stable-7001/turnserver.conf should be the default jitsi config file for coturn:

# jitsi-meet coturn config. Do not modify this line
use-auth-secret
keep-address-family
static-auth-secret=SECRETKEY
realm=JITSI.MYDOMAIN.COM
cert=/cert.pem
pkey=/cert.key
no-multicast-peers
no-cli
no-loopback-peers
no-tcp-relay
no-tcp
listening-port=3478
tls-listening-port=4443
no-tlsv1
no-tlsv1_1
# https://ssl-config.mozilla.org/#server=haproxy&version=2.1&config=intermediate&openssl=1.1.0g&guideline=5.4
cipher-list=ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
# jitsi-meet coturn relay disable config. Do not modify this line
denied-peer-ip=0.0.0.0-0.255.255.255
denied-peer-ip=10.0.0.0-10.255.255.255
denied-peer-ip=100.64.0.0-100.127.255.255
denied-peer-ip=127.0.0.0-127.255.255.255
denied-peer-ip=169.254.0.0-169.254.255.255
denied-peer-ip=127.0.0.0-127.255.255.255
denied-peer-ip=172.16.0.0-172.31.255.255
denied-peer-ip=192.0.0.0-192.0.0.255
denied-peer-ip=192.0.2.0-192.0.2.255
denied-peer-ip=192.88.99.0-192.88.99.255
denied-peer-ip=192.168.0.0-192.168.255.255
denied-peer-ip=198.18.0.0-198.19.255.255
denied-peer-ip=198.51.100.0-198.51.100.255
denied-peer-ip=203.0.113.0-203.0.113.255
denied-peer-ip=240.0.0.0-255.255.255.255
denied-peer-ip=::1
denied-peer-ip=64:ff9b::-64:ff9b::ffff:ffff
denied-peer-ip=::ffff:0.0.0.0-::ffff:255.255.255.255
denied-peer-ip=100::-100::ffff:ffff:ffff:ffff
denied-peer-ip=2001::-2001:1ff:ffff:ffff:ffff:ffff:ffff:ffff
denied-peer-ip=2002::-2002:ffff:ffff:ffff:ffff:ffff:ffff:ffff
denied-peer-ip=fc00::-fdff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
denied-peer-ip=fe80::-febf:ffff:ffff:ffff:ffff:ffff:ffff:ffff
syslog

  • the docker ip range needs to be commented out in the above denied-peer-ip if you wish the coturn docker access your jvb using its internal IP rather than the public IP

  • Note that Jitsi does not show you anything about the TURN server / port in the connection information, and it shows the JVB IP / port . It would be nice if the turn server / port is also shown in the connection information.

@damencho
Copy link
Member

@damencho damencho commented on 7a93978 Mar 15, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@madmatah Adding the "denied-peer-ip" is a security requirement, otherwise, you are exposing your internal network to the world, you better add them. The coturn will be accessing jvb on its public address when you drop the external-ip config from the coturn config.

@saghul
Copy link
Member Author

@saghul saghul commented on 7a93978 Mar 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How are you making sure the HTTPS traffic doesn’t go to coturn?

@mghadam
Copy link

@mghadam mghadam commented on 7a93978 Mar 15, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@madmatah Adding the "denied-peer-ip" is a security requirement, otherwise, you are exposing your internal network to the world, you better add them. The coturn will be accessing jvb on its public address when you drop the external-ip config from the coturn config.

You were right! It worked when I dropped the external-ip line in coturn config. It was not also required for coturn to be on the same network as jvb. I have updated this in my post above.

How are you making sure the HTTPS traffic doesn’t go to coturn?

I'm actually using port 4443 in production. I have updated my post above with the correct port.

@saghul
Copy link
Member Author

@saghul saghul commented on 7a93978 Mar 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually using port 4443 in production.

I think 443 can be used if you configure nginx to proxy the traffic to coturn using SNI for detecting the domain, but I'm not sure if there might be incompatibilities.

@mghadam
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually using port 4443 in production.

I think 443 can be used if you configure nginx to proxy the traffic to coturn using SNI for detecting the domain, but I'm not sure if there might be incompatibilities.

This is apparently not possible according to coturn/coturn#43

@saghul
Copy link
Member Author

@saghul saghul commented on 7a93978 Mar 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it might be possible in certain scenarios (coturn/coturn#702) but not as straightforward as I thought, oh well.

@cedricroijakkers
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@madmatah Adding the "denied-peer-ip" is a security requirement, otherwise, you are exposing your internal network to the world, you better add them. The coturn will be accessing jvb on its public address when you drop the external-ip config from the coturn config.

Interesting. I have Jitsi and coturn running in the same Kubernetes cluster, so I had the same issue that when looking in the connection stats in Jitsi itself for my self-view, I could see the internal 10.x.x.x IP address of the Jitsi JVB pod as the remote address, and the internal 10.x.x.x IP address of the coturn pod as the local address. So I was connecting via TCP to coturn (obviously over the internet, over the public LoadBalancer service in k8s towards coturn) but coturn was connecting via UDP to JVB using the internal k8s connection, hence I could see both the internal IPs in my connection view.

Since then, I have added the denied-peer-ip lines and removed the external-ip line from the turnserver.conf file, which is still working, but now shows a slightly different status in my connection self-view: Remote address is now the public IP address of the JVB service, Local address is still the 10.x.x.x IP address of the coturn pod. So I'm not entirely sure how the connection is now flowing. It is still working fine though for people with UDP blocked, but I would actually expect to see the coturn public IP address in there. Or maybe this is just something that is inherent to docker/k8s internal networking?

Anyway, adding the denied-peer-ip lines and blocking tcp relaying is probably a good idea, otherwise coturn could in theory relay network traffic to my other internal Kubernetes services.

@damencho
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is apparently not possible according to coturn/coturn#43

Not sure this is applicable here, but it can be done behind proxy using a different DNS https://jitsi.github.io/handbook/docs/devops-guide/turn#use-turn-server-on-port-443

@fbezdeka
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mghadam: Thanks for providing 7a93978#commitcomment-68759524. That fixed our setup behind a loadbalancer. Thanks!

@saghul
Copy link
Member Author

@saghul saghul commented on 7a93978 Mar 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is apparently not possible according to coturn/coturn#43

Not sure this is applicable here, but it can be done behind proxy using a different DNS https://jitsi.github.io/handbook/docs/devops-guide/turn#use-turn-server-on-port-443

That's what I mentioned about SNI, but there seem to be some bugs? coturn/coturn#702

Please sign in to comment.