No ws/udp service definition for JVB ? #11
Comments
|
Hey Ryan ! I'm glad you're interested. About the server snippet, we added it because conferences were dropped, but I recently understood that the issue was the default proxy timeout. (It's the same as the default heartbeat of meet over the websocket, this mean that with a little delay, the connection would drop). So, I changed the proxy timeout and removed the snippet in order to avoid duplicating work with upstream. For now, we don't require a JVB service because the UDP port is binded via pod's hostPort and when accessing colibri-ws the client uses the cluster internal pod ip in the url that is then used by nginx (in the web pod) to connect directly to the jvb pod. Do you have network policies that could block traffic not explicitly allowed ? |
|
Hey @hrenard thanks for the quick reply! Yeah I had heard about the proxy-timeout issues with nginx and websocket proxying and needing the proxy annotations set, however my nginx ingress controller only seems to respond to nginx.org/ annotations (maybe this is a recent change?) Thanks for the tip on the colibri-ws nginx config in the web container, i hadn't noticed it being present there. Ok, so I shouldn't need a server snippet in my ingress annotation any more, only the proxy-timeout stuff, and let the web container handle the proxying. Im not sure why yet, but I can see status code 101's for switching protocols on ws upgrade to /xmpp-websocket in my ingress nginx and web container logs, but can just see connections being opened and immediately closed in the prosody logs, so the web UI is just trying to force me to reconnect endlessly currently. I am just running this from inside my home network at the moment, so I don't have any network policies blocking anything afaik. As for the UDP port, yes I see I've got the UDP port bound to hostPort, which is a private ip, so do you expose this to clients on the internet at all, or are you just relying on the websocket connection? I'm guessing the latter? |
|
Which ingress controller are you using ? I only added annotations for this one : https://github.com/kubernetes/ingress-nginx. When you same home network, do you mean being a NAT or firewall ? Because for now, the JVB discover its public IP through STUN. So, if your router doesn't let traffic on port 10000/UDP through to the JVB, it's likely meet refreshes the UI because you are not able to connect to the JVB and in the same process closes the XMPP websocket. |
|
Hi again - sorry for the late reply - been mad busy! Well, it turns out I was using nginx's ingress-nginx off the Rancher marketplace, not kubernetes ingress-nginx (https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx) which annoyingly have the same name but do behave a bit differently. So I've switched to the right one now. The jitsi ingress now has a public IP which is the same one being detected by STUN in jvb, and has the appropriate proxy config in nginx.conf without any additional mods to the ingress annotations. I also use BGP in metallb to advertise the jitsi nginx ingress IP to my opnsense router, so it seems to registers the public IP address there ok. That sorts out the routing side, and then I use firewall to ensure only port 443/tcp and 30100/udp are allowed in on the WAN. Because you're using host port for udp, I thought it might be better if I use udp ingress with nginx to proxy to the request to the jvb, but haven't got that working quite yet. I'm going to have another crack at it again tonight so will keep you posted. |
|
We could also add compatibility for other ingress controllers... Using ingress for jvb is complicated because you'll need an ingress by jvb because prosody tells the client on wich jvb to go. So, autoscaling would become harder. Also, it would add a lot of stress on ingress and an extra hop, for no gain. And finally, k8s is for now quite bad at handling udp traffic in general. A better solution would be to use a service in NodePort mode. And we'll probably do that. We just need to know the nodePort for advertising. But, mostly for historical reason (customers with firewall whitelist) we kept the default jvb port (10000) wich isn't in the node port range by default, so hostPort 😄 . |
|
Hi again - thanks for your feedback. I've decided to switch back to the 'default' simple jitsi manifest and work forward from there again. So I'm using port 10000/udp on jvb now, and I've set up a temporary port forward in my router to allow public ip access to the JVB hostPort. That seems to reach ok from the internet, though if I restart jvb, the node IP changes and I have to update the rule on the router, so its not ideal but just doing it this way for basic testing first of all. The web ingress seems to be working ok up until I need a colibri-ws connection, which 403's after a few seconds. Having googled around extensively, I'm thinking at this point I'm going to need coturn deployed, and get some turncredentials added to prosody to make the p2p stuff work. Still trying to get this working, but I do feel like I'm getting nearer to a working state. |
|
I managed to get the prosody external_services plugin working with the default jitsi stun, and enabling the useStunTurn in config.js via configmap: and then add this into the jitsi crd manifest: Now I just need to get this config into the prosody.cfg.lua as well: I'm guessing this is done through : Thats the bit I'm working on now, and then I'll make my own coturn deployment and use that instead of the jitsi one. I'm pretty pleased I've managed to get something fairly reliable working, and I think I've got what I need now to finish the job. Thanks for your help though, and great operator - well done :) |
|
Hey, Glad you're making it work ! |
|
Yep, thank you - I just added the customProsodyConfigCM and its working now. I notice the current implementation is to completely override the jitsi-meet.cfg.lua file with the content of the file in the configmap. I guess this is ok, as I used what was already there as a template, but I suppose changes in upstream won't be reflected there until you take off the configmap and allow the operator to create a new templated file for you first. Are you interested in a helm chart being created for the operator yet, or are you waiting until the operator has reached a bit more maturity? |
|
Yes, to minimize maintenance burden, we chose to use mostly upstream configurations. But we added some escape hatches like this for experienced admins. I'm not against a helm chart, but I don't believe it to be very useful, as we can install and update the operator with a one-liner : |
Hey all,
Great work on the operator so far, I like where this is going and it is almost in a place where I can use it in my cluster.
However, there seems to be an omission of JVB service definition to allow it to reach the UDP and websocket ports on that service.
For example: https://github.com/jitsi-contrib/jitsi-helm/blob/main/templates/jvb/service.yaml
I have set my ingress up with nginx, and have added the server-snippet into the jitsi CRD ingress annotations like you used to have previously:
jitsi-kubernetes-operator/controllers/ingress.go
Lines 30 to 48 in eb013e7
My CRD in my Rancher cluster so far :
I can now reach the xmpp-websocket ok, but the colibri-ws one is currently not reaching as port 9090 is not exposed in the JVB service, plus also the UDP port is not either.
I'm interested in finding out how you have you achieved this so far?
Thanks, Ryan
The text was updated successfully, but these errors were encountered: