-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WebSocket transport dead lock #732
Comments
We have an old pointers list in place because the object allocation for the WebSockets client is done by the library itself, and it tends to reuse old pointers for that. Since we use those pointers as opaque identifiers for the transport in place, we have to do some magic, specifically making sure we don't handle send requests from the core on a transport we know has gone (e.g., closed connection, so we added the pointer to the "old list"), and making also sure the "old list" is updated whenever we get a new connection (if a pointer previously used for a connection was in the "old list" and the same one is used for a new connection, remove it from the "old list"). This is indeed a bit wacky, and is not an issue anymore in the reference counters branch for instance as in that case the pointer we use to identify transports is a container we create ourselves. That said, not sure what can cause that to deadlock. libwebsockets is single threaded and so if it's stuck there it won't indeed go on, but why it locks there is something we'll have to investigate. Probably a lock somewhere else where we don't unlock in some if/then/else branches? I'll have a look. |
PS: if you find it easy enough to replicate, you can try enabling the locking debug in Janus and try again until it happens. This way, you should be able to see exactly where in the code the double locking happens for you. |
Thanks. I just enabled the lock_debug, and will wait for this to happen again. BTW: when will the new refcount branch will be considered stable? |
As soon as it is! 😉 |
Will try it soon. And I just noticed another issue, if I start a WebSocket connection, close it, then open another connection soon, a timeout message will be sent to the new connection with old session id. I'm not sure if this is a bug, because I can deal with it in client code. |
Yeah, that's something we experimented too, and happens when a new connection arrives as soon as the previous one ends and the two share the same "pointer". In that case, Janus might be a little to slow to notify the timeout, use the pointer it had, and so the event will get to the new connection instead of the old one. Again, something that with the reference counter branch won't happen. To solve that issue in the short term, we might simply add a map in the transport, so that we always feed the core with a different pointer no matter what it points to, as it would be mapped to the actual transport structure. |
@Vecio any update on the lock debugging? |
Not occurred any more, have been running the service for one month. |
Ok, closing the issue then. Feel free to reopen should this reappear. |
I don't have a button to reopen this issue but I am having this issue with the latest master code. It happens about once a day where janus doesn't crash, but it will no longer accept websockets. I'm using the latest libwebsockets. When I build it says using the "new api". Is there an alternative library I can use to work around this issue? I'm going to enable lock_debug. Also using the sip plugin only. BTW in github if a contributor (Iminiero) closes an issue only they can reopen it. |
@shrhoads didn't know about this, reopening then. If you can provide some |
As a side note, you can try changing the |
This issue occurred to me again, will try to get some useful log. |
Recently I have been using the latest refcount branch, there is no this issue any more. |
Same issue with the latest master code , I have this issue when I build Janus in Mac, and it works well in ubuntu. |
It does happen on ubuntu, too, although it hasn't happened again for 1.5 weeks so far. |
If you can readily reproduce on mac, can you add the logging/etc. suggested above? |
It happens on Mac every time and every request, I test both old api and new api of libwebsockets. And http transport works well. @shrhoads This is lock_debug log : https://gist.github.com/zombiecong/e1417c8001b12c8478ed5658d4181d32 I sent a janus "create" request, there was no response from server.After a moment, I saw the session "timeout" log and the session was released. BTW:I build some dependencies libs by myself , not by homebrew. |
Sorry, missed this notification, apparently... bridging #798 here as well, as it's the same issue (SIP or VideoRoom are not the issue). |
I don't think it's a locking issue in Janus, as you get to the |
I am also hitting the deadlock issue, running openssl 1.0.1e, libwebsockets 2.2.0. I tried disabling secure websockets and just used an unsecure websocket with the decryption handled via nginx but still luck. I can repro this issue at least once per day. I will try some more logging in libwebsockets and post an update |
same issue with the latest refcount branch, libwebsockets 2.2.0, OpenSSL 1.0.1t, debian jessie docker container on ubuntu 16.14. If this happens, creating http request also can't work in addition to websocket request. |
Wait, if HTTP requests fail too, then it is indeed a lock in Janus causing this. Can the others confirm? I was under the assumption that the HTTP API was still working when this happened. |
I can confirm that the issue is also in the latest refcount branch. |
I won't be able to do any test before the IETF ends. If you guys can find ways to replicate this in the meanwhile, so that I can check for myself when I get back, that would help. |
I was able to reproduce this with libwebsockets logging at the highest level (4095): https://www.dropbox.com/s/3wwqlhp0ksk4e4f/janus_deadlock.log I have an external process that hits janus over wss on the localhost every minute and this is visible in the log by looking for the message: Message for Session 278: {"id": 0, "request": "watch"} the first appears at 20:54:32, there should have been another one at 20:55:32 and another at 20:55:32. My watchdog thread kills Janus with an exit status of 99 if it doesn't get a new sessions request for 2 minutes which was the case at 20:54:32 |
Can't seem to be able to replicate the issue. Please, if you have an easy way to consistently replicate the issue, then provide the steps here so that I can try and check if it happens to me as well. Logs with the locking debug enabled (either via configuration or via Admin API) when the issue happens will also help me identify which lock is being held and not released, if any. |
Ah, the irony! I had managed to quickly replicate this with the tiny stresser tool I implemented, but as soon as I added the same lines I suggested you, of course Janus is not deadlocking anymore... keeping it up to annoy Janus hoping it will happen soon 😉 |
Finally got it to happen, and from a first look in the log it seems the cause there was the I'll do another round just to check if it happens in the same point, and then I'll evaluate. |
Sorry for the delay, but it will take some time before I can look into this. Today is Labor Day (National holiday in Italy) and tomorrow I'll fly abroad for an event for a few days. Anyway, I did some more dumps of my own too, so I have material to investigate when I can. |
Just came back to the office after attending two different events, so sorry for the lack of feedback. I'll give a look at the logs between today and tomorrow in order to check if I can get to the real source of the issue. |
Contacted the lws developers for info on what might cause I'm now trying to attach via gdb to Janus as soon as it deadlocks, but obviously I've been stressing it for more than an hour and nothing happened yet... not sure if I'm using a different library than when I could replicate the issue at the time: now I'm using the version my Fedora 25 ships in the repos, which apparently has I'll keep you posted in case I manage to deadlock it again and get some gdb info. |
If we use libwebsockets with single thread, won't it attract extra delay? Any ways we started to use libwebsockets with single thread and it is running with out deadlock since 36 hours. |
libwebsockets is conceived to be single thread, so there shouldn't be any issue. If you think about that, node.js is single threaded as well, and there's usually no problem in basing HTTP/WS services on top of that. Considering the WS support in the Janus API is only needed for signalling purposes (no really intensive traffic), I don't expect that to be an issue. Thanks for the feedback on the lack of deadlocks in that case! |
@Vecio @shrhoads @zombiecong @sslivins @atyenoria just pinging you guys as you contributed to the issue with feedback on this happening to you. As explained in a previous note, which addressed a comment from the lws developers, just building your library with |
i will try this today @lminiero and let you know the results over the next 24h...thanks again for tracking this down |
btw the default value of LWS_MAX_SMP is 32 in libwebsockets 2.2 |
Weird, the lws developer told me the default was 1. My Fedora ships a 2.1 rpm in the repos, and I couldn't replicate the issue with that, which makes me think that one is using 1 (seems to be confirmed by looking at the cmake file). |
again, this is i build from source and dont put anything explicit for the cmake...also corresponds with their docs: https://libwebsockets.org/lws-api-doc-master/html/md_README_8coding.html |
so just to update things...,i am still experiencing some deadlocks but it's many many orders of magnitude fewer...i still need to evaluate if these are the same root cause or something else, i'll get you more info on monday |
For me, it won't happen. It seems to be solved. In the past, I can confirm that this happens frequently. |
Something weird I just noticed... I wanted to add a check for |
I faced the same issue. LWS_MAX_SMP is set to 32 by default. I rebuilt the library with LWS_MAX_SMP = 1. |
Adding to this discussion: I am using websockets, i recompiled libwebsockets with LWS_MAX_SMP=1 and i still managed to get the deadlock. But here's the weird behaviour of the deadlock: I am using 2 plugins in my janus installation:
I ran the LOCK/DEADLOCK script and i got these: https://pastebin.com/zAa4iNLC How to reproduce: Create 6 chrome tabs and join a single room. Do the same on pc 2. and connect from 3rd device(in my case its mobile app). Run automation script that refresh -> join room -> unmute for 2 seconds -> refresh -> join.... After a while the audiobridge will deadlock. EDIT: After further crashin, the first lines of the LOCK/UNLOCK script change, depending on the situation where it deadlocked(next time its not in my room_helper:radio_thread, but in list_rooms, etc). So that probably means it deadlocks before and then tries to lock again on already locked mutex. |
If other plugins work, then it's not websockets related (unless the plugin
still works when accessed via another transport like HTTP, for instance,
which I find unlikely), so please let's not confuse issues. It would be
better to open another issue on the audiobridge deadlocking with the info
you found.
Il 04 giu 2017 5:13 PM, "Ivrrimum" <[email protected]> ha scritto:
… Adding to this discussion:
I am using websockets, i recompiled libwebsockets with LWS_MAX_SMP=1 and i
still managed to get the deadlock.
But here's the weird behaviour of the deadlock: I am using 2 plugins in my
janus installation:
1. audiobridge
2. videoroom.
When a deadlock happens on audioroom, i can still access and use
videoroom. I don't know if this is related tho.
Also i wanted to add that i am using janus-gateway version from Dec
20, 2016.
I ran the LOCK/DEADLOCK script and i got these:
https://pastebin.com/zAa4iNLC
How to reproduce: Create 6 chrome tabs and join a single room. Do the same
on pc 2. and connect from 3rd device(in my case its mobile app). Run
automation script that refresh -> join room -> unmute for 2 seconds ->
refresh -> join....
After a while the audiobridge will deadlock.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#732 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADg5vFCJYQShKCiYGNjgEeZv9gz-vc9cks5sAsmrgaJpZM4LbiaO>
.
|
Just reporting in on this, LWS_MAX_SMP=1 seems to have solved the issue for me. |
Closing then, thank you all for the feedback! |
To avoid deadlock in websocket See meetecho/janus-gateway#732 re: LWS_MAX_SMP
I'm using Janus video room plugin to develop a new website, and have found this issue for several times. e.g. the WebSocket won't accept any new client after some tests.
After tracing the code execution, I found the code block at
I have some old clients exited without cleanup, e.g. no detach handle, etc.
Is this expected to happen when clients don't cleanup resources before disconnect?
The text was updated successfully, but these errors were encountered: