-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Janus Crash (SIGSEV) on latest master branch #688
Comments
I am building janus like this: JANUS_REV=master JANUS_REPO=meetecho/janus-gateway SU_CMD=sudo OPENSSL_REV="1.0.2j" JANUS_FEATURE_OPTS=" APT="$SU_CMD apt-get --quiet --assume-yes" if $CONTAINER; then $APT install --no-install-recommends mkdir -p $SRCDIR OPENSSL_TARBALL="openssl-${OPENSSL_REV}.tar.gz" ./config --prefix=$PREFIX --openssldir=$PREFIX/openssl -DPURIFY shared LIBSRTP_TARBALL="libsrtp-${LIBSRTP_REV}.tar.gz" LIBNICE_TARBALL="libnice-${LIBNICE_REV}.tar.gz" $SU_CMD make -C nice install-exec rm -Rf janus-gateway |
Not sure about the implications, but could we move at line 2393 in janus_videoroom.c to be inside the protected janus_mutex_lock(&sessions_mutex); block above? |
That wouldn't solve anything, as if it's race conditions that are causing this, they may happen slightly later and affect other usages. You may want to also give #403 a try as it tries to address issues like those, although it might not be suitable for production environments yet. |
OK, thanks for the reply. The reason I ask, is that for my application and use case,2 separate servers, at separate times, crashed on the same line (2393) if (session->destroyed) So I might want to move the code to be inside the mutex, no matter what, as if the session is "destroyed" the loop continues and the session ofject is NOT used... Thoughts? |
If memory gets corrupted, it's not relevant if it happens before or after you get the object from the table: it means the pointer points to garbage for some reason, which should not happen. The fact they crash on the same line simply means the same object may be getting corrupted, possibly because some property of the object is invalidated in the same part of the code. Compiling with libasan support may give you better info on what's happening, e.g., if a double free is happening or if other things are causing weird behaviours. |
Got it crash with AddressSAnitizer. |
Im experiencing the same issue. Regular crashs in the videoroom plugin at the mentioned code position. Here is my gdb output: http://pastebin.com/rrnz6aCN The likelihood of a crash seems to be related to the load. Janus runs fine for hours at lower loads while it crashes about once per hour at higher loads. Do i understand the issue correctly, that the session is freed by the watchdog code while a message for that session is being processed in janus_videoroom_handler? |
Have you tried to upgrade libsrtp to 2.0.0+ or maybe use websockets? |
Im already using websockets but also tried switching to long poll. Makes no difference... Do you think libsrtp 2.0.0 + would make a difference? Im currently using 1.5.4 compiled from source. |
I would give ot a try yes |
Just a follow up... i experimented a bit with the settings of the HTTP transport. Setting the thread number to 8 instead of the default "unlimited" seems to improve the stability of Janus in regard of the mentioned SIGSEV. Just a workaround i guess but maybe it helps to spot the real cause. |
@sailerinteractive i dont see any options in the http transport to set threads, where did you find this setting? |
@sailerinteractive a thread number of 8 for HTTP will limit the number of concurrent user sessions to about 6-7, as most of the times you'll have one long poll per user keeping one connection (and so thread) busy for a long time, preventing others from injecting other requests. A limit is reasonable, but you have to adapt it to how many users you want to be able to serve at the same time. |
@lminiero thanks for the input and your work on Janus! Im using websockets for all user sessions. The HTTP transport is only in use for some server side admin requests (such as creating video rooms). Just wanted to share my experiences that this seems to limit the number of crashes at this code position. |
Have you guys checked the reference counter branch too to see if if's fixed there? |
Closing as too much has changed in the code. If still an issue, feel free to open a new one with up-to-date details. |
I am using latest Janus ( commit df5b546) on Debian Jessie 64 bit.
The crash happens while using the videoroom plugin, with about 200 watchers, and a few presenters.
Core was generated by `/opt/janus/bin/janus -o'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 janus_videoroom_handler (data=0x0) at plugins/janus_videoroom.c:2393
2393 if(session->destroyed) {
(gdb) bt
#0 janus_videoroom_handler (data=0x0) at plugins/janus_videoroom.c:2393
#1 0x00007f88d7cc5845 in g_thread_proxy (data=0x2051140) at /build/glib2.0-y6934K/glib2.0-2.42.1/./glib/gthread.c:764
#2 0x00007f88d6a4f0a4 in start_thread (arg=0x7f88d8899700) at pthread_create.c:309
#3 0x00007f88d678462d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) frame
#0 janus_videoroom_handler (data=0x0) at plugins/janus_videoroom.c:2393
2393 if(session->destroyed) {
(gdb) info locals
request = 0x0
msg_sdp_type = 0x0
msg_sdp = 0x7f86b32666e0 "P?%?\206\177"
session = 0xa1679af2d1c2a787
FUNCTION = "janus_videoroom_handler"
msg = 0x7f86cd0c1870
error_code = -674938240
error_cause = "No such feed (6059871534061037)", '\000' <repeats 480 times>
Before the crash here part of the log
Here is the log right before the crash:
http://pastebin.com/QRX0y2c1
before that, I see hundreds of
[ERR] [plugins/janus_videoroom.c:janus_videoroom_handler:2618] No such feed (6059871534061037)
[ERR] [plugins/janus_videoroom.c:janus_videoroom_handler:2618] No such feed (6059871534061037)
[ERR] [plugins/janus_videoroom.c
....
The text was updated successfully, but these errors were encountered: