SIGSEGV in librabbitmq

[SadSeep]

Hi,
i am trying out the videoroom plugin with rabbitmq as transport between my server and janus. as the number of participants in the room (who both publish their own stream and subscribe to others) increase, janus crashes. running with gdb, output log displayed is

**[Switching to Thread 0x7fffc7fff700 (LWP 1230)]

(gdb) bt

#0  0x00007ffff56b93b1 in __strlen_sse2 () at /lib64/libc.so.6

#1  0x00007fffccbe6009 in amqp_cstring_bytes () at /lib64/librabbitmq.so.4

#2  0x00007fffccdfcc5d in janus_rmq_out_thread (data=<optimized out>) at transports/janus_rabbitmq.c:770

#3  0x00007ffff6eef4f0 in g_thread_proxy () at /lib64/libglib-2.0.so.0

#4  0x00007ffff5a01dd5 in start_thread () at /lib64/libpthread.so.0

#5  0x00007ffff572aead in clone () at /lib64/libc.so.6**

I have no idea what is going wrong and it seems to occur every now and then. I am unable to scale my videoroom because of this problem.

[lminiero]

Please provide a libasan dump: https://janus.conf.meetecho.com/docs/debug
The trace you pasted is insufficient to investigate the issue.

[SadSeep]

Hi,
Here is the link to the logfile after configuring with libasan-> https://pastebin.com/KsjaSjNG .

[atoppi]

Hi @SadSeep,
if you still have the core file, can you head to #2 frame and print out the value of payload_text ?

[SadSeep]

I do not have the log file, but i should be able to reproduce this again. Please provide me more info on this "payload_text". What exactly am i looking for? what is frame #2?

[SadSeep]

Hi,
I have attached both the log file(from the beginning of room creation to janus crashing) and a screenshot of the libasan output.

In the log file,
The log file is pretty big. Kindly, scroll towards the end of the file. It would seem that the last request being sent is a "hangup" followed by a "detach".

The log file->janus_apr_15_1_debug5.log

In the libasan screenshot,
The libasan output starts with a "heap-use-after-free" error.
The libasan output->

I ran the same case few times. The libasan output is pretty much the same with only the addresses being different every time.

Do get back to me if this is what you had asked for. If not, please provide me more info on how to get the required logs.

[lminiero]

Please don't post screenshots... there's nothing I can do with them. I can't parse them, can't copy, nothing. Paste the content as a gist/pastebin link instead. Thanks.

[lminiero]

That said, it might be a jansson object (JSON) that is being accessed after it's been destroyed, but since there are no line numbers there, no clue on why/where/when.

[SadSeep]

Sorry about the screenshot. Here is a pastebin link to a libasan output of one such case.
https://pastebin.com/kKzKmFHV

[SadSeep]

Hi,
I tried the same using address sanitizer. This link contains the crash log -> https://pastebin.com/viBSW4TE

It points to L:755 in janus_rabbitmq.c. The method being called is "json_dumps". Do advise on how to proceed further.

[atoppi]

@SadSeep are you running a customized plugin or just the plain videoroom plugin ?

[SadSeep]

@atoppi no, I am not using a customized plugin. I am using the plain videoroom plugin.

[atoppi]

@SadSeep try with latest master please, we are looking into this

[SadSeep]

ok, i'll try and let you know. Thank you.

[SadSeep]

Hi,
I tried reproducing the issue using the latest commit. So far, janus hasn't crashed. i'll conduct the same test cases over the next few days and let you know of the results. Thank you for the commit.

[SadSeep]

Hi,
This commit seems to have fixed this problem. My Janus Instance hasn't crashed the past few days. I think we can close this issue. I'll let you know if i come across the same again. Cheers.

[lminiero]

Excellent, closing then! 👍