-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory corruption with Janus running in Docker #2987
Comments
Please provide a libasan stacktrace from a recent commit. |
@atoppi Sorry, I'm not sure what do you mean under recent commit. |
The stack trace you shared is from an unknown 0.11.8 janus version. |
Hm, maybe I am not reading Janus versions correctly but here is link to commit: Isn't it v0.11.8 from master branch version? |
No, 0.11.8 is almost 4 months old: https://github.com/meetecho/janus-gateway/blob/master/CHANGELOG.md#v0118---2022-02-11 The |
I run it using master (1.0.3) and here is link to logs: https://pastebin.com/vcVcMRvZ I am running publisher on the same physical computer as docker container running Janus. Publisher is trying to stream video, audio and data to Janus videoroom plugin. Issue does not repro when publisher is on different computer or when Janus is running in datacenter. With master build it took 2 runs to repro it:
There is one thread that suggests to define PORT environment variable to avoid empty answer, although it was when communication with Janus was through HTTP, not websockets. I'll try it to see if it works. However, it looks like memory issue is still there. I'll also try with 0.12.3 build later and see what is behavior. |
I tried with latest 0.x (0.12.3) and crash resembles the same one I got with 0.11.8. Here is link to logs: https://pastebin.com/9A6bA8Q4 |
Both crashes happened deep into glib, trying to allocate new memory in order to add an element to a linked list. |
Could you please try setting the G_SLICE env var before launching janus?
|
Yes, I am running docker on Windows, with docker engine running on WSL2. It was working fine up to ~4 weeks ago. It does sound like realistic possibility that memory allocation fails and after that system fails. |
I tried to run docker on computer with more memory and issue still reproes. Although I can't eliminate yet issue with Docker, running with G_SLICE setting as you suggested revealed where issue might be: https://pastebin.com/PMb2grNw
|
Some stack traces are too short so we can't track the source of the first memory free.
And start the process adding
|
If you are using the master branch of libnice, I seem to have experienced a similar crash. The crash occurred when using commit After investigating, I thought that a bug may have been introduced in commit I believe it has been fixed in the latest libnice master branch. The crash no longer occurs in my environment. Here is the output from Valgrind when Janus crashed in my environment
|
@futr thanks for the heads up on this and for providing a solution already, I wasn't aware of this issue! |
@bjelovarduck can you check with either the latest master of libnice (if the fix was indeed pushed) or with a previous version (e.g., 0.1.18) to see if you still have the issue? |
I checked and looks like that was an issue. With libnice version 0.1.18 there is no crash. With libnice version 0.1.19 there is crash. Date of change approximately corresponds to date when I started to see the issue. |
Ack, thanks for confirming. I'll close then 👍 |
Recently I started experiencing memory corruption running Janus in Docker. I tried all versions from 0.11.2 - 1.0.0.1.
I followed your instructions how to debug Janus, run it using Address Sanitizer, added output to pastebin: https://pastebin.com/GtfhutH9
The text was updated successfully, but these errors were encountered: