[1.x] Janus pp rec doesn't fill gaps in audio file

[techwipes]

What version of Janus is this happening on?
v 1.0.1

Have you tested a more recent version of Janus too?
Yes, it worked on version 0.12.1

Was this working before?
Janus pp rec fills gaps (muted audio) in audio file

Is there a gdb or libasan trace of the issue?
No

Additional context
Hi Lorenzo and the Meetecho team! I seem to have found a bug in the janus pp-rec converter.

How to produce:

1. Create a room, join as a publisher (video room plugin)
2. Mute sound 5 seconds after start for 5 seconds
3. Turn on the sound
4. Leave the room after 5 seconds.
5. Use janus-pp-rec to convert *-audio-0.mjr to *.opus

Current behavior:
After conversion, you will get a 10 second audio file without the 5 second silence in the middle of the file.

Expected behavior:
After conversion, you will receive an audio file (duration 15 sec, 5 sec silence in the middle)

[lminiero]

janus-pp-rec hasn't changed between 0.x and master, so I don't see how that's possible. Have you tried using the 0.x janus-pp-rec to convert the master recording, and viceversa? If there really is a problem, it may be in how the new VideoRoom creates the recording, rather than in how the tool processes it.

[atoppi]

@techwipes what are you doing to mute the sound ?

[techwipes]

janus-pp-rec hasn't changed between 0.x and master, so I don't see how that's possible. Have you tried using the 0.x janus-pp-rec to convert the master recording, and viceversa? If there really is a problem, it may be in how the new VideoRoom creates the recording, rather than in how the tool processes it.

I didn’t quite understand the question, I didn’t update the converter, I work only with the Janus branch itself.

I wrote my own service for working with files processed by janus-pp-rec. Before switching to 1.0.x, everything worked fine. After the update, the file names changed a little and the timestamps in the file names began to differ, after refactoring I noticed that there was an out of sync between sound and audio, and began to look for the problem.
I saw that the sound was no longer filled with gaps, as it was before.
I'm not sure if I can rollback to version 0.x and see how it all works, my frontend and backend are already adapted for multi-stream

[techwipes]

@techwipes what are you doing to mute the sound ?

Hi Alessandro! I just asked my frontend colleagues, they answer me:
"We are sending this offer:

[lminiero]

You're renegotiating the PeerConnection, which when audio is added back will cause the SSRC to change. As a consequence, the recorder will fix the RTP headers to make the change in sequence number and timestamp seamless, thus making it look like there was no pause. If you just want to mute audio, there are easier and less invasive ways of doing that, like locally muting the audio track (which will result in sending silence).

[techwipes]

You're renegotiating the PeerConnection, which when audio is added back will cause the SSRC to change. As a consequence, the recorder will fix the RTP headers to make the change in sequence number and timestamp seamless, thus making it look like there was no pause. If you just want to mute audio, there are easier and less invasive ways of doing that, like locally muting the audio track (which will result in sending silence).

Thanks for the advice!
In this case, will I be able to get information about the current state of the stream and devices in order to understand which device icon, for example, I should draw to users?
Will there be a message in the websocket or rabbit that the participant has turned off the audio or video?

[lminiero]

When muting locally there's no message, it's up to you to circulate an event out of band.

[techwipes]

Much appreciate, closing the issue

[techwipes]

@lminiero
Hi!
I decided not to produce extra tickets and clarify a couple of questions here.

The local mute method worked, the problem went away.
But this doesn't work very well for the video track, if you turn off the video locally, the camera light stays on, which is confusing.
Is there another way to turn off the video stream so that there is no problem with filling a pause during conversion?

[techwipes]

Please don't take this as an insult, I just want to understand and help

We wrote about this situation in the google group here

I wrote earlier that gaps are not filled during post-processing if we have audio stream renegotiations

But from your last comment it looks like this should work.

[lminiero]

But this doesn't work very well for the video track, if you turn off the video locally, the camera light stays on, which is confusing.

The light will stay on for audio as well, on the tab. You can try using replaceTrack(null) to stop sending, and close the previous track to release the resources; a new getUserMedia to get the audio/video track again can be used for another replaceTrack to restore the stream without a renegotiation.

On the picture you shared, that's probably because or the SSRC change I mentioned, and if so is a consequence of the change that was contributed in #2724 that I probably forgot about when I answered that post. Since a renegotiation will change the SSRC after it's removed and re-added, when the existing recorder calls janus_rtp_header_update the RTP header is updated to make it look like the same stream, and that's why it looks like a simple and immediate pause/resume.

Not sure if/how this should be addressed, since I don't like the idea of always creating a new recording file after a renegotiation, since it depends on what triggered a recording in the first place. If we're recording everything it's one thing, if it's per participant maybe another. I'll give this some thought to see if I can come up with a workable solution.

[lminiero]

Since a renegotiation will change the SSRC after it's removed and re-added, when the existing recorder calls janus_rtp_header_update the RTP header is updated to make it look like the same stream, and that's why it looks like a simple and immediate pause/resume

Actually it may not even be as easy as that, since the Janus core itself calls janus_rtp_header_update when processing incoming RTP packets. This means the plugin may already see the "processed" headers, and so the recorder never sees an SSRC at all, but the effect is the same. This means just "fixing" the recorder code is not viable.

[lminiero]

I just checked with the EchoTest and I see timestamps have the proper gaps when renegotiating a stream out and then back in: I removed video in a renegotiation, and after ~18s added it back in, and from the timestamps in the MJR file I see a 1620000 difference, which corresponds to a 1620000/90000=18.00 difference, so we're writing the right thing in the file. Replaying the processed video with mplayer I see the video being frozen at about 23s, and then recovering at ~42s (when I added video back in). As such, I'm not sure there's anything we need to do. I do see VLC struggling with the same video, so it may be a matter of which tool you're using to handle the media that janus-pp-rec gives you: doing a re-encoding with ffmpeg of the video makes VLC happy, so I guess the same might work for you too.

[techwipes]

I just checked with the EchoTest and I see timestamps have the proper gaps when renegotiating a stream out and then back in: I removed video in a renegotiation, and after ~18s added it back in, and from the timestamps in the MJR file I see a 1620000 difference, which corresponds to a 1620000/90000=18.00 difference, so we're writing the right thing in the file. Replaying the processed video with mplayer I see the video being frozen at about 23s, and then recovering at ~42s (when I added video back in). As such, I'm not sure there's anything we need to do. I do see VLC struggling with the same video, so it may be a matter of which tool you're using to handle the media that janus-pp-rec gives you: doing a re-encoding with ffmpeg of the video makes VLC happy, so I guess the same might work for you too.

Hi Lorenzo! Thank you for your reply.
To be honest, I also like the idea of ​​creating only one file per stream. Your advice for correctly filling audio gaps also works for me (local muting).

I have only one problem left, this is the filling the gaps in the video file when the camera is turned off.

I would like the camera indicator to turn off when the video stream is muted, and this can be done if you send a renegotiation for the stream. But in the file after conversion using the Janus utility, instead of a black frame, the last frame is inserted for the entire duration of the pause. And in the video, the situation looks like the participant is frozen, and did not turn off the camera.

What should happen when you disable and enable video in the current version? Should the pause be filled with a black frame as before, or should the last frame be inserted?

I recorded a video to make it easier to understand me

d230496d-e4c6-11ec-8fd9-0242ac140005.mp4

)

[lminiero]

The frozen frame is what you should expect, because unlike audio we can't generate a black frame (video works in a different way). If you keep track of renegotiations in a metadata file, you can generate a new video out of the recording by adding the black video yourself.

[techwipes]

The frozen frame is what you should expect, because unlike audio we can't generate a black frame (video works in a different way). If you keep track of renegotiations in a metadata file, you can generate a new video out of the recording by adding the black video yourself.

Thanks, it's all clear now

[lminiero]

Notice that you can also try to generate a black screen from the client side, using a canvas element and replaceTrack: this way, when you want to mute video you can stop the webcam and put the canvas video in without renegotiating. Not sure how robust or reliable that would be, but it might work and keep things simpler. We have a couple of demos on how to use a canvas to feed a PeerConnection.