Spatial audio support in AudioBridge via stereo mixing

[lminiero]

As the title suggests, this is a first attempt at using stereo mixing for the purpose of implementing some basic spatial audio support: the idea, basically, is to allow you to place different participants in different parts of the stereo mix. Please notice that the implementation is quite naive, so you shouldn't expect anything as sophisticated as the STEAK project, for instance; that said, it should still be a good starting point, and can always be improved in the future.

By default AudioBridge rooms continue working as before. If you create rooms with a spatial_audio set to true, though, then the mixer is instructed to work with stereo streams instead. You place participants using the spatial_position attribute, which can be set both when joining, and dynamically via configure requests: this spatial_position ranges from 0 (hard left) to 100 (hard right), and defaults to 50 (center), which means you have a good range of positions to play with. As I was saying the implementation is quite naive, and uses L/R gains to simulate this positioning. When a user is in the center, we apply a 50% gain on both sides (half volume on L and R), which changes the more left/right the user is (hard left = 100% gain on L, 0% on R). Not sure this is the right math to perform, and if we should always keep 100% on the side until we cross the center and get gurther from it: this is one of the areas where we can probably improve, and feedback would be welcome.

I've marked this PR as a draft because the work is not complete. Besides the considerations made above, at the moment when using stereo streams we're also completely ignoring the volume gain, for instance, which will of course need to be taken into account in the future. Looking forward to some feedback on whether you think this is useful, and how it can be improved.

[lminiero]

Nobody cares?

[augustblack]

Nice work! At first glance, stereo is working. I can pan left and right on each stream.

setting 'spatial_position' on the stream seems to have the correct effect (even when panning the input stereo stream)

[augustblack]

In testing on a mac, I can send stereo. However, when I open the audiobridge demo, I get stereo in Firefox and Chrome, but mono in Safari.

[lminiero]

Thanks for testing! On Safari, it looks like a browser-specific issue, especially if Chrome and Firefox do their job. If you find out what needs to change in the SDP to make it happy (if stereo WebRTC is supported by Safari at all), I'll fix it.

[augustblack]

What should I be looking for in the SDP?
I notice that for firefox/chrome, I can send and receive in stereo without having to mess with SDP offer.
Should the same behavior be expected for Safari?

[danjenkins]

I'm surprised you haven't had to mess with the sdp for chrome! You need to enable stereo in the sdp in the offer and answer for your publisher and subscriber, pretty sure there's an example of how to do that in the streaming demo

[lminiero]

I'm surprised you haven't had to mess with the sdp for chrome! You need to enable stereo in the sdp in the offer and answer for your publisher and subscriber, pretty sure there's an example of how to do that in the streaming demo

The updated AudioBridge demo tries to do that automatically, now, using customizeSdp (assuming you enabled stereo via the related query string argument).

[augustblack]

I was using the old audiobrdige.js from master.
With the customizeSdp that sets stereo, all seems to be working fine in safari.

[fippo]

see the lengthy discussion in https://bugs.chromium.org/p/webrtc/issues/detail?id=8133 wrt sdp munging.

[nerdbaggy]

This is great. I haven’t been able to confirm this, but it seems like when the incoming stream is stereo, the browser disables the AEC on the users device? It seems kinda hardware and browser dependent, not really sure.

[lminiero]

@nerdbaggy see the link @fippo posted.

[lmangani]

Is there a suggested or best practice "max rate" for firing spatial_position configure events, or events in general?
Asking for a friend who's mapping this to a mouse position event ;)

[lminiero]

You mean, a rate at which clients should send a "configure" to reposition themselves? I don't know as we never stressed that part, but definitely not more than 50 per second, since that's the amount of packets we process per second for each user, and so it would be silly to send 100 packets as half of those positions would never be enforced in the stereo stream.

[lminiero]

I feel this is now ready, even if not perfect. I added some controls to the demo as well to make testing easier, so please play with this and let us know.

[lminiero]

Merging.