Consider hooking up sound source nodes in the API somehow

[cwilso]

There have been requests to add sound to the scope of the WebXR API . There are two aspects to this - first, that we should manage the audio input/outputs associated with an XR device. This is already covered by #98. The second aspect is enabling developers to easily position sound sources in the virtual space, and use an HRTF (Head-Related Transfer Function) or multi-speaker setup to properly "position" the sound.

It is relatively straightforward to use Web Audio's PannerNode to hook up between a posed sound source and the head pose - in fact, three.js does exactly this, with a PositionalAudio source object. However, the problem lies in keeping the headpose (and the sound source pose) updated on a high enough frequency - ideally, letting the audio thread directly get headpose info somehow or the like.

(Note that I don't consider this a high-priority today - Issue #98 is more important, and even that is a future enhancement - but I wanted to capture it.)

[blairmacintyre]

Why wouldn't we use existing web api's (getUserMedia, webAudio)? How are they lacking, in a way that couldn't be solved by updating them?

As with accessing video, it seems like enhancing existing web APIs, or perhaps somehow creating a binding between them, would be preferable to creating a new, different API. For audio, in particular, webAudio seems pretty good, and if the issue is synchronizing the headpose of the audio for spatialization, this seems like something that could be solved with a small additional feature in webAudio.

[Manishearth]

The problem is that you have to manually send the pose data to webAudio, which carries some overhead.

Ideally, WebAudio would have a mode where it could be told "the position of this panner node is to reflect the head pose" and then it uses realtime head pose data.

[cwilso]

@Manishearth hit the nail on the head. We would (presumably) use media streams and Web Audio. Web Audio even already has 3D positioning - and yes, the key missing piece is synchronizing the poses (it's not just headpose - it's also the pose of each individually-placed sound-producing object) - or more to the point, minimizing the latency of keeping those poses updated, and getting them updated in the audio thread on a regular basis.

It's possible this is just advice and best practice for web audio; it's possible we'd want a small feature tying web audio (PannerNode or a derivative) to an XR Session and some poses. It may not turn in to an actual feature in XR - but that all needs to be explored, and this seems like the best place to track it to me.

[blairmacintyre]

@cwilso @Manishearth yes, exactly the approach I was imagining. Getting someone to explore this would be great.