Behavior of window.requestAnimationFrame and window.requestIdleCallback when presenting

[toji]

Short and sweet: When we're in WebVR presentation mode on devices without an external display, should the page continue pumping the window rAF and rIC loops?

For context: In Chrome at least when we kick into VR mode on mobile we stop the main page from running it's normal compositing process for performance reasons. This has a natural side effect of preventing rAF and rIC from running unless we take explicit (and slightly hacky) steps to avoid that. It may make sense, though, to treat this like any other time we hide the page (switching tabs, etc.) and suspend rAF simply because the page isn't actually meant to be updating and rAF is meant to be tied to page animation. (I'm not as sure about the intended behavior for rIC.)

That would create a behavioral disparity with desktop, though, which WILL have the page visible while presenting and does want to keep rAF running. That could create a situation where developers test code primarily on one platform and are surprised by the behavior of the others.

Would love opinions on this from other vendors!

[ddorwin]

As noted in this Chromium commit, there may be a race condition between the window rAF and the beginning of the WebVR mechanism. Whatever the outcome of this issue, we should ensure the behavior is clearly defined and that there is a reliable path for authors.

[toji]

Quick update: This was discussed on the WebVR implementors call today and the general consensus is that if the page isn't visible it shouldn't be pumping rAF. (Didn't discuss rIC as much, because not everyone implements it.)

In the future exceptions would probably need to be made for DOM content that is displayed in VR via a layer or similar mechanism, but at that point it's reasonable to expect that the DOM compositor will have to be running to generate that content anyway, and rAF will naturally run as a result. Some questions came up about what framerate the rAF should run at in that scenario (monitor vs. HMD?) but if the in-VR layer was able to be updated asynchronously from headtracked rendering on the quad it wouldn't be a user comfort issue.

[kearwood]

Also, as a follow-up from the WebVR implementers call, we wish to avoid any artifacts related to video framerate if parts of the page are mirrored into the headset.

For example, if 24fps video was rendered in an HTMLVideoElement it is temporally upsampled to the 2d monitor's 60hz framerate. This is commonly done by repeating some frames, causing a variability in frame duration as the 24hz video catches up to the 60hz output rate. Effectively, half of the video frames are presented for 2 vsync cycles and the other half of the video frames are presented for 3 vsync cycles:

vsync: 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
       30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

video: 00 00 01 01 01 02 02 03 03 03 04 04 05 05 05 06 06 07 07 07 08 08 09 09 09 10 10 11 11 11
       12 12 13 13 13 14 14 15 15 15 16 16 17 17 17 18 18 19 19 19 20 20 21 21 21 22 22 23 23 23

video frame uniformity: 2 3 2 3 2 3 2 3 2 3 2 3 2 3 2 3 2 3 2 3 2 3 2 3

If a piece of the page, containing such a video is then sampled into a VR layer at 60hz, it will be again upsampled. A VR display could be running at 90hz typically:

vr vsync:     00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
              30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59
              60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89

source frame: 00 01 01 02 03 03 04 05 05 06 07 07 08 09 09 10 11 11 12 13 13 14 15 15 16 17 17 18 19 19
              20 21 21 22 23 23 24 25 25 26 27 27 28 29 29 30 31 31 32 33 33 34 35 35 36 37 37 38 39 39
              40 41 41 42 43 43 44 45 45 46 47 47 48 49 49 50 51 51 52 53 53 54 55 55 56 57 57 58 59 59

video frame:  00 00 00 01 01 01 01 02 02 02 03 03 03 03 03 04 04 04 05 05 05 05 06 06 06 07 07 07 07 07
              08 08 08 09 09 09 09 10 10 10 11 11 11 11 11 12 12 12 13 13 13 13 14 14 14 15 15 15 15 15
              16 16 16 17 17 17 17 18 18 18 19 19 19 19 19 20 20 20 21 21 21 21 22 22 22 23 23 23 23 23

video uniformity: 3 4 3 5 3 4 3 5 3 4 3 5 3 4 3 5 3 4 3 5 3 4 3 5 (Janky!)

If we were upsampling the video directly from 24hz to 90hz:

vr vsync:     00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
              30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59
              60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89

video frame:  00 00 00 01 01 01 01 02 02 02 02 03 03 03 03 04 04 04 05 05 05 05 06 06 06 06 07 07 07 07
              08 08 08 09 09 09 09 10 10 10 10 11 11 11 11 12 12 12 13 13 13 13 14 14 14 14 15 15 15 15
              16 16 16 17 17 17 17 18 18 18 18 19 19 19 19 20 20 20 21 21 21 21 22 22 22 22 23 23 23 23

video uniformity: 3 4 4 4 3 4 4 4 3 4 4 4 3 4 4 4 3 4 4 4 3 4 4 4

[kearwood]

The video frame uniformity issues would also apply to CSS animations and WebGL canvases that will be projected onto "floating quad in space" layers or onto textures within the VR presentation.

I believe we should always prioritize the VR experience when a VR presentation is active.

The key question should be if the effective framerate of the 2d page should bump up to the HMD's framerate in the event that parts of it are captured for VR presentation, or if it should remain at 60hz.

This might not be a simple task for browser implementers, and could slow adoption for the WebVR 2.0 API if required.

If the video frame uniformity is the issue affecting users the most, perhaps we should first implement a specialized video VR layer or implement optimized functions for decoding video directly to WebGL textures.