APG guidance for touch devices

[LJWatson]

We need to address the problem of design pattern accessibility on touch screen devices. Touch ATs intercept all touch gestures. Only click/blur/focus are handled reliably - and they're not sufficient to support interaction with complex widgets like trees.

We should look to include guidance that surfaces these shortcomings, and indicate which of the design patterns will/won't work on touch devices with ATs enabled.

[patrickhlauke]

For context (as the above is paraphrasing my recent ranting on twitter), this relates to further thinking around this bug/comment I filed https://lists.w3.org/Archives/Public/public-pfwg-comments/2015AprJun/0005.html / https://www.w3.org/Bugs/Public/show_bug.cgi?id=28694

[stevefaulkner]

Have set up a google spreadsheet to record compatibility issues with design patterns
https://docs.google.com/spreadsheets/d/1gN9oRZPdrJxLDNtT6nVO4fn7E7sn1061L9Xl3__slZ4/edit?usp=sharing

[patrickhlauke]

A simple, concrete example: take an implementation of a "faked" slider like http://hanshillen.github.io/jqtest/#goto_slider - to my knowledge, this implementation follows the advice provided in http://www.w3.org/WAI/PF/aria-practices/#slider which defines the expected keyboard behavior (use of left/right arrow, home/end, page up/down, tab to get in/out of the slider widget itself). Now, as with many other patterns, this: a) assumes that it's working fine for mouse users (so nothing special is defined/mentioned about how a mouse user would need to interacts with it) and b) defines behavior for keyboard users, which the author is expected to include by hooking into keydown/keypress events, check for those defined expected keystrokes, and trigger relevant JS in response.

This simple slider is unusuable for users with touchscreen interfaces and AT. Simply taking the example of iOS/VoiceOver, a user is able to set focus to the slider "knob". VoiceOver announces (because of the role="slider") that it's "adjustable" and that a user can "swipe up or down with one finger to adjust the value". However, doing this has no effect - as the actual functionality that fakes this is attached via keydown events listening for arrow/home/end/etc, and swiping up/down does NOT trigger these (VoiceOver does not fire fake keyboard events that would rattle those event handlers, and I believe it shouldn't either - though it may have to come to this if no better solution is available). The naive solution here would be to also layer touch/gesture events on top of the keyboard listener events, but this will not work because VO will intercept touch events/gestures (similar to role="document" on desktop/keyboard/AT scenarios, and there's no sensible way to force touch/AT like VO to go into role="application" mode). The only way, if there WERE touch/gesture listeners, for a user would be to perform an override/pass-through gesture in VO (double-tap and hold, then swipe) to pass actual swipe gestures to the underlying page/JS, bypassing VO...but this, in my mind, is the philosophical equivalent of saying "keyboard users can just use mouse keys to interact with this" - it's a crutch, not a solution (and not all platforms have this sort of pass-through gesture...for instance, I don't believe Android/Talkback offers anything like this).

In short, even with such a simple widget as a slider, following the current advice/design pattern (which tacitly assumes a false dychotomy of mouse/keyboard only) results in a construct that simply will not work for touch/AT users.

[patrickhlauke]

Another example of a construct that follows ARIA design patterns and fails for touch/AT: http://hanshillen.github.io/jqtest/#goto_menubar which follows http://www.w3.org/WAI/PF/aria-practices/#menu - specifically, if a menu item has a submenu, the pattern currently expects

If the menu item with focus has a submenu, pressing Enter, Space, or the right arrow key opens the submenu and puts focus on the first submenu item.

With iOS/VO, I can open the first level of each menu, set focus to any item, but if the item also has a submenu, i cannot trigger it (in Hans' implementation), as this will NOT send Enter/Space/right arrow, but simply a click event

[patrickhlauke]

A third one, for good measure: http://hanshillen.github.io/jqtest/#goto_tree which implements the TreeView http://www.w3.org/WAI/PF/aria-practices/#TreeView

With iOS/VO, I can set focus to individual visible nodes (and also hidden nodes, but this seems a shortcoming in Hans' implementation, rather than intention), but I cannot open/expand a tree branch, as I cannot fire

Right arrow key expands a closed node, moves to the first child of an open node, or does nothing on an end node.

I can only fire a click event at the actual focused node, which only triggers what I assume is the mouse-based equivalent coded for

Enter key performs the default action on end nodes.

[patrickhlauke]

Additionally, when I originally said that for touch/AT situation we can only currently rely on focus/blur/click ... the situation is actually even murkier, as some OS/Browser combos don't send focus/blur when you'd expect it. see http://patrickhlauke.github.io/touch/tests/results/#mobile-tablet-touchscreen-assistive-technology-events

[patrickhlauke]

Quick additional note, from testing: iOS devices with a paired bluetooth keyboard do not fire key events (whether VO is enabled or not) unless focus is in a text input/textarea (so patterns reliant on key events will fail even for "keyboard" users on iOS); Android with a paired keyboard does act mostly like a traditional "desktop" device with keyboard - no keys are intercepted, fired as normal (in standard Browser or Chrome; some browsers, like Firefox/Android, do provide additional shortcut keys when running Talkback, and these are intercepted)

[mraccess77]

In my experience custom ARIA combo boxes are also problematic with mobile devices such as iOS and Android. Listboxes faired better in my testing. My presentation at CSUN indicated the same challenges with sliders, menus, and trees as discussed above. Spinboxes also were not supported.

[DavidMacDonald]

I'm totally on board with this discussion and we'll be looking at it carefully with the Mobile Task Force

[hshepherd]

It seems like interactions with role="slider" elements are still an issue in iOS Voiceover even when using the recommended aria attributes (aria-valuemin, aria-valuemax, aria-valuenow). Are there any plans to revisit this?

[patrickhlauke]

interactions with ARIA menus that have submenus are still an issue in iOS/VO too. generally, over the last few months, i've actually started advising clients NOT to implement certain ARIA patterns as they fall apart on mobile.

[ZoeBijl]

This issue was moved to w3c/aria-practices#8