Handwriting Recognition API

[wacky6]

HIQaH! QaH! TAG!

I'm requesting a TAG review of Handwriting Recognition API.

Handwriting is a widely used input method, one key usage is to recognize the texts when users are drawing. This feature already exists on many operating systems (e.g. handwriting input methods). However, the web platform as of today doesn't have this capability, the developers need to integrate with third-party libraries (or cloud services), or to develop native apps.

We want to add handwriting recognition capability to the web platform, so developers can use the existing handwriting recognition features available on the operating system.

• Explainer: https://github.com/WICG/handwriting-recognition/blob/main/explainer.md
• Security and Privacy self-review: https://github.com/WICG/handwriting-recognition/blob/main/security-privacy-self-review.md
• GitHub repo: https://github.com/WICG/handwriting-recognition
• Primary contacts (and their relationship to the specification):
  • Jiewei Qian (@wacky6), Google, Editor
  • Matt Giuca (@mgiuca), Google
• Organization/project driving the design: Google
• External status/issue trackers for this feature (publicly visible, e.g. Chrome Status):
  • https://www.chromestatus.com/features/5263213807534080

Further details:

• I have reviewed the TAG's API Design Principles
• The group where the incubation/design work on this is being done (or is intended to be done in the future): WICG
• The group where standardization of this work is intended to be done: unknown
• Existing major pieces of multi-stakeholder review or discussion of this design: GitHub issues
• Major unresolved issues with or opposition to this design:
  • Complex script text segmentation: issue 6
• This work is being funded by: Google

We'd prefer the TAG provide feedback as (please delete all but the desired option):

🐛 open issues in our GitHub repo for each point of feedback

Thanks.

[cynthia]

We briefly looked at this during our F2F today, and had a couple early review questions:

1. How does this work when it comes to writing vertically or right-to-left?
2. Same question, but in the context of code-switching? (e.g. English word in between Arabic text?)
3. Does it make sense for this to be stuck directly on to navigator? Could you let us know why it is there?

@r12a do you have any input on this?

[cynthia]

What's the metric of the cartesians in the explainer? Are they physical pixels, logical pixels, or something else?

[r12a]

@cynthia WICG/handwriting-recognition#4

I also made a bunch of other i18n-related comments (see the issue list).

[wacky6]

Vertical writing.

Here I assume you mean a language that can be written both horizontally and vertically.

Google's recognizer generally returns characters in the order they were written (for the above type of languages). So it works in both writing directions (e.g. rtl, ltr, top-bottom). Our metric shows vertical written isn't commonly used by our users, so this feature hasn't got recent attentions.

We aren't sure how other recognizers work. Some may only work with one direction (and doesn't work at all for vertical writing). Some may ignore the character writing order.

WDYT to have a hint about writing direction? In case some recognizers need this information. Note, some recognizer may disregard this hint altogether.

RTL writing

For RTL languages, the recognizer already knows it should process text from right to left.

For LTR languages, but characters written from right to left (e.g. "hello" written in "olleh" order). It's a rare/uncommon scenario. I'm not sure what's the correct interpretation. The user perhaps want the text to be interpreted as "hello", but it's really up to the recognizer to decide what it will output. Either output can be considered valid IMO.

---

Mixed scripts.

The recognizer could determine the writing direction by looking at each character's written time and their spatial relations. Similarly for context switching.

For example,

• Unidirectional text "ABC". The writing direction can be learned by looking at the order of each character (A->B->C or C->B->A).
• Mixed: "AB cba CD" (upper-case / lower-case are two different scripts), "A->B->C->D->a->b", or, "A->B->C->D->b->a".

This being said, existing recognizers (those available on the market) don't support mixed scripts (e.g. english + arabic). They will recognize text as if the text is written in a single script (e.g. recognize arabic characters as english characters, and give less-ideal results).

I don't think we should try to solve the mixed script problem if the underlying implementations haven't solved it. Our solution may not work for them. Or, if the implementation is advanced, it doesn't care about whether we provide this information / hint).

---

Why navigator object

We choose navigator because it's preferred over alternatives (e.g. window, global constructor):

We expect handwriting recognizer to interact with platform-specific APIs, and support different features (on different platforms). Navigator seems natural based on this consideration of feature differences.

We don't have particular preferences on where the methods are. Are you suggesting we put the methods behind a attribute (e.g. navigator.handwritingService.doSomething())?

---

What's the metric of the cartesians in the explainer

The explainer examples use logical pixels.

The recognizer doesn't particularly care about the measurement unit, as long as all provided coordinates are measured in the same way (i.e. don't mix logical pixels and device pixels).

The recognizer implementation normalizes the coordinates, and perform recognition relatively (e.g. relative to the smallest character / block in the drawing).