regexp/syntax: recognize Unicode category aliases

[rsc]

The Unicode specification defines aliases for some of the general category names. For example the category "L" has alias "Letter".

The regexp package supports \p{L} but not \p{Letter}, because there is nothing in the Unicode tables that lets regexp know about Letter.

Package regexp would be a permitted implementation for use in JSON-API Schema implementations except that there are tests with usage of aliases like \p{Letter} instead of \p{L}.

In #70780 I proposed adding a new CategoryAliases table to package unicode.

If that is accepted, I propose to also recognize the category aliases in regexp/syntax, which will make them work in package regexp.

I also propose to follow https://unicode.org/reports/tr18/#General_Category_Property and add \p{Any}, \p{Assigned}, and \p{ASCII}.

Finally, I propose to make the Unicode names case-insensitive, so that \p{ascii} can be used instead of \p{ASCII}.

[gabyhelp]

Related Issues

• proposal: unicode: add CategoryAliases #70780

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[rsc]

#70780 has been updated to include adding LC and Cn.
These would automatically start working in regexp as well,
as \p{LC} and \p{Cn}.

[gopherbot]

Change https://go.dev/cl/641377 mentions this issue: regexp/syntax: recognize category aliases like \p{Letter}

[rsc]

@BurntSushi pointed out that https://unicode.org/reports/tr18/#General_Category_Property also mentions \p{Any}, \p{Assigned}, and \p{ASCII}. Perhaps these should be included in the change as well.

[BurntSushi]

\p{any} is nice as a substitute for (?s:.) and \p{ascii} is especially nice in its negation. e.g., \P{ascii} when you're hunting around for things that aren't ASCII.

Obviously neither add any new capabilities, but I do use them occasionally myself.

[rsc]

The other potential change, as Andrew just used in his comment, is treating the names as case-insensitive and also ignoring spaces, hyphens, and underscores. If we're going to be more like TR18, it may be worth hitting that at the same time too.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[willfaught]

@rsc What is TR18?

[BurntSushi]

@willfaught A "Technical Standard" for how regular expressions should behave with respect to Unicode.

[rsc]

Have all remaining concerns about this proposal been addressed?

The proposal is to bring the regexp package into further alignment with Unicode TR18 by recognizing the following forms inside \p{...} and \P{...}

1. The new unicode.CategoryAliases, Cn, and LC (unicode: add CategoryAliases, LC, Cn #70780)
2. The pseudo-categories Any (all code points), Assigned (= not Cn), and ASCII (00-7F)
3. As TR18 says, "be lenient as to spaces, casing, hyphens and underbars", meaning accept \p{lu} for \p{Lu}, \p{ascii} for \p{ASCII}, \p{spacing mark} and even \p{spa_cing-ma rk} for \p{Spacing_Mark} (== \p{Mc})

[aclements]

Based on the discussion above, this proposal seems like a likely accept.
— aclements for the proposal review group

The proposal is to bring the regexp package into further alignment with Unicode TR18 by recognizing the following forms inside \p{...} and \P{...}

1. The new unicode.CategoryAliases, Cn, and LC (unicode: add CategoryAliases, LC, Cn #70780)
2. The pseudo-categories Any (all code points), Assigned (= not Cn), and ASCII (00-7F)
3. As TR18 says, "be lenient as to spaces, casing, hyphens and underbars", meaning accept \p{lu} for \p{Lu}, \p{ascii} for \p{ASCII}, \p{spacing mark} and even \p{spa_cing-ma rk} for \p{Spacing_Mark} (== \p{Mc})

[aclements]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— aclements for the proposal review group

The proposal is to bring the regexp package into further alignment with Unicode TR18 by recognizing the following forms inside \p{...} and \P{...}

1. The new unicode.CategoryAliases, Cn, and LC (unicode: add CategoryAliases, LC, Cn #70780)
2. The pseudo-categories Any (all code points), Assigned (= not Cn), and ASCII (00-7F)
3. As TR18 says, "be lenient as to spaces, casing, hyphens and underbars", meaning accept \p{lu} for \p{Lu}, \p{ascii} for \p{ASCII}, \p{spacing mark} and even \p{spa_cing-ma rk} for \p{Spacing_Mark} (== \p{Mc})