Skip to content

Support locales Chromium disables #64

Open

Description

@anarchodin

I notice that workerd appears to be using Chromium's locale data. This is unfortunate, since Chromium deliberately hobbles locale support to be in line with full translations of its own UI. This removes support for a substantial number of languages. Node.js and Edge appear to fix this, presumably by using alternative ICU data files.

This prevents me from using either Cloudflare Workers or workerd for any task that requires sorting strings in my native language, and since the client is just as likely to be Chrome I can't rely on the browser knowing how to do it either. While that can be worked around, the workarounds almost always involve running a server somewhere that does know how to do collation.

Would it be possible to at least investigate including more complete locale data?

Activity

  1. kentonv commented on Sep 30, 2022

    @kentonv
    Member

    Huh, I had no idea about this.

    Why did Chrome make this decision? Is the data file much larger otherwise? If so, how large?

    Is this entirely about the data file, or has Chrome also removed code from their version of libicu? We currently get the code from their repo.

  2. anarchodin commented on Oct 1, 2022

    @anarchodin
    Author

    I have to confess that I'm not familiar enough with Chromium or ICU to know for sure, but from what I've been able to find it looks like it's the data files only. I'd wager the reasoning is based on size; if I understand the setup correctly they're including data for fifty languages and excluding for a hundred. That's based on a file that appears to configure how Chromium builds locale data, at third_party/icu/filters/common.json in the Chromium tree. (Line 821 mentions keeping "only the minimum locale data", followed by a list that includes Icelandic, which I know to be excluded from Chrome. I was directed to that file by the comment I linked above.)

    As for how big the full data is, I don't know, but it appears to be included in the official Node.js binaries.

  3. anarchodin commented on Oct 4, 2022

    @anarchodin
    Author

    So, an update: I replaced workerd's data file with the complete icudt71l.dat from the ICU4C tarball. The resulting binary hasn't crashed yet, and it is quite happy to provide me, among other things, with localised date strings for the locales missing from Chromium. That would suggest that this really is just down to the data files.

    The bad news is that said binary is now 82 megabytes, as opposed to 63 for the other build. Incidentally, this is similar to the difference between a Node.js small-icu (English-only) build and full-icu (which is what the official binaries are). That's probably not a coincidence.

  4. kentonv commented on Oct 4, 2022

    @kentonv
    Member

    Hmm so the data file goes up from ~10MB to ~30MB? That is annoying but maybe not a huge deal for a server binary. We need to verify a few things though:

    • Does ICU scan the data at startup? Or only use it on-demand? The way we embed it, the data will be mmap()ed with the rest of the binary, so pages would actually be loaded from disk on first use. As long as ICU doesn't actually read pages until needed, then, having a larger embed should be "free". If it does scan the data at startup, though, that could be a problem as we care a lot about startup time (moreso than Node does, I think).

    • I imagine the intermediate file bazel-out/k8-fastbuild/bin/icudata-embed.capnp.c++ goes from 57MB to, like, 160MB? That file basically defines one huge byte array constant containing the entire icudata. I wonder if it takes excessive RAM to compile that file? But we can find a better way to do this embedding if it's a problem.

    • Where do you recommend sourcing this file for our build? You mention the "ICU4C tarball", where is that found exactly? Does the data file have to match the particular version of ICU, or is it compatible across versions? At present we get the data file from the git repo that also contains the ICU source code that we use, so there's no chance of version mismatch.

    @jasnell do you have any experience with this in Node? Any thoughts?

  5. anarchodin commented on Oct 4, 2022

    @anarchodin
    Author
    1. From looking at the documentation for ICU and various things, it looks like the data files here originated as serialized in-memory structures. That would also explain their size and compressibility. If that's the case, they're probably not scanning it over. I'd measure the startup time, but I'm not quite sure how. :)

    2. 160M is what ls calls it, yes. I couldn't find a file by that name for the opt build the README suggested; is that build somehow done without generating it? I didn't run into problems with memory, but then again I am using a machine with 32GB RAM.

    3. ICU releases seem to be on GitHub now. The file isn't in the source tree, but is in the -src.{tgz,zip} archives, under /icu/source/data/in/icudt71l.dat. The Chromium build removes the version number from the filename. These files are, I believe, stable for a given major version of ICU but not across them. Updates of the file in the Chromium git tree seem to be of three sorts:

      1. Along with an update of ICU.
      2. Tweaking the locale filters.
      3. Updates to zoneinfo.

      That last one looks like an actual concern, but there are instructions to do those updates.

    Having looked at this whole story, I'm starting to understand why the Unicode Consortium is pushing ICU4X as the preferred model.

  6. gudrodur commented on Sep 8, 2026

    @gudrodur

    Four years on, with one number I don't think anyone has measured, and I think it
    reframes the size question this stalled on.

    Vivaldi ships Icelandic, and it is the same Chromium.

    Measured today on one machine, both browsers on Chromium 152, minutes apart,
    each driven over CDP against a fresh profile:

    Vivaldi 8.2.4 Chrome 152
    Intl.DateTimeFormat("is").resolvedOptions().locale is en-US
    Intl.Collator("is").resolvedOptions().locale is en-US
    DateTimeFormat.supportedLocalesOf(["is"]) ["is"] []
    two-letter locales with DateTimeFormat data 61 60
    two-letter locales with Collator data 54 53
    icudtl.dat on disk 10,957,760 B 10,876,560 B

    One locale of difference. 81,200 bytes — 0.7% of the ICU data both already
    carry. Neither browser ships an Icelandic UI translation, so this is not
    Chromium's "non-UI language" rule behaving differently for Vivaldi; it is a
    downstream vendor deciding the filter costs more than it saves.

    Why that matters for this issue specifically: the 2022 discussion was about
    swapping in the entire icudt*.dat (binary 63 → 82 MB), and the follow-up
    question in #64 — "so the data file goes up from ~10MB to ~30MB?" — has been
    quoted ever since as though it were a measurement. It was a question, and it was
    never answered. The number that was actually missing is the marginal cost of
    one locale
    , and on a shipping Chromium build it is ~81 KB.

    That makes the ask a much smaller one than the one that stalled: not "embed full
    ICU", but "keep full data for the locales people ask for". At ~81 KB each, the
    startup concern raised at the time is hard to make load-bearing — it is 0.7%
    more mmap'd data, not 3x.

    One thing worth weighing alongside it: Collator is the part userland cannot
    fix.
    formatjs ships polyfills for DateTimeFormat, NumberFormat,
    RelativeTimeFormat, ListFormat and PluralRules. There is no Collator
    polyfill and none is in progress. So on workerd today localeCompare(a, b, "is")
    sorts Icelandic names by the English alphabet and there is nothing to install —
    Þórður lands among the T's and Ævar sorts second instead of second to last. We
    ended up hand-writing a collator pinned to Node's full ICU rather than shipping
    a member list in the wrong order. It works, and it should not have to exist.

    Happy to test a build if relaxing the filter for a locale or two is easy to try.

  7. gudrodur commented on Sep 8, 2026

    @gudrodur

    Correction to my own comment above, before anyone acts on it.

    I wrote that "there is no Collator polyfill and none is in progress". That is wrong, and a reader who checks will find it in about ten seconds. @formatjs/intl-collator has been on npm since 2026-05-14 and is at 0.2.7. I asserted the gap instead of checking it, which is exactly the kind of claim I was complaining about elsewhere in that comment.

    Having now measured it, here is what it does, because the picture is more interesting than either "no polyfill" or "problem solved". Node v24.19.0 full ICU as the oracle, polyfill-force, compared against the native Intl.Collator("is") it replaces:

    @formatjs/intl-collator 0.2.7
    the ten-name Icelandic list correct
    Icelandic alphabet + tailored letters, 5,473 ordered pairs 10 divergences — all on ä / ø / å
    every ordered pair of the 450 Latin letters, 202,500 25,766 divergences (12.7%)
    bundled, minified, gzipped 707,603 B

    So it is real and it gets the Icelandic alphabet right, which is more than I gave it credit for. Its own README is candid about the rest: it provides "the ECMA-402 constructor/prototype surface and a deterministic baseline comparator", with "full CLDR/UCA collation data compilation" still tracked as future work. That matches what I measured — the basics land, CLDR exactness does not.

    What this changes about the argument, honestly:

    • Weaker than I put it. "Nothing to install" was false. Something is installable today and will sort ordinary Icelandic names correctly.
    • Still standing, for a narrower reason. 707,603 B gzip to get collation that is 12.7% off ICU across Latin is not a fix a site can reasonably ship, and it still misses the ä/ø/å tailoring that CLDR specifies for Icelandic. A Polish or Turkish name in an Icelandic list is where it shows.

    None of this touches the measurement the comment was actually about: Vivaldi ships Icelandic on Chromium 152 for 81,200 bytes, and coll_tree is in the pending Chromium CL either way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions