Repository navigation
zstd support #54
Description
Activity
Currently libzstd is not linked into Chromium. It would add roughly 700KB to the binary size (maybe can be trimmed a bit). This would be a hard sell if the compression standard is the only user.
It seems that you did manage to trim the size a bit as this change only added 38k. Was that just by dropping everything besides decompression, or were there other tricks you made use of?
On the storage team we are also interested in using zstd for compressing IndexedDB values before sending them over IPC (or storing as blobs). Adding the compression half of zstd appears to add another 200kB.
So that's at least two users of zstd.
We didn't do anything special in Chromium to reduce the size increase, except for only compiling in the decompression code. I think we were just lucky.
I would like to move this specification to WHATWG before we add new features, since it is now implemented in Firefox and Safari and we would like to have consensus.
That means it will probably be some months before there's movement on adding zstd.
I would like to move this specification to WHATWG before we add new features, since it is now implemented in Firefox and Safari and we would like to have consensus.
Why would it take that much to move to WHATWG?
Note: Chrome/Chromium, Firefox, and Opera have agreed to add zstd to content-encoding options. Chrome is planning on General Availability in 118.
Reacted by Jedipedia and VReacted by Kagami Sascha Rosylight and VAnother note: Firefox now supports
Content-Encoding: zstdas of version 126 but there's still no support fornew DecompressionStream("zstd")in either Firefox or Chromium.Please add it to the specification, as it seemingly blocks from implementing
new DecompressionStream("zstd")in browsers:https://issues.chromium.org/issues/348499728
Reacted by Ruslan Burakov, Jedipedia, 第二扩展 and justin0mcateerJust as brotli, we'd be open to ship only DecompressionStream support for zstd, as @jesup reported 600KB size increase to ship the compression side of the library.
Reacted by JedipediaJust as brotli, we'd be open to ship only DecompressionStream support for zstd, as @jesup reported 600KB size increase to ship the compression side of the library.
Thanks. I think this will also be Blink's position.
Bun would be interested in supporting this since we already ship with zstd support internally for other APIs
Reacted by justin0mcateerI heavily use
DecompressionStreamfor gzip. I would love to seebrotliandzstdsupport for this as reading data is drastically more useful/common in browsers than writing. Commenting to understand better the process here. Will this become a priority within the next year or so?We need someone to write an implementation, a PR to update this specification, and web platform tests. Or three separate people, it doesn't matter.
At the last W3C TPAC we reached an agreement in principle to add this to the standard, someone just needs to do the work.
Unfortunately Chromium won't be able to ship this until there is some real-world usage, due to the binary size problem.
Did we? I think we only agreed about brotli.
@saschanaz My recollection is that we had a proper agreement for Brotli, and someone suggested we apply it to zstd too, and there was a general sense of agreement. However, if you think there's more to be done to get agreement for zstd, I'm happy to wait for that.
Reacted by Anne van KesterenI remember (and per the meeting minutes) I checked again about zstd at the end of the meeting and everyone said brotli only... But maybe I'm wrong?
(if others are fine with it, Mozilla would be happy to consider exposing our ongoing zstd implementation)
Reacted by arukiidou and Thomas DagenaisColleagues and I would like to see
zstdsupport inDecompressionStreamin support of the GPU texture "supercompression" format Basis Universal. This file format contains blocks ofzstd-compressed coefficients. It is already used by libraries like Three.js and is a recommended texture format in glTF - both for texturing geometry in 3D web apps. Direct decompression support in the browser would reduce the size of WebAssembly modules which operate on these files. From our standpoint, compression support isn't needed. Thanks in advance for considering this; we're hoping this gets standardized.Reacted by 地図の神様, Hiromichi Goto, marpe, Kanashimia, Theo Paris, Thomas Dagenais and Timothée DenizouBun shipped as
"zstd"in v1.3.3.Firefox added support for
brotliandzstdtoCompressionStream/DecompressionStreamin mdn/browser-compat-data#28619 and linked issues / PRs. Is it possible to move forward this support across browsers?(To be clear Gecko never shipped zstd but only implemented it behind a pref. And decoding only.)
Sorry, I meant to say "behind a pref" - thanks for the correction.
From our partners' standpoint, support only in
DecompressionStreamwould still be helpful, if compression support is difficult, or increases code size too much.Reacted by Vitalik Gordon, roelentless, Keitaroh Kobayashi, Thomas Dagenais and Timothée DenizouNianticLabs recently switched from gzip to zstd compression for Gaussian Splatting Compression. ( nianticlabs/spz#73 )
Edit: Added link to the blog post https://www.nianticspatial.com/blog/spz4For our ($COMPANY) web gaussian splatting viewer we can use DecompressionStream("gzip") in order to not ship a wasm module for decompression and would love to be able to use a DecompressionStream("zstd") for this new version as well.
Every implementation should have zstd decoder so nothing should really block shipping. How much are we concerned about having only decoder?
My standpoint still stays same as in 2024 TPAC: Either we ship both or decoder only, the spec should specify so (and not make it optional) and we should have a good cross-vendor agreement that enough implementations are willing to ship, to prevent webcompat issues.
Reacted by Timothée DenizouWe lack WebKit voice here; filed WebKit/standards-positions#667.
Having spent some time with Firefox's native
new DecompressionStream("zstd")(enablingdom.compression_streams.zstd.enabledinabout:config), our feedback is:Performance is 10x faster than the fzstd JavaScript library we were using before. But if we use a WASM library like the bokuweb port of Facebook's original zstd library we get identical performance to the native
new DecompressionStream("zstd").file size native fzstd bokuweb zstd-wasm-dec 2 MB 5 41 3 5 6 MB 14 120 14 14 5.5 MB 16 121 15 20 11 MB fail 254 37 fail 10 MB fail 195 21 fail More worryingly, native Firefox can only decompress files up to 8 MB in size, otherwise it fails with:
TypeError: zstd decompression error: Frame requires too much memory for decoding
This is fine for decoding HTTP responses but in a web app, we'll want to decode files 200 MB in size or larger, so we're stuck with bundling the WASM library anyway (which can decompress up to 2 GB). Given it has identical performance, there's not much point in using
new DecompressionStream("zstd")until the 8 MB restriction is lifted, except to maybe keep the processing out of the main thread for < 8 MB files.Reacted by 地図の神様 and GTATom
Issue for discussion of adding support for the zstd format to the API.