Repository navigation
WebNN should support int8 quantized models #128
Description
Activity
@wchao1115 @huningxin do you think we should label this as "cr" for #240 purposes?
I think this is important one and support to label as "cr".
@wchao1115 this issue was on the agenda today, but we had to defer due to timing. Let us know your thoughts. I'm planning to bring this up for our next meeting for discussion.
Per discussion at https://www.w3.org/2022/03/24-webmachinelearning-minutes.html#t06 we consider this to be in scope for CR.
We've discussed this feature on our recent meetings:
https://www.w3.org/2022/09/22-webmachinelearning-minutes.html#t05
https://www.w3.org/2022/09/08-webmachinelearning-minutes.html#t05
https://www.w3.org/2022/08/25-webmachinelearning-minutes.html#t06I will label this issue as "v2" due to required implementation experience for the initial CR inclusion. There's a mechanism for us to publish a Candidate Recommendation Draft subsequent to the initial CR that would give us adequate time to properly define, develop and test this feature.
Furthermore, we should soon start discussing WebNN "v2" plan as we look to extend our current charter and this feature could be one concrete feature to highlight. We can continue discuss this feature on our bi-weekly calls when there's new information and revise our position as appropriate.
- added a commit that references this issue
on Dec 12, 2022 It looks like this was added to the spec in 0970115 and we may have some implementation experience at this point. Close, despite it being marked v2 ?
The int8 quantized models may need some extra ops, for example
DynamicQuantizeLinear,DequantizeLinear,ConvIntegerandMatMulInteger, that are missed in current spec.Transformer Models Analysis spread sheet has more details of ops required by int8 quantized model (see columns marked with (int8)).
Reacted by Joshua BellThe int8 quantized models may need some extra ops, for example
DynamicQuantizeLinear,DequantizeLinear,ConvIntegerandMatMulInteger, that are missed in current spec.Indeed, I have those 4 prototyped here (a minimal first four): https://github.com/fdwr/chromium-src-webnn-dml/pull/1/files#diff-e1b2517a6ae8f7c4494c75d17c8650b56e4f8d430f54f5e1f765475f00a5e1f3R427-R433
Seems int4 quantization is also a thing (with negligible impact on output quality). int4 practically halfs the VRAM requirement of the model, and offers a speedup on devices that support them.
Example of a int4 quantization model: https://huggingface.co/01-ai/Yi-6B-Chat-4bits
Should this be considered for v2? Or is int4 too specific? (I'm not sure if 4bit is adequate for image or audio models)
// There's a more aggressive {-1,0,1} quantization. It's fairly new, and I believe it's application is limited to language models.
Reacted by Dwayne RobinsonThe BitNet paper was really cool. https://arxiv.org/abs/2310.11453
Reacted by Dwayne RobinsonThe int8 quantized models may need some extra ops, for example
DynamicQuantizeLinear,DequantizeLinear,ConvIntegerandMatMulInteger, that are missed in current spec.Indeed, I have those 4 prototyped here (a minimal first four): https://github.com/fdwr/chromium-src-webnn-dml/pull/1/files#diff-e1b2517a6ae8f7c4494c75d17c8650b56e4f8d430f54f5e1f765475f00a5e1f3R427-R433
Hey @fdwr - how fresh is your prototype of these? And have you looked at how other backends (CoreML, TFLite) would implement these? Starting with the "minimum viable" quantization support as outlined in #623 is appealing!
Reacted by Dwayne Robinsonhow fresh is your prototype
It's moldy bread by now (but snippets could be reused). The ORT WebNN EP implementation still exists (it was originally added during prototyping) and would light up again once the op is added into Chromium.
And have you looked at how other backends (CoreML, TFLite)
There are differences, but they should be expressible (🤞). For dequantization, most decompose to
output = mul(sub(input, zeroPoint), scale)(except TF full, CoreML MIL's LUT mode, and CoreML's scale&bias form). They have differing broadcasting rules, which I'd like to iron out to be more consistent (consistent with unidirectional broadcasting of its decomposition andexpand).API Name Equation Types TFLite DequantizeOp real = (quantized - zeroPoint) * scale (link) input: uint4, uint8, int8, int16, float16
zeroPoint: uint8
scale: float32
output: float32TF tf.quantization.dequantize output = minRange + (input * (maxRange - minRange) / dataTypeRange) input: uint8
minRange: float32
maxRange: float32
dataTypeRange: int
output: float32CoreML MIL constexpr_affine_dequantize real = (input - zeroPoint) * scale input: uint8, int8
zeroPoint: uint8, int8, float32
scale: same as output
output: float16, float32CoreML MIL constexpr_lut_to_dense real = lut[input] input: uint1, uint2, uint4, uint6, uint8
output: uint8, int8, float16, float32CoreML LinearQuantizationParams ? input * scale + bias ? input: ?
scale: float32
bias: float32
output: ?ONNX DequantizeLinear real = (input - zeroPoint) * scale input: uint4, int4, uint8, int8, uint16, int16, int32, float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz
zeroPoint: same as input
scale: same as output
output: bfloat16, float16, float32DML DEQUANTIZE_LINEAR real = (input - zeroPoint) * scale input: uint4, int4, uint8, int8, uint16, int16, uint32, int32
zeroPoint: same as input
scale: same as output
output: float16, float32
Supporting int8 quantized models is essential for mobile scenarios and in many NPU architectures. TensorFlow (Lite) and ONNX, for instances, have int8 quantization support built-in, and WebNN should to. Related #93