Repository navigation
Specify the operand data type constraints of operation #283
Description
Activity
There are other operations should constrain the operand to floating-point types, e.g., batchNormalization, elu, hardSigmoid, hardSwish, instanceNormalization, leakyRelu, linear, sigmoid, softplus, softsign, tanh.
Reacted by Dwayne Robinson- changed the title
[-]Softmax should only support input of floating-point types[/-][+]Specify the operand type constraints of operation[/+]on Sep 21, 2022 For some element-wise unary ops,
ceilandfloorshould only accept floating-point types.absandnegshould only accept signed types, including signed integer and floating-point types. This feedback is from Chromium CL review. Thanks @wacky6 and @miaobin!Additionally,
cos,exp,log,sinandtanshould accept floating-point types.Reacted by Dwayne RobinsonreduceL2, reduceLogSum, reduceLogSumExp and reduceMean should only accept floating-point types.
That's consistent with DML's data type support too:
REDUCE: ARGMIN, ARGMAX: featureLevel: 4.1 InputDataType: uint8, uint16, uint32, uint64, int8, int16, int32, int64, float16, float32 OutputDataType: uint32, uint64, int32, int64 MinRank: 1 MaxRank: 8 AVERAGE, L2, LOG_SUM, LOG_SUM_EXP: featureLevel: 3.0 InputDataType: float16, float32 OutputDataType: float16, float32 MinRank: 1 MaxRank: 8 L1, SUM_SQUARE: featureLevel: 5.0 InputDataType: uint8, uint16, uint32, uint64, int8, int16, int32, int64, float16, float32 OutputDataType: uint8, uint16, uint32, uint64, int8, int16, int32, int64, float16, float32 MinRank: 1 MaxRank: 8 MIN, MaxRank: featureLevel: 5.0 InputDataType: uint8, uint16, uint32, uint64, int8, int16, int32, int64, float16, float32 OutputDataType: uint8, uint16, uint32, uint64, int8, int16, int32, int64, float16, float32 MinRank: 1 MaxRank: 8 MULTIPLY, SUM: featureLevel: 5.0 InputDataType: uint8, uint16, uint32, uint64, int8, int16, int32, int64, float16, float32 OutputDataType: uint8, uint16, uint32, uint64, int8, int16, int32, int64, float16, float32 MinRank: 1 MaxRank: 8Reacted by Ningxin Hu and lisa0314Just curious, what's the expected behavior for integer overflows for MULTIPLY and SUM?
I guess float overflow should just become
Infinity.Just curious, what's the expected behavior for integer overflows for MULTIPLY and SUM?
I recall surveying a number of libraries a while back for integers, and they all did two's complement wrap rather than saturate. That is what DML does. I presume XNNPack too?
e.g.
import numpy x = numpy.array(200, dtype=numpy.uint8) y = numpy.add(x, x) print("value:", y) print("shape:", y.shape) # Prints: # value: 144 # shape: ()
I guess float overflow should just become Infinity.
👍
[WIP] Summary of the operand data type constraints for current WebNN operations.
argMin/argMax
batchNormalization
- input: float32, float16
- mean: same as input
- variance: same as input
- scale: same as input
- bias: same as input
- output: same as input
clamp
concat
conv2d
convTranspose2d
Element-wise binary operations
add
- a: all supported data types
- b: same as a
- output: same as a
sub
- a: all supported data types
- b: same as a
- output: same as a
mul
- a: all supported data types
- b: same as a
- output: same as a
div
- a: all supported data types
- b: same as a
- output: same as a
min
- a: all supported data types
- b: same as a
- output: same as a
min
- a: all supported data types
- b: same as a
- output: same as a
pow
- a: all supported data types
- b: same as a
- output: same as a
Element-wise unary operations
abs
ceil
cos
exp
floor
log
neg
sin
tan
elu
expand
gather
gelu
gemm
gru
- input: float32, float16
- weight: same as input
- recurrentWeight: same as input
- bias: same as input
- recurrentBias: same as input
- initialHiddenState: same as input
- outputs: same as input
gruCell
- input: float32, float16
- weight: same as input
- recurrentWeight: same as input
- hiddenState: same as input
- bias: same as input
- recurrentBias: same as input
- output: same as input
hardSigmoid
hardSwish
instanceNormalization
layerNormalization
leakyRelu
linear
lstm
- input: float32, float16
- weight: same as input
- recurrentWeight: same as input
- bias: same as input
- recurrentBias: same as input
- peepholeWeight: same as input
- initialHiddenState: same as input
- initialCellState: same as input
- output: same as input
lstmCell
- input: float32, float16
- weight: same as input
- recurrentWeight: same as input
- hiddenState: same as input
- cellState: same as input
- bias: same as input
- recurrentBias: same as input
- peepholeWeight: same as input
- outputs: same as input
matmul
pad
Pooling operations
averagePool2d
l2Pool2d
maxPool2d
prelu
Reduction operations
reduceL1
reduceL2
reduceLogSum
reduceLogSumExp
reduceMax
reduceMean
reduceMin
reduceProduct
reduceSum
reduceSumSquare
relu
resample2d
reshape
sigmoid
slice
softmax
softplus
softsign
split
transpose
triangular
where
Reacted by Dwayne Robinson and shiyi@fdwr , if I read the DML doc correctly, L1, SUM_SQUARE, MULTIPLY, SUM reduce functions only support 32 and 64 bit integers.
@fdwr , if I read the DML doc correctly, L1, SUM_SQUARE, MULTIPLY, SUM reduce functions only support 32 and 64 bit integers.
@huningxin : Correct.
Summary of the operand data type constraints for current WebNN operations
Is this the intersection of DML with XNNPack? (because FL3+ DML ABS supports int16 too, and FL4.1+ supports int8/int16/int32/int64).
- changed the title
[-]Specify the operand type constraints of operation[/-][+]Specify the operand data type constraints of operation[/+]on Jan 31, 2024 @huningxin in the list above, when an op is listed like this:
batchNormalization
input: float32, float16
mean: same as inputShould that "same as input" statement be interpreted as:
(1) mean's data type has the same restriction as input (i.e. float32 or float16); or
(2) the given mean operand's data type must exactly match the given input operantd's data typeI assume the latter (based on other examples) but wanted to confirm.
Should that "same as input" statement be interpreted as: (1) mean's data type has the same restriction as input (i.e. float32 or float16); or (2) the given_mean_ operand's data type must exactly match the given input operantd's data type
@inexorabletash Yep, latter. If
inputis float32, thenmeanmust also be float32.Reacted by Joshua Bell and Ningxin Hu2 remaining items
My read of DML doc is that
conv(fp16) -> fp32isn't permitted.@wacky6 You are correct - DML almost always requires the input and output data types to be the same (some notable exceptions are cast and quantize/dequantize operators).
Does DML use a fp32 accumulator internally, then casts the result to fp16? Or is it fp16 accumulation all the way (might saturate the range and yield Infinity / NaN)?
It depends on the op (reduction needs more intermediate precision than simple addition) and which flags are passed (e.g.
DML_EXECUTION_FLAG_ALLOW_HALF_PRECISION_COMPUTATIONwhich can be faster on some devices, at the cost of precision).caller shouldn't add/multiple matrices of different types
I generally agree, as mixing any multiple different input types would explode the test matrix and increase backend complexity.
The table is missing:
Added the following ops into the table.
- softmax but it kicked off the issue as "float32" and "float16" only
+1
- softplus, softsign, tanh are mentioned in a comment as floating-point only
+1
- gelu
floating-point only
- layerNormalization
floating-point only
- gru, gruCell
floating-point only
- lstm, lstmCell
floating-point only
- argmin/argmax (presumably anything, like most of the reduction ops)
argmin/argmax's output is int64
- expand, gather, slice, transpose, triangular (presumably anything)
gather's indices is uint32 or int64
- where (already in the spec, condition must be uint8, other other must match input)
+1
Reacted by Dwayne RobinsonHi,
I've compared it with what CoreML support, the main difference is:- For almost all the ops that support all types, CoreML support fp32, fp16, int32.
- Argmax/argmin output is int32
- Scalar parameters, like gather
indices, conv2dgroupstype is int32. - For floating points computation, NPU only supports
fp16. CPU & GPU supports bothfp32andfp16. - Overal Model input/output:
fp16,fp32,int32. So if the last op of the model generatesint8, it errors out.
If WebNN declares wider type set, I think we would need some way to feature detect(#463).
Reacted by Dwayne Robinson and Ningxin Hu- added a commit that references this issue
on Apr 26, 2024 For almost all the ops that support all types, CoreML support fp32, fp64, int32.
@philloooo The
float64type is interesting because in contrast, DML doesn't supportfloat64for any math operators (only for bitwise data movement operators like transposing and gather and bitwise and/or, plus cast at the edges of partitions) but DML does supportint64, whereas CoreML supportsfloat64but doesn't supportint64for any operators. 🤹If WebNN declares wider type set, I think we would need some way to feature detect(#463).
Agreed. Differences across implementations will be inevitable (similarly, WebGPU doesn't support all GPUTextureFormats, like
astc-4x4-unormacross all GPU's), but if they can be testable up-front, that avoids wastefully loading a model only to fail later, and if any differences occur broadly (e.g. entire data types missing), that's easier to deal with than sporadic differences (certain operators missing) because you could easily generate two separate models with different data types, but it would be tougher to generate multiple models depending on various permutations of spotty operator support. TheMLContextis probably the right place to ask that before you even make agraphBuilder. e.g.interface MLContext { Promise<MLComputeResult> compute(MLGraph graph, MLNamedArrayBufferViews inputs, MLNamedArrayBufferViews outputs); + boolean isTypeSupported(MLOperandDataType dataType); };I know it's bad form to comment on a closed issue, but... the table lists this for ReLU and PReLU:
- input: float32, float16, int32, int8
@philloooo points out that most other activations support only floats, and #283 (comment) says that prelu should only accept floats?
@huningxin - is this intentional?
@huningxin - is this intentional?
Yes. The intention is having them to accept the signed values.
#283 (comment) says that prelu should only accept floats?
In Chromium CL review, we'd like to prototype
preluby staring from floating point data point in sake of simplicity.- added 2 commits that reference this issue
on May 27, 2024 - added a commit that references this issue
on May 22, 2025 - added a commit that references this issue
on Aug 8, 2025
The current spec doesn't specify the operand type constraints of an operation. However, some operations, e.g.,
softmaxshould only support float32 operand type according to the survey of frameworks and native ML APIs in the following table. The lack of the operand type constraints specification would lead to implementation issue, such as Chromium CL 3856752.Thanks @wacky6 for pointing this out.