Repository navigation
Remove the constant tensor support #940
Description
Activity
The original motivation for constant tensors (issue #760) was to enable reusable immutable weights.
Has that optimization path been abandoned, or is there another reason the original rationale no longer applies? It would be helpful to document that in the issue.
Given the only backend which supported it was removed from the Chromium prototype I think the working group needs to redo the analysis of whether this approach is implementable or if there is an alternative mechanism to achieve the same optimization.
- How likely are we to add it back later? If so, how soon?
- How much additional complexity does keeping it incur?
- Are there really no other backends that benefit? I thought CoreML wanted certain tensors to be constant, or at least could be more efficient if it knew that a tensor was constant
MLTensor.isConstant == true(e.g. able to call dequantize directly rather than emulate through some decomposition).
I thought CoreML wanted certain tensors to be constant, or at least could be more efficient if it knew that a tensor was constant MLTensor.isConstant == true (e.g. able to call dequantize directly rather than emulate through some decomposition).
What Core ML needs is already achieved by using
constant().The "constant tensor support" being referred to here is about allowing constants to be shared between multiple graphs associated with a context. That is not implementable with Core ML unless all of the graphs sharing constants are built at the same time so they can be combined into a single
.mlpackagefile with separate "functions".What Core ML needs is already achieved by using constant().
Oh yeah,
constant()returns anMLOperand(which is missingisConstant) rather than anMLTensor.I think prefill/decode is one such scenario. Both graphs execute the same transformer and reuse the same immutable weights. Looking at ORT,
AddExternalInitializersappears to enable cross-session weight sharing, suggesting this is likely an implementation gap. CoreML suggests it can do it but not with the current API shape.Reacted by Dwayne Robinson
The constant tensor feature allows tensors to be created with initial data and used as graph constants. However, because no backends currently support constant tensors, we should remove this feature to simplify the code.
We should remove the API createconstanttensor and readonly attribute boolean constant.
@huningxin @fdwr @RafaelCintron @reillyeon