Support style-transfer models with changes/additions to the following operations

[wchao1115]

#108

• Extend the conv2d operation to support transposed convolution, an essential upsample tool for encoder-decoder models.
• Add instanceNormalization in addition to batch-normalization. Instance-normalization is a fused operation for a normalization subgraph that computes the mean and variance values per-feature instance on the fly.
• Replace the sqrt unary operation with a more generic pow binary operation used by the normalization process.
• Add pad operation that supports all 4 padding modes found in various frameworks.
• Add resample operation to support both upsampling and downsampling of feature instances. This operation is used in the ONNX version of the style-transfer models.

---

Preview | Diff

[anssiko]

Thanks! Added to the 2020-12-10 agenda for discussion.

As usual, if we get all the reviews completed ahead the call we can merge this earlier than that and recap the changes on the call.

[wchao1115]

@gramalingam

[gramalingam]

I wonder if it would be better to not allow "Operands" in "Options", and make "Operands" explicit (even if optional). I also think it is useful to restrict "Options" to values that are known at the graph-building time (while "Operands" represent values known at run-time).

[wchao1115]

@gramalingam That's a good observation and I have thought about this exact point but ultimately went for the idea that any optional param, whether it's an operand or not, ought to be placed in the options struct because the original motivation for this is to simplify future changes to the API so that no overload will ever be needed in the future.

So even if we are willing to draw an arbitrary line right now barring a non-compile time param from entering the options struct, we will be forced to break that rule anyway in the future at the very first instance of the need to extend the API to support an additional optional operand.

[gramalingam]

"Any optional param ought to be placed in the options struct" seems a bit extreme. In the end, either is functionally equivalent, and it is mostly a matter of style as to which is more readable. I personally prefer making the operand explicit, but am okay with this

[huningxin]

Looks good, thanks for your great efforts @wchao1115 !

[gramalingam]

One general question relating to padding in convolution etc.: ONNX and TF have options like SAME for padding. I don't see something similar here, is there a way to achieve that?

[wchao1115]

One general question relating to padding in convolution etc.: ONNX and TF have options like SAME for padding. I don't see something similar here, is there a way to achieve that?

Algorithmic padding can be handled by the padding array. Here, the SAME padding simply means whether the extra paddings are to be done to the left (lower end) or to the right (upper end) of the input spatial dimensions. ONNX has more flexibility in specifying which side of the input dimensions is prioritized to receive the extra padding first while TensorFlow SAME padding has a fixed pad upper logic.

That said if the input dimensions aren't known at graph building time (in models with free dimensions), then obviously the padding array can't be calculated ahead of time either, similar to the argument that @huningxin made earlier in the thread about the need for outputSizes in the transposed case. We can add an autoPad option to mitigate that issue at runtime.

[wchao1115]

@gramalingam Can you please take a look at the latest commit? Based on your feedback, an autoPad option is added to handle the case where the input shape isn't known at graph construction time.