Support `initial-default` when reading Avro

[Fokko]

I would suggest adding support for initial-default to read V1 Avro files into V2 objects in memory.

For example, V1 does not have sequence numbers, and then need to be set to 0. This is important when you have a V1 table, and then you upgrade it to V2, the underlying Avro files are still V1 until you rewrite everything. See comment here: #79 (comment)

With the initial-default you could just read a 0 if the field is not present.

[ZENOTME]

If we read a V1 Avro file as a V2 object, when we write back, we also need to set format-version in manifest as 2, right?

I realized that I miss the case that read the v1 avro file as v2 object and write it back. I will complete them and solve this issue.

[Fokko]

If we read a V1 Avro file as a V2 object, when we write back, we also need to set format-version in manifest as 2, right?

Yes, this is correct, then the version of the table will be 2. There are two situation then:

• If you do a full overwrite, then manifest will be rewritten and everything will be V2.
• If you do an append, there can still be V1 manifests in the metadata tree.

[ZENOTME]

If we read a V1 Avro file as a V2 object, when we write back, we also need to set format-version in manifest as 2, right?

Yes, this is correct, then the version of the table will be 2. There are two situation then:

• If you do a full overwrite, then manifest will be rewritten and everything will be V2.
• If you do an append, there can still be V1 manifests in the metadata tree.

Thanks for your explanation!