two bugs

[Samrose-Ahmed]

I was testing the basic scan on some real data, found what I think is two bugs

1. One is this line (

   iceberg-rust/crates/iceberg/src/spec/manifest.rs

   Line 1364 in 35f3481

   m.insert(entry.key, entry.value.try_into()?);

   ), it needs to be able to handle negative numbers, I had eg null_value_counts where value was -1, I think that can be mapped to 0 here.

2. In this function (

   iceberg-rust/crates/iceberg/src/spec/manifest.rs

   Line 1333 in 35f3481

   .ok_or_else(|| {

   ) we shouldn't throw this error, but just skip if the field is not found, due to schema evolution.

[liurenjie1024]

I'll take this.

[liurenjie1024]

Hi @Samrose-Ahmed Sorry for late reply.

One is this line (

iceberg-rust/crates/iceberg/src/spec/manifest.rs

Line 1364 in 35f3481

m.insert(entry.key, entry.value.try_into()?);
), it needs to be able to handle negative numbers, I had eg null_value_counts where value was -1, I think that can be mapped to 0 here.

In what case this might be negative value, i.e. used for null_value_counts? Why counts may be negative value? cc @Fokko Any ideas?

In this function (

iceberg-rust/crates/iceberg/src/spec/manifest.rs

Line 1333 in 35f3481

.ok_or_else(|| {
) we shouldn't throw this error, but just skip if the field is not found, due to schema evolution.

Sound reasonable to me. I will fix it.

[Fokko]

Thanks for giving it a try @Samrose-Ahmed 🙌

In what case this might be a negative value, i.e. used for null_value_counts? Why counts may be negative value? cc @Fokko Any ideas?

Negative numbers don't make sense. If there are no statistics, the field-id key should be omitted from the map. I think discarding the entry makes more sense than setting it to 0. @Samrose-Ahmed Do you know how this Iceberg table was created?

[Samrose-Ahmed]

Yeah I think discarding makes sense. this table was written by the Iceberg Java library and I've read it with spark and trino.

[liurenjie1024]

This bug has been fixed, could you have another try? cc @Samrose-Ahmed