fix: empty record dtypes

[yokomotod]

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

• Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
• Ensure the tests and linter pass
• Code coverage does not decrease (if any source code was changed)
• Appropriate docs were updated (if necessary)

Fixes #2148 🦕

Can we increase the minimum supported pandas from 1.1.4 to 1.3.0 ?

[yokomotod]

pandas-dev/pandas#41052 has released with pandas 1.3.0 (as mentioned in above issue: pandas-dev/pandas#41241 (comment))

So if we can increase minimum pandas version to 1.3.0, we don't need this workaround

[tswast]

We should really come up with a consistent policy across all Google packages:

• pandas-gbq supports 1.1.4+ (https://github.com/googleapis/python-bigquery-pandas/blob/b3a9af27960c6e2736835d6a94e116f550f94114/setup.py#L27)
• db-dtypes supports 1.2.0+ (https://github.com/googleapis/python-db-dtypes-pandas/blob/44a5aa24ef3219849e9e0bedeeaebc4b9ee53d9e/setup.py#L34)
• bigframes supports 1.5.3+ (https://github.com/googleapis/python-bigquery-dataframes/blob/7d00be67cf50fdf713c40912f207d14f0f65538f/setup.py#L53)
• Colab and BigQuery Studio Python notebooks are both in the 2.x range now.

CC @chalmerlowe @Linchin @shollyman @parthea

[Linchin]

Indeed, moving to a monorepo might help make the dependencies more consistent, ideally we might want any overlapped dependencies among the handwritten BigQuery projects to be consistent.

But just for the purpose of this PR, I think it's reasonable to increase the minimum pandas version to 1.3.0, as long as it's not breaking anything.

[yokomotod]

I confirmed that pandas==1.2.5 reproduces

FAILED tests/unit/test_table.py::TestRowIterator::test_to_dataframe_w_bqstorage_no_streams - ValueError: need at least one array to concatenate

and pandas==1.3.0 can pass

[tswast]

Thanks for updating the constraint file!

[tswast]

This looks good to me, but will wait for @chalmerlowe or @Linchin to approve.

[tswast]

We should really come up with a consistent policy across all Google packages:

• pandas-gbq supports 1.1.4+ (https://github.com/googleapis/python-bigquery-pandas/blob/b3a9af27960c6e2736835d6a94e116f550f94114/setup.py#L27)
• db-dtypes supports 1.2.0+ (https://github.com/googleapis/python-db-dtypes-pandas/blob/44a5aa24ef3219849e9e0bedeeaebc4b9ee53d9e/setup.py#L34)
• bigframes supports 1.5.3+ (https://github.com/googleapis/python-bigquery-dataframes/blob/7d00be67cf50fdf713c40912f207d14f0f65538f/setup.py#L53)
• Colab and BigQuery Studio Python notebooks are both in the 2.x range now.

CC @chalmerlowe @Linchin @shollyman @parthea

[tswast]

Thanks for updating the constraint file!

[Linchin]

I think we need to remove some liens in the unit test so coverage test can pass, could you fix it for us? Then we should be able to merge the PR

Name                                                         Stmts   Miss Branch BrPart  Cover   Missing
--------------------------------------------------------------------------------------------------------
tests/unit/test_table.py                                      3288      2     86      1    99%   4152-4153

[yokomotod]

@Linchin done ffd6e42

Indeed, pandas>=1.3.0 must have Float64Dtype

https://pandas.pydata.org/pandas-docs/stable/whatsnew/v1.2.0.html#experimental-nullable-data-types-for-float-data

[Linchin]

A different system test is failing, could you fix it too? Thanks

[yokomotod]

Float64 is introduced in pandas 1.2.0, not 1.4. So we can drop this handling now (pandas >= 1.3.0)

https://btx.cloud.google.com/invocations/feb3ac69-f2db-449c-ab07-2122420f069b/targets/cloud-devrel%2Fclient-libraries%2Fpython%2Fgoogleapis%2Fpython-bigquery%2Fpresubmit%2Fsystem-3.9/log

nox > python -m pip freeze
  .
  .
  .
pandas==1.3.0
  .
  .
  .

        if PANDAS_INSTALLED_VERSION >= "1.4":
            assert df.dtypes["float64_col"].name == "Float64"
        else:
>           assert df.dtypes["float64_col"].name == "string"
E           AssertionError: assert 'Float64' == 'string'
E             
E             - string
E             + Float64

https://pandas.pydata.org/pandas-docs/stable/whatsnew/v1.2.0.html#experimental-nullable-data-types-for-float-data

[yokomotod]

@Linchin sorry to overlooking another error, I fixed it too.

Can you try to run Ci test again ?

I have trouble to run the system test on my local (maybe due to old numpy on apple silicon) but the tests should pass.

[Linchin]

LGTM. Thanks for your contribution!