fix(bigquery): use pyarrow fallback for improved schema detection

[plamut]

Closes #9206.

This PR adds additional logic to schema autodetection if pyarrow is available, and the types could not be detected for all columns.

How to test

• Make sure that pyarrow is installed
• Try loading data from a dataframe to a new table when the dataframe contains columns of type string, date, etc. (those that end up being the Pandas dtype "object"). Do not provide an explicit schema to the load job.

Actual result (before the fix):
The backend responds with an error (incompatible types).

Expected result (after the fix):
The load job completes successfully, and column types are correctly detected (strings, dates...).

Misc

The code might not work for all scalar types, depending on how good the pyarrow's schema detection logic is. Which types to we care about the most?

We should update the test case to include all of them (we currently test strings and dates).

[tswast]

I don't think we're supposed to use Arrow type objects as dictionary keys. The factory functions all describe instances of data types (https://arrow.apache.org/docs/python/api/datatypes.html#factory-functions), so I don't think we're guaranteed on equality / consistent hash between two Arrow data types.

That said, I'm not sure what alternatives we have. We could have a function with a whole bunch of if statements calling the type checking functions: https://arrow.apache.org/docs/python/api/datatypes.html#type-checking but that sounds dirty and slow (though more guaranteed to be correct). For pandas, I use the string name of the dtype, but I'm not seeing an equivalent for Arrow.

[plamut]

Each pyarrow type has a numeric ID that we can use instead. It's true that we don't know if these IDs are going to stay the same forever, but the map is built dynamically, and the IDs are consistent within a particular pyarrow version.

What do you think?

Edit: Well, for integers, for example, we assume pyarrow.int64, but an integer in arrow could also be of another type such as pyarrow.int32, and we need to map that to BigQuery's INT64, too.

[plamut]

I extended the types map in a provisional new commit, feel free to scrutinize. :)

[tswast]

Probably best if we make a function for Arrow type + field name -> BQ field now, since we know we'll want to handle repeated / nested fields at some point.

[plamut]

@tswast Updated the code and extended the tests.

Also, I wonder if we should amend our return type to include the converted arrays?

Looking at the code, that would require changing the signatures of the following Pandas helpers:

• currate_schema() - return type,
• dataframe_to_bq_schema() - return type,
• dataframe_to_parquet() and dataframe_to_arrow() - additional parameter holding any dataframe columns already converted pyarrow arrays.

Returning (schema, Optional[arrow_arrays]) does not feel like a clean design, and these changes would be added for a use case that we discourage (and will likely deprecate in the future), i.e. relying on implicit schema detection. And even that only for loading the data into a new table when mode != WRITE_TRUNCATE.

Unless there are serious performance problems reported, we are probably better off promoting the explicit schema usage IMO, and discouraging implicit schemas.