-
Notifications
You must be signed in to change notification settings - Fork 302
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BigQuery to_dataframe() ArrowNotImplementedError #63
Comments
@jvschoen Thanks for the report (I moved the issue from the old repository to here). Which Also, could you share the schema of the source table and the query that fetches data from it? That could also be useful for diagnosing the issue, thanks! |
We may want to consider using Fletcher for struct and array BigQuery data types, since pandas needs to use (slow) Python objects in these cases. |
This is the pip show on my AI notebook
Here's pip show of the google.cloud.bigquery
Here's output of the table schema in a google sheets: I think it has something to do with |
If you look at that sheet and I try to just query the face_data repeated record:
|
The underlying Arrow Issue has been fixed on master and will be available with the next release (https://issues.apache.org/jira/browse/ARROW-7872) |
@emkornfield This sounds good, thanks! Looking forward to trying it out. |
Also would using avro be a short term workaround for this? |
@emkornfield If a release is indeed planned some time in the next few weeks, I think that's soon enough, considering the fact that this limitation has already been around for awhile. I checked the test case from the fix commit, and while it fails in FWIW, I did encounter quite a few problems with dependencies and versions, but eventually managed to compile
import pandas
import pyarrow
series = pandas.Series([1, 2, 3], name="foo")
inner_type = pyarrow.int64()
arrow_type = pyarrow.list_(inner_type)
pyarrow.ListArray.from_pandas(series, type=arrow_type)
# ArrowNotImplementedError: NumPyConverter doesn't implement <list<item: int64>> conversion.
Update: Actually, this "bug" above was an error in my test, there was a mismatch between the schema and the test data, nevermind. |
I think there are now nightly builds getting published. You should be able to use
|
It indeed works, thanks. I was able to install and run a development version with Python 3.7. |
I know this is delayed, but that pyarrow update worked for me. I'm closing this issue. |
I'm working with the Google Vision API and trying to do some analysis in pandas. When I try to get the output into pandas there is an issue and I get the NotImplementedError Below.
Environment details
General, Core, and Other are also allowed as types
Steps to reproduce
Code example
Stack trace
The text was updated successfully, but these errors were encountered: