-
Notifications
You must be signed in to change notification settings - Fork 93
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(csharp/src/Drivers/BigQuery): fix support for large results #1507
Conversation
davidhcoe
commented
Feb 1, 2024
- Adds value for a DestinationTable, which is needed for large results
- Removes old Arrow references in test projects
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems unfortunate for the caller to have to handle this. How does it work via the ODBC or JDBC drivers? Do the callers similarly have to know that they're fetching large data? Does the driver catch that error and then automatically create a temporary table? (And if so, how does the cleanup happen?)
I don't think it's necessarily a problem to commit the current logic and enable a new scenario, but this doesn't seem like the right longer-term approach.
@@ -1014,6 +1032,10 @@ public override AdbcStatement CreateStatement() | |||
{ | |||
options[keyValuePair.Key] = keyValuePair.Value; | |||
} | |||
if(keyValuePair.Key == BigQueryParameters.LargeResultsDestinationTable) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: space before paren to match the other tests
Unrelated to this specific change, does the ADBC spec say anything about case-sensitivity of options? For ODBC/JDBC/ADO.NET, connection string keys are usually case-insensitive.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
~it's up to the driver, but mostly we've been case sensitive so far
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
resolved nit
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
~it's up to the driver, but mostly we've been case sensitive so far
This seems like something we'd want to provide guidance around to improve standardization e.g. "case-sensitive but consistently lower-case keys"
string datasetId = string.Empty; | ||
string tableId = string.Empty; | ||
|
||
List<string> segments = destinationTable.Split('.').Where(x => x.Length > 0).ToList(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think "a....b....c" should be allowed.
In principle, should this follow BigQuery identifier rules for e.g. escaping? Is there a function in the Google libraries to parse table names.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see a method to parse the table names.
BigQuery separates the full table names by periods as well, ie,
SELECT FROM pq-test-398321.AdbcTests.adbc_all_types
LIMIT 1000
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I didn't do a great job explaining this feedback. What I meant was that this test would allow me to pass a..b..c (that is a dot dot be dot dot c) and have it be treated the same way as a.b.c (a dot b dot c) and that seems wrong.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, fixed
@@ -185,6 +186,33 @@ static IArrowReader ReadChunk(BigQueryReadClient readClient, string streamName) | |||
{ | |||
options.AllowLargeResults = true ? keyValuePair.Value.ToLower().Equals("true") : false; | |||
} | |||
if(keyValuePair.Key == BigQueryParameters.LargeResultsDestinationTable) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: space before paren
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
resolved nit