feat(sqllab): add latest partition support for BigQuery #30760
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mistercrunch
commented
Oct 31, 2024
| ) | ||
| assert result == expected_result | ||
|
|
||
| def test_get_indexes(self): |
There was a problem hiding this comment.
turns out BigQuery's implementation of get_indexes is a no-op -> https://github.com/googleapis/python-bigquery-sqlalchemy/blob/150143ce27c02f1cd711b8df4ffe587a98ada288/sqlalchemy_bigquery/base.py#L1266-L1268
dosubot
bot
added
data:connect:googlebigquery
rusackas
reviewed
Nov 11, 2024
| @@ -40,6 +40,7 @@ | |||
| EXAMPLES_HOST = os.getenv("EXAMPLES_HOST") | |||
| EXAMPLES_PORT = os.getenv("EXAMPLES_PORT") | |||
| EXAMPLES_DB = os.getenv("EXAMPLES_DB") | |||
| SHOW_STACKTRACE = True | |||
There was a problem hiding this comment.
Was this intended to be part of the PR, or just for local/temporary dev purposes?
There was a problem hiding this comment.
Let me remove that line, now thinking it may make sense to deprecate this flag as it's not evenly implemented.
Adding db_engine_spec-related features that enables SQL Lab to show the latest partition when using time partitioning in BigQuery as well as applying a WHERE clause on the latest partition by default when fetching the sample dataset. Turns out that `SELECT * FROM {{...}} LIMIT {n}` can be costly against large tables in BigQuery as it results in a full table scan.
mistercrunch
force-pushed
the
bigquery_partitions
branch
from
14e90ba
to
97f0bfa
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adding db_engine_spec-related features that enables SQL Lab to show the latest partition when using time partitioning in BigQuery as well as applying a WHERE clause on the latest partition by default when fetching the sample dataset. Turns out that
SELECT * FROM {{...}} LIMIT {n}can be costly against large tables in BigQuery as it results in a full table scan.closes #17299