-
Notifications
You must be signed in to change notification settings - Fork 14.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Common SQLCheckOperators Various Functionality Update #25164
Common SQLCheckOperators Various Functionality Update #25164
Commits on Jul 18, 2022
-
Add batching to SQL Check Operators
Commit adds a WHERE clause to the sql statement that allows for arbitrary batching in a given table.
Configuration menu - View commit details
-
Copy full SHA for dc01866 - Browse repository at this point
Copy the full SHA dc01866View commit details -
Fix bug with multiple table checks
When multiple table checks are given to the SQLTableCheckOperator and at least one is not a fully aggregate statement, a GROUP BY clause was previously needed. This commit updates the operator to use the get_pandas_df() method instead of _get_first() to return a pandas dataframe object that contains the check names and check results from the new style of query. The new style of query uses UNION ALL to run each test as its own SELECT statement, bypassing the need to do a GROUP BY.
Configuration menu - View commit details
-
Copy full SHA for 7c25227 - Browse repository at this point
Copy the full SHA 7c25227View commit details -
Changed name of method from _get_failed_tests to _get_failed_checks to better match naming, and updated logic of the method to include an optional column param. The query in the column check operator is removed from the failed test exception message, as it was only ever showing the last query, instead of the relevant one(s). This is replaced by the column, which will be more useful in debugging.
Configuration menu - View commit details
-
Copy full SHA for 2a3df61 - Browse repository at this point
Copy the full SHA 2a3df61View commit details
Commits on Jul 21, 2022
-
Add table alias to SQLTableCheckOperator query
Without a table alias, the query does not run on Postgres and other databases. The alias is arbitrary and used only for proper query execution.
Configuration menu - View commit details
-
Copy full SHA for 66922f0 - Browse repository at this point
Copy the full SHA 66922f0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 554e8ba - Browse repository at this point
Copy the full SHA 554e8baView commit details -
Add batching to SQL Check Operators
Commit adds a WHERE clause to the sql statement that allows for arbitrary batching in a given table.
Configuration menu - View commit details
-
Copy full SHA for cf90083 - Browse repository at this point
Copy the full SHA cf90083View commit details -
Fix bug with multiple table checks
When multiple table checks are given to the SQLTableCheckOperator and at least one is not a fully aggregate statement, a GROUP BY clause was previously needed. This commit updates the operator to use the get_pandas_df() method instead of _get_first() to return a pandas dataframe object that contains the check names and check results from the new style of query. The new style of query uses UNION ALL to run each test as its own SELECT statement, bypassing the need to do a GROUP BY.
Configuration menu - View commit details
-
Copy full SHA for 7c20bf6 - Browse repository at this point
Copy the full SHA 7c20bf6View commit details -
Changed name of method from _get_failed_tests to _get_failed_checks to better match naming, and updated logic of the method to include an optional column param. The query in the column check operator is removed from the failed test exception message, as it was only ever showing the last query, instead of the relevant one(s). This is replaced by the column, which will be more useful in debugging.
Configuration menu - View commit details
-
Copy full SHA for d364e96 - Browse repository at this point
Copy the full SHA d364e96View commit details -
Add table alias to SQLTableCheckOperator query
Without a table alias, the query does not run on Postgres and other databases. The alias is arbitrary and used only for proper query execution.
Configuration menu - View commit details
-
Copy full SHA for 3c300e7 - Browse repository at this point
Copy the full SHA 3c300e7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5645b5d - Browse repository at this point
Copy the full SHA 5645b5dView commit details -
Merge branch 'sql_check_operators_various_functionality_update' of gi…
…thub.com:denimalpaca/airflow into sql_check_operators_various_functionality_update
Configuration menu - View commit details
-
Copy full SHA for 98a28c3 - Browse repository at this point
Copy the full SHA 98a28c3View commit details -
Move alias to proper query build statement
The table alias should be in the self.sql query build statement as that is where the table it needs to alias is defined.
Configuration menu - View commit details
-
Copy full SHA for ee697e3 - Browse repository at this point
Copy the full SHA ee697e3View commit details -
Add batching to SQL Check Operators
Commit adds a WHERE clause to the sql statement that allows for arbitrary batching in a given table.
Configuration menu - View commit details
-
Copy full SHA for 1754fdc - Browse repository at this point
Copy the full SHA 1754fdcView commit details -
Fix bug with multiple table checks
When multiple table checks are given to the SQLTableCheckOperator and at least one is not a fully aggregate statement, a GROUP BY clause was previously needed. This commit updates the operator to use the get_pandas_df() method instead of _get_first() to return a pandas dataframe object that contains the check names and check results from the new style of query. The new style of query uses UNION ALL to run each test as its own SELECT statement, bypassing the need to do a GROUP BY.
Configuration menu - View commit details
-
Copy full SHA for 31d0e0e - Browse repository at this point
Copy the full SHA 31d0e0eView commit details -
Changed name of method from _get_failed_tests to _get_failed_checks to better match naming, and updated logic of the method to include an optional column param. The query in the column check operator is removed from the failed test exception message, as it was only ever showing the last query, instead of the relevant one(s). This is replaced by the column, which will be more useful in debugging.
Configuration menu - View commit details
-
Copy full SHA for bc4140e - Browse repository at this point
Copy the full SHA bc4140eView commit details -
Add table alias to SQLTableCheckOperator query
Without a table alias, the query does not run on Postgres and other databases. The alias is arbitrary and used only for proper query execution.
Configuration menu - View commit details
-
Copy full SHA for 24ce964 - Browse repository at this point
Copy the full SHA 24ce964View commit details -
Configuration menu - View commit details
-
Copy full SHA for 987f6c2 - Browse repository at this point
Copy the full SHA 987f6c2View commit details -
Merge branch 'sql_check_operators_various_functionality_update' of gi…
…thub.com:denimalpaca/airflow into sql_check_operators_various_functionality_update
Configuration menu - View commit details
-
Copy full SHA for a27fc2c - Browse repository at this point
Copy the full SHA a27fc2cView commit details -
Bug fixes and updates to test and operator
Fixed bug in test where the dataframe column names did not match the operator's expected dataframe column names. Added more info to the SQLColumnCheckOperator's batch arg. Fixed the location of table aliasing in SQLTableCheckOperator.
Configuration menu - View commit details
-
Copy full SHA for 404eef5 - Browse repository at this point
Copy the full SHA 404eef5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 01bfb2f - Browse repository at this point
Copy the full SHA 01bfb2fView commit details -
Rename parameter batch to partition_clause
Gives a clearer name to the parameter and adds templating to the SQLTableCheckOperator.
Configuration menu - View commit details
-
Copy full SHA for 75d59d1 - Browse repository at this point
Copy the full SHA 75d59d1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0388ac4 - Browse repository at this point
Copy the full SHA 0388ac4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 719a830 - Browse repository at this point
Copy the full SHA 719a830View commit details