feat: build column lineage using sqlglot #46
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* feat: add notes on sap hana queries and file-based lineage yaml file * feat: add new datasource and annotations * fix: added logger.debug, removed references to internal sap hana dev, fixed white spaces * added new docker compose * feat: add integration tests * fixing the check_golden_file * feat: add integration tests * feat: after task fix * feat: adjust test for integration * feat: adjust test for integration, edited pypro and test_hana * feat: adjusted the decorators * Update pyproject.toml * Update pyproject.toml * adding new test and headers * changes to test_hana and other files * added headers to test_helpers and other files * Update ingestion.py to remove the constraints for SYS * Update datahub_sap_hana/ingestion.py Co-authored-by: Lucas Roesler <roesler.lucas@gmail.com> * Update datahub_sap_hana/ingestion.py Co-authored-by: Lucas Roesler <roesler.lucas@gmail.com> * remove the execution block in ingestion.py * Update hana_recipe.yaml to change sink type to console * Update README.md * removed query * Update hana_to_file.yml * Update the testing section * Update pyproject.toml Co-authored-by: Lucas Roesler <roesler.lucas@gmail.com> * Update notes.md * Update notes.md * Update docker-compose.yml * Update test_hana.py * Update docker-compose.yml * Update tests/README.md Co-authored-by: Lucas Roesler <roesler.lucas@gmail.com> * Update tests/docker-compose.yml Co-authored-by: Lucas Roesler <roesler.lucas@gmail.com> * changes to filenames, deleted notes.md * changes to filenames * changed file names * ran task fix * added deepdiff --------- Co-authored-by: kay_alave <kay.alave@gmail.com> Co-authored-by: Lucas Roesler <roesler.lucas@gmail.com>
LucasRoesler
requested changes
Jun 23, 2023
LucasRoesler
requested changes
Jun 23, 2023
Signed-off-by: Lucas Roesler <roesler.lucas@gmail.com>
LucasRoesler
requested changes
Jun 26, 2023
Signed-off-by: Lucas Roesler <roesler.lucas@gmail.com>
This adds two major features 1. the inspector is now described as a protocol, which allows us to define and use an inspector with caching 2. using this caching, we now ensure that the column names used in the lineage use casing matching the value returned from the insepctor. Signed-off-by: Lucas Roesler <roesler.lucas@gmail.com>
Signed-off-by: Lucas Roesler <roesler.lucas@gmail.com>
LucasRoesler
approved these changes
Jun 30, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This extracts and build column level lineage data from a SAP Hana db using sqlglot.
The column lineage is built from sqlglot nodes, which contains upstream and downstream data, before being transformed into workunits to be sent to datahub.
At this point, code is still WIP, with testing and experimentation for edge cases still ongoing.