Column lineage graph endpoint #2124
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pawel-big-lebowski
force-pushed
the
column-lineage-graph-endpoint
branch
from
b2478de
to
75798d1
Compare
Codecov Report
@@ Coverage Diff @@
## main #2124 +/- ##
============================================
+ Coverage 75.82% 76.33% +0.51%
- Complexity 1063 1099 +36
============================================
Files 209 214 +5
Lines 5013 5139 +126
Branches 403 407 +4
============================================
+ Hits 3801 3923 +122
+ Misses 763 762 -1
- Partials 449 454 +5
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
pawel-big-lebowski
force-pushed
the
column-lineage-graph-endpoint
branch
3 times, most recently
from
d47f846
to
f79a812
Compare
pawel-big-lebowski
force-pushed
the
column-lineage-graph-endpoint
branch
2 times, most recently
from
ad91189
to
5260505
Compare
pawel-big-lebowski
force-pushed
the
add-column-level-lineage
branch
from
e18c13e
to
21dac22
Compare
pawel-big-lebowski
force-pushed
the
column-lineage-graph-endpoint
branch
from
5260505
to
e824c61
Compare
pawel-big-lebowski
force-pushed
the
column-lineage-graph-endpoint
branch
from
e824c61
to
39b3add
Compare
pawel-big-lebowski
changed the title
pawel-big-lebowski
force-pushed
the
column-lineage-graph-endpoint
branch
from
39b3add
to
a8a5a41
Compare
| @@ -88,4 +95,59 @@ void doUpsertColumnLineageRow( | |||
| }, | |||
| value = "values") | |||
| List<ColumnLineageRow> rows); | |||
|
|
|||
| @SqlQuery( | |||
There was a problem hiding this comment.
Most important piece of the PR: recursive query to extract column-lineage graph.
Only column_lineage table is used and joined to obtained graph nodes.
Other tables are only used to enrich found nodes.
wslulciuc
reviewed
Sep 30, 2022
wslulciuc
reviewed
Sep 30, 2022
pawel-big-lebowski
force-pushed
the
column-lineage-graph-endpoint
branch
from
a8a5a41
to
c65ecdd
Compare
| @@ -533,8 +533,8 @@ public static class ColumnLineageOutputColumn extends BaseJsonModel { | |||
| @ToString | |||
| public static class ColumnLineageInputField extends BaseJsonModel { | |||
|
|
|||
| @NotNull private String datasetNamespace; | |||
There was a problem hiding this comment.
Fix to previous PR to align with Openlineage spec:
https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/ColumnLineageDatasetFacet.json
pawel-big-lebowski
force-pushed
the
column-lineage-graph-endpoint
branch
from
c65ecdd
to
791e1bf
Compare
pawel-big-lebowski
force-pushed
the
column-lineage-graph-endpoint
branch
from
791e1bf
to
b79d712
Compare
Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>
pawel-big-lebowski
force-pushed
the
column-lineage-graph-endpoint
branch
from
b79d712
to
8e66689
Compare
mobuchowski
approved these changes
Oct 7, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Pawel Leszczynski leszczynski.pawel@gmail.com
Problem
PR #2096 allows storing in database column-lineage information from the events. In this PR we expose column lineage through a graph endpoint according to the proposal (https://github.com/MarquezProject/marquez/blob/main/proposals/2045-column-lineage-endpoint.md)
Closes: #2114
Solution
NodeTypeDATASET_FIELDis added,column-lineageendpoint returns serializedLineageobjects similar to a currently existinglineageendpoint,Checklist
CHANGELOG.mdwith details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary).sqldatabase schema migration according to Flyway's naming convention (if relevant)