Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

openlineage: add opt-in option #37725

Merged
merged 2 commits into from
Mar 26, 2024
Merged

Conversation

JDarDagran
Copy link
Contributor

Problem statement

Currently you can only disable OpenLineage for all DAGs and tasks or selectively disable by operator class. Users report lack of possibility to do so on DAG/task level instead while opting in.

Proposed solution

Add opt_in option as well as enable/disable_lineage methods that work both on DAG and task objects.
Internally it passes constant parameter (Airflow's airflow.models.Param) that is next checked in OpenLineage listener.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@JDarDagran JDarDagran force-pushed the openlineage/opt-in branch 2 times, most recently from aa139e6 to 6472b9f Compare March 8, 2024 14:06
Copy link
Contributor

@mobuchowski mobuchowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just address one @kacpermuda comment before merging.

by default and enable it selectively.

Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
@mobuchowski mobuchowski merged commit 714a933 into apache:main Mar 26, 2024
47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants