Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fill run state of the parent run when it is created by child run #2296

Merged
merged 2 commits into from
Dec 8, 2022

Conversation

fm100
Copy link
Collaborator

@fm100 fm100 commented Dec 8, 2022

Signed-off-by: Minkyu Park minkyu.park.200@gmail.com

Problem

Airflow integration is not sending the run event for the DAG but it is enclosed as a parent facet of task runs events, and when the marquez receives OpenLineage event with parent facet, it creates parent job and run if they don't exist. However, the created parent run is missing its run state, and it makes a little weird when getting the runs for the DAG job. For example when the marquez receives the START event, the the parent run is being created and its status is RUNNING but it does not have a startedAt information.

Solution

This PR adds a run state to the parent run at its creation time, honoring the way that it determines the state of the parent run. If the parent run is created with the START event, its run state becomes RUNNING and it adds started_at to the database. If the parent run is created with the COMPLETE event, its run state becomes COMPLETE and it adds ended_at to the database.

Note: All database schema changes require discussion. Please link the issue for context.

Checklist

  • You've signed-off your work
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
  • You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
  • You've included a header in any source code files (if relevant)

Signed-off-by: Minkyu Park <minkyu.park.200@gmail.com>
@fm100 fm100 requested a review from wslulciuc December 8, 2022 00:44
@boring-cyborg boring-cyborg bot added the api API layer changes label Dec 8, 2022
Copy link
Member

@wslulciuc wslulciuc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 💯 🥇

@wslulciuc wslulciuc enabled auto-merge (squash) December 8, 2022 10:49
@wslulciuc wslulciuc merged commit 1bbcb6f into MarquezProject:main Dec 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api API layer changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants