Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issue reported by the community in slack with lineage in Cosmos 1.1 #526

Merged
merged 8 commits into from
Sep 14, 2023

Conversation

tatiana
Copy link
Collaborator

@tatiana tatiana commented Sep 11, 2023

Fix the following issue, that was reported in slack by a user named Sai:

"After upgrading to cosmos 1.1, I see the below error in Airflow logs. I don't have any OpenLineage integrations. Is there a way to stop emitting this error?"

[2023-09-08, 22:28:07 UTC] {taskinstance.py:1318} INFO - Marking task as SUCCESS. dag_id=high_freq_core_dbt, task_id=dbt_front.daily_company_user_metrics.daily_company_user_metrics_run, execution_date=20230908T183000, start_date=20230908T222730, end_date=20230908T222807
[2023-09-08, 22:28:07 UTC] {manager.py:61} ERROR - Failed to extract metadata 'TaskInstance' object has no attribute 'openlineage_events_completes' task_type=DbtRunLocalOperator airflow_dag_id=high_freq_core_dbt task_id=dbt_front.daily_company_user_metrics.daily_company_user_metrics_run airflow_run_id=scheduled__2023-09-08T18:30:00+00:00 
Traceback (most recent call last):
  File "/home/airflow/python-venvs/data-pipelines/lib/python3.10/site-packages/openlineage/airflow/extractors/manager.py", line 45, in extract_metadata
    task_metadata = extractor.extract_on_complete(task_instance)
  File "/home/airflow/python-venvs/data-pipelines/lib/python3.10/site-packages/openlineage/airflow/extractors/base.py", line 116, in extract_on_complete
    return self._get_openlineage_facets(
  File "/home/airflow/python-venvs/data-pipelines/lib/python3.10/site-packages/openlineage/airflow/extractors/base.py", line 124, in _get_openlineage_facets
    facets: OperatorLineage = get_facets_method(*args)
  File "/home/airflow/python-venvs/data-pipelines/lib/python3.10/site-packages/cosmos/operators/local.py", line 318, in get_openlineage_facets_on_complete
    for completed in task_instance.openlineage_events_completes:
AttributeError: 'TaskInstance' object has no attribute 'openlineage_events_completes'

The issue was reported to happen using:

  • Airflow version: v2.5.1
  • dbt version: 1.5.1
  • without any intentional setup of OpenLineage

Since the issue was reported, I learned that get_openlineage_facets_on_complete is called even if execute fails because get_openlineage_facets_on_failure is not currently implemented.

Closes: #530 (Rust dependency issue reported)

@tatiana tatiana requested a review from a team as a code owner September 11, 2023 14:02
@tatiana tatiana requested a review from a team September 11, 2023 14:02
@netlify
Copy link

netlify bot commented Sep 11, 2023

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name Link
🔨 Latest commit f322db0
🔍 Latest deploy log https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/6502e32595aa2f00084601ae

@tatiana tatiana temporarily deployed to internal September 11, 2023 14:02 — with GitHub Actions Inactive
@tatiana tatiana temporarily deployed to internal September 11, 2023 15:30 — with GitHub Actions Inactive
@tatiana tatiana temporarily deployed to internal September 11, 2023 15:46 — with GitHub Actions Inactive
@tatiana tatiana temporarily deployed to internal September 11, 2023 20:53 — with GitHub Actions Inactive
@tatiana tatiana temporarily deployed to internal September 11, 2023 21:31 — with GitHub Actions Inactive
@codecov
Copy link

codecov bot commented Sep 11, 2023

Codecov Report

Patch coverage: 96.15% and no project coverage change.

Comparison is base (bcf7714) 92.37% compared to head (f322db0) 92.37%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #526   +/-   ##
=======================================
  Coverage   92.37%   92.37%           
=======================================
  Files          49       49           
  Lines        1927     1941   +14     
=======================================
+ Hits         1780     1793   +13     
- Misses        147      148    +1     
Files Changed Coverage Δ
cosmos/operators/local.py 88.37% <96.15%> (+0.21%) ⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tatiana tatiana temporarily deployed to internal September 11, 2023 22:33 — with GitHub Actions Inactive
@tatiana tatiana temporarily deployed to internal September 12, 2023 09:34 — with GitHub Actions Inactive
@tatiana tatiana temporarily deployed to internal September 12, 2023 09:36 — with GitHub Actions Inactive
@tatiana tatiana temporarily deployed to internal September 12, 2023 09:38 — with GitHub Actions Inactive
cosmos/operators/local.py Outdated Show resolved Hide resolved
cosmos/operators/local.py Show resolved Hide resolved
@tatiana tatiana added this to the 1.1.1 milestone Sep 12, 2023
@tatiana tatiana temporarily deployed to internal September 13, 2023 09:47 — with GitHub Actions Inactive
@tatiana tatiana temporarily deployed to internal September 13, 2023 10:27 — with GitHub Actions Inactive
Copy link
Contributor

@harels harels left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! thanks @tatiana

cosmos/operators/local.py Outdated Show resolved Hide resolved
cosmos/operators/local.py Outdated Show resolved Hide resolved
@tatiana tatiana temporarily deployed to internal September 14, 2023 10:20 — with GitHub Actions Inactive
@tatiana tatiana temporarily deployed to internal September 14, 2023 10:41 — with GitHub Actions Inactive
@tatiana tatiana merged commit 665a9b8 into main Sep 14, 2023
40 checks passed
@tatiana tatiana deleted the community-issue branch September 14, 2023 13:57
tatiana added a commit that referenced this pull request Sep 14, 2023
Bug fixes

* Fix attempt of emitting OpenLineage events if task execution fails by @tatiana in #526
* Fix Rust dependency for Windows users by @tatiana in #526
* Fix DbtRunOperationLocalOperator missing flags by @tatiana in #529
* Fix DbtRunLocalOperator to support the full refresh argument by @tatiana in #529
* Remove redundant prefix of task names when test_behavior = TestBehavior.AFTER_EACH by @binhnq94 in #524
* Fix rendering vars in `DbtModel` when using `LoadMode.CUSTOM` by @dojinkimm in #502

Others

* Docs: add `documentation comparing Airflow and dbt concepts <https://astronomer.github.io/astronomer-cosmos/getting_started/dbt-airflow-concepts.html>`_ by @tatiana in #523.
* Update PyPI project links by @tatiana in #528
* pre-commit updates
tatiana added a commit that referenced this pull request Sep 14, 2023
Bug fixes

* Fix attempt of emitting OpenLineage events if task execution fails by @tatiana in #526
* Fix Rust dependency for Windows users by @tatiana in #526
* Fix DbtRunOperationLocalOperator missing flags by @tatiana in #529
* Fix DbtRunLocalOperator to support the full refresh argument by @tatiana in #529
* Remove redundant prefix of task names when test_behavior = TestBehavior.AFTER_EACH by @binhnq94 in #524
* Fix rendering vars in `DbtModel` when using `LoadMode.CUSTOM` by @dojinkimm in #502

Others

* Docs: add `documentation comparing Airflow and dbt concepts <https://astronomer.github.io/astronomer-cosmos/getting_started/dbt-airflow-concepts.html>`_ by @tatiana in #523.
* Update PyPI project links by @tatiana in #528
* pre-commit updates
@tatiana tatiana mentioned this pull request Sep 14, 2023
harels pushed a commit that referenced this pull request Sep 14, 2023
Bug fixes

* Remove redundant prefix of task names when test_behavior =
TestBehavior.AFTER_EACH by @binhnq94 in #524
* Fix rendering vars in `DbtModel` when using `LoadMode.CUSTOM` by
@dojinkimm in #502
* Fix attempt of emitting OpenLineage events if task execution fails by
@tatiana in #526
* Fix Rust dependency for Windows users by @tatiana in #526
* Fix DbtRunOperationLocalOperator missing flags by @tatiana in #529
* Fix DbtRunLocalOperator to support the full refresh argument by
@tatiana in #529

Others

* Docs: add `documentation comparing Airflow and dbt concepts
<https://astronomer.github.io/astronomer-cosmos/getting_started/dbt-airflow-concepts.html>`_
by @tatiana in #523.
* Update PyPI project links by @tatiana in #528
* pre-commit updates
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Remove rust dependency from Cosmos
3 participants