Skip to content

Commit

Permalink
Add openlineage_dataset_uri in databricks db (#1919)
Browse files Browse the repository at this point in the history
currently, integration tests fail with below error so adding
`openlineage_dataset_uri` in databrick delta db

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/openlineage/airflow/extractors/manager.py", line 42, in extract_metadata
    task_metadata = extractor.extract_on_complete(task_instance)
  File "/usr/local/lib/python3.9/site-packages/openlineage/airflow/extractors/base.py", line 112, in extract_on_complete
    return self._get_openlineage_facets(
  File "/usr/local/lib/python3.9/site-packages/openlineage/airflow/extractors/base.py", line 120, in _get_openlineage_facets
    facets: OperatorLineage = get_facets_method(*args)
  File "/home/astro/.local/lib/python3.9/site-packages/astro/sql/operators/load_file.py", line 323, in get_openlineage_facets_on_complete
    name=self.output_table.name, uri=self.output_table.openlineage_dataset_uri()
  File "/home/astro/.local/lib/python3.9/site-packages/astro/table.py", line 170, in openlineage_dataset_uri
    return f"{database.openlineage_dataset_uri(table=self)}"
  File "/home/astro/.local/lib/python3.9/site-packages/astro/databases/base.py", line 894, in openlineage_dataset_uri
    raise NotImplementedError
NotImplementedError
[2023-05-04, 00:35:01 UTC] {standard_task_runner.py:100} ERROR - Failed to execute job 44345 for task load_file__23 (Databricks job failed. Job info ***
```
  • Loading branch information
pankajastro authored May 4, 2023
1 parent ccf277d commit 4c88d5e
Showing 1 changed file with 3 additions and 0 deletions.
3 changes: 3 additions & 0 deletions python-sdk/src/astro/databases/databricks/delta.py
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,9 @@ def openlineage_dataset_name(self, table: BaseTable) -> str:
def openlineage_dataset_namespace(self) -> str:
return ""

def openlineage_dataset_uri(self, table: BaseTable) -> str:
return ""

def create_table_from_select_statement(
self,
statement: str,
Expand Down

0 comments on commit 4c88d5e

Please sign in to comment.