Skip to content

Commit

Permalink
Support new error messages. (#226)
Browse files Browse the repository at this point in the history
### Description

Supports new error messages.

In `SparkAdapter.get_columns_in_relation`, it checks the error message when the specified table or view doesn't exist:

https://github.com/dbt-labs/dbt-spark/blob/c87b6b2c48bcefb0ce52cd64984d3129d6f14ea0/dbt/adapters/spark/impl.py#L223

but, Spark will change the error message in the future release (apache/spark#37887), which causes the function to raise the `dbt.exceptions.RuntimeException` instead of returning an empty list.

The function should also check whether the error message contains `[TABLE_OR_VIEW_NOT_FOUND]` or not.

This will be reverted once dbt-labs/dbt-spark#515 is resolved.
  • Loading branch information
ueshin committed Nov 15, 2022
1 parent a26872c commit 8e420cc
Showing 1 changed file with 29 additions and 0 deletions.
29 changes: 29 additions & 0 deletions dbt/adapters/databricks/impl.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
from dbt.adapters.base.relation import BaseRelation
from dbt.adapters.spark.impl import (
SparkAdapter,
GET_COLUMNS_IN_RELATION_RAW_MACRO_NAME,
KEY_TABLE_OWNER,
KEY_TABLE_STATISTICS,
LIST_RELATIONS_MACRO_NAME,
Expand Down Expand Up @@ -180,6 +181,34 @@ def parse_describe_extended( # type: ignore[override]
for idx, column in enumerate(rows)
]

def get_columns_in_relation( # type: ignore[override]
self, relation: DatabricksRelation
) -> List[DatabricksColumn]:
columns = []
try:
rows: List[Row] = self.execute_macro(
GET_COLUMNS_IN_RELATION_RAW_MACRO_NAME, kwargs={"relation": relation}
)
columns = self.parse_describe_extended(relation, rows)
except dbt.exceptions.RuntimeException as e:
# spark would throw error when table doesn't exist, where other
# CDW would just return and empty list, normalizing the behavior here
errmsg = getattr(e, "msg", "")
if any(
msg in errmsg
for msg in (
"[TABLE_OR_VIEW_NOT_FOUND]",
"Table or view not found",
"NoSuchTableException",
)
):
pass
else:
raise e

# strip hudi metadata columns.
return [x for x in columns if x.name not in self.HUDI_METADATA_COLUMNS]

def parse_columns_from_information( # type: ignore[override]
self, relation: DatabricksRelation
) -> List[DatabricksColumn]:
Expand Down

0 comments on commit 8e420cc

Please sign in to comment.