Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CT-1503] [Feature] Support new error messages in the future Spark. #515

Closed
3 tasks done
ueshin opened this issue Nov 14, 2022 · 5 comments · Fixed by #520
Closed
3 tasks done

[CT-1503] [Feature] Support new error messages in the future Spark. #515

ueshin opened this issue Nov 14, 2022 · 5 comments · Fixed by #520
Labels
enhancement New feature or request

Comments

@ueshin
Copy link
Contributor

ueshin commented Nov 14, 2022 •
edited
Loading

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt-spark functionality, rather than a Big Idea better suited to a discussion

Describe the feature

In SparkAdapter.get_columns_in_relation, it checks the error message when the specified table or view doesn't exist:

dbt-spark/dbt/adapters/spark/impl.py

Line 223 in c87b6b2

if "Table or view not found" in errmsg or "NoSuchTableException" in errmsg:

but, Spark will change the error message in the future release (apache/spark#37887), which causes the function to raise the dbt.exceptions.RuntimeException instead of returning an empty list.

The function should also check whether the error message contains [TABLE_OR_VIEW_NOT_FOUND] or not.

Describe alternatives you've considered

No response

Who will this benefit?

No response

Are you interested in contributing this feature?

No response

Anything else?

The feature should be backported.
@ueshin ueshin added enhancement New feature or request triage labels Nov 14, 2022
@github-actions github-actions bot changed the title [Feature] Support new error messages in the future Spark. [CT-1503] [Feature] Support new error messages in the future Spark. Nov 14, 2022
@ueshin ueshin mentioned this issue Nov 14, 2022
Merged
ueshin added a commit to databricks/dbt-databricks that referenced this issue Nov 15, 2022
@ueshin
Support new error messages. (#226)
09100c8 
### Description

Supports new error messages.

In `SparkAdapter.get_columns_in_relation`, it checks the error message when the specified table or view doesn't exist:

https://github.com/dbt-labs/dbt-spark/blob/c87b6b2c48bcefb0ce52cd64984d3129d6f14ea0/dbt/adapters/spark/impl.py#L223

but, Spark will change the error message in the future release (apache/spark#37887), which causes the function to raise the `dbt.exceptions.RuntimeException` instead of returning an empty list.

The function should also check whether the error message contains `[TABLE_OR_VIEW_NOT_FOUND]` or not.

This will be reverted once dbt-labs/dbt-spark#515 is resolved.
ueshin added a commit to databricks/dbt-databricks that referenced this issue Nov 15, 2022
@ueshin
Support new error messages. (#226)
8e420cc 
### Description

Supports new error messages.

In `SparkAdapter.get_columns_in_relation`, it checks the error message when the specified table or view doesn't exist:

https://github.com/dbt-labs/dbt-spark/blob/c87b6b2c48bcefb0ce52cd64984d3129d6f14ea0/dbt/adapters/spark/impl.py#L223

but, Spark will change the error message in the future release (apache/spark#37887), which causes the function to raise the `dbt.exceptions.RuntimeException` instead of returning an empty list.

The function should also check whether the error message contains `[TABLE_OR_VIEW_NOT_FOUND]` or not.

This will be reverted once dbt-labs/dbt-spark#515 is resolved.
ueshin added a commit to databricks/dbt-databricks that referenced this issue Nov 15, 2022
@ueshin
Support new error messages. (#226)
3884386 
### Description

Supports new error messages.

In `SparkAdapter.get_columns_in_relation`, it checks the error message when the specified table or view doesn't exist:

https://github.com/dbt-labs/dbt-spark/blob/c87b6b2c48bcefb0ce52cd64984d3129d6f14ea0/dbt/adapters/spark/impl.py#L223

but, Spark will change the error message in the future release (apache/spark#37887), which causes the function to raise the `dbt.exceptions.RuntimeException` instead of returning an empty list.

The function should also check whether the error message contains `[TABLE_OR_VIEW_NOT_FOUND]` or not.

This will be reverted once dbt-labs/dbt-spark#515 is resolved.
ueshin added a commit to databricks/dbt-databricks that referenced this issue Nov 15, 2022
@ueshin
Support new error messages. (#226)
5f7a314 
### Description

Supports new error messages.

In `SparkAdapter.get_columns_in_relation`, it checks the error message when the specified table or view doesn't exist:

https://github.com/dbt-labs/dbt-spark/blob/c87b6b2c48bcefb0ce52cd64984d3129d6f14ea0/dbt/adapters/spark/impl.py#L223

but, Spark will change the error message in the future release (apache/spark#37887), which causes the function to raise the `dbt.exceptions.RuntimeException` instead of returning an empty list.

The function should also check whether the error message contains `[TABLE_OR_VIEW_NOT_FOUND]` or not.

This will be reverted once dbt-labs/dbt-spark#515 is resolved.
ueshin added a commit to databricks/dbt-databricks that referenced this issue Nov 15, 2022
@ueshin
Support new error messages. (#226)
df6abc0 
### Description

Supports new error messages.

In `SparkAdapter.get_columns_in_relation`, it checks the error message when the specified table or view doesn't exist:

https://github.com/dbt-labs/dbt-spark/blob/c87b6b2c48bcefb0ce52cd64984d3129d6f14ea0/dbt/adapters/spark/impl.py#L223

but, Spark will change the error message in the future release (apache/spark#37887), which causes the function to raise the `dbt.exceptions.RuntimeException` instead of returning an empty list.

The function should also check whether the error message contains `[TABLE_OR_VIEW_NOT_FOUND]` or not.

This will be reverted once dbt-labs/dbt-spark#515 is resolved.
ueshin added a commit to databricks/dbt-databricks that referenced this issue Nov 15, 2022
@ueshin
Support new error messages. (#226)
96e64a8 
### Description

Supports new error messages.

In `SparkAdapter.get_columns_in_relation`, it checks the error message when the specified table or view doesn't exist:

https://github.com/dbt-labs/dbt-spark/blob/c87b6b2c48bcefb0ce52cd64984d3129d6f14ea0/dbt/adapters/spark/impl.py#L223

but, Spark will change the error message in the future release (apache/spark#37887), which causes the function to raise the `dbt.exceptions.RuntimeException` instead of returning an empty list.

The function should also check whether the error message contains `[TABLE_OR_VIEW_NOT_FOUND]` or not.

This will be reverted once dbt-labs/dbt-spark#515 is resolved.
ueshin added a commit to databricks/dbt-databricks that referenced this issue Nov 15, 2022
@ueshin
Support new error messages. (#226)
f8a5a14 
### Description

Supports new error messages.

In `SparkAdapter.get_columns_in_relation`, it checks the error message when the specified table or view doesn't exist:

https://github.com/dbt-labs/dbt-spark/blob/c87b6b2c48bcefb0ce52cd64984d3129d6f14ea0/dbt/adapters/spark/impl.py#L223

but, Spark will change the error message in the future release (apache/spark#37887), which causes the function to raise the `dbt.exceptions.RuntimeException` instead of returning an empty list.

The function should also check whether the error message contains `[TABLE_OR_VIEW_NOT_FOUND]` or not.

This will be reverted once dbt-labs/dbt-spark#515 is resolved.
@ueshin ueshin mentioned this issue Nov 16, 2022
Merged
6 tasks
@jtcohen6
Copy link
Contributor

Thanks for flagging this, @ueshin!

@jtcohen6 jtcohen6 removed the triage label Nov 17, 2022
@Fleid
Copy link
Contributor

Fleid commented Nov 28, 2022

Hey @ueshin, do you know what version of Spark will receive these changes? Will it be backported to previous versions at all?

@ueshin
Copy link
Contributor Author

ueshin commented Nov 28, 2022

The error message changes will be in Spark 3.4 and also Databricks Runtime 12.0.
Those are not released yet, but DBR 12.0 will be released soon.

This was referenced Nov 29, 2022
Merged
Merged
Merged
Closed
@ueshin
Copy link
Contributor Author

ueshin commented Dec 15, 2022

Hi, Databricks Runtime 12.0 has been released.
What's the plan for releases of the backports?
Thanks.

@ueshin
Copy link
Contributor Author

ueshin commented Jan 6, 2023 •
edited
Loading

Hi, I see dbt-spark 1.1.1 and 1.2.1 were released on Dec 23, but what about 1.3?
Do you have a plan to release it?

@ueshin ueshin mentioned this issue Jan 19, 2023
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Supports new error messages. ueshin/dbt-spark
3 participants
@ueshin @Fleid @jtcohen6