-
Notifications
You must be signed in to change notification settings - Fork 234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Model pre_hook is not executed on ODBC, but it is for HTTP #256
Comments
@dinedal Unfortunately, this is a known limitation for I think this must have to do with In the meantime, there are two viable workarounds to set your spark config (
Unfortunately, neither workaround will give you granular, model-specific behavior. The first should at least give you the ability to change behavior between different invocations, such as by setting its value to an env var. I'm not sure if there's anything we can do to fix this. I think we absolutely need to document the limitation and workarounds here: https://docs.getdbt.com/reference/resource-configs/spark-configs |
I actually tried option 1 and it was also ignored - option 2 is my current work-around as well, but it has it's own disadvantages.
Considering (ref https://docs.getdbt.com/reference/warehouse-profiles/spark-profile ) says:
Yes, I would agree that the preferred method should be documented that not all of dbt functions as expected. I would suggest changing the preferred method to HTTP, but on HTTP anything other than |
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days. |
Issue remains |
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days. |
Issue still remains |
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days. |
Describe the bug
We use a model with a config block like so:
When running with a profile setup like this:
Everything works as expected.
When we run with the profile set up like this:
We get the following error:
Steps To Reproduce
Simple reproduction:
set spark.sql.caseSensitive=trrue
)java.lang.IllegalArgumentException: spark.sql.caseSensitive should be boolean, but was trrue
)More complex way to reproduce:
Expected behavior
I expect on ODBC connections the behavior documented in https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook is correct.
I expect model pre_hooks to execute consistently regardless of the connection method.
Screenshots and log output
If applicable, add screenshots or log output to help explain your problem.
System information
The output of
dbt --version
:The operating system you're using:
Mac OS X 12.0.1, Ubuntu
The output of
python --version
:Python 3.8.6
Additional context
The text was updated successfully, but these errors were encountered: