-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPIKE] [CT-2650] Materialization macros should support dispatch #7799
Comments
Also came up in #7881 (comment) |
Materializations are a variant of Jinja macro (jinja extension) which is created in core/dbt/clients/jinja.py, in MaterializationExtension. They are saved in the manifest.macros dictionary with names such as: "macro.dbt.materialization_incremental_default" In order to use adapter.dispatch and use its macro resolution code we would need to normalize the names. (Could we have synonyms?) (Note: we currently have two macro resolvers, MacroResolver and MacroNamespace.) In the "def execute" method in dbt/task/run.py and dbt/task/clone.py we do manifest.get_materialization_macro_name. Instead we would have to execute adapter.dispatch to get the materialization macro, which would return a MacroGenerator object so it wouldn't need to be created. We don't currently call adapter.dispatch in Python, but the materialization macros are the "launcher" macros that get everything started so this might be an exception. Concerns here are the inconsistency in naming conventions between materializations and "normal" macros and the changes that would need to be made in the adapter repos. It remains to be seen how much incompatibility there would be. Possibly we might be able to have a temporary mapping from old to new materialization macros. But it's also possible that this would be a fairly big breaking change. |
Users define these as:
Lowest-risk solution is add the materialization macro by both names:
The main thing this gets us is allowing someone to write this and have it work:
We should raise a deprecation warning if people are still calling a materialization directly by the old name. In order to do this, we need to update our own materialization lookup logic ( Potential acceptance criteria:
|
On thinking about this a bit more, I think it's probably a bad idea to have duplicated macros with different names. It would complicate partial parsing quite a bit. I'm now thinking that the best way to do this would be to add code in adapter.dispatch so that if we fail to find a materialization macro we use special resolution code which will mangle the name into the "standard" materialization name. |
Having a way to call a materialization as a function (with reliable dispatch + ability to pass in arguments instead of patching the context) would also give us more options for implementing this Mesh UX improvement: |
In conversation with Kshitij just now, it seems that the main changes need to be in core/dbt/context/macros.py, in MacroNamespace.get_from_package where we can add special processing of "materialization" macros. The "get_from_package" method is called from adapter.dispatch, which comes from the adapter "contextproperty" of the ProviderContext in core/dbt/context/providers.py. The actual "dispatch" method is in the BaseDatabaseWrapper. The materialization macro can be found in the "execute" method of the clone task (and probably the same method in runnable.py) by doing context.adapter.dispatch from the constructed context. This should already be a MacroGenerator callable, so we wouldn't need to construct a MacroGenerator as we do further down in "execute". |
Closing in favor of #8031 |
Copied from #4646
Adapter-specific materializations work differently from adapter-specific macros. Namely:
{% materialization table, adapter = 'snowflake' %}
creates a macro namedmaterialization_table_snowflake
, rather than (as one would expect)snowflake__materialization_table
Let's make materialization macro generation / discovery work more like dispatch. We'd need to offer backwards compatibility / avoiding breaking changes for folks who currently rely on the
materialization_<name>_<adapter>
construction.This is important in cases where we want to "call" a materialization macro from within another macro (perhaps from within another materialization). I found a concrete use case for doing this in my spiking on
dbt clone
(#7258), whereby theclone
materialization would in certain instances want to "shell out" to theview
materialization.Specific comment & spot in the spiking code where this is relevant: #7258 (comment)
Depending on the implementation we pursue for #7442, this could also be a relevant capability there.
The text was updated successfully, but these errors were encountered: