Make internal macros use macro dispatch pattern

[ueshin]

Description

Makes internal macros use macro dispatch pattern for backward compatibility for users who override the equivalent macros in their projects.

[jtcohen6]

In cases where these macros are line-for-line identical with their counterparts in dbt-spark, I don't think you actually need to redefine them here. You can still call them within other macros in this plugin, including in materializations. The reason: All macros defined in plugins and in dbt-core's global project are saved in the "internal"/"global" dbt namespace.

In cases where you do need to redefine these macros—e.g. file_format_clause should return delta by default rather than parquet—we should use explicit macro dispatch to deterministically define inheritance order. Docs: https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch

To that end, I think the right solution is to add dispatch calls and spark__ prefixes in dbt-spark, so that you can then add databricks__ prefixes in dbt-databricks. To simplify, the preference order is:

1. Macros in the user's own project — they can always override the builtins, and they can do it by defining a macro named any of these: file_format_clause, databricks__file_format_clause, spark__file_format_clause, default__file_format_clause
2. Built-in macros prefixed databricks__: your version, databricks__file_format_clause, delta by default
3. Built-in macros prefixed spark__: our version, spark__file_format_clause (should probably be parquet by default rather than nothing/text!)
4. Finally, macros prefixed default__, i.e. if file_format_clause were to be a macro built into dbt-core with a sensible default, in the same manner as create_table_as and get_merge_sql

Could I ask you to open that as an issue/PR in dbt-spark? If the CLA is a blocker to contributing actual code, let's talk more and find a way to make that not so

[allisonwang-db]

Looks good!

[allisonwang-db]

Just curious, if a user has re-defined the file_format_clause in an existing project, for example

{% macro file_format_clause() %}
...
{% endmacro %}

Will this change (and the upstream change in dbt-spark) break the existing pipeline when the user upgrades the adapter version?

[ueshin]

According to @jtcohen6's explanation in #72 (comment), the user's file_format_clause will be used.

1. Macros in the user's own project — they can always override the builtins, and they can do it by defining a macro named any of these: file_format_clause, databricks__file_format_clause, spark__file_format_clause, default__file_format_clause

[allisonwang-db]

This is needed because we need to install the latest dbt-spark version first?

[ueshin]

Yes.

[ueshin]

Thanks! merging.