-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reorganize global project (macros) #4154
Merged
Merged
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
e79c7d5
Add integration tests
jtcohen6 5c80d6e
Reorganize + dispatch more global macros
jtcohen6 48b8d53
Reorg materializations subdir
jtcohen6 67523e6
Move around + document generic tests
jtcohen6 838a002
Fix failing tests
jtcohen6 8a6c7a7
Fix merge conflict
jtcohen6 9185c58
Grab fix from #4148
jtcohen6 f4a3fed
PR feedback
jtcohen6 b03f4f4
Fixup
jtcohen6 5a2ea5a
Add load_relation back, it was nice
jtcohen6 ec98984
Last few test fixes
jtcohen6 99b4073
Rm incremental_upsert, now unused
jtcohen6 171b3cc
Add changelog entry
jtcohen6 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
89 changes: 89 additions & 0 deletions
89
core/dbt/include/global_project/macros/adapters/columns.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,89 @@ | ||
{% macro get_columns_in_relation(relation) -%} | ||
{{ return(adapter.dispatch('get_columns_in_relation', 'dbt')(relation)) }} | ||
{% endmacro %} | ||
|
||
{% macro default__get_columns_in_relation(relation) -%} | ||
{{ exceptions.raise_not_implemented( | ||
'get_columns_in_relation macro not implemented for adapter '+adapter.type()) }} | ||
{% endmacro %} | ||
|
||
{# helper for adapter-specific implementations of get_columns_in_relation #} | ||
{% macro sql_convert_columns_in_relation(table) -%} | ||
{% set columns = [] %} | ||
{% for row in table %} | ||
{% do columns.append(api.Column(*row)) %} | ||
{% endfor %} | ||
{{ return(columns) }} | ||
{% endmacro %} | ||
|
||
|
||
{% macro get_columns_in_query(select_sql) -%} | ||
{{ return(adapter.dispatch('get_columns_in_query', 'dbt')(select_sql)) }} | ||
{% endmacro %} | ||
|
||
{% macro default__get_columns_in_query(select_sql) %} | ||
{% call statement('get_columns_in_query', fetch_result=True, auto_begin=False) -%} | ||
select * from ( | ||
{{ select_sql }} | ||
) as __dbt_sbq | ||
where false | ||
limit 0 | ||
{% endcall %} | ||
|
||
{{ return(load_result('get_columns_in_query').table.columns | map(attribute='name') | list) }} | ||
{% endmacro %} | ||
|
||
|
||
{% macro alter_column_type(relation, column_name, new_column_type) -%} | ||
{{ return(adapter.dispatch('alter_column_type', 'dbt')(relation, column_name, new_column_type)) }} | ||
{% endmacro %} | ||
|
||
{% macro default__alter_column_type(relation, column_name, new_column_type) -%} | ||
{# | ||
1. Create a new column (w/ temp name and correct type) | ||
2. Copy data over to it | ||
3. Drop the existing column (cascade!) | ||
4. Rename the new column to existing column | ||
#} | ||
{%- set tmp_column = column_name + "__dbt_alter" -%} | ||
|
||
{% call statement('alter_column_type') %} | ||
alter table {{ relation }} add column {{ adapter.quote(tmp_column) }} {{ new_column_type }}; | ||
update {{ relation }} set {{ adapter.quote(tmp_column) }} = {{ adapter.quote(column_name) }}; | ||
alter table {{ relation }} drop column {{ adapter.quote(column_name) }} cascade; | ||
alter table {{ relation }} rename column {{ adapter.quote(tmp_column) }} to {{ adapter.quote(column_name) }} | ||
{% endcall %} | ||
|
||
{% endmacro %} | ||
|
||
|
||
{% macro alter_relation_add_remove_columns(relation, add_columns = none, remove_columns = none) -%} | ||
{{ return(adapter.dispatch('alter_relation_add_remove_columns', 'dbt')(relation, add_columns, remove_columns)) }} | ||
{% endmacro %} | ||
|
||
{% macro default__alter_relation_add_remove_columns(relation, add_columns, remove_columns) %} | ||
|
||
{% if add_columns is none %} | ||
{% set add_columns = [] %} | ||
{% endif %} | ||
{% if remove_columns is none %} | ||
{% set remove_columns = [] %} | ||
{% endif %} | ||
|
||
{% set sql -%} | ||
|
||
alter {{ relation.type }} {{ relation }} | ||
|
||
{% for column in add_columns %} | ||
add column {{ column.name }} {{ column.data_type }}{{ ',' if not loop.last }} | ||
{% endfor %}{{ ',' if add_columns and remove_columns }} | ||
|
||
{% for column in remove_columns %} | ||
drop column {{ column.name }}{{ ',' if not loop.last }} | ||
{% endfor %} | ||
|
||
{%- endset -%} | ||
|
||
{% do run_query(sql) %} | ||
|
||
{% endmacro %} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
side note: do we need these auto-generated docs here even?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is what's included by default in the dbt-docs site
Overview
, until/unless users provide a custom overview. It could be simpler / have less info in it — is that what you're thinking?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure. but does the
global_project
need it?global_project
doesn't show up in anyones dbt docs, nor are any of the macros documented? I guess this file is just present in all scaffolded dbt projects?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's right - this is how it gets into users' project, by default.
It's an opinionated choice to exclude internal/global macros from the dbt-docs site; they're included in
manifest.json
, just like every other macro. I'd like to document global macros (perhaps a community-sourced effort??), which we can use to power other cool metadata-y things down the line