Added case for handling postgres foreign tables...

[Aesthet]

(tables which are external to current database and are imported into current database from remote data stores by using Foreign Data Wrappers functionality).

This is a:

• bug fix PR with no breaking changes — please ensure the base branch is master
• new functionality — please ensure the base branch is the latest dev/ branch
• a breaking change — please ensure the base branch is the latest dev/ branch

Description & motivation

As described in this issue we are facing errors when we are using Postgres Foreign Data Wrappers functionality - in particular tables which are imported as part of IMPORT FOREIGN SCHEMA command. Those tables are external to database in which they are imported and they have type FOREIGN, which is causing errors during execution of get_relations_by_pattern macros which is getting values from get_tables_by_pattern_sql macros.

Checklist

• I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
  • BigQuery
  • Postgres
  • Redshift
  • Snowflake
• I followed guidelines to ensure that my changes will work on "non-core" adapters by:
  • dispatching any new macro(s) so non-core adapters can also use them (e.g. the star() source)
  • using the limit_zero() macro in place of the literal string: limit 0
  • using dbt_utils.type_* macros instead of explicit datatypes (e.g. dbt_utils.type_timestamp() instead of TIMESTAMP
• I have updated the README.md (if applicable)
• I have added tests & descriptions to my models (and macros if applicable)
• I have added an entry to CHANGELOG.md

[Aesthet]

Of course I could add separate foreign type, which would require adding new relation type here:

class RelationType(StrEnum):
    Table = 'table'
    View = 'view'
    CTE = 'cte'
    MaterializedView = 'materializedview'
    External = 'external'

and/or add this case specifically to postgres__get_tables_by_pattern_sql instead of default__get_tables_by_pattern_sql, but I think that would be an overkill.

[joellabes]

I agree that making a separate foreign type would be overkill, but I don't hate the idea of getting a focused postgres__get_tables_by_pattern_sql version of the macro, or at least a subset.

What about a new macro that generates the case statement for different table types? Something like

{% macro map_table_types(table_type) %}
  {{ return adapter.dispatch(yada yada) }} 
{% endmacro  %}

{% macro default__map_table_types() %}
 case table_type
   when 'BASE TABLE' then 'table'
   ...

{% endmacro %}

{% macro postgres__map_table_types() %}
 case table_type
   when 'BASE TABLE' then 'table'
   ...
   when 'FOREIGN' then 'external'

{% endmacro %}

and then get_tables_by_pattern_sql swaps out the case statement for a call to map_table_types() (I don't like the macro name; feel free to pick something better).
Handling an adapter-specific piece of functionality that other adapters either shouldn't know about or might have an adverse reaction to in the future (living in fear of Snowflake one day adding a FOREIGN type which is some other wild thing 😱) feels like exactly the sort of thing that multiple dispatch lends itself to with minimal overhead.

How does that sound to you?

[Aesthet]

Sounds reasonable to me, will update this PR today.

[joellabes]

Love it! I've added some thoughts based on your comments about how far to take it - I'm not wed to this, but I think it's a good way to balance the competing needs of minimal copy-paste vs not exposing adapter-specific logic to everything else.

[joellabes]

I agree that making a separate foreign type would be overkill, but I don't hate the idea of getting a focused postgres__get_tables_by_pattern_sql version of the macro, or at least a subset.

What about a new macro that generates the case statement for different table types? Something like

{% macro map_table_types(table_type) %}
  {{ return adapter.dispatch(yada yada) }} 
{% endmacro  %}

{% macro default__map_table_types() %}
 case table_type
   when 'BASE TABLE' then 'table'
   ...

{% endmacro %}

{% macro postgres__map_table_types() %}
 case table_type
   when 'BASE TABLE' then 'table'
   ...
   when 'FOREIGN' then 'external'

{% endmacro %}

and then get_tables_by_pattern_sql swaps out the case statement for a call to map_table_types() (I don't like the macro name; feel free to pick something better).
Handling an adapter-specific piece of functionality that other adapters either shouldn't know about or might have an adverse reaction to in the future (living in fear of Snowflake one day adding a FOREIGN type which is some other wild thing 😱) feels like exactly the sort of thing that multiple dispatch lends itself to with minimal overhead.

How does that sound to you?

[joellabes]

Wonderful @Aesthet! Please add yourself to the 0.8.1 CHANGELOG, then we'll get this merged in 🎉