Implement `get_catalog_for_single_relation` macro #1064

aranke · 2024-05-28T10:10:11Z

Problem

We need to implement the get_catalog_for_single_relation macro and GetCatalogForSingleRelation capability for Snowflake.

Solution

Set GetCatalogForSingleRelation capability to Full
Implement get_catalog_for_single_relation macro using show objects

Checklist

I have read the contributing guide and understand what's expected of me
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

aranke · 2024-05-28T10:11:42Z

Current pre-commit failure:

Check for dbt-core usage in an adapter...................................Failed
- hook id: dbt-core-in-adapters-check
- exit code: 1

A dbt-core module is imported in dbt/adapters/snowflake/impl.py: dbt.artifacts.schemas.catalog.TableMetadata

jtcohen6 · 2024-05-28T10:28:00Z

Nice to see that the dbt-core import check is working as intended!

I think that means TableMetadata, StatsDict, StatsItem will need to move into dbt-adapters &/or dbt-common. That might also enable a better return type for the methods introduced in dbt-labs/dbt-adapters#231

aranke · 2024-05-29T20:08:33Z

Tests are currently failing because we haven't merged in dbt-labs/dbt-adapters#231 yet.

jtcohen6 · 2024-05-30T09:45:56Z

Tests are currently failing because we haven't merged in dbt-labs/dbt-adapters#231 yet.

@aranke You can update this line in dev requirements to run full dbt-snowflake CI with your proposed dbt-adapters changes:

git+https://github.com/dbt-labs/dbt-adapters.git@get_relation_metadata

.changes/unreleased/Fixes-20240516-224134.yaml

.gitignore

dbt/include/snowflake/macros/adapters.sql

mikealfare · 2024-06-11T19:51:22Z

dev-requirements.txt

-git+https://github.com/dbt-labs/dbt-adapters.git
-git+https://github.com/dbt-labs/dbt-adapters.git#subdirectory=dbt-tests-adapter
-git+https://github.com/dbt-labs/dbt-common.git
+git+https://github.com/dbt-labs/dbt-adapters.git@get_relation_metadata


Reminder to revert prior to merging.

mikealfare · 2024-06-11T19:52:16Z

setup.py

@@ -57,8 +57,8 @@ def _plugin_version() -> str:
    packages=find_namespace_packages(include=["dbt", "dbt.*"]),
    include_package_data=True,
    install_requires=[
-        "dbt-common>=1.0.4,<2.0",
-        "dbt-adapters>=1.1.1,<2.0",
+        # "dbt-common>=1.2.0,<2.0",


Reminder to revert prior to merging. Also, we need a new lower bound on dbt-adapters and dbt-common for this new feature.

tests/functional/adapter/test_basic.py

dbt/adapters/snowflake/impl.py

dbt/include/snowflake/macros/adapters.sql

dbt/adapters/snowflake/impl.py

mikealfare

I only noticed the MV thing, otherwise it looks good.

dbt/adapters/snowflake/relation_configs/policies.py

dbt/adapters/snowflake/impl.py

aranke added 2 commits May 16, 2024 22:43

Allow enabling/disabling stats collection on catalog table using env var

361d196

Implement get_relation_metadata macro

4970ddf

cla-bot bot added the cla:yes label May 28, 2024

Merge branch 'main' into catalog_enable_stats

bfe8cb1

Discard changes to dbt/include/snowflake/macros/catalog.sql

be990e1

This was referenced May 29, 2024

Move StatsItem, StatsDict, TableMetadata to dbt-common dbt-labs/dbt-common#141

Merged

Add get_catalog_for_single_relation macro and capability dbt-labs/dbt-adapters#231

Merged

aranke added 3 commits May 29, 2024 15:19

Import artifacts from dbt_common

1783849

fix dbt_common import

d19e3ef

Use get_relation_metadata from dbt-common

2213a16

Update Fixes-20240516-224134.yaml

efb3595

aranke marked this pull request as ready for review May 29, 2024 20:12

aranke requested a review from a team as a code owner May 29, 2024 20:12

use snowflakerelation

1944fb7

aranke marked this pull request as draft June 3, 2024 10:43

aranke mentioned this pull request Jun 3, 2024

Move CatalogKey, ColumnMetadata, ColumnMap, CatalogTable to dbt-common dbt-labs/dbt-common#147

Merged

5 tasks

aranke and others added 5 commits June 5, 2024 16:49

add broken test

8fa4f1d

wip

f76e29b

Merge branch 'main' into catalog_enable_stats

4f5336f

add test for get_catalog_for_single_relation

6e09133

Merge branch 'main' into catalog_enable_stats

f7892e1

aranke changed the title ~~Implement get_relation_metadata macro~~ Implement get_catalog_for_single_relation macro Jun 11, 2024

aranke marked this pull request as ready for review June 11, 2024 16:38

mikealfare reviewed Jun 11, 2024

View reviewed changes

aranke added 2 commits June 12, 2024 14:52

Merge branch 'main' into catalog_enable_stats

a30cc8c

wip

9cf3497

aranke added 6 commits June 18, 2024 15:57

wip

8bd2b11

merge from main

b501d16

hardcode macro name

ab51351

fix docs_generate test

2f4510b

address pr comments

5aacd01

update changie, add type sigs

126ba32

aranke marked this pull request as ready for review June 18, 2024 17:22

aranke requested a review from mikealfare June 18, 2024 17:22

colin-rogers-dbt reviewed Jun 18, 2024

View reviewed changes

dbt/include/snowflake/macros/adapters.sql Show resolved Hide resolved

colin-rogers-dbt reviewed Jun 18, 2024

View reviewed changes

dbt/adapters/snowflake/impl.py Outdated Show resolved Hide resolved

Use enum values for comparison

a6cc1c7

aranke requested a review from colin-rogers-dbt June 18, 2024 18:20

mikealfare reviewed Jun 18, 2024

View reviewed changes

dbt/adapters/snowflake/relation_configs/policies.py Outdated Show resolved Hide resolved

dbt/adapters/snowflake/impl.py Outdated Show resolved Hide resolved

remove mv

943be20

aranke requested a review from mikealfare June 18, 2024 20:13

mikealfare approved these changes Jun 18, 2024

View reviewed changes

aranke and others added 4 commits June 18, 2024 21:28

Discard changes to dev-requirements.txt

7a3ea26

Discard changes to setup.py

a269ae2

Merge branch 'main' into catalog_enable_stats

4c014d7

Merge branch 'main' into catalog_enable_stats

1a734c4

colin-rogers-dbt closed this Jun 21, 2024

colin-rogers-dbt reopened this Jun 21, 2024

colin-rogers-dbt enabled auto-merge (squash) June 21, 2024 03:03

Bump pins on dbt-common and dbt-adapters

55ee5c3

aranke disabled auto-merge June 21, 2024 17:45

aranke changed the title ~~Implement get_catalog_for_single_relation macro~~ Implement get_catalog_for_single_relation macro Jun 21, 2024

aranke closed this Jun 21, 2024

aranke reopened this Jun 21, 2024

colin-rogers-dbt merged commit d0a259f into main Jun 21, 2024
29 checks passed

colin-rogers-dbt deleted the catalog_enable_stats branch June 21, 2024 19:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement `get_catalog_for_single_relation` macro #1064

Implement `get_catalog_for_single_relation` macro #1064

aranke commented May 28, 2024 •

edited

Loading

aranke commented May 28, 2024

jtcohen6 commented May 28, 2024

aranke commented May 29, 2024

jtcohen6 commented May 30, 2024

mikealfare Jun 11, 2024

mikealfare Jun 11, 2024

mikealfare left a comment

Implement get_catalog_for_single_relation macro #1064

Implement get_catalog_for_single_relation macro #1064

Conversation

aranke commented May 28, 2024 • edited Loading

Problem

Solution

Checklist

aranke commented May 28, 2024

jtcohen6 commented May 28, 2024

aranke commented May 29, 2024

jtcohen6 commented May 30, 2024

mikealfare Jun 11, 2024

Choose a reason for hiding this comment

mikealfare Jun 11, 2024

Choose a reason for hiding this comment

mikealfare left a comment

Choose a reason for hiding this comment

Implement `get_catalog_for_single_relation` macro #1064

Implement `get_catalog_for_single_relation` macro #1064

aranke commented May 28, 2024 •

edited

Loading