SNOW-1010216: Support more option for create or replace dynamic table #2110

sfc-gh-aalam · 2024-08-16T16:06:00Z

Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

Fixes SNOW-1010216, SNOW-1569878: Creating Dynamic Table using Snowpark does not support all options #2019
Fill out the following pre-review checklist:
- I am adding a new automated test(s) to verify correctness of my new code
  - If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
- I am adding new logging messages
- I am adding a new telemetry message
- I am adding new credentials
- I am adding a new dependency
- If this is a new feature/behavior, I'm adding the Local Testing parity changes.
Please describe how your code solves the related issue.

Adding support to specify the following:

REFERSH_MODE
INITIALIZE
CLUSTER BY
TRANSIENT
DATA_RETENTION_TIME_IN_DAYS
MAX_DATA_EXTENSION_TIME_IN_DAYS

sfc-gh-jrose

Are there any tests that can be added for these new parameters?

docs/source/snowpark/dataframe.rst

Co-authored-by: Jamison Rose <Jamison.Rose@snowflake.com>

sfc-gh-aalam · 2024-08-16T17:11:23Z

Are there any tests that can be added for these new parameters?

Sorry. I was in the process of adding more tests. I have pushed the tests to cover all new cases now.

CHANGELOG.md

github-actions · 2024-08-16T20:02:05Z

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

github-actions · 2024-08-17T01:29:07Z

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

sfc-gh-yixie · 2024-08-21T00:10:05Z

src/snowflake/snowpark/_internal/analyzer/snowflake_plan.py

+                name,
+                warehouse,
+                lag,
+                comment,
+                replace,
+                if_not_exists,
+                refresh_mode,
+                initialize,
+                clustering_keys,
+                is_transient,
+                data_retention_time,
+                max_data_extension_time,
+                x,


I suggest you use named arguments. Positional arguments are error prone when there are so many arguments.

sfc-gh-yixie · 2024-08-21T00:10:12Z

CHANGELOG.md

@@ -6,6 +6,14 @@

 #### New Features
 - Added support for `snowflake.snowpark.testing.assert_dataframe_equal` that is a util function to check the equality of two Snowpark DataFrames.
+- Added support for specifying the following attributes when create `DataFrame.create_or_replace_dynamic_table`:


Added support for specifying the following parameters to DataFrame.create_or_replace_dynamic_table()

sfc-gh-yixie · 2024-08-21T00:11:16Z

src/snowflake/snowpark/_internal/analyzer/analyzer_utils.py

    return (
        CREATE
-        + OR
-        + REPLACE
+        + replace_sql


How about changing to format string instead of +?

sfc-gh-yixie · 2024-08-21T00:13:38Z

src/snowflake/snowpark/dataframe.py

@@ -3435,7 +3444,28 @@ def create_or_replace_dynamic_table(
            lag: specifies the target data freshness
            comment: Adds a comment for the created table. See
                `COMMENT <https://docs.snowflake.com/en/sql-reference/sql/comment>`_.
+            mode: Specifies the behavior of create dynamic table. Allowed values are:


Does a dynamic table have truncate? I guess no. Just to double confirm.

It does not support it in the create or replace dynamic table api. Here we map
overwrite = CREATE OR REPLACE DYNAMIC TABLE
errorifexists = CREATE DYNAMIC TABLE
ignore = CREATE DYNAMIC TABLE IF NOT EXISTS

github-actions · 2024-08-21T18:25:36Z

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

github-actions · 2024-08-21T19:00:05Z

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

src/snowflake/snowpark/_internal/analyzer/snowflake_plan.py

Support more option for create or replace dynamic table

2640e13

sfc-gh-aalam requested a review from a team as a code owner August 16, 2024 16:06

sfc-gh-aalam requested review from sfc-gh-smirzaei, sfc-gh-yuwang and sfc-gh-jrose August 16, 2024 16:06

changelog update

e46caee

sfc-gh-jrose reviewed Aug 16, 2024

View reviewed changes

docs/source/snowpark/dataframe.rst Outdated Show resolved Hide resolved

sfc-gh-aalam and others added 4 commits August 16, 2024 09:37

Update synk requirement to match with setup.py

97771df

add tests

efef7a9

Update docs/source/snowpark/dataframe.rst

8b61e55

Co-authored-by: Jamison Rose <Jamison.Rose@snowflake.com>

add more cases

320277a

sfc-gh-aalam added 3 commits August 16, 2024 10:15

fix type check

a46a657

fix data retention value

701bca5

add mode

8f966b6

sfc-gh-aalam commented Aug 16, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

sfc-gh-aalam added 2 commits August 16, 2024 12:01

self review

dd6435a

skip local test

7cfecdf

refactor

42a5335

sfc-gh-yixie reviewed Aug 21, 2024

View reviewed changes

sfc-gh-aalam added 2 commits August 21, 2024 10:04

merge with main

4560f8c

fix sql generation

f59a333

fix query generation

ce3414f

sfc-gh-jrose approved these changes Aug 21, 2024

View reviewed changes

src/snowflake/snowpark/_internal/analyzer/snowflake_plan.py Show resolved Hide resolved

sfc-gh-yixie approved these changes Aug 22, 2024

View reviewed changes

sfc-gh-aalam merged commit 92dd396 into main Aug 22, 2024
34 checks passed

sfc-gh-aalam deleted the aalam-SNOW-1010216-create-or-replace-dynamic-table-options branch August 22, 2024 17:22

github-actions bot locked and limited conversation to collaborators Aug 22, 2024

sfc-gh-aalam linked an issue Aug 22, 2024 that may be closed by this pull request

SNOW-1569878: Creating Dynamic Table using Snowpark does not support all options #2019

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SNOW-1010216: Support more option for create or replace dynamic table #2110

SNOW-1010216: Support more option for create or replace dynamic table #2110

sfc-gh-aalam commented Aug 16, 2024 •

edited

Loading

sfc-gh-jrose left a comment

sfc-gh-aalam commented Aug 16, 2024

github-actions bot commented Aug 16, 2024

github-actions bot commented Aug 17, 2024

sfc-gh-yixie Aug 21, 2024

sfc-gh-yixie Aug 21, 2024

sfc-gh-yixie Aug 21, 2024

sfc-gh-yixie Aug 21, 2024

sfc-gh-aalam Aug 21, 2024

github-actions bot commented Aug 21, 2024

github-actions bot commented Aug 21, 2024

SNOW-1010216: Support more option for create or replace dynamic table #2110

SNOW-1010216: Support more option for create or replace dynamic table #2110

Conversation

sfc-gh-aalam commented Aug 16, 2024 • edited Loading

sfc-gh-jrose left a comment

Choose a reason for hiding this comment

sfc-gh-aalam commented Aug 16, 2024

github-actions bot commented Aug 16, 2024

github-actions bot commented Aug 17, 2024

sfc-gh-yixie Aug 21, 2024

Choose a reason for hiding this comment

sfc-gh-yixie Aug 21, 2024

Choose a reason for hiding this comment

sfc-gh-yixie Aug 21, 2024

Choose a reason for hiding this comment

sfc-gh-yixie Aug 21, 2024

Choose a reason for hiding this comment

sfc-gh-aalam Aug 21, 2024

Choose a reason for hiding this comment

github-actions bot commented Aug 21, 2024

github-actions bot commented Aug 21, 2024

sfc-gh-aalam commented Aug 16, 2024 •

edited

Loading