Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add Snowflake materialization engine #2948

Merged
merged 1 commit into from
Aug 18, 2022
Merged

feat: Add Snowflake materialization engine #2948

merged 1 commit into from
Aug 18, 2022

Conversation

sfc-gh-madkins
Copy link
Collaborator

@sfc-gh-madkins sfc-gh-madkins commented Jul 16, 2022 •
edited
Loading

What this PR does / why we need it:

Allows Snowflake to be a materialization option when Snowflake is used as an offline store

Which issue(s) this PR fixes:

Fixes #2633

@codecov-commenter
Copy link

codecov-commenter commented Jul 16, 2022 •
edited
Loading

Codecov Report

Merging #2948 (5d0cf95) into master (83c5efb) will decrease coverage by 9.28%.
The diff coverage is 33.42%.

@@            Coverage Diff             @@
##           master    #2948      +/-   ##
==========================================
- Coverage   67.08%   57.80%   -9.29%     
==========================================
  Files         173      209      +36     
  Lines       15132    17380    +2248     
==========================================
- Hits        10152    10046     -106     
- Misses       4980     7334    +2354
Flag Coverage Δ
integrationtests ?
unittests 57.80% <33.42%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...hon/feast/infra/offline_stores/snowflake_source.py 63.04% <14.70%> (-29.27%) ⬇️
...hon/feast/infra/utils/snowflake/snowflake_utils.py 18.96% <20.51%> (ø)
...on/feast/infra/materialization/snowflake_engine.py 34.37% <34.37%> (ø)
...ests/integration/materialization/test_snowflake.py 38.09% <38.09%> (ø)
sdk/python/feast/infra/offline_stores/snowflake.py 37.19% <46.15%> (-47.14%) ⬇️
sdk/python/feast/type_map.py 46.54% <46.66%> (-13.75%) ⬇️
...python/feast/infra/offline_stores/offline_store.py 67.64% <50.00%> (-15.36%) ⬇️
sdk/python/feast/repo_config.py 76.62% <50.00%> (-6.77%) ⬇️
sdk/python/feast/infra/online_stores/snowflake.py 47.66% <71.42%> (-50.51%) ⬇️
.../feature_repos/universal/data_sources/snowflake.py 51.21% <100.00%> (-48.79%) ⬇️
... and 165 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

achals
achals reviewed Jul 22, 2022
View reviewed changes
Copy link
Member

@achals achals left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks reasonable to me

sdk/python/feast/type_map.py Show resolved Hide resolved
sdk/python/feast/infra/utils/snowflake_udfs.py Outdated Show resolved Hide resolved
sdk/python/feast/infra/utils/snowflake_udfs.py Outdated Show resolved Hide resolved
sdk/python/feast/infra/utils/snowflake_udfs.py Outdated Show resolved Hide resolved
kevjumba
kevjumba reviewed Jul 25, 2022
View reviewed changes
Copy link
Collaborator

@kevjumba kevjumba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just need some documentation and a few nits.

sdk/python/feast/infra/materialization/snowflake_engine.py Show resolved Hide resolved
sdk/python/feast/infra/materialization/snowflake_engine.py Show resolved Hide resolved
sdk/python/feast/infra/materialization/snowflake_engine.py Outdated Show resolved Hide resolved
@sfc-gh-madkins sfc-gh-madkins force-pushed the snowflake_materialize branch 2 times, most recently from 905cdb4 to 88984d9 Compare July 30, 2022 13:13
@sfc-gh-madkins sfc-gh-madkins force-pushed the snowflake_materialize branch 2 times, most recently from db2a1b7 to 2c61948 Compare July 31, 2022 02:13
@sfc-gh-madkins sfc-gh-madkins force-pushed the snowflake_materialize branch 5 times, most recently from 747a85f to fe1834e Compare August 2, 2022 02:55
@sfc-gh-madkins sfc-gh-madkins force-pushed the snowflake_materialize branch 2 times, most recently from 456a520 to b42c452 Compare August 16, 2022 04:52
@sfc-gh-madkins sfc-gh-madkins force-pushed the snowflake_materialize branch 6 times, most recently from a0b28b7 to ff3c697 Compare August 17, 2022 17:43
@sfc-gh-madkins sfc-gh-madkins force-pushed the snowflake_materialize branch 4 times, most recently from 7d99d8b to 70eb0f1 Compare August 18, 2022 04:52
achals
achals approved these changes Aug 18, 2022
View reviewed changes
Copy link
Member

@achals achals left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@achals
Copy link
Member

achals commented Aug 18, 2022

Tests are red because of a couple of tests that aren't being skipped correctly (since pr-local-integration-tests is triggered on pull_request_target).

Going to force merge this in.

@feast-ci-bot feast-ci-bot removed the lgtm label Aug 18, 2022
@sfc-gh-madkins
feat: Add Snowflake materialization engine
5d0cf95 
Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>
achals
achals approved these changes Aug 18, 2022
View reviewed changes
Copy link
Member

@achals achals left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@feast-ci-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: achals, kevjumba, sfc-gh-madkins

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@achals achals merged commit f3b522b into feast-dev:master Aug 18, 2022
@sfc-gh-madkins sfc-gh-madkins deleted the snowflake_materialize branch August 19, 2022 03:51
kevjumba pushed a commit that referenced this pull request Aug 25, 2022
@feast-ci-bot
chore(release): release 0.24.0
c1c71da 
# [0.24.0](v0.23.0...v0.24.0) (2022-08-25)

### Bug Fixes

* Check if on_demand_feature_views is an empty list rather than None for snowflake provider ([#3046](#3046)) ([9b05e65](9b05e65))
* FeatureStore.apply applies BatchFeatureView correctly ([#3098](#3098)) ([41be511](41be511))
* Fix Feast Java inconsistency with int64 serialization vs python ([#3031](#3031)) ([4bba787](4bba787))
* Fix feature service inference logic ([#3089](#3089)) ([4310ed7](4310ed7))
* Fix field mapping logic during feature inference ([#3067](#3067)) ([cdfa761](cdfa761))
* Fix incorrect on demand feature view diffing and improve Java tests ([#3074](#3074)) ([0702310](0702310))
* Fix Java helm charts to work with refactored logic. Fix FTS image ([#3105](#3105)) ([2b493e0](2b493e0))
* Fix on demand feature view output in feast plan + Web UI crash ([#3057](#3057)) ([bfae6ac](bfae6ac))
* Fix release workflow to release 0.24.0 ([#3138](#3138)) ([a69aaae](a69aaae))
* Fix Spark offline store type conversion to arrow ([#3071](#3071)) ([b26566d](b26566d))
* Fixing Web UI, which fails for the SQL registry ([#3028](#3028)) ([64603b6](64603b6))
* Force Snowflake Session to Timezone UTC ([#3083](#3083)) ([9f221e6](9f221e6))
* Make infer dummy entity join key idempotent ([#3115](#3115)) ([1f5b1e0](1f5b1e0))
* More explicit error messages ([#2708](#2708)) ([e4d7afd](e4d7afd))
* Parse inline data sources ([#3036](#3036)) ([c7ba370](c7ba370))
* Prevent overwriting existing file during `persist` ([#3088](#3088)) ([69af21f](69af21f))
* Register BatchFeatureView in feature repos correctly ([#3092](#3092)) ([b8e39ea](b8e39ea))
* Return an empty infra object from sql registry when it doesn't exist ([#3022](#3022)) ([8ba87d1](8ba87d1))
* Teardown tables for Snowflake Materialization testing ([#3106](#3106)) ([0a0c974](0a0c974))
* UI error when saved dataset is present in registry. ([#3124](#3124)) ([83cf753](83cf753))
* Update sql.py ([#3096](#3096)) ([2646a86](2646a86))
* Updated snowflake template ([#3130](#3130)) ([f0594e1](f0594e1))

### Features

* Add authentication option for snowflake connector ([#3039](#3039)) ([74c75f1](74c75f1))
* Add Cassandra/AstraDB online store contribution ([#2873](#2873)) ([feb6cb8](feb6cb8))
* Add Snowflake materialization engine ([#2948](#2948)) ([f3b522b](f3b522b))
* Adding saved dataset capabilities for Postgres  ([#3070](#3070)) ([d3253c3](d3253c3))
* Allow passing repo config path via flag ([#3077](#3077)) ([0d2d951](0d2d951))
* Contrib azure provider with synapse/mssql offline store and Azure registry store ([#3072](#3072)) ([9f7e557](9f7e557))
* Custom Docker image for Bytewax batch materialization ([#3099](#3099)) ([cdd1b07](cdd1b07))
* Feast AWS Athena offline store (again) ([#3044](#3044)) ([989ce08](989ce08))
* Implement spark offline store `offline_write_batch` method ([#3076](#3076)) ([5b0cc87](5b0cc87))
* Initial Bytewax materialization engine ([#2974](#2974)) ([55c61f9](55c61f9))
* Refactor feature server helm charts to allow passing feature_store.yaml in environment variables ([#3113](#3113)) ([85ee789](85ee789))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@adchia adchia adchia left review comments

@achals achals achals approved these changes

@kevjumba kevjumba kevjumba approved these changes

Assignees

@achals achals

Labels
approved lgtm ok-to-test size/XXL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Snowflake type behavior is strange
6 participants
@sfc-gh-madkins @codecov-commenter @kevjumba @achals @feast-ci-bot @adchia