-
Notifications
You must be signed in to change notification settings - Fork 1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge feast-snowflake plugin into main repo with documentation (#2193)
* Add backticks to left_table_query_string (#2250) Signed-off-by: david <davidmiller252@gmail.com> Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * Delete entity key from Redis only when all attached feature views are gone (#2240) * Delete entity from redis when the last attached feature view is deleted Signed-off-by: pyalex <moskalenko.alexey@gmail.com> * Delete entity key from Redis only when all attached feature views are gone Signed-off-by: pyalex <moskalenko.alexey@gmail.com> * make lint happy Signed-off-by: pyalex <moskalenko.alexey@gmail.com> * make lint happy Signed-off-by: pyalex <moskalenko.alexey@gmail.com> * one more try with mypy Signed-off-by: pyalex <moskalenko.alexey@gmail.com> Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * historical_field_mappings2 merge for one sign off commit (#2252) Signed-off-by: Michelle Rascati <michelle.rascati@sailpoint.com> Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * Correct inconsistent dependency (#2255) Signed-off-by: Judah Rand <17158624+judahrand@users.noreply.github.com> Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * Add snowflake environment variables to allow testing on snowflake infra (#2258) * add snowflake environment vars to test framework Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * add snowflake environment vars to test framework Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * Return `UNIX_TIMESTAMP` as Python `datetime` (#2244) * Refactor `UNIX_TIMESTAMP` conversion Signed-off-by: Judah Rand <17158624+judahrand@users.noreply.github.com> * Return `UNIX_TIMESTAMP` types as `datetime` to user Signed-off-by: Judah Rand <17158624+judahrand@users.noreply.github.com> * Fix linting errors Signed-off-by: Judah Rand <17158624+judahrand@users.noreply.github.com> * Rename variable to something more sensible Signed-off-by: Judah Rand <17158624+judahrand@users.noreply.github.com> Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * Feast plan clean up (#2256) * Run validation and inference on views and entities during plan Signed-off-by: Felix Wang <wangfelix98@gmail.com> * Do not log objects that are unchanged Signed-off-by: Felix Wang <wangfelix98@gmail.com> * Rename Fco to FeastObject Signed-off-by: Felix Wang <wangfelix98@gmail.com> * Remove useless method Signed-off-by: Felix Wang <wangfelix98@gmail.com> * Lint Signed-off-by: Felix Wang <wangfelix98@gmail.com> * Always initialize registry during feature store initialization Signed-off-by: Felix Wang <wangfelix98@gmail.com> * Fix usage test Signed-off-by: Felix Wang <wangfelix98@gmail.com> * Remove print statements Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * Squash commits Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * Add error type and refactor query execution to have retries Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * Handle more snowflake errors Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * Fix lint errors Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * Fix lint errors Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * Fix lint errors Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * Fix wrong import Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * modify registry.db s3 object initialization to work in S3 subdirectory with Java Feast Server (#2259) Signed-off-by: NalinGHub <nalinm01@gmail.com> Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * clean up docs Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * lint-python Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * fixed historical test Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> * fixed historical test Signed-off-by: sfc-gh-madkins <miles.adkins@snowflake.com> Co-authored-by: David Miller <david@patagona.ca> Co-authored-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> Co-authored-by: Michelle Rascati <44408275+michelle-rascati-sp@users.noreply.github.com> Co-authored-by: Judah Rand <17158624+judahrand@users.noreply.github.com> Co-authored-by: Felix Wang <wangfelix98@gmail.com> Co-authored-by: Danny Chiao <danny@tecton.ai> Co-authored-by: Nalin Mehra <37969183+NalinGHub@users.noreply.github.com>
- Loading branch information
1 parent
2e4f3e5
commit f2bc411
Showing
45 changed files
with
2,005 additions
and
54 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,44 @@ | ||
# Snowflake | ||
|
||
## Description | ||
|
||
Snowflake data sources allow for the retrieval of historical feature values from Snowflake for building training datasets as well as materializing features into an online store. | ||
|
||
* Either a table reference or a SQL query can be provided. | ||
|
||
## Examples | ||
|
||
Using a table reference | ||
|
||
```python | ||
from feast import SnowflakeSource | ||
|
||
my_snowflake_source = SnowflakeSource( | ||
database="FEAST", | ||
schema="PUBLIC", | ||
table="FEATURE_TABLE", | ||
) | ||
``` | ||
|
||
Using a query | ||
|
||
```python | ||
from feast import SnowflakeSource | ||
|
||
my_snowflake_source = SnowflakeSource( | ||
query=""" | ||
SELECT | ||
timestamp_column AS "ts", | ||
"created", | ||
"f1", | ||
"f2" | ||
FROM | ||
`FEAST.PUBLIC.FEATURE_TABLE` | ||
""", | ||
) | ||
``` | ||
|
||
One thing to remember is how Snowflake handles table and column name conventions. | ||
You can read more about quote identifiers [here](https://docs.snowflake.com/en/sql-reference/identifiers-syntax.html) | ||
|
||
Configuration options are available [here](https://rtd.feast.dev/en/latest/index.html#feast.data_source.SnowflakeSource). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
# Snowflake | ||
|
||
## Description | ||
|
||
The Snowflake offline store provides support for reading [SnowflakeSources](../data-sources/snowflake.md). | ||
|
||
* Snowflake tables and views are allowed as sources. | ||
* All joins happen within Snowflake. | ||
* Entity dataframes can be provided as a SQL query or can be provided as a Pandas dataframe. Pandas dataframes will be uploaded to Snowflake in order to complete join operations. | ||
* A [SnowflakeRetrievalJob](https://github.com/feast-dev/feast/blob/bf557bcb72c7878a16dccb48443bbbe9dc3efa49/sdk/python/feast/infra/offline_stores/snowflake.py#L185) is returned when calling `get_historical_features()`. | ||
|
||
## Example | ||
|
||
{% code title="feature_store.yaml" %} | ||
```yaml | ||
project: my_feature_repo | ||
registry: data/registry.db | ||
provider: local | ||
offline_store: | ||
type: snowflake.offline | ||
account: snowflake_deployment.us-east-1 | ||
user: user_login | ||
password: user_password | ||
role: sysadmin | ||
warehouse: demo_wh | ||
database: FEAST | ||
``` | ||
{% endcode %} | ||
Configuration options are available [here](https://github.com/feast-dev/feast/blob/bf557bcb72c7878a16dccb48443bbbe9dc3efa49/sdk/python/feast/infra/offline_stores/snowflake.py#L39). |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.