Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add Snowflake online store (again) #2922

Merged
merged 18 commits into from
Jul 29, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@
* [PostgreSQL (contrib)](reference/offline-stores/postgres.md)
* [Online stores](reference/online-stores/README.md)
* [SQLite](reference/online-stores/sqlite.md)
* [Snowflake](reference/online-stores/snowflake.md)
* [Redis](reference/online-stores/redis.md)
* [Datastore](reference/online-stores/datastore.md)
* [DynamoDB](reference/online-stores/dynamodb.md)
Expand Down
4 changes: 2 additions & 2 deletions docs/reference/offline-stores/snowflake.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Description

The Snowflake offline store provides support for reading [SnowflakeSources](../data-sources/snowflake.md).
The [Snowflake](https://trial.snowflake.com) offline store provides support for reading [SnowflakeSources](../data-sources/snowflake.md).

* Snowflake tables and views are allowed as sources.
* All joins happen within Snowflake.
Expand All @@ -11,7 +11,7 @@ The Snowflake offline store provides support for reading [SnowflakeSources](../d
* This allows you to call
* `to_snowflake` to save the dataset into Snowflake
* `to_sql` to get the SQL query that would execute on `to_df`
* `to_arrow_chunks` to get the result in batches ([Snowflake python connector docs](https://docs.snowflake.com/en/user-guide/python-connector-api.html#get_result_batches))
* `to_arrow_chunks` to get the result in batches ([Snowflake python connector docs](https://docs.snowflake.com/en/user-guide/python-connector-api.html#get_result_batches))

## Example

Expand Down
2 changes: 2 additions & 0 deletions docs/reference/online-stores/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ Please see [Online Store](../../getting-started/architecture-and-components/onli

{% page-ref page="sqlite.md" %}

{% page-ref page="snowflake.md" %}

{% page-ref page="redis.md" %}

{% page-ref page="datastore.md" %}
Expand Down
35 changes: 35 additions & 0 deletions docs/reference/online-stores/snowflake.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Snowflake

## Description

The [Snowflake](https://trial.snowflake.com) online store provides support for materializing feature values into a Snowflake Transient Table for serving online features.

* Only the latest feature values are persisted

The data model for using a Snowflake Transient Table as an online store follows a tall format (one row per feature)):
* "entity_feature_key" (BINARY) -- unique key used when reading specific feature_view x entity combination
* "entity_key" (BINARY) -- repeated key currently unused for reading entity_combination
* "feature_name" (VARCHAR)
* "value" (BINARY)
* "event_ts" (TIMESTAMP)
* "created_ts" (TIMESTAMP)

(This model may be subject to change when Snowflake Hybrid Tables are released)

## Example

{% code title="feature_store.yaml" %}
```yaml
project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
type: snowflake.online
account: SNOWFLAKE_DEPLOYMENT_URL
user: SNOWFLAKE_USER
password: SNOWFLAKE_PASSWORD
role: SNOWFLAKE_ROLE
warehouse: SNOWFLAKE_WAREHOUSE
database: SNOWFLAKE_DATABASE
```
{% endcode %}
3 changes: 2 additions & 1 deletion docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ The list below contains the functionality that contributors are planning to deve
* [x] [In-memory / Pandas](https://docs.feast.dev/reference/offline-stores/file)
* [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/adding-a-new-offline-store)
* **Online Stores**
* [x] [Snowflake](https://docs.feast.dev/reference/online-stores/snowflake)
* [x] [DynamoDB](https://docs.feast.dev/reference/online-stores/dynamodb)
* [x] [Redis](https://docs.feast.dev/reference/online-stores/redis)
* [x] [Datastore](https://docs.feast.dev/reference/online-stores/datastore)
Expand Down Expand Up @@ -59,4 +60,4 @@ The list below contains the functionality that contributors are planning to deve
* [x] Model-centric feature tracking (feature services)
* [x] Amundsen integration (see [Feast extractor](https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/feast_extractor.py))
* [x] DataHub integration (see [DataHub Feast docs](https://datahubproject.io/docs/generated/ingestion/sources/feast/))
* [x] Feast Web UI (Alpha release. See [docs](https://docs.feast.dev/reference/alpha-web-ui))
* [x] Feast Web UI (Alpha release. See [docs](https://docs.feast.dev/reference/alpha-web-ui))
10 changes: 9 additions & 1 deletion docs/tutorials/driver-stats-on-snowflake.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
description: >-
Initial demonstration of Snowflake as an offline store with Feast, using the Snowflake demo template.
Initial demonstration of Snowflake as an offline+online store with Feast, using the Snowflake demo template.
---

# Drivers stats on Snowflake
Expand Down Expand Up @@ -61,6 +61,14 @@ offline_store:
role: ROLE_NAME #case sensitive
warehouse: WAREHOUSE_NAME #case sensitive
database: DATABASE_NAME #case cap sensitive
online_store:
type: snowflake.online
account: SNOWFLAKE_DEPLOYMENT_URL #drop .snowflakecomputing.com
user: USERNAME
password: PASSWORD
role: ROLE_NAME #case sensitive
warehouse: WAREHOUSE_NAME #case sensitive
database: DATABASE_NAME #case cap sensitive
```
{% endcode %}

Expand Down
2 changes: 1 addition & 1 deletion sdk/python/docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = []
html_static_path = ["_static"]


# -- Options for HTMLHelp output ------------------------------------------
Expand Down
8 changes: 8 additions & 0 deletions sdk/python/docs/source/feast.infra.online_stores.rst
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,14 @@ feast.infra.online\_stores.redis module
:undoc-members:
:show-inheritance:

feast.infra.online\_stores.snowflake module
-------------------------------------------

.. automodule:: feast.infra.online_stores.snowflake
:members:
:undoc-members:
:show-inheritance:

feast.infra.online\_stores.sqlite module
----------------------------------------

Expand Down
3 changes: 3 additions & 0 deletions sdk/python/docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -250,18 +250,21 @@ Sqlite Online Store

.. automodule:: feast.infra.online_stores.sqlite
:members:
:noindex:

Datastore Online Store
----------------------

.. automodule:: feast.infra.online_stores.datastore
:members:
:noindex:

DynamoDB Online Store
---------------------

.. automodule:: feast.infra.online_stores.dynamodb
:members:
:noindex:

Redis Online Store
------------------
Expand Down
Loading