feat: Feast Spark Offline Store #2349

kevjumba · 2022-03-01T01:58:05Z

What this PR does / why we need it:

Add spark offline store plugin from Adyen Spark to spark offline store. https://github.com/Adyen/feast-spark-offline-store
Currently 159/194 integration tests for spark source work.

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

Initial cut of Spark based Feast offline store

sdk/python/feast/type_map.py

sdk/python/feast/infra/offline_stores/third_party/spark.py

sdk/python/feast/infra/offline_stores/third_party/spark_source.py

sdk/python/feast/type_map.py

sdk/python/tests/integration/registration/test_universal_types.py

sdk/python/tests/integration/feature_repos/universal/data_sources/spark_data_source_creator.py

sdk/python/feast/infra/offline_stores/third_party/spark.py

codecov-commenter · 2022-03-02T23:31:05Z

Codecov Report

Merging #2349 (1acc088) into master (74f887f) will decrease coverage by 0.14%.
The diff coverage is 46.40%.

@@            Coverage Diff             @@
##           master    #2349      +/-   ##
==========================================
- Coverage   58.41%   58.27%   -0.15%     
==========================================
  Files         117      118       +1     
  Lines        9764     9885     +121     
==========================================
+ Hits         5704     5760      +56     
- Misses       4060     4125      +65

Flag	Coverage Δ
unittests	`58.27% <46.40%> (-0.15%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdk/python/feast/cli.py	`38.65% <ø> (ø)`
sdk/python/feast/inference.py	`60.60% <0.00%> (ø)`
sdk/python/feast/repo_config.py	`77.09% <ø> (ø)`
sdk/python/feast/type_map.py	`39.42% <37.50%> (-0.70%)`	⬇️
...stores/contrib/spark_offline_store/spark_source.py	`47.66% <47.66%> (ø)`
sdk/python/feast/__init__.py	`91.30% <100.00%> (+0.39%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74f887f...1acc088. Read the comment docs.

java/datatypes/src/main/proto/feast/core/DatastoreTable.proto

docs/reference/data-sources/spark.md

docs/reference/offline-stores/README.md

docs/reference/offline-stores/spark.md

sdk/python/feast/infra/offline_stores/third_party/spark_offline_store/spark.py

sdk/python/feast/type_map.py

sdk/python/tests/integration/feature_repos/universal/data_sources/spark_data_source_creator.py

sdk/python/tests/integration/feature_repos/repo_configuration.py

kevjumba

@adchia addressed changes PTAL!

docs/reference/offline-stores/spark.md

sdk/python/feast/infra/offline_stores/third_party/spark_offline_store/spark.py

sdk/python/tests/integration/feature_repos/repo_configuration.py

sdk/python/tests/integration/feature_repos/universal/data_sources/spark_data_source_creator.py

sdk/python/feast/type_map.py

sdk/python/tests/integration/feature_repos/universal/data_sources/spark_data_source_creator.py

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

adchia

/lgtm

feast-ci-bot · 2022-03-04T20:16:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adchia, kevjumba

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [adchia]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

# [0.19.0](v0.18.0...v0.19.0) (2022-03-05) ### Bug Fixes * Added additional value types to UI parser and removed references to registry-bq.json ([#2361](#2361)) ([d202d51](d202d51)) * Fix Redshift bug that stops waiting on statements after 5 minutes ([#2363](#2363)) ([74f887f](74f887f)) * Method _should_use_plan only returns true for local sqlite provider ([#2344](#2344)) ([fdb5f21](fdb5f21)) * Remove redis service to prevent more conflicts and add redis node to master_only ([#2354](#2354)) ([993616f](993616f)) * Rollback Redis-py to Redis-py-cluster ([#2347](#2347)) ([1ba86fb](1ba86fb)) * Update github workflow to prevent redis from overlapping ports. ([#2350](#2350)) ([c2a6c6c](c2a6c6c)) ### Features * Add owner field to Entity and rename labels to tags ([412d625](412d625)) * Allow all snowflake python connector connection methods to be available to Feast ([#2356](#2356)) ([ec7385c](ec7385c)) * Allowing password based authentication and SSL for Redis in Java feature server ([0af8adb](0af8adb)) * Event timestamps response ([#2355](#2355)) ([5481caf](5481caf)) * Feast Spark Offline Store ([#2349](#2349)) ([98b8d8d](98b8d8d)) * Initial merge of Web UI logic ([#2352](#2352)) ([ce3bc59](ce3bc59)) * Key ttl setting for redis online store ([#2341](#2341)) ([236a108](236a108)) * Metadata changes & making data sources top level objects to power Feast UI ([#2336](#2336)) ([43da230](43da230))

woop · 2022-03-05T06:13:04Z

🎉 This PR is included in version 0.19.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

* State of feast Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up changes Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix random incorrect changes Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix build errors Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add spark offline store components to test against current integration tests Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Rename to pass checks Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix issues Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix type checking issues Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up print statements for first review Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix flake 8 lint tests Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add warnings for alpha version release Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Format Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Address review Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Address review Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add file store functionality Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add example feature repo Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update data source creator Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Make cli work for feast init with spark Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update the docs Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up code Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up more code Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Uncomment repo configs Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix setup.py Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update dependencies Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix ci dependencies Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Screwed up rebase Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Screwed up rebase Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Screwed up rebase Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Realign with master Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix accidental changes Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Make type map change cleaner Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Address review comments Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix tests accidentally broken Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add comments Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Reformat Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix logger Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Remove unused imports Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix imports Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix CI dependencies Signed-off-by: Danny Chiao <danny@tecton.ai> * Prefix destinations with project name Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update comment Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix 3.8 Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * temporary fix Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * rollback Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * update Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update ci? Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Move third party to contrib Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix imports Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Remove third_party refactor Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Revert ci requirements and update comment in type map Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Revert 3.8-requirements Signed-off-by: Kevin Zhang <kzhang@tecton.ai> Co-authored-by: Danny Chiao <danny@tecton.ai>

# [0.19.0](feast-dev/feast@v0.18.0...v0.19.0) (2022-03-05) ### Bug Fixes * Added additional value types to UI parser and removed references to registry-bq.json ([feast-dev#2361](feast-dev#2361)) ([d202d51](feast-dev@d202d51)) * Fix Redshift bug that stops waiting on statements after 5 minutes ([feast-dev#2363](feast-dev#2363)) ([74f887f](feast-dev@74f887f)) * Method _should_use_plan only returns true for local sqlite provider ([feast-dev#2344](feast-dev#2344)) ([fdb5f21](feast-dev@fdb5f21)) * Remove redis service to prevent more conflicts and add redis node to master_only ([feast-dev#2354](feast-dev#2354)) ([993616f](feast-dev@993616f)) * Rollback Redis-py to Redis-py-cluster ([feast-dev#2347](feast-dev#2347)) ([1ba86fb](feast-dev@1ba86fb)) * Update github workflow to prevent redis from overlapping ports. ([feast-dev#2350](feast-dev#2350)) ([c2a6c6c](feast-dev@c2a6c6c)) ### Features * Add owner field to Entity and rename labels to tags ([412d625](feast-dev@412d625)) * Allow all snowflake python connector connection methods to be available to Feast ([feast-dev#2356](feast-dev#2356)) ([ec7385c](feast-dev@ec7385c)) * Allowing password based authentication and SSL for Redis in Java feature server ([0af8adb](feast-dev@0af8adb)) * Event timestamps response ([feast-dev#2355](feast-dev#2355)) ([5481caf](feast-dev@5481caf)) * Feast Spark Offline Store ([feast-dev#2349](feast-dev#2349)) ([98b8d8d](feast-dev@98b8d8d)) * Initial merge of Web UI logic ([feast-dev#2352](feast-dev#2352)) ([ce3bc59](feast-dev@ce3bc59)) * Key ttl setting for redis online store ([feast-dev#2341](feast-dev#2341)) ([236a108](feast-dev@236a108)) * Metadata changes & making data sources top level objects to power Feast UI ([feast-dev#2336](feast-dev#2336)) ([43da230](feast-dev@43da230)) Signed-off-by: Achal Shah <achals@gmail.com>

* State of feast Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up changes Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix random incorrect changes Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix build errors Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add spark offline store components to test against current integration tests Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Rename to pass checks Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix issues Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix type checking issues Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up print statements for first review Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix flake 8 lint tests Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add warnings for alpha version release Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Format Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Address review Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Address review Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add file store functionality Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add example feature repo Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update data source creator Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Make cli work for feast init with spark Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update the docs Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up code Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up more code Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Uncomment repo configs Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix setup.py Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update dependencies Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix ci dependencies Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Screwed up rebase Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Screwed up rebase Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Screwed up rebase Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Realign with master Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix accidental changes Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Make type map change cleaner Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Address review comments Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix tests accidentally broken Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add comments Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Reformat Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix logger Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Remove unused imports Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix imports Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix CI dependencies Signed-off-by: Danny Chiao <danny@tecton.ai> * Prefix destinations with project name Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update comment Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix 3.8 Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * temporary fix Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * rollback Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * update Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update ci? Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Move third party to contrib Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix imports Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Remove third_party refactor Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Revert ci requirements and update comment in type map Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Revert 3.8-requirements Signed-off-by: Kevin Zhang <kzhang@tecton.ai> Co-authored-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com>

# [0.19.0](feast-dev/feast@v0.18.0...v0.19.0) (2022-03-05) ### Bug Fixes * Added additional value types to UI parser and removed references to registry-bq.json ([feast-dev#2361](feast-dev#2361)) ([d202d51](feast-dev@d202d51)) * Fix Redshift bug that stops waiting on statements after 5 minutes ([feast-dev#2363](feast-dev#2363)) ([74f887f](feast-dev@74f887f)) * Method _should_use_plan only returns true for local sqlite provider ([feast-dev#2344](feast-dev#2344)) ([fdb5f21](feast-dev@fdb5f21)) * Remove redis service to prevent more conflicts and add redis node to master_only ([feast-dev#2354](feast-dev#2354)) ([993616f](feast-dev@993616f)) * Rollback Redis-py to Redis-py-cluster ([feast-dev#2347](feast-dev#2347)) ([1ba86fb](feast-dev@1ba86fb)) * Update github workflow to prevent redis from overlapping ports. ([feast-dev#2350](feast-dev#2350)) ([c2a6c6c](feast-dev@c2a6c6c)) ### Features * Add owner field to Entity and rename labels to tags ([412d625](feast-dev@412d625)) * Allow all snowflake python connector connection methods to be available to Feast ([feast-dev#2356](feast-dev#2356)) ([ec7385c](feast-dev@ec7385c)) * Allowing password based authentication and SSL for Redis in Java feature server ([0af8adb](feast-dev@0af8adb)) * Event timestamps response ([feast-dev#2355](feast-dev#2355)) ([5481caf](feast-dev@5481caf)) * Feast Spark Offline Store ([feast-dev#2349](feast-dev#2349)) ([98b8d8d](feast-dev@98b8d8d)) * Initial merge of Web UI logic ([feast-dev#2352](feast-dev#2352)) ([ce3bc59](feast-dev@ce3bc59)) * Key ttl setting for redis online store ([feast-dev#2341](feast-dev#2341)) ([236a108](feast-dev@236a108)) * Metadata changes & making data sources top level objects to power Feast UI ([feast-dev#2336](feast-dev#2336)) ([43da230](feast-dev@43da230)) Signed-off-by: Achal Shah <achals@gmail.com>

* State of feast Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up changes Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix random incorrect changes Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix build errors Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add spark offline store components to test against current integration tests Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Rename to pass checks Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix issues Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix type checking issues Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up print statements for first review Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix flake 8 lint tests Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add warnings for alpha version release Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Format Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Address review Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Address review Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add file store functionality Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add example feature repo Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update data source creator Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Make cli work for feast init with spark Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update the docs Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up code Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up more code Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Uncomment repo configs Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix setup.py Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update dependencies Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix ci dependencies Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Screwed up rebase Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Screwed up rebase Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Screwed up rebase Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Realign with master Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix accidental changes Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Make type map change cleaner Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Address review comments Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix tests accidentally broken Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add comments Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Reformat Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix logger Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Remove unused imports Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix imports Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix CI dependencies Signed-off-by: Danny Chiao <danny@tecton.ai> * Prefix destinations with project name Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update comment Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix 3.8 Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * temporary fix Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * rollback Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * update Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update ci? Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Move third party to contrib Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix imports Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Remove third_party refactor Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Revert ci requirements and update comment in type map Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Revert 3.8-requirements Signed-off-by: Kevin Zhang <kzhang@tecton.ai> Co-authored-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com>

* Added additional value types to UI parser and removed references to registry-bq.json ([feast-dev#2361](feast-dev#2361)) ([d202d51](feast-dev@d202d51)) * Fix Redshift bug that stops waiting on statements after 5 minutes ([feast-dev#2363](feast-dev#2363)) ([74f887f](feast-dev@74f887f)) * Method _should_use_plan only returns true for local sqlite provider ([feast-dev#2344](feast-dev#2344)) ([fdb5f21](feast-dev@fdb5f21)) * Remove redis service to prevent more conflicts and add redis node to master_only ([feast-dev#2354](feast-dev#2354)) ([993616f](feast-dev@993616f)) * Rollback Redis-py to Redis-py-cluster ([feast-dev#2347](feast-dev#2347)) ([1ba86fb](feast-dev@1ba86fb)) * Update github workflow to prevent redis from overlapping ports. ([feast-dev#2350](feast-dev#2350)) ([c2a6c6c](feast-dev@c2a6c6c)) * Add owner field to Entity and rename labels to tags ([412d625](feast-dev@412d625)) * Allow all snowflake python connector connection methods to be available to Feast ([feast-dev#2356](feast-dev#2356)) ([ec7385c](feast-dev@ec7385c)) * Allowing password based authentication and SSL for Redis in Java feature server ([0af8adb](feast-dev@0af8adb)) * Event timestamps response ([feast-dev#2355](feast-dev#2355)) ([5481caf](feast-dev@5481caf)) * Feast Spark Offline Store ([feast-dev#2349](feast-dev#2349)) ([98b8d8d](feast-dev@98b8d8d)) * Initial merge of Web UI logic ([feast-dev#2352](feast-dev#2352)) ([ce3bc59](feast-dev@ce3bc59)) * Key ttl setting for redis online store ([feast-dev#2341](feast-dev#2341)) ([236a108](feast-dev@236a108)) * Metadata changes & making data sources top level objects to power Feast UI ([feast-dev#2336](feast-dev#2336)) ([43da230](feast-dev@43da230)) Signed-off-by: Achal Shah <achals@gmail.com>

* Remove sdk/go Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * Add interfaces and overall skeleton (incl working protobufs) Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Wire up FeatureStore & OnlineStore, implement RedisOnlineStore constructor, add tests Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Introduce RepoConfig struct instead of using raw map[string]interface{} Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Pass feast project name to RedisOnlineStore Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Add Redis Online Read Test Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Implement gRPC server start logic Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Update OnlineRead interface Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Redis reader with hashing implemented Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * added implementation for GetOnlineFeatures in featurestore and a simple test Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Iterate on key building Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Implement gRPC server Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Fix some warnings Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Fix bugs & tests Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Implement redis key builder and reader Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Add more TODO comments Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Rebuilt protos Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Ignore read test Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Add more TODO comments Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Clean up serialize function Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * fixed GetOnlineFeatures to so that each EntityKey is a row in the first param of OnlineRead, added values to SerializeValue Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Remove comment Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * return copy of Features from OnlineStore instead of reusing original copy + add response for OnlineStore Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Delete go/protos directory from git, since it's in .gitignore Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Fix go/server compilation Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Add integration test Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * added http json endpoint + add validation for feature names and entity keys for each requested feature view in GetOnlineFeatures Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Call go subprocess from Python Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * fixed GetOnlineFeatures bug Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * removed grpc-gateway + add go http server on fs.serve() Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * experimenting python connector Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * working version of python connector Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * fixed python connector error + add a script to build plugin binary Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * added a simple connector config type to RepoConfig in python sdk to support feast alpha enable go_feature_server Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * migrating python's sdk get_online_features and its helper functions Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * added redis pipeline + passed go tests, haven't tested feature service Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * added integration test w/o odfv + dummy entity Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * accidentally removed entity from request Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * passed integration test Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * first go server clean version, keep python functions. Next commit will diverge from python function calls for more optimized calls Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * added 'optimized' functions Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * added goroutines to OnlineRead Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * added goroutines to redis OnlineRead, getFeaturesToUse, getEntityMaps and entity struct Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Ensure the correct tests are run for the go feature server Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Format and lint Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Remove Go feature server implementation of Python feature server Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Remove incorrect comments Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Clean up Go feature server test logic Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Add docstrings for GoServer class Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Remove unused GoServerError Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Remove unused ConnectorOnlineStore Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Run Go feature server tests separately from main integration tests Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * lint-go + cleaned go code + handle BrokenPipeError in go_server.py Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * added create ./go/protos to compile-protos-go in Makefile Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * removed 1 unneeded comment Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * fix small bug in go_server raise if error is not brokenpipe in stop function Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Add Snowflake back in for Go feature server tests Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * formatted go Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Add CI for Go integration tests Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Modify setup.py to build Go protos and clean up Makefile Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Move go build logic to a different file to not trigger on pip install Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Fix Go protoc command Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * separated create config from json and yaml + changed go server _connect method name + add TODO to user proper logging + fix go_server exception on stop() + change go layout Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * added go server lifecycle test + local infra for go server + experiment with new go server implementations Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Only run Go tests on FileSource Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Fix Makefile commands Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * fixed test to use go_feature_server flag in test + added option to 2 go server implementations + fix parsedKind Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * fixed goserverlifecyle test implementation Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * fixed goserverlifecycle --- still need to review this test Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * added a command to test goserver with thread, works in reasonable time now but need to review go server thread implementation Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Print output Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Clean up Makefile and fix errors in setup.py Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Pin Golang protobuf dependency to specific version Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Format + lint Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Revert previous change Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Add back Python version Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Fix test errors due to rebase Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Add Go version to unit test workflow Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Fix .gitignore Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Format + lint Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Add setup Go action to set correct Go version for unit tests Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Test Gopath in workflow Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Add Github action for installing protoc during unit tests Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Print stderr for Go proto generation Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Check for PATH and GOBIN in unit tests Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Lint Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Switch from using protoc binary for Go protos to grpcio-tools Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Add psutil==5.9.0 dependency for test_go_server_life_cycle test Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Specify Go version during Github actions Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Add back protoc setup Github action Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Add GOPATH to GITHUB_PATH so that protoc-gen-go is discovered Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Set PATH so that protoc-gen-go can be found Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Update Python3.7 ci-requirements file Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Update all requirements Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Fix test Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Fix protoc-gen-go bug for all workflows Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Clean up existing test and mark new test as integration Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Lint Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Correctly set GOPATH in workflows Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * Modify AWS Lambda Dockerfile to copy the necessary Go files Signed-off-by: Felix Wang <wangfelix98@gmail.com> Signed-off-by: Achal Shah <achals@gmail.com> * merge infra package into feast package Signed-off-by: Achal Shah <achals@gmail.com> * optimized go server + address some PR comments Signed-off-by: Ly Cao <lycao@Lys-MacBook-Pro.local> Signed-off-by: Achal Shah <achals@gmail.com> * added newServingServiceServer to comply with the rest of the code style Signed-off-by: Ly Cao <lycao@Lys-MacBook-Pro.local> Signed-off-by: Achal Shah <achals@gmail.com> * removed an extra err check Signed-off-by: Ly Cao <lycao@Lys-MacBook-Pro.local> Signed-off-by: Achal Shah <achals@gmail.com> * added pre-allocation to online response metadata Signed-off-by: Ly Cao <lycao@Lys-MacBook-Pro.local> Signed-off-by: Achal Shah <achals@gmail.com> * Fix protoc compilation for go and use a unix domain socket file Signed-off-by: Achal Shah <achals@gmail.com> * Use a flag for compilation Signed-off-by: Achal Shah <achals@gmail.com> * Change flag in Signed-off-by: Achal Shah <achals@gmail.com> * fix: Remove redis service to prevent more conflicts and add redis node to master_only (#2354) * Fix redis on master.yml Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Remove hack in make_feature_store_yaml Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix error Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix tests Signed-off-by: Kevin Zhang <kzhang@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * fix: Added additional value types to UI parser and removed references to registry-bq.json (#2361) Signed-off-by: Yun Nan Liu <yunnanl@twitter.com> Co-authored-by: Yun Nan Liu <yunnanl@twitter.com> Signed-off-by: Achal Shah <achals@gmail.com> * fix:revert back to jdk 11 for datatypes package Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * fix:upgrade feast ui packages to fix critical vulnerabilities Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * ci: Add support for semantic release (#2332) * Add support for semantic release Signed-off-by: Willem Pienaar <git@willem.co> * Fix typos Signed-off-by: Willem Pienaar <git@willem.co> * Require a Personal Access Token Signed-off-by: Willem Pienaar <git@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * feat: Allow all snowflake python connector connection methods to be available to Feast (#2356) * Allow all snowflake python connector connection methods to be available to feast Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * format/lint Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * feat: have feast using snowflake python connector for authentication Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> * fix random print Signed-off-by: Miles Adkins <miles.adkins@snowflake.com> Signed-off-by: Achal Shah <achals@gmail.com> * fix: Fix Redshift bug that stops waiting on statements after 5 minutes (#2363) Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * feat: Feast Spark Offline Store (#2349) * State of feast Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up changes Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix random incorrect changes Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix build errors Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add spark offline store components to test against current integration tests Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Rename to pass checks Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix issues Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix type checking issues Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up print statements for first review Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix flake 8 lint tests Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add warnings for alpha version release Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Format Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Address review Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Address review Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add file store functionality Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add example feature repo Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update data source creator Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Make cli work for feast init with spark Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update the docs Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up code Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up more code Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Uncomment repo configs Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix setup.py Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update dependencies Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix ci dependencies Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Screwed up rebase Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Screwed up rebase Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Screwed up rebase Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Realign with master Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix accidental changes Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Make type map change cleaner Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Address review comments Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix tests accidentally broken Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add comments Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Reformat Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix logger Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Remove unused imports Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix imports Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix CI dependencies Signed-off-by: Danny Chiao <danny@tecton.ai> * Prefix destinations with project name Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update comment Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix 3.8 Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * temporary fix Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * rollback Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * update Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update ci? Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Move third party to contrib Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix imports Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Remove third_party refactor Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Revert ci requirements and update comment in type map Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Revert 3.8-requirements Signed-off-by: Kevin Zhang <kzhang@tecton.ai> Co-authored-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * ci: Update versions of helm charts + java pom files Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * feat: Event timestamps response (#2355) * ability to get event timestamps from online response Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com> * fix event timestamp bugs Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com> * python formatting Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com> * optional param to retrieve event_timestamp in online_reponse Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com> * formatting Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com> * renaming param Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com> Signed-off-by: Achal Shah <achals@gmail.com> * feat: Metadata changes & making data sources top level objects to power Feast UI (#2336) * Squash commits for metadata changes Signed-off-by: Danny Chiao <danny@tecton.ai> * tests Signed-off-by: Danny Chiao <danny@tecton.ai> * Add more tests Signed-off-by: Danny Chiao <danny@tecton.ai> * lint Signed-off-by: Danny Chiao <danny@tecton.ai> * Add apply test Signed-off-by: Danny Chiao <danny@tecton.ai> * Add apply test Signed-off-by: Danny Chiao <danny@tecton.ai> * Add apply test Signed-off-by: Danny Chiao <danny@tecton.ai> * lint Signed-off-by: Danny Chiao <danny@tecton.ai> * fix bigquery source Signed-off-by: Danny Chiao <danny@tecton.ai> * fix test Signed-off-by: Danny Chiao <danny@tecton.ai> * fix spark source Signed-off-by: Danny Chiao <danny@tecton.ai> * fix spark source Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * ci: Remove buggy branch identification Signed-off-by: Willem Pienaar <git@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * fix action and set go version Signed-off-by: Achal Shah <achals@gmail.com> * Complie python protos correctly Signed-off-by: Achal Shah <achals@gmail.com> * Set up go in the pr integration test workflow Signed-off-by: Achal Shah <achals@gmail.com> * Nuke go requirements for python ci (for the moment) Signed-off-by: Achal Shah <achals@gmail.com> * Update redis port Signed-off-by: Achal Shah <achals@gmail.com> * ci: Add version bump script Signed-off-by: Willem Pienaar <git@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * ci: Fix broken release preparation Signed-off-by: Willem Pienaar <git@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * ci: Add ci bot as committer and fix typo in dry-run config Signed-off-by: feast-ci-bot <feast-ci-bot@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * chore(release): release 0.19.0 # [0.19.0](v0.18.0...v0.19.0) (2022-03-05) ### Bug Fixes * Added additional value types to UI parser and removed references to registry-bq.json ([#2361](#2361)) ([d202d51](d202d51)) * Fix Redshift bug that stops waiting on statements after 5 minutes ([#2363](#2363)) ([74f887f](74f887f)) * Method _should_use_plan only returns true for local sqlite provider ([#2344](#2344)) ([fdb5f21](fdb5f21)) * Remove redis service to prevent more conflicts and add redis node to master_only ([#2354](#2354)) ([993616f](993616f)) * Rollback Redis-py to Redis-py-cluster ([#2347](#2347)) ([1ba86fb](1ba86fb)) * Update github workflow to prevent redis from overlapping ports. ([#2350](#2350)) ([c2a6c6c](c2a6c6c)) ### Features * Add owner field to Entity and rename labels to tags ([412d625](412d625)) * Allow all snowflake python connector connection methods to be available to Feast ([#2356](#2356)) ([ec7385c](ec7385c)) * Allowing password based authentication and SSL for Redis in Java feature server ([0af8adb](0af8adb)) * Event timestamps response ([#2355](#2355)) ([5481caf](5481caf)) * Feast Spark Offline Store ([#2349](#2349)) ([98b8d8d](98b8d8d)) * Initial merge of Web UI logic ([#2352](#2352)) ([ce3bc59](ce3bc59)) * Key ttl setting for redis online store ([#2341](#2341)) ([236a108](236a108)) * Metadata changes & making data sources top level objects to power Feast UI ([#2336](#2336)) ([43da230](43da230)) Signed-off-by: Achal Shah <achals@gmail.com> * chore: Fix changelog and disable release comments Signed-off-by: Willem Pienaar <git@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * fix: Fix unhashable Snowflake and Redshift sources Signed-off-by: Willem Pienaar <git@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * chore: Remove needs-kind and release not requirement Signed-off-by: Willem Pienaar <git@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * chore: Remove release note block Signed-off-by: Willem Pienaar <git@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * ci: Uncheck prerelease status for patch releases Signed-off-by: Willem Pienaar <git@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * fix: Fix default feast apply path without any extras (#2373) * fix: Fix default feast apply path without any extras Signed-off-by: Danny Chiao <danny@tecton.ai> * revert removing ge Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * docs: Add short section about github actions (#2315) * Add short section about github actions Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Move to contributing.md Signed-off-by: Kevin Zhang <kzhang@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * fix: Use the correct dockerhub image tag when building feature servers (#2372) * fix: Use the correct dockerhub image tag when building feature servers Signed-off-by: Achal Shah <achals@gmail.com> * Print attempt Signed-off-by: Achal Shah <achals@gmail.com> * Increase sleep interval Signed-off-by: Achal Shah <achals@gmail.com> * More logging and sleep Signed-off-by: Achal Shah <achals@gmail.com> * change logging to debug Signed-off-by: Achal Shah <achals@gmail.com> * Change the dockerimage name Signed-off-by: Achal Shah <achals@gmail.com> * Change version to have periods Signed-off-by: Achal Shah <achals@gmail.com> * Change version entirely Signed-off-by: Achal Shah <achals@gmail.com> * Undo unintended changes Signed-off-by: Achal Shah <achals@gmail.com> * Remove dead branch Signed-off-by: Achal Shah <achals@gmail.com> * compile go as part of python ci setup Signed-off-by: Achal Shah <achals@gmail.com> * set path with gopath/bin Signed-off-by: Achal Shah <achals@gmail.com> * Set path for commands correctly Signed-off-by: Achal Shah <achals@gmail.com> * Set up go for lint-go Signed-off-by: Achal Shah <achals@gmail.com> * Remove makefile stuff Signed-off-by: Achal Shah <achals@gmail.com> * Fix workflows and Makefile commands Signed-off-by: Achal Shah <achals@gmail.com> * protoc-gen-go-grpc Signed-off-by: Achal Shah <achals@gmail.com> * install go deps first Signed-off-by: Achal Shah <achals@gmail.com> * Compile go protos as well Signed-off-by: Achal Shah <achals@gmail.com> * Remove go mod tidy from the deps target Signed-off-by: Achal Shah <achals@gmail.com> * Better python tooling Signed-off-by: Achal Shah <achals@gmail.com> * Remove container Signed-off-by: Achal Shah <achals@gmail.com> * Upgrade lint go as well Signed-off-by: Achal Shah <achals@gmail.com> * Upgrade lint go as well Signed-off-by: Achal Shah <achals@gmail.com> * More logging for unit-go Signed-off-by: Achal Shah <achals@gmail.com> * Install using python setup.py develop Signed-off-by: Achal Shah <achals@gmail.com> * install-python-ci-dependencies Signed-off-by: Achal Shah <achals@gmail.com> * Skip tests that need a feature repo Signed-off-by: Achal Shah <achals@gmail.com> * Skip a scary forking test Signed-off-by: Achal Shah <achals@gmail.com> * Remove all the crusty port stuff and disable the lifecycle test Signed-off-by: Achal Shah <achals@gmail.com> * Remove more crusty port stuff Signed-off-by: Achal Shah <achals@gmail.com> * Update skip message Signed-off-by: Achal Shah <achals@gmail.com> * Introduce a sleep before reading values Signed-off-by: Achal Shah <achals@gmail.com> * Disable one more test configuration since background thread-based monitoring doesn't currently work Signed-off-by: Achal Shah <achals@gmail.com> * Disable the test entirely Signed-off-by: Achal Shah <achals@gmail.com> * ci: Add support for semantic release (#2332) * Add support for semantic release Signed-off-by: Willem Pienaar <git@willem.co> * Fix typos Signed-off-by: Willem Pienaar <git@willem.co> * Require a Personal Access Token Signed-off-by: Willem Pienaar <git@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * feat: Feast Spark Offline Store (#2349) * State of feast Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up changes Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix random incorrect changes Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix build errors Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add spark offline store components to test against current integration tests Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Rename to pass checks Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix issues Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix type checking issues Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up print statements for first review Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix flake 8 lint tests Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add warnings for alpha version release Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Format Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Address review Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Address review Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add file store functionality Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * lint Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add example feature repo Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update data source creator Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Make cli work for feast init with spark Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update the docs Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up code Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up more code Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Uncomment repo configs Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix setup.py Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update dependencies Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix ci dependencies Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Screwed up rebase Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Screwed up rebase Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Screwed up rebase Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Realign with master Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix accidental changes Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Make type map change cleaner Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Address review comments Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix tests accidentally broken Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add comments Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Reformat Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix logger Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Remove unused imports Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix imports Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix CI dependencies Signed-off-by: Danny Chiao <danny@tecton.ai> * Prefix destinations with project name Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update comment Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix 3.8 Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * temporary fix Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * rollback Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * update Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update ci? Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Move third party to contrib Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix imports Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Remove third_party refactor Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Revert ci requirements and update comment in type map Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Revert 3.8-requirements Signed-off-by: Kevin Zhang <kzhang@tecton.ai> Co-authored-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * ci: Update versions of helm charts + java pom files Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * feat: Metadata changes & making data sources top level objects to power Feast UI (#2336) * Squash commits for metadata changes Signed-off-by: Danny Chiao <danny@tecton.ai> * tests Signed-off-by: Danny Chiao <danny@tecton.ai> * Add more tests Signed-off-by: Danny Chiao <danny@tecton.ai> * lint Signed-off-by: Danny Chiao <danny@tecton.ai> * Add apply test Signed-off-by: Danny Chiao <danny@tecton.ai> * Add apply test Signed-off-by: Danny Chiao <danny@tecton.ai> * Add apply test Signed-off-by: Danny Chiao <danny@tecton.ai> * lint Signed-off-by: Danny Chiao <danny@tecton.ai> * fix bigquery source Signed-off-by: Danny Chiao <danny@tecton.ai> * fix test Signed-off-by: Danny Chiao <danny@tecton.ai> * fix spark source Signed-off-by: Danny Chiao <danny@tecton.ai> * fix spark source Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * ci: Remove buggy branch identification Signed-off-by: Willem Pienaar <git@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * ci: Add version bump script Signed-off-by: Willem Pienaar <git@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * ci: Fix broken release preparation Signed-off-by: Willem Pienaar <git@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * ci: Add ci bot as committer and fix typo in dry-run config Signed-off-by: feast-ci-bot <feast-ci-bot@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * chore(release): release 0.19.0 * Added additional value types to UI parser and removed references to registry-bq.json ([#2361](#2361)) ([d202d51](d202d51)) * Fix Redshift bug that stops waiting on statements after 5 minutes ([#2363](#2363)) ([74f887f](74f887f)) * Method _should_use_plan only returns true for local sqlite provider ([#2344](#2344)) ([fdb5f21](fdb5f21)) * Remove redis service to prevent more conflicts and add redis node to master_only ([#2354](#2354)) ([993616f](993616f)) * Rollback Redis-py to Redis-py-cluster ([#2347](#2347)) ([1ba86fb](1ba86fb)) * Update github workflow to prevent redis from overlapping ports. ([#2350](#2350)) ([c2a6c6c](c2a6c6c)) * Add owner field to Entity and rename labels to tags ([412d625](412d625)) * Allow all snowflake python connector connection methods to be available to Feast ([#2356](#2356)) ([ec7385c](ec7385c)) * Allowing password based authentication and SSL for Redis in Java feature server ([0af8adb](0af8adb)) * Event timestamps response ([#2355](#2355)) ([5481caf](5481caf)) * Feast Spark Offline Store ([#2349](#2349)) ([98b8d8d](98b8d8d)) * Initial merge of Web UI logic ([#2352](#2352)) ([ce3bc59](ce3bc59)) * Key ttl setting for redis online store ([#2341](#2341)) ([236a108](236a108)) * Metadata changes & making data sources top level objects to power Feast UI ([#2336](#2336)) ([43da230](43da230)) Signed-off-by: Achal Shah <achals@gmail.com> * chore: Fix changelog and disable release comments Signed-off-by: Willem Pienaar <git@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * ci: Uncheck prerelease status for patch releases Signed-off-by: Willem Pienaar <git@willem.co> Signed-off-by: Achal Shah <achals@gmail.com> * fix: Fix default feast apply path without any extras (#2373) * fix: Fix default feast apply path without any extras Signed-off-by: Danny Chiao <danny@tecton.ai> * revert removing ge Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * fix: Making a name for data sources not a breaking change (#2379) * fix: Making a name for data sources not a breaking change Signed-off-by: Danny Chiao <danny@tecton.ai> * fix test Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * fix: Update some fields optional in UI parser (#2380) Signed-off-by: Yun Nan Liu <yunnanl@twitter.com> Co-authored-by: Yun Nan Liu <yunnanl@twitter.com> Co-authored-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * fix: Fix spark docs (#2382) * Fix spark docs Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix readme.md Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix md Signed-off-by: Kevin Zhang <kzhang@tecton.ai> Co-authored-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * feat: Makefile for contrib for Issue #2364 (#2366) * Add make file Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Fix makefile Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Add documentation Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Clean up Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * Update documentation Signed-off-by: Kevin Zhang <kzhang@tecton.ai> * update make target Signed-off-by: Kevin Zhang <kzhang@tecton.ai> Co-authored-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Achal Shah <achals@gmail.com> * Closes threadpool resources upon datastore online_write_batch completion (#2386) Signed-off-by: Pamela Toman <ptoman@paloaltonetworks.com> Signed-off-by: Achal Shah <achals@gmail.com> * CR comments Signed-off-by: Achal Shah <achals@gmail.com> Co-authored-by: Tsotne Tabidze <tsotne@tecton.ai> Co-authored-by: Willem Pienaar <willem@Willems-MacBook-Pro.local> Co-authored-by: Ly Cao <lycao@Lys-MacBook-Pro.local> Co-authored-by: Willem Pienaar <git@willem.co> Co-authored-by: Ly Cao <lycao@lys-macbook-pro.myfiosgateway.com> Co-authored-by: Felix Wang <wangfelix98@gmail.com> Co-authored-by: Achal Shah <achals@gmail.com> Co-authored-by: Kevin Zhang <kzhang@tecton.ai> Co-authored-by: mickey-liu <mickey.y.liu@gmail.com> Co-authored-by: Yun Nan Liu <yunnanl@twitter.com> Co-authored-by: Danny Chiao <danny@tecton.ai> Co-authored-by: Willem Pienaar <6728866+woop@users.noreply.github.com> Co-authored-by: sfc-gh-madkins <82121043+sfc-gh-madkins@users.noreply.github.com> Co-authored-by: Vitaly Sergeyev <vsergeyev@better.com> Co-authored-by: feast-ci-bot <feast-ci-bot@willem.co> Co-authored-by: ptoman-pa <95256508+ptoman-pa@users.noreply.github.com>

# [0.20.0](v0.19.0...v0.20.0) (2022-04-14) ### Bug Fixes * Add inlined data sources to the top level registry ([#2456](#2456)) ([356788a](356788a)) * Add new value types to types.ts for web ui ([#2463](#2463)) ([ad5694e](ad5694e)) * Add PushSource proto and Python class ([#2428](#2428)) ([9a4bd63](9a4bd63)) * Add spark to lambda dockerfile ([#2480](#2480)) ([514666f](514666f)) * Added private_key auth for Snowflake ([#2508](#2508)) ([c42c9b0](c42c9b0)) * Added Redshift and Spark typecheck to data_source event_timestamp_col inference ([#2389](#2389)) ([04dea73](04dea73)) * Building of go extension fails ([#2448](#2448)) ([7d1efd5](7d1efd5)) * Bump the number of versions bumps expected to 27 ([#2549](#2549)) ([ecc9938](ecc9938)) * Create __init__ files for the proto-generated python dirs ([#2410](#2410)) ([e17028d](e17028d)) * Don't prevent apply from running given duplicate empty names in data sources. Also fix repeated apply of Spark data source. ([#2415](#2415)) ([b95f441](b95f441)) * Dynamodb deduplicate batch write request by partition keys ([#2515](#2515)) ([70d4a13](70d4a13)) * Ensure that __init__ files exist in proto dirs ([#2433](#2433)) ([9b94f7b](9b94f7b)) * Fix DataSource constructor to unbreak custom data sources ([#2492](#2492)) ([712653e](712653e)) * Fix default feast apply path without any extras ([#2373](#2373)) ([6ba7fc7](6ba7fc7)) * Fix definitions.py with new definition ([#2541](#2541)) ([eefc34a](eefc34a)) * Fix entity row to use join key instead of name ([#2521](#2521)) ([c22fa2c](c22fa2c)) * Fix Java Master ([#2499](#2499)) ([e083458](e083458)) * Fix registry proto ([#2435](#2435)) ([ea6a9b2](ea6a9b2)) * Fix some inconsistencies in the docs and comments in the code ([#2444](#2444)) ([ad008bf](ad008bf)) * Fix spark docs ([#2382](#2382)) ([d4a606a](d4a606a)) * Fix Spark template to work correctly on feast init -t spark ([#2393](#2393)) ([ae133fd](ae133fd)) * Fix the feature repo fixture used by java tests ([#2469](#2469)) ([32e925e](32e925e)) * Fix unhashable Snowflake and Redshift sources ([cd8f1c9](cd8f1c9)) * Fixed bug in passing config file params to snowflake python connector ([#2503](#2503)) ([34f2b59](34f2b59)) * Fixing Spark template to include source name ([#2381](#2381)) ([a985f1d](a985f1d)) * Make name a keyword arg for the Entity class ([#2467](#2467)) ([43847de](43847de)) * Making a name for data sources not a breaking change ([#2379](#2379)) ([71d7ae2](71d7ae2)) * Minor link fix in `CONTRIBUTING.md` ([#2481](#2481)) ([2917e27](2917e27)) * Preserve ordering of features in _get_column_names ([#2457](#2457)) ([495b435](495b435)) * Relax click python requirement to >=7 ([#2450](#2450)) ([f202f92](f202f92)) * Remove date partition column field from datasources that don't s… ([#2478](#2478)) ([ce35835](ce35835)) * Remove docker step from unit test workflow ([#2535](#2535)) ([6f22f22](6f22f22)) * Remove spark from the AWS Lambda dockerfile ([#2498](#2498)) ([6abae16](6abae16)) * Request data api update ([#2488](#2488)) ([0c9e5b7](0c9e5b7)) * Schema update ([#2509](#2509)) ([cf7bbc2](cf7bbc2)) * Simplify DataSource.from_proto logic ([#2424](#2424)) ([6bda4d2](6bda4d2)) * Snowflake api update ([#2487](#2487)) ([1181a9e](1181a9e)) * Support passing batch source to streaming sources for backfills ([#2523](#2523)) ([90db1d1](90db1d1)) * Timestamp update ([#2486](#2486)) ([bf23111](bf23111)) * Typos in Feast UI error message ([#2432](#2432)) ([e14369d](e14369d)) * Update feature view APIs to prefer keyword args ([#2472](#2472)) ([7c19cf7](7c19cf7)) * Update file api ([#2470](#2470)) ([83a11c6](83a11c6)) * Update Makefile to cd into python dir before running commands ([#2437](#2437)) ([ca32155](ca32155)) * Update redshift api ([#2479](#2479)) ([4fa73a9](4fa73a9)) * Update some fields optional in UI parser ([#2380](#2380)) ([cff7ac3](cff7ac3)) * Use a single version of jackson libraries and upgrade to 2.12.6.1 ([#2473](#2473)) ([5be1cc6](5be1cc6)) * Use dateutil parser to parse materialization times ([#2464](#2464)) ([6c55e49](6c55e49)) * Use the correct dockerhub image tag when building feature servers ([#2372](#2372)) ([0d62c1d](0d62c1d)) ### Features * Add `/write-to-online-store` method to the python feature server ([#2423](#2423)) ([d2fb048](d2fb048)) * Add description, tags, owner fields to all feature view classes ([#2440](#2440)) ([ed5e928](ed5e928)) * Add DQM Logging on GRPC Server with FileLogStorage for Testing ([#2403](#2403)) ([57a97d8](57a97d8)) * Add Feast types in preparation for changing type system ([#2475](#2475)) ([4864252](4864252)) * Add Field class ([#2500](#2500)) ([1279612](1279612)) * Add support for DynamoDB online_read in batches ([#2371](#2371)) ([702ec49](702ec49)) * Add Support for DynamodbOnlineStoreConfig endpoint_url parameter ([#2485](#2485)) ([7b863d1](7b863d1)) * Add templating for dynamodb table name ([#2394](#2394)) ([f591088](f591088)) * Allow local feature server to use Go feature server if enabled ([#2538](#2538)) ([a2ef375](a2ef375)) * Allow using entity's join_key in get_online_features ([#2420](#2420)) ([068c765](068c765)) * Data Source Api Update ([#2468](#2468)) ([6b96b21](6b96b21)) * Go server ([#2339](#2339)) ([d12e7ef](d12e7ef)), closes [#2354](#2354) [#2361](#2361) [#2332](#2332) [#2356](#2356) [#2363](#2363) [#2349](#2349) [#2355](#2355) [#2336](#2336) [#2361](#2361) [#2363](#2363) [#2344](#2344) [#2354](#2354) [#2347](#2347) [#2350](#2350) [#2356](#2356) [#2355](#2355) [#2349](#2349) [#2352](#2352) [#2341](#2341) [#2336](#2336) [#2373](#2373) [#2315](#2315) [#2372](#2372) [#2332](#2332) [#2349](#2349) [#2336](#2336) [#2361](#2361) [#2363](#2363) [#2344](#2344) [#2354](#2354) [#2347](#2347) [#2350](#2350) [#2356](#2356) [#2355](#2355) [#2349](#2349) [#2352](#2352) [#2341](#2341) [#2336](#2336) [#2373](#2373) [#2379](#2379) [#2380](#2380) [#2382](#2382) [#2364](#2364) [#2366](#2366) [#2386](#2386) * Graduate write_to_online_store out of experimental status ([#2426](#2426)) ([e7dd4b7](e7dd4b7)) * Make feast PEP 561 compliant ([#2405](#2405)) ([3c41f94](3c41f94)), closes [#2420](#2420) [#2418](#2418) [#2425](#2425) [#2426](#2426) [#2427](#2427) [#2431](#2431) [#2433](#2433) [#2420](#2420) [#2418](#2418) [#2425](#2425) [#2426](#2426) [#2427](#2427) [#2431](#2431) [#2433](#2433) * Makefile for contrib for Issue [#2364](#2364) ([#2366](#2366)) ([a02325b](a02325b)) * Support on demand feature views in go feature server ([#2494](#2494)) ([6edd274](6edd274)) * Switch from Feature to Field ([#2514](#2514)) ([6a03bed](6a03bed)) * Use a daemon thread to monitor the go feature server exclusively ([#2391](#2391)) ([0bb5e8c](0bb5e8c))

rehevkor5 · 2024-05-13T17:13:28Z

sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py

+                    SELECT {field_string},
+                    ROW_NUMBER() OVER({partition_by_join_key_string} ORDER BY {timestamp_desc_string}) AS feast_row_
+                    FROM {from_expression} t1
+                    WHERE {event_timestamp_column} BETWEEN TIMESTAMP('{start_date_str}') AND TIMESTAMP('{end_date_str}')


I'm not sure if it was intentional that both the start and the end are inclusive, but IMO the starting point should be inclusive and the ending point should be exclusive. That way, if you have ranges of tuples like (a, b) and (b, c), they are contiguous and non-overlapping. By using BETWEEN the ranges will instead overlap at b.

kevjumba added kind/feature New feature or request do-not-merge/work-in-progress labels Mar 1, 2022

kevjumba requested a review from a team as a code owner March 1, 2022 01:58

kevjumba requested review from pyalex and removed request for a team March 1, 2022 01:58

feast-ci-bot added release-note-none and removed do-not-merge/work-in-progress labels Mar 1, 2022

github-actions bot assigned adchia Mar 1, 2022

feast-ci-bot added the size/XL label Mar 1, 2022

kevjumba force-pushed the feast_offline_store branch from cd74282 to 4eeef08 Compare March 1, 2022 02:01

kevjumba unassigned adchia Mar 1, 2022

kevjumba requested review from adchia and removed request for pyalex March 1, 2022 02:02

kevjumba changed the title ~~feat: [WIP] Feast Spark Offline Store~~ feat: Feast Spark Offline Store Mar 1, 2022

kevjumba added the ok-to-test label Mar 1, 2022

kevjumba commented Mar 1, 2022

View reviewed changes

sdk/python/feast/type_map.py Outdated Show resolved Hide resolved

sdk/python/feast/type_map.py Show resolved Hide resolved

adchia reviewed Mar 1, 2022

View reviewed changes

feast-ci-bot added size/XXL and removed size/XL labels Mar 2, 2022

kevjumba added the do-not-merge/work-in-progress label Mar 2, 2022

kevjumba force-pushed the feast_offline_store branch from 733d6bb to 0e76808 Compare March 2, 2022 22:19

kevjumba force-pushed the feast_offline_store branch from 3279f1f to 49c7dbf Compare March 3, 2022 00:28

pyalex reviewed Mar 3, 2022

View reviewed changes

java/datatypes/src/main/proto/feast/core/DatastoreTable.proto Outdated Show resolved Hide resolved

adchia reviewed Mar 3, 2022

View reviewed changes

kevjumba commented Mar 3, 2022

View reviewed changes

adchia reviewed Mar 3, 2022

View reviewed changes

sdk/python/feast/type_map.py Outdated Show resolved Hide resolved

sdk/python/tests/integration/feature_repos/universal/data_sources/spark_data_source_creator.py Outdated Show resolved Hide resolved

kevjumba removed the do-not-merge/work-in-progress label Mar 3, 2022

kevjumba force-pushed the feast_offline_store branch from cbbc57f to 12eb130 Compare March 3, 2022 23:24

feast-ci-bot removed the size/XXL label Mar 3, 2022

kevjumba added 3 commits March 4, 2022 12:05

Fix imports

05aaeb8

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Remove third_party refactor

29a60d7

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Revert ci requirements and update comment in type map

b43417e

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

kevjumba force-pushed the feast_offline_store branch from 9c6f6fa to b43417e Compare March 4, 2022 20:05

Revert 3.8-requirements

1acc088

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

adchia approved these changes Mar 4, 2022

View reviewed changes

feast-ci-bot assigned adchia Mar 4, 2022

feast-ci-bot added the lgtm label Mar 4, 2022

feast-ci-bot added approved release-note and removed release-note-none labels Mar 4, 2022

kevjumba added ok-to-test and removed ok-to-test labels Mar 4, 2022

adchia merged commit 98b8d8d into feast-dev:master Mar 4, 2022

rehevkor5 reviewed May 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Feast Spark Offline Store #2349

feat: Feast Spark Offline Store #2349

kevjumba commented Mar 1, 2022 •

edited by adchia

Loading

codecov-commenter commented Mar 2, 2022 •

edited

Loading

kevjumba left a comment

adchia left a comment

feast-ci-bot commented Mar 4, 2022

woop commented Mar 5, 2022

rehevkor5 May 13, 2024

feat: Feast Spark Offline Store #2349

feat: Feast Spark Offline Store #2349

Conversation

kevjumba commented Mar 1, 2022 • edited by adchia Loading

codecov-commenter commented Mar 2, 2022 • edited Loading

Codecov Report

kevjumba left a comment

Choose a reason for hiding this comment

adchia left a comment

Choose a reason for hiding this comment

feast-ci-bot commented Mar 4, 2022

woop commented Mar 5, 2022

rehevkor5 May 13, 2024

Choose a reason for hiding this comment

kevjumba commented Mar 1, 2022 •

edited by adchia

Loading

codecov-commenter commented Mar 2, 2022 •

edited

Loading