Skip to content

Commit

Permalink
feat: Go server (#2339)
Browse files Browse the repository at this point in the history 
* Remove sdk/go

Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* Add interfaces and overall skeleton (incl working protobufs)

Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Wire up FeatureStore & OnlineStore, implement RedisOnlineStore constructor, add tests

Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Introduce RepoConfig struct instead of using raw map[string]interface{}

Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Pass feast project name to RedisOnlineStore

Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Add Redis Online Read Test

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Implement gRPC server start logic

Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Update OnlineRead interface

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Redis reader with hashing implemented

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* added implementation for GetOnlineFeatures in featurestore and a simple test

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Iterate on key building

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Implement gRPC server

Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Fix some warnings

Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Fix bugs & tests

Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Implement redis key builder and reader

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Add more TODO comments

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Rebuilt protos

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Ignore read test

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Add more TODO comments

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Clean up serialize function

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* fixed GetOnlineFeatures to so that each EntityKey is a row in the first param of OnlineRead, added values to SerializeValue

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Remove comment

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* return copy of Features from OnlineStore instead of reusing original copy + add response for OnlineStore

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Delete go/protos directory from git, since it's in .gitignore

Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Fix go/server compilation

Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Add integration test

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* added http json endpoint + add validation for feature names and entity keys for each requested feature view in GetOnlineFeatures

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Call go subprocess from Python

Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* fixed GetOnlineFeatures bug

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* removed grpc-gateway + add go http server on fs.serve()

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* experimenting python connector

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* working version of python connector

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* fixed python connector error + add a script to build plugin binary

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* added a simple connector config type to RepoConfig in python sdk to support feast alpha enable go_feature_server

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* migrating python's sdk get_online_features and its helper functions

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* added redis pipeline + passed go tests, haven't tested feature service

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* added integration test w/o odfv + dummy entity

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* accidentally removed entity from request

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* passed integration test

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* first go server clean version, keep python functions. Next commit will diverge from python function calls for more optimized calls

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* added 'optimized' functions

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* added goroutines to OnlineRead

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* added goroutines to redis OnlineRead, getFeaturesToUse, getEntityMaps and entity struct

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Ensure the correct tests are run for the go feature server

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Format and lint

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Remove Go feature server implementation of Python feature server

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Remove incorrect comments

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Clean up Go feature server test logic

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Add docstrings for GoServer class

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Remove unused GoServerError

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Remove unused ConnectorOnlineStore

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Run Go feature server tests separately from main integration tests

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* lint-go + cleaned go code + handle BrokenPipeError in go_server.py

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* added create ./go/protos to compile-protos-go in Makefile

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* removed 1 unneeded comment

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* fix small bug in go_server raise if error is not brokenpipe in stop function

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Add Snowflake back in for Go feature server tests

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* formatted go

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Add CI for Go integration tests

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Modify setup.py to build Go protos and clean up Makefile

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Move go build logic to a different file to not trigger on pip install

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Fix Go protoc command

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* separated create config from json and yaml + changed go server _connect method name + add TODO to user proper logging + fix go_server exception on stop() + change go layout

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* added go server lifecycle test + local infra for go server + experiment with new go server implementations

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Only run Go tests on FileSource

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Fix Makefile commands

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* fixed test to use go_feature_server flag in test + added option to 2 go server implementations + fix parsedKind

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* fixed goserverlifecyle test implementation

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* fixed goserverlifecycle --- still need to review this test

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* added a command to test goserver with thread, works in reasonable time now but need to review go server thread implementation

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Print output

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Clean up Makefile and fix errors in setup.py

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Pin Golang protobuf dependency to specific version

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Format + lint

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Revert previous change

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Add back Python version

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Fix test errors due to rebase

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Add Go version to unit test workflow

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Fix .gitignore

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Format + lint

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Add setup Go action to set correct Go version for unit tests

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Test Gopath in workflow

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Add Github action for installing protoc during unit tests

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Print stderr for Go proto generation

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Check for PATH and GOBIN in unit tests

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Lint

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Switch from using protoc binary for Go protos to grpcio-tools

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Add psutil==5.9.0 dependency for test_go_server_life_cycle test

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Specify Go version during Github actions

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Add back protoc setup Github action

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Add GOPATH to GITHUB_PATH so that protoc-gen-go is discovered

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Set PATH so that protoc-gen-go can be found

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Update Python3.7 ci-requirements file

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Update all requirements

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Fix test

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Fix protoc-gen-go bug for all workflows

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Clean up existing test and mark new test as integration

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Lint

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Correctly set GOPATH in workflows

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* Modify AWS Lambda Dockerfile to copy the necessary Go files

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* merge infra package into feast package

Signed-off-by: Achal Shah <achals@gmail.com>

* optimized go server + address some PR comments

Signed-off-by: Ly Cao <lycao@Lys-MacBook-Pro.local>
Signed-off-by: Achal Shah <achals@gmail.com>

* added newServingServiceServer to comply with the rest of the code style

Signed-off-by: Ly Cao <lycao@Lys-MacBook-Pro.local>
Signed-off-by: Achal Shah <achals@gmail.com>

* removed an extra err check

Signed-off-by: Ly Cao <lycao@Lys-MacBook-Pro.local>
Signed-off-by: Achal Shah <achals@gmail.com>

* added pre-allocation to online response metadata

Signed-off-by: Ly Cao <lycao@Lys-MacBook-Pro.local>
Signed-off-by: Achal Shah <achals@gmail.com>

* Fix protoc compilation for go and use a unix domain socket file

Signed-off-by: Achal Shah <achals@gmail.com>

* Use a flag for compilation

Signed-off-by: Achal Shah <achals@gmail.com>

* Change flag in

Signed-off-by: Achal Shah <achals@gmail.com>

* fix: Remove redis service to prevent more conflicts and add redis node to master_only (#2354)

* Fix redis on master.yml

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Remove hack in make_feature_store_yaml

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix error

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix tests

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* fix: Added additional value types to UI parser and removed references to registry-bq.json (#2361)

Signed-off-by: Yun Nan Liu <yunnanl@twitter.com>

Co-authored-by: Yun Nan Liu <yunnanl@twitter.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* fix:revert back to jdk 11 for datatypes package

Signed-off-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* fix:upgrade feast ui packages to fix critical vulnerabilities

Signed-off-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* ci: Add support for semantic release (#2332)

* Add support for semantic release

Signed-off-by: Willem Pienaar <git@willem.co>

* Fix typos

Signed-off-by: Willem Pienaar <git@willem.co>

* Require a Personal Access Token

Signed-off-by: Willem Pienaar <git@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* feat: Allow all snowflake python connector connection methods to be available to Feast (#2356)

* Allow all snowflake python connector connection methods to be available to feast

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

* format/lint

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

* feat: have feast using snowflake python connector for authentication

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>

* fix random print

Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* fix: Fix Redshift bug that stops waiting on statements after 5 minutes (#2363)

Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* feat: Feast Spark Offline Store (#2349)

* State of feast

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Clean up changes

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix random incorrect changes

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix lint

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix build errors

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix lint

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Add spark offline store components to test against current integration tests

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix lint

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Rename to pass checks

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix issues

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix type checking issues

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix lint

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Clean up print statements for first review

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix lint

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix flake 8 lint tests

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Add warnings for alpha version release

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Format

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Address review

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Address review

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix lint

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Add file store functionality

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* lint

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Add example feature repo

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Update data source creator

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Make cli work for feast init with spark

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Update the docs

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Clean up code

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Clean up more code

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Uncomment repo configs

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix setup.py

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Update dependencies

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix ci dependencies

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Screwed up rebase

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Screwed up rebase

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Screwed up rebase

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Realign with master

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix accidental changes

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Make type map change cleaner

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Address review comments

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix tests accidentally broken

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Add comments

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Reformat

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix logger

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Remove unused imports

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix imports

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix CI dependencies

Signed-off-by: Danny Chiao <danny@tecton.ai>

* Prefix destinations with project name

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Update comment

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix 3.8

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* temporary fix

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* rollback

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* update

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Update ci?

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Move third party to contrib

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix imports

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Remove third_party refactor

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Revert ci requirements and update comment in type map

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Revert 3.8-requirements

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Co-authored-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* ci: Update versions of helm charts + java pom files

Signed-off-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* feat: Event timestamps response (#2355)

* ability to get event timestamps from online response

Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com>

* fix event timestamp bugs

Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com>

* python formatting

Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com>

* optional param to retrieve event_timestamp in online_reponse

Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com>

* formatting

Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com>

* renaming param

Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* feat: Metadata changes & making data sources top level objects to power Feast UI (#2336)

* Squash commits for metadata changes

Signed-off-by: Danny Chiao <danny@tecton.ai>

* tests

Signed-off-by: Danny Chiao <danny@tecton.ai>

* Add more tests

Signed-off-by: Danny Chiao <danny@tecton.ai>

* lint

Signed-off-by: Danny Chiao <danny@tecton.ai>

* Add apply test

Signed-off-by: Danny Chiao <danny@tecton.ai>

* Add apply test

Signed-off-by: Danny Chiao <danny@tecton.ai>

* Add apply test

Signed-off-by: Danny Chiao <danny@tecton.ai>

* lint

Signed-off-by: Danny Chiao <danny@tecton.ai>

* fix bigquery source

Signed-off-by: Danny Chiao <danny@tecton.ai>

* fix test

Signed-off-by: Danny Chiao <danny@tecton.ai>

* fix spark source

Signed-off-by: Danny Chiao <danny@tecton.ai>

* fix spark source

Signed-off-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* ci: Remove buggy branch identification

Signed-off-by: Willem Pienaar <git@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* fix action and set go version

Signed-off-by: Achal Shah <achals@gmail.com>

* Complie python protos correctly

Signed-off-by: Achal Shah <achals@gmail.com>

* Set up go in the pr integration test workflow

Signed-off-by: Achal Shah <achals@gmail.com>

* Nuke go requirements for python ci (for the moment)

Signed-off-by: Achal Shah <achals@gmail.com>

* Update redis port

Signed-off-by: Achal Shah <achals@gmail.com>

* ci: Add version bump script

Signed-off-by: Willem Pienaar <git@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* ci: Fix broken release preparation

Signed-off-by: Willem Pienaar <git@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* ci: Add ci bot as committer and fix typo in dry-run config

Signed-off-by: feast-ci-bot <feast-ci-bot@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* chore(release): release 0.19.0

# [0.19.0](v0.18.0...v0.19.0) (2022-03-05)

### Bug Fixes

* Added additional value types to UI parser and removed references to registry-bq.json ([#2361](#2361)) ([d202d51](d202d51))
* Fix Redshift bug that stops waiting on statements after 5 minutes ([#2363](#2363)) ([74f887f](74f887f))
* Method _should_use_plan only returns true for local sqlite provider ([#2344](#2344)) ([fdb5f21](fdb5f21))
* Remove redis service to prevent more conflicts and add redis node to master_only ([#2354](#2354)) ([993616f](993616f))
* Rollback Redis-py to Redis-py-cluster ([#2347](#2347)) ([1ba86fb](1ba86fb))
* Update github workflow to prevent redis from overlapping ports. ([#2350](#2350)) ([c2a6c6c](c2a6c6c))

### Features

* Add owner field to Entity and rename labels to tags ([412d625](412d625))
* Allow all snowflake python connector connection methods to be available to Feast ([#2356](#2356)) ([ec7385c](ec7385c))
* Allowing password based authentication and SSL for Redis in Java feature server ([0af8adb](0af8adb))
* Event timestamps response ([#2355](#2355)) ([5481caf](5481caf))
* Feast Spark Offline Store ([#2349](#2349)) ([98b8d8d](98b8d8d))
* Initial merge of Web UI logic ([#2352](#2352)) ([ce3bc59](ce3bc59))
* Key ttl setting for redis online store ([#2341](#2341)) ([236a108](236a108))
* Metadata changes & making data sources top level objects to power Feast UI ([#2336](#2336)) ([43da230](43da230))

Signed-off-by: Achal Shah <achals@gmail.com>

* chore: Fix changelog and disable release comments

Signed-off-by: Willem Pienaar <git@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* fix: Fix unhashable Snowflake and Redshift sources

Signed-off-by: Willem Pienaar <git@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* chore: Remove needs-kind and release not requirement

Signed-off-by: Willem Pienaar <git@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* chore: Remove release note block

Signed-off-by: Willem Pienaar <git@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* ci: Uncheck prerelease status for patch releases

Signed-off-by: Willem Pienaar <git@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* fix: Fix default feast apply path without any extras (#2373)

* fix: Fix default feast apply path without any extras

Signed-off-by: Danny Chiao <danny@tecton.ai>

* revert removing ge

Signed-off-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* docs: Add short section about github actions (#2315)

* Add short section about github actions

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Move to contributing.md

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* fix: Use the correct dockerhub image tag when building feature servers (#2372)

* fix: Use the correct dockerhub image tag when building feature servers

Signed-off-by: Achal Shah <achals@gmail.com>

* Print attempt

Signed-off-by: Achal Shah <achals@gmail.com>

* Increase sleep interval

Signed-off-by: Achal Shah <achals@gmail.com>

* More logging and sleep

Signed-off-by: Achal Shah <achals@gmail.com>

* change logging to debug

Signed-off-by: Achal Shah <achals@gmail.com>

* Change the dockerimage name

Signed-off-by: Achal Shah <achals@gmail.com>

* Change version to have periods

Signed-off-by: Achal Shah <achals@gmail.com>

* Change version entirely

Signed-off-by: Achal Shah <achals@gmail.com>

* Undo unintended changes

Signed-off-by: Achal Shah <achals@gmail.com>

* Remove dead branch

Signed-off-by: Achal Shah <achals@gmail.com>

* compile go as part of python ci setup

Signed-off-by: Achal Shah <achals@gmail.com>

* set path with gopath/bin

Signed-off-by: Achal Shah <achals@gmail.com>

* Set path for commands correctly

Signed-off-by: Achal Shah <achals@gmail.com>

* Set up go for lint-go

Signed-off-by: Achal Shah <achals@gmail.com>

* Remove makefile stuff

Signed-off-by: Achal Shah <achals@gmail.com>

* Fix workflows and Makefile commands

Signed-off-by: Achal Shah <achals@gmail.com>

* protoc-gen-go-grpc

Signed-off-by: Achal Shah <achals@gmail.com>

* install go deps first

Signed-off-by: Achal Shah <achals@gmail.com>

* Compile go protos as well

Signed-off-by: Achal Shah <achals@gmail.com>

* Remove go mod tidy from the deps target

Signed-off-by: Achal Shah <achals@gmail.com>

* Better python tooling

Signed-off-by: Achal Shah <achals@gmail.com>

* Remove container

Signed-off-by: Achal Shah <achals@gmail.com>

* Upgrade lint go as well

Signed-off-by: Achal Shah <achals@gmail.com>

* Upgrade lint go as well

Signed-off-by: Achal Shah <achals@gmail.com>

* More logging for unit-go

Signed-off-by: Achal Shah <achals@gmail.com>

* Install using python setup.py develop

Signed-off-by: Achal Shah <achals@gmail.com>

* install-python-ci-dependencies

Signed-off-by: Achal Shah <achals@gmail.com>

* Skip tests that need a feature repo

Signed-off-by: Achal Shah <achals@gmail.com>

* Skip a scary forking test

Signed-off-by: Achal Shah <achals@gmail.com>

* Remove all the crusty port stuff and disable the lifecycle test

Signed-off-by: Achal Shah <achals@gmail.com>

* Remove more crusty port stuff

Signed-off-by: Achal Shah <achals@gmail.com>

* Update skip message

Signed-off-by: Achal Shah <achals@gmail.com>

* Introduce a sleep before reading values

Signed-off-by: Achal Shah <achals@gmail.com>

* Disable one more test configuration since background thread-based monitoring doesn't currently work

Signed-off-by: Achal Shah <achals@gmail.com>

* Disable the test entirely

Signed-off-by: Achal Shah <achals@gmail.com>

* ci: Add support for semantic release (#2332)

* Add support for semantic release

Signed-off-by: Willem Pienaar <git@willem.co>

* Fix typos

Signed-off-by: Willem Pienaar <git@willem.co>

* Require a Personal Access Token

Signed-off-by: Willem Pienaar <git@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* feat: Feast Spark Offline Store (#2349)

* State of feast

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Clean up changes

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix random incorrect changes

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix lint

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix build errors

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix lint

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Add spark offline store components to test against current integration tests

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix lint

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Rename to pass checks

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix issues

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix type checking issues

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix lint

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Clean up print statements for first review

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix lint

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix flake 8 lint tests

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Add warnings for alpha version release

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Format

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Address review

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Address review

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix lint

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Add file store functionality

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* lint

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Add example feature repo

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Update data source creator

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Make cli work for feast init with spark

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Update the docs

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Clean up code

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Clean up more code

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Uncomment repo configs

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix setup.py

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Update dependencies

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix ci dependencies

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Screwed up rebase

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Screwed up rebase

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Screwed up rebase

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Realign with master

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix accidental changes

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Make type map change cleaner

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Address review comments

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix tests accidentally broken

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Add comments

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Reformat

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix logger

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Remove unused imports

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix imports

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix CI dependencies

Signed-off-by: Danny Chiao <danny@tecton.ai>

* Prefix destinations with project name

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Update comment

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix 3.8

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* temporary fix

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* rollback

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* update

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Update ci?

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Move third party to contrib

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix imports

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Remove third_party refactor

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Revert ci requirements and update comment in type map

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Revert 3.8-requirements

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Co-authored-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* ci: Update versions of helm charts + java pom files

Signed-off-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* feat: Metadata changes & making data sources top level objects to power Feast UI (#2336)

* Squash commits for metadata changes

Signed-off-by: Danny Chiao <danny@tecton.ai>

* tests

Signed-off-by: Danny Chiao <danny@tecton.ai>

* Add more tests

Signed-off-by: Danny Chiao <danny@tecton.ai>

* lint

Signed-off-by: Danny Chiao <danny@tecton.ai>

* Add apply test

Signed-off-by: Danny Chiao <danny@tecton.ai>

* Add apply test

Signed-off-by: Danny Chiao <danny@tecton.ai>

* Add apply test

Signed-off-by: Danny Chiao <danny@tecton.ai>

* lint

Signed-off-by: Danny Chiao <danny@tecton.ai>

* fix bigquery source

Signed-off-by: Danny Chiao <danny@tecton.ai>

* fix test

Signed-off-by: Danny Chiao <danny@tecton.ai>

* fix spark source

Signed-off-by: Danny Chiao <danny@tecton.ai>

* fix spark source

Signed-off-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* ci: Remove buggy branch identification

Signed-off-by: Willem Pienaar <git@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* ci: Add version bump script

Signed-off-by: Willem Pienaar <git@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* ci: Fix broken release preparation

Signed-off-by: Willem Pienaar <git@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* ci: Add ci bot as committer and fix typo in dry-run config

Signed-off-by: feast-ci-bot <feast-ci-bot@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* chore(release): release 0.19.0

* Added additional value types to UI parser and removed references to registry-bq.json ([#2361](#2361)) ([d202d51](d202d51))
* Fix Redshift bug that stops waiting on statements after 5 minutes ([#2363](#2363)) ([74f887f](74f887f))
* Method _should_use_plan only returns true for local sqlite provider ([#2344](#2344)) ([fdb5f21](fdb5f21))
* Remove redis service to prevent more conflicts and add redis node to master_only ([#2354](#2354)) ([993616f](993616f))
* Rollback Redis-py to Redis-py-cluster ([#2347](#2347)) ([1ba86fb](1ba86fb))
* Update github workflow to prevent redis from overlapping ports. ([#2350](#2350)) ([c2a6c6c](c2a6c6c))

* Add owner field to Entity and rename labels to tags ([412d625](412d625))
* Allow all snowflake python connector connection methods to be available to Feast ([#2356](#2356)) ([ec7385c](ec7385c))
* Allowing password based authentication and SSL for Redis in Java feature server ([0af8adb](0af8adb))
* Event timestamps response ([#2355](#2355)) ([5481caf](5481caf))
* Feast Spark Offline Store ([#2349](#2349)) ([98b8d8d](98b8d8d))
* Initial merge of Web UI logic ([#2352](#2352)) ([ce3bc59](ce3bc59))
* Key ttl setting for redis online store ([#2341](#2341)) ([236a108](236a108))
* Metadata changes & making data sources top level objects to power Feast UI ([#2336](#2336)) ([43da230](43da230))

Signed-off-by: Achal Shah <achals@gmail.com>

* chore: Fix changelog and disable release comments

Signed-off-by: Willem Pienaar <git@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* ci: Uncheck prerelease status for patch releases

Signed-off-by: Willem Pienaar <git@willem.co>
Signed-off-by: Achal Shah <achals@gmail.com>

* fix: Fix default feast apply path without any extras (#2373)

* fix: Fix default feast apply path without any extras

Signed-off-by: Danny Chiao <danny@tecton.ai>

* revert removing ge

Signed-off-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* fix: Making a name for data sources not a breaking change (#2379)

* fix: Making a name for data sources not a breaking change

Signed-off-by: Danny Chiao <danny@tecton.ai>

* fix test

Signed-off-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* fix: Update some fields optional in UI parser (#2380)

Signed-off-by: Yun Nan Liu <yunnanl@twitter.com>

Co-authored-by: Yun Nan Liu <yunnanl@twitter.com>
Co-authored-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* fix: Fix spark docs (#2382)

* Fix spark docs

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix readme.md

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix md

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Co-authored-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* feat: Makefile for contrib for Issue #2364 (#2366)

* Add make file

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Fix makefile

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Clean up

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Add documentation

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Clean up

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* Update documentation

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

* update make target

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Co-authored-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Achal Shah <achals@gmail.com>

* Closes threadpool resources upon datastore online_write_batch completion (#2386)

Signed-off-by: Pamela Toman <ptoman@paloaltonetworks.com>
Signed-off-by: Achal Shah <achals@gmail.com>

* CR comments

Signed-off-by: Achal Shah <achals@gmail.com>

Co-authored-by: Tsotne Tabidze <tsotne@tecton.ai>
Co-authored-by: Willem Pienaar <willem@Willems-MacBook-Pro.local>
Co-authored-by: Ly Cao <lycao@Lys-MacBook-Pro.local>
Co-authored-by: Willem Pienaar <git@willem.co>
Co-authored-by: Ly Cao <lycao@lys-macbook-pro.myfiosgateway.com>
Co-authored-by: Felix Wang <wangfelix98@gmail.com>
Co-authored-by: Achal Shah <achals@gmail.com>
Co-authored-by: Kevin Zhang <kzhang@tecton.ai>
Co-authored-by: mickey-liu <mickey.y.liu@gmail.com>
Co-authored-by: Yun Nan Liu <yunnanl@twitter.com>
Co-authored-by: Danny Chiao <danny@tecton.ai>
Co-authored-by: Willem Pienaar <6728866+woop@users.noreply.github.com>
Co-authored-by: sfc-gh-madkins <82121043+sfc-gh-madkins@users.noreply.github.com>
Co-authored-by: Vitaly Sergeyev <vsergeyev@better.com>
Co-authored-by: feast-ci-bot <feast-ci-bot@willem.co>
Co-authored-by: ptoman-pa <95256508+ptoman-pa@users.noreply.github.com>
  • Loading branch information
@ctkhanhly @woop
@felixwang9817 @achals @kevjumba @mickey-liu @adchia @sfc-gh-madkins @feast-ci-bot @ptoman-pa
17 people authored Mar 8, 2022
1 parent 40593c6 commit d12e7ef
Show file tree
Hide file tree
Showing 110 changed files with 3,724 additions and 24,109 deletions.
27 changes: 23 additions & 4 deletions .github/workflows/linter.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@ jobs:
with:
python-version: "3.7"
architecture: x64
- name: Setup Go
id: setup-go
uses: actions/setup-go@v2
with:
go-version: 1.17.7
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1"
Expand All @@ -35,16 +40,30 @@ jobs:
- name: Install pip-tools
run: pip install pip-tools
- name: Install dependencies
run: make install-python-ci-dependencies
run: |
make compile-protos-go
make install-python-ci-dependencies
- name: Lint python
run: make lint-python

lint-go:
container: gcr.io/kf-feast/feast-ci:latest
runs-on: [ubuntu-latest]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Setup Go
id: setup-go
uses: actions/setup-go@v2
with:
go-version: 1.17.7
- name: Setup Python
id: setup-python
uses: actions/setup-python@v2
with:
python-version: 3.7
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1"
- name: Install dependencies
run: make install-go-ci-dependencies
run: make install-go-ci-dependencies
- name: Lint go
run: make lint-go
11 changes: 8 additions & 3 deletions .github/workflows/master_only.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,13 +54,14 @@ jobs:
docker push $ECR_REGISTRY/$ECR_REPOSITORY:${{ steps.image-tag.outputs.DOCKER_IMAGE_TAG }}
outputs:
DOCKER_IMAGE_TAG: ${{ steps.image-tag.outputs.DOCKER_IMAGE_TAG }}
integration-test-python:
integration-test-python-and-go:
needs: build-lambda-docker-image
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: [ 3.7, 3.8, 3.9 ]
go-version: [ 1.17.0 ]
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
Expand All @@ -73,6 +74,11 @@ jobs:
with:
python-version: ${{ matrix.python-version }}
architecture: x64
- name: Setup Go
id: setup-go
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go-version }}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v0
with:
Expand Down Expand Up @@ -113,13 +119,12 @@ jobs:
- name: Start Redis
uses: supercharge/redis-github-action@1.4.0
with:
redis-version: ${{ matrix.redis-version }}
redis-port: 12345
- name: Setup Redis Cluster
run: |
docker pull vishnunair/docker-redis-cluster:latest
docker run -d -p 6001:6379 -p 6002:6380 -p 6003:6381 -p 6004:6382 -p 6005:6383 -p 6006:6384 --name redis-cluster vishnunair/docker-redis-cluster
- name: Test python
- name: Test python and go
env:
FEAST_SERVER_DOCKER_IMAGE_TAG: ${{ needs.build-lambda-docker-image.outputs.DOCKER_IMAGE_TAG }}
FEAST_USAGE: "False"
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/pr_integration_tests.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,9 @@ jobs:
- name: Install pip-tools
run: pip install pip-tools
- name: Install dependencies
run: make install-python-ci-dependencies
run: |
make compile-protos-go
make install-python-ci-dependencies
- name: Setup Redis Cluster
run: |
docker pull vishnunair/docker-redis-cluster:latest
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,4 +40,4 @@ jobs:
- name: Release
if: github.event.inputs.dry_run == 'false'
run: |
npx -p @semantic-release/changelog -p @semantic-release/git -p @semantic-release/exec -p semantic-release semantic-release
npx -p @semantic-release/changelog -p @semantic-release/git -p @semantic-release/exec -p semantic-release semantic-release
23 changes: 21 additions & 2 deletions .github/workflows/unit_tests.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ jobs:
with:
python-version: ${{ matrix.python-version }}
architecture: x64
- name: Setup Go
id: setup-go
uses: actions/setup-go@v2
with:
go-version: 1.17.7
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1"
Expand All @@ -40,7 +45,9 @@ jobs:
- name: Install pip-tools
run: pip install pip-tools
- name: Install dependencies
run: make install-python-ci-dependencies
run: |
make compile-protos-go
make install-python-ci-dependencies
- name: Test Python
env:
SNOWFLAKE_CI_DEPLOYMENT: ${{ secrets.SNOWFLAKE_CI_DEPLOYMENT }}
Expand All @@ -61,9 +68,21 @@ jobs:

unit-test-go:
runs-on: ubuntu-latest
container: gcr.io/kf-feast/feast-ci:latest
steps:
- uses: actions/checkout@v2
- name: Setup Python
id: setup-python
uses: actions/setup-python@v2
with:
python-version: 3.7
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1"
- name: Setup Go
id: setup-go
uses: actions/setup-go@v2
with:
go-version: 1.17.7
- name: Install dependencies
run: make install-go-ci-dependencies
- name: Compile protos
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,7 @@ dmypy.json
sdk/python/docs/html
sdk/python/feast/protos/
sdk/go/protos/
go/protos/

#benchmarks
.benchmarks
Expand Down Expand Up @@ -218,3 +219,5 @@ ui/.vercel
**/yarn-debug.log*
**/yarn-error.log*

# Go subprocess binaries (built during feast pip package building)
sdk/python/feast/binaries/
1 change: 0 additions & 1 deletion .releaserc.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -74,4 +74,3 @@ module.exports = {
}],
]
}

42 changes: 25 additions & 17 deletions Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,6 @@

ROOT_DIR := $(shell dirname $(realpath $(firstword $(MAKEFILE_LIST))))
MVN := mvn -f java/pom.xml ${MAVEN_EXTRA_OPTS}
PROTO_TYPE_SUBDIRS = core serving types storage
PROTO_SERVICE_SUBDIRS = core serving
OS := linux
ifeq ($(shell uname -s), Darwin)
OS = osx
Expand All @@ -35,13 +33,11 @@ protos: compile-protos-go compile-protos-python compile-protos-docs

build: protos build-java build-docker build-html

install-ci-dependencies: install-python-ci-dependencies install-java-ci-dependencies install-go-ci-dependencies

# Python SDK

install-python-ci-dependencies:
install-python-ci-dependencies: install-go-ci-dependencies
cd sdk/python && python -m piptools sync requirements/py$(PYTHON)-ci-requirements.txt
cd sdk/python && python setup.py develop
cd sdk/python && COMPILE_GO=true python setup.py develop

lock-python-ci-dependencies:
cd sdk/python && python -m piptools compile -U --extra ci --output-file requirements/py$(PYTHON)-ci-requirements.txt
Expand All @@ -50,8 +46,7 @@ package-protos:
cp -r ${ROOT_DIR}/protos ${ROOT_DIR}/sdk/python/feast/protos

compile-protos-python:
@$(foreach dir,$(PROTO_TYPE_SUBDIRS),cd ${ROOT_DIR}/protos; python -m grpc_tools.protoc -I. --grpc_python_out=../sdk/python/feast/protos/ --python_out=../sdk/python/feast/protos/ --mypy_out=../sdk/python/feast/protos/ feast/$(dir)/*.proto;)
@$(foreach dir,$(PROTO_TYPE_SUBDIRS),grep -rli 'from feast.$(dir)' sdk/python/feast/protos | xargs -I@ sed -i.bak 's/from feast.$(dir)/from feast.protos.feast.$(dir)/g' @;)
python setup.py build_python_protos

install-python:
cd sdk/python && python -m piptools sync requirements/py$(PYTHON)-requirements.txt
Expand Down Expand Up @@ -81,6 +76,14 @@ test-python-universal-local:
test-python-universal:
FEAST_USAGE=False IS_TEST=True python -m pytest -n 8 --integration --universal sdk/python/tests

test-python-go-server:
go build -o ${ROOT_DIR}/sdk/python/feast/binaries/goserver github.com/feast-dev/feast/go/cmd/goserver
FEAST_USAGE=False IS_TEST=True python -m pytest -n 8 --integration --goserver sdk/python/tests

test-python-go-server-lifecycle:
go build -o ${ROOT_DIR}/sdk/python/feast/binaries/goserver github.com/feast-dev/feast/go/cmd/goserver
FEAST_USAGE=False IS_TEST=True python -m pytest -n 8 --integration --goserverlifecycle sdk/python/tests

format-python:
# Sort
cd ${ROOT_DIR}/sdk/python; python -m isort feast/ tests/
Expand Down Expand Up @@ -123,20 +126,25 @@ build-java-no-tests:
# Go SDK

install-go-ci-dependencies:
go get -u github.com/golang/protobuf/protoc-gen-go
go get -u golang.org/x/lint/golint
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26.0
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1.0

compile-protos-go: install-go-ci-dependencies
pip install grpcio-tools==1.34.0
python sdk/python/setup.py build_go_protos

compile-protos-go:
$(foreach dir,types serving core storage,cd ${ROOT_DIR}/protos; protoc -I/usr/local/include -I. --go_out=plugins=grpc,paths=source_relative:../sdk/go/protos feast/$(dir)/*.proto;)
compile-go-feature-server: compile-protos-go
go mod tidy
go build -o ${ROOT_DIR}/sdk/python/feast/binaries/goserver github.com/feast-dev/feast/go/cmd/goserver

test-go:
cd ${ROOT_DIR}/sdk/go; go test ./...
test-go: install-go-ci-dependencies
go test ./...

format-go:
cd ${ROOT_DIR}/sdk/go; gofmt -s -w *.go
gofmt -s -w go/

lint-go:
cd ${ROOT_DIR}/sdk/go; go vet
lint-go: compile-protos-go
go vet ./go/internal/feast ./go/cmd/goserver

# Docker

Expand Down
54 changes: 22 additions & 32 deletions go.mod
View file
Original file line number Diff line number Diff line change
@@ -1,39 +1,29 @@
module github.com/feast-dev/feast

go 1.17

require (
github.com/Masterminds/goutils v1.1.0 // indirect
github.com/Masterminds/semver v1.5.0 // indirect
github.com/Masterminds/sprig v2.22.0+incompatible // indirect
github.com/feast-dev/feast/sdk/go v0.0.0-20200516052424-09ff3dda724c // indirect
github.com/ghodss/yaml v1.0.0
github.com/gogo/protobuf v1.3.1 // indirect
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
github.com/golang/mock v1.2.0
github.com/go-redis/redis/v8 v8.11.4
github.com/golang/protobuf v1.5.2
github.com/google/go-cmp v0.5.5
github.com/huandu/xstrings v1.2.0 // indirect
github.com/lyft/protoc-gen-validate v0.1.0 // indirect
github.com/mitchellh/copystructure v1.0.0 // indirect
github.com/mitchellh/go-homedir v1.1.0
github.com/mwitkow/go-proto-validators v0.2.0 // indirect
github.com/pseudomuto/protoc-gen-doc v1.3.0 // indirect
github.com/pseudomuto/protokit v0.2.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/spf13/cobra v0.0.4
github.com/spf13/viper v1.4.0
github.com/woop/protoc-gen-doc v1.3.0 // indirect
go.opencensus.io v0.22.3 // indirect
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 // indirect
golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 // indirect
golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f
golang.org/x/tools v0.1.8 // indirect
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 // indirect
google.golang.org/grpc v1.29.1
google.golang.org/protobuf v1.27.1 // indirect
gopkg.in/russross/blackfriday.v2 v2.0.0 // indirect
gopkg.in/yaml.v2 v2.2.4
istio.io/gogo-genproto v0.0.0-20191212213402-78a529a42cd8 // indirect
istio.io/tools v0.0.0-20191228030621-c4eb6a11039c // indirect
github.com/google/uuid v1.1.2
github.com/kelseyhightower/envconfig v1.4.0
github.com/spaolacci/murmur3 v1.1.0
github.com/stretchr/testify v1.7.0
google.golang.org/grpc v1.44.0
google.golang.org/protobuf v1.27.1
)

go 1.13
require (
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/google/go-cmp v0.5.7 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
golang.org/x/net v0.0.0-20210428140749-89ef3d95e781 // indirect
golang.org/x/sys v0.0.0-20210510120138-977fb7262007 // indirect
golang.org/x/text v0.3.6 // indirect
google.golang.org/genproto v0.0.0-20220118154757-00ab72f36ad5 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c // indirect
)
Loading

0 comments on commit d12e7ef

Please sign in to comment.