From 94147b176bbb0562d5d417d7f4e4da36fc77f7ff Mon Sep 17 00:00:00 2001 From: Felix Wang Date: Wed, 13 Apr 2022 14:25:28 -0700 Subject: [PATCH 1/8] Add docs for Go feature server Signed-off-by: Felix Wang --- docs/SUMMARY.md | 1 + .../feature-servers/go-feature-server.md | 23 +++++++++++++++++++ 2 files changed, 24 insertions(+) create mode 100644 docs/reference/feature-servers/go-feature-server.md diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 90e40a24c7..af651c7e43 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -82,6 +82,7 @@ * [.feastignore](reference/feature-repository/feast-ignore.md) * [Feature servers](reference/feature-servers/README.md) * [Local feature server](reference/feature-servers/local-feature-server.md) + * [Go feature server](reference/feature-servers/go-feature-server.md) * [\[Alpha\] Data quality monitoring](reference/dqm.md) * [\[Alpha\] On demand feature view](reference/alpha-on-demand-feature-view.md) * [\[Alpha\] Stream ingestion](reference/alpha-stream-ingestion.md) diff --git a/docs/reference/feature-servers/go-feature-server.md b/docs/reference/feature-servers/go-feature-server.md new file mode 100644 index 0000000000..a1e1636d72 --- /dev/null +++ b/docs/reference/feature-servers/go-feature-server.md @@ -0,0 +1,23 @@ +# Go feature server + +## Overview + +The Go feature server is a Go implementation of the core feature serving logic, embedded in the Python SDK. It supports retrieval of feature references, feature services, and on demand feature views, and can be used either through the Python SDK or the [Python feature server](local-feature-server.md). + +The Go feature server currently only supports Redis as an online store; support for other online stores will be added soon. We also plan on adding support for the Java feature server (e.g. the capability to call into the Go feature server and execute Java UDFs). Initial benchmarks indicate that the Go feature server is significantly faster than the Python feature server. We plan to release a more comprehensive set of benchmarks. For more details, see the [RFC](https://docs.google.com/document/d/1Lgqv6eWYFJgQ7LA_jNeTh8NzOPhqI9kGTeyESRpNHnE). + +## Usage + +To enable the Go feature server, set `go_feature_server: True` in your `feature_store.yaml`. + +{% code title="feature_store.yaml" %} +```yaml +project: my_feature_repo +registry: data/registry.db +provider: local +online_store: + type: redis + connection_string: "localhost:6379" +go_feature_server: True +``` +{% endcode %} From 134212c746e13cef877be03422f6e725a66ff950 Mon Sep 17 00:00:00 2001 From: Kevin Zhang Date: Wed, 13 Apr 2022 16:51:41 -0700 Subject: [PATCH 2/8] Update go feature server docs Signed-off-by: Kevin Zhang --- docs/reference/feature-servers/go-feature-server.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/reference/feature-servers/go-feature-server.md b/docs/reference/feature-servers/go-feature-server.md index a1e1636d72..75cbfc640e 100644 --- a/docs/reference/feature-servers/go-feature-server.md +++ b/docs/reference/feature-servers/go-feature-server.md @@ -4,7 +4,9 @@ The Go feature server is a Go implementation of the core feature serving logic, embedded in the Python SDK. It supports retrieval of feature references, feature services, and on demand feature views, and can be used either through the Python SDK or the [Python feature server](local-feature-server.md). -The Go feature server currently only supports Redis as an online store; support for other online stores will be added soon. We also plan on adding support for the Java feature server (e.g. the capability to call into the Go feature server and execute Java UDFs). Initial benchmarks indicate that the Go feature server is significantly faster than the Python feature server. We plan to release a more comprehensive set of benchmarks. For more details, see the [RFC](https://docs.google.com/document/d/1Lgqv6eWYFJgQ7LA_jNeTh8NzOPhqI9kGTeyESRpNHnE). +Currently, the Go feature server only supports online serving and does not have an offline component including APIs to create feast feature repositories or apply configuration to the registry to facilitate online materialization. It also does not expose its own dedicated cli to perform feast actions. Currently, the Go feature server is only meant to expose an online serving API that can be called through the python SDK to facilitate faster online feature retrieval. + +The Go feature server currently only supports Redis and Sqlite as online stores; support for other online stores will be added soon. Initial benchmarks indicate that the Go feature server is significantly faster than the Python feature server. We plan to release a more comprehensive set of benchmarks. For more details, see the [RFC](https://docs.google.com/document/d/1Lgqv6eWYFJgQ7LA_jNeTh8NzOPhqI9kGTeyESRpNHnE). ## Usage @@ -21,3 +23,10 @@ online_store: go_feature_server: True ``` {% endcode %} + +## Future/Current Work + +The Go feature server online feature logging for Data Quality Monitoring is currently in development. More information can be found [here](https://docs.google.com/document/d/110F72d4NTv80p35wDSONxhhPBqWRwbZXG4f9mNEMd98/edit#heading=h.9gaqqtox9jg6). + +We also plan on adding support for the Java feature server (e.g. the capability to call into the Go feature server and execute Java UDFs). + From 89a85b96c651756dd37d7dc1d4fb0e4689e99c7e Mon Sep 17 00:00:00 2001 From: Kevin Zhang Date: Thu, 14 Apr 2022 10:49:34 -0700 Subject: [PATCH 3/8] Address review components Signed-off-by: Kevin Zhang --- .../feature-servers/go-feature-server.md | 20 +++++++++++-------- sdk/python/feast/repo_config.py | 2 +- sdk/python/setup.py | 2 +- .../integration_test_repo_config.py | 2 +- .../feature_repos/repo_configuration.py | 6 +++--- 5 files changed, 18 insertions(+), 14 deletions(-) diff --git a/docs/reference/feature-servers/go-feature-server.md b/docs/reference/feature-servers/go-feature-server.md index 75cbfc640e..0fbc6f5166 100644 --- a/docs/reference/feature-servers/go-feature-server.md +++ b/docs/reference/feature-servers/go-feature-server.md @@ -1,16 +1,20 @@ -# Go feature server +# Go-based Feature Retrieval ## Overview -The Go feature server is a Go implementation of the core feature serving logic, embedded in the Python SDK. It supports retrieval of feature references, feature services, and on demand feature views, and can be used either through the Python SDK or the [Python feature server](local-feature-server.md). +The Go Feature Retrieval component is a Go implementation of the core feature serving logic, embedded in the Python SDK. It supports retrieval of feature references, feature services, and on demand feature views, and can be used either through the Python SDK or the [Python feature server](local-feature-server.md). -Currently, the Go feature server only supports online serving and does not have an offline component including APIs to create feast feature repositories or apply configuration to the registry to facilitate online materialization. It also does not expose its own dedicated cli to perform feast actions. Currently, the Go feature server is only meant to expose an online serving API that can be called through the python SDK to facilitate faster online feature retrieval. +Currently, this component only supports online serving and does not have an offline component including APIs to create feast feature repositories or apply configuration to the registry to facilitate online materialization. It also does not expose its own dedicated cli to perform feast actions. Furthermore, this component is only meant to expose an online serving API that can be called through the python SDK to facilitate faster online feature retrieval. -The Go feature server currently only supports Redis and Sqlite as online stores; support for other online stores will be added soon. Initial benchmarks indicate that the Go feature server is significantly faster than the Python feature server. We plan to release a more comprehensive set of benchmarks. For more details, see the [RFC](https://docs.google.com/document/d/1Lgqv6eWYFJgQ7LA_jNeTh8NzOPhqI9kGTeyESRpNHnE). +The Go Feature Retrieval component currently only supports Redis and Sqlite as online stores; support for other online stores will be added soon. Initial benchmarks indicate that it is significantly faster than the Python feature server for online feature retrieval. We plan to release a more comprehensive set of benchmarks. For more details, see the [RFC](https://docs.google.com/document/d/1Lgqv6eWYFJgQ7LA_jNeTh8NzOPhqI9kGTeyESRpNHnE). + +## Installation + +To install, run `export COMPILE_GO=true` before you run `pip install feast`. ## Usage -To enable the Go feature server, set `go_feature_server: True` in your `feature_store.yaml`. +To enable the Go online feature retrieval component, set `go_feature_retrieval: True` in your `feature_store.yaml`. This will direct all online feature retrieval to Go instead of Python. This flag will be enabled by default in the future. {% code title="feature_store.yaml" %} ```yaml @@ -20,13 +24,13 @@ provider: local online_store: type: redis connection_string: "localhost:6379" -go_feature_server: True +go_feature_retrieval: True ``` {% endcode %} ## Future/Current Work -The Go feature server online feature logging for Data Quality Monitoring is currently in development. More information can be found [here](https://docs.google.com/document/d/110F72d4NTv80p35wDSONxhhPBqWRwbZXG4f9mNEMd98/edit#heading=h.9gaqqtox9jg6). +The Go feature retrieval online feature logging for Data Quality Monitoring is currently in development. More information can be found [here](https://docs.google.com/document/d/110F72d4NTv80p35wDSONxhhPBqWRwbZXG4f9mNEMd98/edit#heading=h.9gaqqtox9jg6). -We also plan on adding support for the Java feature server (e.g. the capability to call into the Go feature server and execute Java UDFs). +We also plan on adding support for the Java feature server (e.g. the capability to call into the Go component and execute Java UDFs). diff --git a/sdk/python/feast/repo_config.py b/sdk/python/feast/repo_config.py index a4a04b28b1..c86a42a8bd 100644 --- a/sdk/python/feast/repo_config.py +++ b/sdk/python/feast/repo_config.py @@ -118,7 +118,7 @@ class RepoConfig(FeastBaseModel): repo_path: Optional[Path] = None - go_feature_server: Optional[bool] = False + go_feature_retrieval: Optional[bool] = False def __init__(self, **data: Any): super().__init__(**data) diff --git a/sdk/python/setup.py b/sdk/python/setup.py index cc883da95e..0b6a1ce582 100644 --- a/sdk/python/setup.py +++ b/sdk/python/setup.py @@ -107,7 +107,7 @@ CI_REQUIRED = ( [ - "cryptography==3.3.2", + "cryptography==3.4.8", "flake8", "black==19.10b0", "isort>=5", diff --git a/sdk/python/tests/integration/feature_repos/integration_test_repo_config.py b/sdk/python/tests/integration/feature_repos/integration_test_repo_config.py index 99e8512007..f8cd66a619 100644 --- a/sdk/python/tests/integration/feature_repos/integration_test_repo_config.py +++ b/sdk/python/tests/integration/feature_repos/integration_test_repo_config.py @@ -27,7 +27,7 @@ class IntegrationTestRepoConfig: full_feature_names: bool = True infer_features: bool = False python_feature_server: bool = False - go_feature_server: bool = False + go_feature_retrieval: bool = False def __repr__(self) -> str: if not self.online_store_creator: diff --git a/sdk/python/tests/integration/feature_repos/repo_configuration.py b/sdk/python/tests/integration/feature_repos/repo_configuration.py index a8868a26f2..cc108b0a14 100644 --- a/sdk/python/tests/integration/feature_repos/repo_configuration.py +++ b/sdk/python/tests/integration/feature_repos/repo_configuration.py @@ -116,7 +116,7 @@ ), # Go implementation for online retrieval IntegrationTestRepoConfig( - online_store=REDIS_CONFIG, go_feature_server=True, + online_store=REDIS_CONFIG, go_feature_retrieval=True, ), # TODO(felixwang9817): Enable this test once https://github.com/feast-dev/feast/issues/2544 is resolved. # IntegrationTestRepoConfig( @@ -154,7 +154,7 @@ GO_REPO_CONFIGS = [ - IntegrationTestRepoConfig(online_store=REDIS_CONFIG, go_feature_server=True,), + IntegrationTestRepoConfig(online_store=REDIS_CONFIG, go_feature_retrieval=True,), ] @@ -423,7 +423,7 @@ def construct_test_environment( online_store=online_store, repo_path=repo_dir_name, feature_server=feature_server, - go_feature_server=test_repo_config.go_feature_server, + go_feature_retrieval=test_repo_config.go_feature_retrieval, ) # Create feature_store.yaml out of the config From cda50ab11973eca277ceb7317ddd62d2d63de781 Mon Sep 17 00:00:00 2001 From: Kevin Zhang Date: Thu, 14 Apr 2022 10:50:09 -0700 Subject: [PATCH 4/8] Fix Signed-off-by: Kevin Zhang --- .../{go-feature-server.md => go-feature-retrieval.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/reference/feature-servers/{go-feature-server.md => go-feature-retrieval.md} (100%) diff --git a/docs/reference/feature-servers/go-feature-server.md b/docs/reference/feature-servers/go-feature-retrieval.md similarity index 100% rename from docs/reference/feature-servers/go-feature-server.md rename to docs/reference/feature-servers/go-feature-retrieval.md From 8a09826947f79c4cbda11f110e1dee0dbd41b447 Mon Sep 17 00:00:00 2001 From: Kevin Zhang Date: Thu, 14 Apr 2022 11:05:41 -0700 Subject: [PATCH 5/8] Fix links Signed-off-by: Kevin Zhang --- docs/SUMMARY.md | 2 +- sdk/python/feast/feature_store.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index af651c7e43..e73996665e 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -82,7 +82,7 @@ * [.feastignore](reference/feature-repository/feast-ignore.md) * [Feature servers](reference/feature-servers/README.md) * [Local feature server](reference/feature-servers/local-feature-server.md) - * [Go feature server](reference/feature-servers/go-feature-server.md) + * [Go-based feature retrieval](reference/feature-servers/go-feature-retrieval.md) * [\[Alpha\] Data quality monitoring](reference/dqm.md) * [\[Alpha\] On demand feature view](reference/alpha-on-demand-feature-view.md) * [\[Alpha\] Stream ingestion](reference/alpha-stream-ingestion.md) diff --git a/sdk/python/feast/feature_store.py b/sdk/python/feast/feature_store.py index 1aa4cef602..33d297f3ca 100644 --- a/sdk/python/feast/feature_store.py +++ b/sdk/python/feast/feature_store.py @@ -1311,7 +1311,7 @@ def _get_online_features( } # If Go feature server is enabled, send request to it instead of going through regular Python logic - if self.config.go_feature_server: + if self.config.go_feature_retrieval: from feast.embedded_go.online_features_service import ( EmbeddedOnlineFeatureServer, ) From 3c177199cf296f22fc2165b3f6b2fc6112711ee1 Mon Sep 17 00:00:00 2001 From: Kevin Zhang Date: Thu, 14 Apr 2022 11:09:59 -0700 Subject: [PATCH 6/8] Revert Signed-off-by: Kevin Zhang --- sdk/python/setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdk/python/setup.py b/sdk/python/setup.py index 0b6a1ce582..cc883da95e 100644 --- a/sdk/python/setup.py +++ b/sdk/python/setup.py @@ -107,7 +107,7 @@ CI_REQUIRED = ( [ - "cryptography==3.4.8", + "cryptography==3.3.2", "flake8", "black==19.10b0", "isort>=5", From 86ac5baf5313a5f3f231623f2b3f31ec155da931 Mon Sep 17 00:00:00 2001 From: Kevin Zhang Date: Thu, 14 Apr 2022 11:27:25 -0700 Subject: [PATCH 7/8] Fix Signed-off-by: Kevin Zhang --- docs/reference/feature-servers/go-feature-retrieval.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/reference/feature-servers/go-feature-retrieval.md b/docs/reference/feature-servers/go-feature-retrieval.md index 0fbc6f5166..999a142c07 100644 --- a/docs/reference/feature-servers/go-feature-retrieval.md +++ b/docs/reference/feature-servers/go-feature-retrieval.md @@ -10,7 +10,9 @@ The Go Feature Retrieval component currently only supports Redis and Sqlite as o ## Installation -To install, run `export COMPILE_GO=true` before you run `pip install feast`. +As long as you are running macOS or linux x86 with python version 3.7-3.10, the go component comes pre-compiled when you run install feast. + +For developers, if you want to build from source, run `make compile-go-lib` to build and compile the go server. ## Usage From fbf4cca035506c07d35a622a217957c8c6226ec0 Mon Sep 17 00:00:00 2001 From: Felix Wang Date: Thu, 14 Apr 2022 15:06:35 -0700 Subject: [PATCH 8/8] Fix test config Signed-off-by: Felix Wang --- .../tests/integration/feature_repos/repo_configuration.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdk/python/tests/integration/feature_repos/repo_configuration.py b/sdk/python/tests/integration/feature_repos/repo_configuration.py index cc108b0a14..1db8a06b17 100644 --- a/sdk/python/tests/integration/feature_repos/repo_configuration.py +++ b/sdk/python/tests/integration/feature_repos/repo_configuration.py @@ -122,7 +122,7 @@ # IntegrationTestRepoConfig( # online_store=REDIS_CONFIG, # python_feature_server=True, - # go_feature_server=True, + # go_feature_retrieval=True, # ), ] )