feat: SQLAlchemy Registry Support #2734

achals · 2022-05-26T17:41:15Z

Signed-off-by: Achal Shah achals@gmail.com

What this PR does / why we need it:

This is the first pass for the proposal in https://docs.google.com/document/d/1tCZzClj3H8CfhJzccCytWK-bNDw_lkZk4e3fUbPYIP0/edit#.

Some caveats:

We don't expose an API to allow setting user metadta; this will come in a subsequent PR.
There's some methods that are not implemented in the Registry class, which will need to be implemented there as well as the SqlRegistry classes.

Which issue(s) this PR fixes:

Related to #2594.

codecov-commenter · 2022-05-26T17:59:44Z

Codecov Report

Merging #2734 (9e6d84b) into master (44a3f05) will increase coverage by 0.07%.
The diff coverage is 85.97%.

@@            Coverage Diff             @@
##           master    #2734      +/-   ##
==========================================
+ Coverage   80.40%   80.48%   +0.07%     
==========================================
  Files         171      173       +2     
  Lines       14571    14936     +365     
==========================================
+ Hits        11716    12021     +305     
- Misses       2855     2915      +60

Flag	Coverage Δ
integrationtests	`70.53% <57.50%> (-0.35%)`	⬇️
unittests	`59.35% <77.43%> (+0.32%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
setup.py	`0.00% <0.00%> (ø)`
sdk/python/feast/infra/registry_stores/sql.py	`79.11% <79.11%> (ø)`
sdk/python/feast/feature_store.py	`86.76% <80.00%> (-0.08%)`	⬇️
...ests/integration/registration/test_sql_registry.py	`93.20% <93.20%> (ø)`
sdk/python/feast/repo_config.py	`88.88% <100.00%> (+0.09%)`	⬆️
sdk/python/feast/stream_feature_view.py	`90.90% <0.00%> (-4.60%)`	⬇️
sdk/python/feast/repo_operations.py	`52.85% <0.00%> (-1.11%)`	⬇️
sdk/python/feast/registry.py	`81.44% <0.00%> (-0.17%)`	⬇️
sdk/python/feast/infra/online_stores/sqlite.py	`94.48% <0.00%> (ø)`
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 44a3f05...9e6d84b. Read the comment docs.

Signed-off-by: Achal Shah <achals@gmail.com>

.github/workflows/unit_tests.yml

sdk/python/feast/infra/registry_stores/sql.py

sdk/python/tests/integration/registration/test_sql_registry.py

Signed-off-by: Achal Shah <achals@gmail.com>

felixwang9817

/lgtm

feast-ci-bot · 2022-06-03T17:27:06Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: achals, felixwang9817

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [achals,felixwang9817]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sdk/python/feast/infra/registry_stores/sql.py

kevjumba · 2022-06-03T17:18:30Z

sdk/python/feast/infra/registry_stores/sql.py

+    def apply_feature_view(
+        self, feature_view: BaseFeatureView, project: str, commit: bool = True
+    ):
+        if isinstance(feature_view, FeatureView):


can u add a comment here that sfvs will be handled as fvs for the time being

kevjumba · 2022-06-03T17:20:58Z

sdk/python/feast/repo_config.py

@@ -82,6 +82,10 @@ class Config:
 class RegistryConfig(FeastBaseModel):
    """Metadata Store Configuration. Configuration that relates to reading from and writing to the Feast registry."""

+    registry_type: StrictStr = "file"


why do we have this if registry store type does the same thing

Store type doesn't do the same thing cause it abstracts over where the registry proto is saved. It's API is basically save_registry_proto and get_registry_proto which is not as fine grained as the individual Registry operations.

kevjumba · 2022-06-03T17:26:39Z

sdk/python/tests/integration/registration/test_sql_registry.py

+    registry_batch_source = sql_registry.list_data_sources(project)[0]
+    assert registry_batch_source == batch_source
+
+    sql_registry.teardown()


mybe also an integration test that tests apply?

Is there a reason we need an integration test equivalent? Since the tests are running against a local instance of pg/mysql?

Signed-off-by: Achal Shah <achals@gmail.com>

kevjumba · 2022-06-03T17:45:50Z

/lgtm

# [0.22.0](v0.21.0...v0.22.0) (2022-06-21) ### Bug Fixes * Add columns for user metadata in the tables ([#2760](#2760)) ([269055e](269055e)) * Add project columns in the SQL Registry ([#2784](#2784)) ([336fdd1](336fdd1)) * Add S3FS dependency (which Dask depends on for S3 files) ([#2701](#2701)) ([5d6fa94](5d6fa94)) * Bugfixes for how registry is loaded ([#2768](#2768)) ([ecb8b2a](ecb8b2a)) * Conversion of null timestamp from proto to python ([#2814](#2814)) ([cb23648](cb23648)) * Correct feature statuses during feature logging test ([#2709](#2709)) ([cebf609](cebf609)) * Dynamodb drops missing entities when batching ([#2802](#2802)) ([a2e9209](a2e9209)) * Enable faulthandler and disable flaky tests ([#2815](#2815)) ([4934d84](4934d84)) * Fix broken roadmap links ([#2690](#2690)) ([b3ba8aa](b3ba8aa)) * Fix bugs in applying stream feature view and retrieving online features ([#2754](#2754)) ([d024e5e](d024e5e)) * Fix Feast UI failure with new way of specifying entities ([#2773](#2773)) ([0d1ac01](0d1ac01)) * Fix feature view __getitem__ for feature services ([#2769](#2769)) ([88cc47d](88cc47d)) * Fix issue when user specifies a port for feast ui ([#2692](#2692)) ([1c621fe](1c621fe)) * Fix on demand feature view crash from inference when it uses df.apply ([#2713](#2713)) ([c5539fd](c5539fd)) * Fix SparkKafkaProcessor `query_timeout` parameter ([#2789](#2789)) ([a8d282d](a8d282d)) * Fixed custom S3 endpoint read fail ([#2786](#2786)) ([6fec431](6fec431)) * Hydrate infra object in the sql registry proto() method ([#2782](#2782)) ([452dcd3](452dcd3)) * Implement apply_materialization and infra methods in sql registry ([#2775](#2775)) ([4ed107c](4ed107c)) * Minor refactor to format exception message ([#2764](#2764)) ([da763c6](da763c6)) * Python server is not correctly starting in integration tests ([#2706](#2706)) ([7583a0b](7583a0b)) * Random port allocation for python server in tests ([#2710](#2710)) ([dee8090](dee8090)) * Refactor test to reuse LocalRegistryFile ([#2763](#2763)) ([4339c0a](4339c0a)) * Support push sources in stream feature views ([#2704](#2704)) ([0d60eaa](0d60eaa)) * Update udf tests and add base functions to streaming fcos and fix some nonetype errors ([#2776](#2776)) ([331a214](331a214)) ### Features * Add file write_to_offline_store functionality ([#2808](#2808)) ([c0e2ad7](c0e2ad7)) * Add http endpoint to the Go feature server ([#2658](#2658)) ([3347a57](3347a57)) * Add StreamProcessor and SparkKafkaProcessor as contrib ([#2777](#2777)) ([83ab682](83ab682)) * Added Spark support for Delta and Avro ([#2757](#2757)) ([7d16516](7d16516)) * CLI interface for validation of logged features ([#2718](#2718)) ([c8b11b3](c8b11b3)) * Enable stream feature view materialization ([#2798](#2798)) ([a06700d](a06700d)) * Enable stream feature view materialization ([#2807](#2807)) ([7d57724](7d57724)) * Scaffold for unified push api ([#2796](#2796)) ([1bd0930](1bd0930)) * SQLAlchemy Registry Support ([#2734](#2734)) ([b3fe39c](b3fe39c)) * Stream Feature View FCOS ([#2750](#2750)) ([0cf3c92](0cf3c92)) * Update stream fcos to have watermark and sliding interval ([#2765](#2765)) ([3256952](3256952)) * Validating logged features via Python SDK ([#2640](#2640)) ([2874fc5](2874fc5))

# [0.22.0](v0.21.0...v0.22.0) (2022-06-23) ### Bug Fixes * Add columns for user metadata in the tables ([#2760](#2760)) ([269055e](269055e)) * Add project columns in the SQL Registry ([#2784](#2784)) ([336fdd1](336fdd1)) * Add S3FS dependency (which Dask depends on for S3 files) ([#2701](#2701)) ([5d6fa94](5d6fa94)) * Bugfixes for how registry is loaded ([#2768](#2768)) ([ecb8b2a](ecb8b2a)) * Conversion of null timestamp from proto to python ([#2814](#2814)) ([cb23648](cb23648)) * Correct feature statuses during feature logging test ([#2709](#2709)) ([cebf609](cebf609)) * Dynamodb drops missing entities when batching ([#2802](#2802)) ([a2e9209](a2e9209)) * Enable faulthandler and disable flaky tests ([#2815](#2815)) ([4934d84](4934d84)) * Explicitly translate errors when instantiating the go fs ([#2842](#2842)) ([7a2c4cd](7a2c4cd)) * Fix broken roadmap links ([#2690](#2690)) ([b3ba8aa](b3ba8aa)) * Fix bugs in applying stream feature view and retrieving online features ([#2754](#2754)) ([d024e5e](d024e5e)) * Fix Feast UI failure with new way of specifying entities ([#2773](#2773)) ([0d1ac01](0d1ac01)) * Fix feature view __getitem__ for feature services ([#2769](#2769)) ([88cc47d](88cc47d)) * Fix issue when user specifies a port for feast ui ([#2692](#2692)) ([1c621fe](1c621fe)) * Fix on demand feature view crash from inference when it uses df.apply ([#2713](#2713)) ([c5539fd](c5539fd)) * Fix SparkKafkaProcessor `query_timeout` parameter ([#2789](#2789)) ([a8d282d](a8d282d)) * Fixed custom S3 endpoint read fail ([#2786](#2786)) ([6fec431](6fec431)) * Hydrate infra object in the sql registry proto() method ([#2782](#2782)) ([452dcd3](452dcd3)) * Implement apply_materialization and infra methods in sql registry ([#2775](#2775)) ([4ed107c](4ed107c)) * Minor refactor to format exception message ([#2764](#2764)) ([da763c6](da763c6)) * Prefer installing gopy from feast's fork as opposed to upstream ([#2839](#2839)) ([34c997d](34c997d)) * Python server is not correctly starting in integration tests ([#2706](#2706)) ([7583a0b](7583a0b)) * Random port allocation for python server in tests ([#2710](#2710)) ([dee8090](dee8090)) * Refactor test to reuse LocalRegistryFile ([#2763](#2763)) ([4339c0a](4339c0a)) * Support push sources in stream feature views ([#2704](#2704)) ([0d60eaa](0d60eaa)) * Update roadmap with stream feature view rfc ([#2824](#2824)) ([fc8f890](fc8f890)) * Update udf tests and add base functions to streaming fcos and fix some nonetype errors ([#2776](#2776)) ([331a214](331a214)) ### Features * Add file write_to_offline_store functionality ([#2808](#2808)) ([c0e2ad7](c0e2ad7)) * Add http endpoint to the Go feature server ([#2658](#2658)) ([3347a57](3347a57)) * Add StreamProcessor and SparkKafkaProcessor as contrib ([#2777](#2777)) ([83ab682](83ab682)) * Added Spark support for Delta and Avro ([#2757](#2757)) ([7d16516](7d16516)) * CLI interface for validation of logged features ([#2718](#2718)) ([c8b11b3](c8b11b3)) * Enable stream feature view materialization ([#2798](#2798)) ([a06700d](a06700d)) * Enable stream feature view materialization ([#2807](#2807)) ([7d57724](7d57724)) * Implement `offline_write_batch` for BigQuery and Snowflake ([#2840](#2840)) ([97444e4](97444e4)) * Offline push endpoint for pushing to offline stores ([#2837](#2837)) ([a88cd30](a88cd30)) * Push to Redshift batch source offline store directly ([#2819](#2819)) ([5748a8b](5748a8b)) * Scaffold for unified push api ([#2796](#2796)) ([1bd0930](1bd0930)) * SQLAlchemy Registry Support ([#2734](#2734)) ([b3fe39c](b3fe39c)) * Stream Feature View FCOS ([#2750](#2750)) ([0cf3c92](0cf3c92)) * Update stream fcos to have watermark and sliding interval ([#2765](#2765)) ([3256952](3256952)) * Validating logged features via Python SDK ([#2640](#2640)) ([2874fc5](2874fc5))

# [0.22.0](v0.21.0...v0.22.0) (2022-06-24) ### Bug Fixes * Add columns for user metadata in the tables ([#2760](#2760)) ([269055e](269055e)) * Add project columns in the SQL Registry ([#2784](#2784)) ([336fdd1](336fdd1)) * Add S3FS dependency (which Dask depends on for S3 files) ([#2701](#2701)) ([5d6fa94](5d6fa94)) * Bugfixes for how registry is loaded ([#2768](#2768)) ([ecb8b2a](ecb8b2a)) * Conversion of null timestamp from proto to python ([#2814](#2814)) ([cb23648](cb23648)) * Correct feature statuses during feature logging test ([#2709](#2709)) ([cebf609](cebf609)) * Dynamodb drops missing entities when batching ([#2802](#2802)) ([a2e9209](a2e9209)) * Enable faulthandler and disable flaky tests ([#2815](#2815)) ([4934d84](4934d84)) * Explicitly translate errors when instantiating the go fs ([#2842](#2842)) ([7a2c4cd](7a2c4cd)) * Fix broken roadmap links ([#2690](#2690)) ([b3ba8aa](b3ba8aa)) * Fix bugs in applying stream feature view and retrieving online features ([#2754](#2754)) ([d024e5e](d024e5e)) * Fix Feast UI failure with new way of specifying entities ([#2773](#2773)) ([0d1ac01](0d1ac01)) * Fix feature view __getitem__ for feature services ([#2769](#2769)) ([88cc47d](88cc47d)) * Fix issue when user specifies a port for feast ui ([#2692](#2692)) ([1c621fe](1c621fe)) * Fix on demand feature view crash from inference when it uses df.apply ([#2713](#2713)) ([c5539fd](c5539fd)) * Fix SparkKafkaProcessor `query_timeout` parameter ([#2789](#2789)) ([a8d282d](a8d282d)) * Fixed custom S3 endpoint read fail ([#2786](#2786)) ([6fec431](6fec431)) * Hydrate infra object in the sql registry proto() method ([#2782](#2782)) ([452dcd3](452dcd3)) * Implement apply_materialization and infra methods in sql registry ([#2775](#2775)) ([4ed107c](4ed107c)) * Minor refactor to format exception message ([#2764](#2764)) ([da763c6](da763c6)) * Prefer installing gopy from feast's fork as opposed to upstream ([#2839](#2839)) ([34c997d](34c997d)) * Python server is not correctly starting in integration tests ([#2706](#2706)) ([7583a0b](7583a0b)) * Random port allocation for python server in tests ([#2710](#2710)) ([dee8090](dee8090)) * Refactor test to reuse LocalRegistryFile ([#2763](#2763)) ([4339c0a](4339c0a)) * Revert "chore(release): release 0.22.0" ([#2852](#2852)) ([e6a4636](e6a4636)) * Support push sources in stream feature views ([#2704](#2704)) ([0d60eaa](0d60eaa)) * Update roadmap with stream feature view rfc ([#2824](#2824)) ([fc8f890](fc8f890)) * Update udf tests and add base functions to streaming fcos and fix some nonetype errors ([#2776](#2776)) ([331a214](331a214)) ### Features * Add file write_to_offline_store functionality ([#2808](#2808)) ([c0e2ad7](c0e2ad7)) * Add http endpoint to the Go feature server ([#2658](#2658)) ([3347a57](3347a57)) * Add StreamProcessor and SparkKafkaProcessor as contrib ([#2777](#2777)) ([83ab682](83ab682)) * Added Spark support for Delta and Avro ([#2757](#2757)) ([7d16516](7d16516)) * CLI interface for validation of logged features ([#2718](#2718)) ([c8b11b3](c8b11b3)) * Enable stream feature view materialization ([#2798](#2798)) ([a06700d](a06700d)) * Enable stream feature view materialization ([#2807](#2807)) ([7d57724](7d57724)) * Implement `offline_write_batch` for BigQuery and Snowflake ([#2840](#2840)) ([97444e4](97444e4)) * Offline push endpoint for pushing to offline stores ([#2837](#2837)) ([a88cd30](a88cd30)) * Push to Redshift batch source offline store directly ([#2819](#2819)) ([5748a8b](5748a8b)) * Scaffold for unified push api ([#2796](#2796)) ([1bd0930](1bd0930)) * SQLAlchemy Registry Support ([#2734](#2734)) ([b3fe39c](b3fe39c)) * Stream Feature View FCOS ([#2750](#2750)) ([0cf3c92](0cf3c92)) * Update stream fcos to have watermark and sliding interval ([#2765](#2765)) ([3256952](3256952)) * Validating logged features via Python SDK ([#2640](#2640)) ([2874fc5](2874fc5))

# [0.22.0](v0.21.0...v0.22.0) (2022-06-24) ### Bug Fixes * Add columns for user metadata in the tables ([#2760](#2760)) ([269055e](269055e)) * Add project columns in the SQL Registry ([#2784](#2784)) ([336fdd1](336fdd1)) * Add S3FS dependency (which Dask depends on for S3 files) ([#2701](#2701)) ([5d6fa94](5d6fa94)) * Bugfixes for how registry is loaded ([#2768](#2768)) ([ecb8b2a](ecb8b2a)) * Conversion of null timestamp from proto to python ([#2814](#2814)) ([cb23648](cb23648)) * Correct feature statuses during feature logging test ([#2709](#2709)) ([cebf609](cebf609)) * Correctly generate projects-list.json when calling feast ui and using postgres as a source ([#2845](#2845)) ([bee8076](bee8076)) * Dynamodb drops missing entities when batching ([#2802](#2802)) ([a2e9209](a2e9209)) * Enable faulthandler and disable flaky tests ([#2815](#2815)) ([4934d84](4934d84)) * Explicitly translate errors when instantiating the go fs ([#2842](#2842)) ([7a2c4cd](7a2c4cd)) * Fix broken roadmap links ([#2690](#2690)) ([b3ba8aa](b3ba8aa)) * Fix bugs in applying stream feature view and retrieving online features ([#2754](#2754)) ([d024e5e](d024e5e)) * Fix Feast UI failure with new way of specifying entities ([#2773](#2773)) ([0d1ac01](0d1ac01)) * Fix feature view __getitem__ for feature services ([#2769](#2769)) ([88cc47d](88cc47d)) * Fix issue when user specifies a port for feast ui ([#2692](#2692)) ([1c621fe](1c621fe)) * Fix on demand feature view crash from inference when it uses df.apply ([#2713](#2713)) ([c5539fd](c5539fd)) * Fix SparkKafkaProcessor `query_timeout` parameter ([#2789](#2789)) ([a8d282d](a8d282d)) * Fixed custom S3 endpoint read fail ([#2786](#2786)) ([6fec431](6fec431)) * Hydrate infra object in the sql registry proto() method ([#2782](#2782)) ([452dcd3](452dcd3)) * Implement apply_materialization and infra methods in sql registry ([#2775](#2775)) ([4ed107c](4ed107c)) * Minor refactor to format exception message ([#2764](#2764)) ([da763c6](da763c6)) * Prefer installing gopy from feast's fork as opposed to upstream ([#2839](#2839)) ([34c997d](34c997d)) * Python server is not correctly starting in integration tests ([#2706](#2706)) ([7583a0b](7583a0b)) * Random port allocation for python server in tests ([#2710](#2710)) ([dee8090](dee8090)) * Refactor test to reuse LocalRegistryFile ([#2763](#2763)) ([4339c0a](4339c0a)) * Revert "chore(release): release 0.22.0" ([#2852](#2852)) ([e6a4636](e6a4636)) * Support push sources in stream feature views ([#2704](#2704)) ([0d60eaa](0d60eaa)) * Update roadmap with stream feature view rfc ([#2824](#2824)) ([fc8f890](fc8f890)) * Update udf tests and add base functions to streaming fcos and fix some nonetype errors ([#2776](#2776)) ([331a214](331a214)) ### Features * Add feast repo-upgrade for automated repo upgrades ([#2733](#2733)) ([a3304d4](a3304d4)) * Add file write_to_offline_store functionality ([#2808](#2808)) ([c0e2ad7](c0e2ad7)) * Add http endpoint to the Go feature server ([#2658](#2658)) ([3347a57](3347a57)) * Add StreamProcessor and SparkKafkaProcessor as contrib ([#2777](#2777)) ([83ab682](83ab682)) * Added Spark support for Delta and Avro ([#2757](#2757)) ([7d16516](7d16516)) * CLI interface for validation of logged features ([#2718](#2718)) ([c8b11b3](c8b11b3)) * Enable stream feature view materialization ([#2798](#2798)) ([a06700d](a06700d)) * Enable stream feature view materialization ([#2807](#2807)) ([7d57724](7d57724)) * Implement `offline_write_batch` for BigQuery and Snowflake ([#2840](#2840)) ([97444e4](97444e4)) * Offline push endpoint for pushing to offline stores ([#2837](#2837)) ([a88cd30](a88cd30)) * Push to Redshift batch source offline store directly ([#2819](#2819)) ([5748a8b](5748a8b)) * Scaffold for unified push api ([#2796](#2796)) ([1bd0930](1bd0930)) * SQLAlchemy Registry Support ([#2734](#2734)) ([b3fe39c](b3fe39c)) * Stream Feature View FCOS ([#2750](#2750)) ([0cf3c92](0cf3c92)) * Update stream fcos to have watermark and sliding interval ([#2765](#2765)) ([3256952](3256952)) * Validating logged features via Python SDK ([#2640](#2640)) ([2874fc5](2874fc5))

# [0.22.0](v0.21.0...v0.22.0) (2022-06-25) ### Bug Fixes * Add columns for user metadata in the tables ([#2760](#2760)) ([269055e](269055e)) * Add project columns in the SQL Registry ([#2784](#2784)) ([336fdd1](336fdd1)) * Add S3FS dependency (which Dask depends on for S3 files) ([#2701](#2701)) ([5d6fa94](5d6fa94)) * Bugfixes for how registry is loaded ([#2768](#2768)) ([ecb8b2a](ecb8b2a)) * Conversion of null timestamp from proto to python ([#2814](#2814)) ([cb23648](cb23648)) * Correct feature statuses during feature logging test ([#2709](#2709)) ([cebf609](cebf609)) * Correctly generate projects-list.json when calling feast ui and using postgres as a source ([#2845](#2845)) ([bee8076](bee8076)) * Dynamodb drops missing entities when batching ([#2802](#2802)) ([a2e9209](a2e9209)) * Enable faulthandler and disable flaky tests ([#2815](#2815)) ([4934d84](4934d84)) * Explicitly translate errors when instantiating the go fs ([#2842](#2842)) ([7a2c4cd](7a2c4cd)) * Fix broken roadmap links ([#2690](#2690)) ([b3ba8aa](b3ba8aa)) * Fix bugs in applying stream feature view and retrieving online features ([#2754](#2754)) ([d024e5e](d024e5e)) * Fix Feast UI failure with new way of specifying entities ([#2773](#2773)) ([0d1ac01](0d1ac01)) * Fix feature view __getitem__ for feature services ([#2769](#2769)) ([88cc47d](88cc47d)) * Fix issue when user specifies a port for feast ui ([#2692](#2692)) ([1c621fe](1c621fe)) * Fix on demand feature view crash from inference when it uses df.apply ([#2713](#2713)) ([c5539fd](c5539fd)) * Fix SparkKafkaProcessor `query_timeout` parameter ([#2789](#2789)) ([a8d282d](a8d282d)) * Fixed custom S3 endpoint read fail ([#2786](#2786)) ([6fec431](6fec431)) * Hydrate infra object in the sql registry proto() method ([#2782](#2782)) ([452dcd3](452dcd3)) * Implement apply_materialization and infra methods in sql registry ([#2775](#2775)) ([4ed107c](4ed107c)) * Minor refactor to format exception message ([#2764](#2764)) ([da763c6](da763c6)) * Prefer installing gopy from feast's fork as opposed to upstream ([#2839](#2839)) ([34c997d](34c997d)) * Python server is not correctly starting in integration tests ([#2706](#2706)) ([7583a0b](7583a0b)) * Random port allocation for python server in tests ([#2710](#2710)) ([dee8090](dee8090)) * Refactor test to reuse LocalRegistryFile ([#2763](#2763)) ([4339c0a](4339c0a)) * Revert "chore(release): release 0.22.0" ([#2852](#2852)) ([e6a4636](e6a4636)) * Support push sources in stream feature views ([#2704](#2704)) ([0d60eaa](0d60eaa)) * Update roadmap with stream feature view rfc ([#2824](#2824)) ([fc8f890](fc8f890)) * Update udf tests and add base functions to streaming fcos and fix some nonetype errors ([#2776](#2776)) ([331a214](331a214)) ### Features * Add feast repo-upgrade for automated repo upgrades ([#2733](#2733)) ([a3304d4](a3304d4)) * Add file write_to_offline_store functionality ([#2808](#2808)) ([c0e2ad7](c0e2ad7)) * Add http endpoint to the Go feature server ([#2658](#2658)) ([3347a57](3347a57)) * Add StreamProcessor and SparkKafkaProcessor as contrib ([#2777](#2777)) ([83ab682](83ab682)) * Added Spark support for Delta and Avro ([#2757](#2757)) ([7d16516](7d16516)) * CLI interface for validation of logged features ([#2718](#2718)) ([c8b11b3](c8b11b3)) * Enable stream feature view materialization ([#2798](#2798)) ([a06700d](a06700d)) * Enable stream feature view materialization ([#2807](#2807)) ([7d57724](7d57724)) * Implement `offline_write_batch` for BigQuery and Snowflake ([#2840](#2840)) ([97444e4](97444e4)) * Offline push endpoint for pushing to offline stores ([#2837](#2837)) ([a88cd30](a88cd30)) * Push to Redshift batch source offline store directly ([#2819](#2819)) ([5748a8b](5748a8b)) * Scaffold for unified push api ([#2796](#2796)) ([1bd0930](1bd0930)) * SQLAlchemy Registry Support ([#2734](#2734)) ([b3fe39c](b3fe39c)) * Stream Feature View FCOS ([#2750](#2750)) ([0cf3c92](0cf3c92)) * Update stream fcos to have watermark and sliding interval ([#2765](#2765)) ([3256952](3256952)) * Validating logged features via Python SDK ([#2640](#2640)) ([2874fc5](2874fc5))

# [0.22.0](v0.21.0...v0.22.0) (2022-06-28) ### Bug Fixes * Add columns for user metadata in the tables ([#2760](#2760)) ([269055e](269055e)) * Add project columns in the SQL Registry ([#2784](#2784)) ([336fdd1](336fdd1)) * Add S3FS dependency (which Dask depends on for S3 files) ([#2701](#2701)) ([5d6fa94](5d6fa94)) * Bugfixes for how registry is loaded ([#2768](#2768)) ([ecb8b2a](ecb8b2a)) * Conversion of null timestamp from proto to python ([#2814](#2814)) ([cb23648](cb23648)) * Correct feature statuses during feature logging test ([#2709](#2709)) ([cebf609](cebf609)) * Correctly generate projects-list.json when calling feast ui and using postgres as a source ([#2845](#2845)) ([bee8076](bee8076)) * Dynamodb drops missing entities when batching ([#2802](#2802)) ([a2e9209](a2e9209)) * Enable faulthandler and disable flaky tests ([#2815](#2815)) ([4934d84](4934d84)) * Explicitly translate errors when instantiating the go fs ([#2842](#2842)) ([7a2c4cd](7a2c4cd)) * Fix broken roadmap links ([#2690](#2690)) ([b3ba8aa](b3ba8aa)) * Fix bugs in applying stream feature view and retrieving online features ([#2754](#2754)) ([d024e5e](d024e5e)) * Fix Feast UI failure with new way of specifying entities ([#2773](#2773)) ([0d1ac01](0d1ac01)) * Fix feature view __getitem__ for feature services ([#2769](#2769)) ([88cc47d](88cc47d)) * Fix issue when user specifies a port for feast ui ([#2692](#2692)) ([1c621fe](1c621fe)) * Fix on demand feature view crash from inference when it uses df.apply ([#2713](#2713)) ([c5539fd](c5539fd)) * Fix SparkKafkaProcessor `query_timeout` parameter ([#2789](#2789)) ([a8d282d](a8d282d)) * Fixed custom S3 endpoint read fail ([#2786](#2786)) ([6fec431](6fec431)) * Go install gopy instead using go mod tidy ([#2863](#2863)) ([2f2b519](2f2b519)) * Hydrate infra object in the sql registry proto() method ([#2782](#2782)) ([452dcd3](452dcd3)) * Implement apply_materialization and infra methods in sql registry ([#2775](#2775)) ([4ed107c](4ed107c)) * Minor refactor to format exception message ([#2764](#2764)) ([da763c6](da763c6)) * Prefer installing gopy from feast's fork as opposed to upstream ([#2839](#2839)) ([34c997d](34c997d)) * Python server is not correctly starting in integration tests ([#2706](#2706)) ([7583a0b](7583a0b)) * Random port allocation for python server in tests ([#2710](#2710)) ([dee8090](dee8090)) * Refactor test to reuse LocalRegistryFile ([#2763](#2763)) ([4339c0a](4339c0a)) * Revert "chore(release): release 0.22.0" ([#2852](#2852)) ([e6a4636](e6a4636)) * Support push sources in stream feature views ([#2704](#2704)) ([0d60eaa](0d60eaa)) * Update roadmap with stream feature view rfc ([#2824](#2824)) ([fc8f890](fc8f890)) * Update udf tests and add base functions to streaming fcos and fix some nonetype errors ([#2776](#2776)) ([331a214](331a214)) ### Features * Add feast repo-upgrade for automated repo upgrades ([#2733](#2733)) ([a3304d4](a3304d4)) * Add file write_to_offline_store functionality ([#2808](#2808)) ([c0e2ad7](c0e2ad7)) * Add http endpoint to the Go feature server ([#2658](#2658)) ([3347a57](3347a57)) * Add StreamProcessor and SparkKafkaProcessor as contrib ([#2777](#2777)) ([83ab682](83ab682)) * Added Spark support for Delta and Avro ([#2757](#2757)) ([7d16516](7d16516)) * CLI interface for validation of logged features ([#2718](#2718)) ([c8b11b3](c8b11b3)) * Enable stream feature view materialization ([#2798](#2798)) ([a06700d](a06700d)) * Enable stream feature view materialization ([#2807](#2807)) ([7d57724](7d57724)) * Implement `offline_write_batch` for BigQuery and Snowflake ([#2840](#2840)) ([97444e4](97444e4)) * Offline push endpoint for pushing to offline stores ([#2837](#2837)) ([a88cd30](a88cd30)) * Push to Redshift batch source offline store directly ([#2819](#2819)) ([5748a8b](5748a8b)) * Scaffold for unified push api ([#2796](#2796)) ([1bd0930](1bd0930)) * SQLAlchemy Registry Support ([#2734](#2734)) ([b3fe39c](b3fe39c)) * Stream Feature View FCOS ([#2750](#2750)) ([0cf3c92](0cf3c92)) * Update stream fcos to have watermark and sliding interval ([#2765](#2765)) ([3256952](3256952)) * Validating logged features via Python SDK ([#2640](#2640)) ([2874fc5](2874fc5)) ### Reverts * Revert "Create main.yml" (#2867) ([47922a4](47922a4)), closes [#2867](#2867)

# [0.22.0](v0.21.0...v0.22.0) (2022-06-28) ### Bug Fixes * Add columns for user metadata in the tables ([#2760](#2760)) ([269055e](269055e)) * Add project columns in the SQL Registry ([#2784](#2784)) ([336fdd1](336fdd1)) * Add S3FS dependency (which Dask depends on for S3 files) ([#2701](#2701)) ([5d6fa94](5d6fa94)) * Bugfixes for how registry is loaded ([#2768](#2768)) ([ecb8b2a](ecb8b2a)) * Conversion of null timestamp from proto to python ([#2814](#2814)) ([cb23648](cb23648)) * Correct feature statuses during feature logging test ([#2709](#2709)) ([cebf609](cebf609)) * Correctly generate projects-list.json when calling feast ui and using postgres as a source ([#2845](#2845)) ([bee8076](bee8076)) * Dynamodb drops missing entities when batching ([#2802](#2802)) ([a2e9209](a2e9209)) * Enable faulthandler and disable flaky tests ([#2815](#2815)) ([4934d84](4934d84)) * Explicitly translate errors when instantiating the go fs ([#2842](#2842)) ([7a2c4cd](7a2c4cd)) * Fix broken roadmap links ([#2690](#2690)) ([b3ba8aa](b3ba8aa)) * Fix bugs in applying stream feature view and retrieving online features ([#2754](#2754)) ([d024e5e](d024e5e)) * Fix Feast UI failure with new way of specifying entities ([#2773](#2773)) ([0d1ac01](0d1ac01)) * Fix feature view __getitem__ for feature services ([#2769](#2769)) ([88cc47d](88cc47d)) * Fix issue when user specifies a port for feast ui ([#2692](#2692)) ([1c621fe](1c621fe)) * Fix on demand feature view crash from inference when it uses df.apply ([#2713](#2713)) ([c5539fd](c5539fd)) * Fix SparkKafkaProcessor `query_timeout` parameter ([#2789](#2789)) ([a8d282d](a8d282d)) * Fix workflow syntax error ([#2869](#2869)) ([fae45a1](fae45a1)) * Fixed custom S3 endpoint read fail ([#2786](#2786)) ([6fec431](6fec431)) * Go install gopy instead using go mod tidy ([#2863](#2863)) ([2f2b519](2f2b519)) * Hydrate infra object in the sql registry proto() method ([#2782](#2782)) ([452dcd3](452dcd3)) * Implement apply_materialization and infra methods in sql registry ([#2775](#2775)) ([4ed107c](4ed107c)) * Minor refactor to format exception message ([#2764](#2764)) ([da763c6](da763c6)) * Prefer installing gopy from feast's fork as opposed to upstream ([#2839](#2839)) ([34c997d](34c997d)) * Python server is not correctly starting in integration tests ([#2706](#2706)) ([7583a0b](7583a0b)) * Random port allocation for python server in tests ([#2710](#2710)) ([dee8090](dee8090)) * Refactor test to reuse LocalRegistryFile ([#2763](#2763)) ([4339c0a](4339c0a)) * Revert "chore(release): release 0.22.0" ([#2852](#2852)) ([e6a4636](e6a4636)) * Support push sources in stream feature views ([#2704](#2704)) ([0d60eaa](0d60eaa)) * Sync publish and build_wheels workflow to fix verify wheel error. ([#2871](#2871)) ([b0f050a](b0f050a)) * Update roadmap with stream feature view rfc ([#2824](#2824)) ([fc8f890](fc8f890)) * Update udf tests and add base functions to streaming fcos and fix some nonetype errors ([#2776](#2776)) ([331a214](331a214)) ### Features * Add feast repo-upgrade for automated repo upgrades ([#2733](#2733)) ([a3304d4](a3304d4)) * Add file write_to_offline_store functionality ([#2808](#2808)) ([c0e2ad7](c0e2ad7)) * Add http endpoint to the Go feature server ([#2658](#2658)) ([3347a57](3347a57)) * Add StreamProcessor and SparkKafkaProcessor as contrib ([#2777](#2777)) ([83ab682](83ab682)) * Added Spark support for Delta and Avro ([#2757](#2757)) ([7d16516](7d16516)) * CLI interface for validation of logged features ([#2718](#2718)) ([c8b11b3](c8b11b3)) * Enable stream feature view materialization ([#2798](#2798)) ([a06700d](a06700d)) * Enable stream feature view materialization ([#2807](#2807)) ([7d57724](7d57724)) * Implement `offline_write_batch` for BigQuery and Snowflake ([#2840](#2840)) ([97444e4](97444e4)) * Offline push endpoint for pushing to offline stores ([#2837](#2837)) ([a88cd30](a88cd30)) * Push to Redshift batch source offline store directly ([#2819](#2819)) ([5748a8b](5748a8b)) * Scaffold for unified push api ([#2796](#2796)) ([1bd0930](1bd0930)) * SQLAlchemy Registry Support ([#2734](#2734)) ([b3fe39c](b3fe39c)) * Stream Feature View FCOS ([#2750](#2750)) ([0cf3c92](0cf3c92)) * Update stream fcos to have watermark and sliding interval ([#2765](#2765)) ([3256952](3256952)) * Validating logged features via Python SDK ([#2640](#2640)) ([2874fc5](2874fc5)) ### Reverts * Revert "chore(release): release 0.22.0" (#2870) ([ffb0892](ffb0892)), closes [#2870](#2870) * Revert "Create main.yml" (#2867) ([47922a4](47922a4)), closes [#2867](#2867)

# [0.22.0](v0.21.0...v0.22.0) (2022-06-29) ### Bug Fixes * Add columns for user metadata in the tables ([#2760](#2760)) ([269055e](269055e)) * Add project columns in the SQL Registry ([#2784](#2784)) ([336fdd1](336fdd1)) * Add S3FS dependency (which Dask depends on for S3 files) ([#2701](#2701)) ([5d6fa94](5d6fa94)) * Bugfixes for how registry is loaded ([#2768](#2768)) ([ecb8b2a](ecb8b2a)) * Conversion of null timestamp from proto to python ([#2814](#2814)) ([cb23648](cb23648)) * Correct feature statuses during feature logging test ([#2709](#2709)) ([cebf609](cebf609)) * Correctly generate projects-list.json when calling feast ui and using postgres as a source ([#2845](#2845)) ([bee8076](bee8076)) * Dynamodb drops missing entities when batching ([#2802](#2802)) ([a2e9209](a2e9209)) * Enable faulthandler and disable flaky tests ([#2815](#2815)) ([4934d84](4934d84)) * Explicitly translate errors when instantiating the go fs ([#2842](#2842)) ([7a2c4cd](7a2c4cd)) * Fix broken roadmap links ([#2690](#2690)) ([b3ba8aa](b3ba8aa)) * Fix bugs in applying stream feature view and retrieving online features ([#2754](#2754)) ([d024e5e](d024e5e)) * Fix Feast UI failure with new way of specifying entities ([#2773](#2773)) ([0d1ac01](0d1ac01)) * Fix feature view __getitem__ for feature services ([#2769](#2769)) ([88cc47d](88cc47d)) * Fix issue when user specifies a port for feast ui ([#2692](#2692)) ([1c621fe](1c621fe)) * Fix macos wheel version for 310 and also checkout edited go files ([#2890](#2890)) ([bdf170f](bdf170f)) * Fix on demand feature view crash from inference when it uses df.apply ([#2713](#2713)) ([c5539fd](c5539fd)) * Fix SparkKafkaProcessor `query_timeout` parameter ([#2789](#2789)) ([a8d282d](a8d282d)) * Fix workflow syntax error ([#2869](#2869)) ([fae45a1](fae45a1)) * Fixed custom S3 endpoint read fail ([#2786](#2786)) ([6fec431](6fec431)) * Go install gopy instead using go mod tidy ([#2863](#2863)) ([2f2b519](2f2b519)) * Hydrate infra object in the sql registry proto() method ([#2782](#2782)) ([452dcd3](452dcd3)) * Implement apply_materialization and infra methods in sql registry ([#2775](#2775)) ([4ed107c](4ed107c)) * Minor refactor to format exception message ([#2764](#2764)) ([da763c6](da763c6)) * Prefer installing gopy from feast's fork as opposed to upstream ([#2839](#2839)) ([34c997d](34c997d)) * Python server is not correctly starting in integration tests ([#2706](#2706)) ([7583a0b](7583a0b)) * Random port allocation for python server in tests ([#2710](#2710)) ([dee8090](dee8090)) * Refactor test to reuse LocalRegistryFile ([#2763](#2763)) ([4339c0a](4339c0a)) * Revert "chore(release): release 0.22.0" ([#2852](#2852)) ([e6a4636](e6a4636)) * Stop running go mod tidy in setup.py ([#2877](#2877)) ([676ecbb](676ecbb)), closes [/github.com/pypa/cibuildwheel/issues/189#issuecomment-549933912](https://github.com//github.com/pypa/cibuildwheel/issues/189/issues/issuecomment-549933912) * Support push sources in stream feature views ([#2704](#2704)) ([0d60eaa](0d60eaa)) * Sync publish and build_wheels workflow to fix verify wheel error. ([#2871](#2871)) ([b0f050a](b0f050a)) * Update roadmap with stream feature view rfc ([#2824](#2824)) ([fc8f890](fc8f890)) * Update udf tests and add base functions to streaming fcos and fix some nonetype errors ([#2776](#2776)) ([331a214](331a214)) ### Features * Add feast repo-upgrade for automated repo upgrades ([#2733](#2733)) ([a3304d4](a3304d4)) * Add file write_to_offline_store functionality ([#2808](#2808)) ([c0e2ad7](c0e2ad7)) * Add http endpoint to the Go feature server ([#2658](#2658)) ([3347a57](3347a57)) * Add simple TLS support in Go RedisOnlineStore ([#2860](#2860)) ([521488d](521488d)) * Add StreamProcessor and SparkKafkaProcessor as contrib ([#2777](#2777)) ([83ab682](83ab682)) * Added Spark support for Delta and Avro ([#2757](#2757)) ([7d16516](7d16516)) * CLI interface for validation of logged features ([#2718](#2718)) ([c8b11b3](c8b11b3)) * Enable stream feature view materialization ([#2798](#2798)) ([a06700d](a06700d)) * Enable stream feature view materialization ([#2807](#2807)) ([7d57724](7d57724)) * Implement `offline_write_batch` for BigQuery and Snowflake ([#2840](#2840)) ([97444e4](97444e4)) * Offline push endpoint for pushing to offline stores ([#2837](#2837)) ([a88cd30](a88cd30)) * Push to Redshift batch source offline store directly ([#2819](#2819)) ([5748a8b](5748a8b)) * Scaffold for unified push api ([#2796](#2796)) ([1bd0930](1bd0930)) * SQLAlchemy Registry Support ([#2734](#2734)) ([b3fe39c](b3fe39c)) * Stream Feature View FCOS ([#2750](#2750)) ([0cf3c92](0cf3c92)) * Update stream fcos to have watermark and sliding interval ([#2765](#2765)) ([3256952](3256952)) * Validating logged features via Python SDK ([#2640](#2640)) ([2874fc5](2874fc5)) ### Reverts * Revert "chore(release): release 0.22.0" (#2891) ([e5abf58](e5abf58)), closes [#2891](#2891) * Revert "chore(release): release 0.22.0" (#2870) ([ffb0892](ffb0892)), closes [#2870](#2870) * Revert "Create main.yml" (#2867) ([47922a4](47922a4)), closes [#2867](#2867)

feast-ci-bot added do-not-merge/work-in-progress approved size/L labels May 26, 2022

achals requested review from mavysavydav, felixwang9817 and adchia May 26, 2022 17:41

achals force-pushed the achal/postgres-registry branch from 342bfa1 to 71dfaf5 Compare May 31, 2022 18:42

feast-ci-bot added size/XL and removed size/L labels Jun 2, 2022

achals marked this pull request as ready for review June 2, 2022 20:41

feast-ci-bot removed the do-not-merge/work-in-progress label Jun 2, 2022

achals changed the title ~~feat: WIP SQLAlchemy Registry Support~~ feat: SQLAlchemy Registry Support Jun 2, 2022

achals added 7 commits June 2, 2022 14:02

feat: WIP SQLAlchemy Registry Support

3d88358

Signed-off-by: Achal Shah <achals@gmail.com>

hack hack hack

f9e6bb0

Signed-off-by: Achal Shah <achals@gmail.com>

fix repo config

99a6b3e

Signed-off-by: Achal Shah <achals@gmail.com>

reduce duplication

64c3971

Signed-off-by: Achal Shah <achals@gmail.com>

simplify

dcd88bd

Signed-off-by: Achal Shah <achals@gmail.com>

postgres tests

9add91d

Signed-off-by: Achal Shah <achals@gmail.com>

tests for pg and mysql

b7e33ee

Signed-off-by: Achal Shah <achals@gmail.com>

achals force-pushed the achal/postgres-registry branch from 2e82fde to b7e33ee Compare June 2, 2022 21:06

fix requirements

4b92e9b

Signed-off-by: Achal Shah <achals@gmail.com>

feast-ci-bot added size/XXL and removed size/XL labels Jun 2, 2022

achals added 6 commits June 2, 2022 15:11

pin protobuf again

7aaf4c0

Signed-off-by: Achal Shah <achals@gmail.com>

fix macos test

2555e8a

Signed-off-by: Achal Shah <achals@gmail.com>

fix macos test

29c6b1c

Signed-off-by: Achal Shah <achals@gmail.com>

fix quotes

4b7aed1

Signed-off-by: Achal Shah <achals@gmail.com>

add macos

16e8944

Signed-off-by: Achal Shah <achals@gmail.com>

remove macos

6a33f46

Signed-off-by: Achal Shah <achals@gmail.com>

install mysql library but don't try to run tests

629083c

Signed-off-by: Achal Shah <achals@gmail.com>

achals assigned felixwang9817 Jun 3, 2022

felixwang9817 reviewed Jun 3, 2022

View reviewed changes

Fix CR comments

4a2c84c

Signed-off-by: Achal Shah <achals@gmail.com>

felixwang9817 approved these changes Jun 3, 2022

View reviewed changes

feast-ci-bot added the lgtm label Jun 3, 2022

kevjumba reviewed Jun 3, 2022

View reviewed changes

felixwang9817 removed the lgtm label Jun 3, 2022

add a comment

9e6d84b

Signed-off-by: Achal Shah <achals@gmail.com>

feast-ci-bot assigned kevjumba Jun 3, 2022

feast-ci-bot added the lgtm label Jun 3, 2022

achals added the ok-to-test label Jun 3, 2022

feast-ci-bot merged commit b3fe39c into feast-dev:master Jun 3, 2022

achals deleted the achal/postgres-registry branch June 3, 2022 18:25

adchia mentioned this pull request Jun 14, 2022

Add support for concurrent operations on the registry #2176

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: SQLAlchemy Registry Support #2734

feat: SQLAlchemy Registry Support #2734

achals commented May 26, 2022 •

edited

Loading

codecov-commenter commented May 26, 2022 •

edited

Loading

felixwang9817 left a comment

feast-ci-bot commented Jun 3, 2022

kevjumba Jun 3, 2022

achals Jun 3, 2022

kevjumba Jun 3, 2022

achals Jun 3, 2022

kevjumba Jun 3, 2022

achals Jun 3, 2022

kevjumba commented Jun 3, 2022

feat: SQLAlchemy Registry Support #2734

feat: SQLAlchemy Registry Support #2734

Conversation

achals commented May 26, 2022 • edited Loading

codecov-commenter commented May 26, 2022 • edited Loading

Codecov Report

felixwang9817 left a comment

Choose a reason for hiding this comment

feast-ci-bot commented Jun 3, 2022

kevjumba Jun 3, 2022

Choose a reason for hiding this comment

achals Jun 3, 2022

Choose a reason for hiding this comment

kevjumba Jun 3, 2022

Choose a reason for hiding this comment

achals Jun 3, 2022

Choose a reason for hiding this comment

kevjumba Jun 3, 2022

Choose a reason for hiding this comment

achals Jun 3, 2022

Choose a reason for hiding this comment

kevjumba commented Jun 3, 2022

achals commented May 26, 2022 •

edited

Loading

codecov-commenter commented May 26, 2022 •

edited

Loading