-
Notifications
You must be signed in to change notification settings - Fork 996
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Postgres supported as Registry, Online store, and Offline store #2401
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nossrannug looking good overall! left some comments for you (no rush at all)
would also suggest that you add in a template so that users can easily initialize a postgres-based feature repo with feast init -t postgres
, see https://github.com/feast-dev/feast/pull/2349/files for an example of Spark PR adding in a template
would also recommend adding some docs so folks know how to use postgres properly! once again see Spark PR for how to add docs
sdk/python/feast/infra/offline_stores/contrib/contrib_repo_configuration.py
Outdated
Show resolved
Hide resolved
sdk/python/tests/integration/feature_repos/universal/data_sources/postgres_data_source.py
Outdated
Show resolved
Hide resolved
sdk/python/feast/infra/offline_stores/contrib/contrib_repo_configuration.py
Outdated
Show resolved
Hide resolved
Hi @nossrannug, are you still planning on working on this PR? Is there anything we can do to help? |
Yes, I would like to see this PR completed. Real life stuff and lack of motivation have been my biggest blockers. I'll take a stab at it today and see how it goes. |
Totally understand - let us know if there's anything we can do to help! |
Thanks, @achals, I greatly appreciate it. I took a quick look at it over easter. I'm able to run:
I can also run I think the main thing left now is to update the version lock files. I'll see if I can get that done later today and then I don't know if there is much left. I'll give you and @felixwang9817 a ping when I think I've covered everything to ask for your review. |
bb5e1cb
to
4da90a5
Compare
@nossrannug let me see if I can get the unit test builds to work really quickly. |
4da90a5
to
b0ae6df
Compare
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## master #2401 +/- ##
==========================================
- Coverage 83.68% 82.24% -1.45%
==========================================
Files 147 154 +7
Lines 12353 12735 +382
==========================================
+ Hits 10338 10474 +136
- Misses 2015 2261 +246
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Unit tests should be working now. |
Signed-off-by: Gunnar Sv Sigurbjörnsson <gunnar.sigurbjornsson@gmail.com>
Signed-off-by: Gunnar Sv Sigurbjörnsson <gunnar.sigurbjornsson@gmail.com>
Signed-off-by: Gunnar Sv Sigurbjörnsson <gunnar.sigurbjornsson@gmail.com>
Signed-off-by: Gunnar Sv Sigurbjörnsson <gunnar.sigurbjornsson@gmail.com>
Signed-off-by: Gunnar Sv Sigurbjörnsson <gunnar.sigurbjornsson@gmail.com>
Signed-off-by: Kevin Zhang <kzhang@tecton.ai>
Signed-off-by: Kevin Zhang <kzhang@tecton.ai>
Signed-off-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Danny Chiao <danny@tecton.ai>
sdk/python/feast/infra/offline_stores/contrib/postgres_offline_store/postgres.py
Outdated
Show resolved
Hide resolved
sdk/python/feast/infra/offline_stores/contrib/postgres_repo_configuration.py
Show resolved
Hide resolved
sdk/python/feast/infra/offline_stores/contrib/postgres_offline_store/postgres_source.py
Show resolved
Hide resolved
sdk/python/feast/infra/offline_stores/contrib/postgres_offline_store/postgres_source.py
Show resolved
Hide resolved
Signed-off-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Gunnar Sv Sigurbjörnsson <gunnar.sigurbjornsson@gmail.com>
Signed-off-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Danny Chiao <danny@tecton.ai>
Signed-off-by: Danny Chiao <danny@tecton.ai>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: achals, nossrannug The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
# [0.21.0](v0.20.0...v0.21.0) (2022-05-13) ### Bug Fixes * Addresses ZeroDivisionError when materializing file source with same timestamps ([#2551](#2551)) ([1e398d9](1e398d9)) * Asynchronously refresh registry for the feast ui command ([#2672](#2672)) ([1b09ca2](1b09ca2)) * Build platform specific python packages with ci-build-wheel ([#2555](#2555)) ([b10a4cf](b10a4cf)) * Delete data sources from registry when using the diffing logic ([#2669](#2669)) ([fc00ca8](fc00ca8)) * Enforce kw args featureservice ([#2575](#2575)) ([160d7b7](160d7b7)) * Enforce kw args in datasources ([#2567](#2567)) ([0b7ec53](0b7ec53)) * Feature logging to Redshift is broken ([#2655](#2655)) ([479cd51](479cd51)) * Feature service to templates ([#2649](#2649)) ([1e02066](1e02066)) * Feature with timestamp type is incorrectly interpreted by Go FS ([#2588](#2588)) ([e3d9588](e3d9588)) * Fix `__hash__` methods ([#2556](#2556)) ([ebb7dfe](ebb7dfe)) * Fix AWS bootstrap template ([#2604](#2604)) ([c94a69c](c94a69c)) * Fix broken proto conversion methods for data sources ([#2603](#2603)) ([00ed65a](00ed65a)) * Fix case where on demand feature view tab is broken if no custom tabs are passed. ([#2682](#2682)) ([01d3568](01d3568)) * Fix DynamoDB fetches when there are entities that are not found ([#2573](#2573)) ([7076fe0](7076fe0)) * Fix Feast UI parser to work with new APIs ([#2668](#2668)) ([8d76751](8d76751)) * Fix java server after odfv update ([#2602](#2602)) ([0ca6297](0ca6297)) * Fix materialization with ttl=0 bug ([#2666](#2666)) ([ab78702](ab78702)) * Fix push sources and add docs / tests pushing via the python feature server ([#2561](#2561)) ([e8e418e](e8e418e)) * Fixed data mapping errors for Snowflake ([#2558](#2558)) ([53c2ce2](53c2ce2)) * Forcing ODFV udfs to be __main__ module and fixing false positive duplicate data source warning ([#2677](#2677)) ([2ce33cd](2ce33cd)) * Include the ui/build directory, and remove package data ([#2681](#2681)) ([0384f5f](0384f5f)) * Infer features for feature services when they depend on feature views without schemas ([#2653](#2653)) ([87c194c](87c194c)) * Pin dependencies to nearest major version ([#2647](#2647)) ([bb72b7c](bb72b7c)) * Pin pip<22.1 to get around breaking change in pip==22.1 ([#2678](#2678)) ([d3e01bc](d3e01bc)) * Punt deprecation warnings and clean up some warnings. ([#2670](#2670)) ([f775d2e](f775d2e)) * Reject undefined features when using `get_historical_features` or `get_online_features` ([#2665](#2665)) ([36849fb](36849fb)) * Remove ci extra from the feature transformation server dockerfile ([#2618](#2618)) ([25613b4](25613b4)) * Remove incorrect call to logging.basicConfig ([#2676](#2676)) ([8cbf51c](8cbf51c)) * Small typo in CLI ([#2578](#2578)) ([f372981](f372981)) * Switch from `join_key` to `join_keys` in tests and docs ([#2580](#2580)) ([d66c931](d66c931)) * Teardown trino container correctly after tests ([#2562](#2562)) ([72f1558](72f1558)) * Update build_go_protos to use a consistent python path ([#2550](#2550)) ([f136f8c](f136f8c)) * Update data source timestamp inference error message to make sense ([#2636](#2636)) ([3eaf6b7](3eaf6b7)) * Update field api to add tag parameter corresponding to labels in Feature. ([#2610](#2610)) ([689d20b](689d20b)) * Update java integration tests and add more logging ([#2637](#2637)) ([10e23b4](10e23b4)) * Update on demand feature view api ([#2587](#2587)) ([38cd7f9](38cd7f9)) * Update RedisCluster to use redis-py official implementation ([#2554](#2554)) ([ce5606f](ce5606f)) * Use cwd when getting module path ([#2577](#2577)) ([b550e59](b550e59)) * Use ParquetDataset for Schema Inference ([#2686](#2686)) ([4f85e3e](4f85e3e)) * Use timestamp type when converting unixtimestamp feature type to arrow ([#2593](#2593)) ([c439611](c439611)) ### Features * Add hbase online store support in feast ([#2590](#2590)) ([c9eda79](c9eda79)) * Adding SSL options for Postgres ([#2644](#2644)) ([0e809c2](0e809c2)) * Allow Feast UI to be spun up with CLI command: feast ui ([#2667](#2667)) ([44ca9f5](44ca9f5)) * Allow to pass secrets and environment variables to transformation service ([#2632](#2632)) ([ffa33ad](ffa33ad)) * CLI command 'feast serve' should start go-based server if flag is enabled ([#2617](#2617)) ([f3ff812](f3ff812)) * Create stream and batch feature view abstractions ([#2559](#2559)) ([d1f76e5](d1f76e5)) * Postgres supported as Registry, Online store, and Offline store ([#2401](#2401)) ([ed2f979](ed2f979)) * Support entity fields in feature view `schema` parameter by dropping them ([#2568](#2568)) ([c8fcc35](c8fcc35)) * Write logged features to an offline store (Python API) ([#2574](#2574)) ([134dc5f](134dc5f)) * Write logged features to Offline Store (Go - Python integration) ([#2621](#2621)) ([ccad832](ccad832)) ### Reverts * Revert "chore: Deprecate value type (#2611)" (#2643) ([4fbdfb1](4fbdfb1)), closes [#2611](#2611) [#2643](#2643)
Signed-off-by: Gunnar Sv Sigurbjörnsson gunnar.sigurbjornsson@gmail.com
What this PR does / why we need it:
Adds support for postgres as Registry, and Online and Offline stores.