fix: Fix DynamoDB fetches when there are entities that are not found #2573
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Danny Chiao <danny@tecton.ai>
achals
reviewed
Apr 19, 2022
sdk/python/tests/unit/infra/online_store/test_dynamodb_online_store.py
Outdated
Show resolved
Hide resolved
…st for different unknowns Signed-off-by: Danny Chiao <danny@tecton.ai>
Codecov Report
@@ Coverage Diff @@
## master #2573 +/- ##
==========================================
- Coverage 83.29% 83.27% -0.03%
==========================================
Files 145 145
Lines 12076 12096 +20
==========================================
+ Hits 10059 10073 +14
- Misses 2017 2023 +6
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
achals
approved these changes
Apr 19, 2022
Comment on lines
+79
to
+81
| Data is returned as a list, one item per entity key in the original order as the entity_keys argument. | ||
| Each item in the list is a tuple of event_ts for the row, and the feature data as a dict from feature names | ||
| to values. Values are returned as Value proto message. |
There was a problem hiding this comment.
This should probably be highlighted - also smells like a fragile interface. But that discussion is out of scope for this PR.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: achals, adchia The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
adchia
added a commit
that referenced
this pull request
Apr 20, 2022
…2573) * fix: Fix DynamoDB fetches when there are entities that are not found Signed-off-by: Danny Chiao <danny@tecton.ai> * remove sort_keys from dynamo since they must be sorted. Add better test for different unknowns Signed-off-by: Danny Chiao <danny@tecton.ai>
adchia
pushed a commit
that referenced
this pull request
Apr 20, 2022
## [0.20.1](v0.20.0...v0.20.1) (2022-04-20) ### Bug Fixes * Addresses ZeroDivisionError when materializing file source with same timestamps ([#2551](#2551)) ([5539c51](5539c51)) * Build platform specific python packages with ci-build-wheel ([#2555](#2555)) ([1757639](1757639)) * Enforce kw args featureservice ([#2575](#2575)) ([4dce254](4dce254)) * Enforce kw args in datasources ([#2567](#2567)) ([6374634](6374634)) * Fix `__hash__` methods ([#2556](#2556)) ([dd8b854](dd8b854)) * Fix DynamoDB fetches when there are entities that are not found ([#2573](#2573)) ([882328f](882328f)) * Fix push sources and add docs / tests pushing via the python feature server ([#2561](#2561)) ([c5006c2](c5006c2)) * Fixed data mapping errors for Snowflake ([#2558](#2558)) ([abd6be7](abd6be7)) * Small typo in CLI ([#2578](#2578)) ([8717bc8](8717bc8)) * Switch from `join_key` to `join_keys` in tests and docs ([#2580](#2580)) ([6130b80](6130b80)) * Update build_go_protos to use a consistent python path ([#2550](#2550)) ([1c523bf](1c523bf)) * Update RedisCluster to use redis-py official implementation ([#2554](#2554)) ([c47fa2a](c47fa2a)) * Use cwd when getting module path ([#2577](#2577)) ([28752f2](28752f2))
achals
pushed a commit
that referenced
this pull request
May 13, 2022
# [0.21.0](v0.20.0...v0.21.0) (2022-05-13) ### Bug Fixes * Addresses ZeroDivisionError when materializing file source with same timestamps ([#2551](#2551)) ([1e398d9](1e398d9)) * Asynchronously refresh registry for the feast ui command ([#2672](#2672)) ([1b09ca2](1b09ca2)) * Build platform specific python packages with ci-build-wheel ([#2555](#2555)) ([b10a4cf](b10a4cf)) * Delete data sources from registry when using the diffing logic ([#2669](#2669)) ([fc00ca8](fc00ca8)) * Enforce kw args featureservice ([#2575](#2575)) ([160d7b7](160d7b7)) * Enforce kw args in datasources ([#2567](#2567)) ([0b7ec53](0b7ec53)) * Feature logging to Redshift is broken ([#2655](#2655)) ([479cd51](479cd51)) * Feature service to templates ([#2649](#2649)) ([1e02066](1e02066)) * Feature with timestamp type is incorrectly interpreted by Go FS ([#2588](#2588)) ([e3d9588](e3d9588)) * Fix `__hash__` methods ([#2556](#2556)) ([ebb7dfe](ebb7dfe)) * Fix AWS bootstrap template ([#2604](#2604)) ([c94a69c](c94a69c)) * Fix broken proto conversion methods for data sources ([#2603](#2603)) ([00ed65a](00ed65a)) * Fix case where on demand feature view tab is broken if no custom tabs are passed. ([#2682](#2682)) ([01d3568](01d3568)) * Fix DynamoDB fetches when there are entities that are not found ([#2573](#2573)) ([7076fe0](7076fe0)) * Fix Feast UI parser to work with new APIs ([#2668](#2668)) ([8d76751](8d76751)) * Fix java server after odfv update ([#2602](#2602)) ([0ca6297](0ca6297)) * Fix materialization with ttl=0 bug ([#2666](#2666)) ([ab78702](ab78702)) * Fix push sources and add docs / tests pushing via the python feature server ([#2561](#2561)) ([e8e418e](e8e418e)) * Fixed data mapping errors for Snowflake ([#2558](#2558)) ([53c2ce2](53c2ce2)) * Forcing ODFV udfs to be __main__ module and fixing false positive duplicate data source warning ([#2677](#2677)) ([2ce33cd](2ce33cd)) * Include the ui/build directory, and remove package data ([#2681](#2681)) ([0384f5f](0384f5f)) * Infer features for feature services when they depend on feature views without schemas ([#2653](#2653)) ([87c194c](87c194c)) * Pin dependencies to nearest major version ([#2647](#2647)) ([bb72b7c](bb72b7c)) * Pin pip<22.1 to get around breaking change in pip==22.1 ([#2678](#2678)) ([d3e01bc](d3e01bc)) * Punt deprecation warnings and clean up some warnings. ([#2670](#2670)) ([f775d2e](f775d2e)) * Reject undefined features when using `get_historical_features` or `get_online_features` ([#2665](#2665)) ([36849fb](36849fb)) * Remove ci extra from the feature transformation server dockerfile ([#2618](#2618)) ([25613b4](25613b4)) * Remove incorrect call to logging.basicConfig ([#2676](#2676)) ([8cbf51c](8cbf51c)) * Small typo in CLI ([#2578](#2578)) ([f372981](f372981)) * Switch from `join_key` to `join_keys` in tests and docs ([#2580](#2580)) ([d66c931](d66c931)) * Teardown trino container correctly after tests ([#2562](#2562)) ([72f1558](72f1558)) * Update build_go_protos to use a consistent python path ([#2550](#2550)) ([f136f8c](f136f8c)) * Update data source timestamp inference error message to make sense ([#2636](#2636)) ([3eaf6b7](3eaf6b7)) * Update field api to add tag parameter corresponding to labels in Feature. ([#2610](#2610)) ([689d20b](689d20b)) * Update java integration tests and add more logging ([#2637](#2637)) ([10e23b4](10e23b4)) * Update on demand feature view api ([#2587](#2587)) ([38cd7f9](38cd7f9)) * Update RedisCluster to use redis-py official implementation ([#2554](#2554)) ([ce5606f](ce5606f)) * Use cwd when getting module path ([#2577](#2577)) ([b550e59](b550e59)) * Use ParquetDataset for Schema Inference ([#2686](#2686)) ([4f85e3e](4f85e3e)) * Use timestamp type when converting unixtimestamp feature type to arrow ([#2593](#2593)) ([c439611](c439611)) ### Features * Add hbase online store support in feast ([#2590](#2590)) ([c9eda79](c9eda79)) * Adding SSL options for Postgres ([#2644](#2644)) ([0e809c2](0e809c2)) * Allow Feast UI to be spun up with CLI command: feast ui ([#2667](#2667)) ([44ca9f5](44ca9f5)) * Allow to pass secrets and environment variables to transformation service ([#2632](#2632)) ([ffa33ad](ffa33ad)) * CLI command 'feast serve' should start go-based server if flag is enabled ([#2617](#2617)) ([f3ff812](f3ff812)) * Create stream and batch feature view abstractions ([#2559](#2559)) ([d1f76e5](d1f76e5)) * Postgres supported as Registry, Online store, and Offline store ([#2401](#2401)) ([ed2f979](ed2f979)) * Support entity fields in feature view `schema` parameter by dropping them ([#2568](#2568)) ([c8fcc35](c8fcc35)) * Write logged features to an offline store (Python API) ([#2574](#2574)) ([134dc5f](134dc5f)) * Write logged features to Offline Store (Go - Python integration) ([#2621](#2621)) ([ccad832](ccad832)) ### Reverts * Revert "chore: Deprecate value type (#2611)" (#2643) ([4fbdfb1](4fbdfb1)), closes [#2611](#2611) [#2643](#2643)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Before this, having an unknown entity (e.g. such as is commonly the case in the feast-benchmarks repo) in a batch would result in an exception because
https://github.com/feast-dev/feast/blob/master/sdk/python/feast/feature_store.py#L1789-L1789
expects that the number of response rows is equal to the number of requested entities. Dynamo will only return values for entities that are found, so there could be a mismatch and that causes protobuf to complain because we mix None values and Value values.
This also reverts the change to allow users to not sort feature values. Feast expects the results to be in the original order to return the correct values in a GetOnlineFeatures response.
Which issue(s) this PR fixes:
Fixes #