fix: Version entity serialization mechanism and fix issue with int64 vals #2944

achals · 2022-07-15T06:05:12Z

This PR should supercedes #2609. The PR is kinda WIP (specifically, fixing java and go deserialization and adding more tests), but mostly ready for review.

Signed-off-by: Achal Shah achals@gmail.com

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #1908, #2345, #2942

protos/feast/core/FeatureView.proto

sdk/python/feast/base_feature_view.py

sdk/python/feast/feature_view.py

sdk/python/feast/infra/key_encoding_utils.py

sdk/python/tests/unit/infra/test_key_encoding_utils.py

codecov-commenter · 2022-07-16T04:22:37Z

Codecov Report

Merging #2944 (82259d4) into master (b69eadc) will decrease coverage by 19.17%.
The diff coverage is 82.00%.

@@             Coverage Diff             @@
##           master    #2944       +/-   ##
===========================================
- Coverage   78.66%   59.48%   -19.18%     
===========================================
  Files         184      186        +2     
  Lines       16313    16339       +26     
===========================================
- Hits        12833     9720     -3113     
- Misses       3480     6619     +3139

Flag	Coverage Δ
integrationtests	`?`
unittests	`59.48% <82.00%> (+0.06%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...east/infra/materialization/lambda/lambda_engine.py	`46.87% <0.00%> (-45.84%)`	⬇️
.../online_stores/contrib/hbase_online_store/hbase.py	`37.25% <0.00%> (ø)`
...thon/feast/infra/online_stores/contrib/postgres.py	`32.69% <0.00%> (ø)`
sdk/python/feast/infra/online_stores/datastore.py	`40.27% <0.00%> (-49.08%)`	⬇️
sdk/python/feast/infra/online_stores/redis.py	`28.39% <0.00%> (-66.67%)`	⬇️
sdk/python/feast/infra/utils/hbase_utils.py	`41.17% <ø> (ø)`
sdk/python/feast/on_demand_feature_view.py	`74.33% <ø> (-6.80%)`	⬇️
sdk/python/feast/registry.py	`68.13% <ø> (-14.92%)`	⬇️
...n/tests/integration/online_store/test_e2e_local.py	`100.00% <ø> (ø)`
...t/infra/online_store/test_dynamodb_online_store.py	`100.00% <ø> (ø)`
... and 93 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b69eadc...82259d4. Read the comment docs.

…64 vals Signed-off-by: Achal Shah <achals@gmail.com>

Signed-off-by: Achal Shah <achals@gmail.com>

go/internal/feast/onlinestore/redisonlinestore.go

sdk/python/feast/infra/key_encoding_utils.py

sdk/python/feast/repo_config.py

Signed-off-by: Achal Shah <achals@gmail.com>

kevjumba

will let @felixwang9817 take a pass and stamp

felixwang9817

/lgtm

feast-ci-bot · 2022-07-19T23:05:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: achals, felixwang9817, kevjumba

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [achals,felixwang9817,kevjumba]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

# [0.23.0](v0.22.0...v0.23.0) (2022-08-02) ### Bug Fixes * Add dummy alias to pull_all_from_table_or_query ([#2956](#2956)) ([5e45228](5e45228)) * Bump version of Guava to mitigate cve ([#2896](#2896)) ([51df8be](51df8be)) * Change numpy version on setup.py and upgrade it to resolve dependabot warning ([#2887](#2887)) ([80ea7a9](80ea7a9)) * Change the feature store plan method to public modifier ([#2904](#2904)) ([0ec7d1a](0ec7d1a)) * Deprecate 3.7 wheels and fix verification workflow ([#2934](#2934)) ([040c910](040c910)) * Do not allow same column to be reused in data sources ([#2965](#2965)) ([661c053](661c053)) * Fix build wheels workflow to install apache-arrow correctly ([#2932](#2932)) ([bdeb4ae](bdeb4ae)) * Fix file offline store logic for feature views without ttl ([#2971](#2971)) ([26f6b69](26f6b69)) * Fix grpc and update protobuf ([#2894](#2894)) ([86e9efd](86e9efd)) * Fix night ci syntax error and update readme ([#2935](#2935)) ([b917540](b917540)) * Fix nightly ci again ([#2939](#2939)) ([1603c9e](1603c9e)) * Fix the go build and use CgoArrowAllocator to prevent incorrect garbage collection ([#2919](#2919)) ([130746e](130746e)) * Fix typo in CONTRIBUTING.md ([#2955](#2955)) ([8534f69](8534f69)) * Fixing broken links to feast documentation on java readme and contribution ([#2892](#2892)) ([d044588](d044588)) * Fixing Spark min / max entity df event timestamps range return order ([#2735](#2735)) ([ac55ce2](ac55ce2)) * Move gcp back to 1.47.0 since grpcio-tools 1.48.0 got yanked from pypi ([#2990](#2990)) ([fc447eb](fc447eb)) * Refactor testing and sort out unit and integration tests ([#2975](#2975)) ([2680f7b](2680f7b)) * Remove hard-coded integration test setup for AWS & GCP ([#2970](#2970)) ([e4507ac](e4507ac)) * Resolve small typo in README file ([#2930](#2930)) ([16ae902](16ae902)) * Revert "feat: Add snowflake online store ([#2902](#2902))" ([#2909](#2909)) ([38fd001](38fd001)) * Snowflake_online_read fix ([#2988](#2988)) ([651ce34](651ce34)) * Spark source support table with pattern "db.table" ([#2606](#2606)) ([3ce5139](3ce5139)), closes [#2605](#2605) * Switch mysql log string to use regex ([#2976](#2976)) ([5edf4b0](5edf4b0)) * Update gopy to point to fork to resolve github annotation errors. ([#2940](#2940)) ([ba2dcf1](ba2dcf1)) * Version entity serialization mechanism and fix issue with int64 vals ([#2944](#2944)) ([d0d27a3](d0d27a3)) ### Features * Add an experimental lambda-based materialization engine ([#2923](#2923)) ([6f79069](6f79069)) * Add column reordering to `write_to_offline_store` ([#2876](#2876)) ([8abc2ef](8abc2ef)) * Add custom JSON table tab w/ formatting ([#2851](#2851)) ([0159f38](0159f38)) * Add CustomSourceOptions to SavedDatasetStorage ([#2958](#2958)) ([23c09c8](23c09c8)) * Add Go option to `feast serve` command ([#2966](#2966)) ([a36a695](a36a695)) * Add interfaces for batch materialization engine ([#2901](#2901)) ([38b28ca](38b28ca)) * Add pages for individual Features to the Feast UI ([#2850](#2850)) ([9b97fca](9b97fca)) * Add snowflake online store ([#2902](#2902)) ([f758f9e](f758f9e)), closes [#2903](#2903) * Add Snowflake online store (again) ([#2922](#2922)) ([2ef71fc](2ef71fc)), closes [#2903](#2903) * Add to_remote_storage method to RetrievalJob ([#2916](#2916)) ([109ee9c](109ee9c)) * Support retrieval from multiple feature views with different join keys ([#2835](#2835)) ([056cfa1](056cfa1))

feast-ci-bot added approved size/L labels Jul 15, 2022

achals changed the title ~~fix: version the entity serialization and fix issue with int64 vals~~ fix: Version entity serialization mechanism and fix issue with int64 vals Jul 15, 2022

kevjumba reviewed Jul 15, 2022

View reviewed changes

feast-ci-bot added size/XL and removed size/L labels Jul 18, 2022

achals added 16 commits July 18, 2022 21:11

fix: version the entity serialization mechanism to fix issue with int…

af4ccef

…64 vals Signed-off-by: Achal Shah <achals@gmail.com>

fix tests

c6dabd1

Signed-off-by: Achal Shah <achals@gmail.com>

Add a test

b709511

Signed-off-by: Achal Shah <achals@gmail.com>

Add a test

0b81975

Signed-off-by: Achal Shah <achals@gmail.com>

fix test

b91bf17

Signed-off-by: Achal Shah <achals@gmail.com>

fix test

4bc7c80

Signed-off-by: Achal Shah <achals@gmail.com>

fix test

13712bb

Signed-off-by: Achal Shah <achals@gmail.com>

simplify

de47286

Signed-off-by: Achal Shah <achals@gmail.com>

simplify

3aac0f6

Signed-off-by: Achal Shah <achals@gmail.com>

feature_store.yaml

da9b407

Signed-off-by: Achal Shah <achals@gmail.com>

fix tests

7ab893c

Signed-off-by: Achal Shah <achals@gmail.com>

remove protos

2e1b123

Signed-off-by: Achal Shah <achals@gmail.com>

fix tests

fd1e8eb

Signed-off-by: Achal Shah <achals@gmail.com>

update feature_store.yaml templates

ec4a099

Signed-off-by: Achal Shah <achals@gmail.com>

fix java

37e32e4

Signed-off-by: Achal Shah <achals@gmail.com>

fix java test

e1a47fc

Signed-off-by: Achal Shah <achals@gmail.com>

achals force-pushed the achal/int64-serialization branch from e4dbfee to e1a47fc Compare July 19, 2022 04:50

kevjumba reviewed Jul 19, 2022

View reviewed changes

go/internal/feast/onlinestore/redisonlinestore.go Outdated Show resolved Hide resolved

sdk/python/feast/infra/key_encoding_utils.py Show resolved Hide resolved

sdk/python/feast/repo_config.py Outdated Show resolved Hide resolved

achals added 3 commits July 19, 2022 14:31

docs

bbceb68

Signed-off-by: Achal Shah <achals@gmail.com>

docs

4aa2ba7

Signed-off-by: Achal Shah <achals@gmail.com>

docs

82259d4

Signed-off-by: Achal Shah <achals@gmail.com>

kevjumba approved these changes Jul 19, 2022

View reviewed changes

felixwang9817 approved these changes Jul 19, 2022

View reviewed changes

feast-ci-bot assigned felixwang9817 Jul 19, 2022

feast-ci-bot added the lgtm label Jul 19, 2022

feast-ci-bot merged commit d0d27a3 into feast-dev:master Jul 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Version entity serialization mechanism and fix issue with int64 vals #2944

fix: Version entity serialization mechanism and fix issue with int64 vals #2944

achals commented Jul 15, 2022 •

edited

Loading

codecov-commenter commented Jul 16, 2022 •

edited

Loading

kevjumba left a comment

felixwang9817 left a comment

feast-ci-bot commented Jul 19, 2022

fix: Version entity serialization mechanism and fix issue with int64 vals #2944

fix: Version entity serialization mechanism and fix issue with int64 vals #2944

Conversation

achals commented Jul 15, 2022 • edited Loading

codecov-commenter commented Jul 16, 2022 • edited Loading

Codecov Report

kevjumba left a comment

Choose a reason for hiding this comment

felixwang9817 left a comment

Choose a reason for hiding this comment

feast-ci-bot commented Jul 19, 2022

achals commented Jul 15, 2022 •

edited

Loading

codecov-commenter commented Jul 16, 2022 •

edited

Loading