Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add support for DynamoDB online_read in batches #2371

Merged
Merged
Changes from 1 commit
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
c34bf3e
feat: dynamodb onlin read in batches
TremaMiguel Mar 6, 2022
56d2d32
run linters and format
TremaMiguel Mar 6, 2022
7b98faf
feat: batch_size parameter
TremaMiguel Mar 6, 2022
94abfb6
Merge branch 'feast-dev:master' into feat/dynamo_db_online_write_read
TremaMiguel Mar 6, 2022
b5c1a3d
docs: typo in batch_size description
TremaMiguel Mar 6, 2022
5a12856
trailing white space
TremaMiguel Mar 6, 2022
8bd2a84
fix: batch_size is last argument
TremaMiguel Mar 7, 2022
fb6eacb
test: dynamodb online store online_read in batches
TremaMiguel Mar 7, 2022
1bbd5dc
Merge branch 'master' into feat/dynamo_db_online_write_read
adchia Mar 7, 2022
307bab9
test: mock dynamodb behavior
TremaMiguel Mar 7, 2022
e52a895
feat: batch_size value must be less than 40
TremaMiguel Mar 8, 2022
29b5cf6
Merge branch 'master' into feat/dynamo_db_online_write_read
adchia Mar 9, 2022
97fd71f
feat: batch_size defaults to 40
TremaMiguel Mar 10, 2022
ddb3f0a
Merge branch 'feat/dynamo_db_online_write_read' of github.com:TremaMi…
TremaMiguel Mar 10, 2022
12064e4
feat: sort dynamodb responses
TremaMiguel Mar 11, 2022
449f60d
merge branch master into feat/dynamo_db_online_write_read
TremaMiguel Mar 12, 2022
3f72228
resolve merge conflicts
TremaMiguel Mar 12, 2022
7a4edbd
test online response proto with redshift:dynamodb
TremaMiguel Mar 12, 2022
3843a02
feat: consistency in batch_size process
TremaMiguel Mar 15, 2022
88e183e
fix: return batch_size times None
TremaMiguel Mar 16, 2022
23cb49a
remove debug code
TremaMiguel Mar 16, 2022
44f97e7
Merge branch 'feast-dev:master' into feat/dynamo_db_online_write_read
TremaMiguel Mar 22, 2022
eaf4940
typo in docstring
TremaMiguel Mar 22, 2022
5bc54d3
batch_size in onlineconfigstore
TremaMiguel Mar 23, 2022
c7ab086
Merge branch 'master' into feat/dynamo_db_online_write_read
TremaMiguel Mar 23, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion sdk/python/feast/online_response.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,18 @@

TIMESTAMP_POSTFIX: str = "__ts"

import logging
import sys

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)


class OnlineResponse:
"""
Expand Down Expand Up @@ -57,7 +69,12 @@ def to_dict(self, include_event_timestamps: bool = False) -> Dict[str, Any]:

for result in self.proto.results:
for idx, feature_ref in enumerate(self.proto.metadata.feature_names.val):
native_type_value = feast_value_type_to_python_type(result.values[idx])
try:
native_type_value = feast_value_type_to_python_type(result.values[idx])
except IndexError as e:
logger.info(f'ERROR -- {e}')
logger.info(f'ONLINE RESPONSE PROTO: {self.proto}')
TremaMiguel marked this conversation as resolved.
Show resolved Hide resolved
raise IndexError(e)
if feature_ref not in response:
response[feature_ref] = [native_type_value]
else:
Expand Down