-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Redis cache support for enrichment transform #30307
Conversation
c668d69
to
71beae2
Compare
Assigning reviewers. If you would like to opt out of this review, comment R: @tvalentyn for label python. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
71beae2
to
7a721dd
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Haven't gotten through the tests yet, but on the whole this looks pretty good to me; left some comments
cache_request = self.source_caller.get_cache_request(element) | ||
# check if the caller is a enrichment handler. EnrichmentHandler | ||
# provides the request format for cache. | ||
if cache_request: | ||
encoded_request = self.request_coder.encode(cache_request) | ||
else: | ||
encoded_request = self.request_coder.encode(element) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: I think this logic is shared and can be extracted out of the if/else
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The write part is similar but it has tuples as input
sdks/python/apache_beam/transforms/enrichment_handlers/bigtable.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Had one small nit, otherwise this looks good to me once tests pass
Thank you both Danny and Valentyn for your reviews! |
Thanks @tvalentyn! Made changes from some of your suggestions but kept a few of them as is. If this looks good I'll drop the postcommit file once tests pass and merge. Thanks! |
521acbb
to
66a7abd
Compare
@tvalentyn I ended up using the All tests passed. Dropping the postcommit file |
This PR implements Redis caching support for
RequestResponseIO
and integrates it with Enrichment transform. It also enables the test for RRIO and Enrichment with HTTP Mock API.Part of #29787
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123
), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>
instead.CHANGES.md
with noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.