-
Notifications
You must be signed in to change notification settings - Fork 55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upgrade Apache Beam to 2.20.0. #583
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Tural for updating our Beam.
Please make sure our integration tests pass.
I remember when I tried to update to 2.19.0 I had an issue with DirectRunner tests that was most likely due to this:
BEAM-8402 Backwards incompatible change related to how Environments are represented in Python DirectRunner.
I am not sure if this is still problematic, just wanted to let you know.
# Note that adding 'google-api-python-client>=1.6' causes some dependency | ||
# mismatch issues. This is fatal if using 'setup.py install', but works on | ||
# 'pip install .' as it ignores conflicting versions. See Issue #71. | ||
'google-api-python-client>=1.6,<1.7.12', | ||
'intervaltree>=2.1.0,<2.2.0', | ||
'mmh3<2.6', | ||
# Refer to issue #528 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is #528 resolved? If yes, then let's refer to it in this PR and close it as we land this. If not then I don't think we should remove that comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well seems like pip built everything for me correctly and unit tests passed (although integration tests are still running). That issue is already closed tho.
setup.py
Outdated
# Refer to issue #528 | ||
'google-cloud-storage<1.23.0', | ||
'google-cloud-storage', | ||
'google-resumable-media<0.6dev,>=0.5.0', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why we are adding this new dependency while we don't use at all? This seems to be still related to #528
If that's the case, cannot we resolve this issue the same way we resolved it before: pin to a version of google-cloud-storage
that is compatible to updated beam dependency:
'google-cloud-bigquery>=1.6.0,<=1.24.0',
to avoid the error message?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Huh seems I don't need it anymore - I removed the google-cloud-bigquery
version later, so I guess I only needed it for <1.23.0. Removed.
No description provided.