Upgrade Apache Beam to 2.20.0. #583
Conversation
There was a problem hiding this comment.
Thanks Tural for updating our Beam.
Please make sure our integration tests pass.
I remember when I tried to update to 2.19.0 I had an issue with DirectRunner tests that was most likely due to this:
BEAM-8402 Backwards incompatible change related to how Environments are represented in Python DirectRunner.
I am not sure if this is still problematic, just wanted to let you know.
| # Note that adding 'google-api-python-client>=1.6' causes some dependency | ||
| # mismatch issues. This is fatal if using 'setup.py install', but works on | ||
| # 'pip install .' as it ignores conflicting versions. See Issue #71. | ||
| 'google-api-python-client>=1.6,<1.7.12', | ||
| 'intervaltree>=2.1.0,<2.2.0', | ||
| 'mmh3<2.6', | ||
| # Refer to issue #528 |
There was a problem hiding this comment.
Is #528 resolved? If yes, then let's refer to it in this PR and close it as we land this. If not then I don't think we should remove that comment.
There was a problem hiding this comment.
Well seems like pip built everything for me correctly and unit tests passed (although integration tests are still running). That issue is already closed tho.
setup.py
Outdated
| # Refer to issue #528 | ||
| 'google-cloud-storage<1.23.0', | ||
| 'google-cloud-storage', | ||
| 'google-resumable-media<0.6dev,>=0.5.0', |
There was a problem hiding this comment.
Why we are adding this new dependency while we don't use at all? This seems to be still related to #528
If that's the case, cannot we resolve this issue the same way we resolved it before: pin to a version of google-cloud-storage that is compatible to updated beam dependency:
'google-cloud-bigquery>=1.6.0,<=1.24.0',
to avoid the error message?
There was a problem hiding this comment.
Huh seems I don't need it anymore - I removed the google-cloud-bigquery version later, so I guess I only needed it for <1.23.0. Removed.
No description provided.