[Failing Test]: Python PostCommit failing hdfsIntegrationTest in generate_external_transform_wrappers

[Abacn]

What happened?

Caused by #29834

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Abacn]

Find two breakage on installing Python SDK from source.

#30198 #29834 in both cases, it assumes a full beam repository present. website / java SDK. However this is not the case in hdfsIntegrationTest, where only Python SDK source (as well as protos) are present. Then install beam will fail when executing setup.py

CC: @robertwb @ahmedabu98 should we make copy_tests_from_docs or generate_external_transform_wrappers optional?

[robertwb]

sdist should provide a valid bundle for install (e.g. by copying the relevant files from the larger repository, similar to how it generates protos). From here one should be able to install from source using setup.py (which should verify the required files are present, and fail otherwise. This should not be optional as it may result in a partial install).

[ahmedabu98]

I see the script copies over necessary files for protos here:

beam/sdks/python/apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh

Line 40 in 2b8a737

cp -r ${ROOT_DIR}/model ${CONTEXT_DIR}/

[Abacn]

There is another new failure:

:sdks:python:test-suites:dataflow:py38:tensorRTtests

 ERROR:apache_beam.runners.dataflow.dataflow_runner:: JOB_MESSAGE_ERROR: Traceback (most recent call last):
   File "apache_beam/runners/common.py", line 1435, in apache_beam.runners.common.DoFnRunner.process
   File "apache_beam/runners/common.py", line 636, in apache_beam.runners.common.SimpleInvoker.invoke_process
 
   File "apache_beam/runners/common.py", line 1611, in apache_beam.runners.common._OutputHandler.handle_process_outputs
   File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1555, in process
     initial_restriction = self.restriction_provider.initial_restriction(
   File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/io/iobase.py", line 1631, in initial_restriction
     range_tracker = element_source.get_range_tracker(None, None)
   File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/io/filebasedsource.py", line 206, in get_range_tracker
     return self._get_concat_source().get_range_tracker(
   File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/options/value_provider.py", line 193, in _f
 > Task :sdks:python:test-suites:dataflow:py38:tensorRTtests
     return fnc(self, *args, **kwargs)
   File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/io/filebasedsource.py", line 144, in _get_concat_source
     match_result = FileSystems.match([pattern])[0]
   File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/io/filesystems.py", line 204, in match
     return filesystem.match(patterns, limits)
   File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.8/site-packages/apache_beam/io/filesystem.py", line 804, in match
     raise BeamIOError("Match operation failed", exceptions)
 apache_beam.io.filesystem.BeamIOError: Match operation failed with exceptions {'gs://apache-beam-ml/testing/inputs/tensorrt_image_file_names.txt': AttributeError("'Credentials' object has no attribute 'universe_domain'")}

Update: Caused by googleapis/google-cloud-python#12254 but only surfaced to Beam until March 4th, likely transient dependency upgrade