Add flags to build multiarch Python SDK containers and test #27723

celeste-zeng · 2023-07-27T23:34:05Z

Use new flags to build and push multi-arch Python containers(will not affect the current building of x86 containers if flags are not specified during build);
Add a validatescontainer arm script so it builds and pushes a multi-arch python container, and use it to run a dataflow wordcount pipeline;
Change the script to adjust base image requirements for Python SDK containers on ARM.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

codecov · 2023-07-27T23:49:46Z

Codecov Report

Merging #27723 (fa423f0) into master (36e8254) will increase coverage by 0.03%.
Report is 12 commits behind head on master.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #27723      +/-   ##
==========================================
+ Coverage   70.91%   70.95%   +0.03%     
==========================================
  Files         860      860              
  Lines      104860   104866       +6     
==========================================
+ Hits        74364    74404      +40     
+ Misses      28937    28902      -35     
- Partials     1559     1560       +1

Flag	Coverage Δ
go	`53.65% <ø> (+0.09%)`	⬆️
python	`80.00% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 8 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-actions · 2023-07-28T00:50:10Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @damccorm for label python.
R: @Abacn for label build.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

… with other yml files.

.github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml

sdks/python/apache_beam/examples/wordcount_it_test.py

sdks/python/container/run_validatescontainer_arm.sh

sdks/python/apache_beam/examples/wordcount_it_test.py

sdks/python/container/run_generate_requirements.sh

sdks/python/container/run_validatescontainer.sh

tvalentyn · 2023-08-01T17:16:17Z

.github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml

+        run: |
+          PY_VER=${{ matrix.python_version }}
+          PY_VER_CLEAN=${PY_VER//.}
+          echo "py_ver_clean=$PY_VER_CLEAN" >> $GITHUB_OUTPUT


nit: debugging leftover?

I think all python GH tests now has this step, added by #27712

.github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml

sdks/python/container/run_validatescontainer.sh

sdks/python/test-suites/dataflow/common.gradle

tvalentyn · 2023-08-01T17:53:46Z

sdks/python/container/run_generate_requirements.sh

@@ -104,4 +104,13 @@ EOT
 # Remove pkg_resources to guard against
 # https://stackoverflow.com/questions/39577984/what-is-pkg-resources-0-0-0-in-output-of-pip-freeze-command
 pip freeze | grep -v pkg_resources >> "$REQUIREMENTS_FILE"
+
+if grep -q "tensorflow==" "$REQUIREMENTS_FILE"; then


i recommend to move changes to this file into a separate PR.

Done. #27789

sdks/python/container/run_validatescontainer.sh

…7723)

celeste-zeng added 5 commits July 27, 2023 21:11

[New Python Arm Tests] Create yml file, change README file.

00b7148

Add new script, adjust dependent test files, rename yml file.

661c866

Add steps to build multi-arch Python containers.

4fc4f1a

Adjust error message.

b838a2f

Regenerate base image requirement, add test suite label.

91a368b

github-actions bot added python build examples docker labels Jul 27, 2023

Merge branch 'apache:master' into gh-py-arm

c3f932e

github-actions bot added the Next Action: Reviewers label Jul 28, 2023

tvalentyn self-requested a review July 28, 2023 17:26

celeste-zeng added 2 commits July 28, 2023 14:39

Merge branch 'master' into gh-py-arm

f7e810f

Add Gcloud authentication, make minor changes to maintain consistency…

0b43e66

… with other yml files.

tvalentyn reviewed Jul 28, 2023

View reviewed changes

celeste-zeng added 2 commits July 31, 2023 17:21

Reuse old script, fix formatting, remove unnecessary changes.

d605499

fix formatting.

b0ab6ab

tvalentyn reviewed Jul 31, 2023

View reviewed changes

sdks/python/apache_beam/examples/wordcount_it_test.py Outdated Show resolved Hide resolved

tvalentyn reviewed Jul 31, 2023

View reviewed changes

sdks/python/container/run_generate_requirements.sh Outdated Show resolved Hide resolved

celeste-zeng added 6 commits July 31, 2023 20:44

Made changes based on comments.

2d2121b

Merge branch 'master' into gh-py-arm

d8cf065

Remove unused script, set current time as build tag.

a29f105

Fix formatting errors.

e945ea1

Formatting.

17830a1

Revert some changes.

2d7ca2f

tvalentyn reviewed Jul 31, 2023

View reviewed changes

sdks/python/container/run_validatescontainer.sh Show resolved Hide resolved

celeste-zeng added 2 commits August 1, 2023 01:23

Add comment about pushed containers.

b00ac72

Update comments on ARM test suites.

756f8c3

tvalentyn reviewed Aug 1, 2023

View reviewed changes

celeste-zeng added 3 commits August 1, 2023 19:01

Made changes based on comments.

264d1d0

Make ARCH a variable set by gradle not GH.

baf47fe

Fix typo

e7c2dd1

celeste-zeng mentioned this pull request Aug 1, 2023

[Feature Request]: Add ARM Support to All Beam Container Images #27674

Closed

15 tasks

Added more detailed comments for script arguments.

fa423f0

tvalentyn reviewed Aug 1, 2023

View reviewed changes

sdks/python/container/run_validatescontainer.sh Show resolved Hide resolved

tvalentyn approved these changes Aug 1, 2023

View reviewed changes

tvalentyn merged commit d4f09da into apache:master Aug 1, 2023
75 checks passed

bullet03 pushed a commit to akvelon/beam that referenced this pull request Aug 11, 2023

Add flags to build multiarch Python SDK containers and test (apache#2…

4bf7eda

…7723)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add flags to build multiarch Python SDK containers and test #27723

Add flags to build multiarch Python SDK containers and test #27723

celeste-zeng commented Jul 27, 2023 •

edited

Loading

codecov bot commented Jul 27, 2023 •

edited

Loading

github-actions bot commented Jul 28, 2023

tvalentyn Aug 1, 2023

celeste-zeng Aug 1, 2023

tvalentyn Aug 1, 2023

celeste-zeng Aug 1, 2023

Add flags to build multiarch Python SDK containers and test #27723

Add flags to build multiarch Python SDK containers and test #27723

Conversation

celeste-zeng commented Jul 27, 2023 • edited Loading

GitHub Actions Tests Status (on master branch)

codecov bot commented Jul 27, 2023 • edited Loading

Codecov Report

github-actions bot commented Jul 28, 2023

tvalentyn Aug 1, 2023

Choose a reason for hiding this comment

celeste-zeng Aug 1, 2023

Choose a reason for hiding this comment

tvalentyn Aug 1, 2023

Choose a reason for hiding this comment

celeste-zeng Aug 1, 2023

Choose a reason for hiding this comment

celeste-zeng commented Jul 27, 2023 •

edited

Loading

codecov bot commented Jul 27, 2023 •

edited

Loading