Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add flags to build multiarch Python SDK containers and test #27723

Merged
merged 22 commits into from
Aug 1, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
00b7148
[New Python Arm Tests] Create yml file, change README file.
celeste-zeng Jul 27, 2023
661c866
Add new script, adjust dependent test files, rename yml file.
celeste-zeng Jul 27, 2023
4fc4f1a
Add steps to build multi-arch Python containers.
celeste-zeng Jul 27, 2023
b838a2f
Adjust error message.
celeste-zeng Jul 27, 2023
91a368b
Regenerate base image requirement, add test suite label.
celeste-zeng Jul 27, 2023
c3f932e
Merge branch 'apache:master' into gh-py-arm
celeste-zeng Jul 27, 2023
f7e810f
Merge branch 'master' into gh-py-arm
celeste-zeng Jul 28, 2023
0b43e66
Add Gcloud authentication, make minor changes to maintain consistency…
celeste-zeng Jul 28, 2023
d605499
Reuse old script, fix formatting, remove unnecessary changes.
celeste-zeng Jul 31, 2023
b0ab6ab
fix formatting.
celeste-zeng Jul 31, 2023
2d2121b
Made changes based on comments.
celeste-zeng Jul 31, 2023
d8cf065
Merge branch 'master' into gh-py-arm
celeste-zeng Jul 31, 2023
a29f105
Remove unused script, set current time as build tag.
celeste-zeng Jul 31, 2023
e945ea1
Fix formatting errors.
celeste-zeng Jul 31, 2023
17830a1
Formatting.
celeste-zeng Jul 31, 2023
2d7ca2f
Revert some changes.
celeste-zeng Jul 31, 2023
b00ac72
Add comment about pushed containers.
celeste-zeng Aug 1, 2023
756f8c3
Update comments on ARM test suites.
celeste-zeng Aug 1, 2023
264d1d0
Made changes based on comments.
celeste-zeng Aug 1, 2023
baf47fe
Make ARCH a variable set by gradle not GH.
celeste-zeng Aug 1, 2023
e7c2dd1
Fix typo
celeste-zeng Aug 1, 2023
fa423f0
Added more detailed comments for script arguments.
celeste-zeng Aug 1, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,6 @@ jobs:
-Ppush-containers
env:
MULTIARCH_TAG: ${{ env.MULTIARCH_TAG }}
ARCH: ARM
USER: github-actions
- name: Archive code coverage results
uses: actions/upload-artifact@v3
Expand Down
7 changes: 4 additions & 3 deletions sdks/python/container/run_validatescontainer.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@

echo "This script must be executed in the root of beam project. Please set GCS_LOCATION, PROJECT and REGION as desired."

if [[ $# != 2 ]]; then
printf "Usage: \n$> ./sdks/python/container/run_validatescontainer.sh <python_version> <sdk_location>"
if [[ $# < 2 ]]; then
printf "Usage: \n$> ./sdks/python/container/run_validatescontainer.sh <python_version> <sdk_location> <optional_archtecture>"
printf "\n\tpython_version: [required] Python version used for container build and run tests."
printf " Sample value: 3.9"
exit 1
Expand All @@ -53,6 +53,7 @@ REGION=${REGION:-us-central1}
IMAGE_PREFIX="$(grep 'docker_image_default_repo_prefix' gradle.properties | cut -d'=' -f2)"
SDK_VERSION="$(grep 'sdk_version' gradle.properties | cut -d'=' -f2)"
PY_VERSION=$1
ARCH=${3:-"x86"}
IMAGE_NAME="${IMAGE_PREFIX}python${PY_VERSION}_sdk"
CONTAINER_PROJECT="sdks:python:container:py${PY_VERSION//.}" # Note: we substitute away the dot in the version.
PY_INTERPRETER="python${PY_VERSION}"
Expand All @@ -74,7 +75,7 @@ CONTAINER=us.gcr.io/$PROJECT/$USER/$IMAGE_NAME
PREBUILD_SDK_CONTAINER_REGISTRY_PATH=us.gcr.io/$PROJECT/$USER/prebuild_python${PY_VERSION//.}_sdk
echo "Using container $CONTAINER"

if [[ "$ARCH" != "ARM" ]]; then
if [[ "$ARCH" == "x86" ]]; then
celeste-zeng marked this conversation as resolved.
Show resolved Hide resolved
# Verify docker image has been built.
docker images | grep "apache/$IMAGE_NAME" | grep "$SDK_VERSION"

Expand Down
3 changes: 2 additions & 1 deletion sdks/python/test-suites/dataflow/common.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -358,7 +358,8 @@ task validatesContainerARM() {
executable 'sh'
args '-c', ". ${envdir}/bin/activate && cd ${rootDir} && ${runScriptsPath} " +
"${project.ext.pythonVersion} " +
"${project.ext.sdkLocation}"
"${project.ext.sdkLocation}" +
"ARM"
}
}
}
Expand Down
Loading