Skip to content

Commit

Permalink
merge master into fix_comment_trigger_against_pr_branch
Browse files Browse the repository at this point in the history
  • Loading branch information
andreydevyatkin committed Aug 3, 2023
2 parents eac9d19 + 42cdd61 commit 0ee2dc7
Show file tree
Hide file tree
Showing 115 changed files with 4,460 additions and 360 deletions.
1 change: 0 additions & 1 deletion .github/REVIEWERS.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,6 @@ labels:
- Abacn
- ahmedabu98
- bvolpato
- manavgarg
exclusionList: []
- name: spanner
reviewers:
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@ Please note that jobs with matrix need to have matrix element in the comment. Ex
| Workflow name | Matrix | Trigger Phrase | Cron Status |
|:-------------:|:------:|:--------------:|:-----------:|
| [ PostCommit Go Dataflow ARM](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Go_Dataflow_ARM.yml) | N/A |`Run Go PostCommit Dataflow ARM`| [![.github/workflows/beam_PostCommit_Go_Dataflow_ARM](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Go_Dataflow_ARM.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Go_Dataflow_ARM.yml) |
| [ PostCommit Java Examples Dataflow ARM ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml) | N/A |`Run Java_Examples_Dataflow_ARM PostCommit`| [![PostCommit Java Examples Dataflow ARM](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml) |
| [ PreCommit Community Metrics ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_CommunityMetrics.yml) | N/A |`Run CommunityMetrics PreCommit`| [![.github/workflows/beam_PreCommit_CommunityMetrics.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_CommunityMetrics.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_CommunityMetrics.yml) |
| [ PreCommit Go ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Go.yml) | N/A |`Run Go PreCommit`| [![.github/workflows/beam_PreCommit_Go.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Go.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Go.yml) |
| [ PreCommit Java Examples Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow.yml) | N/A |`Run Java_Examples_Dataflow PreCommit`| [![.github/workflows/beam_PreCommit_Java_Examples_Dataflow.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow.yml) |
Expand All @@ -137,4 +138,5 @@ Please note that jobs with matrix need to have matrix element in the comment. Ex
| [ PreCommit Website ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website.yml) | N/A |`Run Website PreCommit`| [![.github/workflows/beam_PreCommit_Website.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website.yml) |
| [ PreCommit Website Stage GCS ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website_Stage_GCS.yml) | N/A |`Run Website_Stage_GCS PreCommit`| [![PreCommit Website Stage GCS](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website_Stage_GCS.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website_Stage_GCS.yml) |
| [ PreCommit Whitespace ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Whitespace.yml) | N/A |`Run Whitespace PreCommit`| [![.github/workflows/beam_PreCommit_Whitespace.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Whitespace.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Whitespace.yml) |
| [ Python Validates Container Dataflow ARM ](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) | ['3.8','3.9','3.10','3.11'] | `Run Python Validates Container Dataflow ARM (matrix_element)`| [![.github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) |
| [ Python Validates Container Dataflow ARM ](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) | ['3.8','3.9','3.10','3.11'] | `Run Python ValidatesContainer Dataflow ARM (matrix_element)`| [![.github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) |
| [ PreCommit GoPortable ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_GoPortable.yml) | N/A |`Run GoPortable PreCommit`| [![.github/workflows/beam_PreCommit_GoPortable.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_GoPortable.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_GoPortable.yml) |
2 changes: 1 addition & 1 deletion .github/workflows/beam_PostCommit_Go_Dataflow_ARM.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ permissions:

# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.head_ref || github.ref }}'
group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.sender.login }}-${{ github.event.schedule }}'
cancel-in-progress: true

jobs:
Expand Down
143 changes: 143 additions & 0 deletions .github/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

name: PostCommit Java Examples Dataflow ARM

on:
push:
tags: ['v*']
branches: ['master', 'release-*']
paths:
- 'model/**'
- 'sdks/java/**'
- 'runners/google-cloud-dataflow-java/**'
- 'examples/java/**'
- 'examples/kotlin/**'
- 'release/**'
- '.github/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml'
issue_comment:
types: [created]
schedule:
- cron: '0 */6 * * *'
workflow_dispatch:

# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.sender.login }}-${{ github.event.schedule }}'
cancel-in-progress: true

#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
permissions:
actions: write
pull-requests: read
checks: read
contents: read
deployments: read
id-token: none
issues: read
discussions: read
packages: read
pages: read
repository-projects: read
security-events: read
statuses: read

env:
TAG: $(date +'%Y%m%d-%H%M%S%N')

jobs:
beam_PostCommit_Java_Examples__Dataflow_ARM:
name: beam_PostCommit_Java_Examples__Dataflow_ARM (Run Java_Examples_Dataflow_ARM PostCommit)
runs-on: [self-hosted, ubuntu-20.04, main]
strategy:
fail-fast: false
if: |
github.event_name == 'push' ||
github.event_name == 'schedule' ||
github.event.comment.body == 'Run Java_Examples_Dataflow_ARM PostCommit' ||
github. event_name == 'workflow_dispatch'
steps:
- name: Check out repository code
uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Rerun on comment
if: github.event.comment.body == 'Run Java_Examples_Dataflow_ARM PostCommit'
uses: ./.github/actions/rerun-job-action
with:
pull_request_url: ${{ github.event.issue.pull_request.url }}
github_repository: ${{ github.repository }}
github_token: ${{ secrets.GITHUB_TOKEN }}
github_job: "beam_PostCommit_Java_Examples__Dataflow_ARM (Run Java_Examples_Dataflow_ARM PostCommit)"
github_current_run_id: ${{ github.run_id }}
- name: Setup self-hosted
uses: ./.github/actions/setup-self-hosted-action
with:
requires-py-38: false
requires-py-39: false
requires-go: false
- name: Setup Gradle
uses: gradle/gradle-build-action@v2
with:
cache-read-only: false
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Authenticate on GCP
id: auth
uses: google-github-actions/setup-gcloud@v0
with:
service_account_email: ${{ secrets.GCP_SA_EMAIL }}
service_account_key: ${{ secrets.GCP_SA_KEY }}
project_id: ${{ secrets.GCP_PROJECT_ID }}
export_default_credentials: true
- name: GCloud Docker credential helper
run: |
gcloud auth configure-docker us.gcr.io
- name: run javaExamplesDataflowARMPostCommit script for Java8
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :runners:google-cloud-dataflow-java:arm:examplesJavaRunnerV2IntegrationTestARM
arguments: |
-Pcontainer-architecture-list=arm64,amd64 \
-Ppush-containers \
-Pdocker-repository-root=us.gcr.io/apache-beam-testing/java-postcommit-it/arm \
-Pdocker-tag=${{env.TAG}} \
- name: run javaExamplesDataflowARMPostCommit script for Java11
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :runners:google-cloud-dataflow-java:arm:examplesJavaRunnerV2IntegrationTestARM
arguments: |
-PcompileAndRunTestsWithJava11 \
-Pcontainer-architecture-list=arm64,amd64 \
-Ppush-containers \
-Pdocker-repository-root=us.gcr.io/apache-beam-testing/java-postcommit-it/arm \
-Pdocker-tag=${{env.TAG}} \
- name: run javaExamplesDataflowARMPostCommit script for Java17
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :runners:google-cloud-dataflow-java:arm:examplesJavaRunnerV2IntegrationTestARM
arguments: |
-PcompileAndRunTestsWithJava17 \
-Pcontainer-architecture-list=arm64,amd64 \
-Ppush-containers \
-Pdocker-repository-root=us.gcr.io/apache-beam-testing/java-postcommit-it/arm \
-Pdocker-tag=${{env.TAG}} \
- name: Upload test report
uses: actions/upload-artifact@v3
with:
name: java-code-coverage-report
path: "**/build/test-results/**/*.xml"
85 changes: 85 additions & 0 deletions .github/workflows/beam_PreCommit_GoPortable.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

name: PreCommit GoPortable

on:
push:
tags: ['v*']
branches: ['master', 'release-*']
paths: ['model/**', 'sdks/go.**', 'release/**','.github/workflows/beam_PreCommit_GoPortable.yml']
pull_request_target:
branches: ['master', 'release-*']
paths: ['model/**', 'sdks/go.**', 'release/**']
issue_comment:
types: [created]
schedule:
- cron: '0 */6 * * *'
workflow_dispatch:

# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.sender.login }}-${{ github.event.schedule }}'
cancel-in-progress: true

# Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
permissions:
actions: write
pull-requests: read
checks: read
contents: read
deployments: read
id-token: none
issues: read
discussions: read
packages: read
pages: read
repository-projects: read
security-events: read
statuses: read

jobs:
beam_PreCommit_GoPortable:
timeout-minutes: 120
if: |
github.event_name == 'push' ||
github.event_name == 'pull_request_target' ||
github.event.comment.body == 'Run GoPortable PreCommit' ||
github.event_name == 'schedule'
runs-on: [self-hosted, ubuntu-20.04, main]
name: beam_PreCommit_GoPortable (Run GoPortable PreCommit)
steps:
- name: Git checkout
uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Rerun on comment
if: github.event.comment.body == 'Run GoPortable PreCommit'
uses: ./.github/actions/rerun-job-action
with:
pull_request_url: ${{ github.event.issue.pull_request.url }}
github_repository: ${{ github.repository }}
github_token: ${{ secrets.GITHUB_TOKEN }}
github_job: ${{ github.job }}
github_current_run_id: ${{ github.run_id }}
- name: Setup self-hosted
uses: ./.github/actions/setup-self-hosted-action
with:
requires-py-39: false
requires-go: false
- name: Run goPortablePreCommit script
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :goPortablePreCommit
23 changes: 13 additions & 10 deletions .github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,20 +44,21 @@ permissions:

# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.head_ref || github.ref }}'
group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.sender.login }}-${{ github.event.schedule }}'
cancel-in-progress: true

jobs:
beam_Python_ValidatesContainer_Dataflow_ARM:
name: beam_Python_ValidatesContainer_Dataflow_ARM
name: beam_Python_ValidatesContainer_Dataflow_ARM (Run Python ValidatesContainer Dataflow ARM 3.x)
strategy:
fail-fast: false
matrix:
python_version: ['3.8','3.9','3.10','3.11']
if: |
github.event_name == 'push' ||
startsWith(github.event.comment.body, 'Run Python ValidatesContainer Dataflow ARM') ||
github.event_name == 'schedule'
github.event_name == 'schedule' ||
github.event_name == 'workflow_dispatch'
runs-on: [self-hosted, ubuntu-20.04, main]
env:
MULTIARCH_TAG: $(date +'%Y%m%d-%H%M%S%N')
Expand All @@ -72,7 +73,8 @@ jobs:
with:
comment_phrase: ${{ steps.set_comment_body.outputs.comment_body }}
github_token: ${{ secrets.GITHUB_TOKEN }}
github_job: "${{ github.job }} (${{ matrix.python_version }})"
github_job: "beam_Python_ValidatesContainer_Dataflow_ARM (Run Python ValidatesContainer Dataflow ARM 3.x) (${{ matrix.python_version }})"
github_current_run_id: ${{ github.run_id }}
- name: Install Python
uses: actions/setup-python@v4
with:
Expand Down Expand Up @@ -101,19 +103,20 @@ jobs:
echo "py_ver_clean=$PY_VER_CLEAN" >> $GITHUB_OUTPUT
- name: run Python validatesContainerARM script
id: run_script
run: |
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :sdks:python:test-suites:dataflow:py${{steps.set_py_ver_clean.outputs.py_ver_clean}}:validatesContainerARM
arguments: |
-PpythonVersion=${{ matrix.python_version }} \
-Pcontainer-architecture-list=arm64,amd64 \
-Pdocker-repository=us.gcr.io/apache-beam-testing/github-actions \
-Pdocker-tag=${{ env.MULTIARCH_TAG }} \
-Ppush-containers
env:
MULTIARCH_TAG: ${{ env.MULTIARCH_TAG }}
USER: github-actions
-Ppush-containers \
env:
MULTIARCH_TAG: ${{ env.MULTIARCH_TAG }}
USER: github-actions
- name: Archive code coverage results
uses: actions/upload-artifact@v3
with:
name: python-code-coverage-report
path: "**/pytest*.xml"
path: "**/pytest*.xml"
2 changes: 1 addition & 1 deletion .test-infra/tools/stale_dataflow_prebuilt_image_cleaner.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ set -euo pipefail
# Images more than 5 day old and not the latest (either has latest label or newest)

PUBLIC_REPOSITORIES=(beam-sdk beam_portability)
PRIVATE_REPOSITORIES=(java-postcommit-it python-postcommit-it jenkins)
PRIVATE_REPOSITORIES=(java-postcommit-it python-postcommit-it jenkins github-actions)
# set as the same as 6-week release period
DELETE_BEFORE_DAY=$(date --iso-8601=s -d '6 weeks ago')

Expand Down
Loading

0 comments on commit 0ee2dc7

Please sign in to comment.