Skip to content

Commit

Permalink
Add regression check automation
Browse files Browse the repository at this point in the history
This commit adds benchmark_regression_test.yml workflow that runs a
regression test by comparing the benchmark results of the current branch
with the previous benchmark results of the code in main branch.

Signed-off-by: Arjun Raja Yogidas <arjunry@amazon.com>
  • Loading branch information
coderbirju committed Aug 14, 2023
1 parent bde429d commit 64fca1a
Showing 1 changed file with 68 additions and 0 deletions.
68 changes: 68 additions & 0 deletions .github/workflows/benchmark_regression_test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
name: Benchmark Regression Check

on:
pull_request:
branches: [ main ]
paths:
- '**.go'
- 'go.*'
- 'cmd/go.*'
- 'Makefile'
- 'Dockerfile'
- 'integration/**'
- 'scripts/**'
- '.github/workflows/**'

jobs:
benchmark-and-fetch-previous-results_and_compare:
name: Run Benchmark on current Branch
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v3
- uses: actions/setup-go@v4
with:
go-version: '1.18.10'
- run: make
- name: Run benchmark
run: make benchmarks-perf-test
- name: Make previous directory
run: mkdir -v ${{ github.workspace }}/previous
- name: Download previous run artifact
id: download-artifact
uses: dawidd6/action-download-artifact@v2
with:
github_token: ${{secrets.GITHUB_TOKEN}}
name: benchmark-result-artifact
name_is_regexp: true
path: ${{ github.workspace }}/previous
repo: ${{ github.repository }}
check_artifacts: false
search_artifacts: true
skip_unpack: false
if_no_artifact_found: fail
workflow: benchmark_visualization.yml
- name: Perform Comparison and log results
id: run-compare
run: |
sudo chmod +x ${{ github.workspace }}/scripts/check_regression.sh
if sudo ${{ github.workspace }}/scripts/check_regression.sh ${{ github.workspace }}/previous/benchmark-result-artifact/results.json ${{github.workspace}}/benchmark/performanceTest/output/results.json; then
echo "Comparison successful. All P90 values are within the acceptable range."
else
echo "Comparison failed. Current P90 values exceed 110% of the corresponding past values."
echo "regression-detected=true" >> $GITHUB_OUTPUT
fi
- name: Stop the workflow if regression is detected
if: steps.run-compare.outputs.regression-detected == 'true'
uses: actions/github-script@v6
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const comment = `
:warning: **Regression Detected** :warning:
The benchmark comparison indicates that there has been a performance regression.
Please investigate and address the issue.
To Investigate check logs of the previous job above.
`;
core.setFailed(comment);

0 comments on commit 64fca1a

Please sign in to comment.