Skip to content

Commit

Permalink
Report tests to datadog from Job that executed them
Browse files Browse the repository at this point in the history
Previously, each test run was associated with the Job
that uploaded them.
This makes it harder to understand under which configuration
a specific test failed.
Now we immediately upload test results from the
respective job.

This increases feedback time slightly by up to 1 minute
(the slowest single upload time).
I'd say this is an acceptable tradeoff given our
current CI times.

This reduces the time until a "Retry failed jobs"
is avaiable since we no longer need to pass the test
results between jobs.
  • Loading branch information
eps1lon committed Dec 11, 2024
1 parent e279a4a commit 4b1b811
Show file tree
Hide file tree
Showing 2 changed files with 21 additions and 52 deletions.
41 changes: 0 additions & 41 deletions .github/workflows/build_and_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -520,47 +520,6 @@ jobs:
stepName: 'test-ppr-prod-${{ matrix.group }}'
secrets: inherit

report-test-results-to-datadog:
needs:
[
'changes',
'test-unit',
'test-dev',
'test-prod',
'test-integration',
'test-ppr-dev',
'test-ppr-prod',
'test-ppr-integration',
'test-turbopack-dev',
'test-turbopack-integration',
'test-turbopack-production',
'test-turbopack-production-integration',
]
if: ${{ always() && needs.changes.outputs.docs-only == 'false' && !github.event.pull_request.head.repo.fork }}

runs-on: ubuntu-latest
name: report test results to datadog
steps:
- name: Download test report artifacts
id: download-test-reports
uses: actions/download-artifact@v4
with:
pattern: test-reports-*
path: test
merge-multiple: true

- name: Upload test report to datadog
run: |
if [ -d ./test/test-junit-report ]; then
# Add a `test.type` tag to distinguish between turbopack and next.js runs
DD_ENV=ci npx @datadog/datadog-ci@2.23.1 junit upload --tags test.type:nextjs --service nextjs ./test/test-junit-report
fi
if [ -d ./test/turbopack-test-junit-report ]; then
# Add a `test.type` tag to distinguish between turbopack and next.js runs
DD_ENV=ci npx @datadog/datadog-ci@2.23.1 junit upload --tags test.type:turbopack --service nextjs ./test/turbopack-test-junit-report
fi
tests-pass:
needs:
[
Expand Down
32 changes: 21 additions & 11 deletions .github/workflows/build_reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,12 @@ env:
DATADOG_API_KEY: ${{ secrets.DATA_DOG_API_KEY }}
NEXT_JUNIT_TEST_REPORT: 'true'
DD_ENV: 'ci'
DD_CI_JOB_NAME: ${{ inputs.stepName }}
DD_GITHUB_JOB_NAME: ${{ inputs.stepName }}
# Better than Datadog's default (e.g. https://github.com/vercel/next.js/commit/3dd3d19ba63c7ca790f3bf39e0e15152b597547c/checks)
# Job id is not provided by GitHub. Using workflow url as a fallback.
# We could derived the job id from GH API response but this is just more network slowness.
DD_CI_JOB_URL: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}/attempts/${{ github.run_attempt }}
TEST_TIMINGS_TOKEN: ${{ secrets.TEST_TIMINGS_TOKEN }}
NEXT_TEST_JOB: 1
VERCEL_TEST_TOKEN: ${{ secrets.VERCEL_TEST_TOKEN }}
Expand Down Expand Up @@ -213,18 +219,22 @@ jobs:
name: webpack bundle analysis stats-${{ steps.var.outputs.input_step_key }}
path: packages/next/dist/compiled/next-server/report.*.html

- name: Upload test report artifacts
uses: actions/upload-artifact@v4
- name: Upload test report to datadog
if: ${{ inputs.afterBuild && always() }}
with:
name: test-reports-${{ steps.var.outputs.input_step_key }}
path: |
test/test-junit-report
test/turbopack-test-junit-report
if-no-files-found: ignore

# upload playwright snapshots from failed tests
- name: Upload test report artifacts
run: |
echo "DD_CI_JOB_NAME: ${{ env.DD_CI_JOB_NAME }}"
echo "DD_CI_JOB_URL: ${{ env.DD_CI_JOB_URL }}"
if [ -d ./test/test-junit-report ]; then
# Add a `test.type` tag to distinguish between turbopack and next.js runs
npx @datadog/datadog-ci@2.45.1 junit upload --service nextjs --tags test.type:nextjs ./test/test-junit-report
fi
if [ -d ./test/turbopack-test-junit-report ]; then
# Add a `test.type` tag to distinguish between turbopack and next.js runs
npx @datadog/datadog-ci@2.45.1 junit upload --service nextjs --tags test.type:turbopack ./test/turbopack-test-junit-report
fi
- name: Upload Playwright Snapshots
uses: actions/upload-artifact@v4
if: ${{ inputs.afterBuild && always() }}
with:
Expand Down

0 comments on commit 4b1b811

Please sign in to comment.