Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TT-1834] Adds Splunk Flakeguard uploads #15718

Merged
merged 21 commits into from
Dec 20, 2024
+40 −15
Merged

[TT-1834] Adds Splunk Flakeguard uploads #15718

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
21485c8
Adds Splunk Flakeguard uploads
kalverra Dec 16, 2024
0efff77
Fix data path
kalverra Dec 16, 2024
7f6f044
Secrets
kalverra Dec 16, 2024
f94b88a
Proper secrets
kalverra Dec 16, 2024
eb0150d
Add proper splunk events
kalverra Dec 17, 2024
61c3d24
Fix JQ
kalverra Dec 17, 2024
da961e2
Better debugging for failing calls
kalverra Dec 17, 2024
37739df
Properly format for Splunk events
kalverra Dec 17, 2024
689af11
Fix flakeguard decoding
kalverra Dec 17, 2024
51d7d4e
Compact and debug
kalverra Dec 18, 2024
29a657e
Merge branch 'develop' of github.com:smartcontractkit/chainlink into …
kalverra Dec 18, 2024
b67d937
Updated go version
kalverra Dec 19, 2024
d4f5b39
Input splunk params
kalverra Dec 19, 2024
ba560b6
Update event types
kalverra Dec 19, 2024
1f0a2fc
Fix typo
kalverra Dec 19, 2024
3ede8e2
Batch splunk sends
kalverra Dec 19, 2024
a8a7486
Fix if triggers
kalverra Dec 19, 2024
9cb2c16
Use success failure
kalverra Dec 19, 2024
7f8b93c
Revert name change
kalverra Dec 20, 2024
8c5444e
Merge branch 'develop' of github.com:smartcontractkit/chainlink into …
kalverra Dec 20, 2024
18c45b4
Fix splunk event
kalverra Dec 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/ci-core.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -561,6 +561,8 @@ jobs:
secrets:
SLACK_BOT_TOKEN: ${{ secrets.QA_SLACK_API_KEY }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
FLAKEGUARD_SPLUNK_ENDPOINT: ${{ secrets.FLAKEGUARD_SPLUNK_ENDPOINT }}
FLAKEGUARD_SPLUNK_HEC: ${{ secrets.FLAKEGUARD_SPLUNK_HEC }}

trigger-flaky-test-detection-for-deployment-project:
name: Flakeguard Deployment Project
Expand All @@ -580,6 +582,9 @@ jobs:
secrets:
SLACK_BOT_TOKEN: ${{ secrets.QA_SLACK_API_KEY }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
FLAKEGUARD_SPLUNK_ENDPOINT: ${{ secrets.FLAKEGUARD_SPLUNK_ENDPOINT }}
FLAKEGUARD_SPLUNK_HEC: ${{ secrets.FLAKEGUARD_SPLUNK_HEC }}


clean:
name: Clean Go Tidy & Generate
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/flakeguard-nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,5 @@ jobs:
secrets:
SLACK_BOT_TOKEN: ${{ secrets.QA_SLACK_API_KEY }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

FLAKEGUARD_SPLUNK_ENDPOINT: ${{ secrets.FLAKEGUARD_SPLUNK_ENDPOINT }}
FLAKEGUARD_SPLUNK_HEC: ${{ secrets.FLAKEGUARD_SPLUNK_HEC }}
3 changes: 2 additions & 1 deletion .github/workflows/flakeguard-on-demand.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,4 +69,5 @@ jobs:
secrets:
SLACK_BOT_TOKEN: ${{ secrets.QA_SLACK_API_KEY }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

FLAKEGUARD_SPLUNK_ENDPOINT: ${{ secrets.FLAKEGUARD_SPLUNK_ENDPOINT }}
FLAKEGUARD_SPLUNK_HEC: ${{ secrets.FLAKEGUARD_SPLUNK_HEC }}
44 changes: 31 additions & 13 deletions .github/workflows/flakeguard.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ on:
repoUrl:
required: true
type: string
default: 'https://github.com/smartcontractkit/chainlink'
description: 'The URL of the repository to compare changes for detecting flaky tests.'
projectPath:
required: true
Expand Down Expand Up @@ -54,6 +55,12 @@ on:
required: false
GH_TOKEN:
required: true
FLAKEGUARD_SPLUNK_ENDPOINT:
description: "The Splunk HTTP Event Collector (HEC) endpoint."
required: true
FLAKEGUARD_SPLUNK_HEC:
description: "The Splunk HTTP Event Collector (HEC) token."
required: true

env:
GIT_BASE_REF: ${{ inputs.baseRef }}
Expand Down Expand Up @@ -116,14 +123,15 @@ jobs:
echo "git_base_sha=" >> $GITHUB_OUTPUT
fi

- name: Set up Go 1.21.9
uses: actions/setup-go@v5.0.2
- name: Setup Go
uses: ./.github/actions/setup-go
with:
cache: false
restore-build-cache-only: "true"

- name: Install flakeguard
if: ${{ inputs.runAllTests == false }}
shell: bash
run: go install github.com/smartcontractkit/chainlink-testing-framework/tools/flakeguard@ea4ffd8c51ce02efebf5ea6bca503fe10b6cee92 # flakguard@0.1.0
run: go install github.com/smartcontractkit/chainlink-testing-framework/tools/flakeguard@7c45cff27ac6b0d4244754660661cdbfcfaf2f9e # flakguard@0.1.0

- name: Find new or updated test packages
if: ${{ inputs.runAllTests == false }}
Expand Down Expand Up @@ -282,7 +290,7 @@ jobs:

- name: Install flakeguard
shell: bash
run: go install github.com/smartcontractkit/chainlink-testing-framework/tools/flakeguard@ea4ffd8c51ce02efebf5ea6bca503fe10b6cee92 # flakguard@0.1.0
run: go install github.com/smartcontractkit/chainlink-testing-framework/tools/flakeguard@7c45cff27ac6b0d4244754660661cdbfcfaf2f9e # flakguard@0.1.0

- name: Run tests with flakeguard
shell: bash
Expand Down Expand Up @@ -310,6 +318,10 @@ jobs:
uses: actions/checkout@9bb56186c3b09b4f86b1c65136769dd318469633 # v4.1.2
with:
ref: ${{ env.GIT_HEAD_REF }}
- name: Setup Go
uses: ./.github/actions/setup-go
with:
restore-build-cache-only: "true"

- name: Set Pretty Project Path
id: set_project_path_pretty
Expand All @@ -329,7 +341,7 @@ jobs:

- name: Install flakeguard
shell: bash
run: go install github.com/smartcontractkit/chainlink-testing-framework/tools/flakeguard@ea4ffd8c51ce02efebf5ea6bca503fe10b6cee92 # flakguard@0.1.0
run: go install github.com/smartcontractkit/chainlink-testing-framework/tools/flakeguard@7c45cff27ac6b0d4244754660661cdbfcfaf2f9e # flakguard@0.1.0

- name: Aggregate Flakeguard Results
id: results
Expand All @@ -354,7 +366,11 @@ jobs:
--repo-url "${{ inputs.repoUrl }}" \
--base-sha "${{ needs.get-tests.outputs.git_base_sha }}" \
--head-sha "${{ needs.get-tests.outputs.git_head_sha }}" \
--github-workflow-name "${{ github.workflow }}"
--github-workflow-name "${{ github.workflow }}" \
--github-workflow-run-url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
--splunk-url "${{ secrets.FLAKEGUARD_SPLUNK_ENDPOINT }}" \
--splunk-token "${{ secrets.FLAKEGUARD_SPLUNK_HEC }}" \
--splunk-event "${{ github.event }}"

# Print out the summary file
echo -e "\nFlakeguard Summary:"
Expand All @@ -365,23 +381,23 @@ jobs:
echo "summary=$summary" >> $GITHUB_OUTPUT

- name: Upload All Test Results as Artifact
if: ${{ fromJSON(steps.results.outputs.summary).total_tests > 0 }}
if: ${{ (success() || failure()) && fromJSON(steps.results.outputs.summary).total_tests > 0 }}
uses: actions/upload-artifact@v4.4.3
with:
path: ./flakeguard-report/all-test-results.json
name: all-test-results.json
retention-days: 90

- name: Upload Failed Test Results as Artifact
if: ${{ fromJSON(steps.results.outputs.summary).failed_runs > 0 }}
if: ${{ (success() || failure()) && fromJSON(steps.results.outputs.summary).failed_runs > 0 }}
uses: actions/upload-artifact@v4.4.3
with:
path: ./flakeguard-report/failed-test-results.json
name: failed-test-results.json
retention-days: 90

- name: Upload Failed Test Results With Logs as Artifact
if: ${{ fromJSON(steps.results.outputs.summary).failed_runs > 0 }}
if: ${{ (success() || failure()) && fromJSON(steps.results.outputs.summary).failed_runs > 0 }}
uses: actions/upload-artifact@v4.4.3
with:
path: ./flakeguard-report/failed-test-results-with-logs.json
Expand All @@ -390,6 +406,7 @@ jobs:

- name: Generate Flakeguard Reports
shell: bash
if: success() || failure()
env:
GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
run: |
Expand Down Expand Up @@ -432,6 +449,7 @@ jobs:
fi

- name: Add Github Summary
if: (success() || failure())
run: |
FILE_SIZE=$(wc -c < ./flakeguard-report/all-test-summary.md)
echo "File size: $FILE_SIZE bytes"
Expand All @@ -446,7 +464,7 @@ jobs:
fi

- name: Post comment on PR if flaky tests found
if: ${{ fromJSON(steps.results.outputs.summary).flaky_tests > 0 && github.event_name == 'pull_request' }}
if: ${{ (success() || failure()) && fromJSON(steps.results.outputs.summary).flaky_tests > 0 && github.event_name == 'pull_request' }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this different from always()?

Suggested change
if: ${{ (success() || failure()) && fromJSON(steps.results.outputs.summary).flaky_tests > 0 && github.event_name == 'pull_request' }}
if: ${{ always() && fromJSON(steps.results.outputs.summary).flaky_tests > 0 && github.event_name == 'pull_request' }}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, always() will also trigger if the workflow has been cancelled, which we would rather not do here. It's not a huge difference, but could stop some confusion.

uses: actions/github-script@v7
continue-on-error: true
with:
Expand All @@ -463,7 +481,7 @@ jobs:
});

- name: Send Slack message for failed tests
if: ${{ inputs.slackNotificationAfterTestsChannelId != '' && fromJSON(steps.results.outputs.summary).flaky_tests > 0 }}
if: ${{ (success() || failure()) && inputs.slackNotificationAfterTestsChannelId != '' && fromJSON(steps.results.outputs.summary).flaky_tests > 0 }}
uses: slackapi/slack-github-action@6c661ce58804a1a20f6dc5fbee7f0381b469e001 # v1.25.0
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
Expand Down Expand Up @@ -516,7 +534,7 @@ jobs:

- name: Send general Slack message
uses: slackapi/slack-github-action@6c661ce58804a1a20f6dc5fbee7f0381b469e001 # v1.25.0
if: ${{ inputs.slackNotificationAfterTestsChannelId != '' && fromJSON(steps.results.outputs.summary).flaky_tests == 0 && fromJSON(steps.results.outputs.summary).total_tests > 0 }}
if: ${{ (success() || failure()) && inputs.slackNotificationAfterTestsChannelId != '' && fromJSON(steps.results.outputs.summary).flaky_tests == 0 && fromJSON(steps.results.outputs.summary).total_tests > 0 }}
id: slack
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
Expand Down
Loading