Make the containers new-e2e tests send datadog events to help debugging #20508
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
L3n41c
added
team/containers
dev/testing
changelog/no-changelog
[deprecated] qa/skip-qa - use other qa/ labels
L3n41c
force-pushed
the
lenaic/CONTINT-52_CONTINT-258_CONTINT-3310_CONTINT-3314_new-e2e_send_events
branch
from
da3a6be
to
7180b8e
Compare
CONTINT-52, CONTINT-258, CONTINT-3310, CONTINT-3314
L3n41c
force-pushed
the
lenaic/CONTINT-52_CONTINT-258_CONTINT-3310_CONTINT-3314_new-e2e_send_events
branch
from
7180b8e
to
1fb9d12
Compare
pducolin
approved these changes
Oct 31, 2023
There was a problem hiding this comment.
This is amazing. Helps a lot with visibility of tests and debugging. Can we link it easily to CI Visibility ? I can investigate which tags the event need to share with the ci visibility payload to get the link under a test execution.
Currently test stack is cut which makes it hard to investigate issues from CI Visibility
| suite.EventuallyWithTf(func(collect *assert.CollectT) { | ||
| myCollect := &myCollectT{ |
There was a problem hiding this comment.
💭 thought
Could possibly be skipped after stretchr/testify#1481 is merged
There was a problem hiding this comment.
Maybe not, I see we would not have access to collect errors.
L3n41c
deleted the
lenaic/CONTINT-52_CONTINT-258_CONTINT-3310_CONTINT-3314_new-e2e_send_events
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Emit a datadog event each time a metric is asserted in the new-e2e test framework.
Motivation
Knowing that the agent is broken because an e2e test is failing is nice.
But knowing what it broken and why because e2e test failures are giving a maximum of information regarding the failure is better.
In that context, the
containerse2e tests are already providing, at the end of their execution, a link to a dashboard containing all the metrics that are produced and asserted by the tests.This PR adds an event overlay on top of those graph so that we can visually see when the assertion ran.
It is useful to visualize when a metric eventually show up after the retry loop ended, or when we do too many useless retries, etc.
Additional Notes
Here is what the dashboard looks like:
The events contain information about what was looked for and what was missing:
Possible Drawbacks / Trade-offs
Describe how to test/QA your changes
Reviewer's Checklist
Triagemilestone is set.major_changelabel if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.changelog/no-changeloglabel has been applied.qa/skip-qalabel is not applied.team/..label has been applied, indicating the team(s) that should QA this change.need-change/operatorandneed-change/helmlabels have been applied.k8s/<min-version>label, indicating the lowest Kubernetes version compatible with this feature.