Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add goroutine stack dump in support bundle #5538

Merged
merged 1 commit into from
Oct 11, 2023
Merged

Add goroutine stack dump in support bundle #5538

merged 1 commit into from
Oct 11, 2023

Conversation

aniketraj1947
Copy link
Contributor

@aniketraj1947 aniketraj1947 commented Oct 1, 2023

Fixes #5243

This PR adds goroutine stack dump in support bundle for both agent and controller. Sample shown below for controller.

antrea % kubectl get pods -l app=antrea -n kube-system
NAME                                 READY   STATUS    RESTARTS   AGE
antrea-agent-lvnl6                   2/2     Running   0          57m
antrea-controller-668f6cc866-b4gxt   1/1     Running   0          57m

antrea % kubectl exec -it antrea-controller-668f6cc866-b4gxt -n kube-system -c antrea-controller bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@minikube:/# antctl supportbundle
Created bundle under /tmp
Expire time: 2023-10-01 17:07:18 +0000 UTC
root@minikube:/# tar -xzvf /tmp/bundle_269813909.tar.gz 
addressgroups
appliedtogroups
controllerinfo
goroutinestacks
logs/controller/antrea-controller.minikube.root.log.INFO.20231001-150949.1
memprofile
networkpolicies

root@minikube:/# head goroutinestacks 
goroutine 22252 [running]:
runtime/pprof.writeGoroutineStacks({0x2ab1a00, 0xc0003ac000})
        /usr/local/go/src/runtime/pprof/pprof.go:703 +0x6a
runtime/pprof.writeGoroutine({0x2ab1a00?, 0xc0003ac000?}, 0x4f3ff6?)
        /usr/local/go/src/runtime/pprof/pprof.go:692 +0x25
runtime/pprof.(*Profile).WriteTo(0x269762a?, {0x2ab1a00?, 0xc0003ac000?}, 0x4f2cbb?)
        /usr/local/go/src/runtime/pprof/pprof.go:329 +0x146
antrea.io/antrea/pkg/support.DumpGoroutinePprof({0x2af0860, 0x3f871a0}, {0xc000d211c0?, 0x19?})
        /antrea/pkg/support/dump.go:106 +0x1d5

The original issue also talks about addition of /metrics API response to support bundle but I hope it'd be fine if it's done in a follow-up PR.

Signed-off-by: Aniket Raj <info.aniketraj@gmail.com>
Copy link
Contributor

@antoninbas antoninbas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM. I will wait for @tnqn to review since he opened the corresponding Github issue. In particular, Quan was originally asking for this to be optional (with an antctl command-line flag). But at the time, I don't think he was aware that we already collected the heap profile unconditionally. So let's see what he says.

@aniketraj1947
Copy link
Contributor Author

@tnqn Did you get a chance to look at this? LMK if a CLI flag needs to be added.

Copy link
Member

@tnqn tnqn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, the patch is really timely, even yesterday I was debugging a deadlock issue: #5565. With this enhancement, it could have been much easier to identify the root cause.

Now I don't think we need an extra flag since memprofile is already there, and even the other information included in the bundle could have something users don't want to be publicly visible, like address and log. We should perhaps always suggest users to share supportbundle in a private way.

@tnqn tnqn added the action/release-note Indicates a PR that should be included in release notes. label Oct 11, 2023
@tnqn tnqn added this to the Antrea v1.14 release milestone Oct 11, 2023
@tnqn
Copy link
Member

tnqn commented Oct 11, 2023

/test-e2e
/skip-networkpolicy
/skip-conformance

@tnqn tnqn merged commit 960e376 into antrea-io:main Oct 11, 2023
44 checks passed
antoninbas added a commit to antoninbas/antrea that referenced this pull request Feb 21, 2024
Fix the TestVMAgent/testExternalNodeSupportBundleCollection test.
The test case was not updated correctly after antrea-io#5538 was merged.

Signed-off-by: Antonin Bas <antonin.bas@broadcom.com>
antoninbas added a commit to antoninbas/antrea that referenced this pull request Feb 21, 2024
Fix the TestVMAgent/testExternalNodeSupportBundleCollection test.
The test case was not updated correctly after antrea-io#5538 was merged.

Signed-off-by: Antonin Bas <antonin.bas@broadcom.com>
antoninbas added a commit that referenced this pull request Feb 21, 2024
Fix the TestVMAgent/testExternalNodeSupportBundleCollection test.
The test case was not updated correctly after #5538 was merged.

Signed-off-by: Antonin Bas <antonin.bas@broadcom.com>
luolanzone pushed a commit to luolanzone/antrea that referenced this pull request Mar 21, 2024
Fix the TestVMAgent/testExternalNodeSupportBundleCollection test.
The test case was not updated correctly after antrea-io#5538 was merged.

Signed-off-by: Antonin Bas <antonin.bas@broadcom.com>
tnqn pushed a commit that referenced this pull request Mar 21, 2024
Fix the TestVMAgent/testExternalNodeSupportBundleCollection test.
The test case was not updated correctly after #5538 was merged.

Signed-off-by: Antonin Bas <antonin.bas@broadcom.com>
luolanzone pushed a commit to luolanzone/antrea that referenced this pull request Mar 22, 2024
Fix the TestVMAgent/testExternalNodeSupportBundleCollection test.
The test case was not updated correctly after antrea-io#5538 was merged.

Signed-off-by: Antonin Bas <antonin.bas@broadcom.com>
tnqn pushed a commit that referenced this pull request Mar 25, 2024
Fix the TestVMAgent/testExternalNodeSupportBundleCollection test.
The test case was not updated correctly after #5538 was merged.

Signed-off-by: Antonin Bas <antonin.bas@broadcom.com>
luolanzone pushed a commit to luolanzone/antrea that referenced this pull request Mar 26, 2024
Fix the TestVMAgent/testExternalNodeSupportBundleCollection test.
The test case was not updated correctly after antrea-io#5538 was merged.

Signed-off-by: Antonin Bas <antonin.bas@broadcom.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
action/release-note Indicates a PR that should be included in release notes.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add more information to support bundle
3 participants