Error and panic in kubeapps-internal-apis deployment

[sde-melo]

Describe the bug
During the deployment of kubeapps on a Kubernetes cluster using a Helm chart with default values, an error and panic occur, resulting in a segmentation violation. The deployment process fails, and the application does not start as expected.

To Reproduce
Steps to reproduce the behavior:

1. Install the kubeapps-apis chart on a Kubernetes cluster using the default Helm values.
2. Observe the logs during the deployment process.

I0719 16:03:45.944454       1 root.go:36] "kubeapps-apis has been configured with serverOptions" serverOptions={"Port":50051,"PluginDirs":["/plugins/helm-packages","/plugins/resources"],"ClustersConfigPath":"/config/clusters.conf","PluginConfigPath":"/config/kubeapps-apis/plugins.conf","PinnipedProxyURL":"http://kubeapps-internal-pinniped-proxy.kubeapps:3333","PinnipedProxyCACert":"","GlobalHelmReposNamespace":"default","UnsafeLocalDevKubeconfig":false,"QPS":50,"Burst":100}
I0719 16:03:46.348623       1 server.go:103] +helm NewServer(globalPackagingCluster: [default], globalPackagingNamespace: [default], pluginConfigPath: [/config/kubeapps-apis/plugins.conf]
I0719 16:03:46.348969       1 server.go:115] +helm using custom config: [{{3 3 3} 300 }]
I0719 16:03:46.349015       1 server.go:127] +helm NewServer effective globalPackagingNamespace: [default]
I0719 16:03:46.350190       1 plugins.go:155] "Successfully registered plugin" pluginPath="/plugins/helm-packages/helm-packages-v1alpha1-plugin.so"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1e5d210]
goroutine 1 [running]:
...

Expected behavior
The kubeapps-apis should deploy successfully without any errors or panics. The application should start and be ready to serve requests.

Desktop (please complete the following information):

• Version: 2.8.0
• Kubernetes version: v1.27.3-eks-a5565ad
• Package version: 12.4.7

[absoludity]

Hi @sde-melo . Can you please include the rest of the lines after the panic? It will help us see exactly where the code is panicking (since we're not seeing this in our own environments, it will help us know how we can reproducue it). Thanks!

[sde-melo]

Hi @absoludity
Of course, here they are:

I0719 16:03:45.944454       1 root.go:36] "kubeapps-apis has been configured with serverOptions" serverOptions={"Port":50051,"PluginDirs":["/plugins/helm-packages","/plugins/resources"],"ClustersConfigPath":"/config/clusters.conf","PluginConfigPath":"/config/kubeapps-apis/plugins.conf","PinnipedProxyURL":"http://kubeapps-internal-pinniped-proxy.kubeapps:3333/","PinnipedProxyCACert":"","GlobalHelmReposNamespace":"default","UnsafeLocalDevKubeconfig":false,"QPS":50,"Burst":100}
I0719 16:03:46.348623       1 server.go:103] +helm NewServer(globalPackagingCluster: [default], globalPackagingNamespace: [default], pluginConfigPath: [/config/kubeapps-apis/plugins.conf]
I0719 16:03:46.348969       1 server.go:115] +helm using custom config: [{{3 3 3} 300 }]
I0719 16:03:46.349015       1 server.go:127] +helm NewServer effective globalPackagingNamespace: [default]
I0719 16:03:46.350190       1 plugins.go:155] "Successfully registered plugin" pluginPath="/plugins/helm-packages/helm-packages-v1alpha1-plugin.so"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1e5d210]
goroutine 1 [running]:
k8s.io/client-go/discovery.convertAPIResource(...)
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/pkg/mod/k8s.io/client-go@v0.26.1/discovery/aggregated_discovery.go:88
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc00038c5e8, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/pkg/mod/k8s.io/client-go@v0.26.1/discovery/aggregated_discovery.go:69 +0x5f0
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc0007b7770, 0x15}, {0xc00088e2e0, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/pkg/mod/k8s.io/client-go@v0.26.1/discovery/aggregated_discovery.go:35 +0x2f8
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0x7f9024f6d9d0?)
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:310 +0x47c
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0x3?)
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:198 +0x5c
k8s.io/client-go/discovery.ServerGroupsAndResources({0x28ce410, 0xc0004b2d80})
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:392 +0x59
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1()
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:356 +0x25
k8s.io/client-go/discovery.withRetries(0x2, 0xc0006531a8)
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:621 +0x72
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0xc0006a5368?)
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:355 +0x3a
k8s.io/client-go/restmapper.GetAPIGroupResources({0x28ce410?, 0xc0004b2d80?})
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/pkg/mod/k8s.io/client-go@v0.26.1/restmapper/discovery.go:148 +0x42
plugin/unnamed-dcfd69cbbce91170f13515084dbf2737ec5d9323.createRESTMapper(0x42480000, 0x64)
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/src/github.com/vmware-tanzu/kubeapps/cmd/kubeapps-apis/plugins/resources/v1alpha1/server.go:96 +0x1be
plugin/unnamed-dcfd69cbbce91170f13515084dbf2737ec5d9323.NewServer(0x0?, 0x42480000, 0x2?, {0x7ffe0beed651, 0x22}, {{0xc000059690, 0x7}, {0x7ffe0beed68d, 0x7}, {0x2652a17, ...}, ...}, ...)
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/src/github.com/vmware-tanzu/kubeapps/cmd/kubeapps-apis/plugins/resources/v1alpha1/server.go:104 +0x5c
plugin/unnamed-dcfd69cbbce91170f13515084dbf2737ec5d9323.RegisterWithGRPCServer({0xc00013c8c0, {{0xc000059690, 0x7}, {0x7ffe0beed68d, 0x7}, {0x2652a17, 0x35}, {0x0, 0x0}, 0xc0000c7f20}, ...})
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/src/github.com/vmware-tanzu/kubeapps/cmd/kubeapps-apis/plugins/resources/v1alpha1/main.go:34 +0x9c
github.com/vmware-tanzu/kubeapps/cmd/kubeapps-apis/core/plugins/v1alpha1.(*PluginsServer).registerGRPC(0xc00018c3c0, 0xc0004b20c0, 0x7f90278b7300, 0xc00013c8c0, {0xc383, {0xc0000b4a80, 0x2, 0x2}, {0x7ffe0beed626, 0x15}, ...}, ...)
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/src/github.com/vmware-tanzu/kubeapps/cmd/kubeapps-apis/core/plugins/v1alpha1/plugins.go:181 +0x2fa
github.com/vmware-tanzu/kubeapps/cmd/kubeapps-apis/core/plugins/v1alpha1.(*PluginsServer).registerPlugins(0xc00018c3c0, {0xc000153020, 0x2, 0x203000?}, {{0x28c54b8, 0xc0003aa2c0}, 0xc0000bc980, {0xc000210cf8, 0x6}, {0xc000063290, ...}}, ...)
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/src/github.com/vmware-tanzu/kubeapps/cmd/kubeapps-apis/core/plugins/v1alpha1/plugins.go:142 +0x2fd
github.com/vmware-tanzu/kubeapps/cmd/kubeapps-apis/core/plugins/v1alpha1.NewPluginsServer({0xc383, {0xc0000b4a80, 0x2, 0x2}, {0x7ffe0beed626, 0x15}, {0x7ffe0beed651, 0x22}, {0x2652a17, 0x35}, ...}, ...)
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/src/github.com/vmware-tanzu/kubeapps/cmd/kubeapps-apis/core/plugins/v1alpha1/plugins.go:91 +0x378
github.com/vmware-tanzu/kubeapps/cmd/kubeapps-apis/server.Serve({0xc383, {0xc0000b4a80, 0x2, 0x2}, {0x7ffe0beed626, 0x15}, {0x7ffe0beed651, 0x22}, {0x2652a17, 0x35}, ...})
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/src/github.com/vmware-tanzu/kubeapps/cmd/kubeapps-apis/server/server.go:97 +0x345
github.com/vmware-tanzu/kubeapps/cmd/kubeapps-apis/cmd.newRootCmd.func2(0xc000120300?, {0x26029d4?, 0x9?, 0x9?})
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/src/github.com/vmware-tanzu/kubeapps/cmd/kubeapps-apis/cmd/root.go:39 +0x45
github.com/spf13/cobra.(*Command).execute(0xc000120300, {0xc00012e010, 0x9, 0x9})
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc000120300)
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992
github.com/vmware-tanzu/kubeapps/cmd/kubeapps-apis/cmd.Execute()
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/src/github.com/vmware-tanzu/kubeapps/cmd/kubeapps-apis/cmd/root.go:48 +0x25
main.main()
 /bitnami/blacksmith-sandox/kubeapps-apis-2.8.0/src/github.com/vmware-tanzu/kubeapps/cmd/kubeapps-apis/main.go:12 +0x17

[sde-melo]

For now I am stuck in 2.6.2. I have tried all upper versions and I encouter the same crash. However, if I force the 2.6.2-debian-11-r0 tag for the kubeapps-internal-kubeappsapis image in the latest helm chart, I get rid of the error.

[absoludity]

Thanks @sde-melo . So it is an error when Kubeapps tries to initialise a standard piece for interacting with the Kubernetes infrastructure (a REST mapper, which maps what APIs are available in the k8s api server). Kubeapps handles any error as it should there, but the 3rd party code itself isn't.

Digging down, I see lots of OSS projects had this issue in the past few months (for e.g. look at this report and the links to it) which point back to an error in the Kubernetes client code itself at: kubernetes/kubernetes#116603 . So the reason that you're seeing this issue while our CI and environments don't, is because of an extra aggregated API you have running on your k8s API server which is returning nil GroupVersionKind's (possibly the metrics server?). Anyway, they have already updated and released the fix, in k8s.io/client-go 1.26.4, which we already updated to, but not before the 2.8.0 release.

So you should be able to verify that your problem is also fixed with the latest kubeapps-apis development image (though don't run your prod systems with that since we've not QA'd it through our release process - just wait for the 2.8.1 kubeapps release or similar). Thanks for the extra information!

EDIT: or by removing/upgrading the aggregate APIs that are returning nil GroupVersionKind's, but I'm guessing that's less practical :)

[sde-melo]

Thanks for the detailed explanations @absoludity! I'll wait for the 2.8.1 release then.