Some Chart versions hidden from dropdown when deploying an application #3588

cristian-aldea · 2021-10-13T16:09:53Z

Description:

We’re running into a very strange bug and need help since it’s affecting customers of our infrastructure product.

We use Kubeapps for customers to deploy applications, and we use Chartmuseum to store charts for them to use. However, when we want to select a version of an application to deploy, some versions are hidden.

I’ve done some testing, and all the versions we offer are indeed available in chartmuseum, they show up when we GET on /api/charts/chart-name, and they can be pulled using “helm pull” to get the tar bundle. However it seems that Kubeapps is failing to show all these versions in the dropdown. For example, we start off by seeing the versions 1.19.4, 1.19.5, and 1.19.6:

but when a test chart like 1.19.7-CCT-XXXX (a test chart created for testing changes introduced by a ticket) is uploaded, some versions start disappearing.

However, I know for sure that Kubeapps knows about these hidden versions, since older applications deployed using these versions can still be accessed:

as opposed to showing us an error about the chart version not existing:

This leads me to believe that the dropdown is bugged in some way. We just started running into this issue, and it seems (not 100% sure on this) to start occurring when test charts (X.X.X-ABC-XXXX) are uploaded. Please tell us if you know anything about why this is happening. thank you for the help

Slack message: https://kubernetes.slack.com/archives/C9D3TSUG4/p1634139332089700

Our version history:
oct 6: we updated kubeapps chart to 7.5.4, app version 2.4.0
sept 9: we updated kubeapps chart to 7.4.0, app version 2..4.0
july 7, we updated kubeapps chart to 7.1.2, app version 2.3.2

Steps to reproduce the issue:

Deploy chartmuseum and kubeapps
configure chartmuseum as an app repository for kubeapps
upload charts to chartmuseum
refresh the app repository on kubeapps ui, or by running the sync cronjob

Describe the results you received:

Some chart versions don't show in the dropdown

Describe the results you expected:

All chart versions show up in the dropdown

Additional information you deem important (e.g. issue happens only occasionally):

We use particular version names when deploying test charts (chart which include changes related to a ticket). These charts have the format X.X.X-TICKET. This might be unexpected and might be partially causing the issue. but we've been doing this for years without running into this issue before.

Version of Helm, Kubeapps and Kubernetes:

Output of helm version:

version.BuildInfo{Version:"v3.6.3", GitCommit:"d506314abfb5d21419df8c7e7e68012379db2354", GitTreeState:"dirty", GoVersion:"go1.16.6"}

Output of helm list -n <kubeapps-namespace> <kubeapps-release-name>:

NAME    NAMESPACE       REVISION        UPDATED STATUS  CHART   APP VERSION
# nothing, we don't deploy kubeapps through helm install, but through argocd

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.8", GitCommit:"5575935422cc1cf5169dfc8847cb587aa47bac5a", GitTreeState:"clean", BuildDate:"2021-06-16T13:00:45Z", GoVersion:"go1.15.13", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.10-gke.301", GitCommit:"17ad7bd6afa01033d7bd3f02ce5de56f940a915d", GitTreeState:"clean", BuildDate:"2021-08-24T05:18:54Z", GoVersion:"go1.15.15b5", Compiler:"gc", Platform:"linux/amd64"}

The text was updated successfully, but these errors were encountered:

absoludity · 2021-10-14T00:25:48Z

Hey @cristian-aldea . Thanks for the report. Yes, as part of switching to the new API, we've switched to returning a summary of the available versions rather than the complete history (which was huge for some repos). This summary currently has the latest 3 patch versions of the latest 3 minor versions of the latest 3 major versions (See an explicit test of this). But this isn't the end goal (yet).

The main reason for this was to move away from returning all versions in the API response as for repos with years of history, it gets very large and is not normally what people want.

The ideal situation is a UX that presents a useful (customizable) summary by default, but still allows access to any version. This would require two further changes I think:

Allowing the number of major, minor and patch versions in the summary to be customizable in the chart. So for example, setting versionSummary: { major: 2, minor: 1, patch: 10 } would ensure that the UX receives (and displays) the latest 10 patch versions for the latest 2 major versions only, etc., perhaps with 0 meaning all so the previous behavior can also be obtained.
Updating the UX to also allow querying. So for example, entering 2.1 would then display all versions that start with 2.1, in reverse semver order (most recent first).

From what you've said, I think (1) would solve your immediate problem Cristian? (as you could restore the previous behavior or some variation that better suits your repositories usage). (1) is going to be something we can do quickly, perhaps for the next release, while (2) could take a bit longer to schedule and get done with our current resources.

Let me know what you think.

cristian-aldea · 2021-10-14T13:19:13Z

I see, thank you so much for the quick response explaining the issue. Yes (1) would allow us to quickly address the issue for now and (2) would be nice, since indeed some older versions aren't as relevant, but might still be useful to deploy for testing and such.

cristian-aldea · 2021-10-14T13:25:40Z

Just quickly, for my own information, when was this change in the api made?

cristian-aldea · 2021-10-14T13:32:49Z

Also, let us know when these features will be implemented so we can organize the upgrade on our side. Now that we know what the behavior actually is we'll be able to manage it for now.

absoludity · 2021-10-18T05:35:49Z

Just quickly, for my own information, when was this change in the api made?

We switched the create and update UX functions over to the new API with https://github.com/kubeapps/kubeapps/releases/tag/v2.4.1 so I assume it was that release (so 10 days ago).

Also, let us know when these features will be implemented so we can organize the upgrade on our side. Now that we know what the behavior actually is we'll be able to manage it for now.

Will do, we'll discuss it in our next planning meeting (one week from now) and update it then (perhaps doing the first part in the next iteration). Let us know if it's not tenable to manage the issue and we'll give it a higher priority.

cristian-aldea · 2021-10-28T14:27:01Z

Hello, please let me know through this issue when the Allowing the number of major, minor and patch versions in the summary to be customizable in the chart fix has been implemented so we know when to upgrade

absoludity · 2021-10-29T00:32:36Z

Will do @cristian-aldea . One of our new team members has started looking at this issue. Once you see the issue closed then it'll be in the following release.

absoludity · 2021-10-29T01:44:29Z

@satya-dillikar Some pre-implementation notes about passing config for plugins, based on our earlier calls. The main issue here is that we don't want to simply add an --versions-in-summary option or similar directly to the kubeapps-apis binary, given that this configuration option is specific to packaging plugins and in general, other future configuration options may be specific to one particular plugin and should not pollute the args to the binary when the related plugin may not even be used.

So we want to enable configuration to be passed to any configured plugin (or implementations of a core API) without the kubeapps-apis binary caring about the format itself (it'll be up to the plugin to validate the config).

One solution that we talked about in our call was to enable, in the values.yaml, a pluginConfig key that is populated with config per plugin, such as:

kubeappsapis:
  enabledPlugins:
    - helm
   pluginConfig:
    core:
      packages:
        v1alpha1:
          versionsInSummary:
            major: 6
    helm:
        packages:
          v1alpha1:
            someOtherConfig: some-value

This will be turned into json data within a config map and mounted as a volume for the kubeapps-apis deployment (the same way that we do for the clusters configuration in a few deployments, including the kubeapps-apis)

Originally (during our call) I had thought we should send only the relevant plugin config sections to each plugin, but reality is all plugins have disk access so it's enough for each plugin to just parse the bits they're interested in.

The kubeapps-apis binary itself will be updated to include an arg, --plugin-config-path and so can simply pass this path to each plugin when registering a plugin (ie. the signature will change to include the extra string, with registration functions for each plugin updated.

This will enable each plugin to read the (read-only) file and parse just the bits they are interested in. As an example, for the helm plugin it would do something like the following: https://play.golang.org/p/B92n0zD3yyc

This allows us to have config for each version of each plugin if necessary, without spilling over into the kubeapps-apis binary args . Let me know if you have any questions or objections Satya :)

absoludity · 2021-10-29T01:55:34Z

In terms of breaking that up into small PRs, one possibility is:

Update the signature of RegisterWithGRPCServer in plugins as well as where the registration happens, but pass an empty string ("") for now during registration.
Update the helm.packages.v1 plugin only, adding a parseConfig(configPath string), using this to configure the versionsInSummaryoption of the server (should be a field on the server struct, not a package level var) duringNewServer()`. 1 & 2 could be combined if small enough.
Update kubeapps-apis binary with the new flag (storing the path in serverOptions) and updating the chart to pass this via the volume mount as above.

cristian-aldea · 2021-11-04T01:02:22Z

Thank you so much for implementing a fix for our issue!

Looking forward to the new release :)

linhvuntu · 2023-02-15T16:45:51Z

Hi there I'm using kubeapps helm version 12.2.4/ App version 2.6.2 and the issue still happens.
I have updated the pluginConfig as below (increase all version display to 5), but the Package Versions dropdownlist still only show 3 major/minor/patch releases. Is there any param that I need to config?

pluginConfig:
core:
packages:
v1alpha1:
versionsInSummary:
major: 5
minor: 5
patch: 5

absoludity · 2023-02-16T09:20:11Z

Hi @linhvuntu . Just to double-check, the pluginConfig is an option of kubeappsapis, not a root option, as per the values.yaml:

kubeapps/chart/kubeapps/values.yaml

Lines 1494 to 1514 in a82349e

    
           ## @section kubeappsapis parameters 
        
           kubeappsapis: 
        
             ## @param kubeappsapis.enabledPlugins Manually override which plugins are enabled for the Kubeapps-APIs service 
        
             ## 
        
             ## NOTE: normally this should remain blank, with the top-level `packaging` 
        
             ## value automatically determining which plugins should be enabled. Only 
        
             ## set this value if you want to manually override the list of plugins 
        
             ## enabled for the service. 
        
             ## 
        
             enabledPlugins: [] 
        
             pluginConfig: 
        
               core: 
        
                 packages: 
        
                   v1alpha1: 
        
                     versionsInSummary: 
        
                       ## @param kubeappsapis.pluginConfig.core.packages.v1alpha1.versionsInSummary.major Number of major versions to display in the summary 
        
                       major: 3 
        
                       ## @param kubeappsapis.pluginConfig.core.packages.v1alpha1.versionsInSummary.minor Number of minor versions to display in the summary 
        
                       minor: 3 
        
                       ## @param kubeappsapis.pluginConfig.core.packages.v1alpha1.versionsInSummary.patch Number of patch versions to display in the summary 
        
                       patch: 3

It looks as though you're seeing the default (of 3) because your specified values are having no effect (which makes sense if the pluginConfig isn't specified as an option of kubeappsapis). Let us know if that's the problem or we need to dig further. Thanks.

linhvuntu · 2023-02-22T11:55:13Z

Hi @absoludity , how to specify the pluginConfig as an option of kubeappsapis ?

I have tried to set the enabledPlugins: [helm-packages, resources, core] but there is no directory /plugins/core in the pod. What should be the correct way then? Thanks

absoludity · 2023-02-22T22:34:16Z

Hi @linhvuntu . You shouldn't need to set enabledPlugins at all, just leave that one blank. To specify the pluginConfig, it's just a matter of setting that value when deploying the chart, so for example, your values file should contain:

kubeappsapis:
  pluginConfig:
    core:
      packages:
        v1alpha1:
          versionsInSummary:
            major: 5
            minor: 5
            patch: 5

From your previous example above, it looked like you thought pluginConfig was a root or global field in the values.yaml, as you'd not specified the kubeappsapis before it, which would mean that Helm just happily ignored your pluginConfig. Hope that's clearer.

linhvuntu · 2023-02-23T01:10:32Z

Hi @absoludity , ok now I get what it means by root option.. Actually I skipped the kubeappsapis in my example, the actual values that I set is exactly as you mentioned above. Because I just use default values.yaml and update the numbers.

kubeappsapis:
  enabledPlugins: []
  pluginConfig:
    core:
      packages:
        v1alpha1:
          versionsInSummary:
            major: 5
            minor: 5
            patch: 5
          timeoutSeconds: 300

And it's still not working. When checking the startup log, it show like this
+helm using custom config: [{{5 5 5} 300 false }]

Does it mean the custom config is "false" to setup?

absoludity · 2023-02-23T03:15:45Z

So that config looks fine then. If I run Kubeapps without that config and check the logs of the kubeapps-apis server pod, I see that the helm custom config is [{{3 3 3} 300 }] as expected:

I0223 03:10:34.820621       1 root.go:36] "kubeapps-apis has been configured with serverOptions" serverOptions={Port:50051 PluginDirs:[/plugins/helm-packages /plugins/resources] ClustersConfigPath:/config/clusters.conf PluginConfigPath:/config/kubeapps-apis/plugins.conf PinnipedProxyURL:http://kubeapps-internal-pinniped-proxy.kubeapps:3333 PinnipedProxyCACert: GlobalHelmReposNamespace:kubeapps UnsafeLocalDevKubeconfig:false QPS:50 Burst:100}
I0223 03:10:35.125202       1 server.go:101] +helm NewServer(globalPackagingCluster: [default], globalPackagingNamespace: [kubeapps], pluginConfigPath: [/config/kubeapps-apis/plugins.conf]
I0223 03:10:35.125331       1 server.go:112] +helm using custom config: [{{3 3 3} 300 }]
I0223 03:10:35.125346       1 server.go:124] +helm NewServer effective globalPackagingNamespace: [kubeapps]

If I update to use your above config, I see [{{5 5 5} 300 }] as expected:

I0223 03:12:44.517948       1 root.go:36] "kubeapps-apis has been configured with serverOptions" serverOptions={Port:50051 PluginDirs:[/plugins/helm-packages /plugins/resources] ClustersConfigPath:/config/clusters.conf PluginConfigPath:/config/kubeapps-apis/plugins.conf PinnipedProxyURL:http://kubeapps-internal-pinniped-proxy.kubeapps:3333 PinnipedProxyCACert: GlobalHelmReposNamespace:kubeapps UnsafeLocalDevKubeconfig:false QPS:50 Burst:100}
I0223 03:12:44.818952       1 server.go:101] +helm NewServer(globalPackagingCluster: [default], globalPackagingNamespace: [kubeapps], pluginConfigPath: [/config/kubeapps-apis/plugins.conf]
I0223 03:12:44.819086       1 server.go:112] +helm using custom config: [{{5 5 5} 300 }]
I0223 03:12:44.819101       1 server.go:124] +helm NewServer effective globalPackagingNamespace: [kubeapps]
I0223 03:12:44.819407       1 plugins.go:152] "Successfully registered plugin" pluginPath="/plugins/helm-packages/helm-packages-v1alpha1-plugin.so"

Can you please check the logs of your kubeapps-apis server to see whether it's reading the config in and getting {5 5 5} as expected? That will help us to know where to look next.

linhvuntu · 2023-02-23T05:10:04Z

So I tried to create a brand new kubeapps with Helm Package version 12.2.4 / App Version 2.6.3, and with the config of {10 10 10} and the logs are as below

I0223 04:40:53.660198       1 root.go:36] "kubeapps-apis has been configured with serverOptions" serverOptions={Port:50051 PluginDirs:[/plugins/helm-packages /plugins/resources] ClustersConfigPath:/config/clusters.conf PluginConfigPath:/config/kubeapps-apis/plugi │ns.conf PinnipedProxyURL:http://kubeapps-internal-pinniped-proxy.kubeapps:3333 PinnipedProxyCACert: GlobalHelmReposNamespace:kubeapps-v3 UnsafeLocalDevKubeconfig:false QPS:50 Burst:100}                                                                               │
I0223 04:40:53.941472       1 server.go:101] +helm NewServer(globalPackagingCluster: [default], globalPackagingNamespace: [kubeapps-v3], pluginConfigPath: [/config/kubeapps-apis/plugins.conf]                                                                         
I0223 04:40:53.941671       1 server.go:112] +helm using custom config: [{{10 10 10} 300 }]                                                                                                                                                                             
I0223 04:40:54.029400       1 server.go:124] +helm NewServer effective globalPackagingNamespace: [kubeapps-v3]                                                                                                                                                          
I0223 04:40:54.030557       1 plugins.go:152] "Successfully registered plugin" pluginPath="/plugins/helm-packages/helm-packages-v1alpha1-plugin.so"                                                                                                                     │
I0223 04:40:54.332092       1 server.go:119] +resources using custom config: [{{ }}]

So it seems updated successfully, but when checking the Package Version dropdown list, it still display in with the setting of {3 3 3}

The Helm package I'm checking is the public ingress-nginx chart from Kubernetes here: https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx

When I check the this chart versions from terminal, it shows full list of versions as below

helm search repo ingress-nginx/ingress-nginx -l

NAME                       	CHART VERSION	APP VERSION	DESCRIPTION
ingress-nginx/ingress-nginx	4.5.2        	1.6.4      	Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx	4.5.0        	1.6.3      	Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx	4.4.2        	1.5.1      	Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx	4.4.1         	1.5.2      	Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx	4.4.0        	1.5.1      	Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx	4.3.0        	1.4.0      	Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx	4.2.5        	1.3.1      	Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx	4.2.4        	1.3.1      	Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx	4.2.3        	1.3.0      	Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx	4.2.2        	1.3.0      	Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx	4.2.1         	1.3.0      	Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx	4.2.0        	1.3.0      	Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx	4.1.4         	1.2.1      	Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx	4.1.3         	1.2.1      	Ingress controller for Kubernetes using NGINX a...

... and more ..

But from the Package Version dropdown list in the Kubeapps Dashboards, it show only

4.5.2 
4.5.0
4.4.2
4.4.1
4.4.0
4.3.0
3.41.0
3.40.0
3.39.0
.. and more ..

For some specific reason, we need to use the Chart version 4.2.5 but it is not displayed.

Do I need to config anything else at the kubeappsController or elsewhere? I have cleared cache, use another laptop to check but it's still the same

absoludity · 2023-02-23T05:56:25Z

Thanks for the detail, I'm turning that into a separate bug report to investigate the regression - see #6020. Let's discuss it there.

absoludity added component/apis-server Issue related to kubeapps api-server priority/high kind/feature An issue that reports a feature (approved) to be implemented labels Oct 18, 2021

antgamdia added this to the Community requests milestone Oct 19, 2021

satya-dillikar mentioned this issue Oct 28, 2021

parameterize VersionsInSummary #3683

Closed

ppbaena assigned satya-dillikar Oct 28, 2021

satya-dillikar mentioned this issue Oct 29, 2021

parameterize VersionsInSummary - complete fix #3703

Merged

absoludity closed this as completed in #3703 Nov 4, 2021

antgamdia mentioned this issue Nov 9, 2021

Add implementation of GetAvailablePackageVersions for kapp-controller. #3734

Merged

absoludity mentioned this issue Feb 23, 2023

[regression] Cannot control which chart versions are displayed #6020

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Some Chart versions hidden from dropdown when deploying an application #3588

Some Chart versions hidden from dropdown when deploying an application #3588

cristian-aldea commented Oct 13, 2021 •

edited

Loading

absoludity commented Oct 14, 2021

cristian-aldea commented Oct 14, 2021 •

edited

Loading

cristian-aldea commented Oct 14, 2021

cristian-aldea commented Oct 14, 2021

absoludity commented Oct 18, 2021

cristian-aldea commented Oct 28, 2021

absoludity commented Oct 29, 2021

absoludity commented Oct 29, 2021

absoludity commented Oct 29, 2021

cristian-aldea commented Nov 4, 2021

linhvuntu commented Feb 15, 2023

absoludity commented Feb 16, 2023

linhvuntu commented Feb 22, 2023 •

edited

Loading

absoludity commented Feb 22, 2023

linhvuntu commented Feb 23, 2023 •

edited

Loading

absoludity commented Feb 23, 2023

linhvuntu commented Feb 23, 2023

absoludity commented Feb 23, 2023

Some Chart versions hidden from dropdown when deploying an application #3588

Some Chart versions hidden from dropdown when deploying an application #3588

Comments

cristian-aldea commented Oct 13, 2021 • edited Loading

Description:

Steps to reproduce the issue:

Describe the results you received:

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Version of Helm, Kubeapps and Kubernetes:

absoludity commented Oct 14, 2021

cristian-aldea commented Oct 14, 2021 • edited Loading

cristian-aldea commented Oct 14, 2021

cristian-aldea commented Oct 14, 2021

absoludity commented Oct 18, 2021

cristian-aldea commented Oct 28, 2021

absoludity commented Oct 29, 2021

absoludity commented Oct 29, 2021

absoludity commented Oct 29, 2021

cristian-aldea commented Nov 4, 2021

linhvuntu commented Feb 15, 2023

absoludity commented Feb 16, 2023

linhvuntu commented Feb 22, 2023 • edited Loading

absoludity commented Feb 22, 2023

linhvuntu commented Feb 23, 2023 • edited Loading

absoludity commented Feb 23, 2023

linhvuntu commented Feb 23, 2023

absoludity commented Feb 23, 2023

cristian-aldea commented Oct 13, 2021 •

edited

Loading

cristian-aldea commented Oct 14, 2021 •

edited

Loading

linhvuntu commented Feb 22, 2023 •

edited

Loading

linhvuntu commented Feb 23, 2023 •

edited

Loading