[kube-prometheus-stack] Unable to install chart from OCI via Flux

[thezanke]

Describe the bug

Applying the manifest found at https://github.com/fluxcd/flux2/manifests/monitoring/kube-prometheus-stack, which uses the OCI registry for the "41.x" branch for KPS, my HelmChart resource for kube-prometheus-stack is unhealthy and gives me the following reason:

chart verification error: failed to verify oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack:41.7.3: no matching signatures:                                
  error getting target ctfe.pub by usage: error verifying local metadata; local cache may be corrupt: tuf: failed to decode snapshot.json: expired at 2023-01-22 00:08:42 +0000 UTC

As far as I can tell this is the only chart having any issues.

What's your helm version?

version.BuildInfo{Version:"v3.9.1", GitCommit:"a7c043acb5ff905c261cfdc923a35776ba5e66e4", GitTreeState:"clean", GoVersion:"go1.18.4"}

What's your kubectl version?

Client Version: v1.24.3
Kustomize Version: v4.5.4
Server Version: v1.22.15-eks-fb459a0

Which chart?

kube-prometheus-stack

What's the chart version?

41.x

What happened?

Received the error:

chart verification error: failed to verify oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack:41.7.3: no matching signatures:                                
  error getting target ctfe.pub by usage: error verifying local metadata; local cache may be corrupt: tuf: failed to decode snapshot.json: expired at 2023-01-22 00:08:42 +0000 UTC

What you expected to happen?

Synchronize HelmChart successfully

How to reproduce it?

Add resources to flux-powered cluster:

apiVersion: v1
kind: List
items:
  - apiVersion: source.toolkit.fluxcd.io/v1beta2
    kind: GitRepository
    metadata:
      name: flux-monitoring
      namespace: flux-system
    spec:
      interval: 30m0s
      ref:
        branch: main
      url: https://github.com/fluxcd/flux2
  - apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
    kind: Kustomization
    metadata:
      name: kube-prometheus-stack
      namespace: flux-system
    spec:
      interval: 1h0m0s
      path: ./manifests/monitoring/kube-prometheus-stack
      prune: true
      sourceRef:
        kind: GitRepository
        name: flux-monitoring

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

flux reconcile kustomization kube-prometheus-stack -n flux-system

Anything else we need to know?

No response

[migueleliasweb]

I don't know much about Flux but I am using ArgoCD to install the kube-prometheus-stack.

Make sure you're using the server-side apply options for this as the manifests are too big and will cause issues with the last-applied-configuration fields.

Also, I've had reconcile loop problems due to some configs creating never ending diffs in the API Server. Have a look if the manifests keep changing in your cluster. That might be due to a reconcile loop.

[thezanke]

@migueleliasweb Are you using the OCI manifest repo (oci://ghcr.io/prometheus-community/charts) or are you using Git (https://prometheus-community.github.io/helm-charts
)? My problem is with the former; it appears the signature being validated by TUF expired 2023-01-22 00:08:42 +0000 UTC. Though I'm not really sure how all of that works, and google has very little to say about any of this. The only link I find when googling is to this previous issue which turned out to be a CI problem in this repo.

[monotek]

oci uploads are deactivated at the moment. see: #2841

[thezanke]

oci uploads are deactivated at the moment

Aha! Thank you! I swear I tried to search first!