[bug] Omni overrides secret objects when included in extraManifests

[tman5]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When using the extraManifests config object on the omni cluster, to apply secret objects to the cluster, everytime we bootstrap the cluster, the secrets change. Here is an example manifest:

apiVersion: v1
kind: Secret
metadata:
  name: secret
  namespace: "kube-system"
  annotations:
    kubernetes.io/service-account.name: secret
type: kubernetes.io/service-account-token

I've also seen this for certain CNI manifests

Expected Behavior

No changes to the secret tokens. What is happening is that omni sees the secret objects have changes by the cluster, but it wants to override when the cluster changes the manifest

Steps To Reproduce

1. Create an extraManifest configuration in the cluster config that includes a secret
2. Apply the cluster config
3. Notice that Omni will apply it, but then will always mark it as a change in the bootstrap manifest section because the kubernetes cluster has attached a token to that secret and Omni wants to override it

What browsers are you seeing the problem on?

Firefox, Chrome

Anything else?

No response

[Unix4ever]

I don't think it has anything to do with Omni to be honest.
I think that's Kubernetes internals doing that.
@smira should know that better.

But actually it feels like you're trying to do something weird. What's your usecase for this scenario?

[tman5]

This extraManifest that it is applying is 2 objects for service accounts. The issue is the Secret object since when the bootstrap runs again after applying it the first time it will change the secret data

---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: kube-system
  name: testadmin
---
apiVersion: v1
kind: Secret
metadata:
  name: test-admin-token
  namespace: "kube-system"
  annotations:
    kubernetes.io/service-account.name: test
type: kubernetes.io/service-account-token

[smira]

I don't know, the only thing we can do is to skip/filter out such updates.

Bootstrap manifests have limited support for rersources to be created, you can also remove that Secret from the config patch after the bootstrap as a workaround.