Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vpa-up.sh reliably fails to create secrets, but only the first time you run it. #7270

Open
rsalmond opened this issue Sep 9, 2024 · 3 comments
Open

vpa-up.sh reliably fails to create secrets, but only the first time you run it. #7270

rsalmond opened this issue Sep 9, 2024 · 3 comments
Assignees
@omerap12
Labels
area/vertical-pod-autoscaler kind/bug Categorizes issue or PR as related to a bug.

Comments

@rsalmond
Copy link

rsalmond commented Sep 9, 2024

Which component are you using?: VPA

What version of the component are you using?:

Component version: 1.2

What k8s version are you using (kubectl version)?:

kubectl version Output
$ kubectl version
Client Version: v1.29.5
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.6-gke.1326000

What environment is this in?:

GKE

What did you expect to happen?:

./hack/vpa-up.sh should execute gencerts.sh every time it is executed.

What happened instead?:

./hack/vpa-up.sh skips the execution of gencerts.sh the first time it is run. Subsequent runs work correctly.

How to reproduce it (as minimally and precisely as possible):

  1. start a new VM
  2. clone kubernetes/autoscaler
  3. run ./hack/vpa-up.sh

Anything else we need to know?:

Steps executed on a brand new machine:

Run 1

$ git clone https://github.com/kubernetes/autoscaler.git
Cloning into 'autoscaler'...
remote: Enumerating objects: 207306, done.
remote: Counting objects: 100% (2838/2838), done.
remote: Compressing objects: 100% (1875/1875), done.
remote: Total 207306 (delta 1581), reused 1499 (delta 918), pack-reused 204468 (from 1)
Receiving objects: 100% (207306/207306), 239.69 MiB | 20.82 MiB/s, done.
Resolving deltas: 100% (132221/132221), done.
Updating files: 100% (20815/20815), done.
lab[default] student@50:~
$ cd autoscaler/vertical-pod-autoscaler/
lab[default] student@50:~/autoscaler/vertical-pod-autoscaler
$ uptime
 13:44:17 up 11 min,  1 user,  load average: 0.37, 0.20, 0.14
lab[default] student@50:~/autoscaler/vertical-pod-autoscaler
$ ./hack/vpa-up.sh
HEAD is now at 6569b7734 Merge pull request #7178 from raywainman/vpa-release-1.2
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalercheckpoints.autoscaling.k8s.io created
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalers.autoscaling.k8s.io created
clusterrole.rbac.authorization.k8s.io/system:metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:vpa-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-status-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:evictioner created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-actor created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-actor created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-target-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-target-reader-binding created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-evictioner-binding created
serviceaccount/vpa-admission-controller created
serviceaccount/vpa-recommender created
serviceaccount/vpa-updater created
clusterrole.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrole.rbac.authorization.k8s.io/system:vpa-status-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-reader-binding created
role.rbac.authorization.k8s.io/system:leader-locking-vpa-updater created
rolebinding.rbac.authorization.k8s.io/system:leader-locking-vpa-updater created
role.rbac.authorization.k8s.io/system:leader-locking-vpa-recommender created
rolebinding.rbac.authorization.k8s.io/system:leader-locking-vpa-recommender created
deployment.apps/vpa-updater created
deployment.apps/vpa-recommender created
deployment.apps/vpa-admission-controller created
service/vpa-webhook created

Run 2

$ ./hack/vpa-up.sh
HEAD is now at 6569b7734 Merge pull request #7178 from raywainman/vpa-release-1.2
Error from server (AlreadyExists): error when creating "STDIN": customresourcedefinitions.apiextensions.k8s.io "verticalpodautoscalercheckpoints.autoscaling.k8s.io" already exists
Error from server (AlreadyExists): error when creating "STDIN": customresourcedefinitions.apiextensions.k8s.io "verticalpodautoscalers.autoscaling.k8s.io" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:metrics-reader" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:vpa-actor" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:vpa-status-actor" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:vpa-checkpoint-actor" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:evictioner" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterrolebindings.rbac.authorization.k8s.io "system:metrics-reader" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterrolebindings.rbac.authorization.k8s.io "system:vpa-actor" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterrolebindings.rbac.authorization.k8s.io "system:vpa-status-actor" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterrolebindings.rbac.authorization.k8s.io "system:vpa-checkpoint-actor" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:vpa-target-reader" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterrolebindings.rbac.authorization.k8s.io "system:vpa-target-reader-binding" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterrolebindings.rbac.authorization.k8s.io "system:vpa-evictioner-binding" already exists
Error from server (AlreadyExists): error when creating "STDIN": serviceaccounts "vpa-admission-controller" already exists
Error from server (AlreadyExists): error when creating "STDIN": serviceaccounts "vpa-recommender" already exists
Error from server (AlreadyExists): error when creating "STDIN": serviceaccounts "vpa-updater" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:vpa-admission-controller" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterrolebindings.rbac.authorization.k8s.io "system:vpa-admission-controller" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterroles.rbac.authorization.k8s.io "system:vpa-status-reader" already exists
Error from server (AlreadyExists): error when creating "STDIN": clusterrolebindings.rbac.authorization.k8s.io "system:vpa-status-reader-binding" already exists
Error from server (AlreadyExists): error when creating "STDIN": roles.rbac.authorization.k8s.io "system:leader-locking-vpa-updater" already exists
Error from server (AlreadyExists): error when creating "STDIN": rolebindings.rbac.authorization.k8s.io "system:leader-locking-vpa-updater" already exists
Error from server (AlreadyExists): error when creating "STDIN": roles.rbac.authorization.k8s.io "system:leader-locking-vpa-recommender" already exists
Error from server (AlreadyExists): error when creating "STDIN": rolebindings.rbac.authorization.k8s.io "system:leader-locking-vpa-recommender" already exists
Error from server (AlreadyExists): error when creating "STDIN": deployments.apps "vpa-updater" already exists
Error from server (AlreadyExists): error when creating "STDIN": deployments.apps "vpa-recommender" already exists
Generating certs for the VPA Admission Controller in /tmp/vpa-certs.
Certificate request self-signature ok
subject=CN=vpa-webhook.kube-system.svc
Uploading certs to the cluster.
secret/vpa-tls-certs created
Deleting /tmp/vpa-certs.
Error from server (AlreadyExists): error when creating "STDIN": deployments.apps "vpa-admission-controller" already exists
Error from server (AlreadyExists): error when creating "STDIN": services "vpa-webhook" already exists
@rsalmond rsalmond added the kind/bug Categorizes issue or PR as related to a bug. label Sep 9, 2024
@omerap12
Copy link

/assign

@voelzmo
Copy link
Contributor

voelzmo commented Sep 11, 2024

/area vertical-pod-autoscaler

@adrianmoisey
Copy link
Member

I figured out what's happening.
On the first run you're on the master branch.

autoscaler/vertical-pod-autoscaler/hack/vpa-up.sh

Line 29 in 59aba2b

$SCRIPT_ROOT/hack/vpa-process-yamls.sh apply $*

vpa-up.sh is passing apply to vpa-process-yamls.sh.

vpa-process-yamls.sh only generates a certificate when create is passed in, see:

autoscaler/vertical-pod-autoscaler/hack/vpa-process-yamls.sh

Lines 66 to 67 in 59aba2b

if [ ${ACTION} == create ] ; then
(bash ${SCRIPT_ROOT}/pkg/admission-controller/gencerts.sh || true)

However, part of vpa-up.sh is to change to the latest tag, which is vertical-pod-autoscaler-1.2.1.
In that tag, vpa-up.sh passes in apply to vpa-process-yamls, which then follows the path to create the certs.

autoscaler/vertical-pod-autoscaler/hack/vpa-up.sh

Line 29 in 6569b77

$SCRIPT_ROOT/hack/vpa-process-yamls.sh create $*

The change from create to apply happened in #7199

@voelzmo @raywainman thoughts? Should we revert that change or fix the logic in vpa-up.sh?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@omerap12 omerap12

Labels
area/vertical-pod-autoscaler kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@adrianmoisey @voelzmo @rsalmond @k8s-ci-robot @omerap12