Upgrade Karpenter to v0.27.5 #15144

anthonyhaussman · 2023-02-12T16:52:21Z

feat(karpenter): Upgrade to version 0.27.0

Upgrade Karpenter to the current last stable version 0.27.0. Templates have been updated to use the same templates as the Helm chart.

Some noticeable changes:

Rename karpenter-config-logging ConfigMap to config-logging
Some environment variables configurations are now in the karpenter-global-settings ConfigMap
Karpenter use now a common pod for the controller and webhook.

feat(karpenter): Use AWSNodeTemplate for launchTemplate

To set Launch Templates is deprecated into the provisioner, it is recommended to use the AWSNodeTemplate to set it.
Ref:

https://karpenter.sh/v0.27.0/concepts/node-templates/

feat(karpenter): Drop the karpenter feature flag

Drop the Karpenter feature flag. it's fairly well-supported now.

k8s-ci-robot · 2023-02-12T16:52:29Z

Hi @anthonyhaussman. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

olemarkus

With these massive changes, I think we need an upgrade e2e test with Karpenter. There are quite a few potential breakages here.

upup/models/cloudup/resources/addons/karpenter.sh/k8s-1.19.yaml.template

olemarkus · 2023-02-13T07:59:26Z

/ok-to-test

anthonyhaussman · 2023-02-14T09:41:03Z

/retest

anthonyhaussman · 2023-02-14T11:42:33Z

/retest

anthonyhaussman · 2023-02-14T13:56:13Z

@olemarkus I have fixed most of the templates et done some tests on a private environment.
Karpenter starts correctly and new nodes appear but do not join the cluster on my side (I think it's a different problem as I have another "classic" InstanceGroup that does not appear also).

On the e2e test, I see the Karpenter pod staying in Pending status and I'm not really sure why.
I don't have any experience with e2e unfortunately 😕

I would like to have your and/or the community's helps on this 🙏

anthonyhaussman · 2023-02-14T17:13:54Z

I have locally backported modifications on kops-1.26.0-beta.2 to try a bootstrap.
Karpenter works like a charm 👌

olemarkus · 2023-02-14T18:02:44Z

You can see a list of all the pod resources here: https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kops/15144/pull-kops-e2e-aws-karpenter/1625454977755385856/artifacts/cluster-info/kube-system/pods.yaml

Includes their status and reason.

upup/models/cloudup/resources/addons/karpenter.sh/k8s-1.19.yaml.template

hakman · 2023-06-08T19:55:13Z

/test pull-kops-e2e-aws-karpenter

anthonyhaussman · 2023-06-09T07:12:51Z

Looks like --instance-manager flag has been rebased out of the PR again.

@olemarkus I had re-pushed all commits concerning extra flags after my past bad rebase into 294f343

I have rechecked my reflog and didn't find a commit about the --intance-manager flag. 😕
If I can have your help on this. 🙏

hakman · 2023-06-09T07:18:03Z

@anthonyhaussman Could you also take care of the indentation diffs inside the template? I think you are using a different indentation than before.

hakman · 2023-06-09T07:26:14Z

@anthonyhaussman Could you also take care of the indentation diffs inside the template? I think you are using a different indentation than before. It should reduce the diff a little.

ltellesfl · 2023-06-22T20:59:27Z

Any updates on this? Karpenter is already at version 0.28.1

olemarkus · 2023-06-24T06:20:20Z

I think it's time to get this one in. @anthonyhaussman can you squash/clean up your commits and update the version in the PR title? Then we're good to go.

hakman · 2023-06-24T07:13:18Z

@anthonyhaussman & @olemarkus Also #15144 (comment) please.

hakman · 2023-06-30T04:52:09Z

/retest

hakman · 2023-06-30T05:06:43Z

/test all

k8s-ci-robot · 2023-06-30T05:20:42Z

@anthonyhaussman: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kops-e2e-aws-upgrade-k123-ko125-to-k124-kolatest-karpenter	`d281cbd`	link	false	`/test pull-kops-e2e-aws-upgrade-k123-ko125-to-k124-kolatest-karpenter`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2023-06-30T05:35:34Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [hakman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* feat(karpenter): Upgrade to version 0.27.0 Upgrade Karpenter to current last stable version `0.27.0`. Template have been updated to use the same templates than the Helm chart. * feat(karpenter): Use AWSNodeTemplate for launchTemplate To set Launch Templates is deprecated into the provisioner, it is recommends using the `AWSNodeTemplate` to set it. Ref: - https://karpenter.sh/v0.27.0/concepts/node-templates/ * feat(karpenter): Enable pruning addon * Use extra flags in upgrade-ab scenario test * feat(karpenter): Drop `karpenter` feature flag * feat(karpenter): Add release note for `1.27` * feat(karpenter): Upgrade to version 0.27.3 * feat(karpenter): fix template * feat(karpenter): Upgrade to version 0.27.5 * Update Karpenter documentation with depending kops version * Delete KOPS_FEATURE_FLAGS from e2e test `run-test` * Run hack/update-expected.sh --------- Co-authored-by: Anthony Hausman <anthonyhausman@gmail.com>

anthonyhaussman · 2023-07-03T08:00:44Z

@olemarkus / @hakman Sorry for the late response. 🙏
I was on vacation and was not able to have a look at the PR during that time.

Really sorry about that. Thanks for the merge. 🙏

hakman · 2023-07-03T08:11:26Z

@anthonyhaussman No worries, just wanted to get it in 1.27. Great work!

BTW, if you get a chance, maybe you can take a look at #15585.

* feat(karpenter): Upgrade to version 0.27.0 Upgrade Karpenter to current last stable version `0.27.0`. Template have been updated to use the same templates than the Helm chart. * feat(karpenter): Use AWSNodeTemplate for launchTemplate To set Launch Templates is deprecated into the provisioner, it is recommends using the `AWSNodeTemplate` to set it. Ref: - https://karpenter.sh/v0.27.0/concepts/node-templates/ * feat(karpenter): Enable pruning addon * Use extra flags in upgrade-ab scenario test * feat(karpenter): Drop `karpenter` feature flag * feat(karpenter): Add release note for `1.27` * feat(karpenter): Upgrade to version 0.27.3 * feat(karpenter): fix template * feat(karpenter): Upgrade to version 0.27.5 * Update Karpenter documentation with depending kops version * Delete KOPS_FEATURE_FLAGS from e2e test `run-test` * Run hack/update-expected.sh

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 12, 2023

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Feb 12, 2023

k8s-ci-robot requested review from johngmyers and zetaab February 12, 2023 16:52

k8s-ci-robot added area/addons area/documentation labels Feb 12, 2023

anthonyhaussman marked this pull request as ready for review February 13, 2023 06:48

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 13, 2023

k8s-ci-robot requested review from hakman and olemarkus February 13, 2023 06:49

anthonyhaussman force-pushed the feat/kops/karpenter_upgrade_0.24.0 branch from 3fdb37c to 47c1d79 Compare February 13, 2023 07:07

olemarkus requested changes Feb 13, 2023

View reviewed changes

k8s-ci-robot assigned olemarkus Feb 13, 2023

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 13, 2023

hakman changed the title ~~feat(karpenter): Upgrade to version 0.24.0~~ Upgrade Karpenter to v0.24.0 Feb 13, 2023

anthonyhaussman requested review from olemarkus and removed request for johngmyers, hakman and zetaab February 14, 2023 13:38

olemarkus requested changes Feb 14, 2023

View reviewed changes

upup/models/cloudup/resources/addons/karpenter.sh/k8s-1.19.yaml.template Outdated Show resolved Hide resolved

upup/models/cloudup/resources/addons/karpenter.sh/k8s-1.19.yaml.template Show resolved Hide resolved

anthonyhaussman added 2 commits June 8, 2023 20:45

Update Karpenter documentation with depending kops version

3371d3c

Delete KOPS_FEATURE_FLAGS from e2e test run-test

5b95d30

anthonyhaussman force-pushed the feat/kops/karpenter_upgrade_0.24.0 branch from 10679cb to 5b95d30 Compare June 8, 2023 18:46

k8s-ci-robot added area/api area/documentation and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jun 8, 2023

Run hack/update-expected.sh

d281cbd

hakman changed the title ~~Upgrade Karpenter to v0.27.1~~ Upgrade Karpenter to v0.27.5 Jun 30, 2023

hakman added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Jun 30, 2023

hakman approved these changes Jun 30, 2023

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 30, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 30, 2023

k8s-ci-robot merged commit c2ed4b6 into kubernetes:master Jun 30, 2023

hakman mentioned this pull request Jul 1, 2023

Automated cherry pick of #15144: Upgrade Karpenter to v0.27.5 #15571

Merged

anthonyhaussman deleted the feat/kops/karpenter_upgrade_0.24.0 branch July 3, 2023 07:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade Karpenter to v0.27.5 #15144

Upgrade Karpenter to v0.27.5 #15144

anthonyhaussman commented Feb 12, 2023 •

edited

Loading

k8s-ci-robot commented Feb 12, 2023

olemarkus left a comment

olemarkus commented Feb 13, 2023

anthonyhaussman commented Feb 14, 2023

anthonyhaussman commented Feb 14, 2023

anthonyhaussman commented Feb 14, 2023

anthonyhaussman commented Feb 14, 2023

olemarkus commented Feb 14, 2023

hakman commented Jun 8, 2023

anthonyhaussman commented Jun 9, 2023

hakman commented Jun 9, 2023

hakman commented Jun 9, 2023 •

edited

Loading

ltellesfl commented Jun 22, 2023

olemarkus commented Jun 24, 2023

hakman commented Jun 24, 2023

hakman commented Jun 30, 2023

hakman commented Jun 30, 2023

k8s-ci-robot commented Jun 30, 2023

k8s-ci-robot commented Jun 30, 2023

anthonyhaussman commented Jul 3, 2023

hakman commented Jul 3, 2023

Upgrade Karpenter to v0.27.5 #15144

Upgrade Karpenter to v0.27.5 #15144

Conversation

anthonyhaussman commented Feb 12, 2023 • edited Loading

feat(karpenter): Upgrade to version 0.27.0

feat(karpenter): Use AWSNodeTemplate for launchTemplate

feat(karpenter): Drop the karpenter feature flag

k8s-ci-robot commented Feb 12, 2023

olemarkus left a comment

Choose a reason for hiding this comment

olemarkus commented Feb 13, 2023

anthonyhaussman commented Feb 14, 2023

anthonyhaussman commented Feb 14, 2023

anthonyhaussman commented Feb 14, 2023

anthonyhaussman commented Feb 14, 2023

olemarkus commented Feb 14, 2023

hakman commented Jun 8, 2023

anthonyhaussman commented Jun 9, 2023

hakman commented Jun 9, 2023

hakman commented Jun 9, 2023 • edited Loading

ltellesfl commented Jun 22, 2023

olemarkus commented Jun 24, 2023

hakman commented Jun 24, 2023

hakman commented Jun 30, 2023

hakman commented Jun 30, 2023

k8s-ci-robot commented Jun 30, 2023

k8s-ci-robot commented Jun 30, 2023

anthonyhaussman commented Jul 3, 2023

hakman commented Jul 3, 2023

anthonyhaussman commented Feb 12, 2023 •

edited

Loading

hakman commented Jun 9, 2023 •

edited

Loading