Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Kubernetes v1.29 clusters #382

Merged
merged 1 commit into from
Mar 5, 2024
Merged

Conversation

thunderboltsid
Copy link
Contributor

@thunderboltsid thunderboltsid commented Feb 22, 2024

This required two changes:

How has this been tested?
Create a cluster with Kubernetes v1.29.2

$ make test-cluster-create
clusterctl version
clusterctl version: &version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"", GitTreeState:"", BuildDate:"", GoVersion:"go1.21.4", Compiler:"gc", Platform:"darwin/amd64"}
Using configuration File="/Users/sid.shukla/.cluster-api/clusterctl.yaml"

New clusterctl version available: v1.6.0 -> v1.6.2
sigs.k8s.io/cluster-api
clusterctl config repositories | grep nutanix
Using configuration File="/Users/sid.shukla/.cluster-api/clusterctl.yaml"
nutanix        InfrastructureProvider   file:///Users/sid.shukla/.cluster-api/overrides/infrastructure-nutanix/v1.4.99/             infrastructure-components.yaml
Using configuration File="/Users/sid.shukla/.cluster-api/clusterctl.yaml"

New clusterctl version available: v1.6.0 -> v1.6.2
sigs.k8s.io/cluster-api
clusterctl generate cluster mycluster-without-topology -i nutanix:v1.4.99 --list-variables -v 10
Using configuration File="/Users/sid.shukla/.cluster-api/clusterctl.yaml"
Fetching File="cluster-template.yaml" Provider="nutanix" Type="InfrastructureProvider" Version="v1.4.99"
Required Variables:
  - CONTROL_PLANE_ENDPOINT_IP
  - NUTANIX_ENDPOINT
  - NUTANIX_MACHINE_TEMPLATE_IMAGE_NAME
  - NUTANIX_PASSWORD
  - NUTANIX_PRISM_ELEMENT_CLUSTER_NAME
  - NUTANIX_SSH_AUTHORIZED_KEY
  - NUTANIX_SUBNET_NAME
  - NUTANIX_USER

Optional Variables:
  - CCM_CUSTOM_LABEL                 (defaults to "false")
  - CCM_REPO                         (defaults to "ghcr.io/nutanix-cloud-native/cloud-provider-nutanix/controller")
  - CCM_TAG                          (defaults to "v0.3.2")
  - CLUSTER_NAME                     (defaults to mycluster-without-topology)
  - CONTROL_PLANE_ENDPOINT_PORT      (defaults to "6443")
  - CONTROL_PLANE_MACHINE_COUNT      (defaults to 1)
  - KUBERNETES_VERSION               (defaults to v1.29.2)
  - KUBEVIP_LB_ENABLE                (defaults to "false")
  - KUBEVIP_SVC_ELECTION             (defaults to "false")
  - KUBEVIP_SVC_ENABLE               (defaults to "false")
  - NAMESPACE                        (defaults to current Namespace in the KubeConfig file)
  - NUTANIX_ADDITIONAL_TRUST_BUNDLE  (defaults to "")
  - NUTANIX_INSECURE                 (defaults to "false")
  - NUTANIX_MACHINE_BOOT_TYPE        (defaults to "legacy")
  - NUTANIX_MACHINE_MEMORY_SIZE      (defaults to "4Gi")
  - NUTANIX_MACHINE_VCPU_PER_SOCKET  (defaults to "1")
  - NUTANIX_MACHINE_VCPU_SOCKET      (defaults to "2")
  - NUTANIX_PORT                     (defaults to "9440")
  - NUTANIX_SYSTEMDISK_SIZE          (defaults to "40Gi")
  - TLS_CIPHER_SUITES                (defaults to "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256")
  - WORKER_MACHINE_COUNT             (defaults to 0)

Using configuration File="/Users/sid.shukla/.cluster-api/clusterctl.yaml"

New clusterctl version available: v1.6.0 -> v1.6.2
sigs.k8s.io/cluster-api
clusterctl generate cluster mycluster-without-topology -i nutanix:v1.4.99 --target-namespace ns-topology  -v 10 > ./mycluster-without-topology.yaml
Using configuration File="/Users/sid.shukla/.cluster-api/clusterctl.yaml"
Fetching File="cluster-template.yaml" Provider="nutanix" Type="InfrastructureProvider" Version="v1.4.99"
Using configuration File="/Users/sid.shukla/.cluster-api/clusterctl.yaml"

New clusterctl version available: v1.6.0 -> v1.6.2
sigs.k8s.io/cluster-api
kubectl create ns ns-topology --dry-run=client -oyaml | kubectl apply --server-side -f -
namespace/ns-topology serverside-applied
kubectl apply --server-side -f ./mycluster-without-topology.yaml
configmap/mycluster-without-topology-pc-trusted-ca-bundle serverside-applied
configmap/nutanix-ccm serverside-applied
secret/mycluster-without-topology serverside-applied
secret/nutanix-ccm-secret serverside-applied
clusterresourceset.addons.cluster.x-k8s.io/nutanix-ccm-crs serverside-applied
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/mycluster-without-topology-kcfg-0 serverside-applied
cluster.cluster.x-k8s.io/mycluster-without-topology serverside-applied
machinedeployment.cluster.x-k8s.io/mycluster-without-topology-wmd serverside-applied
machinehealthcheck.cluster.x-k8s.io/mycluster-without-topology-mhc serverside-applied
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/mycluster-without-topology-kcp serverside-applied
nutanixcluster.infrastructure.cluster.x-k8s.io/mycluster-without-topology serverside-applied
nutanixmachinetemplate.infrastructure.cluster.x-k8s.io/mycluster-without-topology-mt-0 serverside-applied

and see the cluster come up

$ k -n ns-topology get machines
NAME                                         CLUSTER                      NODENAME                                     PROVIDERID                                       PHASE     AGE     VERSION
mycluster-without-topology-kcp-gkngc         mycluster-without-topology   mycluster-without-topology-kcp-gkngc         nutanix://c594b0f6-d000-43ab-a25d-b65fdfa1158f   Running   2m18s   v1.29.2
mycluster-without-topology-wmd-r9jlq-64726   mycluster-without-topology   mycluster-without-topology-wmd-r9jlq-64726   nutanix://063027f2-dfff-497e-876c-2512fa1ceb2f   Running   2m22s   v1.29.2
mycluster-without-topology-wmd-r9jlq-bxjqr   mycluster-without-topology   mycluster-without-topology-wmd-r9jlq-bxjqr   nutanix://4d1df71c-b118-433c-820e-8a57a34054a4   Running   2m22s   v1.29.2

@thunderboltsid
Copy link
Contributor Author

/hold

@deepakm-ntnx
Copy link
Contributor

could you please also add references on why these changes are required, may be a ref to opensource link on these suggestions

@thunderboltsid
Copy link
Contributor Author

/test e2e-capx-controller-upgrade

@thunderboltsid
Copy link
Contributor Author

/test e2e-k8s-upgrade

1 similar comment
@thunderboltsid
Copy link
Contributor Author

/test e2e-k8s-upgrade

@thunderboltsid
Copy link
Contributor Author

/test e2e-capx-conformance

3 similar comments
@thunderboltsid
Copy link
Contributor Author

/test e2e-capx-conformance

@thunderboltsid
Copy link
Contributor Author

/test e2e-capx-conformance

@thunderboltsid
Copy link
Contributor Author

/test e2e-capx-conformance

@thunderboltsid
Copy link
Contributor Author

Seems, the conformance jobs are consecutively failing on the CSI job.

@thunderboltsid
Copy link
Contributor Author

/test e2e-capx-conformance

@thunderboltsid
Copy link
Contributor Author

/test e2e-capx-controller-upgrade

@thunderboltsid
Copy link
Contributor Author

/test e2e-capx-conformance

4 similar comments
@thunderboltsid
Copy link
Contributor Author

/test e2e-capx-conformance

@thunderboltsid
Copy link
Contributor Author

/test e2e-capx-conformance

@thunderboltsid
Copy link
Contributor Author

/test e2e-capx-conformance

@thunderboltsid
Copy link
Contributor Author

/test e2e-capx-conformance

This required two changes:
- Remove hostAliases from kube-vip podspec
    This is addressed by adding entries directly to the /etc/hosts
- Do a super-admin.conf switcheroo for the kube-vip static pod
    Add pre and post kubeadm commands for handling kubernetes
    versions v1.29.0+. The prekubeadm command checks if kubeadm
    init has been run and if it is, it replaces the kubeconfig
    hostPath in kube-vip static pod from admin.conf to the
    super-admin.conf. The postkubeadm command checks if
    kubeadm init has been run and if it is, it changes the
    hostPath in kube-vip static pod from super-admin.conf
    back to admin.conf.
@thunderboltsid
Copy link
Contributor Author

/retest

@thunderboltsid
Copy link
Contributor Author

/test e2e-capx-conformance

3 similar comments
@thunderboltsid
Copy link
Contributor Author

/test e2e-capx-conformance

@thunderboltsid
Copy link
Contributor Author

/test e2e-capx-conformance

@thunderboltsid
Copy link
Contributor Author

/test e2e-capx-conformance

Copy link

codecov bot commented Feb 27, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 20.37%. Comparing base (974299f) to head (d5ff104).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #382   +/-   ##
=======================================
  Coverage   20.37%   20.37%           
=======================================
  Files          17       17           
  Lines        1212     1212           
=======================================
  Hits          247      247           
  Misses        965      965           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@deepakm-ntnx
Copy link
Contributor

/approve
/lgtm

@nutanix-cn-prow-bot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deepakm-ntnx, thunderboltsid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [deepakm-ntnx,thunderboltsid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

templates/base/kcp.yaml Show resolved Hide resolved
templates/cluster-template-clusterclass.yaml Show resolved Hide resolved
templates/cluster-template-clusterclass.yaml Show resolved Hide resolved
templates/base/kcp.yaml Show resolved Hide resolved
@thunderboltsid thunderboltsid merged commit aec5951 into main Mar 5, 2024
10 checks passed
@thunderboltsid thunderboltsid deleted the jira/krbn-8029 branch March 5, 2024 19:21
thunderboltsid added a commit that referenced this pull request Mar 13, 2024
This required two changes:
- Remove hostAliases from kube-vip podspec
    This is addressed by adding entries directly to the /etc/hosts
- Do a super-admin.conf switcheroo for the kube-vip static pod
    Add pre and post kubeadm commands for handling kubernetes
    versions v1.29.0+. The prekubeadm command checks if kubeadm
    init has been run and if it is, it replaces the kubeconfig
    hostPath in kube-vip static pod from admin.conf to the
    super-admin.conf. The postkubeadm command checks if
    kubeadm init has been run and if it is, it changes the
    hostPath in kube-vip static pod from super-admin.conf
    back to admin.conf.
thunderboltsid added a commit that referenced this pull request Mar 13, 2024
This required two changes:
- Remove hostAliases from kube-vip podspec
    This is addressed by adding entries directly to the /etc/hosts
- Do a super-admin.conf switcheroo for the kube-vip static pod
    Add pre and post kubeadm commands for handling kubernetes
    versions v1.29.0+. The prekubeadm command checks if kubeadm
    init has been run and if it is, it replaces the kubeconfig
    hostPath in kube-vip static pod from admin.conf to the
    super-admin.conf. The postkubeadm command checks if
    kubeadm init has been run and if it is, it changes the
    hostPath in kube-vip static pod from super-admin.conf
    back to admin.conf.
thunderboltsid added a commit that referenced this pull request Mar 13, 2024
This required two changes:
- Remove hostAliases from kube-vip podspec
    This is addressed by adding entries directly to the /etc/hosts
- Do a super-admin.conf switcheroo for the kube-vip static pod
    Add pre and post kubeadm commands for handling kubernetes
    versions v1.29.0+. The prekubeadm command checks if kubeadm
    init has been run and if it is, it replaces the kubeconfig
    hostPath in kube-vip static pod from admin.conf to the
    super-admin.conf. The postkubeadm command checks if
    kubeadm init has been run and if it is, it changes the
    hostPath in kube-vip static pod from super-admin.conf
    back to admin.conf.
thunderboltsid added a commit that referenced this pull request Mar 13, 2024
This required two changes:
- Remove hostAliases from kube-vip podspec
    This is addressed by adding entries directly to the /etc/hosts
- Do a super-admin.conf switcheroo for the kube-vip static pod
    Add pre and post kubeadm commands for handling kubernetes
    versions v1.29.0+. The prekubeadm command checks if kubeadm
    init has been run and if it is, it replaces the kubeconfig
    hostPath in kube-vip static pod from admin.conf to the
    super-admin.conf. The postkubeadm command checks if
    kubeadm init has been run and if it is, it changes the
    hostPath in kube-vip static pod from super-admin.conf
    back to admin.conf.
thunderboltsid added a commit that referenced this pull request Mar 13, 2024
This required two changes:
- Remove hostAliases from kube-vip podspec
    This is addressed by adding entries directly to the /etc/hosts
- Do a super-admin.conf switcheroo for the kube-vip static pod
    Add pre and post kubeadm commands for handling kubernetes
    versions v1.29.0+. The prekubeadm command checks if kubeadm
    init has been run and if it is, it replaces the kubeconfig
    hostPath in kube-vip static pod from admin.conf to the
    super-admin.conf. The postkubeadm command checks if
    kubeadm init has been run and if it is, it changes the
    hostPath in kube-vip static pod from super-admin.conf
    back to admin.conf.
thunderboltsid added a commit that referenced this pull request Mar 13, 2024
* Add support for Kubernetes v1.29 clusters (#382)

This required two changes:
- Remove hostAliases from kube-vip podspec
    This is addressed by adding entries directly to the /etc/hosts
- Do a super-admin.conf switcheroo for the kube-vip static pod
    Add pre and post kubeadm commands for handling kubernetes
    versions v1.29.0+. The prekubeadm command checks if kubeadm
    init has been run and if it is, it replaces the kubeconfig
    hostPath in kube-vip static pod from admin.conf to the
    super-admin.conf. The postkubeadm command checks if
    kubeadm init has been run and if it is, it changes the
    hostPath in kube-vip static pod from super-admin.conf
    back to admin.conf.

* Update controller-gen version to 0.14.0 and update makefile

Go1.22 broke controller-gen version 0.8.0 through 0.13.0.
Go1.22 drops support for GO111MODULE=off go get -u in legacy gopath
mode.
@thunderboltsid thunderboltsid added the enhancement New feature or request label Apr 8, 2024
thunderboltsid added a commit that referenced this pull request Apr 25, 2024
This required two changes:
- Remove hostAliases from kube-vip podspec
    This is addressed by adding entries directly to the /etc/hosts
- Do a super-admin.conf switcheroo for the kube-vip static pod
    Add pre and post kubeadm commands for handling kubernetes
    versions v1.29.0+. The prekubeadm command checks if kubeadm
    init has been run and if it is, it replaces the kubeconfig
    hostPath in kube-vip static pod from admin.conf to the
    super-admin.conf. The postkubeadm command checks if
    kubeadm init has been run and if it is, it changes the
    hostPath in kube-vip static pod from super-admin.conf
    back to admin.conf.
thunderboltsid added a commit that referenced this pull request Apr 26, 2024
This required two changes:
- Remove hostAliases from kube-vip podspec
    This is addressed by adding entries directly to the /etc/hosts
- Do a super-admin.conf switcheroo for the kube-vip static pod
    Add pre and post kubeadm commands for handling kubernetes
    versions v1.29.0+. The prekubeadm command checks if kubeadm
    init has been run and if it is, it replaces the kubeconfig
    hostPath in kube-vip static pod from admin.conf to the
    super-admin.conf. The postkubeadm command checks if
    kubeadm init has been run and if it is, it changes the
    hostPath in kube-vip static pod from super-admin.conf
    back to admin.conf.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved enhancement New feature or request lgtm
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants