Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use channel to manage goroutine when upgrade nodes #588

Merged
merged 2 commits into from
Jan 15, 2021

Conversation

oldthreefeng
Copy link
Collaborator

@oldthreefeng oldthreefeng commented Jan 14, 2021

Signed-off-by: oldthreefeng louisehong4168@gmail.com

[SKIP CI]seaols: 一句话简短描述该PR内容

升级最多并发升级两个节点.

01:02:37 [INFO] [upgrade.go:77] UpgradeMaster0
01:02:37 [INFO] [upgrade.go:106] first: to drain master node k8s-dev
01:02:37 [DEBG] [ssh.go:58] [192.168.60.136:22] kubectl drain k8s-dev --ignore-daemonsets --delete-local-data
01:02:38 [INFO] [ssh.go:51] [192.168.60.136:22] node/k8s-dev already cordoned
01:02:38 [INFO] [ssh.go:51] [192.168.60.136:22] WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-hrgfv, kube-system/kube-proxy-bpqfs, kube-system/upgrade-prepull-kube-apiserver-cc7hc, kube-system/upgrade-prepull-kube-controller-manager-pbp99, kube-system/upgrade-prepull-kube-scheduler-vmzcn
01:02:38 [INFO] [ssh.go:51] [192.168.60.136:22] node/k8s-dev drained
01:02:38 [INFO] [upgrade.go:117] second: to exec kubeadm upgrade node on k8s-dev
01:02:38 [DEBG] [ssh.go:58] [192.168.60.136:22] kubeadm upgrade apply --certificate-renewal=false  --yes v1.18.13
01:02:39 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/config] Making sure the configuration is correct:
01:02:39 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/config] Reading configuration from the cluster...
01:02:39 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
01:02:39 [INFO] [ssh.go:51] [192.168.60.136:22] [preflight] Running pre-flight checks.
01:02:39 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade] Running cluster health checks
01:02:42 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/version] You have chosen to change the cluster version to "v1.18.13"
01:02:42 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/versions] Cluster version: v1.17.0
01:02:42 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/versions] kubeadm version: v1.18.13
01:02:42 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
01:02:42 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/prepull] Prepulling image for component etcd.
01:02:42 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/prepull] Prepulling image for component kube-apiserver.
01:02:42 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/prepull] Prepulling image for component kube-controller-manager.
01:02:42 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/prepull] Prepulling image for component kube-scheduler.
01:02:42 [INFO] [ssh.go:51] [192.168.60.136:22] [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
01:02:42 [INFO] [ssh.go:51] [192.168.60.136:22] [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd
01:02:42 [INFO] [ssh.go:51] [192.168.60.136:22] [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
01:02:42 [INFO] [ssh.go:51] [192.168.60.136:22] [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
01:02:42 [INFO] [ssh.go:51] [192.168.60.136:22] [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd
01:02:47 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/prepull] Prepulled image for component kube-apiserver.
01:02:48 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/prepull] Prepulled image for component kube-controller-manager.
01:02:48 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/prepull] Prepulled image for component etcd.
01:02:48 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/prepull] Prepulled image for component kube-scheduler.
01:02:48 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/prepull] Successfully prepulled the images for all the control plane components
01:02:48 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.18.13"...
01:02:48 [INFO] [ssh.go:51] [192.168.60.136:22] Static pod: kube-apiserver-k8s-dev hash: 9ae9890169769060f0536fb1fffe09ec
01:02:49 [INFO] [ssh.go:51] [192.168.60.136:22] Static pod: kube-controller-manager-k8s-dev hash: 38a99425d6fa5318ad762d490fff7d82
01:02:49 [INFO] [ssh.go:51] [192.168.60.136:22] Static pod: kube-scheduler-k8s-dev hash: 9d6a9179c391d22675521c1de81c4493
01:02:49 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/etcd] Upgrading to TLS for etcd
01:02:49 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/etcd] Non fatal issue encountered during upgrade: the desired etcd version for this Kubernetes version "v1.18.13" is "3.4.3-0", but the current etcd version is "3.4.3". Won't downgrade etcd, instead just continue
01:02:49 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests223136939"
01:02:49 [INFO] [ssh.go:51] [192.168.60.136:22] W0115 01:02:49.978214   14539 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
01:02:50 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/staticpods] Preparing for "kube-apiserver" upgrade
01:02:50 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-01-15-01-02-48/kube-apiserver.yaml"
01:02:50 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/staticpods] Waiting for the kubelet to restart the component
01:02:50 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
01:02:50 [INFO] [ssh.go:51] [192.168.60.136:22] Static pod: kube-apiserver-k8s-dev hash: 9ae9890169769060f0536fb1fffe09ec
01:02:58 [INFO] [ssh.go:51] [192.168.60.136:22] Static pod: kube-apiserver-k8s-dev hash: 9ae9890169769060f0536fb1fffe09ec
01:02:58 [INFO] [ssh.go:51] [192.168.60.136:22] Static pod: kube-apiserver-k8s-dev hash: 9ae9890169769060f0536fb1fffe09ec
01:02:58 [INFO] [ssh.go:51] [192.168.60.136:22] Static pod: kube-apiserver-k8s-dev hash: 9ae9890169769060f0536fb1fffe09ec
01:02:59 [INFO] [ssh.go:51] [192.168.60.136:22] Static pod: kube-apiserver-k8s-dev hash: 9ae9890169769060f0536fb1fffe09ec
01:02:59 [INFO] [ssh.go:51] [192.168.60.136:22] Static pod: kube-apiserver-k8s-dev hash: 9ae9890169769060f0536fb1fffe09ec
01:03:00 [INFO] [ssh.go:51] [192.168.60.136:22] Static pod: kube-apiserver-k8s-dev hash: 9ae9890169769060f0536fb1fffe09ec
01:03:03 [INFO] [ssh.go:51] [192.168.60.136:22] Static pod: kube-apiserver-k8s-dev hash: 24a5ea3278e61a3b71a1e41866c04c1f
01:03:03 [INFO] [ssh.go:51] [192.168.60.136:22] [apiclient] Found 1 Pods for label selector component=kube-apiserver
01:03:10 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
01:03:10 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
01:03:10 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-01-15-01-02-48/kube-controller-manager.yaml"
01:03:10 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/staticpods] Waiting for the kubelet to restart the component
01:03:10 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
01:03:10 [INFO] [ssh.go:51] [192.168.60.136:22] Static pod: kube-controller-manager-k8s-dev hash: 38a99425d6fa5318ad762d490fff7d82
01:03:11 [INFO] [ssh.go:51] [192.168.60.136:22] Static pod: kube-controller-manager-k8s-dev hash: 0211507dd35608ae25cb3dedb59ddc8c
01:03:11 [INFO] [ssh.go:51] [192.168.60.136:22] [apiclient] Found 1 Pods for label selector component=kube-controller-manager
01:03:13 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
01:03:13 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/staticpods] Preparing for "kube-scheduler" upgrade
01:03:13 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-01-15-01-02-48/kube-scheduler.yaml"
01:03:13 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/staticpods] Waiting for the kubelet to restart the component
01:03:13 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
01:03:13 [INFO] [ssh.go:51] [192.168.60.136:22] Static pod: kube-scheduler-k8s-dev hash: 9d6a9179c391d22675521c1de81c4493
01:03:13 [INFO] [ssh.go:51] [192.168.60.136:22] Static pod: kube-scheduler-k8s-dev hash: 5e06824c8bd74e7e067ede11e7295da8
01:03:13 [INFO] [ssh.go:51] [192.168.60.136:22] [apiclient] Found 1 Pods for label selector component=kube-scheduler
01:03:15 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
01:03:15 [INFO] [ssh.go:51] [192.168.60.136:22] [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
01:03:15 [INFO] [ssh.go:51] [192.168.60.136:22] [kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
01:03:15 [INFO] [ssh.go:51] [192.168.60.136:22] [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
01:03:15 [INFO] [ssh.go:51] [192.168.60.136:22] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
01:03:15 [INFO] [ssh.go:51] [192.168.60.136:22] [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
01:03:15 [INFO] [ssh.go:51] [192.168.60.136:22] [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
01:03:15 [INFO] [ssh.go:51] [192.168.60.136:22] [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
01:03:15 [INFO] [ssh.go:51] [192.168.60.136:22] [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
01:03:17 [INFO] [ssh.go:51] [192.168.60.136:22] [addons] Applied essential addon: CoreDNS
01:03:18 [INFO] [ssh.go:51] [192.168.60.136:22] [addons] Applied essential addon: kube-proxy
01:03:18 [INFO] [ssh.go:51] [192.168.60.136:22] 
01:03:18 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.18.13". Enjoy!
01:03:18 [INFO] [ssh.go:51] [192.168.60.136:22] 
01:03:18 [INFO] [ssh.go:51] [192.168.60.136:22] [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
01:03:18 [INFO] [upgrade.go:137] third: to restart kubelet on k8s-dev
01:03:18 [DEBG] [ssh.go:58] [192.168.60.136:22] systemctl daemon-reload && systemctl restart kubelet
01:03:31 [INFO] [upgrade.go:147] fourth:  k8s-dev nodes is ready
01:03:31 [INFO] [upgrade.go:154] fifth: to uncordon node, 10 seconds to wait for k8s-dev uncordon
01:03:31 [INFO] [upgrade.go:84] UpgradeNodes
01:03:31 [INFO] [upgrade.go:113] first: to print upgrade node k8s-node2
01:03:31 [INFO] [upgrade.go:117] second: to exec kubeadm upgrade node on k8s-node2
01:03:31 [DEBG] [ssh.go:58] [192.168.60.139:22] kubeadm upgrade node --certificate-renewal=false
01:03:31 [INFO] [upgrade.go:113] first: to print upgrade node k8s-node1
01:03:31 [INFO] [upgrade.go:117] second: to exec kubeadm upgrade node on k8s-node1
01:03:31 [DEBG] [ssh.go:58] [192.168.60.138:22] kubeadm upgrade node --certificate-renewal=false
01:03:33 [INFO] [ssh.go:51] [192.168.60.138:22] [upgrade] Reading configuration from the cluster...
01:03:33 [INFO] [ssh.go:51] [192.168.60.138:22] [upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
01:03:33 [INFO] [ssh.go:51] [192.168.60.139:22] [upgrade] Reading configuration from the cluster...
01:03:33 [INFO] [ssh.go:51] [192.168.60.139:22] [upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
01:03:33 [INFO] [ssh.go:51] [192.168.60.138:22] [upgrade] Skipping phase. Not a control plane node.
01:03:33 [INFO] [ssh.go:51] [192.168.60.138:22] [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
01:03:33 [INFO] [ssh.go:51] [192.168.60.138:22] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
01:03:33 [INFO] [ssh.go:51] [192.168.60.138:22] [upgrade] The configuration for this node was successfully updated!
01:03:33 [INFO] [ssh.go:51] [192.168.60.138:22] [upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
01:03:33 [INFO] [ssh.go:51] [192.168.60.139:22] [upgrade] Skipping phase. Not a control plane node.
01:03:33 [INFO] [ssh.go:51] [192.168.60.139:22] [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
01:03:33 [INFO] [upgrade.go:137] third: to restart kubelet on k8s-node1
01:03:33 [DEBG] [ssh.go:58] [192.168.60.138:22] systemctl daemon-reload && systemctl restart kubelet
01:03:33 [INFO] [ssh.go:51] [192.168.60.139:22] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
01:03:33 [INFO] [ssh.go:51] [192.168.60.139:22] [upgrade] The configuration for this node was successfully updated!
01:03:33 [INFO] [ssh.go:51] [192.168.60.139:22] [upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
01:03:33 [INFO] [upgrade.go:137] third: to restart kubelet on k8s-node2
01:03:33 [DEBG] [ssh.go:58] [192.168.60.139:22] systemctl daemon-reload && systemctl restart kubelet
01:03:44 [INFO] [upgrade.go:147] fourth:  k8s-node2 nodes is ready
01:03:44 [ALRT] [drain.go:58] Node k8s-node2 is already cordoned: false
01:03:44 [INFO] [upgrade.go:154] fifth: to uncordon node, 10 seconds to wait for k8s-node2 uncordon
01:03:44 [INFO] [upgrade.go:113] first: to print upgrade node k8s-node3
01:03:44 [INFO] [upgrade.go:117] second: to exec kubeadm upgrade node on k8s-node3
01:03:44 [DEBG] [ssh.go:58] [192.168.60.140:22] kubeadm upgrade node --certificate-renewal=false
01:03:44 [INFO] [upgrade.go:147] fourth:  k8s-node1 nodes is ready
01:03:44 [ALRT] [drain.go:58] Node k8s-node1 is already cordoned: false
01:03:44 [INFO] [upgrade.go:154] fifth: to uncordon node, 10 seconds to wait for k8s-node1 uncordon
01:03:45 [INFO] [ssh.go:51] [192.168.60.140:22] [upgrade] Reading configuration from the cluster...
01:03:45 [INFO] [ssh.go:51] [192.168.60.140:22] [upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
01:03:45 [INFO] [ssh.go:51] [192.168.60.140:22] [upgrade] Skipping phase. Not a control plane node.
01:03:45 [INFO] [ssh.go:51] [192.168.60.140:22] [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
01:03:45 [INFO] [ssh.go:51] [192.168.60.140:22] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
01:03:45 [INFO] [ssh.go:51] [192.168.60.140:22] [upgrade] The configuration for this node was successfully updated!
01:03:45 [INFO] [ssh.go:51] [192.168.60.140:22] [upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
01:03:45 [INFO] [upgrade.go:137] third: to restart kubelet on k8s-node3
01:03:45 [DEBG] [ssh.go:58] [192.168.60.140:22] systemctl daemon-reload && systemctl restart kubelet
01:03:56 [INFO] [upgrade.go:147] fourth:  k8s-node3 nodes is ready
01:03:56 [ALRT] [drain.go:58] Node k8s-node3 is already cordoned: false
01:03:56 [INFO] [upgrade.go:154] fifth: to uncordon node, 10 seconds to wait for k8s-node3 uncordon

fix to Info not Alert on drain nodes.
add ip in logs to upgrade logs

Signed-off-by: oldthreefeng <louisehong4168@gmail.com>
Signed-off-by: oldthreefeng <louisehong4168@gmail.com>
Comment on lines 111 to +117
if strings.Contains(str, ApiServer) {
logger.Info("local %s is already exists %s", etcHostPath, ApiServer)
return nil
}
if err == io.EOF {
break
}
Copy link
Collaborator Author

@oldthreefeng oldthreefeng Jan 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果是最后一行, 就直接break了。 所以把判断是否存在放在前面。

@@ -11,7 +11,7 @@ func (s *SealosInstaller) SendPackage() {
// rm old sealos in package avoid old version problem. if sealos not exist in package then skip rm
kubeHook := fmt.Sprintf("cd /root && rm -rf kube && tar zxvf %s && cd /root/kube/shell && rm -f ../bin/sealos && bash init.sh", pkg)
deletekubectl := `sed -i '/kubectl/d;/sealos/d' /root/.bashrc `
completion := "echo 'command -v kubectl &>/dev/null && source <(kubectl completion bash)' >> /root/.bashrc && echo 'command -v sealos &>/dev/null && source <(sealos completion bash)' >> /root/.bashrc && source /root/.bashrc"
completion := "echo 'command -v kubectl &>/dev/null && source <(kubectl completion bash)' >> /root/.bashrc && echo '[ -x /usr/bin/sealos ] && source <(sealos completion bash)' >> /root/.bashrc && source /root/.bashrc"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

避免 permission deny 的问题。 如果 不能执行 sealos 就不需要执行 补全。

}(i)
}
pool.Wait()
println(runtime.NumGoroutine())
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -55,7 +55,7 @@ func CordonUnCordon(k8sClient *kubernetes.Clientset, nodeName string, cordoned b
return err
}
if node.Spec.Unschedulable == cordoned {
logger.Alert("Node %s is already cordoned: %v", nodeName, cordoned)
logger.Info("Node %s is already Uncordoned, skip...", nodeName)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果已经是ready。 则INFO 而不是Alert

@oldthreefeng oldthreefeng merged commit 9ec95b9 into labring:develop Jan 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant