staging for cgroupdriver , fix #582 #589

oldthreefeng · 2021-01-16T17:53:53Z

Signed-off-by: oldthreefeng louisehong4168@gmail.com

[SKIP CI]seaols: 一句话简短描述该PR内容

本次 PR的主要内容：

使用kubeadm init/join 前，判断当前主机cgroupdriver 并在kubeadm-config.yaml 写入cgroupdriver.

PR的优势:

可以兼容节点不同的cgroupdriver . 比如有 1/2/3三台机器 , 1的cgroup Driver 为 systemd, 2/3 为 cgroupfs. 当然这个不推荐.
可以减少一些因为cgroup不一致, 导致的安装报错问题. 增加安装的丝滑.
使用阿里云的kubelet的config.yaml(目前使用默认配置)配置. 减少默认配置造成的一些不必要的问题.

测试用例

需要测试sealos init / join

1master 节点 1 node节点，一致/不一致测试。 centos / ubuntu
3master节点测试. 一致测试
3master节点测试. cgroup driver 不一致测试.
1master 3node 测试, cgroup driver 一致测试.
1master 3node cgroup driver 不一致测试.
1master 3node cgroup driver 不一致测试. 并且使用自定义的 kubeadm-config.yaml 进行init.

oldthreefeng · 2021-01-17T01:52:25Z

install/constants.go

+	ContainerdShell = `if grep "SystemdCgroup = true"  /etc/containerd/config.toml &> /dev/nul; then  
+driver=systemd
+else
+driver=cgroupfs
+fi


因为 containerd 默认是 cgroupfs.
kubernetes 默认是 systemd. (这个他们好像在弄了, 现在还是cgroupfs).

oldthreefeng · 2021-01-17T01:54:46Z

install/init.go

@@ -54,9 +54,26 @@ func BuildInit() {
 	i.PrintFinish()
 }

+func (s *SealosInstaller) IsCgroupDriver(h string) bool {
+	var output string
+	if For120(Version) {


sealos init kubernetes >=1.20 , 则使用containerd. 否则使用 docker

oldthreefeng · 2021-01-18T08:12:07Z

install/constants.go

+driver=cgroupfs
+fi
+echo ${driver}`
+	DockerShell = `cgroupDriver=$(docker info|grep Cg)


driver=$(docker info -f "{{.CgroupDriver}}")好像更方便点.

driver=$(docker info -f "{{.CgroupDriver}}") 这样优雅多了，以前写法就是垃圾

改成 driver=$(docker info -f "{{.CgroupDriver}}") 这个吧

fanux · 2021-01-18T12:17:34Z

LGTM

fanux

把docker获取cgroup driver换成你的写法，果然我shell就是个渣渣

oldthreefeng

这个地方还是有可能会有问题. 如果已经安装了docker. docker会自动安装contianerd. 需要多次测试.

oldthreefeng · 2021-01-18T14:49:52Z

install/constants.go

+cgroupDriver: {{ .CgroupDriver}}`)
+
+const (
+	ContainerdShell = `if grep "SystemdCgroup = true"  /etc/containerd/config.toml &> /dev/null; then  


这个地方的处理只适用sealos安装的，如果自定义安装contianerd，并且不是通过配置文件修改cgroupdriver，使用containerd 命令行 --cgroup-manager flag启动.那么判断可能会有问题.

Signed-off-by: oldthreefeng <louisehong4168@gmail.com>

zhangguanzhang · 2021-03-29T03:15:26Z

containerd没有类似docker info -f "{{.CgroupDriver}}"的命令行获取信息的操作吗

oldthreefeng · 2021-03-29T06:16:25Z

containerd没有类似docker info -f "{{.CgroupDriver}}"的命令行获取信息的操作吗

我找了半天，没找到。看看这个老外的解释
kubernetes-sigs/cri-tools#728 (comment)

fanux · 2021-07-06T06:45:06Z

@oldthreefeng 这个any update? 已经写了把它收尾掉？

oldthreefeng · 2021-07-06T07:13:33Z

好。

* rewrite install and delete for app, app.tar don't send if already exist * read byte to remote file by sftp * add etcd health check * restore only one node to local * use subcommad instead Flag * single node save, healthcheck and restore test ok * recovery kube system when err happend by restore * use tar to compress instead of zip. * slove in docker use with save. * add --docker bool flag , if true , this will auto add unix timestamp to your snapshot suffix. * add save snapshot to oss. sealos config add oss-conf * feat(develop): 修复 #430 * fix ETCD CaCert or key file is not exist occurs panic, #427 * add kubernetes cronjob example yaml file * fix #441 * fix issue #443 * implementaion for sealos exec #429 * fix bug --label & --node will exec cmd twice and add example exec cmd * support exec scp local dir to remote * refactor exec command and use get ip by node name & by labelselector method to avoid for loop * fix sealos etcd health check for mutil master * add bash/zsh completion for sealos. from kubectl * fix kubeadm not found on old sealos package && fix port in exec cmd #469 && fix --service-cidr on old version && delete route cmd * fix 1.19.1 kube-controller-manager and kube-scheduler use the LocalAPIEndpoint instead of the ControlPlaneEndpoint. * add sealos route cmd docs * rm original sealos in old package to aviod some problem. add multi network install docs stage add upx in dockerfile, add upx in drone. change image to golang:15.2-alpine * dockerfile add upx stage build * fix versiontointall when version like v1.16.14 >= 1191 * add test record for upgrade cmd * when init , do not send twice, if valid copy md5 success ,do not logger * sepreate install master0 and other master when send ca and key and kubeconfig * fix --config when use customer config. * validate copy kubetarball * fix #499 * fix #509. * fix #534 only for 1.19.1 and 1.19.2 * /root/.kube/config设置为600，否则默认其它组有r权限，导致helm工具执行有警告 * when kubernetes gt 1.20, use Containerd instead of docker, #540 suport 1.20 containerd * fix #566, sealos clean --node 不小心写了 masterip add cleanCmd example * fix #571. handle unexpected error * fix #577, join node use config file, fix ipformat comment to oss when push to develop, only to tag to release Fix 1.14.x has no kubeadm.k8s.io/v1beta2 by use cli kubeadm join --xxxx. * fix #586, drain node is too danger for prod use; do not drain nodes drain worker node is too danger for prod use; do not drain nodes if worker nodes~ * fix build status (#610) * fix arm64 tags * fix #613, delete -i for cp command * fix #621. join node or master should not exsit in kubernetes. * feat(develop): fix ipip param not set false (#653) * # 决绝路径取消拼接，防止出错。 (#654) * feat(develop): fix cni config too long (#655) * fix version 3.19.1 yaml file lint error. (#656) * fix calico (#657) * fix calico version nil yaml file retrun null. (#658) * Update upgrade.md (#665) * use new const for kubeletconfig (#589) Signed-off-by: oldthreefeng <louisehong4168@gmail.com> * [WIP]Sealos kubeadm 1.23 v1beta3 (#673) * fix #671 * feat(develop): fix ci dir for sealos (#735) * fix bootstrapToken (#737) * feat(develop): rc6 release (#738) * ci(develop) fix golint for code and lic (#736) Co-authored-by: steven <fhtjob@hotmail.com> Co-authored-by: oldthreefeng <louisehong4168@gmail.com> Co-authored-by: 中弈 <zhongyi.fht@alibaba-inc.com> Co-authored-by: Ryan <yangchuansheng33@gmail.com> Co-authored-by: Louis <38313565+oldthreefeng@users.noreply.github.com> Co-authored-by: ysicing <i@ysicing.me> Co-authored-by: huizhi.szh <huizhi.szh@alibaba-inc.com> Co-authored-by: aiyijing <aiyijing@live.com> Co-authored-by: scott lewis <33612882+dk-lockdown@users.noreply.github.com> Co-authored-by: wenshihong <wsh@justep.com> Co-authored-by: wisheen <wisheen@163.com> Co-authored-by: Cluas <Cluas@live.cn> Co-authored-by: currycan <ansandyzzt@gmail.com> Co-authored-by: zhangzhitao <zhangzhitao@lakala.com> Co-authored-by: rick <rick@jenkins-zh.cn> Co-authored-by: panda-lab <64345457+panda-lab@users.noreply.github.com> Co-authored-by: 付亮 <48649837+hellolittlewei@users.noreply.github.com> Co-authored-by: SorryMaker <41097970+CodeSorryMaker@users.noreply.github.com>

* ci(develop) fix golint for code and lic (#736) * merge to master (#739) * rewrite install and delete for app, app.tar don't send if already exist * read byte to remote file by sftp * add etcd health check * restore only one node to local * use subcommad instead Flag * single node save, healthcheck and restore test ok * recovery kube system when err happend by restore * use tar to compress instead of zip. * slove in docker use with save. * add --docker bool flag , if true , this will auto add unix timestamp to your snapshot suffix. * add save snapshot to oss. sealos config add oss-conf * feat(develop): 修复 #430 * fix ETCD CaCert or key file is not exist occurs panic, #427 * add kubernetes cronjob example yaml file * fix #441 * fix issue #443 * implementaion for sealos exec #429 * fix bug --label & --node will exec cmd twice and add example exec cmd * support exec scp local dir to remote * refactor exec command and use get ip by node name & by labelselector method to avoid for loop * fix sealos etcd health check for mutil master * add bash/zsh completion for sealos. from kubectl * fix kubeadm not found on old sealos package && fix port in exec cmd #469 && fix --service-cidr on old version && delete route cmd * fix 1.19.1 kube-controller-manager and kube-scheduler use the LocalAPIEndpoint instead of the ControlPlaneEndpoint. * add sealos route cmd docs * rm original sealos in old package to aviod some problem. add multi network install docs stage add upx in dockerfile, add upx in drone. change image to golang:15.2-alpine * dockerfile add upx stage build * fix versiontointall when version like v1.16.14 >= 1191 * add test record for upgrade cmd * when init , do not send twice, if valid copy md5 success ,do not logger * sepreate install master0 and other master when send ca and key and kubeconfig * fix --config when use customer config. * validate copy kubetarball * fix #499 * fix #509. * fix #534 only for 1.19.1 and 1.19.2 * /root/.kube/config设置为600，否则默认其它组有r权限，导致helm工具执行有警告 * when kubernetes gt 1.20, use Containerd instead of docker, #540 suport 1.20 containerd * fix #566, sealos clean --node 不小心写了 masterip add cleanCmd example * fix #571. handle unexpected error * fix #577, join node use config file, fix ipformat comment to oss when push to develop, only to tag to release Fix 1.14.x has no kubeadm.k8s.io/v1beta2 by use cli kubeadm join --xxxx. * fix #586, drain node is too danger for prod use; do not drain nodes drain worker node is too danger for prod use; do not drain nodes if worker nodes~ * fix build status (#610) * fix arm64 tags * fix #613, delete -i for cp command * fix #621. join node or master should not exsit in kubernetes. * feat(develop): fix ipip param not set false (#653) * # 决绝路径取消拼接，防止出错。 (#654) * feat(develop): fix cni config too long (#655) * fix version 3.19.1 yaml file lint error. (#656) * fix calico (#657) * fix calico version nil yaml file retrun null. (#658) * Update upgrade.md (#665) * use new const for kubeletconfig (#589) Signed-off-by: oldthreefeng <louisehong4168@gmail.com> * [WIP]Sealos kubeadm 1.23 v1beta3 (#673) * fix #671 * feat(develop): fix ci dir for sealos (#735) * fix bootstrapToken (#737) * feat(develop): rc6 release (#738) * ci(develop) fix golint for code and lic (#736) Co-authored-by: steven <fhtjob@hotmail.com> Co-authored-by: oldthreefeng <louisehong4168@gmail.com> Co-authored-by: 中弈 <zhongyi.fht@alibaba-inc.com> Co-authored-by: Ryan <yangchuansheng33@gmail.com> Co-authored-by: Louis <38313565+oldthreefeng@users.noreply.github.com> Co-authored-by: ysicing <i@ysicing.me> Co-authored-by: huizhi.szh <huizhi.szh@alibaba-inc.com> Co-authored-by: aiyijing <aiyijing@live.com> Co-authored-by: scott lewis <33612882+dk-lockdown@users.noreply.github.com> Co-authored-by: wenshihong <wsh@justep.com> Co-authored-by: wisheen <wisheen@163.com> Co-authored-by: Cluas <Cluas@live.cn> Co-authored-by: currycan <ansandyzzt@gmail.com> Co-authored-by: zhangzhitao <zhangzhitao@lakala.com> Co-authored-by: rick <rick@jenkins-zh.cn> Co-authored-by: panda-lab <64345457+panda-lab@users.noreply.github.com> Co-authored-by: 付亮 <48649837+hellolittlewei@users.noreply.github.com> Co-authored-by: SorryMaker <41097970+CodeSorryMaker@users.noreply.github.com> * Revert "merge to master (#739)" (#741) This reverts commit c8349b0. * Update README.md * ci(master): add dockerfile * hotfix(master): clean panic fix by lock (#750) * refactor(ci): add auto invite (#762) * refactor(ci): add auto invite (#763) * refactor(ci): add auto invite * docs: readme align * refactor(dev): fix docs site (#773) * refactor(master): cloud,app feature close (#774) * refactor(master): cloud,app feature close * 缺陷: 最新版本sealos init 安装k8s 失败 (#778) Fixes #691 * refactor(master): release rc.8 (#782) * refactor(master): changelog (#784) * update changelog to master (#785) * refactor(master): changelog Co-authored-by: steven <fhtjob@hotmail.com> Co-authored-by: oldthreefeng <louisehong4168@gmail.com> Co-authored-by: 中弈 <zhongyi.fht@alibaba-inc.com> Co-authored-by: Ryan <yangchuansheng33@gmail.com> Co-authored-by: Louis <38313565+oldthreefeng@users.noreply.github.com> Co-authored-by: ysicing <i@ysicing.me> Co-authored-by: huizhi.szh <huizhi.szh@alibaba-inc.com> Co-authored-by: aiyijing <aiyijing@live.com> Co-authored-by: scott lewis <33612882+dk-lockdown@users.noreply.github.com> Co-authored-by: wenshihong <wsh@justep.com> Co-authored-by: wisheen <wisheen@163.com> Co-authored-by: Cluas <Cluas@live.cn> Co-authored-by: currycan <ansandyzzt@gmail.com> Co-authored-by: zhangzhitao <zhangzhitao@lakala.com> Co-authored-by: rick <rick@jenkins-zh.cn> Co-authored-by: panda-lab <64345457+panda-lab@users.noreply.github.com> Co-authored-by: 付亮 <48649837+hellolittlewei@users.noreply.github.com> Co-authored-by: SorryMaker <41097970+CodeSorryMaker@users.noreply.github.com> Co-authored-by: jiangyanfei <749985146@qq.com> Co-authored-by: ldseraph <ld.seraph@163.com>

oldthreefeng commented Jan 17, 2021

View reviewed changes

oldthreefeng requested a review from cuisongliu January 17, 2021 01:55

oldthreefeng force-pushed the cgroupdriver branch from 0e53a84 to 5edfd53 Compare January 17, 2021 08:20

oldthreefeng commented Jan 18, 2021

View reviewed changes

fanux reviewed Jan 18, 2021

View reviewed changes

oldthreefeng requested a review from fanux January 18, 2021 12:23

oldthreefeng commented Jan 20, 2021

View reviewed changes

oldthreefeng mentioned this pull request Feb 3, 2021

缺陷: Sealos部署的集群不能正确处理node下线状态 #596

Closed

oldthreefeng force-pushed the cgroupdriver branch 2 times, most recently from 20275df to b8aee3f Compare February 3, 2021 10:24

use new const for kubeletconfig

7f04f51

Signed-off-by: oldthreefeng <louisehong4168@gmail.com>

oldthreefeng force-pushed the cgroupdriver branch from 4e869b5 to 7f04f51 Compare March 11, 2021 12:21

cuisongliu approved these changes Mar 28, 2021

View reviewed changes

oldthreefeng changed the title ~~[wip] staging for cgroupdriver , fix #582~~ [need to test] staging for cgroupdriver , fix #582 Aug 6, 2021

cuisongliu merged commit b6144f3 into labring:develop Sep 2, 2021

cuisongliu mentioned this pull request Sep 2, 2021

using systemd as default cgroup driver sealerio/sealer#489

Closed

oldthreefeng changed the title ~~[need to test] staging for cgroupdriver , fix #582~~ staging for cgroupdriver , fix #582 Sep 9, 2021

cuisongliu mentioned this pull request Dec 9, 2021

merge to master (#739) #740

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

staging for cgroupdriver , fix #582 #589

staging for cgroupdriver , fix #582 #589

oldthreefeng commented Jan 16, 2021 •

edited

Loading

oldthreefeng Jan 17, 2021

oldthreefeng Jan 17, 2021

oldthreefeng Jan 18, 2021 •

edited

Loading

fanux Jan 18, 2021

fanux Jan 18, 2021

oldthreefeng Jan 18, 2021

fanux commented Jan 18, 2021

fanux left a comment

oldthreefeng left a comment

oldthreefeng Jan 18, 2021

zhangguanzhang commented Mar 29, 2021

oldthreefeng commented Mar 29, 2021

fanux commented Jul 6, 2021

oldthreefeng commented Jul 6, 2021

staging for cgroupdriver , fix #582 #589

staging for cgroupdriver , fix #582 #589

Conversation

oldthreefeng commented Jan 16, 2021 • edited Loading

测试用例

oldthreefeng Jan 17, 2021

Choose a reason for hiding this comment

oldthreefeng Jan 17, 2021

Choose a reason for hiding this comment

oldthreefeng Jan 18, 2021 • edited Loading

Choose a reason for hiding this comment

fanux Jan 18, 2021

Choose a reason for hiding this comment

fanux Jan 18, 2021

Choose a reason for hiding this comment

oldthreefeng Jan 18, 2021

Choose a reason for hiding this comment

fanux commented Jan 18, 2021

fanux left a comment

Choose a reason for hiding this comment

oldthreefeng left a comment

Choose a reason for hiding this comment

oldthreefeng Jan 18, 2021

Choose a reason for hiding this comment

zhangguanzhang commented Mar 29, 2021

oldthreefeng commented Mar 29, 2021

fanux commented Jul 6, 2021

oldthreefeng commented Jul 6, 2021

oldthreefeng commented Jan 16, 2021 •

edited

Loading

oldthreefeng Jan 18, 2021 •

edited

Loading