Fix multi-node cluster not working after restarting docker #2671

tnqn · 2022-03-11T15:04:02Z

In a multi-node cluster with single controlplane node, if the
controlplane node's IP changes, kube-controller-manager and
kube-scheduler would fail to connect kube-apiserver. Updating the
server address to new IP doesn't work because the API server's
certificate isn't valid for it.

This patch uses "patches" option of kubeadm to replace the server
address in the kubeconfig files of kube-controller-manager and
kube-scheduler with loopback address, which is an alternative address
of the API server's certificate.

Signed-off-by: Quan Tian qtian@vmware.com

For #1685

Depends on kubernetes/kubernetes#109886

k8s-ci-robot · 2022-03-11T15:04:10Z

Welcome @tnqn!

It looks like this is your first PR to kubernetes-sigs/kind 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kind has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2022-03-11T15:04:11Z

Hi @tnqn. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

aojea · 2022-03-11T15:13:02Z

images/base/files/usr/local/bin/entrypoint

+      # kubeconfig should be updated. However, the server certificate isn't valid for the new IP. We update it to
+      # loopback address as it's an alternative address of the certificate.
+      if [[ "${old_ipv4}" != "${curr_ipv4}" ]]; then
+        sed -i "s#${old_ipv4}#127.0.0.1#" /etc/kubernetes/controller-manager.conf || true


hmm, @neolit123 how are these files generated?
can we influence them?

it seems easier to generate them with the localhost IP instead of the node IP, they will only contact the local apiserver ... in HA we have one of these in each CP

what I'm saying is to modify the code in KIND so the scheduler and the controller-manager always use API server ip address = localhost , maybe there is a knob or a flag to do it

Thanks @aojea. I agree with you and I actually tried it but it seems more complicated.
The address in controller-manager.conf and scheduler.conf comes from localAPIEndpoint.advertiseAddress:

kind/pkg/cluster/internal/kubeadm/config.go

Lines 210 to 211 in b4ab02c

localAPIEndpoint:

advertiseAddress: "{{ .AdvertiseAddress }}"

https://github.com/kubernetes/kubernetes/blob/cdee77a4a9ba77a01bab6b462c1780d7a52af8c3/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go#L436

But localAPIEndpoint.advertiseAddress is used in many places in kubeadm so we can't simply replace it with localhost IP.

the commit says kubernetes/kubernetes@d944190

Instead of using the CPE for these components, use the LocalAPIEndpoint. This guarantees that the components would talk to the local kube-apiserver, which should be the same version, unless the user explicitly patched manifests.

wonder if we can patch it

I guess kubeadm uses the local node IP instead of localhost because kubeadm doesn't add localhost to alternative addresses of the server certificate by default.
kind adds it for specific reason:

kind/pkg/cluster/internal/kubeadm/config.go

Lines 167 to 171 in b4ab02c

# on docker for mac we have to expose the api server via port forward,

# so we need to ensure the cert is valid for localhost so we can talk

# to the cluster after rewriting the kubeconfig to point to localhost

apiServer:

certSANs: [localhost, "{{.APIServerAddress}}"]

.

localAPIEndpoint.advertiseAddress is an IP because that's what kube-apiserver supports. Advertise / bind addresses for k8s components are IPs...they cannot be hostnames. So it must be 127.0.0.1 and not localhost. Just wanted to be clear about that.

I think I've seen users binding everything to loopback and it should work. I haven't tried it. If something in kubeadm is blocking you let me know. But also check if the manifest patches work for you ALA initconfiguration.patches

nah, this is not about advertise addess, is about setting the destination address for the controller-manager and the scheduler ... but I understand now, it uses the advertise address for that , so I found the circle now ...
If kubeadm was fine until now this way I don't think we should modify it just for this

aojea · 2022-03-17T10:10:27Z

/ok-to-test

I would prefer to do less magic
https://github.com/kubernetes-sigs/kind/pull/2671/files#r824831420
@neolit123 if the goal is to talk to localhost why not be able to indicate localhost directly? do you have some context?

aojea · 2022-03-17T10:12:42Z

images/base/files/usr/local/bin/entrypoint

+      # controller-manager and scheduler connect to local API endpoint. If the Node's IP changes, server address in
+      # kubeconfig should be updated. However, the server certificate isn't valid for the new IP. We update it to
+      # loopback address as it's an alternative address of the certificate.
+      if [[ "${old_ipv4}" != "${curr_ipv4}" ]]; then


we are unconditionally changing the IP to 127.0.0.1 , do we need the if?

I added this check because if docker restart doesn't change the Node's IP, it can continue working with the original IP. I'm not sure which is better: use the trick when it has to, or use it unconditionally as long as the container is restarted. Would like to hear your opinion.

yeah, that sounds safer, we don't change the current behavior

aojea · 2022-03-17T11:04:16Z

/lgtm
/assign @BenTheElder

I can't think in a better solution , we are already doing "magic" for reboots, this is building on that so 🤷

BenTheElder · 2022-03-23T19:04:10Z

Thank you for this!

I think we should probably be altering the config during the initial bringup if at all possible, rather than on-reboot, where we have static changes like this (substituting loopback). If kubeadm config doesn't enable this directly, we may have to move into using the phases and generating, which will be trickier but probably the better longterm approach 🤔

tnqn · 2022-04-01T12:55:08Z

I think we should probably be altering the config during the initial bringup if at all possible, rather than on-reboot, where we have static changes like this (substituting loopback). If kubeadm config doesn't enable this directly, we may have to move into using the phases and generating, which will be trickier but probably the better longterm approach 🤔

@BenTheElder Thanks for the suggestion. I investigated several approaches to alter the config in the beginning but found they seem not ideal:

Set --master in extraArgs of kubeadm ConfigTemplate to override server address in kubeconfig files. However, it only overrides the server address in the main kubeconfig but not --authentication-kubeconfig and --authorization-kubeconfig, there would be continuous error logs like the following after Node IP change:

W0401 09:08:40.383960       1 reflector.go:324] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:205: failed to list *v1.ConfigMap: Get "https://[fc00:f853:ccd:e793::3]:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp [fc00:f853:ccd:e793::3]:6443: connect: connection refused
E0401 09:08:40.384763       1 reflector.go:138] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:205: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://[fc00:f853:ccd:e793::3]:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp [fc00:f853:ccd:e793::3]:6443: connect: connection refused

initconfiguration.patches doesn't help much as it patches the static pod manifest, not the kubeconfig file. It's not much different from the 1st approach.
Use phases: However, even the kubeconfig phase command doesn't accept loopback address as --apiserver-advertise-address with the following error. So we will have to use substituting command in-between phase commands. Is it what you suggested?

# kubeadm init phase kubeconfig controller-manager --apiserver-advertise-address 127.0.0.1
unable to select an IP from lo network interface

for _, cmd := range []exec.Cmd {
	phaseCommand("kubelet-start"),
	phaseCommand("certs", "all"),
	phaseCommand("kubeconfig", "all"),
	replaceServerCommand(nodeAddress, "127.0.0.1", "/etc/kubernetes/controller-manager.conf"),
	replaceServerCommand(nodeAddress, "127.0.0.1", "/etc/kubernetes/scheduler.conf"),
	phaseCommand("control-plane", "all"),
	...
} {
	lines, err := exec.CombinedOutputLines(cmd)
	ctx.Logger.V(3).Info(strings.Join(lines, "\n"))
	if err != nil {
		return errors.Wrap(err, "failed to init node with kubeadm")
	}
}

BenTheElder · 2022-05-03T19:20:55Z

Use phases: However, even the kubeconfig phase command doesn't accept loopback address as --apiserver-advertise-address with the following error. So we will have to use substituting command in-between phase commands. Is it what you suggested?

yes, sorry.

this change will be more involved, alternatively if we can make it possible to specify everything to kubeadm between patches and other inputs that would be reasonable.

BenTheElder · 2022-05-03T19:23:07Z

[I just realized I never replied to this, I don't know how it slipped through ... so many notifications, I apologize]

I think most ideal is one of:

kuebeadm makes it possible to specify this directly
kind uses phases to work around kubeadm

Doing it only upon reboot will be trickier to debug / surprising and be more likely to leave this broken in the future as we won't be doing that in CI.

tnqn · 2022-05-07T17:14:31Z

@BenTheElder @neolit123 @aojea I didn't find a suitable option name to specify the server address for kube-controller-manager and kube-scheduler separately in kubeadm, and the change may sound too specific to kind. So I followed @neolit123's suggestion at #2671 (comment) and extended kubeadm's patching support to kubeconfig files (kubernetes/kubernetes#109886). This PR has been updated accordingly. Please let me know your opinions.

k8s-ci-robot · 2022-05-07T17:15:36Z

New changes are detected. LGTM label has been removed.

k8s-ci-robot · 2022-05-07T17:15:47Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tnqn
To complete the pull request process, please ask for approval from aojea after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

In a multi-node cluster with single controlplane node, if the controlplane node's IP changes, kube-controller-manager and kube-scheduler would fail to connect kube-apiserver. Updating the server address to new IP doesn't work because the API server's certificate isn't valid for it. This patch uses "patches" option of kubeadm to replace the server address in the kubeconfig files of kube-controller-manager and kube-scheduler with loopback address, which is an alternative address of the API server's certificate. Signed-off-by: Quan Tian <qtian@vmware.com>

neolit123 · 2022-05-09T07:53:38Z

The kubeadm patches feature has a KEP so ideally we should document the change and rationale before adding it. Personally i do not see a problem with adding kubeconfig patches but i need to double check.

Phases should already support what is required here:

Call kubeconfig phase on demand
modify the generated files
call kubeadm but skip kubeconfig phase

Then again, one of the reasons we added patches was because users did not like phases for some customizations.

I can comment more on kubernetes/kubernetes#109886

Trade off - If it merges, only kubeadm 1.25 will support this, while using phases is supported for a number of old kubeadm versions already.

tnqn · 2022-05-09T09:29:23Z

@neolit123 thanks for your comment. I thought about using phases but found:

For init phases, we would need to call certs first then kubeconfig, otherwise the latter would fail due of missing cert, so it would become calling two phases first, modify the generated files, call kubeadm but skip the two phases
For join phases, it would depend on whether the node is a controlplane Node to decide whether to call control-plane-prepare and modify the generated files.

It seems it would become a bit coupled with the content of phases and may need some adjustment when kubeadm phases change, not sure if it's a suitable long term solution.

For kubernetes/kubernetes#109886, if we agree to this approach, I would be happy to follow your guidance to add documentation for the change.

neolit123 · 2022-05-09T12:29:27Z

@tnqn

can you clarify what is the blocker for using this in kind via {Init|Join}Configuration[.controlPlane].localApiEndpoint?

localAPIEndpoint: 
   advertiseAddress: 127.0.0.1 # or v6

#2671 (comment)

errors are not returned for ip.IsLoopback(),
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/config/common.go#L142-L161

so unless i'm missing something kubeadm should allow you to bind all components and kubeconfigs (except kubelets) to loopback by setting it in advertiseAddress.

neolit123 · 2022-05-09T12:40:44Z

For init phases, we would need to call certs first then kubeconfig, otherwise the latter would fail due of missing cert, so it would become calling two phases first, modify the generated files, call kubeadm but skip the two phases

For join phases, it would depend on whether the node is a controlplane Node to decide whether to call control-plane-prepare and modify the generated files.

these seem tolerable. patches also have complexity as kubeadm would need to load files from disk (i.e. from the kind nodes).

It seems it would become a bit coupled with the content of phases and may need some adjustment when kubeadm phases change, not sure if it's a suitable long term solution.

phases are CLI and have a GA deprecation policy (unless marked as experimental) in kubeadm (> 1 year / change if it happens).

For kubernetes/kubernetes#109886, if we agree to this approach, I would be happy to follow your guidance to add documentation for the change.

i think the biggest issue is this trade off that i've mentioned:

Trade off - If it merges, only kubeadm 1.25 will support this, while using phases is supported for a number of old kubeadm versions already.

tnqn · 2022-05-09T12:51:03Z

@tnqn

can you clarify what is the blocker for using this in kind via {Init|Join}Configuration[.controlPlane].localApiEndpoint?
localAPIEndpoint: 
   advertiseAddress: 127.0.0.1 # or v6
#2671 (comment)

errors are not returned for ip.IsLoopback(), https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/config/common.go#L142-L161

so unless i'm missing something kubeadm should allow you to bind all components and kubeconfigs (except kubelets) to loopback by setting it in advertiseAddress.

It would return the error "unable to select an IP from lo network interface" because setting loopback address has a special meaning as explained by the below comment and it expects another global IP on loopback device.
https://github.com/kubernetes/kubernetes/blob/4717a59097d7365ddcea578dae9aa29ade78398a/cmd/kubeadm/app/util/config/initconfiguration.go#L138-L141

neolit123 · 2022-05-09T13:41:24Z

It would return the error "unable to select an IP from lo network interface" because setting loopback address has a special meaning as explained by the below comment and it expects another global IP on loopback device.
https://github.com/kubernetes/kubernetes/blob/4717a59097d7365ddcea578dae9aa29ade78398a/cmd/kubeadm/app/util/config/initconfiguration.go#L138-L141

i vaguely recall when that logic was added and also recall the "-1"s from people when "just allowing loopback" was discussed.

so one question here would be whether we want to change the logic and just allow loopback addresses.
but if we do that we have to somehow preserve the current bahavior "if loopback address -> ask LO interface -> pick global address".

tnqn · 2022-05-09T15:19:52Z

i vaguely recall when that logic was added and also recall the "-1"s from people when "just allowing loopback" was discussed.

so one question here would be whether we want to change the logic and just allow loopback addresses. but if we do that we have to somehow preserve the current bahavior "if loopback address -> ask LO interface -> pick global address".

It seems even kubeadm allows loopback address, kube-apiserver validation doesn't allow it (when it tries to sync kubernetes service): https://github.com/kubernetes/kubernetes/blob/543893cbb0948bf5e4fad5eb71d8e70537453698/pkg/apis/core/validation/validation.go#L6056-L6076

F W0509 15:10:24.123176       1 lease.go:250] Resetting endpoints for master service "kubernetes" to [127.0.0.1]
F E0509 15:10:24.128276       1 controller.go:241] unable to sync kubernetes service: Endpoints "kubernetes" is invalid: subsets[0].addresses[0].ip: Invalid value: "127.0.0.1": may not be in the loopback range (127.0.0.0/8, ::1/128)

neolit123 · 2022-05-09T15:45:08Z

It seems even kubeadm allows loopback address, kube-apiserver validation doesn't allow it (when it tries to sync kubernetes service):

Does this mean that even if we use kubeadm phases or patches, apiserver will just fail if we use loopback address? How would the kind multi CP node restart feature work in that case?

aojea · 2022-05-09T15:51:21Z

I lost track and could not read all messages, so I may miss some context

, kube-apiserver validation doesn't allow it (when it tries to sync kubernetes service):

that validation is for Services and Endpoints, those IPs should be reachable, localhost is not valid, but it's orthogonal to what we want to do , that is tell controller-manager and scheduler to use the 127.0.0.1 to communicate with the apiserver.

The thing is that the apiserver address is obtained from localAPIEndpoint.advertiseAddress ... but we should not touch that, just add the way of telling those components to use localhost to reach apiserver, because we know that those components can communicate through localhost because are in the same host

... it is something like controller-manager, scheduler --use-loopback

tnqn · 2022-05-09T16:00:30Z

Yes, what the patch does only affects which address kube-controller-manager and kube-scheduler use to connect to kube-apiserver, not the address kube-apiserver listens to.

BenTheElder · 2022-05-09T22:55:41Z

I lost track and could not read all messages, so I may miss some context

Incidentally, this is why our contributing guide asks that features first be discussed in issues prior to implementation, it's hard to keep up with review + approach discussion on PRs.

https://kind.sigs.k8s.io/docs/contributing/getting-started/

I think this is also the case with the kubeadm OWNERS.

I really appreciate the effort on this though.

BenTheElder · 2022-05-09T22:55:35Z

pkg/cluster/internal/create/actions/config/config.go

+
+	patches := map[string]string{}
+	// Kubernetes older than v1.25 don't support patching kubeconfig files.
+	if ver.AtLeast(version.MustParseSemantic("v1.25.0")) {


we'll want to make this more precise to the version that supports this, so we can start testing it sooner. we can get a version after merge if it merges with hack/print-workspace-status.sh

BenTheElder · 2022-05-09T23:00:35Z

pkg/cluster/internal/create/actions/config/config.go

+	if ver.AtLeast(version.MustParseSemantic("v1.25.0")) {
+		// controller-manager and scheduler connect to local API endpoint, which defaults to the advertise address of the
+		// API server. If the Node's IP changes (which could happen after docker restarts), the server address in KubeConfig
+		// should be updated. However, the server certificate isn't valid for the new IP. To resolve it, we update the


we should probably fix this, perhaps in the entrypoint we can re-roll the cert if it exists?

Thanks for the suggestion. It reminds me that since we have the CA cert and key, we could just re-roll the cert and replace the server address in kubeconfig with the new Node IP like the existing magic:

kind/images/base/files/usr/local/bin/entrypoint

Lines 413 to 414 in 1a6bfa7

# kubernetes manifests are only present on control-plane nodes

sed -i "s#${old_ipv4}#${curr_ipv4}#" /etc/kubernetes/manifests/*.yaml || true

Then everything should just work in theory. Do you think this is a better solution?

I tried the above approach and it worked as expected, created issue #2759 to discuss the approaches.

aojea · 2022-05-10T08:29:44Z

pkg/cluster/internal/create/actions/config/config.go

+		loopbackAddress := "127.0.0.1"
+		if data.IPFamily == config.IPv6Family {
+			loopbackAddress = "[::1]"
+		}
+		jsonPatch := fmt.Sprintf(`[{"op": "replace", "path": "/clusters/0/cluster/server", "value": "https://%s:%d"}]`, loopbackAddress, data.APIBindPort)


Suggested change

loopbackAddress := "127.0.0.1"

if data.IPFamily == config.IPv6Family {

loopbackAddress = "[::1]"

}

jsonPatch := fmt.Sprintf(`[{"op": "replace", "path": "/clusters/0/cluster/server", "value": "https://%s:%d"}]`, loopbackAddress, data.APIBindPort)

loopbackAddress := "127.0.0.1"

if data.IPFamily == config.IPv6Family {

loopbackAddress = "::1"

}

jsonPatch := fmt.Sprintf(`[{"op": "replace", "path": "/clusters/0/cluster/server", "value": "https://%s"}]`, net.JoinHostPort(loopbackAddress, strconv.Itoa(data.APIBindPort)))

k8s-ci-robot · 2022-05-10T21:54:08Z

@tnqn: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kind-e2e-kubernetes-1-24	`4d6b0c8`	link	true	`/test pull-kind-e2e-kubernetes-1-24`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2022-05-10T21:54:15Z

@tnqn: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BenTheElder · 2022-05-25T23:53:51Z

#2775, thanks!

SiegfredRodriguez · 2023-05-02T09:29:19Z

This seems to still occur on v0.18.0, is there a specific usage change needed to use #2775 ?

BenTheElder · 2023-05-02T17:40:00Z

@SiegfredRodriguez please file a bug with cluster logs etc unless it's #1689 which is a known limitation.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 11, 2022

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Mar 11, 2022

k8s-ci-robot requested a review from amwat March 11, 2022 15:04

k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Mar 11, 2022

k8s-ci-robot requested a review from aojea March 11, 2022 15:04

tnqn mentioned this pull request Mar 11, 2022

When after restart docker, kind cluster could't connect #1685

Closed

aojea reviewed Mar 11, 2022

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 17, 2022

aojea reviewed Mar 17, 2022

View reviewed changes

k8s-ci-robot assigned BenTheElder and aojea Mar 17, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 17, 2022

This was referenced May 3, 2022

multi-node: Kubernetes cluster does not start after Docker re-assigns node's IP addresses after (Docker) restart #2045

Closed

HA clusters don't reboot properly #1689

Open

tnqn mentioned this pull request May 7, 2022

kubeadm: support patching kubeconfig files kubernetes/kubernetes#109886

Closed

tnqn force-pushed the fix-docker-restart branch from 99b7c18 to 1b14fe8 Compare May 7, 2022 17:15

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 7, 2022

k8s-ci-robot removed the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label May 7, 2022

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label May 7, 2022

tnqn force-pushed the fix-docker-restart branch from 1b14fe8 to 4d6b0c8 Compare May 7, 2022 17:17

BenTheElder mentioned this pull request May 9, 2022

Add start and stop command to kind. #2715

Open

BenTheElder reviewed May 9, 2022

View reviewed changes

aojea reviewed May 10, 2022

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 10, 2022

tnqn mentioned this pull request May 12, 2022

Support docker restart for clusters with multiple nodes (single controlplane node) #2759

Closed

BenTheElder closed this May 25, 2022

	localAPIEndpoint:
	advertiseAddress: "{{ .AdvertiseAddress }}"

	# on docker for mac we have to expose the api server via port forward,
	# so we need to ensure the cert is valid for localhost so we can talk
	# to the cluster after rewriting the kubeconfig to point to localhost
	apiServer:
	certSANs: [localhost, "{{.APIServerAddress}}"]

	# kubernetes manifests are only present on control-plane nodes
	sed -i "s#${old_ipv4}#${curr_ipv4}#" /etc/kubernetes/manifests/*.yaml \|\| true

Fix multi-node cluster not working after restarting docker #2671

Fix multi-node cluster not working after restarting docker #2671

Conversation

tnqn commented Mar 11, 2022 • edited Loading

k8s-ci-robot commented Mar 11, 2022

k8s-ci-robot commented Mar 11, 2022

aojea Mar 11, 2022 • edited Loading

Choose a reason for hiding this comment

aojea Mar 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnqn Mar 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aojea commented Mar 17, 2022

aojea Mar 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aojea commented Mar 17, 2022

BenTheElder commented Mar 23, 2022

tnqn commented Apr 1, 2022 • edited Loading

BenTheElder commented May 3, 2022

BenTheElder commented May 3, 2022

tnqn commented May 7, 2022

k8s-ci-robot commented May 7, 2022

k8s-ci-robot commented May 7, 2022

neolit123 commented May 9, 2022

tnqn commented May 9, 2022

neolit123 commented May 9, 2022 • edited Loading

neolit123 commented May 9, 2022 • edited Loading

tnqn commented May 9, 2022

neolit123 commented May 9, 2022

tnqn commented May 9, 2022 • edited Loading

neolit123 commented May 9, 2022 • edited Loading

aojea commented May 9, 2022 • edited Loading

tnqn commented May 9, 2022

BenTheElder commented May 9, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented May 10, 2022

k8s-ci-robot commented May 10, 2022

BenTheElder commented May 25, 2022

SiegfredRodriguez commented May 2, 2023

BenTheElder commented May 2, 2023

tnqn commented Mar 11, 2022 •

edited

Loading

aojea Mar 11, 2022 •

edited

Loading

aojea Mar 11, 2022 •

edited

Loading

tnqn Mar 11, 2022 •

edited

Loading

aojea Mar 17, 2022 •

edited

Loading

tnqn commented Apr 1, 2022 •

edited

Loading

neolit123 commented May 9, 2022 •

edited

Loading

neolit123 commented May 9, 2022 •

edited

Loading

tnqn commented May 9, 2022 •

edited

Loading

neolit123 commented May 9, 2022 •

edited

Loading

aojea commented May 9, 2022 •

edited

Loading