Generated kubeconfig cannot be used (certificate signed by unknown authority)

[eirik-talberg]

Disclaimer: I'm new to Terraform and am just getting started provisioning a small k3s setup at home.

After running apply, the cluster seems to be bootstrapped as expected and seems to be working. However, the output kube_config does not appear to generate a valid config file that allows communication with the cluster itself:

$ kubectl get node
Unable to connect to the server: x509: certificate signed by unknown authority

If I scp the k3s kubeconfig to my machine and replace the IP, it works just as expected and I am able to communicate with the cluster.

I'll attach my main.tf, which is extremely barebones for now, to show my setup.

terraform {
  required_providers {
    
  }
}

module "k3s" {
    source  = "xunleii/k3s/module"
    version = "3.2.0"
    k3s_version = "v1.25.2+k3s1"
    use_sudo = true

    servers = {
        k3s-01 = {
            ip = "<ip>"
            connection = {
                user = "<username>"
                private_key = "${file("/home/<me>/.ssh/id_rsa")}"
    
           }
        }
    }  
}

output "kubeconfig" {
    value = module.k3s.kube_config
    sensitive = true
}

resource "local_sensitive_file" "kubeconfig_file" {
    content = module.k3s.kube_config
    filename = "/home/<me>/.kube/config"
}

Am I missing something? Is this an issue with the k3s version I'm using being newer than tested configurations? I would expect a generated kubeconfig from this module to be useable against the generated cluster.

Any help would be appreciated, this module seems like a great way to bootstrap a cluster without using something like ansible, which tends to not handle state changes as well as terraform it seems.

[xunleii]

Hi @eirik-talberg

First of all, thank you for using this module. I haven't used this module in a while, but indeed, one can easily reproduce this problem by running one of the examples.

On my side, I have this error message (from TF, but I think you probably have the same using kubectl):

Error: Post "https://65.109.132.232:6443/api/v1/namespaces/default/serviceaccounts": x509: certificate is valid for 10.254.1.1, 10.254.1.2, 10.254.1.3, 10.43.0.1, 127.0.0.1, ::1, not 65.109.132.232

However, I can't explain why we have this issue; I can read the certificate used by the Kubernetes API by using openssl, but it seems correct:

$ openssl s_client -connect 65.109.132.232:6443 -showcerts < /dev/null | openssl x509 -text
depth=1 CN = kubernetes-server-ca
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 CN = kubernetes-server-ca
verify return:1
depth=0 O = k3s, CN = k3s
verify return:1
DONE
Certificate:
...
            X509v3 Subject Alternative Name: 
                DNS:k3s-control-plane-0, DNS:k3s-control-plane-1, DNS:k3s-control-plane-2, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:localhost, IP Address:10.254.1.1, IP Address:10.254.1.2, IP Address:10.254.1.3, IP Address:10.43.0.1, IP Address:127.0.0.1, IP Address:65.109.132.232, IP Address:0:0:0:0:0:0:0:1
...

We can see that we have the 65.109.132.232 inside the SANs list.

EDIT: add information after some investigations on my side

---

I found something interesting; the "public" IP does not exist in the SAN list at first. However, if you wait long enough, it will be added and commands using kubectl will work.

To continue this investigation, I need a little more information 😅:

• What type of node are you using for your Kubernetes cluster (powerful server? SBC like RaspberryPi?).
• If you wait at least 5 minutes (or maybe more, depending on your machine), does it work? (using the example with Hetzner cloud, I ran the Terrraform target 3 times and only had the problem the first time)

At least, if it works after waiting some time, it could be "fixed" by modifying how the output kubernetes_ready works.

[FalcoSuessgott]

I can verify that I observed the same issues. First the Public IP was missing in the SANs after some time it seems to be added lol (v1.27.1).

[xunleii]

For people experiencing this problem and using k3s > 1.28, please check k3s-io/k3s#8085.