null_resource.copy-worker Still creating... with Fedora CoreOS

[dghubble]

Fedora CoreOS 33 has stronger crypto defaults (faq) while the Go language (and therefore Terraform plugins) hasn't yet added support for RSA SHA-2 (see golang/go#37278).

Short/medium term, follow the Fedora CoreOS guidance. You can either use a non-RSA key (except Azure) or apply the workaround recommended by Fedora CoreOS as a snippet.

variant: fcos
version: 1.1.0
storage:
  files:
    - path: /etc/ssh/sshd_config.d/10-insecure-rsa-keysig.conf
      mode: 0600
      contents:
        inline: |
          PubkeyAcceptedKeyTypes=+ssh-rsa

This issue is not tracking any planned action on Typhoon's part. But rather anticipating users not seeing the CHANGELOG.

Symptom

You'll notice that provisioning clusters with Fedora CoreOS 33 hangs when trying to us an RSA key, because of the now stronger policies in Fedora.

module.foo.null_resource.copy-worker-secrets[0]: Still creating....
....

Bear in mind, if you are a new user, there are plenty of other reasons your SSH connections to bootstrap machines might not be working (e.g. ssh-agent, see docs).

Related: coreos/fedora-coreos-tracker#699

[dghubble]

Terraform is still tracking this hashicorp/terraform#30134

[adaliszk]

Before opening a new issue, I am wondering if this is why the SSH connection does not even try using a key?

module.cluster.module.typhoon.null_resource.copy-controller-secrets[0]: Still creating... [9m30s elapsed]
2022-04-16T22:06:27.840Z [DEBUG] Connecting to some-node.some-domain.tld:22 for SSH
2022-04-16T22:06:27.862Z [DEBUG] Connection established. Handshaking for user core
2022-04-16T22:06:27.989Z [WARN]  SSH authentication failed (core@some-node.some-domain.tld:22): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain
2022-04-16T22:06:27.989Z [WARN]  retryable error: SSH authentication failed (core@some-node.some-domain.tld:22): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain

To be sure, I've configured my local client to be able to automatically just log in, but that seem to be ignored in this case.

[sedlund]

Before opening a new issue, I am wondering if this is why the SSH connection does not even try using a key?

Terraform cannot (specifically golang/go#49952) produce a valid fingerprint for RSA SHA-2 keys to be seen as valid to the SSH server, so it will not be a valid method to use. I generated and use ed25519 keys in the meantime.

[dghubble]

Go addressed this issue in the crypto library and Terraform v1.2+ releases have the fix. RSA SHA2 keys can be used with Fedora CoreOS again, which removes the need for a workaround (esp on Azure where ed25519 isn't allowed) in my testing. hashicorp/terraform#30134