F33: updated crypto defaults: SSH tooling may fail

[dghubble]

Describe the bug

Provisioning an host with FCOS testing (33.20201201.2.1), manual ssh'ing works (uses ssh-agent), but connections via Terraform automation now fail (also uses ssh-agent).

Repeating with FCOS 32.20201104.3.0 (stable), this problem doesn't appear.

Reproduction steps

resource "null_resource" "example" {

  connection {
    type    = "ssh"
    host    = PUBLIC_IP
    user    = "core"
    timeout = "15m"
  }

  provisioner "remote-exec" {
    inline = [
      "echo 'hello world'",
    ]
  }
}

export TF_LOG=debug
terraform apply

Expected behavior

A clear and concise description of what you expected to happen.

Actual behavior

2020-12-16T01:07:10.498-0800 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/12/16 01:07:10 [WARN] retryable error: SSH authentication failed (core@18.189.192.190:22): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Manually sshing to a how we can observe failing attempts.

Dec 16 08:52:02 ip-10-0-12-25 audit[14168]: USER_ERR pid=14168 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=redacted addr=redacted terminal=ssh res=failed'

System details

• AWS, DO tried
• 33.20201201.2.1 (testing)

Ignition config

{
  "ignition": {
    "config": {
      "replace": {
        "source": null,
        "verification": {}
      }
    },
    "security": {
      "tls": {}
    },
    "timeouts": {},
    "version": "3.0.0"
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "ssh-rsa PUB_KEY_GOES_HERE"
        ]
      }
    ]
  },
  "storage": {},
  "systemd": {}
}

Additional information
Add any other information about the problem here.

[lucab]

Thanks for the report. From the logs, it sounds like sshd is rejecting the key.
Do you maybe have multiple agents running, or multiple keys enrolled in the agent? Does the agent log the fingerprint/keygrip of the offered keys in both cases?
Can you maybe try raising the LogLevelsetting in sshd config to check how the daemon is reacting in both cases?

[dustymabe]

I'm almost certain this is related to the Fedora Change adopting tighter security settings by default. We've encountered this when originally moving to Fedora 33 and worked around it a few times (see details in coreos/coreos-assembler#1772).

For terraform (golang based) you're probably bumping into golang/go#37278 (comment). Golang needs to update their library.

I need to publish a FAQ entry for this we can point people to.

[dghubble]

Yeah, with a ed25519 key the problem doesn't appear, so it looks like this, thanks! For users, I think pivoting SSH keys is sometimes a good bit of work, so if there are stop-gaps that'd be great.

Sounds like the longer term plan is waiting for that Go change to land, getting a Go release out, and then https://github.com/hashicorp/terraform-provider-null updating and releasing a new plugin (responsible for these connections).

[dustymabe]

We're going to keep in line with the Fedora defaults on this one so let's attempt to solve this with documentation: coreos/fedora-coreos-docs#220

[dghubble]

Sounds fair to me. Thanks for the docs on this!

[adriangabura]

@dustymabe Can you please explain why I am able to log in to Fedora CoreOS via ssh application using an ssh-rsa key that is clearly deprecated. I can provision it with Terraform docker but I can't use other forms of Terraform ssh transactions? I mean it either is deprecated or is not, why does it work partially? Thanks

[dustymabe]

Hey @adriangabura - I'm not a crypto expert, but ssh-rsa keys weren't deprecated altogether, just the key signature algorithm. We have a FAQ entry on the topic which covers some of the details: https://docs.fedoraproject.org/en-US/fedora-coreos/faq/#_why_does_ssh_stop_working_after_upgrading_to_fedora_33

[adriangabura]

Hey @adriangabura - I'm not a crypto expert, but ssh-rsa keys weren't deprecated altogether, just the key signature algorithm. We have a FAQ entry on the topic which covers some of the details: https://docs.fedoraproject.org/en-US/fedora-coreos/faq/#_why_does_ssh_stop_working_after_upgrading_to_fedora_33

Oh so SHA2 isn't deprecated then? Hmm but in sshd -T ssh-rsa means rsa-sha1? I saw in sshd -T that rsa-sha2 is available

Good god, this is relate to this dumb issue.

Also this naming scheme gave me trouble ssh-rsa where it's not clear what refers to what, this is like that old adage in programming - name your variables properly

Also I don't understand this
so the format of openssh public key convention is like ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIBmhLUTJiP[and so on]== somename-20060227 but it doesn't specify the type or smth? Is there any way to specify it so it doesn't give trouble to CoreOS? Like rsa-sha2-256 or smth?

Sorry for wasting your time like this, I am just trying to avoid people who will use Hashicorp ecosystem with Fedora CoreOS hours and hours of headaches.

[bgilbert]

ssh-rsa keys are not deprecated. The ssh-rsa handshake on the wire is deprecated because it uses SHA-1 signatures. The replacement, rsa-sha2-256, also uses ssh-rsa keys.

Go's x/crypto/ssh has now been updated to support the new handshake (just one day before GitHub disabled the old one) so building Terraform with the newest version of that package will likely address the issue.

[adriangabura]

Nice thank you for your explanation!