Ansible not working #26890

Kompromittiert93 · 2023-05-25T11:52:34Z

Kompromittiert93
May 25, 2023

Hi,

I am new to teleport and deployed a single teleport proxy and connected some nodes with the shell script, which is provided in the webif under "Enroll new ressources". I can connect to these nodes in tunnled mode while using tsh, webif or teleport connect.

After that I want to combine Ansible with teleport and found a guide for that already in the docs. But I cannot make a connection. Ansible itself shows this error: fatal: [GM-CHEF-STASH-01.teleport.gg-management.de]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection timed out during banner exchange\r\nConnection to UNKNOWN port 65535 timed out", "unreachable": true}

Then I troubleshoot the problem like described in the docs with the ssh client in verbose mode, I found out, that the teleport proxy tries to connect to the node with port 3022/tcp, so I added "-p 22" to the ssh client (so I think I have to adjust the ssh.cfg which was build by "tsh config" and add the "-p 22)

But in the end it's still not working, because now, I can see in the logs from the teleport proxy, the error message:
failed: ssh: handshake failed: remote host presented a public key, expected a host certificate

I hoped, that the teleport on the nodes will configure the openssh-server by itself and add the CA and Host Cert automatically. So if I can guess, I think I have to do manually the openssh config for the legacy ssh client? Before I can use ansible? Am I right?

Best Regards

Answered by webvictim

Oct 20, 2023

The other thing worth trying if you have uppercase hostnames is to set case_insensitive_routing: true in the auth_service section of your /etc/teleport.yaml config file and restart.

(Teleport Cloud users can tsh login to their cluster as a user with editor privileges, run tctl get cluster_networking_config > cnc.yaml, and add the same case_insensitive_routing: true under spec: and then tctl create -f cnc.yaml)

View full answer

webvictim · 2023-05-26T18:05:14Z

webvictim
May 26, 2023
Collaborator

Which guide did you use? This one? https://goteleport.com/docs/server-access/guides/ansible/

What happens if you run ssh -vv login@GM-CHEF-STASH-01.teleport.gg-management.de from the command prompt?

The ProxyCommand in the output of tsh config does indeed reference port 3022, but don't worry - if the Teleport proxy knows that it has a reverse tunnel to that host, it will automatically use that instead and ignore port 3022. Your error is likely something else, and the output of ssh -vv can help us figure out what that is.

5 replies

Kompromittiert93 May 26, 2023
Author

Yes, this is the right guide.

ssh -vv gives the same error as found in the logs of the teleport proxy: failed: ssh: handshake failed: remote host presented a public key, expected a host certificate.

And the error message is correct, the openssh-Server is not configured with cert auth. There is teleport installed and I did a tsh config > ssh.conf like described in the guide, so normally the tsh ssh proxy should be used if I am right…

If I do not use the parameter -p 22, I get another error message, but I can tell you only next Tuesday (Monday is a holiday).

webvictim May 26, 2023
Collaborator

If your nodes were connected with the shell script, they're running the Teleport ssh_service process (which is why they appear in tsh ls and work with tsh ssh etc). This handles the generation and maintenance of the host certificate, reverse tunnel etc for you.

The way the connection flow should work is:

you run ssh -F ./ssh.cfg login@GM-CHEF-STASH-01.teleport.gg-management.de at the command prompt (or Ansible runs the equivalent for you)
ssh reads ./ssh.cfg (which you wrote with tsh config > ssh.cfg) and knows that to access any host matching the pattern *.teleport.gg-management.de, it needs to first call tsh proxy ssh login@GM-CHEF-STASH-01.teleport.gg-management.de (which is configured by the ProxyCommand in ssh.cfg)
tsh proxy ssh connects to the Teleport proxy, uses your local identity from tsh login (which is also present in your running local ssh-agent - see ssh-add -l) and requests to be connected to GM-CHEF-STASH-01.teleport.gg-management.de via its reverse tunnel, which is established by the running Teleport process on that host
Teleport proxies the connection and SSH is connected directly to the listening ssh_service process on your host
ssh then authenticates to Teleport directly, using the identity from your SSH agent

At this point the connection should be established and you'll have a shell on the remote machine - which either you or Ansible can use to run commands.

Some part of this process is failing, and the output of ssh -vv -F ./ssh.cfg login@GM-CHEF-STASH-01.teleport.gg-management.de should be able to give some good detail. Feel free to share as much of that as you're comfortable with - the only sensitive information in there is the hostnames being connected to, which you've already shared in full as far as I can see.

Kompromittiert93 May 30, 2023
Author

Now, I had time to post the requested output:

ssh -vvv -F ssh.cfg root@GM-CHEF-STASH-01.teleport.gg-management.de
OpenSSH_9.2p1 Debian-2, OpenSSL 3.0.8 7 Feb 2023
debug1: Reading configuration data ssh.cfg
debug1: ssh.cfg line 4: Applying options for *.teleport.gg-management.de
debug1: ssh.cfg line 11: Applying options for *.teleport.gg-management.de
debug1: Executing proxy command: exec "/usr/local/bin/tsh" proxy ssh --cluster=teleport.gg-management.de --proxy=teleport.gg-management.de root@gm-chef-stash-01.teleport.gg-management.de:3022
debug1: identity file /home/kali/.tsh/keys/teleport.gg-management.de/k.*** type 0
debug1: certificate file /home/kali/.tsh/keys/teleport.gg-management.de/k.***-ssh/teleport.gg-management.de-cert.pub type 4
debug1: Local version string SSH-2.0-OpenSSH_9.2p1 Debian-2
ERROR: ssh: subsystem request failed

kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

on teleport proxy side:

May 30 16:39:59 teleport.gg-management.de teleport[821]: 2023-05-30T16:39:59+02:00 WARN [ROUTER]    server lookup failed: using default=gm-chef-stash-01:3022 pid:821.1 proxy/router.go:273
May 30 16:40:29 teleport.gg-management.de teleport[821]: 2023-05-30T16:40:29+02:00 WARN [PROXY]     "Subsystem request proxySubsys(cluster=default/teleport.gg-management.de, host=gm-chef-stash-01, port=3022) failed: Teleport proxy failed to conne
ct to \"node\" agent \"gm-chef-stash-01:3022\" over direct dial:\n\n  dial tcp 192.168.131.33:3022: i/o timeout\n\nThis usually means that the agent is offline or has disconnected. Check the\nagent logs and, if the issue persists, try restarting
it or re-registering it\nwith the cluster.." id:581 local:192.168.135.30:443 login:root remote:10.242.139.3:11220 teleportUser:k.***regular/sshserver.go:1823
May 30 16:40:29 teleport.gg-management.de teleport[821]: 2023-05-30T16:40:29+02:00 ERRO             failure handling SSH "subsystem" request error:[
May 30 16:40:29 teleport.gg-management.de teleport[821]: ERROR REPORT:
May 30 16:40:29 teleport.gg-management.de teleport[821]: Original Error: trace.aggregate Teleport proxy failed to connect to &#34;node&#34; agent &#34;gm-chef-stash-01:3022&#34; over direct dial:
May 30 16:40:29 teleport.gg-management.de teleport[821]:   dial tcp 192.168.131.33:3022: i/o timeout
May 30 16:40:29 teleport.gg-management.de teleport[821]: This usually means that the agent is offline or has disconnected. Check the
May 30 16:40:29 teleport.gg-management.de teleport[821]: agent logs and, if the issue persists, try restarting it or re-registering it
May 30 16:40:29 teleport.gg-management.de teleport[821]: with the cluster.
May 30 16:40:29 teleport.gg-management.de teleport[821]: Stack Trace:
May 30 16:40:29 teleport.gg-management.de teleport[821]:         github.com/gravitational/teleport/lib/reversetunnel/localsite.go:354 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).dialAndForward.func1
May 30 16:40:29 teleport.gg-management.de teleport[821]:         github.com/gravitational/teleport/lib/reversetunnel/localsite.go:363 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).dialAndForward
May 30 16:40:29 teleport.gg-management.de teleport[821]:         github.com/gravitational/teleport/lib/reversetunnel/localsite.go:250 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).Dial
May 30 16:40:29 teleport.gg-management.de teleport[821]:         github.com/gravitational/teleport/lib/proxy/router.go:292 github.com/gravitational/teleport/lib/proxy.(*Router).DialHost
May 30 16:40:29 teleport.gg-management.de teleport[821]:         github.com/gravitational/teleport/lib/srv/regular/proxy.go:270 github.com/gravitational/teleport/lib/srv/regular.(*proxySubsys).proxyToHost
May 30 16:40:29 teleport.gg-management.de teleport[821]:         github.com/gravitational/teleport/lib/srv/regular/proxy.go:237 github.com/gravitational/teleport/lib/srv/regular.(*proxySubsys).Start
May 30 16:40:29 teleport.gg-management.de teleport[821]:         github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1822 github.com/gravitational/teleport/lib/srv/regular.(*Server).handleSubsystem
May 30 16:40:29 teleport.gg-management.de teleport[821]:         github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1567 github.com/gravitational/teleport/lib/srv/regular.(*Server).dispatch
May 30 16:40:29 teleport.gg-management.de teleport[821]:         github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1526 github.com/gravitational/teleport/lib/srv/regular.(*Server).handleSessionRequests
May 30 16:40:29 teleport.gg-management.de teleport[821]:         runtime/asm_amd64.s:1598 runtime.goexit
May 30 16:40:29 teleport.gg-management.de teleport[821]: User Message: Teleport proxy failed to connect to &#34;node&#34; agent &#34;gm-chef-stash-01:3022&#34; over direct dial:
May 30 16:40:29 teleport.gg-management.de teleport[821]:   dial tcp 192.168.131.33:3022: i/o timeout
May 30 16:40:29 teleport.gg-management.de teleport[821]: This usually means that the agent is offline or has disconnected. Check the
May 30 16:40:29 teleport.gg-management.de teleport[821]: agent logs and, if the issue persists, try restarting it or re-registering it
May 30 16:40:29 teleport.gg-management.de teleport[821]: with the cluster.] regular/sshserver.go:2033

Kompromittiert93 May 30, 2023
Author

So as I can see in the sophos firewall, the teleport proxy does not use the reverse tunnel to the node, instead it tries to connect to port 3022/tcp where no service is listening...

webvictim May 31, 2023
Collaborator

I understand. Presumably the hostname that appears in tsh ls is GM-CHEF-STASH-01.teleport.gg-management.de (with the hostname in UPPERCASE)?

(this may be important, because Teleport's internal hostname resolution is case-sensitive as per #3112)

One other thing you can try is changing the ProxyCommand line in your ssh.cfg file to this:

ProxyCommand "/usr/local/bin/tsh" proxy ssh --cluster=teleport.gg-management.de --proxy=teleport.gg-management.de %r@%h.teleport.gg-management.de:%p

(essentially repeat the cluster name again as part of the command - this is necessary due to this bug: #19524)

then running ssh -vvv -F ssh.cfg root@GM-CHEF-STASH-01.teleport.gg-management.de again.

cfeller · 2023-10-17T17:10:43Z

cfeller
Oct 17, 2023

ProxyCommand "/usr/local/bin/tsh" proxy ssh --cluster=teleport.gg-management.de --proxy=teleport.gg-management.de %r@%h.teleport.gg-management.de:%p
(essentially repeat the cluster name again as part of the command - this is necessary due to this bug: #19524)

I was having a similar issue, and this fixed it for me. Thanks so much for this!
(Teleport version 13.3.2)

0 replies

webvictim · 2023-10-20T11:47:32Z

webvictim
Oct 20, 2023
Collaborator

The other thing worth trying if you have uppercase hostnames is to set case_insensitive_routing: true in the auth_service section of your /etc/teleport.yaml config file and restart.

(Teleport Cloud users can tsh login to their cluster as a user with editor privileges, run tctl get cluster_networking_config > cnc.yaml, and add the same case_insensitive_routing: true under spec: and then tctl create -f cnc.yaml)

3 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ansible not working #26890

{{title}}

Replies: 3 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Ansible not working #26890

Kompromittiert93 May 25, 2023

Replies: 3 comments · 8 replies

webvictim May 26, 2023 Collaborator

Kompromittiert93 May 26, 2023 Author

webvictim May 26, 2023 Collaborator

Kompromittiert93 May 30, 2023 Author

Kompromittiert93 May 30, 2023 Author

webvictim May 31, 2023 Collaborator

cfeller Oct 17, 2023

webvictim Oct 20, 2023 Collaborator

davidwebber Sep 17, 2024

davidwebber Sep 17, 2024

webvictim Sep 19, 2024 Collaborator

Kompromittiert93
May 25, 2023

Replies: 3 comments 8 replies

webvictim
May 26, 2023
Collaborator

Kompromittiert93 May 26, 2023
Author

webvictim May 26, 2023
Collaborator

Kompromittiert93 May 30, 2023
Author

Kompromittiert93 May 30, 2023
Author

webvictim May 31, 2023
Collaborator

cfeller
Oct 17, 2023

webvictim
Oct 20, 2023
Collaborator

webvictim Sep 19, 2024
Collaborator