add foreman::shell param #742
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Could you elaborate a bit on what's exactly broken? It works fine here and also in our end-to-end tests, so there must be more difference than just the shell? Regardless of that, I would prefer not to expose it at the |
jhoblitt
force-pushed
the
feature/user-shell
branch
from
c805e62
to
f879cbf
Compare
|
@evgeni It looks either a bug in the openssh client or a configuration problem. The same ssh client config is being used on literally 100s of hosts for years, so I'm leaning towards the former. Which looks like an ssh key problem except that it doesn't get so far as even opening a socket. After turning on sshd debugging on the target host side I finally resorted to trying The ssh client config is fairly typical for an ipa env: $ cat /etc/ssh/ssh_config
# File managed by Puppet
GlobalKnownHostsFile /var/lib/sss/pubconf/known_hosts
PubkeyAuthentication yes
ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h
GSSAPIAuthentication yes
GSSAPIDelegateCredentials yes
Host *
HashKnownHosts yes
SendEnv LANG LC_*and the ssh packages (along with every other centos package) are up to date. $ rpm -qa | grep openssh
openssh-7.4p1-22.el7_9.x86_64
openssh-server-7.4p1-22.el7_9.x86_64
openssh-clients-7.4p1-22.el7_9.x86_64 |
|
Would you mind trying without the IPA related entries? |
|
Yes, that gets ssh working. I may be able to use user match rules to have a different default for the foreman-proxy user... but ssh should not be hanging like that. However, I'm more concerned about the next error in that all rex attempts fail with and this trace in the foreman-proxy log |
|
This looks like a legit bug to me -- @adamruzicka do you think we could fix that in rex_ssh somehow? The usermatch way looks like a rather odd hammer. As for the script busy -- I've seen others reporting that, but no idea. |
|
User match rules in the ssh client config feel a bit like a future landmine... which is why I thought it was reasonable to add a knob to change the |
|
Something else I haven't yet tested is if this same ssh glitch occurs on EL8. I am working on updating foreman 2.4 -> 3.2 as a prelude to migrating from EL7 -> EL8 but my puppet tree isn't yet working on EL8. |
jhoblitt
force-pushed
the
feature/user-shell
branch
from
f879cbf
to
96d8b40
Compare
|
I've updated the PR to use |
jhoblitt
force-pushed
the
feature/user-shell
branch
2 times, most recently
from
91f0f20
to
8c52b3b
Compare
|
Argh. The mysterious "other problem" has gone away after yet another |
|
and after another reboot it is able to run but never actually invokes ssh. So maybe there is some sort of strange state problem going on too... but I think this PR is still needed... |
|
Err... so it looks like the IPA So this PR may not be needed at all unless there is another use case for setting the shell. |
What exactly is "that" in this context? |
|
FWIW openssh 8.8 seems to honor SHELL env variable, so we could try forcing the shell to /bin/sh or something in sp-rex-ssh |
Sorry, context matters, yeah… I am reading @jhoblitt posts in the following way:
So I wonder if we (well, REX) should just force Edit: @jhoblitt does the following config also make it work? |
|
@jhoblitt Could you try this patch for me https://github.com/theforeman/smart_proxy_remote_execution_ssh/compare/master...adamruzicka:force-shell.patch without changing |
|
Will do. |
|
Related to working around this problem, here is support for adding |
|
I gave the patch a try on a test vm, which is running the latest rpm release of 0.5.3. With the patch applied, the service restarted, and the It is safe to say at that the patch does not work with |
|
Restarting REX is working again with the patch installed after restoring the ssh match rules and restarting |
Meaning we don't really need to change the shell for foreman-proxy user? |
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
May 4, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
May 10, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
May 12, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
May 17, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jun 23, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jun 23, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jun 23, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jun 23, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jun 23, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jun 23, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jun 23, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jun 23, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jun 24, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jun 28, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jun 29, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jun 29, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jun 29, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jun 29, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jun 30, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jul 8, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jul 14, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jul 15, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
jhoblitt
added a commit
to lsst-it/lsst-control
that referenced
this pull request
Jul 19, 2022
When the ssh client is configured to for sssd/ipa host key tracking using `/usr/bin/sss_ssh_knownhostsproxy` as a `ProxyCommand`, this somehow breaks foreman-proxy REX. Without a valid shell set in `/etc/passwd` or via the `SHELL` env var, the ssh client errors claiming there is a key exchange failure but never gets as far as opening a network socket. With a valid `SHELL`, ssh sessions are able to be established but it appears that no stdout/stderr is piped back to the foreman-proxy daemon. The only identified work around is remove the `ProxyCommand` completely for the `foreman-proxy` user. There is some discussion of this problem on theforeman/puppet-foreman_proxy#742
|
Making a note here: it appears a similar matter can surface when using
@adamruzicka perhaps we'll need to revisit setting |
|
or disallow its use completely in REX. I don't like either option, but we should address it somehow |
|
Disallowing ProxyJump? :( |
|
There's a case (03276701) where the workaround to make ProxyJump work is |
|
In theory we could do that in rex/ansible and set SHELL to /bin/sh for the newly forked process so that we don't compromise the whole user? |
|
That's what I was thinking of as well. Should we open a new issue so it can be properly tracked? |
|
There was a syntax error in the |
|
Correct. Changing the shell allows the ssh session to connect to the remote host but REX is still broken. The only fix seems to be to remove the proxy command for the foreman-proxy user.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To allow the login shell of the foreman-proxy user to be overridden.
On EL7, after the
smart_proxy_remote_execution_sshgem switched fromNet::SSHto shelling out to runssh, REX is completely broken for me with theforeman-proxyuser's shell set to/bin/false. Changing the shell to/bin/bashallows ssh to function.