Cannot run jj git push: using ssh_key_from_agent loop

[prestonvanloon]

Description

Steps to Reproduce the Problem

1. jj git push

Not sure how I can provide a repro example. Just started using jj yesterday on an existing git repo.

Expected Behavior

It pushes

Actual Behavior

jj git push -v
2023-08-03T20:04:20.124904Z  INFO jj_cli::cli_util: verbose logging enabled
Branch changes to push to origin:
  Move branch assertion-api from eba925600601 to 2e9b7b55bbeb
2023-08-03T20:04:20.420168Z  INFO run_command: jj_lib::git: using ssh_key_from_agent username="git"
2023-08-03T20:04:20.443888Z  INFO run_command: jj_lib::git: using ssh_key_from_agent username="git"
2023-08-03T20:04:20.467704Z  INFO run_command: jj_lib::git: using ssh_key_from_agent username="git"
... 

The log statement INFO run_command: jj_lib::git: using ssh_key_from_agent username="git" is sent repeatedly in an apparent infinite loop.

Specifications

• Platform: linux amd64
• Version: jj 0.8.0-6ee8feea861c95f8246aee9e720ed5db66161006

[martinvonz]

Weird! I wonder if that's libssh2 doing retries for some reason (trying different keys? but you probably don't have that many different keys for it to try). Does ssh -F /dev/null git@<hostname> work and what does it output?

[prestonvanloon]

@martinvonz

ssh -F /dev/null git@github.com                      
PTY allocation request failed on channel 0
Hi prestonvanloon! You've successfully authenticated, but GitHub does not provide shell access.
Connection to github.com closed.

Not sure if this is related or not, but the repository was recently renamed and I still have the old remote url.
The command works for me on other repositories.

[martinvonz]

The ssh output looks as expected.

Not sure if this is related or not, but the repository was recently renamed and I still have the old remote url.
The command works for me on other repositories.

That's curious. Even if it's not related to the repo rename, it's a useful piece of data that it works in other repos. Does it work in a fresh clone of the repo?

[prestonvanloon]

Yes -- It works with a fresh clone of the repo. It also works if I clone it under the old remote url too.

[sullyj3]

same issue on jj 0.8.0-8c9fc9880c1796988f853d055b683f1ab70514b6. In an existing git repo.

[martinvonz]

Does it work in a fresh clone for you too, @sullyj3? I have no idea what might cause it.

[sullyj3]

Yeah, it did work. Bizarre.

[sullyj3]

Still works regardless of whether it was cloned with git clone followed by jj init --git-repo . or jj git clone.

[martinvonz]

Bizarre indeed. I don't know how to troubleshoot it more, so I'm going to rely on you and @prestonvanloon to try to figure out what's different between the old and the new repo. There is no retrying in jj's code anyway. I don't know how best to debug it even if i had access to your repos and environments. Maybe adding println!() statements in jj's code, or maybe libgit2 or libssh2 have some logging you can turn on. Or maybe even Wireshark.

[sullyj3]

If I run into it again I might become sufficiently curious to investigate further.

[jbcdnr]

Hello, I am hitting the same problem on a fresh repo #1979. There was no recent renaming of the repo I am trying to fetch or pull from.

[martinvonz]

I just looked up what libgit2 does and found https://libgit2.org/docs/guides/authentication/#:~:text=The%20callback%20drives%20the%20authentication,which%20authentication%20methods%20are%20available, which includes this:

If the server doesn’t accept the credentials from the callback, it will be called again. The callback drives the authentication loop. It will get called until it returns credentials accepted by the server or it returns an error code.

I suppose that's the loop you're running into. Can someone report what what ssh-add -l says?

[jbcdnr]

In my case:

$ ssh-add -l
The agent has no identities.

[martinvonz]

In my case:

$ ssh-add -l
The agent has no identities.

That would explain why jj can't get a key from it then. If you run ssh-add (and enter your password if it asks for it), does jj git clone work after that?

I don't know how git would find the key. Is your key password protected?

[jbcdnr]

Yes, it works now by adding explicitly the SSH key ! I don't know if git tries all of them or if I had configured which one to use at some point (but I don't think so).

[jbcdnr]

Is your key password protected?

No it is not.

And I don't have anything special in ~/.ssh/config.

Thanks a lot for the help and the great tool. Happy to try some things in my setup (ssh-add -d breaks things again) if you want to investigate.

[martinvonz]

Is your key password protected?

No it is not.

Ah, so maybe what happens is that jj finds $SSH_AUTH_SOCK and uses only ssh-agent credential, while git first tries ssh-agent and then falls back to check if any password-less keys are available (or maybe it does it in the opposite order). We should do the same.

[prestonvanloon]

Another data point: I changed nothing (that i can recall) and today it works.

My ssh-add -l has two identities in it. One of them (the first one in the list) is the identity linked to my github. The second one in the list is not linked to github.

[prestonvanloon]

I'm now experiencing this problem on another repo. Not sure how what further information I can add, except that the problem seems to happen sometimes but not other times.

[martinvonz]

It was actually easy to reproduce this by just unsetting $SSH_AUTH_SOCK and $SSH_AUTH_PID. It looks like we keep trying to use ssh-agent still, even though it fails. I suspect it's a bug in 104f8e1.

@prestonvanloon, did you say you have $SSH_AUTO_SOCK exported (I would think so since ssh-add -l works), and ssh-add -l includes the right key? And jj git fetch still hangs in that shell?

[prestonvanloon]

$SSH_AUTO_SOCK and $SSH_AUTH_PID are not set. It might have been when I commented on Aug 4, but it's not right now (also ssh-add -l also fails today Could not open a connection to your authentication agent.)

[martinvonz]

$SSH_AUTO_SOCK and $SSH_AUTH_PID are not set. It might have been when I commented on Aug 4, but it's not right now (also ssh-add -l also fails today Could not open a connection to your authentication agent.)

Sounds like what I was able to reproduce then. So you should be able to work around the bug by making sure that $SSH_AUTH_SOCK (note that my $SSH_AUTO_SOCK above was a typo!). If ssh-add -l works in your shell and jj git fetch still hangs, let me know (I'd also be happy to hear if you get it to work).

[prestonvanloon]

@martinvonz no worries about the typo. I did printenv | grep SSH and it wasn't there either.

git fetch / pull / push etc works fine without the ssh-agent running with those environment variables. Is it going to be a requirement to have the agent running? I started it with

eval `ssh-agent -s`

and this shows no identities when running ssh-agent -l ssh-add -l. I'm not sure how it was working on Aug 4th, but my expectation is that jj git commands should work if git commands work. What do you think?

[martinvonz]

git fetch / pull / push etc works fine without the ssh-agent running with those environment variables. Is it going to be a requirement to have the agent running?

No, it was just a workaround. The bug is that we repeatedly try the ssh-agent even if it's not there, so we never end up trying the password-less key.

I started it with

eval `ssh-agent -s`

and this shows no identities when running ssh-agent -l.

You'll need to run ssh-add and then ssh-add -l (not ssh-agent -l, but maybe that was also just a typo :)) should include your keys. Once ssh-add -l lists your keys, I would expect jj git fetch to work.

I'll work on a fix for the bug.

[yuja]

I suspect it's a bug in 104f8e1.

Indeed. git2::Cred::ssh_key_from_agent(username) does nothing other than setting up an empty credential object, and _git_ssh_setup_conn() runs request_creds() and _git_ssh_authenticate_session() in a loop. We'll probably need a callback that remembers we've tried agent.

[martinvonz]

We'll probably need a callback that remembers we've tried agent.

Yes, that's what I tried and that seemed to work. I'm thinking I should remove the special case for ssh-agent and have the callback go through a list of methods to attempt. Sounds good?

[yuja]

Yes, that's what I tried and that seemed to work. I'm thinking I should remove the special case for ssh-agent and have the callback go through a list of methods to attempt. Sounds good?

SGTM. As far as I can tell, Remote::download() nor Remote::push() doesn't set up SSH connection more than once, so we can assume the previous credential attempt would be failed.

[martinvonz]

I'm thinking I should remove the special case for ssh-agent and have the callback go through a list of methods to attempt.

That seemed more complicated than I had hoped, so I went with the special case for ssh-agent.

[martinvonz]

This is hopefully fixed with #2008 merged. Can you try again with a new binary built from head? If you're on Linux or Mac, use this to build from source:

cargo install --git https://github.com/martinvonz/jj.git --locked --bin jj jj-cli

Please let me know whether it works.

[prestonvanloon]

Interestingly, cargo install command had a similar issue when ssh agent was not running.

cargo install --git https://github.com/martinvonz/jj.git --locked --bin jj jj-cli

    Updating git repository `https://github.com/martinvonz/jj.git`
error: failed to fetch into: /home/preston/.cargo/git/db/jj-1467e3dd78fc324a

Caused by:
  failed to authenticate when downloading repository: ssh://git@github.com/martinvonz/jj.git

  * attempted ssh-agent authentication, but no usernames succeeded: `git`

  if the git CLI succeeds then `net.git-fetch-with-cli` may help here
  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli

Caused by:
  no authentication methods succeeded

Once installed successfully, I now have an error rather than infinite loop.

jj git fetch   
Error: authentication required but no callback set; class=Ssh (23); code=Auth (-16)
Hint: Jujutsu uses libssh2, which doesn't respect ~/.ssh/config. Does `ssh -F /dev/null` to the host work?

[martinvonz]

Hmm, it's supposed to try with the key after ssh-agent fails. Your key is not password protected, is it? What does jj git fetch -v say? And just to make sure, does regular git fetch work?

[prestonvanloon]

2023-08-08T14:14:22.998837Z  INFO jj_cli::cli_util: verbose logging enabled
2023-08-08T14:14:23.027602Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [], remotes: [] }}:fetch{remote_name="origin" branch_name_globs=None git_settings=GitSettings { auto_local_branch: true }}: jj_lib::git: remote.download
2023-08-08T14:14:23.278809Z  INFO run_command:cmd_git_fetch{args=GitFetchArgs { branch: [], remotes: [] }}:fetch{remote_name="origin" branch_name_globs=None git_settings=GitSettings { auto_local_branch: true }}: jj_lib::git: using ssh_key_from_agent username="git"
2023-08-08T14:14:23.300430Z  INFO run_command:cmd_git_fetch{args=GitFetchArgs { branch: [], remotes: [] }}:fetch{remote_name="origin" branch_name_globs=None git_settings=GitSettings { auto_local_branch: true }}:get_ssh_key{_username="git"}: jj_cli::commands::git: no ssh key found path="/home/preston/.ssh/id_rsa"
2023-08-08T14:14:23.300453Z  INFO run_command:cmd_git_fetch{args=GitFetchArgs { branch: [], remotes: [] }}:fetch{remote_name="origin" branch_name_globs=None git_settings=GitSettings { auto_local_branch: true }}: jj_lib::git: using default
Error: authentication required but no callback set; class=Ssh (23); code=Auth (-16)
Hint: Jujutsu uses libssh2, which doesn't respect ~/.ssh/config. Does `ssh -F /dev/null` to the host work?

I don't use an RSA ssh key. I think it should use /home/preston/.ssh/id_ed25519?

[martinvonz]

I think it should use /home/preston/.ssh/id_ed25519?

Ah, right that's another limitation I had almost forgotten (#440). I'll see what I can do about that too. Closing this now since the hang has been fixed and we do try to use password-less id_rsa keys afterwards now.

[sullyj3]

Ah yeah, I'm also using passwordless id_ed25519.pub with no ssh_agent