I also ensured the key has proper permissions chmod 600 $HOME/.ssh/id_rsa

Ahh this is due to the lack of an SSH agent running. I suppose we can manually detect id_rsa and include it, but that's a bit odd... hmm

hmm... I am seeing this still happens with I specify an SSH config

coder@coder-dev:~/.ssh$ cat config
~/.ssh/config
Host github.com
    IdentityFile ~/.ssh/id_rsa
    IdentitiesOnly yes

Interesting. Maybe our SSH command isn't properly passing the identity file... I'll test!

Ahh, I see. It's because we explicitly specify our SSH identity, even if one is already passed in. Testing a fix now.

Alright, so I believe I understand the issue.

Our Git SSH feature passes -i to the SSH command which targets the built-in identity, removing the ability for id_rsa to be used.

During implementation, I expected multiple identities could be added, but I'm wrong. Seems like a clean way to handle this is allowing the disabling of gitssh in entirety. @mafredri @johnstcn I'd be curious for your takes here (as resident idiomatic Unix philosophy experts 🤓).

We could use an agent like we do in v1, Golang has an agent server implementation in golang.org/x/crypto/ssh/agent which would work very easily on linux and darwin, and for windows we could use a named pipe listener and the same package I believe

@deansheather I kind of like that idea, but our current implementation has the benefit of working in tandem with SSH agent forwarding, if we used our own agent, the forwarded agent would disable the Coder SSH key. It could be considered acceptable behavior/tradeoff but I suspect it would be unexpected for the user.

@kylecarbs To my knowledge passing -i will only disable the default entries (like ~/.ssh/id_rsa), the user should still be able to specify the identity file manually in the ~/.ssh/config (and/or with additional -i flags). So I'm not exactly sure why this is an issue currently.

The only thing is that the -i flag takes precedence (will be tried first). But usually an SSH server allows up to 6(?) keys to be provided and tried, in the default configuration anyway.

One thing we could do is parse the users SSH config using ssh -G github.com (or whatever hostname we receive). When no specific identity is defined for that host, the output should contain something like:

identityfile ~/.ssh/id_rsa
identityfile ~/.ssh/id_dsa
identityfile ~/.ssh/id_ecdsa
identityfile ~/.ssh/id_ecdsa_sk
identityfile ~/.ssh/id_ed25519
identityfile ~/.ssh/id_ed25519_sk
identityfile ~/.ssh/id_xmss

These are the default values (for me anyway), i.e. user did not change anything. What we could do here is check if any of these files exist -- if yes, add them as -i flags to the SSH command (before ours, just to be safe).

The output is reduced to as many identity files as have been defined for the host, e.g. with a single entry in ~/.ssh/config, it'd be:

identityfile ~/.ssh/id_ed25519_home.pub

I don't know if we should take the value of identitiesonly yes into account (no is default).

PS. If the user wants to to work around the issue right now, one option would be to unset GIT_SSH_COMMAND (they could even put it in their ~/.profile, or similar).

Based on the above, this would be my proposal:

1. Let keys be an empty array
2. Let output be the output from ssh -G [host]
3. On error, set output to the list of default identities (i.e. too old version of OpenSSH)
4. Parse output
5. For every identityfile, if file exists, add to keys
6. Add coder identityfile to keys
7. Run ssh with all keys as -i args

I kind of like that idea, but our current implementation has the benefit of working in tandem with SSH agent forwarding, if we used our own agent, the forwarded agent would disable the Coder SSH key. It could be considered acceptable behavior/tradeoff but I suspect it would be unexpected for the user.

In SSH sessions we could detect agent forwarding and proxy the requests to the original agent and tack on our own keys.

@mafredri's proposal of adding more -i arguments sounds simplest -- we'd want to make sure we have a way to view the command that's being executed though for debugging purposes.

Agent forwarding detection + proxying sounds complicated.

I do think @deansheather's suggestion has merit, but I also agree with @johnstcn, there may be unexpected complications. I once tried to write a multi-agent (I have two locally and sometimes I'd like to forward both) and noticed that some agents allow you to re-use the socket connection between invocations whereas others require you to re-connect to the socket every time you use it. (I abandoned that project though, for other reasons.)

Writing the agent is fairly easy using golang.org/x/crypto/ssh/agent, but there may be side-effects we can't foresee and we may be limiting the functionality of the original agent. Agents could have 10s of keys, do we place ours first? Will a user bring their own ~/.ssh/config that disables the agent? Etc.

I've unassigned myself! This one is up for grabs 😎

Added to the EE milestone since we're working with a group who'd like this.

I can grab this one, I'll go with the simple approach of adding -i arguments for now. We can always go the proxy agent route if we find problems with the simple approach.

It just dawned upon me that there would be one benefit to going the proxy-agent route. It would allow us to enable git commit signing via SSH key without exporting the private key to the filesystem. Might be an alternative approach to #3318.

The user should in theory only need to to update their git config:

git config --global gpg.format ssh
git config --global user.signingkey "$(coder pubkey --export)"

(Note we should implement --export, or remove the noise from coder pubkey command for this to work.)

Since we'd be setting SSH_AUTH_SOCK, this should just work ™️. With this, would there be any reason to access the private key in templates? This may be worth exploring.

• https://docs.github.com/en/authentication/managing-commit-signature-verification/telling-git-about-your-signing-key#telling-git-about-your-ssh-key

My previous comment inspired me to look deeper into the SSH agent (proxy) method. Oh boy.

When it comes to Windows, the SSH agent seems like a minefield. First and foremost, agent forwarding on Windows doesn't seem to work (or at least properly in all scenarios, see PowerShell/Win32-OpenSSH#1865, the issue tracker has many more like it).

When I say a minefield, check out this graph (borrowed from masahide/OmniSSHAgent):

So take-aways here are that:

• Implementing our own SSH agent on Windows is fraught with peril (perhaps we can take learnings from OmniSSHAgent, or find similar libs that abstract it away)
• Our current SSH agent forwarding most likely doesn't work on Windows at all (I tested Windows 10, OpenSSH_for_Windows_8.1p1, didn't work), or perhaps it might in an environment where unix sockets are supported (although my understanding is that Windows OpenSSH is incompatible with std OpenSSH agent)
• More research (and testing a potential implementation in various settings) is needed

This makes me confident we should go ahead with the -i approach for now.