why can't I connect to my ssh server UNLESS I enter eval "$(ssh-agent -s)" first?

dysprosium@lemmy.dbzer0.com · edit-2 3 months ago

why can't I connect to my ssh server UNLESS I enter eval "$(ssh-agent -s)" first?

INeedMana@lemmy.world · 3 months ago

The whole point of ssh-agent is to remember your passphrase. If you don’t want to do that your problem might be that for some reason ssh client doesn’t pick up your key. Try defining it for the host

Also, there’s -v flag for ssh. Use it to debug what’s going on when it doesn’t try to use your key

kolorafa@lemmy.world · 3 months ago

The only reason ssh client would “hang” without any output is when it’s waiting for external key storage to allow access. It’s designed that way to give user some time to approve access to key storage.

It sometimes happen that the installed key storage is broken in a way that it fails to show user modal, for any reason (showing on wrong screen, wrong desktop, wrong activity, wrong framebuffer, …)

One solution (that you already did) is to change the SSH agent env variable to point to different key storage.

Another would be (if possible) to uninstall the broken key storage if you don’t use it. But it is sometimes needed/used by other apps.

It’s overall good to notify/open bug on your distro issue tracker to notify that some packages are missconfigured (maybe have missing dependencies) or conflicts with other ones.

gedhrel@lemmy.world · 3 months ago

Without the ssh-agent invocation:

what does ssh-add -L show?
what is the original SSH_AUTH_SOCK value?
what is listening to that? (Use lsof)

This kind of stuff often happens because there’s a ton of terrible advice online about managing ssh-agent - make sure there’s none if that baked into your shellrc.

dysprosium@lemmy.dbzer0.com · edit-2 3 months ago

ssh-add -L
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqS5l(redacted)f0phb8x+fUV1w== username@computername

echo $SSH_AUTH_SOCK
/run/user/1000/gcr/ssh

lsof $SSH_AUTH_SOCK
COMMAND      PID    USER FD   TYPE             DEVICE SIZE/OFF    NODE NAME
gcr-ssh-a 778406 username  3u  unix 0x000000007e25ee6b      0t0   30290 /run/user/1000/gcr/ssh type=STREAM (LISTEN)
gcr-ssh-a 778406 username  6u  unix 0x0000000020f5b559      0t0 2096642 /run/user/1000/gcr/ssh type=STREAM (CONNECTED)
gcr-ssh-a 778406 username 12u  unix 0x00000000a6756d60      0t0 2100347 /run/user/1000/gcr/ssh type=STREAM (CONNECTED)
gcr-ssh-a 778406 username 15u  unix 0x00000000625cb05a      0t0 2261237 /run/user/1000/gcr/ssh type=STREAM (CONNECTED)
gcr-ssh-a 778406 username 21u  unix 0x00000000d0b214f9      0t0 2261238 /run/user/1000/gcr/ssh type=STREAM (CONNECTED)
gcr-ssh-a 778406 username 23u  unix 0x00000000a2f197fe      0t0 2349665 /run/user/1000/gcr/ssh type=STREAM (CONNECTED)
gcr-ssh-a 778406 username 27u  unix 0x00000000da22a130      0t0 2349668 /run/user/1000/gcr/ssh type=STREAM (CONNECTED)
gcr-ssh-a 778406 username 29u  unix 0x000000004f7a1723      0t0 2365382 /run/user/1000/gcr/ssh type=STREAM (CONNECTED)
gcr-ssh-a 778406 username 33u  unix 0x00000000e26976b3      0t0 2365389 /run/user/1000/gcr/ssh type=STREAM (CONNECTED)
gcr-ssh-a 778406 username 35u  unix 0x00000000b8185a8a      0t0 2375648 /run/user/1000/gcr/ssh type=STREAM (CONNECTED)
gcr-ssh-a 778406 username 39u  unix 0x00000000ba41030c      0t0 2375649 /run/user/1000/gcr/ssh type=STREAM (CONNECTED)
gcr-ssh-a 778406 username 41u  unix 0x000000006867cb01      0t0 2380999 /run/user/1000/gcr/ssh type=STREAM (CONNECTED)
gcr-ssh-a 778406 username 45u  unix 0x0000000091384b95      0t0 2381008 /run/user/1000/gcr/ssh type=STREAM (CONNECTED)
gcr-ssh-a 778406 username 47u  unix 0x00000000d5b28b08      0t0 3729149 /run/user/1000/gcr/ssh type=STREAM (CONNECTED)
gcr-ssh-a 778406 username 51u  unix 0x00000000f65088aa      0t0 3731006 /run/user/1000/gcr/ssh type=STREAM (CONNECTED)

All before issuing the ssh-agent

gedhrel@lemmy.world · 3 months ago

It’s the gnome key ring ssh agent.

It’s possible that this has popped up a window asking gor permission / a passphrase / something and you’re not seeing that.

gedhrel@lemmy.world · 3 months ago

Okay, that agent process is running but it looks wedged: multiple connections to the socket seem to be opened, probably your other attempts to use ssh.

The ssh-add output looks like it’s responding a bit, however.

I’d use your package manager to work out what owns it and go looking for open bugs in the tool.

(Getting a trace of that process itself would be handy, while you’re trying again. There may be a clue in its behaviour.)

The server reaponse seems like the handshake process is close to completing. It’s not immediately clear what’s up there I’m afraid.

gedhrel@lemmy.world · 3 months ago

Is this problem a recurring one after a reboot?

If it is it warrants more effort.

If not and you’re happy with rhe lack of closure, you can potentially fix this: kill the old agent (watch out to see if it respawns; if it does and that works, fine). If it doesn’t, you can (a) remove the socket file (b) launch ssh-agent with the righr flag (-a $SSH_AGENT_SOCK iirc) to listen at the same place, then future terminal sessions that inherit the env var will still look in the right place. Unsatisfactory but it’ll get you going again.

dysprosium@lemmy.dbzer0.com · edit-2 3 months ago

reboot makes no difference. A new terminal gives the symptoms from the start.

I think I found a bad workaround. If I add this script to ~/.zshrc (because I’m not using bash but zsh)

SSH_AUTH_SOCK=/tmp/ssh-agent-$USER-socket
export SSH_AUTH_SOCK
if [ ! -S "$SSH_AUTH_SOCK" ]; then
    eval $(ssh-agent -a "$SSH_AUTH_SOCK")
fi

then it works. But I think I’m still using the ssh agent which I actually should not be using. At least it’s asking for the passphrase every time, which is nice. Even in the same terminal after ssh logout.

EDIT: The first two lines do the trick as well:

SSH_AUTH_SOCK=/tmp/ssh-agent-$USER-socket
export SSH_AUTH_SOCK

EDIT: If I change this SSH_AUTH_SOCK to ANYTHING else, it also works. So /run/user/1000/gcr/ssh does not work. I gave ample permission to this file, so that cannot be the problem. Perhaps BECAUSE this is a file. I think the SSH_AUTH_SOCK should point to a nonexisting file because then it makes temporarily a special file that it needs. Ok I’m just shooting in the dark.

gedhrel@lemmy.world · 3 months ago

Minimise your windows one at a time and check that the gnome keyring hasn’t popped up a dialog box sonewhere behind everything else that’s asking you if it’s okay to proceed.

dysprosium@lemmy.dbzer0.com · 3 months ago

No unfortunately not… Would’ve been a real pain.

mvirts@lemmy.world · 3 months ago

Have you considered storing your keys unencrypted? In this case ssh doesn’t need the agent or a password.

Yes it’s not as secure, but for me it’s good enough considering my systems at home are not doing anything important. If you have an encrypted home partition it’s just as secure when your partition is unmounted.

elmicha@feddit.org · 3 months ago

Search for /run/user/1000/gcr/ssh on the Internet. I’m on my phone and didn’t find the solution, but I’m sure you’ll find it.

dysprosium@lemmy.dbzer0.com · 3 months ago

I searched. When I change this variable (path), it works. So in the startup script for my terminal (~/.zshrc) I added this:

SSH_AUTH_SOCK=/tmp/ssh-agent-$USER-socket
export SSH_AUTH_SOCK

Now it works, but I’m not sure why. Anything BUT /run/user/1000/gcr/ssh works I think

Uberflussig@sh.itjust.works · 3 months ago

Can you post the result of the env command as well? It sounds like your config is very minimal, but the fact that it’s looking for a local Unix socket in the strace output is weird.

dysprosium@lemmy.dbzer0.com · 3 months ago

SYSTEMD_EXEC_PID=3980
SSH_AUTH_SOCK=/run/user/1000/gcr/ssh
SESSION_MANAGER=local/computername:@/tmp/.ICE-unix/3857,unix/computername:/tmp/.ICE-unix/3857
GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/668d083f_5746_4268_beac_1ab8e7e69305
GTK3_MODULES=xapp-gtk3-module
XDG_CURRENT_DESKTOP=GNOME
LANG=en_GB.UTF-8
GRADLE_HOME=/usr/share/java/gradle
LC_IDENTIFICATION=en_GB.UTF-8
DISPLAY=:1
QT_QPA_PLATFORMTHEME=qt5ct
COLORTERM=truecolor
QT_IM_MODULE=ibus
USER=username
LC_MEASUREMENT=en_GB.UTF-8
XDG_MENU_PREFIX=gnome-
HOME=/home/username
PWD=/home/username
DESKTOP_SESSION=gnome
XDG_SESSION_CLASS=user
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
LC_NUMERIC=en_GB.UTF-8
XDG_DATA_DIRS=/home/username/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share/:/usr/share/
WINDOWPATH=2
XDG_SESSION_DESKTOP=gnome
VTE_VERSION=7602
LC_TIME=en_GB.UTF-8
MAIL=/var/spool/mail/username
DEBUGINFOD_URLS=https://debuginfod.archlinux.org 
QT_AUTO_SCREEN_SCALE_FACTOR=1
LC_PAPER=en_GB.UTF-8
LOGNAME=username
MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.SettingsDaemon.MediaKeys.service/memory.pressure
MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=
XMODIFIERS=@im=ibus
GNOME_TERMINAL_SERVICE=:1.870
XDG_SESSION_TYPE=x11
SHELL=/bin/zsh
XDG_RUNTIME_DIR=/run/user/1000
LC_MONETARY=en_GB.UTF-8
EDITOR=/usr/bin/nano
PATH=/home/username/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/var/lib/flatpak/exports/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
USERNAME=username
GNOME_KEYRING_CONTROL=/run/user/1000/keyring
LC_TELEPHONE=en_GB.UTF-8
GDM_LANG=en_GB.UTF-8
LC_NAME=en_GB.UTF-8
MOTD_SHOWN=pam
TERM=xterm-256color
GDMSESSION=gnome
XAUTHORITY=/run/user/1000/gdm/Xauthority
LC_ADDRESS=en_GB.UTF-8
SHLVL=1
OLDPWD=/home/username
LESS_TERMCAP_mb=
LESS_TERMCAP_md=
LESS_TERMCAP_me=
LESS_TERMCAP_se=
LESS_TERMCAP_so=
LESS_TERMCAP_ue=
LESS_TERMCAP_us=
LESS=-R
LS_OPTIONS=--color=auto
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.7z=01;31:*.ace=01;31:*.alz=01;31:*.apk=01;31:*.arc=01;31:*.arj=01;31:*.bz=01;31:*.bz2=01;31:*.cab=01;31:*.cpio=01;31:*.crate=01;31:*.deb=01;31:*.drpm=01;31:*.dwm=01;31:*.dz=01;31:*.ear=01;31:*.egg=01;31:*.esd=01;31:*.gz=01;31:*.jar=01;31:*.lha=01;31:*.lrz=01;31:*.lz=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.lzo=01;31:*.pyz=01;31:*.rar=01;31:*.rpm=01;31:*.rz=01;31:*.sar=01;31:*.swm=01;31:*.t7z=01;31:*.tar=01;31:*.taz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tgz=01;31:*.tlz=01;31:*.txz=01;31:*.tz=01;31:*.tzo=01;31:*.tzst=01;31:*.udeb=01;31:*.war=01;31:*.whl=01;31:*.wim=01;31:*.xz=01;31:*.z=01;31:*.zip=01;31:*.zoo=01;31:*.zst=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:
P9K_SSH=0
_P9K_SSH_TTY=/dev/pts/0
P9K_TTY=old
_P9K_TTY=/dev/pts/0
_=/usr/bin/env

Uberflussig@sh.itjust.works · 3 months ago

What happens if you do:

unset  SSH_AUTH_SOCK

And then try to connect?

dysprosium@lemmy.dbzer0.com · 3 months ago

Hey that works too! Same effect as my previous workaround, that I just posted yesterday.

I do have to repeat this command everytime, so I had to put it into ~/.zshrc so it’s executed beforehand in every new terminal.

It still does feel lile a workaround since it ‘resets’ itself (as I said) with every new terminal.

Uberflussig@sh.itjust.works · 3 months ago

So, this is set somewhere in your config files, I think. Maybe try:

grep -r SSH_AUTH_SOCK /etc
grep -r SSH_AUTH_SOCK ~/.*

Just to see where it’s being set.

flux@lemmy.ml · 3 months ago

As mentioned, -v (or -vv) helps to analyze the situation.

My theory is that you already have something providing ssh agent service, but that process is somehow stuck, and when ssh tries to connect it, it doesn’t respond to the connect, or it accepts the connection but doesn’t actually interact with ssh. Quite possibly ssh doesn’t have a timeout for interacting with ssh-agent.

Using eval $(ssh-agent -s) starts a new ssh agent and replaces the environment variables in question with the new ones, therefore avoiding the use of the stuck process.

If this is the actual problem here, then before running the eval, echo $SSH_AUTH_SOCK would show the path of the existing ssh agent socket. If this is the case, then you can use lsof $SSH_AUTH_SOCK to see what that process is. Quite possibly it’s provided by gnome-keyring-daemon if you’re running Gnome. As to why that process would not be working I don’t have ideas.

Another way to analyze the problem is strace -o logfile -f ssh .. and then check out what is at the end of the logfile. If the theory applies, then it would likely be a connect call for the ssh-agent.

beeng@discuss.tchncs.de · 3 months ago

which ssh

which ssh-agent

flux@lemmy.ml · 3 months ago

I guess it’s worth checking if those names point to the expected binaries, but I also think it would be highly unlikely they would be anything else than just /usr/bin/ssh and /usr/bin/ssh-agent.

dysprosium@lemmy.dbzer0.com · edit-2 3 months ago

I didn’t really follow the former part, but I can give you this:

strace -o logfile -f ssh -p 8322 pi@192.168.2.223 of when I get blank

flux@lemmy.ml · 3 months ago

At the end of the log you find:

822413 connect(4, {sa_family=AF_UNIX, sun_path="/run/user/1000/gcr/ssh"}, 110) = 0
...
822413 read(4,

meaning it’s trying to interact with the ssh-agent, but it (finally) doesn’t give a response.

Use the lsof command to figure out which program is providing the agent service and try to resolve issue that way. If it’s not the OpenSSH ssh-agent, then maybe you can disable its ssh-agent functionality and use real ssh-agent in its place…

My wild guess is that the program might be trying to interactively verify the use of the key from you, but it is not succeeding in doing that for some reason.

dysprosium@lemmy.dbzer0.com · edit-2 3 months ago

I am not sure I “solved” this but when I add this to my startup script for my terminal (~/.zshrc):

SSH_AUTH_SOCK=/tmp/ssh-agent-$USER-socket
export SSH_AUTH_SOCK

it works then. I am not sure I’m still using the ssh agent, but at least it also does not cache my passphrase/private key

flux@lemmy.ml · 3 months ago

Do you have that file? If not, then unset SSH_AUTH_SOCK will work just as well.

If it does exist, then I suppose it has good chances of working correctly :). ssh-add -l will try to use that socket and list your keys in the service (or list nothing if there are no keys, but it would still work without error).