Cannot SSH using IPv6 address until I ping it with the correct address

I'm having a weird problem when trying to SSH into a wireless SBC running Armbian (Ubuntu 22.04.01 LTS).

When I try to SSH using the full domain name it doesn't respond:

vmplayer@vm-play:~$ ssh -l potato potato-blue.mostly-harmless.wlan
^C

If I force IPv4 then it'll connect

vmplayer@vm-play:~$ ssh -4 -l potato potato-blue.mostly-harmless.wlan
potato@potato-blue.mostly-harmless.wlan's password:

Doing a lookup reveals both IPv4 and IPv6 addresses:

vmplayer@vm-play:~$ nslookup potato-blue.mostly-harmless.wlan
Server:        127.0.0.53
Address:    127.0.0.53#53

Non-authoritative answer:
Name:    potato-blue.mostly-harmless.wlan
Address: 10.20.30.246
Name:    potato-blue.mostly-harmless.wlan
Address: fdff:9cb4:dff::4bb

Looking at the router, I can see that matches:

Active DHCPv6 Leases
Host         IPv6-Address
vm-play      fdff:9cb4:dff::444
potato-blue  fdff:9cb4:dff::4bb

If I try to SSH using the absolute IPv6 address it doesn't respond:

vmplayer@vm-play:~$ ssh -l potato fdff:9cb4:dff::4bb
^C

Pinging also doesn't respond:

vmplayer@vm-play:~$ ping fdff:9cb4:dff::4bb
PING fdff:9cb4:dff::4bb(fdff:9cb4:dff::4bb) 56 data bytes
^C
--- fdff:9cb4:dff::4bb ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6147ms

Getting into the wireless device and getting the IP address, I'm getting several IPv6 addresses:

potato@potato-blue:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether ba:cd:d0:79:a7:3e brd ff:ff:ff:ff:ff:ff
3: wlx3c3300206d7c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 3c:33:00:20:6d:7c brd ff:ff:ff:ff:ff:ff
    inet 10.20.30.246/24 brd 10.20.30.255 scope global dynamic noprefixroute wlx3c3300206d7c
       valid_lft 43105sec preferred_lft 43105sec
    inet6 fdff:9cb4:dff::4bb/128 scope global noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fdff:9cb4:dff:0:e94:aec2:3e03:6800/64 scope global temporary dynamic
       valid_lft 604707sec preferred_lft 86280sec
    inet6 fdff:9cb4:dff:0:4a0f:3507:b82:c02a/64 scope global mngtmpaddr noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::ff7c:79d3:28c8:2736/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

This is where it gets weird, if I try to SSH using fdff:9cb4:dff:0:e94:aec2:3e03:6800, it doesn't respond:

vmplayer@vm-play:~$ ssh -l potato fdff:9cb4:dff:0:e94:aec2:3e03:6800
^C

If I ping it, it responds:

vmplayer@vm-play:~$ ping fdff:9cb4:dff:0:e94:aec2:3e03:6800
PING fdff:9cb4:dff:0:e94:aec2:3e03:6800(fdff:9cb4:dff:0:e94:aec2:3e03:6800) 56 data bytes
64 bytes from fdff:9cb4:dff:0:e94:aec2:3e03:6800: icmp_seq=1 ttl=64 time=107 ms
64 bytes from fdff:9cb4:dff:0:e94:aec2:3e03:6800: icmp_seq=2 ttl=64 time=1.36 ms
64 bytes from fdff:9cb4:dff:0:e94:aec2:3e03:6800: icmp_seq=3 ttl=64 time=1.13 ms
64 bytes from fdff:9cb4:dff:0:e94:aec2:3e03:6800: icmp_seq=4 ttl=64 time=1.19 ms

If I try to SSH using that address, that works now (but I don't want the IP address added):

vmplayer@vm-play:~$ ssh -l potato fdff:9cb4:dff:0:e94:aec2:3e03:6800
The authenticity of host 'fdff:9cb4:dff:0:e94:aec2:3e03:6800 (fdff:9cb4:dff:0:e94:aec2:3e03:6800)' can't be established.
ECDSA key fingerprint is SHA256:mJSIyTtITjN4muTp58tlfien/QOC7v41V0/qlcBS5MY.
Are you sure you want to continue connecting (yes/no/[fingerprint])? no
Host key verification failed.

Now if I try to SSH using the full domain name, it works:

vmplayer@vm-play:~$ ssh -l potato potato-blue.mostly-harmless.wlan
potato@potato-blue.mostly-harmless.wlan's password:
Last login: Thu Dec  1 00:11:21 2022 from fdff:9cb4:dff:0:fdea:5e03:da6f:3d2e

I have no idea what's happening behind the scenes and I don't know why I can't connect at one point, then I can. Can anyone help or shed some light on this?

Edited December 3, 20223 yr by wphyer
Fix formatting