Long story short, in case other people also have this problem: I found the solution. I believe it is caused by Armbian's Rock S0 device tree enabling the HS200 high speed mode for the emmc device.  (source)
I guess not all emmc chips reliably support this. To fix it, I had to remove the line in the device tree linked in the source above. Actually this should probably be done with a proper device tree modification, but in my case, just to test, I modified the boot scripts. In /boot/boot.cmd, add these lines near the bottom, but ABOVE the "booti ..." line:
 

fdt rm /mmc@ff490000 mmc-hs200-1_8v
fdt set /mmc@ff490000 max-frequency <0x02faf080>

Then recompile in terminal with:
 

mkimage -C none -A arm -T script -d /boot/boot.cmd /boot/boot.scr

If you update the kernel/armbian distro these script will probably get overwritten, hence why it should preferably done with a custom device tree instead. But in my case I have frozen updates, and either way this will serve as a starting point for others having the same issue.

Actually, I cannot see this troublesome patch on the 6.18 distro source, so maybe this is old news and no longer a problem on modern images anyway. But my image is still on 6.12, so I figured I'd share anyway. 

Basically, download the attached .dts file, then run

sudo armbian-add-overlay rk3308-spi2-spidev.dts

I also edited /boot/armbianEnv.txt and added this line, but I'm unsure if that's actually necessary?

param_spidev_spi_bus=2

This worked on Armbian 24.5.1 Noble, creating /dev/spidev2.0. You can verify that it works using this tool: https://github.com/rm-hull/spidev-test

The dts file is based on this one from radxa, however changed to "fragment@" syntax like other armbian overlays: https://github.com/radxa-pkg/radxa-overlays/blob/main/arch/arm64/boot/dts/rockchip/overlays/rk3308-spi2-spidev.dts

Maybe this will also work on Rock Pi S, but haven't tested.

rk3308-spi2-spidev.dts

Basically, it is caused by the SoC being very sensitive to voltage drops before the USB controller interprets it as the connection being lost. It's far more sensitive than it needs to be, given that my device is bus-powered and so the OS never actually needs to worry about a connection loss. If the connection is lost it is because I have unplugged the cable and so the device would be shut off regardless.

Anyway, the solution lies deep within the RK3308's control registers, where I found the following: Three bits named "B_validsession reference tuning". There's no other explanation on what these bits does in the manual, but it sounded vaguely promising since I had been able to establish that "b-session valid" is actually the name of the status/interrupt that the vbus voltage affects in OTG device mode. Other registers related to the b-session valid status didn't work, however, including filter times, interrupt masks, force high bits, etc. But luckily this one register finally did the trick.

The default value of these three bits are 000. Without any documentation I basically just had to guess what to do, so I tried setting them to 111 instead. And that seemed to work. It significantly lowered the voltage on the vbus pin required to trigger a disconnect signal. My connection was finally stable.

Here's how you can do it yourself. First install memtool via apt, then run: 

memtool mw 0xFF008018 0x1C001C00

Here it is: rk3308-user-leds.dts

What I needed this for is ethernet port LEDs, since the original connector does not include them. To get standard ethernet LED behavior on these newly defined LEDs, you can do this (you need su access):
 

modprobe ledtrig-netdev
echo "netdev" > /sys/class/leds/rock-s0\:orange\:user1/trigger
echo "netdev" > /sys/class/leds/rock-s0\:green\:user2/trigger
echo "1" > /sys/class/leds/rock-s0\:orange\:user1/rx
echo "1" > /sys/class/leds/rock-s0\:green\:user2/link
echo "end0" > /sys/class/leds/rock-s0\:orange\:user1/device_name
echo "end0" > /sys/class/leds/rock-s0\:green\:user2/device_name

(Change end0 to eth0 in the last two lines if you use ubuntu-based image instead of debian-based (minimal).