Fuses are a good idea, but we're talking about 24V here. Enough to make a spark if you've got a big enough source (e.g. car battery) but in low currents it should be entirely safe to use.
650mA * 24V is 15.5W, which is the same limit as normal 802.3af. If they are not a one-time fuse, I would guess they're a polyfuse, meaning they are heat activated. The same kind of fuses are used to protect USB ports from over current.
However looking at the photos on Amazon, I can't see these fuses anywhere. If you buy this injector I would verify that there are actually fuses installed.
200mA * 24V is only 5W. In my opinion this is cutting it too close, you're very likely to have the fuse burned out at some point due to a current spike (e.g. if you have a power failure and all Orange Pi Zero's restart at the same time). I also think that 3W for 4 Orange Pi units with Ethernet and WiFi is too conservative.
I would recommend using a larger fuse. Something around 1A (24W) will provide both enough head room for all Orange Pi Zero units to run and yet still provide the protection you're seeking.
24W won't generate enough heat to cause a fire, even if there's a direct short.