Orange Pi3 USB host controller stuck

Short issue description: sometimes after pulling a USB device out it makes OPi3 USB host controller stuck. One or more USB ports are getting unresponsive (sometimes all four of them) and no messages in dmesg about connection/disconnection. No matter what device is used. 
This thing happens only on the first host controller, the second rear one is functioning and responsive.
It happens more often if we push in and pull out four devices in a short period of time. 
Wherein /sys/kernel/debug/usb/devices is stuck too, we cannot read the info from it and all the utils (vi, cat, etc.) are getting the D state (uninterruptible sleep).
"lsusb" is also not working, but can be killed using ctrl+c.
sys/bus/usb/drivers/usb unbind + bind is not working on the first USB host controller but works fine on the second one. If we try to unbind the first host controller, it becomes stuck too.
The only thing that helps is the device reboot.

We tried to use usbmon module and when this thing happens, we're getting these messages from usbmon:

ffff000004c35500 2548827842 S Ii:1:002:1 -115:2048 1 <
ffff000004c35500 2549083792 C Ii:1:002:1 0:2048 1 = 0e

and so on.

We can bring more usbmon log if it is needed.
These interrupt messages don't stop and keep coming to usbmon.

Our setup is:

Orange Pi3
Armbian 21.05.1
Kernel 5.10.34