ferro

You are not alone seeing this problem, check this topic https://forum.armbian.com/topic/31682-poor-network-performance-send-only/?do=findComment&comment=235234 I think your finding points to the same solution mentioned there, where changing tx_delay to 0x35 resolved the speed issue.

Hi, Many thanks for your hard work on the Mesa patches and for adapting the stack to the RK3568 platform. It's great to see it running on the ODROID-M1. Keep it up! 🙂 I managed to get it working on my ODROID-M1 (8 GB) systems running my own Armbian build based on v26.08 and 7.2.0-rc7-bleedingedge-rockchip64 with this iommu patch: (npu-test) fero@odroidtest:~/rk3568-npu$ sudo dmesg | grep -w npu [ 9.743716] rockchip-pm-domain fdd90000.power-management:power-controller: failed to get ack on domain 'npu', val=0x1ee [ 10.753390] platform fde40000.npu: Adding to iommu group 3 [ 14.203903] rocket fde40000.npu: npu-supply enabled at 850000 uV [ 14.204531] rocket fde40000.npu: TEST scmi npu clk: enable=0 set_rate=0 rate=800000000 [ 14.215487] rocket fde40000.npu: raw VERSION=0x00000000 VERSION_NUM=0x00000000 [ 14.216241] rocket fde40000.npu: Rockchip NPU core 0 version: 0 and here are the results from some of your test scripts: (npu-test) fero@odroidtest:~/rk3568-npu$ python tests/suite.py models/layer-c28.tflite INFO: Created TensorFlow Lite XNNPACK delegate for CPU. layer-c28.tflite f0:1/0.09 f128:1/0.22 f255:1/0.06 colgrad:1/0.06 rowgrad:1/0.07 chan:1/0.20 mix:1/0.06 (npu-test) fero@odroidtest:~/rk3568-npu$ python tests/mob.py INFO: Created TensorFlow Lite XNNPACK delegate for CPU. mobilenet_v1_1.0_224_quant.tflite cpu 111ms npu 5.4ms top5 5/5 logit diff max 16 mean 0.03 | npu top5 [412, 742, 736, 886, 912]

Update: I found that the original IOMMU patch (see 1st post) - Armbian PR #9403 was incomplete for my case! I created an updated IOMMU patch based on the original upstream discussion. The important difference is that iommu_data_ops_v2 is forced back to 32-bit DMA addressing and GFP_DMA32 allocations: - .dma_bit_mask = DMA_BIT_MASK(40), - .gfp_flags = 0, + .dma_bit_mask = DMA_BIT_MASK(32), + .gfp_flags = GFP_DMA32, After rebuilding the kernel with this updated patch, the NPU now works correctly with the IOMMU enabled. root@odroidtest:~# dmesg | grep -i iommu [ 0.147669] iommu: Default domain type: Translated [ 0.147794] iommu: DMA domain TLB invalidation policy: strict mode [ 2.495423] platform fde40000.npu: Adding to iommu group 0 [ 2.498138] platform fdea0000.video-codec: Adding to iommu group 1 [ 2.500670] platform fdee0000.video-codec: Adding to iommu group 2 [ 2.503131] platform fe040000.vop: Adding to iommu group 3 root@odroidtest:~# root@odroidtest:~# dmesg | grep -i rkn [ 6.142497] [drm] Initialized rknpu 0.9.8 for fde40000.npu on minor 1 [ 6.143858] RKNPU fde40000.npu: RKNPU: DKMS: /dev/rknpu registered (DMA-BUF import only) [ 6.160736] RKNPU fde40000.npu: RKNPU: SCMI clock 198 MHz [ 6.169001] RKNPU fde40000.npu: RKNPU: devfreq active (SCMI-only, OPP 200-1000 MHz) [ 6.180806] RKNPU fde40000.npu: RKNPU: thermal throttling enabled [ 6.188054] RKNPU fde40000.npu: RKNPU: sram region: [0x00000000fdcc0000, 0x00000000fdccb000), sram size: 0xb000

Hi, As I mentioned earlier, after updating the DTS I was able to get the driver loaded and the NPU detected. I then started testing it using rknn-toolkit2 and a model from rknn_model_zoo. The RKNN runtime was able to load and inspect the model successfully, but the first NPU job submission failed: ./rknn_yolov5_demo model/yolov5s_relu.rknn model/bus.jpg ... rknn_run E RKNN: failed to submit, op id: 1, op name: Conv:/model.0/convsp/Conv ... rknn_run fail! ret=-1 init_yolov5_model fail! ret=-1 After a lot of testing and troubleshooting, I found that the failure appears to be related to the NPU IOMMU path. After disabling the NPU IOMMU in the DTS: iommu@fde4b000 { ... status = "disabled"; ... }; the NPU started working correctly. The demo completed successfully and produced valid object detection results: (base) fero@odroidtest:~/rk356x_linux_aarch64/rknn_yolov5_demo$ ./rknn_yolov5_demo model/yolov5s_relu.rknn model/bus.jpg load lable ./model/coco_80_labels_list.txt model input num: 1, output num: 3 input tensors: index=0, name=images, n_dims=4, dims=[1, 640, 640, 3], n_elems=1228800, size=1228800, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922 output tensors: index=0, name=output0, n_dims=4, dims=[1, 255, 80, 80], n_elems=1632000, size=1632000, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922 index=1, name=286, n_dims=4, dims=[1, 255, 40, 40], n_elems=408000, size=408000, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922 index=2, name=288, n_dims=4, dims=[1, 255, 20, 20], n_elems=102000, size=102000, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922 model is NHWC input fmt model input height=640, width=640, channel=3 origin size=640x640 crop size=640x640 input image: 640 x 640, subsampling: 4:2:0, colorspace: YCbCr, orientation: 1 scale=1.000000 dst_box=(0 0 639 639) allow_slight_change=1 _left_offset=0 _top_offset=0 padding_w=0 padding_h=0 convert image use cpu finish rknn_run person @ (209 243 286 510) 0.880 person @ (479 238 560 526) 0.871 person @ (109 238 231 534) 0.840 bus @ (91 129 555 464) 0.692 person @ (79 353 121 517) 0.301 write_image path: out.png width=640 height=640 channel=3 data=0x3d269000 The remaining issue is understanding why inference fails when the NPU IOMMU is enabled. Nevertheless, the NPU now appears to be fully usable in non-IOMMU mode. I will continue testing with additional RKNN models and workloads while investigating the underlying IOMMU problem.

Hi, Finally, I made some good progress. I updated the DTS by starting with the DTS from Armbian 26.5.1 for current 6.18.x kernels and then merged the missing NPU and Power Domain nodes from the original DTS provided by @m1zfs. While comparing the original DTS with the current Armbian DTS, I also found that one of the phandle references in the NPU power-domain definition needed to be adjusted to match the phandle numbering used in the current Armbian DTS. After fixing that, the system booted successfully and the NPU was detected correctly. I'm now testing it with rknn_toolkit2 to verify that the NPU is fully functional. root@odroidtest:~# uname -a Linux odroidtest 6.18.35-current-rockchip64 #2 SMP PREEMPT Tue Jun 9 10:28:53 UTC 2026 aarch64 GNU/Linux root@odroidtest:~# root@odroidtest:~# lsmod | grep -E '^(rknpu|dma32_heap)\b' rknpu 94208 0 dma32_heap 12288 0 root@odroidtest:~# root@odroidtest:~# ls -l /dev/rknpu /dev/dri/by-path/platform-fde40000.npu-render /dev/dma_heap/system /dev/dma_heap/dma32 crw------- 1 root root 251, 3 Jul 25 21:19 /dev/dma_heap/dma32 lrwxrwxrwx 1 root root 19 Jul 25 21:19 /dev/dma_heap/system -> /dev/dma_heap/dma32 lrwxrwxrwx 1 root root 13 Jul 25 21:19 /dev/dri/by-path/platform-fde40000.npu-render -> ../renderD128 crw------- 1 root root 10, 261 Jul 25 21:19 /dev/rknpu root@odroidtest:~# root@odroidtest:~# dmesg | grep -i rknpu [ 6.108962] [drm] Initialized rknpu 0.9.8 for fde40000.npu on minor 1 [ 6.110169] RKNPU fde40000.npu: RKNPU: DKMS: /dev/rknpu registered (DMA-BUF import only) [ 6.132583] RKNPU fde40000.npu: RKNPU: SCMI clock 198 MHz [ 6.133433] RKNPU fde40000.npu: RKNPU: devfreq active (SCMI-only, OPP 200-1000 MHz) [ 6.134599] RKNPU fde40000.npu: RKNPU: thermal throttling enabled [ 6.139965] RKNPU fde40000.npu: RKNPU: sram region: [0x00000000fdcc0000, 0x00000000fdccb000), sram size: 0xb000 root@odroidtest:~#

Hello, I believe @m1zfs's work on the DKMS modules is available here: https://github.com/vitalijborissow/rk3568-npu-ODROID-M1 I have spent many hours trying to get the NPU working on my ODROID-M1 (8 GB) running Armbian 26.5.1 with current kernel 6.18.x as well as edge kernels 6.19 and 7.1. I also tried building the latest Armbian image from source (July 19 build including U-Boot 2026.07), but so far I have not been able to get the device tree / overlay and DKMS modules from the repository working on any of these combinations. The system always fails to boot. Fortunately, I can capture the boot process through the serial console and see where it fails. The log shows a kernel exception related to enabling the RK3568 NPU power domain: rockchip-pm-domain fdd90000.power-management:power-controller: failed to get ack on domain 'npu', val=0x1ee platform fde40000.npu: Adding to iommu group 3 Internal error: synchronous external abort ... Workqueue: pm genpd_power_off_work_fn ... rockchip_pd_power From the log it appears that the crash occurs before rknpu.ko is loaded. This makes me believe the issue is more likely related to the device tree, overlay, power-domain, regulator, or clock configuration rather than to the DKMS module itself. Has anyone seen a similar issue on recent Armbian kernels, or can suggest a direction for further troubleshooting? I am still willing to spend some time investigating this and would appreciate any ideas or hints.

Hi, I can confirm that the issue is present in Armbian v26.5.1 with current kernel versions 6.18.x, and that it is resolved in the edge kernels 6.19.x and 7.1.x. Armbian v26.5.1 - current kernel 6.18.x - NOT OK (many retrans and ~368 Mb/sec): root@odroidtest:~# grep -a --null-data U-Boot /dev/mtd0ro U-Boot SPL 2026.01_armbian-2026.01-S127a-Pb401-H8652-Vab81-Bd0d2-R448a (May 20 2026 - 22:28:40 +0000) root@odroidtest:~# uname -a Linux odroidtest 6.18.35-current-rockchip64 #2 SMP PREEMPT Tue Jun 9 10:28:53 UTC 2026 aarch64 GNU/Linux root@odroidtest:~# iperf3 -c 192.168.0.1 Connecting to host 192.168.0.1, port 5201 [ 5] local 192.168.0.15 port 42940 connected to 192.168.0.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 43.0 MBytes 360 Mbits/sec 199 26.9 KBytes [ 5] 1.00-2.00 sec 44.9 MBytes 376 Mbits/sec 172 26.9 KBytes [ 5] 2.00-3.00 sec 41.8 MBytes 350 Mbits/sec 188 39.6 KBytes [ 5] 3.00-4.00 sec 45.1 MBytes 379 Mbits/sec 178 33.9 KBytes [ 5] 4.00-5.00 sec 41.9 MBytes 351 Mbits/sec 185 17.0 KBytes [ 5] 5.00-6.00 sec 46.5 MBytes 390 Mbits/sec 168 35.4 KBytes [ 5] 6.00-7.00 sec 42.8 MBytes 359 Mbits/sec 170 24.0 KBytes [ 5] 7.00-8.00 sec 47.0 MBytes 394 Mbits/sec 164 31.1 KBytes [ 5] 8.00-9.00 sec 41.0 MBytes 344 Mbits/sec 178 18.4 KBytes [ 5] 9.00-10.00 sec 44.8 MBytes 375 Mbits/sec 167 15.6 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 439 MBytes 368 Mbits/sec 1769 sender [ 5] 0.00-10.00 sec 438 MBytes 368 Mbits/sec receiver iperf Done. Armbian v26.5.1 - edge kernel 6.19 - OK root@odroidtest:~# grep -a --null-data U-Boot /dev/mtd0ro U-Boot SPL 2026.01_armbian-2026.01-S127a-Pb401-H8652-Vab81-Bd0d2-R448a (May 20 2026 - 22:28:40 +0000) root@odroidtest:~# uname -a Linux odroidtest 6.19.0-edge-rockchip64 #1 SMP PREEMPT Sun Feb 8 21:03:27 UTC 2026 aarch64 GNU/Linux root@odroidtest:~# iperf3 -c 192.168.0.1 Connecting to host 192.168.0.1, port 5201 [ 5] local 192.168.0.15 port 35634 connected to 192.168.0.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 113 MBytes 950 Mbits/sec 0 1.81 MBytes [ 5] 1.00-2.00 sec 111 MBytes 933 Mbits/sec 0 1.81 MBytes [ 5] 2.00-3.00 sec 112 MBytes 944 Mbits/sec 0 1.81 MBytes [ 5] 3.00-4.00 sec 113 MBytes 947 Mbits/sec 0 1.81 MBytes [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 0 1.81 MBytes [ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 0 1.81 MBytes [ 5] 6.00-7.00 sec 113 MBytes 948 Mbits/sec 0 1.81 MBytes [ 5] 7.00-8.00 sec 111 MBytes 934 Mbits/sec 1 929 KBytes [ 5] 8.00-9.00 sec 113 MBytes 948 Mbits/sec 0 929 KBytes [ 5] 9.00-10.00 sec 113 MBytes 944 Mbits/sec 0 929 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 1 sender [ 5] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec receiver iperf Done. Armbian v26.5.1 - edge kernel 7.1 - OK root@odroidtest:~# grep -a --null-data U-Boot /dev/mtd0ro U-Boot SPL 2026.01_armbian-2026.01-S127a-Pb401-H8652-Vab81-Bd0d2-R448a (May 20 2026 - 22:28:40 +0000) root@odroidtest:~# uname -a Linux odroidtest 7.1.0-edge-rockchip64 #1 SMP PREEMPT Sun Jun 14 14:58:38 UTC 2026 aarch64 GNU/Linux root@odroidtest:~# iperf3 -c 192.168.0.1 Connecting to host 192.168.0.1, port 5201 [ 5] local 192.168.0.15 port 37256 connected to 192.168.0.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 114 MBytes 958 Mbits/sec 0 1.72 MBytes [ 5] 1.00-2.00 sec 111 MBytes 933 Mbits/sec 0 1.72 MBytes [ 5] 2.00-3.00 sec 113 MBytes 948 Mbits/sec 0 1.72 MBytes [ 5] 3.00-4.00 sec 112 MBytes 944 Mbits/sec 0 1.72 MBytes [ 5] 4.00-5.00 sec 112 MBytes 936 Mbits/sec 0 1.72 MBytes [ 5] 5.00-6.00 sec 113 MBytes 947 Mbits/sec 0 1.72 MBytes [ 5] 6.00-7.00 sec 112 MBytes 935 Mbits/sec 0 1.72 MBytes [ 5] 7.00-8.00 sec 113 MBytes 945 Mbits/sec 0 1.72 MBytes [ 5] 8.00-9.00 sec 112 MBytes 943 Mbits/sec 0 2.19 MBytes [ 5] 9.00-10.00 sec 112 MBytes 937 Mbits/sec 0 2.19 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver iperf Done.

Hello, After successfully installed and booted Armbian 23.02.2 Jammy from SD card on Odroid M1, I ran armbian-install script with using option "boot from SD, system on SSD" to move system to SSD. The process finished successfully, see the logs from armbianmonitor (taken before reboot): https://paste.armbian.com/folivedane After reboot, system failed to boot automatically from SSD due to error: [ 13.766854] 8644d43f-c78d-4586-ab61-14dfa2e740ea: Can't open blockdev [ 13.774855] systemd[1]: Starting Rule-based Manager for Device Events and Files... [ 13.781860] systemd[1]: media-mmcboot.mount: Mount process exited, code=exited, status=32/n/a [ 13.781889] systemd[1]: media-mmcboot.mount: Failed with result 'exit-code'. [ 13.783141] systemd[1]: Failed to mount /media/mmcboot. [ 13.785392] systemd[1]: Dependency failed for /boot. It asked for root password to go to maintenance mode. I found, boot failed because of wrong entry in /etc/fstab: root@odroidm1:~# cat /etc/fstab # <file system> <mount point> <type> <options> <dump> <pass> tmpfs /tmp tmpfs defaults,nosuid 0 0 8644d43f-c78d-4586-ab61-14dfa2e740ea /media/mmcboot ext4 defaults,noatime,commit=600,errors=remount-ro,x-gvfs-hide 0 1 /media/mmcboot/boot /boot none bind 0 0 UUID=832c2ac5-7666-4725-a85f-c8a015d9995a / ext4 defaults,noatime,commit=600,errors=remount-ro,x-gvfs-hide 0 1 as you can see there is missing "UUID=" before uuid string for /media/mmcboot entry. After I fixed this problem by mounting /media/mmcboot and /boot (bind mount) manually and exited from maintenance mode it booted OK. I collected another armbianmonitor logs after boot, see: https://paste.armbian.com/qohidenowu When the broken line is fixed in the /etc/fstab: UUID=8644d43f-c78d-4586-ab61-14dfa2e740ea /media/mmcboot ext4 defaults,noatime,commit=600,errors=remount-ro,x-gvfs-hide 0 1 system is booting normally. The same issue is present also when I used image Armbian 23.02.2 Bullseye Could you fix this bug, please? I think problem in nand-sata-install script (line 371). Thanks in advance!