iav

@amreo — no HDDs and no PCIe controller at all, both with and without the patch, is a different story: the overlay only changes reset timing on a link that otherwise trains. Could you post dmesg | grep -i pcie and your kernel version?

@BipBip1981 — on the missing cpufreq_dt module: nothing broke. It was switched from module to built-in in Armbian on 2026-02-28 (commit 6868e1c9, "orangepi5: fix slow boot on current kernel"), so rockchip64 current/edge carry CONFIG_CPUFREQ_DT=y. lsmod will never show it, and policy0/policy4 being present means the driver is active. Nothing to restore. More useful is what your own tests show: cpufreq.off=1 is what gets the board through boot, with either overlay or none. That points at CPU DVFS/OPP, not PCIe. The PERST# overlay did apply on your board (gpio-28 (|ep) out hi) and did not change your crashes — so your failure class is not the boot SError I was chasing.

@Boardcon_yang, thanks for the heads-up on the SBC3568 build — the timeline is fine. The branch matters more than the delay: `accel/rocket` is a mainline DRM/accel driver and does not exist in the Rockchip legacy BSP 5.10 tree, so even a clean BSP .debs set won't carry this stack. The README targets Armbian `bleedingedge` rockchip64 7.2-rc7, and the .debs in release v2026.08.22 are from that. A full image build isn't needed either — `./compile.sh kernel BRANCH=edge` with your board config is enough, and it avoids most of the WSL2 friction: loop devices and binfmt only matter for the rootfs/image stages.

That's more than I could test myself — the 70 °C soak and the radio coex sharing an IOMMU group with the NPU are exactly the two angles I have no hardware for; everything I measured was one board at room temperature. I'd be glad to add your results to mine: a Tested-by line in the v2026.08.22 README plus your full report (SHA-256s and traces), linked back to this post. No Mesa PR from me, though — I have no resources for upstreaming. The branch is public under the same licence as Mesa; if someone else wants to carry it upstream I'd be happy, and I'll answer questions on the details, but I won't be driving it.

@ferro, thanks for the run — 7/7 layer probes and v1 at 5.4 ms with top-5 5/5 on a different board and kernel build is exactly the confirmation that was missing. Your iommu patch is the same GFP_DMA32 fix baked into the released .debs (`iommu_data_ops_v2` page tables land above 4 GB on 8 GB boards without it); it now ships in the repo as `kernel-patches/` so nobody has to hunt for it.

Thanks. A run on SBC3568 would help — use release v2026.08.22 + README, we'll compare output sha256s. task_number: vendor rknpu_drv.c sets pc_task_number_bits=12 for RK3568, same as RK3588, and the register confirms it — write 0xfff, read back 0xfff. "Bit-identical" is at a clock pinned via SCMI (800 MHz) — added to the post above. No upstream submission from my side — no resources to work with upstream. The branch is public, anyone is welcome to pick it up.

The NPU on the ODROID-M1 can execute ordinary quantized TFLite models through the standard TFLite delegate mechanism, on a fully open stack: the `accel/rocket` kernel driver (with local fixes) plus the Mesa Teflon delegate. An application loads a regular .tflite file — no model-conversion step, no proprietary runtime. Image classification (mobilenet v1/v2, resnet18) and object detection (yolov8n) work today, and other convolutional networks take the same path. This complements the vendor-stack setup from [the DKMS topic](https://forum.armbian.com/topic/58036-odroid-m1-npu-fully-working-on-armbian-618x/), on the same board. What runs, measured against CPU-only TFLite on the same board: - **mobilenet_v1** (224×224 quant): **5.4 ms vs 117 ms (~22×)** — convolutions, depthwise, the fully-connected final layer and average pooling all on the NPU; top-5 classes match the CPU reference, mean absolute logit difference 0.03, bit-identical results across repeated runs. - **mobilenet_v2**: **6.3 ms vs 76 ms (~12×)** — including all ten fused residual additions; top-5 matches the CPU reference including order. - **resnet18**: **11.2 ms vs 345 ms (~31×)** — end-to-end in a single NPU partition: convolutions, residual adds, max pooling, global average pooling and the fully-connected classifier, the poolings executed by the NPU's dedicated PPU unit. - **yolov8n** (320×320 int8, the plain ultralytics TFLite export): **14.9 ms vs 70 ms (~4.7×)** — backbone, neck and the three detection heads on the NPU in two partitions, including all 57 SiLU activations (fused into the convolutions through the DPU lookup table, as the vendor stack does) and the two nearest-neighbour upsamples (the DPU's unpooling mode); the NPU itself takes 8 ms, the rest is the [1, 84, 2100] DFL tail on the CPU. On bus.jpg the same five detections as the CPU (boxes, classes, scores). - Single-operation probe layers (regular, depthwise, stride-2, the 3-channel first layer, fully-connected, concat, standalone add, SiLU, slice/pad/requantize views, upsample — uint8 and int8, per-tensor and per-channel weights) match the TFLite CPU reference within 1 LSB. - For the classifiers only reshape and softmax stay on the CPU. Timings are at the NPU clock pinned to 800 MHz through SCMI (the MAC array is clocked from the SCMI/PVTPLL rate, not the CRU clock the driver sets; a module parameter selects the rate). "Bit-identical" means: same board, fixed clock, repeated runs; results from other RK3568 boards are welcome. How the hardware is driven: the whole delegated graph is submitted as one job. Each task's command stream is patched to chain into the next, the NPU's PC unit walks the chain in hardware and raises a single interrupt at the end — the same submit model the vendor driver uses. Per-job overhead is ~0.2 ms, so avoiding a round-trip per layer matters: it is most of the difference between these numbers and a naive per-task submit. The NPU cannot address memory above 4 GiB, but no `mem=4G` boot restriction is needed: GFP_DMA32 patches in the IOMMU page-table path and in the driver's buffer allocation keep the full 8 GB usable. This builds on the RK3588 Mesa branch (MR !42134). RK3568 differs: 8-channel feature atomics instead of 16 (strides in 8-byte units), 8×32 KiB CBUF banks, a different weight layout (with compact tails for partial channel slices and kernel groups), depthwise above 32 channels running as 32-channel-group tasks, a packed-RGB first-layer mode, weight streaming for FC-shaped layers, requantization through the DPU BS stream, a feature-prefetch (FEATURE_GRAINS) formula that grows on narrow feature maps, the element-wise unit configuration for residual adds — including the surface-notch addressing that band-split add tasks need — the PPU/PPU_RDMA pooling programming, the DPU lookup table (SiLU; the tables travel with the job and the kernel writes them over MMIO, loading them from the command stream lands only the first one) and the DPU unpooling mode (2× upsample), all derived from byte-level comparison against captured vendor command streams (resnet18 and probe models). Details are in the commit messages. Code and reproduction (step-by-step README, kernel .debs in the release): - https://github.com/iav/rk3568-npu — kernel module, DT overlay, test harness, probe models, README - https://github.com/iav/mesa (branch rk3568-test-session-20260820) - release with the kernel .debs / rocket.ko / dtbo: https://github.com/iav/rk3568-npu/releases/tag/v2026.08.22 References used: MR !42134 (https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/42134), the vendor-stack topic above (incl. the >4 GB IOMMU discussion), kiln (https://github.com/gahingwoo/kiln), rknn-toolkit2 (https://github.com/airockchip/rknn-toolkit2 — only needed for further RE, not for running). Limitations: residual ±1-per-layer rounding drift (the hardware rounding pipeline differs from TFLite's reference requantization); PPU pooling covers kernels ≤ 8 with equal strides and equal input/output quantization, other average poolings fall back to a depthwise-convolution lowering; softmax and the 3D detection tail on the CPU; convolution/depthwise/add/pooling/concat/fully-connected with fused ReLU/ReLU6/SiLU, slice/pad/requantize as addressing, nearest 2× upsample; uint8/int8 activations (per-tensor), per-tensor or per-channel weights; feature maps up to 2047 wide/high. A rare (~2% of fresh processes) one-pixel race between chained tasks shows on tiny probe graphs only; the real networks are bit-identical run to run. The Mesa branch head is ahead of the patch series in the rk3568-npu repo; use the branch for the latest state.

@SymbiosisSystems @BipBip1981 Reproduced a boot panic on my Helios64: `SError` in `rockchip_pcie_rd_conf` (config read of the JMB585 right after link-up), roughly 1 in 10 warm reboots on 6.18.43. The driver code is identical in 6.18.10 / 6.18.43 / 7.1, so this is not version-related. `pcie_rockchip_host.bus_scan_delay=1000` does not help. Likely cause: since 2020 Armbian drops `ep-gpios` from the DT (Kobol patch), so the driver never asserts PERST# to the SATA controller at probe; after a warm reboot the JMB585 sometimes brings the link up but does not answer config requests. Restoring `ep-gpios` via an overlay: 0 panics in 15 reboots vs 3 in 35 without it. Please test if you crash at boot (Symbiosis — exactly this SError; BipBip — the cold-boot panic): 1. `helios64-pcie-perst.dtbo` (attached) → `/boot/overlay-user/` 2. add `user_overlays=helios64-pcie-perst` to `/boot/armbianEnv.txt` 3. reboot; check: `sudo grep ' ep ' /sys/kernel/debug/gpio` → `gpio-28 (ep) out hi`. Caveat: Kobol removed PERST# because on some boards the driver reset broke link training (kernel 5.8, 2020). If your SATA disks disappear with the overlay, remove the `user_overlays` line and report here. This does not address runtime crashes under load — I cannot reproduce those on 6.18.43 or 7.1 (30 min fio on 3 disks, 0 errors). Overlay source: ``` /dts-v1/; /plugin/; &pcie0 { ep-gpios = <&gpio2 28 0>; }; ``` helios64-pcie-perst.dtbo

@BipBip1981 I compared everything that changed between 6.18.10 and 6.18.35+. The device tree (OPP tables) and the Helios64 patches are identical — both your DTB variants apply correctly. So the regression is in kernel code or kernel config, not in the voltage patch. Two suspects: 1. Since 2026-02-28 the cpufreq driver is built into the kernel (it was a module). DVFS is now active from very early boot — including LUKS unlock, when cryptsetup loads the CPU to maximum. 6.18.10 was built before this change; all your bad kernels after it. 2. The window also contains SATA/PCIe changes: 6.18.24 added a DMA quirk for the JMB585 (the Helios64 SATA controller). This could cause SError panics under disk I/O — independent of voltage, which would explain why both DTB variants fail the same way. To tell these apart, could you: 1. Capture a serial log of a crash — USB-C port, 1500000 baud ([Kobol wiki](https://wiki.kobol.io/helios64/usb/)). One log would answer this immediately. 2. On a bad kernel, add `extraargs=cpufreq.off=1` to `/boot/armbianEnv.txt` and reboot. If it becomes stable → suspect 1. 3. On your stable 6.18.10, post the output of `ls /sys/devices/system/cpu/cpufreq/` and `lsmod | grep cpufreq`.

@SymbiosisSystems, Thanks for testing and reporting back. Before diving into the panic itself, a bit of context: the CPU stability overlay we packaged from the prahal/ebin-dev recipe is opt-in precisely because, thanks to the RK3399 silicon lottery, most Helios64 units don't need it. The very fact that you do need it already hints that your particular unit is sitting closer to the edge of its operating margin than a typical one. That's not blaming the hardware — just an observation that units with marginal stability often expose other weak spots that stay invisible on an "average" board. Now to the crash itself. `Asynchronous SError` in `rockchip_pcie_rd_conf` during PCI enumeration is a known bus-level abort on the RK3399 PCIe controller, not on the CPU. Our overlay has nothing to do with CPU cluster voltages here: it only raises `vdd_cpu_b` on opp00..opp06 of the A72 cluster, while PCIe sits on `vdd_logic` and separate PCIe rails which we don't touch. The PCIe x2 Gen2 lane from the RK3399 in Helios64 is wired directly to the soldered-on JMicron JMB585 SATA controller ([Kobol Wiki — SATA](https://wiki.kobol.io/helios64/sata/)), which provides the five SATA ports. So the only possible endpoint where bus scan is currently failing is the JMB585. The typical scenario goes like this: the endpoint replies with CRS (Configuration Request Retry Status) or simply doesn't respond within the completion timeout window → the root complex raises a completion abort → AArch64 sees it as an async SError. This class of bugs on the RK3399 is well documented: - [\[BUG\] rk3399-rockpro64 pcie synchronous external abort](https://lore.kernel.org/linux-pci/CAMdYzYrYHtiEXwiKxwWcKSV7Re6dG4zTvkKtZxvso+fLBRYbPQ@mail.gmail.com/T/) — same class of bus-level abort on the same SoC. - [RFC patch: configurable bus scan delay](https://www.spinics.net/lists/linux-pci/msg103350.html) — the general nature of CRS/timeout problems (the specific HBA delays from that thread don't directly apply to Helios64, since your endpoint is a fixed JMB585, not a swappable card). - [rockchip-linux/kernel#116](https://github.com/rockchip-linux/kernel/issues/116) and [#118](https://github.com/rockchip-linux/kernel/issues/118) — "Some PCIe devices cause Rockchip PCIe controller to crash in bus scan". - The [v5 series fixing the RK3399 PCIe endpoint driver](https://patchwork.ozlabs.org/project/devicetree-bindings/cover/20230418074700.1083505-1-rick.wertenbroek@gmail.com/#3135632) — CRS handling and the Configuration Enable bit. - Armbian has known about RK3399 PCIe quirks for a long time: [commit edb45f9 — "Make PCIe reset optional"](https://github.com/armbian/build/commit/edb45f9acf322fc4b27bf7efba3f14a3a432b617). So your stack trace falls into a family of long-known RK3399 PCIe problems and is unrelated to the voltage overlay. If someone — from the rockchip64 maintainers or from the community — picks up your case, here's what is usually expected to be attached to a report like this, otherwise diagnosis will inevitably hit a wall of missing data: 1. A full serial bootlog via USB-UART (Helios64 → 1500000 baud), both with the overlay and without it (just comment out `overlays=` in `armbianEnv.txt`) — to see whether the picture differs or this is an independent problem. 2. State of the SATA bays: try booting with the bays completely empty (or one drive at a time) — if the PCIe link to the JMB585 comes up cleanly without any SATA load, the problem narrows down to one of the drives rather than the link itself. 3. u-boot and BL31 (ATF) versions — `dmesg | grep -iE "u-boot|psci|bl31"`, plus `strings /boot/u-boot.bin | grep -m1 "U-Boot 20"` if you have access. 4. The full `dmesg` up to the moment of the crash — even if truncated, whatever made it out to serial. 5. The entire contents of `/boot/armbianEnv.txt`. 6. Try two extra kernel boot parameter options — this is the simplest experiment and often helps on unstable RK3399 PCIe. **a) Disable PCIe power management** (a known workaround for the broken PM on RK3399). Open `/boot/armbianEnv.txt`: ``` sudo nano /boot/armbianEnv.txt ``` Find a line starting with `extraargs=`. Two cases: - **If such a line exists** — append the parameters at its end, separated by a space. For example, it was: ``` extraargs=console=tty1 ``` Becomes: ``` extraargs=console=tty1 pcie_aspm=off pcie_port_pm=off ``` - **If there is no such line at all** — just add a new line at the very end of the file: ``` extraargs=pcie_aspm=off pcie_port_pm=off ``` Save (Ctrl+O, Enter, Ctrl+X), `sudo reboot`, try booting. To revert: remove what you appended (or delete the added line entirely) and reboot again. **b)** You can try downgrading PCIe from Gen2 to Gen1 (RK3399 Gen2 link training is regularly unstable), but that requires writing your own device tree overlay — there is no ready-made one in Armbian. And finally, to be honest: we are not PCIe stack experts and not rockchip64

Hi @ebin-dev, @SymbiosisSystems, and everyone in this thread, Following the recurring instability reports here and in the older topic ( https://forum.armbian.com/topic/30074-helios64-armbian-2308-bookworm-issues-solved/ ), I've packaged your opp-microvolt workaround as an opt-in DT overlay in the Armbian build framework. PR: https://github.com/armbian/build/pull/9822 — adds the overlay to both Armbian kernel trees: `rockchip64-current` (6.18) and `rockchip64-edge` (7.0). What you get once this lands: - `rockchip-rk3399-helios64-cpu-stability.dtbo` ships inside the regular `linux-dtb-{current,edge}-rockchip64` package. No hand-patching of DTBs, no separate downloads; `apt upgrade` keeps the overlay in sync with whatever DTB your kernel ships. - **Not enabled by default** — for the people whose Helios64 "just works", the mainline OPPs stay untouched. I don't want to push a tree-wide voltage bump onto every user when only some units exhibit the instability. - Activation is the standard Armbian way, either: armbian-config → System → Kernel → Manage device tree overlays → [*] rk3399-helios64-cpu-stability → save → reboot or manually, by adding the overlay name to the `overlays=` line in `/boot/armbianEnv.txt` (the `rockchip-` prefix is implicit, because `overlay_prefix=rockchip` is already set on this board): overlays=rk3399-helios64-cpu-stability Then reboot. Voltages are exactly the ones from your post in this thread — https://forum.armbian.com/topic/58597-helios64-armbian-trixie-with-linux-618-incl-opp-microvolt-patch/?do=findComment&comment=237456 (opp00..opp06 raised to 900 / 900 / 900 / 950 / 1025 / 1100 / 1175 mV; opp07 left at the mainline 1.20 V; `max` everywhere kept at 1.25 V). Frequencies are not touched. End-to-end verified on my Helios64 with both kernels: - current / 6.18.30, Trixie SD-card image - edge / 7.0.7, Trixie SD-card image (locally built) After enabling the overlay and a reboot: for n in 0 1 2 3 4 5 6 7; do od -An -tx4 --endian=big \ /sys/firmware/devicetree/base/opp-table-1/opp0$n/opp-microvolt done opp00 000dbba0 000dbba0 001312d0 opp01 000dbba0 000dbba0 001312d0 opp02 000dbba0 000dbba0 001312d0 opp03 000e7ef0 000e7ef0 001312d0 opp04 000fa3e8 000fa3e8 001312d0 opp05 0010c8e0 0010c8e0 001312d0 opp06 0011edd8 0011edd8 001312d0 opp07 00124f80 00124f80 001312d0 ...which matches your table 1:1. U-boot log line on boot: Applying kernel provided DT overlay rockchip-rk3399-helios64-cpu-stability.dtbo confirms that u-boot picks up the `.dtbo` from `/boot/dtb/.../rockchip/overlay/` and applies it via `fdt apply` before the kernel starts. Ready-to-flash **current/6.18** images, built from the PR branch by the official Armbian builder workflow: https://fi.mirror.armbian.de/incoming/iav/helios64/archive/ - Armbian_26.5.0_Helios64_resolute_current_6.18.30_minimal.img.xz - Armbian_26.5.0_Helios64_resolute_current_6.18.30_xfce_desktop.img.xz - Armbian_26.5.0_Helios64_trixie_current_6.18.30_minimal.img.xz If any of you can grab one of those (or wait for a nightly after the PR is merged) and confirm the workaround applies cleanly through the overlay path on your board, that would help the PR land. Bug reports are welcome too. Attribution lives in the overlay README block (look for the heading `### rk3399-helios64-cpu-stability`), pointing back to forum topics 30074 and 58597, prahal and ebin-dev: https://github.com/armbian/build/blob/feat/helios64-cpu-stability-overlay/patch/kernel/archive/rockchip64-6.18/overlay/README.rockchip-overlays https://github.com/armbian/build/blob/feat/helios64-cpu-stability-overlay/patch/kernel/archive/rockchip64-7.0/overlay/README.rockchip-overlays Thanks!

Hi @ebin-dev, @SymbiosisSystems, and everyone in this thread, Following the recurring instability reports here and in the older topic 30074, I've packaged your opp-microvolt workaround as an opt-in DT overlay in the Armbian build framework. PR: **armbian/build#9822** — adds the overlay to both Armbian kernel trees: `rockchip64-current` (6.18) and `rockchip64-edge` (7.0). What you get once this lands: - `rockchip-rk3399-helios64-cpu-stability.dtbo` ships inside the regular `linux-dtb-{current,edge}-rockchip64` package. No hand-patching of DTBs, no separate downloads; `apt upgrade` keeps the overlay in sync with whatever DTB your kernel ships. - **Not enabled by default** — for the people whose Helios64 "just works", the mainline OPPs stay untouched. I don't want to push a tree-wide voltage bump onto every user when only some units exhibit the instability. - Activation is the standard Armbian way, either: armbian-config → System → Kernel → Manage device tree overlays → [*] rk3399-helios64-cpu-stability → save → reboot or manually, by adding the overlay name to the `overlays=` line in `/boot/armbianEnv.txt` (the `rockchip-` prefix is implicit, because `overlay_prefix=rockchip` is already set on this board): overlays=rk3399-helios64-cpu-stability Then reboot. Voltages are exactly the ones from post #237456 (opp00..opp06 raised to 900 / 900 / 900 / 950 / 1025 / 1100 / 1175 mV; opp07 left at the mainline 1.20 V; `max` everywhere kept at 1.25 V). Frequencies are not touched. End-to-end verified on my Helios64 with both kernels: - current / 6.18.30, Trixie SD-card image - edge / 7.0.7, Trixie SD-card image (locally built) After enabling the overlay and a reboot: for n in 0 1 2 3 4 5 6 7; do od -An -tx4 --endian=big \ /sys/firmware/devicetree/base/opp-table-1/opp0$n/opp-microvolt done opp00 000dbba0 000dbba0 001312d0 opp01 000dbba0 000dbba0 001312d0 opp02 000dbba0 000dbba0 001312d0 opp03 000e7ef0 000e7ef0 001312d0 opp04 000fa3e8 000fa3e8 001312d0 opp05 0010c8e0 0010c8e0 001312d0 opp06 0011edd8 0011edd8 001312d0 opp07 00124f80 00124f80 001312d0 ...which matches your table 1:1. U-boot log line on boot: Applying kernel provided DT overlay rockchip-rk3399-helios64-cpu-stability.dtbo confirms that u-boot picks up the `.dtbo` from `/boot/dtb/.../rockchip/overlay/` and applies it via `fdt apply` before the kernel starts. Ready-to-flash **current/6.18** images, built from the PR branch by the official Armbian builder workflow: https://fi.mirror.armbian.de/incoming/iav/helios64/archive/ - Armbian_26.5.0_Helios64_resolute_current_6.18.30_minimal.img.xz - Armbian_26.5.0_Helios64_resolute_current_6.18.30_xfce_desktop.img.xz - Armbian_26.5.0_Helios64_trixie_current_6.18.30_minimal.img.xz If any of you can grab one of those (or wait for a nightly after the PR is merged) and confirm the workaround applies cleanly through the overlay path on your board, that would help the PR land. Bug reports are welcome too. Attribution lives in `patch/kernel/archive/rockchip64-{6.18,7.0}/overlay/README.rockchip-overlays` under `### rk3399-helios64-cpu-stability`, pointing back to topics 30074 and 58597, prahal and ebin-dev. Thanks!

Can you give a link to the patches you are talking about? Preferably not on a dropbox or pastebin-like, but to their original source — a repository or mailing list.

Wrote a PR to remedy the situation https://github.com/armbian/build/pull/9800

Thank you. I couldn't even think of such a thing. Lord, why is it necessary to preserve the state of indicators that are needed precisely to show the state of something?