Omer Ozgur Cetinoglu

# Orange Pi Zero 3 (1.5GB) Random Crashes — Root Cause Found & Fixed (No Recompilation Required) **Board:** Orange Pi Zero 3 — 1.5GB LPDDR4 variant **SoC:** Allwinner H618 **OS:** Armbian Trixie **Kernels tested:** 6.6.75-legacy, 6.18.33-current, 6.16.8-edge **Status:** ✅ SOLVED — 72+ hours stable uptime confirmed --- ## The Problem My Orange Pi Zero 3 (1.5GB model) was randomly crashing every few hours under all three Armbian kernel branches. The symptoms were: - System freezes with no response (SSH drops, ping still works briefly) - Kernel oops logged: `kswapd0 → ext4_es_scan → paging request fault` - Random segfaults on normal binaries (`sudo`, `curl`, `apt`) - Corrupted library names in memory (e.g. `libGap-Jg.sK.0` instead of `libcrypto.so.3`) - `memtester` passed clean (2 loops, 1GB) - `badblocks` found zero bad blocks - `fsck` found no filesystem errors - Crashes happened even at full idle, with only ~200MB RAM in use The crashes occurred on **all three kernel branches** — legacy, current, and edge — ruling out a kernel-specific bug as the root cause. --- ## Investigation ### What we ruled out (in order) | Suspect | Test | Result | |---|---|---| | SD card physical failure | `badblocks -sv` | 0 bad blocks | | Filesystem corruption | `fsck -f` | Clean | | RAM hardware failure | `memtester 1G x2` | No errors | | eth-optimize.sh (ring buffer 1024, txqueuelen 10000) | Removed from crontab | Extended stability but didn't fix | | Kernel version | Tested 6.6.75, 6.18.33, 6.16.8 | All crashed | | DRAM size detection bug | `dmesg \| grep Memory:` | Consistently showed 1572864K (correct 1.5GB) | ### The kernel oops — the real clue ``` kernel: Unable to handle kernel paging request at virtual address 14ae620800000018 kernel: CPU: 0 PID: 56 Comm: kswapd0 kernel: Call trace: kernel: __es_tree_search.isra.0+0x20/0xa4 kernel: es_reclaim_extents+0x58/0xf0 kernel: ext4_es_scan+0xf0/0x29c kernel: do_shrink_slab+0x174/0x298 kernel: shrink_slab+0xb4/0x2f0 kernel: kswapd+0x18c/0x3b8 ``` The address `14ae620800000018` is in "address between user and kernel address ranges" — a classic corrupted/invalid pointer. This happens when the kernel accesses memory that physically doesn't exist or wraps around. ### DRAM address wraparound — the root cause The H618 1.5GB LPDDR4 variant has a known DRAM address wraparound issue, documented in the linux-sunxi community (notably by **ag123** in Armbian forums and **Jernej Skrabec's** u-boot size scan patches): > *"It is quite possible the addresses wrap around in the 1.5GB LPDDR4 DRAM chips and the 'test' for memory there returns a false positive."* In this specific case, the **size** is correctly detected as 1.5GB — but the **upper address region** (above 1GB physical) exhibits wraparound behavior. When the kernel accesses the upper ~512MB region (via page cache, kswapd reclaim, or slab allocation), writes to those addresses silently corrupt data in the lower region — because the addresses "wrap" back. This explains why: - `memtester` passes (it tests sequentially in a contiguous region, doesn't trigger wraparound patterns) - Size appears correct (1572864K = 1.5GB every boot) - Crashes are random in timing (depends on when page cache/kswapd reaches the upper region) - ALL kernels are affected (this is a u-boot/hardware level issue, not kernel-specific) --- ## The Fix ### Standard approach (requires u-boot recompilation) The proper fix is `CONFIG_DRAM_SUN50I_H616_TRIM_SIZE=y` in u-boot, or Jernej Skrabec's adjusted size scan procedure (columns-first scanning). This correctly detects and excludes the wraparound region, allowing full 1.5GB usage. ### Our approach — kernel `mem=` parameter (no recompilation needed) Since the wraparound occurs in the **upper address region**, we can simply tell the kernel to only use the safe lower 1GB: Edit `/boot/armbianEnv.txt` and add `mem=1024M` to `extraargs`: ```bash sudo nano /boot/armbianEnv.txt ``` Change: ``` extraargs=... existing params ... ``` To: ``` extraargs=mem=1024M ... existing params ... ``` Reboot and verify: ```bash cat /proc/cmdline | grep -o "mem=1024M" cat /proc/iomem | grep "System RAM" free -h ``` Expected output: ``` mem=1024M 40080000-7fffffff : System RAM total used free Mem: 974Mi ... ``` The `iomem` map confirms the kernel's physical RAM range is **exactly** `0x40080000–0x7fffffff` — stopping at the 1GB boundary. The upper wraparound region (`0x80000000` and above) is completely excluded from the kernel's memory map. ### Why this works The wraparound region begins at or above `0x80000000` (1GB physical offset). With `mem=1024M`: - Kernel allocator, page cache, slab, and all processes stay within `0x40000000–0x80000000` - `kswapd` never reclaims pages from the problematic upper region - The corrupted-pointer crash path is never triggered ### Tradeoffs vs. u-boot fix | | mem=1024M | u-boot TRIM fix | |---|---|---| | Requires recompilation | ❌ No | ✅ Yes | | RAM available | ~974MB | ~1.4GB | | Works on all kernels | ✅ Yes | ✅ Yes | | Fixes root cause | Workaround | Proper fix | For SDR servers, home automation, and similar light workloads where ~974MB is more than sufficient (our system uses ~130–200MB), this is a perfectly viable permanent solution. --- ## Results | Metric | Before fix | After fix | |---|---|---| | Uptime | 1–8 hours max | **72+ hours** (and counting) | | Kernel oops | Multiple per session | **Zero** | | Segfaults | Random, on any binary | **Zero** | | `sudo apt update` | Crashed frequently | Runs perfectly | | Memory corruption | Frequent bit-flips | **None observed** | System configuration at time of testing: - **Kernel:** 6.16.8-edge-sunxi64 - **Kernel param:** `mem=1024M` - **CPU governor:** ondemand - **Workload:** OpenWebRX+, SpyServer, RTL-TCP (x2), radiosonde_auto_rx, Xray proxy --- ## How to apply ```bash # Backup first sudo cp /boot/armbianEnv.txt /boot/armbianEnv.txt.bak # Add mem=1024M to extraargs sudo sed -i 's/^extraargs=/extraargs=mem=1024M /' /boot/armbianEnv.txt # Verify grep extraargs /boot/armbianEnv.txt # Reboot sudo reboot ``` After reboot, confirm: ```bash # Should show mem=1024M cat /proc/cmdline | grep -o "mem=1024M" # Should show ~974Mi total free -h # Should show System RAM ending at 0x7fffffff cat /proc/iomem | grep "System RAM" ``` --- ## References - ag123's analysis of 1.5GB wraparound: Armbian Community Forums - Jernej Skrabec's u-boot DRAM size scan fix: [linux-sunxi mailing list](https://www.mail-archive.com/u-boot@lists.denx.de/msg516769.html) - H618 LPDDR4 support patch: [u-boot mailing list](https://lore.kernel.org/all/20231111091000.26744-2-iuncuim@gmail.com/T/) - ag88's 1.5GB u-boot fix (alternative approach): GitHub --- ## Notes This fix was developed and tested on a **1.5GB LPDDR4** Orange Pi Zero 3. The 1GB, 2GB, and 4GB variants use different DRAM configurations and may not be affected by this specific issue. If you have a 1.5GB board and experience similar random crashes, try `mem=1024M` before spending time on kernel debugging or hardware replacement. It may save you days of investigation. **Tested and confirmed working by:** Özgür Çetinoğlu **Location:** Athens, Greece **Setup:** 24/7 SDR server (OpenWebRX+, radiosonde tracking) **Date:** June 2026

After nearly one year of kernel crashes, I finally found the stable configuration for Orange Pi Zero 3 as a 24/7 SDR server. The critical discovery: legacy kernel 6.6.75-sunxi64 is the only stable option. The current kernel crashes within hours under SDR load. Additional fixes: - Blacklist aw859a WiFi driver (causes kernel instability) - Blacklist DVB modules for RTL-SDR - Lock libairspyhfSupport.so with chattr +i (SpyServer/SoapySDR conflict) - Mask systemd-networkd-wait-online - CPU governor: schedutil - usbcore.autosuspend=-1 for USB SDR dongles Running 3 SDRs simultaneously: RTL-SDR Blog V4, Airspy HF+, dedicated ADS-B. 4 antennas via HMC241 GPIO RF switch. 30+ decoders. Full Android control. Complete guide: https://ozcet.github.io/sdr-server-guide