TrevorH

I have a recently commissioned Radxa Rock-5B that I use as my PXE host. It has a number of iso images that are loopback mounted on directories like /var/www/html/mirrors/Fedora-44. I recently attempted an install of Fedora 44 on a VM and it crashed, complaining that it was unable to download a package from /var/www/html/mirrors/Fedora-44/Packages/p/parted-3.6-14.fc44.x86_64.rpm. I checked the directory on the Rock-5B and it was indeed missing but it was also giving me a lot of output that looked like -r--r--r-- 1 root root 264876 Feb 2 2026 policycoreutils-3.10-1.fc44.x86_64.rpm ?????????? ? ? ? ? ? policycoreutils-newrole-3.10-1.fc44.x86_64.rpm all files from here are the same ??? I checked the sha256sum of the Fedora 44 iso against the one published on the website and it matches. I renamed the file and downloaded a fresh copy, just in case, same size, same sha256sum, same errors. I rebooted the Rock-5B to see if it was a transient problem but it was still broken on reboot. I copied the iso file to a different system running an older kernel and mounted it there and the content was correct and there were no errors and more importantly the parted-3.6 rpm file it complained about was present. I removed linux-image-bleedingedge-rockchip64 and rebooted into Linux rock-5b 6.18.53-current-rockchip64 and the problem was gone. It seems something about loopback mounting iso images is broken in 7.3.0-rc4. I have not tried a more recent bleedingedge version yet.

I don't think 192.168.1.255 is a very good choice of ip address as it will be the broadcast address for the 192.168.1.0/24 subnet. It's very unlikely to be a cause for the problem you have but it's probably something you should change.

I had a similar problem with asymmetric iperf3 speeds on my Rock 5B. I was getting 2.35Gbps up and a few hundred Mbps down. If I tested on a 1Gbps link then it was fine and got about the same speeds in both directions. I believe I've found the cause for my problem (self inflicted it seems!) and that was installing and running irqbalance. If I stop irqbalance and e.g `echo ff | sudo tee /proc/irq/114/smp_affinity` (or disable it and reboot) then I get 2.35/2.29Gbps.

https://www.phoronix.com/news/Linux-7.3-Scheduler

I had not tested 7.3 as it's still in RC status but I have now and it looks like this bug is fixed there. The same openssl command ran for 8m22s and kept all 8 cores at 100% with CONFIG_SCHED_CLUSTER=y. The numbers from the 2 runs now differ by fractions of a %, the highest I found was 0.25% better on the CLUSTER=n older kernel than on 7.3 with CLUSTER=y but 0.25% is measurement error and more importantly I can see that it uses all 8 cores all of the time rather than just orphaning one at random.

So I ran `openssl speed -multi 8 rsa md5 sha256 sha512 aes sha1 camellia des rmd160` on 2 builds of the 6.18.52 kernel, one from the repos and one self-built with CONFIG_SCHED_CLUSTER=n and then ran diff against the final 21 lines of output where the stats are. You can see that the all of the md5 tests and 4 of the 6 sha1 tests are almost the same. These are run at the start of the run and I notice that the distro kernel runs all 8 cores for the first 20-30s so that explains why those first results are more or less the same. After that it starts using only 7 cores and the numbers diverge more widely. I checked a few and mostly there seems to be an 8-11% advantage in favour of the CONFIG_SCHED_CLUSTER=n version (i.e non-distro kernel). So CLUSTER=n results are the lefthand side and CLUSTER=y are on the right. trevor@rock-5b:~$ diff -y -W 190 openssl-speed-nocluster.txt openssl-speed-cluster.txt md5 206646.70k 682082.69k 1627733.76k 2515694.25k 3003411.11k 3047533.23k | md5 207893.48k 681828.80k 1624703.66k 2514480.47k 3003370.15k 3047145.47k sha1 257265.18k 945564.91k 2826501.12k 5732223.32k 8417888.94k 8727172.44k | sha1 257001.97k 944134.21k 2818819.67k 5729986.56k 8258613.03k 7898929.54k rmd160 168775.57k 482578.01k 1047489.96k 1490576.73k 1703146.84k 1721171.97k | rmd160 157569.82k 446793.28k 963480.75k 1360659.80k 1549350.23k 1565089.82k sha256 256593.18k 949375.79k 2850635.69k 5781916.67k 8441995.26k 8740929.54k | sha256 241875.40k 894078.82k 2667497.15k 5343714.37k 7705458.01k 7961350.38k sha512 139462.71k 558478.93k 1124027.14k 1811831.47k 2209688.23k 2246688.77k | sha512 130139.93k 520255.22k 1040071.77k 1665936.38k 2023230.12k 2055602.18k des-cbc 0.00 0.00 0.00 0.00 0.00 0.00 des-cbc 0.00 0.00 0.00 0.00 0.00 0.00 des-ede3 132712.38k 138286.40k 139840.77k 140261.44k 140372.65k 140245.64k | des-ede3 122223.67k 126929.64k 128361.86k 128859.14k 128809.72k 128849.24k aes-128-cbc 2887098.42k 6649536.64k 10139807.06k 11986084.86k 12734630.57k 12803970.39k | aes-128-cbc 2737165.32k 6205807.32k 9220290.30k 10736674.05k 11360862.21k 11410365.80k aes-192-cbc 2751396.09k 5874383.27k 8473588.05k 9655262.55k 10161837.40k 10203824.13k | aes-192-cbc 2609197.06k 5462457.41k 7717889.27k 8693462.87k 9127941.46k 9163740.50k aes-256-cbc 2674905.48k 5340891.67k 7373585.83k 8255187.97k 8579080.19k 8609371.48k | aes-256-cbc 2531961.38k 4973017.60k 6722701.40k 7453166.55k 7728204.46k 7748107.85k camellia-128-cbc 582227.33k 679838.68k 710770.01k 721328.47k 723978.92k 724489.56 | camellia-128-cbc 537508.58k 621512.36k 648128.60k 655762.20k 658434.73k 658625.88 camellia-192-cbc 468416.18k 530513.96k 550038.44k 556270.25k 557826.05k 558000.81 | camellia-192-cbc 433255.91k 484146.24k 500183.47k 506022.57k 507199.49k 507164.29 camellia-256-cbc 467775.22k 530535.47k 550066.77k 556213.59k 556876.29k 557940.74 | camellia-256-cbc 430679.27k 483797.80k 500581.66k 505632.77k 507202.22k 507150.34 sign verify encrypt decrypt sign/s verify/s encr./s decr./s sign verify encrypt decrypt sign/s verify/s encr./s decr./s rsa 512 bits 0.000016s 0.000001s 0.000002s 0.000017s 64304.1 755378.7 645816.8 57525.3 | rsa 512 bits 0.000017s 0.000001s 0.000002s 0.000019s 58896.7 690687.0 594987.3 52892.2 rsa 1024 bits 0.000083s 0.000004s 0.000005s 0.000086s 11990.6 233562.6 218317.7 11679.9 | rsa 1024 bits 0.000092s 0.000005s 0.000005s 0.000094s 10854.4 211381.1 198357.9 10589.4 rsa 2048 bits 0.000571s 0.000016s 0.000016s 0.000574s 1752.4 64301.8 62614.2 1743.0 | rsa 2048 bits 0.000633s 0.000017s 0.000018s 0.000636s 1580.0 57918.6 56515.5 1573.2 rsa 3072 bits 0.001760s 0.000034s 0.000035s 0.001765s 568.1 29361.9 28935.6 566.5 | rsa 3072 bits 0.001957s 0.000038s 0.000038s 0.001959s 511.0 26431.1 26051.8 510.3 rsa 4096 bits 0.003995s 0.000060s 0.000060s 0.004000s 250.3 16738.8 16563.8 250.0 | rsa 4096 bits 0.004442s 0.000066s 0.000067s 0.004449s 225.1 15041.4 14899.8 224.8 rsa 7680 bits 0.031106s 0.000207s 0.000208s 0.031131s 32.1 4839.0 4814.8 32.1 | rsa 7680 bits 0.034616s 0.000230s 0.000231s 0.034496s 28.9 4351.1 4323.8 29.0 rsa 15360 bits 0.189898s 0.000820s 0.000821s 0.184244s 5.3 1219.4 1217.3 5.4 | rsa 15360 bits 0.211541s 0.000911s 0.000917s 0.211495s 4.7 1097.8 1090.9 4.7

Well I've just rebuilt the 6.18.52 current kernel without CONFIG_SCHED_CLUSTER set and have rebooted into it and am now testing with stress --cpu 8. So far it's using all 8 cores but it's not been running for long enough to prove it yet. Will update later once it's run and broken/not/broken. root@rock-5b:~# grep CONFIG_SCHED_CL /boot/config-6.18.52-current-rockchip64 # CONFIG_SCHED_CLASS_EXT is not set # CONFIG_SCHED_CLUSTER is not set Edit: 15 minutes in and it's still using all 8 cores at 100% so this does seem to be a bug in the CONFIG_SCHED_CLUSTER code path. It's never lasted for as long as this without pinning processes to specific cores. Edit2: 40 minutes into stress --cpu 8 and it's still using all 8 cores at 100% so I think that proves there is a bug in the SCHED_CLUSTER stuff. Since stress never ends, I'm going to kill that now as 40 minutes sounds like enough.

I am suspecting this is a bug in the linux kernel scheduler that only gets triggered on machines with an asymmetrical cpu configuration. I'm pretty sure that it will always happen on the Rock 5B as I've had reports from at least 2 other 5B users to say they can reproduce this. It would be interesting to see any results from people running machines with a mix of big:little cores that are not the rk3588 to see if the problem is more widespread (I suspect it is).

armbianmonitor -u -> https://paste.armbian.com/upemosixet

I have an interesting discovery that smells suspiciously like a kernel bug. I find it easiest to see this bug by running btop which shows a historical per core cpu usage graph so it's easy to see when one stops being used. You can see the same thing in ordinary top but it's less easy to see than with btop. I have a Radxa Rock 5B with an RK3588 chip that I have been stress testing to make sure the hardware is OK before I start to use it in anger. As part of that I ran `openssl speed -multi 8` to spin up 8 threads using 100% cpu each. On all other machines that I've ever done that with it runs all X cores at 100% until it runs out of tests or is Ctrl-C'ed. Watching top on the Rock5B shows something very odd happens. When it first starts it correctly seems to use all 8 cores at 100% but after a while, maybe 20s, maybe a couple of minutes, it suddenly just stops using one core, sometimes more than one core. This behaviour is also true if running `stress --cpu 8` which I used to make sure this was not an openssl bug, same symptoms, 8x100% cpu for a while then entire cores go completely idle. I have also run these tests on an Odroid HC1 which uses a similar big:little set of cores (Samsung Exynos 5422) as the Rock 5B - 4 smaller slower low power cores, 4 higher power faster cores. That test was run on plain Debian 13 with a 6.12.107 kernel which is why I ran a test on the Rock 5B using armbian 6.12.58 kernel to see if this was a kernel regression. On the Odroid it just works and needs no special actions to make it use all 8 cores simultaneously. I left it to run for the full 1h+ that the openssl speed run takes to complete. It also works correctly on an rpi5 though I only ran that with 4 threads to match its cores. If I change my test to pin a separate process to individual cores using `for core in {0..7}; do taskset -c $core openssl speed & done` then it quite happily runs all 8 of them on all cores until they finish so this appears to me to show that it's not a hardware problem as all 8 processes run on all 8 cores for the entire hour or so it takes for an openssl speed test to complete. I have tested this with various Armbian kernels starting with 6.18.44 then 7.1.8 and 7.2.5 from the beta channel and also reverting to 6.12.58. All show the same symptoms - it just abandons running tasks on one physical core. I have also tested with higher numbers of threads (9-12) and the problem still exhibits itself but takes longer, the more threads, the longer it takes to happen but it does happen eventually. If I run the same 8 process version of the test but omit the taskset -c $core so the system can schedule the task where it likes then it also shows the idle core problem. I am suspecting a kernel scheduler bug since when I manually pin a process to a core so that each one is effectively dedicated to running just that one task then the problem does not occur. I have asked a couple of other Rock 5B users and they also have the same symptoms. I've also seen the symptoms on mine extend to 2 idle cores. These cores are inactive even when all openssl/stress processes show that they are ready to run in the output of ps fax. I do not think this is temperature related as my Rock 5B was consistently reporting itself at less than 57C during these tests. This is what sar reports for openssl/stress with 8 threads: 21:00:51 CPU %user %nice %system %iowait %steal %idle 21:10:43 all 87.39 0.00 0.11 0.00 0.00 12.50 21:20:51 all 87.39 0.00 0.11 0.00 0.00 12.50 21:30:51 all 87.39 0.00 0.11 0.00 0.00 12.50 21:40:43 all 87.38 0.00 0.11 0.00 0.00 12.50 21:50:51 all 87.39 0.00 0.11 0.00 0.00 12.50 and for the taskset -c $core version of the test: 16:26:20 CPU %user %nice %system %iowait %steal %idle 16:30:20 all 7.72 0.00 0.21 11.54 0.00 80.52 16:40:20 all 99.63 0.00 0.37 0.00 0.00 0.00 16:50:20 all 99.81 0.00 0.19 0.00 0.00 0.00 17:00:22 all 99.82 0.00 0.18 0.00 0.00 0.00 17:10:22 all 99.85 0.00 0.15 0.00 0.00 0.00 17:20:20 all 99.86 0.00 0.14 0.00 0.00 0.00 17:30:20 all 99.52 0.00 0.14 0.04 0.00 0.30 17:40:22 all 99.60 0.00 0.40 0.00 0.00 0.00 17:50:20 all 99.83 0.00 0.17 0.00 0.00 0.00 18:00:20 all 99.83 0.00 0.17 0.00 0.00 0.00 The first sample there will be where it was not running for the entire 10 minute period. I've posted similar on the linux-rockchip mailing list and have received a reply from someone at rock-chips.com asking me to rebuild the kernel with CONFIG_SCHED_CLUSTER=n so I'd really like to try to find the deb-src package for linux-image-edge-rockchip64 so that I can rebuild it with that option (not) set. Edit: this appears to be fixed in the 7.3.0 kernel series, specifically tested -rc3.