Frequent `rcu: INFO: rcu_sched detected stalls on CPUs/tasks` events logged.

April 1, 20242 yr

This started happening since accepting the kernel when I used apt to upgrade the BananaPi Pro sbc I've got.

armbianmonitor report:

https://paste.armbian.com/izuboducul

@teknoid are you aware of this?

Quote

0

April 14, 20242 yr

I can confirm this issue on an old BananaPi 1 from Lemaker. It began with Kernel 6.6.

After returning to the older 6.1 the stalls disappeared.

Apr 13 23:02:02 eisbaer kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Apr 13 23:02:02 eisbaer kernel: rcu:         0-....: (5249 ticks this GP) idle=56dc/1/0x40000002 softirq=5480265/5480265 fqs=2606
Apr 13 23:02:02 eisbaer kernel: rcu:         (t=5250 jiffies g=9317845 q=1167 ncpus=2)
Apr 13 23:02:02 eisbaer kernel: CPU: 0 PID: 27606 Comm: htop Tainted: G         C         6.6.16-current-sunxi #1
Apr 13 23:02:02 eisbaer kernel: Hardware name: Allwinner sun7i (A20) Family
Apr 13 23:02:02 eisbaer kernel: PC is at stmmac_get_stats64+0x26/0x128
Apr 13 23:02:02 eisbaer kernel: LR is at 0xc2e6b000
Apr 13 23:02:02 eisbaer kernel: pc : [<c078c5ce>]    lr : [<c2e6b000>]    psr: 80010033
Apr 13 23:02:02 eisbaer kernel: sp : f1621c68  ip : c2e68000  fp : 00000001
Apr 13 23:02:02 eisbaer kernel: r10: f1621e68  r9 : c2ee3c48  r8 : c2e68000
Apr 13 23:02:02 eisbaer kernel: r7 : 00000000  r6 : 00000001  r5 : 00000000  r4 : 80000000
Apr 13 23:02:02 eisbaer kernel: r3 : 07c697cd  r2 : c2e6ae48  r1 : f1621d28  r0 : c2e68000
Apr 13 23:02:02 eisbaer kernel: Flags: Nzcv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
Apr 13 23:02:02 eisbaer kernel: Control: 50c5387d  Table: 46a2c06a  DAC: 00000051
Apr 13 23:02:02 eisbaer kernel:  stmmac_get_stats64 from dev_get_stats+0x27/0xd0
Apr 13 23:02:02 eisbaer kernel:  dev_get_stats from dev_seq_printf_stats+0x21/0x124
Apr 13 23:02:02 eisbaer kernel:  dev_seq_printf_stats from dev_seq_show+0x11/0x24
Apr 13 23:02:02 eisbaer kernel:  dev_seq_show from seq_read_iter+0x281/0x35c
Apr 13 23:02:02 eisbaer kernel:  seq_read_iter from seq_read+0x61/0x84
Apr 13 23:02:02 eisbaer kernel:  seq_read from proc_reg_read+0x71/0x90
Apr 13 23:02:02 eisbaer kernel:  proc_reg_read from vfs_read+0x75/0x1e4
Apr 13 23:02:02 eisbaer kernel:  vfs_read from ksys_read+0x45/0x9c
Apr 13 23:02:02 eisbaer kernel:  ksys_read from ret_fast_syscall+0x1/0x5c
Apr 13 23:02:02 eisbaer kernel: Exception stack(0xf1621fa8 to 0xf1621ff0)
Apr 13 23:02:02 eisbaer kernel: 1fa0:                   01141aa0 000005e8 00000004 011c7200 00000400 00000001
Apr 13 23:02:02 eisbaer kernel: 1fc0: 01141aa0 000005e8 b6e13888 00000003 0000000a bedcddd4 00000000 00000000
Apr 13 23:02:02 eisbaer kernel: 1fe0: 00000003 bedcdcd0 b6dae2bb b6d27616

Quote

0

April 18, 20242 yr

Hello!
I have a Banana-Pi M1 (Allwinner A20 SoC), where the same issue occurred since March 22nd, 2024.
I upgraded from 6.1.63-current-sunxi to 6.6.16-current-sunxi at this date.
I could observe in the logs how after some booted time the sysstat-collect services started by systemd took 20-40 seconds to complete.

Normally it takes <100ms to do so, and after some time the cpu stalls completely, and the system becomes unresponsive if not rebooted.

After reboot, everything fime for some hours, then issue occurs again.

Switched to armbian edge (6.7.4-sunxi) kernel, issue persisted.

Switched to armbian legacy (6.1.77-sunxi) kernel, issue disappeared.

I hope the culprit can be found out, i am willing to assist in error reporting, since my system is non-critical.

Thanks for the error report, this way i know i am not the only one!

Quote

0

May 10, 20242 yr

rcu: INFO: rcu_sched self-detected stall on CPU
rcu: 0-....: (5249 ticks this GP) idle=5ec4/1/0x40000002 softirq=471875/471875 fqs=2602
rcu: (t=5250 jiffies g=738401 q=352 ncpus=4)
CPU: 0 PID: 14211 Comm: octoprint Tainted: G C 6.6.16-current-sunxi #1
Hardware name: Allwinner sun8i Family
PC is at stmmac_get_stats64+0x24/0x128
LR is at 0xc32bb000
pc : [<c078c5cc>] lr : [<c32bb000>] psr: a0010033
sp : e1fed990 ip : c32b8000 fp : 00000001
r10: 00000000 r9 : c95dee40 r8 : 00000000
r7 : 00000000 r6 : 00000001 r5 : 00000000 r4 : 80000000
r3 : 0093d4a3 r2 : c32bae48 r1 : c50ba6b8 r0 : c32b8000
Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA Thumb Segment none
Control: 50c5387d Table: 497a006a DAC: 00000051
stmmac_get_stats64 from dev_get_stats+0x27/0xd0
dev_get_stats from rtnl_fill_stats+0x25/0xb0
rtnl_fill_stats from rtnl_fill_ifinfo+0x4cd/0xde0
rtnl_fill_ifinfo from rtnl_dump_ifinfo+0x1ff/0x49c
rtnl_dump_ifinfo from netlink_dump+0xcd/0x270
netlink_dump from __netlink_dump_start+0x15b/0x1ec
__netlink_dump_start from rtnetlink_rcv_msg+0x177/0x25c
rtnetlink_rcv_msg from netlink_rcv_skb+0x75/0xb0
netlink_rcv_skb from netlink_unicast+0x1c1/0x204
netlink_unicast from netlink_sendmsg+0x185/0x354
netlink_sendmsg from __sock_sendmsg+0x27/0x48
__sock_sendmsg from __sys_sendto+0x7f/0xac
__sys_sendto from ret_fast_syscall+0x1/0x5c
Exception stack(0xe1fedfa8 to 0xe1fedff0)
dfa0: aa039794 0000000c 00000029 aa0397b0 00000014 00000000
dfc0: aa039794 0000000c aa03b920 00000122 00000000 00000029 00000014 00000000
dfe0: 00000122 aa038738 b6dd4325 b6d3db06

Same issue here.

I am using Orange pi one.

Problem afer installing 6.6.16-current-sunxi

Huge utylization of CPU / system freze, I also using, mjpg_streamer, I am metionig about this, because I can only make it working/unfreze only after getting feed from camera on that hardware. I know weird, but it is how it is.

Quote

0

May 13, 20242 yr

Just tested and after revert to Linux 6.1.77-legacy-sunxi no problems on my orangepi.

Quote

0

May 17, 20242 yr

I have got the same issue.

Orange Pi Zero H2+

Linux moblink 6.6.16-current-sunxi #1 SMP Fri Feb 23 08:25:28 UTC 2024 armv7l GNU/Linux

Armbian 24.2.1

I used some software which is

1) playing some sound into embedded sound card

2) processing interrupts on gpio (gpio event with GPIO API V2)

During sysstat-collect.service failing for 1min 47.293s:

+sound playing OK

- gpio interrupts proccessing freeze

I had to stop and disable sysstat-collect.service

Also apt remove sysstat

It helped.

Quote

2024-05-17T03:00:02.397369+03:00 moblink systemd[1]: Starting sysstat-collect.service - system activity accounting tool...
2024-05-17T03:00:23.412541+03:00 moblink kernel: [620821.695893] rcu: INFO: rcu_sched self-detected stall on CPU
2024-05-17T03:00:23.412672+03:00 moblink kernel: [620821.695932] rcu:   2-....: (5250 ticks this GP) idle=83fc/1/0x40000002 softirq=2045859/2045859 fqs=2625
2024-05-17T03:00:23.412698+03:00 moblink kernel: [620821.695958] rcu:   (t=5251 jiffies g=6039277 q=138 ncpus=4)
2024-05-17T03:00:23.412796+03:00 moblink kernel: [620821.695977] CPU: 2 PID: 21477 Comm: sadc Not tainted 6.6.16-current-sunxi #1
2024-05-17T03:00:23.412820+03:00 moblink kernel: [620821.695991] Hardware name: Allwinner sun8i Family
2024-05-17T03:00:23.412835+03:00 moblink kernel: [620821.696000] PC is at stmmac_get_stats64+0x26/0x128
2024-05-17T03:00:23.412848+03:00 moblink kernel: [620821.696035] LR is at 0xc3453000
2024-05-17T03:00:23.412860+03:00 moblink kernel: [620821.696046] pc : [<c078c5ce>]    lr : [<c3453000>]    psr: a00f0033
2024-05-17T03:00:23.412873+03:00 moblink kernel: [620821.696057] sp : d1c3dc68 ip : c3450000 fp : 00000001
2024-05-17T03:00:23.412886+03:00 moblink kernel: [620821.696066] r10: d1c3de68 r9 : c4d78108 r8 : c3450000
2024-05-17T03:00:23.412898+03:00 moblink kernel: [620821.696077] r7 : 00000000 r6 : 00000001 r5 : 00000000 r4 : 80000000
2024-05-17T03:00:23.412911+03:00 moblink kernel: [620821.696087] r3 : 0048fc8f r2 : c3452e48 r1 : d1c3dd28 r0 : c3450000
2024-05-17T03:00:23.412924+03:00 moblink kernel: [620821.696099] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA Thumb Segment none
2024-05-17T03:00:23.412936+03:00 moblink kernel: [620821.696114] Control: 50c5387d Table: 458b406a DAC: 00000051
2024-05-17T03:00:23.412948+03:00 moblink kernel: [620821.696134] stmmac_get_stats64 from dev_get_stats+0x27/0xd0
2024-05-17T03:00:23.412964+03:00 moblink kernel: [620821.696165] dev_get_stats from dev_seq_printf_stats+0x21/0x124
2024-05-17T03:00:23.412978+03:00 moblink kernel: [620821.696191] dev_seq_printf_stats from dev_seq_show+0x11/0x24
2024-05-17T03:00:23.412990+03:00 moblink kernel: [620821.696211] dev_seq_show from seq_read_iter+0x281/0x35c
2024-05-17T03:00:23.413003+03:00 moblink kernel: [620821.696240] seq_read_iter from seq_read+0x61/0x84
2024-05-17T03:00:23.413017+03:00 moblink kernel: [620821.696263] seq_read from proc_reg_read+0x71/0x90
2024-05-17T03:00:23.413029+03:00 moblink kernel: [620821.696288] proc_reg_read from vfs_read+0x75/0x1e4
2024-05-17T03:00:23.413043+03:00 moblink kernel: [620821.696312] vfs_read from ksys_read+0x45/0x9c
2024-05-17T03:00:23.413056+03:00 moblink kernel: [620821.696330] ksys_read from ret_fast_syscall+0x1/0x5c
2024-05-17T03:00:23.413068+03:00 moblink kernel: [620821.696350] Exception stack(0xd1c3dfa8 to 0xd1c3dff0)
2024-05-17T03:00:23.413080+03:00 moblink kernel: [620821.696366] dfa0:                   0209b1e8 000005e8 00000003 02093678 00000400 00000001
2024-05-17T03:00:23.413093+03:00 moblink kernel: [620821.696381] dfc0: 0209b1e8 000005e8 b6ed3888 00000003 0000000a be87c314 00000000 00000000
2024-05-17T03:00:23.413105+03:00 moblink kernel: [620821.696394] dfe0: 00000003 be87c268 b6e6e2bb b6de7616
2024-05-17T03:01:10.692536+03:00 moblink kernel: [620868.976559] rcu: INFO: rcu_sched self-detected stall on CPU
2024-05-17T03:01:10.692648+03:00 moblink kernel: [620868.976596] rcu:   3-....: (5250 ticks this GP) idle=0934/1/0x40000002 softirq=2017949/2017949 fqs=2454
2024-05-17T03:01:10.692672+03:00 moblink kernel: [620868.976622] rcu:   (t=5251 jiffies g=6039281 q=215 ncpus=4)
2024-05-17T03:01:10.692689+03:00 moblink kernel: [620868.976640] CPU: 3 PID: 21477 Comm: sadc Not tainted 6.6.16-current-sunxi #1
2024-05-17T03:01:10.692712+03:00 moblink kernel: [620868.976654] Hardware name: Allwinner sun8i Family
2024-05-17T03:01:10.692729+03:00 moblink kernel: [620868.976663] PC is at stmmac_get_stats64+0x26/0x128
2024-05-17T03:01:10.692741+03:00 moblink kernel: [620868.976698] LR is at 0xc3453000
2024-05-17T03:01:10.692753+03:00 moblink kernel: [620868.976709] pc : [<c078c5ce>]    lr : [<c3453000>]    psr: 800f0033
2024-05-17T03:01:10.692766+03:00 moblink kernel: [620868.976719] sp : d1c3dc68 ip : c3450000 fp : 00000001
2024-05-17T03:01:10.692779+03:00 moblink kernel: [620868.976729] r10: d1c3de68 r9 : c4dfb3d8 r8 : c3450000
2024-05-17T03:01:10.692791+03:00 moblink kernel: [620868.976740] r7 : 00000000 r6 : 00000001 r5 : 00000000 r4 : 80000000
2024-05-17T03:01:10.692802+03:00 moblink kernel: [620868.976750] r3 : 0048fc95 r2 : c3452e48 r1 : d1c3dd28 r0 : c3450000
2024-05-17T03:01:10.692814+03:00 moblink kernel: [620868.976762] Flags: Nzcv IRQs on FIQs on Mode SVC_32 ISA Thumb Segment none
2024-05-17T03:01:10.692827+03:00 moblink kernel: [620868.976776] Control: 50c5387d Table: 458b406a DAC: 00000051
2024-05-17T03:01:10.692839+03:00 moblink kernel: [620868.976797] stmmac_get_stats64 from dev_get_stats+0x27/0xd0
2024-05-17T03:01:10.692851+03:00 moblink kernel: [620868.976828] dev_get_stats from dev_seq_printf_stats+0x21/0x124
2024-05-17T03:01:10.692863+03:00 moblink kernel: [620868.976853] dev_seq_printf_stats from dev_seq_show+0x11/0x24
2024-05-17T03:01:10.692875+03:00 moblink kernel: [620868.976874] dev_seq_show from seq_read_iter+0x281/0x35c
2024-05-17T03:01:10.692887+03:00 moblink kernel: [620868.976902] seq_read_iter from seq_read+0x61/0x84
2024-05-17T03:01:10.692901+03:00 moblink kernel: [620868.976925] seq_read from proc_reg_read+0x71/0x90
2024-05-17T03:01:10.692913+03:00 moblink kernel: [620868.976950] proc_reg_read from vfs_read+0x75/0x1e4
2024-05-17T03:01:10.692925+03:00 moblink kernel: [620868.976972] vfs_read from ksys_read+0x45/0x9c
2024-05-17T03:01:10.692937+03:00 moblink kernel: [620868.976990] ksys_read from ret_fast_syscall+0x1/0x5c
2024-05-17T03:01:10.692949+03:00 moblink kernel: [620868.977010] Exception stack(0xd1c3dfa8 to 0xd1c3dff0)
2024-05-17T03:01:10.692962+03:00 moblink kernel: [620868.977026] dfa0:                   0209b1e8 000005e8 00000004 02093a28 00000400 00000001
2024-05-17T03:01:10.692974+03:00 moblink kernel: [620868.977041] dfc0: 0209b1e8 000005e8 b6ed3888 00000003 0000000a be87c1fc 00000000 00000000
2024-05-17T03:01:10.693097+03:00 moblink kernel: [620868.977054] dfe0: 00000003 be87c128 b6e6e2bb b6de7616
2024-05-17T03:01:49.688246+03:00 moblink systemd[1]: sysstat-collect.service: Deactivated successfully.
2024-05-17T03:01:49.689170+03:00 moblink systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
2024-05-17T03:01:49.691186+03:00 moblink systemd[1]: sysstat-collect.service: Consumed 1min 47.293s CPU time.

cat armbian-image-release

Quote

# PLEASE DO NOT EDIT THIS FILE
BOARD=orangepizero
BOARD_NAME="Orange Pi Zero"
BOARDFAMILY=sun8i
BUILD_REPOSITORY_URL=https://github.com/armbian/build
BUILD_REPOSITORY_COMMIT=edd866b8c
LINUXFAMILY=sunxi
ARCH=arm
IMAGE_TYPE=stable
BOARD_TYPE=conf
INITRD_ARCH=arm
KERNEL_IMAGE_TYPE=zImage
FORCE_BOOTSCRIPT_UPDATE=
FORCE_UBOOT_UPDATE=
VENDOR="Armbian"
VENDORDOCS="https://docs.armbian.com"
VENDORURL="https://www.armbian.com"
VENDORSUPPORT="https://forum.armbian.com"
VENDORBUGS="https://www.armbian.com/bugs"
BOOTSCRIPT_FORCE_UPDATE="no"
BOOTSCRIPT_DST="boot.cmd"
VERSION=24.2.1
REVISION=24.2.1
IMAGE_UUID=ace87232-ab6a-44d2-afb6-8f24225ba9f7

cat armbian-release

Quote

# PLEASE DO NOT EDIT THIS FILE
BOARD=orangepizero
BOARD_NAME="Orange Pi Zero"
BOARDFAMILY=sun8i
BUILD_REPOSITORY_URL=https://github.com/armbian/build
BUILD_REPOSITORY_COMMIT=edd866b8c
LINUXFAMILY=sunxi
ARCH=arm
IMAGE_TYPE=stable
BOARD_TYPE=conf
INITRD_ARCH=arm
KERNEL_IMAGE_TYPE=zImage
FORCE_BOOTSCRIPT_UPDATE=
FORCE_UBOOT_UPDATE=
VENDOR="Armbian"
VENDORDOCS="https://docs.armbian.com"
VENDORURL="https://www.armbian.com"
VENDORSUPPORT="https://forum.armbian.com"
VENDORBUGS="https://www.armbian.com/bugs"
BOOTSCRIPT_FORCE_UPDATE="no"
BOOTSCRIPT_DST="boot.cmd"
VERSION=24.2.1
REVISION=24.2.1
BRANCH=current

cat os-release

Quote

PRETTY_NAME="Armbian 24.2.1 bookworm"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.armbian.com"
SUPPORT_URL="https://forum.armbian.com"
BUG_REPORT_URL="https://www.armbian.com/bugs"
ARMBIAN_PRETTY_NAME="Armbian 24.2.1 bookworm"

Quote

0

May 28, 20242 yr

Author
Solution

So... Taken the latest Armbian system update and on kernel version 6.6.31 no more CPU stalls. 🙂

Whoever fixed this, thank-you.

Edited May 28, 20242 yr by Myron

Quote

2

June 13, 20242 yr

CVE summary: https://www.cve.org/CVERecord?id=CVE-2024-26690

Link to a thread on kernel mailing list with bug report and proposed fix: https://lore.kernel.org/lkml/Zb8QwgYSh9sX8zBi@xhacker/T/

Quote

0

Frequent `rcu: INFO: rcu_sched detected stalls on CPUs/tasks` events logged.

Featured Replies

Solved by Myron

Join the conversation

Similar Content

Important Information

Account

Navigation

Search

Configure browser push notifications

Chrome (Android)

Chrome (Desktop)

Safari (iOS 16.4+)

Safari (macOS)

Edge (Android)

Edge (Desktop)

Firefox (Android)

Firefox (Desktop)