Seems like the OS "malfunctions" after a few hours to a couple of days. The server is still responding to Pings, but services are all down, such as SSH.  Doing a power cycle would make the system available back online, until it becomes non-responsive again.  Moving back to 23.11.1 solves the stability issue.  To add, I'm using a couple of instances for primary and secondary PiHole DNS servers.

 

Probably similar issue on this thread -

 

I'm a bit of a noob, if someone can redirect me to pasting logs. Can those be found on /var/logs as I haven't explored much.  I have downgraded both my OPiOnes to 23.11.1 and Frozen the Armbian firmware updates, so I can't provide logs for now at least until I try to update one of the instances.

 

 

Edited March 24, 20242 yr by ER Samson

For a quick test I ran sbc-bench every 20 minutes overnight, about 11 hours. The system is still running well.

See the attached log.

sbc-bench v0.9.65 Xunlong Orange Pi.txt

 

You might want to connect to the serial port logged in as root to see if it is available after a system hang.  There may be some messages.

I may have found the culprit after monitoring htop. "vnstat -n" process eats up CPU cycles, sometimes reaching up to "6+" load-average (even if this OrangePi is quad-core equipped) 

 

This only happens intermittently and I can't seem to find the pattern.  Sharing my observations here:

• Only affects  Linux 6.6.16-current-sunxi and higher (such as edge), legacy firmware is not affected (stable for more than a week)
• Installing Pihole with cloudflared DoH package seems to trigger high CPU usage of vnstat
• Reboot fixes it (or a power cycle)
• Having PiHole with dnscrypt-proxy does not trigger the issue (so far, this test instance is already at the 16-hr mark, will keep on updating uptime)
  • Since it was stable overnight I now have two instances with this configuration, hoping for the best! 

Not sure if helpful, but this is all that I have for now. Thanks!

I can confirm that new armbian-firmware make instabilities. (orange pi one board, Linux 6.6.16-current-sunxi)

can someone point out the location of repository (3rd party is ok) of previous armbian-firmware releases ? I need to downgrade it from version 24.2.1 to version 21

1 hour ago, Joko Christian said:

location of repository

@Joko Christian

For the archive, go to the image download page for the orangepione and look at the line just below the image icons. 

 

It would be very useful to know how your system is failing.  Could you describe what is running on the machine at the time it fails?  Please provide output of:

 

sudo armbianmonitor -u

I have remove the armbian-firmware
#apt remove armbian-firmware
just need to restart it once today.
I still trying to find the deb package for older version of armbian-firmware
couldn't find it on https://mirror.yandex.ru/mirrors/armbian/archive/orangepione/archive/

 

Quote

chris@opi-jcc:~$ sudo armbianmonitor -u
System diagnosis information will now be uploaded to <!doctype html>
<html lang=en>
<head>
    <meta charset=utf-8>
    <title>&nbsp;</title>
</head>
<style>
    body {
        font-family: monospace;
        margin: 2em;
    }
</style>
<body>
    <p>ix.io is taking a break &#127867;</p>
    <img src="/underconstruction.gif" width="200px">
</body>
</html>

for info, I run the board for Pi-Hole using the  cloudflared. The second service is a smbd server for low throughput usage in LAN
the symtomps after armbian-firmware update to 24.2.1 is a hot cpu, so it might be the system cannot cope with the temperature and just hangs (I do not put any cooling / heatsink for 2 years and its working fine because of the low usages)

any suggestion to a armbian-firmware.deb file that i can dpkg is appreciated. thanks

@Joko Christian I can't find any archive for armbian-firmware.  It might be built right into the image.  Have you tried to install an older image?

 

Have you looked at @ER Samson post above.  He seems to be running the same PIHole and has some work arounds.

 

Have you looked at top to see what is running to cause a hot cpu?

thanks for replying @Stephen Graf. 
I rather want to try using the older armbian-firmware package and keep the newer kernel, since the problem only shows up after I upgrade the firmware.
currently it still hangs after 15-20 hours.
can I extract the firmware from and image?

the post from @ER Samson mention vnstat. I do not use it, and the package is not installed.
 

i did use htop to check what's using the cpu. but nothing extensive. I haven't check the governor prior the deletion of firmware. currently is "on-demand"

@Joko Christian I am not sure that mixing versions of firmware and kernel versions is advisable.

 

I took note from @ER Samson that when he removed cloudfare and replaced it with an equivalent package, the problems went away.

I am not able to do much with your issue as cloudfare is not part of Armbian, nor of debian for that matter.

 

I would recommend you try the most recent current Armbian package and the latest release of cloudfare. Try to run without making changes to the firmware unless you need extra drivers for hardware that is not supported by the Armbian system as built.

 

Will try, I guess i'll revert back to bullseye.

Using DNScrypt does reduce the instabilities with Linux 6.6.16-current-sunxians higher, though it still did have slowdowns/high CPU usage after 2-3 days.

 

I had to move the back up instance to legacy kernel and so my network has a functioning DNS when the other conks up. Legacy has been rock stable.

Edited May 27, 20242 yr by ER Samson
Description

@ER Samson I'm sorry to hear that you are still having problems. However as I mentioned to @Joko Christian, the applications you are running are not Debian or Armbian supported. I would recommend you try support from the individual software suppliers.

hi @Stephen Graf No worries, all good. I am currently trying out the newest freshly installed Armbian (24.5.1) on the primary instance (still on Pihole + DNSCrypt).  One good thing I'm seeing is that using "ps -ef" the vncstats is not visible.  Let's see how it holds.  Will report back.

Just so add, so far so good (10 hrs and counting, no CPU spikes). Armbian 24.5.1 with 6.6.31 Linux Kernel is super stable so far.

 

Update :
I haven't downgrade to bullseye, but seeing there's a new update for the bookworm, I try it.
it seems this latest update is making it stable:
linux-dtb-current-sunxi:armhf (24.2.1, 24.5.1), linux-u-boot-orangepione-current:armhf (24.2.1, 24.5.1), linux-image-current-sunxi:armhf (24.2.1, 24.5.1)
been running for 48 hours without error.
I even tried to install the newest armbian-firmware yesterday.

Yup, the newest Armbian 24.5.1 release is really good,  Seems like all issues may have been resolved.  I'll mark this thread as solved.

 

 

 

Edited June 3, 20242 yr by ER Samson
Adding more detail.

2 hours ago, ER Samson said:

Yup, the newest Armbian 24.5.1 release is really good,  Seems like all issues may have been resolved.  I'll mark this thread as solved.

Thank you for your testing and update.