July 30, 20251 yr Since about four weeks we encounter out of memory issues when building an Armbian image on GitHub with a customize-image.sh script. We had no big modifications on the build system since then, we have to rebuild from time to time to get the latest version of our application into the image. We already use a more powerful runner (32GB RAM, 8 cores). Our compile call: ./compile.sh \ BOARD=rockpi-4b \ BRANCH=edge \ RELEASE=bookworm \ KERNEL_CONFIGURE=no \ BUILD_MINIMAL=yes \ NETWORKING_STACK="network-manager" \ COMPRESS_OUTPUTIMAGE=xz \ KERNEL_BTF=no \ SHARE_LOG=yes Our build is based on the tag v25.8.0-trunk.260 Any ideas for this? Thanks in advance Florian Edited July 30, 20251 yr by sans-ltd
July 31, 20251 yr Hello @sans-ltd, thank you for providing the logs. 32GB sounds like it should be plenty to compile the image successfully. Have you ever tried to reproduce this on bare metal? Would adding swap space to the github runner to mask the issue be an option? Does the runner always error out at the same step? It's a bit odd for apt to be running out of memory like this. Have you been able to monitor memory consumption during a failed build? Can you share your github workflow definition?
July 31, 20251 yr Author We saw that Github has no swap activated. So we activated it using thejerrybao/setup-swap-space@v1 step with 64GB which should be plenty enough. It happens sporadically, but when it happens it happens always at the same place, during this second apt update. Here is something else interesting: 2025-07-31T09:58:48.5188156Z [0m[1;30m[🐳|[0m🔨[0m] [0m total used free shared buff/cache available 2025-07-31T09:58:48.5189232Z [0m[1;30m[🐳|[0m🔨[0m] [0mMem: 31Gi 3.8Gi 15Gi 2.3Gi 14Gi 27Gi 2025-07-31T09:58:48.5190178Z [0m[1;30m[🐳|[0m🔨[0m] [0mSwap: 63Gi 0B 63Gi 2025-07-31T09:58:48.5643835Z [0m[1;30m[🐳|[0m🔨[0m] [0mFilesystem Size Used Avail Use% Mounted on 2025-07-31T09:58:48.5645400Z [0m[1;30m[🐳|[0m🔨[0m] [0mtmpfs 32G 2.3G 29G 8% / 2025-07-31T09:58:48.5646148Z [0m[1;30m[🐳|[0m🔨[0m] [0mtmpfs 32G 12M 32G 1% /tmp 2025-07-31T09:58:48.5646870Z [0m[1;30m[🐳|[0m🔨[0m] [0mtmpfs 32G 0 32G 0% /var/tmp 2025-07-31T09:58:48.5647587Z [0m[1;30m[🐳|[0m🔨[0m] [0mtmpfs 32G 0 32G 0% /run/user/0 2025-07-31T09:58:48.5648279Z [0m[1;30m[🐳|[0m🔨[0m] [0mtmpfs 64M 0 64M 0% /dev 2025-07-31T09:58:48.5648972Z [0m[1;30m[🐳|[0m🔨[0m] [0m/dev/root 290G 105G 186G 37% /tmp/overlay 2025-07-31T09:58:48.5669589Z [0m[1;30m[🐳|[0m🔨[0m] [0m------------------------------------------------------------------ 2025-07-31T09:58:48.5670721Z [0m[1;30m[🐳|[0m🔨[0m] [0m------------------------------------------------------------------------------------ 2025-07-31T09:58:48.5671415Z [0m[1;30m[🐳|[0m🔨[0m] [0mInstalling NodeRED 2025-07-31T09:58:49.1670560Z [0m[1;30m[🐳|[0m🔨[0m] [0mHit:1 http://deb.debian.org/debian bookworm InRelease 2025-07-31T09:58:49.1677442Z [0m[1;30m[🐳|[0m🔨[0m] [0mHit:2 http://github.armbian.com/configng stable InRelease 2025-07-31T09:58:49.1680378Z [0m[1;30m[🐳|[0m🔨[0m] [0mHit:3 http://deb.debian.org/debian bookworm-updates InRelease 2025-07-31T09:58:49.1682868Z [0m[1;30m[🐳|[0m🔨[0m] [0mHit:4 http://deb.debian.org/debian bookworm-backports InRelease 2025-07-31T09:58:49.1685589Z [0m[1;30m[🐳|[0m🔨[0m] [0mHit:5 http://security.debian.org bookworm-security InRelease 2025-07-31T09:58:58.1575506Z [0m[1;30m[🐳|[0m🔨[0m] [0mReading package lists... 2025-07-31T09:58:58.2559413Z [0m[1;30m[🐳|[0m🔨[0m] [0mE: Could not read from /var/lib/apt/lists/deb.debian.org_debian_dists_bookworm-backports_InRelease - getline (12: Cannot allocate memory) 2025-07-31T09:58:58.2561244Z [0m[1;30m[🐳|[0m🔨[0m] [0mE: The repository 'http://deb.debian.org/debian bookworm-backports InRelease' provides only weak security information. We added a free -h and a df -h command. It seems that the memleak is in apt-get update because there is around 78GBi of RAM available. No we did not build on bare metal for a long time.
July 31, 20251 yr 10 minutes ago, sans-ltd said: No we did not build on bare metal for a long time. Thank you for following up. I just did build on bare metal and checked on the result a few minutes ago and came here now to see you already responded. I can confirm this is not a github issue as I ran into the exact same problem on my Thinkpad 16GB RAM and 16GB swap. Never had an OOM issue with any armbian builds before. Thank you for bringing this to our attention.
August 1, 20251 yr Not sure if the target board makes a difference. But FWIW, my compilations for rockpi-4b and orangepi5-plus exhibited the issue while the one for bananapim2plus did not.
August 2, 20251 yr an orangepi5-plus compilation succeeded just fine now, so this error is not deterministic
August 3, 20251 yr Another report of this issue. Went away on a second build attempt according to the user @uablrek.
April 8Apr 8 Try sysctl vm.overcommit_memory=1 QEMU is probably trying to commit lots of virtual memory, but not using it. Don't know why it behaves like this.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.