Hi

I have upgraded an existing x86 Armbian system with kernel 6.6.63 and zfs 2.3.2 to the suggested kernel 6.12.32-current-x86, but I cannot get the zfs kernel module loaded after the latest updates to Armbian 25.5.1 noble. I also tried on other kernels, only the old 6.6.63 seems to work.

 

Normally when you select a different kernel in the armbian-config menu, it should automatically compile the zfs modules if it is activated. It just skips it. But you can of course compile it with the provided DKMS system:

Quote

 

dkms build -m zfs -v 2.3.2 -k 6.12.32-current-x86

 

 

 

the build just finishes fine, without any error, also the install, but when you try to load the module:

Quote

modprobe zfs

it responds with:

 

Quote

 

modprobe: ERROR: could not insert 'zfs': Exec format error

 

 

 

dmesg:

Quote

[Mon Jun 16 12:40:38 2025] module spl: .gnu.linkonce.this_module section size must match the kernel's built struct module size at run time

I have zfs 2.3.2 on kernel 6.12.32-current-bcm2711 running just fine on a Raspberry Pi 4, so I tested what would happen if I just build a small kernel driver module:

root@armbian:/usr/src/dummydriver# cat dummydriver.c
 

Quote

 

#include <linux/module.h>
#include <linux/version.h>
#include <linux/kernel.h>

static int __init driver_init(void)
{
    printk(KERN_INFO "Goodbye World!");

    return 0;
}

static void __exit driver_exit(void)
{
    printk(KERN_INFO "Hello World!");
}

module_init(driver_init);
module_exit(driver_exit);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("dude");
MODULE_DESCRIPTION("driver");

 

 

 

root@armbian:/usr/src/dummydriver# cat Makefile
 

Quote

 

obj-m = dummydriver.o
KVERSION = $(shell uname -r)
all:
    make -C /lib/modules/$(KVERSION)/build M=$(shell pwd) modules

clean:
    make -C /lib/modules/$(KVERSION)/build M=$(shell pwd) clean

 

So when I run this on the  Raspberry Pi 4, I can load the module and unload

Quote

 

insmod dummydriver.ko

rmmod dummydriver

 

 

But when I run this on the x86 machine I get the same error:

Quote

root@armbian:/usr/src/dummydriver# insmod dummydriver.ko 
insmod: ERROR: could not insert module dummydriver.ko: Invalid module format
root@armbinan:/usr/src/dummydriver# [Mon Jun 16 12:13:21 2025] module dummydriver: .gnu.linkonce.this_module section size must match the kernel's built struct module size at run time

root@armbian:/usr/src/dummydriver# modinfo dummydriver.ko
 

Quote

 

filename:       /usr/src/dummydriver/dummydriver.ko
description:    driver
author:         dude
license:        GPL
srcversion:     4510E7795B8BE551B44A0E5
depends:        
name:           dummydriver
retpoline:      Y
vermagic:       6.12.32-current-x86 SMP preempt mod_unload modversions 
 

 

 

 

root@armbian:/usr/src/dummydriver# gcc --version
 

Quote

 

gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 

 

 

 

I also tried gcc 11.4.0

 

Here is my armbianmonitor -u:

 

https://paste.armbian.com/orepumazam

 

Any help welcome!

Philippe

 

 

Edited June 16, 20251 yr by phiser678

Hi

ok, I got a bit deeper. I installed the Armbian build framework and compiled the kernel.

 

./compile.sh build BOARD=uefi-x86 BRANCH=current BUILD_DESKTOP=no BUILD_MINIMAL=yes KERNEL_CONFIGURE=yes RELEASE=noble ENABLE_EXTENSIONS=

Then I got 3 deb files:

-

Quote

 

rw-rw-r-- 1 ubuntu root 551588032 Jun 17 11:08 linux-image-current-x86_25.08.0-trunk_amd64__6.12.33-Se03c-Dd997-P0000-C999999Hf8af-HK01ba-Vf5f1-Ba566-R448a.deb
-rw-rw-r-- 1 ubuntu root 103751872 Jun 17 11:08 linux-headers-current-x86_25.08.0-trunk_amd64__6.12.33-Se03c-Dd997-P0000-C999999Hf8af-HK01ba-Vf5f1-Ba566-R448a.deb
-rw-rw-r-- 1 ubuntu root   7649472 Jun 17 11:08 linux-libc-dev-current-x86_25.08.0-trunk_amd64__6.12.33-Se03c-Dd997-P0000-C999999Hf8af-HK01ba-Vf5f1-Ba566-R448a.deb

 

 

I installed the first 2 (libc-dev was complaining) and then I could compile and load the zfs module (and also the dummydriver)!

At first I used KERNEL_CONFIGURE=no with the compile script, and gave me 3 deb files from the ghcr.io/armbian/os/ repository without compiling the kernel, but these don't work! 

Quote

-rw-rw-r-- 1 ubuntu root  95467896 Jun 17 10:35 linux-image-current-x86_25.08.0-trunk_amd64__6.12.33-Se03c-Dd997-P0000-Ce553Hf8af-HK01ba-Vf5f1-Ba566-R448a.deb
-rw-rw-r-- 1 ubuntu root  14898368 Jun 17 10:35 linux-headers-current-x86_25.08.0-trunk_amd64__6.12.33-Se03c-Dd997-P0000-Ce553Hf8af-HK01ba-Vf5f1-Ba566-R448a.deb
-rw-rw-r-- 1 ubuntu root   1395792 Jun 17 10:35 linux-libc-dev-current-x86_25.08.0-trunk_amd64__6.12.33-Se03c-Dd997-P0000-Ce553Hf8af-HK01ba-Vf5f1-Ba566-R448a.deb
 

 

Notice also the sizes of the deb files are much smaller. I compared the 2 .config files of the one from the repository and the one that was compiled:

Quote

 diff linux-headers-6.12.33-current-x86-orig/.config linux-headers-6.12.33-current-x86/.config
24c24
< CONFIG_PAHOLE_VERSION=0
---
> CONFIG_PAHOLE_VERSION=125
138a139
> # CONFIG_SCHED_CLASS_EXT is not set
11911a11913,11917
> CONFIG_DEBUG_INFO_BTF=y
> CONFIG_PAHOLE_HAS_SPLIT_BTF=y
> CONFIG_PAHOLE_HAS_LANG_EXCLUDE=y
> CONFIG_DEBUG_INFO_BTF_MODULES=y
> # CONFIG_MODULE_ALLOW_BTF_MISMATCH is not set
12134a12141
> CONFIG_PROBE_EVENTS_BTF_ARGS=y
 

 

so, you are good now?

I think I solved it! In the upgraded system there was no package pahole! 

After I installed the pahole and reinstalled the package linux-headers-6.12.33-current-x86, I got the BTF stuff  as described above in the .config file and I can compile modules!

Pfff, finally!

 

I checked the Raspberry Pi 4 version, I see no pahole package installed there, and I can compile modules and zfs. I don't see any BTF stuff in the .config there also.

 

So, I don't know what is really going on here, I should install pahole on arm64 systems as well? And should pahole not be included when you upgrade your system to 25.5.1 ?

 

Edited June 18, 20251 yr by phiser678

Yes.

We should add `pahole` as a dependency of the linux-headers package, otherwise during headers install, the `.config` is different from the image build, leading to all kinds of sad.

While we might fix the future, unfortunately, installing pahole manually (and removing/re-installing the linux-headers package) is the only way forward for versions already out of the door.