random Oom on cubietruck

March 4, 20179 yr

Hi,

since one the last two kernel updates I experience random out of memory scenarious on my Cubietruck. Up until then all was fine. When Oom Killer kicks in the following is logged:

........................................

Spoiler


Mar  4 06:54:23 cubietruck kernel: [80988.962548] sh invoked oom-killer: gfp_mask=0x24000c0(GFP_KERNEL), nodemask=0, order=2, oom_score_adj=0
Mar  4 06:54:23 cubietruck kernel: [80988.962563] sh cpuset=/ mems_allowed=0
Mar  4 06:54:23 cubietruck kernel: [80988.962584] CPU: 1 PID: 10543 Comm: sh Not tainted 4.9.12-sunxi #4
Mar  4 06:54:23 cubietruck kernel: [80988.962588] Hardware name: Allwinner sun7i (A20) Family
Mar  4 06:54:23 cubietruck kernel: [80988.962625] [<c010c8dd>] (unwind_backtrace) from [<c01096a5>] (show_stack+0x11/0x14)
Mar  4 06:54:23 cubietruck kernel: [80988.962643] [<c01096a5>] (show_stack) from [<c04fd277>] (dump_stack+0x67/0x74)
Mar  4 06:54:23 cubietruck kernel: [80988.962660] [<c04fd277>] (dump_stack) from [<c0207e27>] (dump_header+0x63/0x174)
Mar  4 06:54:23 cubietruck kernel: [80988.962679] [<c0207e27>] (dump_header) from [<c01c3a75>] (oom_kill_process+0x261/0x3e4)
Mar  4 06:54:23 cubietruck kernel: [80988.962693] [<c01c3a75>] (oom_kill_process) from [<c01c3e19>] (out_of_memory+0xc5/0x224)
Mar  4 06:54:23 cubietruck kernel: [80988.962706] [<c01c3e19>] (out_of_memory) from [<c01c75f5>] (__alloc_pages_nodemask+0x939/0x950)
Mar  4 06:54:23 cubietruck kernel: [80988.962720] [<c01c75f5>] (__alloc_pages_nodemask) from [<c01c761d>] (__get_free_pages+0x11/0x20)
Mar  4 06:54:23 cubietruck kernel: [80988.962734] [<c01c761d>] (__get_free_pages) from [<c01137ad>] (pgd_alloc+0x19/0xfc)
Mar  4 06:54:23 cubietruck kernel: [80988.962747] [<c01137ad>] (pgd_alloc) from [<c0116b17>] (mm_init+0x97/0x118)
Mar  4 06:54:23 cubietruck kernel: [80988.962760] [<c0116b17>] (mm_init) from [<c020fe6b>] (do_execveat_common+0x1db/0x540)
Mar  4 06:54:23 cubietruck kernel: [80988.962773] [<c020fe6b>] (do_execveat_common) from [<c021038d>] (SyS_execve+0x25/0x28)
Mar  4 06:54:23 cubietruck kernel: [80988.962786] [<c021038d>] (SyS_execve) from [<c01060c1>] (ret_fast_syscall+0x1/0x4c)
Mar  4 06:54:23 cubietruck kernel: [80988.962791] Mem-Info:
Mar  4 06:54:23 cubietruck kernel: [80988.962814] active_anon:12204 inactive_anon:45615 isolated_anon:0
Mar  4 06:54:23 cubietruck kernel: [80988.962814]  active_file:38662 inactive_file:237639 isolated_file:31
Mar  4 06:54:23 cubietruck kernel: [80988.962814]  unevictable:337 dirty:53 writeback:0 unstable:0
Mar  4 06:54:23 cubietruck kernel: [80988.962814]  slab_reclaimable:125926 slab_unreclaimable:41611
Mar  4 06:54:23 cubietruck kernel: [80988.962814]  mapped:19018 shmem:15015 pagetables:745 bounce:0
Mar  4 06:54:23 cubietruck kernel: [80988.962814]  free:8795 free_pcp:1 free_cma:588
Mar  4 06:54:23 cubietruck kernel: [80988.962833] Node 0 active_anon:48816kB inactive_anon:182460kB active_file:154648kB inactive_file:950556kB unevictable:1348kB isolated(anon):0kB isolated(file):124kB mapped:76072kB dirty:212kB writeback:0kB shmem:60060kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Mar  4 06:54:23 cubietruck kernel: [80988.962853] Normal free:17656kB min:3468kB low:4332kB high:5196kB active_anon:488kB inactive_anon:3588kB active_file:54324kB inactive_file:88kB unevictable:0kB writepending:212kB present:786432kB managed:753376kB mlocked:0kB slab_reclaimable:503704kB slab_unreclaimable:166444kB kernel_stack:1424kB pagetables:16kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB
Mar  4 06:54:23 cubietruck kernel: lowmem_reserve[]: 0 10088 10088
Mar  4 06:54:23 cubietruck kernel: [80988.962884] HighMem free:17524kB min:512kB low:2000kB high:3488kB active_anon:48328kB inactive_anon:178872kB active_file:100356kB inactive_file:950336kB unevictable:1348kB writepending:0kB present:1307648kB managed:1307648kB mlocked:1348kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:2964kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:2352kB
Mar  4 06:54:23 cubietruck kernel: lowmem_reserve[]: 0 0 0
Mar  4 06:54:23 cubietruck kernel: [80988.962898] Normal: 3077*4kB (UME) 671*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 17676kB
Mar  4 06:54:23 cubietruck kernel: HighMem: 2026*4kB (UMC) 615*8kB (UMC) 121*16kB (UMC) 43*32kB (UMC) 16*64kB (UM) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 17616kB
Mar  4 06:54:23 cubietruck kernel: 291594 total pagecache pages
Mar  4 06:54:23 cubietruck kernel: [80988.962989] 0 pages in swap cache
Mar  4 06:54:23 cubietruck kernel: [80988.962994] Swap cache stats: add 0, delete 0, find 0/0
Mar  4 06:54:23 cubietruck kernel: [80988.962997] Free swap  = 524284kB
Mar  4 06:54:23 cubietruck kernel: [80988.963000] Total swap = 524284kB
Mar  4 06:54:23 cubietruck kernel: [80988.963003] 523520 pages RAM
Mar  4 06:54:23 cubietruck kernel: [80988.963006] 326912 pages HighMem/MovableOnly
Mar  4 06:54:23 cubietruck kernel: [80988.963009] 8264 pages reserved
Mar  4 06:54:23 cubietruck kernel: [80988.963012] 4096 pages cma reserved
Mar  4 06:54:23 cubietruck kernel: [80988.963016] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Mar  4 06:54:23 cubietruck kernel: [80988.963184] [  256]     0   256     2640      569       6       0        0         -1000 udevd
Mar  4 06:54:23 cubietruck kernel: [80988.963199] [  417]     0   417      780      310       5       0        0             0 lircd
Mar  4 06:54:23 cubietruck kernel: [80988.963215] [ 1538]   999  1538     4835     4368      13       0        0             0 perl
Mar  4 06:54:23 cubietruck kernel: [80988.963233] [ 1554]     0  1554     7850      544       9       0        0             0 rsyslogd
Mar  4 06:54:23 cubietruck kernel: [80988.963251] [ 1577]     0  1577     4438      978      12       0        0             0 nmbd
Mar  4 06:54:23 cubietruck kernel: [80988.963268] [ 1590]     0  1590      543       32       6       0        0             0 atd
Mar  4 06:54:23 cubietruck kernel: [80988.963286] [ 1645]   105  1645     1146      426       6       0        0             0 dbus-daemon
Mar  4 06:54:23 cubietruck kernel: [80988.963317] [ 1677]     0  1677     1797     1037       7       0        0             0 haveged
Mar  4 06:54:23 cubietruck kernel: [80988.963334] [ 1694]     0  1694      803      489       6       0        0             0 incrond
Mar  4 06:54:23 cubietruck kernel: [80988.963354] [ 1875]    33  1875     3391     1441      10       0        0             0 lighttpd
Mar  4 06:54:23 cubietruck kernel: [80988.963367] [ 1876]    33  1876     6490     4474      17       0        0             0 munin-cgi-graph
Mar  4 06:54:23 cubietruck kernel: [80988.963383] [ 1886]    33  1886     6490     4443      16       0        0             0 munin-cgi-graph
Mar  4 06:54:23 cubietruck kernel: [80988.963400] [ 1898]    33  1898     6490     4395      17       0        0             0 munin-cgi-graph
Mar  4 06:54:23 cubietruck kernel: [80988.963417] [ 1908]    33  1908     6490     4429      18       0        0             0 munin-cgi-graph
Mar  4 06:54:23 cubietruck kernel: [80988.963435] [ 1912]    33  1912    50765     5648      29       0        0             0 php-cgi
Mar  4 06:54:23 cubietruck kernel: [80988.963452] [ 1914]     0  1914      360      227       5       0        0             0 mysqld_safe
Mar  4 06:54:23 cubietruck kernel: [80988.963470] [ 2015]    33  2015    52050     7973      37       0        0             0 php-cgi
Mar  4 06:54:23 cubietruck kernel: [80988.963487] [ 2016]    33  2016    52007     7874      37       0        0             0 php-cgi
Mar  4 06:54:23 cubietruck kernel: [80988.963504] [ 2018]    33  2018    51810     7441      37       0        0             0 php-cgi
Mar  4 06:54:23 cubietruck kernel: [80988.963522] [ 2019]    33  2019    51985     7447      37       0        0             0 php-cgi
Mar  4 06:54:23 cubietruck kernel: [80988.963538] [ 2296]   108  2296    81749    10711      46       0        0             0 mysqld
Mar  4 06:54:23 cubietruck kernel: [80988.963548] [ 2343]   106  2343     1123      734       6       0        0             0 ntpd
Mar  4 06:54:23 cubietruck kernel: [80988.963564] [ 2449]     0  2449    19665     2038      19       0        0             0 fail2ban-server
Mar  4 06:54:23 cubietruck kernel: [80988.963578] [ 2460]   109  2460    39311     3817      18       0        0          -900 postgres
Mar  4 06:54:23 cubietruck kernel: [80988.963587] [ 2473]     0  2473     7317     2790      19       0        0             0 smbd
Mar  4 06:54:23 cubietruck kernel: [80988.963598] [ 2475]   109  2475    39311     1665      21       0        0             0 postgres
Mar  4 06:54:23 cubietruck kernel: [80988.963608] [ 2476]   109  2476    39311     1153      14       0        0             0 postgres
Mar  4 06:54:23 cubietruck kernel: [80988.963618] [ 2477]   109  2477    39311     1952      14       0        0             0 postgres
Mar  4 06:54:23 cubietruck kernel: [80988.963628] [ 2478]   109  2478    39380     1253      14       0        0             0 postgres
Mar  4 06:54:23 cubietruck kernel: [80988.963642] [ 2480]   109  2480     4375      918      11       0        0             0 postgres
Mar  4 06:54:23 cubietruck kernel: [80988.963654] [ 2529]     0  2529     1635      560       6       0        0         -1000 sshd
Mar  4 06:54:23 cubietruck kernel: [80988.963664] [ 2707]   114  2707     4101     2100      13       0        0             0 unbound
Mar  4 06:54:23 cubietruck kernel: [80988.963673] [ 2755]     0  2755     1601      438       8       0        0             0 cron
Mar  4 06:54:23 cubietruck kernel: [80988.963684] [ 2769]     0  2769      693      487       6       0        0             0 dovecot
Mar  4 06:54:23 cubietruck kernel: [80988.963693] [ 2774]   111  2774     1048      853       6       0        0             0 imap-login
Mar  4 06:54:23 cubietruck kernel: [80988.963703] [ 2775]   110  2775      576      405       7       0        0             0 anvil
Mar  4 06:54:23 cubietruck kernel: [80988.963712] [ 2777]     0  2777      610      433       6       0        0             0 log
Mar  4 06:54:23 cubietruck kernel: [80988.963723] [ 2779]   111  2779     1048      782       7       0        0             0 imap-login
Mar  4 06:54:23 cubietruck kernel: [80988.963732] [ 2781]     0  2781      961      677       6       0        0             0 config
Mar  4 06:54:23 cubietruck kernel: [80988.963742] [ 2798]     0  2798      346      240       5       0        0             0 startpar
Mar  4 06:54:23 cubietruck kernel: [80988.963752] [ 3009]     0  3009     1339      637       6       0        0             0 master
Mar  4 06:54:23 cubietruck kernel: [80988.963761] [ 3016]   112  3016     1367      656       7       0        0             0 qmgr
Mar  4 06:54:23 cubietruck kernel: [80988.963771] [ 3062]     0  3062      356      340       6       0        0         -1000 watchdog
Mar  4 06:54:23 cubietruck kernel: [80988.963783] [29502]   112 29502     1358      635       6       0        0             0 pickup
Mar  4 06:54:23 cubietruck kernel: [80988.963792] [ 1721]     0  1721     1795      463       8       0        0             0 cron
Mar  4 06:54:23 cubietruck kernel: [80988.963802] [ 1724]     0  1724      360      105       4       0        0             0 sh
Mar  4 06:54:23 cubietruck kernel: [80988.963811] [ 1726]     0  1726      343      228       6       0        0             0 run-parts
Mar  4 06:54:23 cubietruck kernel: [80988.963822] [ 7608]     0  7608     3799     2400      11       0        0             0 munin-node
Mar  4 06:54:23 cubietruck kernel: [80988.963832] [ 8055]   110  8055      782      488       7       0        0             0 auth
Mar  4 06:54:23 cubietruck kernel: [80988.963843] [ 9645]     0  9645      576      124       6       0        0             0 ssl-params
Mar  4 06:54:23 cubietruck kernel: [80988.963853] [ 9682]   110  9682      578      140       6       0        0             0 indexer
Mar  4 06:54:23 cubietruck kernel: [80988.963863] [ 9683]     0  9683     1861     1452       7       0        0             0 indexer-worker
Mar  4 06:54:23 cubietruck kernel: [80988.963873] [10255]   107 10255     4629     3009      12       0        0             0 munin-html
Mar  4 06:54:23 cubietruck kernel: [80988.963885] [10264]   112 10264     1375      658       8       0        0             0 cleanup
Mar  4 06:54:23 cubietruck kernel: [80988.963895] [10269]   112 10269     1360      636       6       0        0             0 trivial-rewrite
Mar  4 06:54:23 cubietruck kernel: [80988.963904] [10272]     0 10272     1366      858       6       0        0             0 local
Mar  4 06:54:23 cubietruck kernel: [80988.963914] [10280]   112 10280     1430      680       5       0        0             0 lmtp
Mar  4 06:54:23 cubietruck kernel: [80988.963924] [10284]     0 10284     2406     1763       9       0        0             0 lmtp
Mar  4 06:54:23 cubietruck kernel: [80988.963933] [10290]     0 10290      885      607       6       0        0             0 auth
Mar  4 06:54:23 cubietruck kernel: [80988.963943] [10302]  1000 10302      587      446       4       0        0             0 mvmail
Mar  4 06:54:23 cubietruck kernel: [80988.963952] [10368]  1000 10368      343       79       5       0        0             0 flock
Mar  4 06:54:23 cubietruck kernel: [80988.963962] [10369]  1000 10369      344       87       5       0        0             0 sleep
Mar  4 06:54:23 cubietruck kernel: [80988.963972] [10536]     0 10536      360       96       4       0        0             0 man-db
Mar  4 06:54:23 cubietruck kernel: [80988.963982] [10542]     6 10542      360      105       4       0        0             0 sh
Mar  4 06:54:23 cubietruck kernel: [80988.963990] [10543]     6 10543      360       16       4       0        0             0 sh
Mar  4 06:54:23 cubietruck kernel: [80988.963999] [10544]     6 10544      360       16       4       0        0             0 sh
Mar  4 06:54:23 cubietruck kernel: [80988.964006] Out of memory: Kill process 2296 (mysqld) score 16 or sacrifice child
Mar  4 06:54:23 cubietruck kernel: [80988.964141] Killed process 2296 (mysqld) total-vm:326996kB, anon-rss:35548kB, file-rss:7296kB, shmem-rss:0kB

........................................

Is there a possibility to rollback the kernel and associated packages to 5.23? Where to find these and how to downgrade then? I am currently running 5.26 and am pretty sure that this started with 5.25. Yes, the system was unchanged - I also removed some services but can not nail it down.

Any help to get the root cause for this is also greately appreciated - what logs / command output to provide here?

Thanks!

Edited March 11, 20179 yr by thom_arm
code under spoiler

March 6, 20179 yr

Which kernel-version are you running? There was a bug in kernel 4.7/4.8:

https://encrypted.google.com/search?hl=de&q=oom-killer kernel bug 4.7

There is also a memory-parameter that configures how much % of memory is used before the oom-killer is invoked. That might help you until the kernel-bug is fixed. Look for vm.overcommit_ratio in the last post here:

http://serverfault.com/questions/362589/effects-of-configuring-vm-overcommit-memory

March 10, 20179 yr

I have the same issue as thom_arm: Since an update from 5.23 to 5.25 (now on kernel 4.9.12), I get frequent oom errors resulting in oom-killer killing my unbound, system-journal etc etc daemons.

I have an ssd attached and tried the following:

moved /tmp from tmpfs to the ssd
set vm/swappiness from 0 to 30
increased swap to 3 GByte (2 GByte physical RAM available in total on my cubietruck board)
set vm/overcommit to 2 and vm/overcommit_ratio to 100

None of that helped to improve the situation. Except the armbian update mentioned above, nothing changed on the system, that I am aware of.

The oom's occur during cron-daily jobs (e.g. texpire) or fetching news (leafnode). But that was working perfectly before the armbian updates

Any suggestions what I should do?

Sample logging:

------------------------------------------

Spoiler


Mar 10 06:55:39 nari kernel: [79423.760842] texpire invoked oom-killer: gfp_mask=0x26000d0(GFP_TEMPORARY|__GFP_NOTRACK), nodemask=0, order=0, oom_score_adj=0
Mar 10 06:55:39 nari kernel: [79423.760859] texpire cpuset=/ mems_allowed=0
Mar 10 06:55:39 nari kernel: [79423.760881] CPU: 0 PID: 4368 Comm: texpire Not tainted 4.9.12-sunxi #4
Mar 10 06:55:39 nari kernel: [79423.760885] Hardware name: Allwinner sun7i (A20) Family
Mar 10 06:55:39 nari kernel: [79423.760926] [<c010c8dd>] (unwind_backtrace) from [<c01096a5>] (show_stack+0x11/0x14)
Mar 10 06:55:39 nari kernel: [79423.760944] [<c01096a5>] (show_stack) from [<c04fd277>] (dump_stack+0x67/0x74)
Mar 10 06:55:39 nari kernel: [79423.760960] [<c04fd277>] (dump_stack) from [<c0207e27>] (dump_header+0x63/0x174)
Mar 10 06:55:39 nari kernel: [79423.760981] [<c0207e27>] (dump_header) from [<c01c3a75>] (oom_kill_process+0x261/0x3e4)
Mar 10 06:55:39 nari kernel: [79423.760994] [<c01c3a75>] (oom_kill_process) from [<c01c3e19>] (out_of_memory+0xc5/0x224)
Mar 10 06:55:39 nari kernel: [79423.761007] [<c01c3e19>] (out_of_memory) from [<c01c75f5>] (__alloc_pages_nodemask+0x939/0x950)
Mar 10 06:55:39 nari kernel: [79423.761025] [<c01c75f5>] (__alloc_pages_nodemask) from [<c01f94bf>] (new_slab+0x303/0x3f8)
Mar 10 06:55:39 nari kernel: [79423.761038] [<c01f94bf>] (new_slab) from [<c01fa813>] (___slab_alloc.constprop.10+0x1a3/0x204)
Mar 10 06:55:39 nari kernel: [79423.761051] [<c01fa813>] (___slab_alloc.constprop.10) from [<c01faded>] (kmem_cache_alloc+0x131/0x154)
Mar 10 06:55:39 nari kernel: [79423.761065] [<c01faded>] (kmem_cache_alloc) from [<c021c7b5>] (__d_alloc+0x25/0x120)
Mar 10 06:55:39 nari kernel: [79423.761076] [<c021c7b5>] (__d_alloc) from [<c021c8c1>] (d_alloc+0x11/0x58)
Mar 10 06:55:39 nari kernel: [79423.761086] [<c021c8c1>] (d_alloc) from [<c021cc2f>] (d_alloc_parallel+0x3b/0x338)
Mar 10 06:55:39 nari kernel: [79423.761098] [<c021cc2f>] (d_alloc_parallel) from [<c0211c51>] (lookup_slow+0x3d/0xe4)
Mar 10 06:55:39 nari kernel: [79423.761110] [<c0211c51>] (lookup_slow) from [<c0213f09>] (walk_component+0x14d/0x1c4)
Mar 10 06:55:39 nari kernel: [79423.761123] [<c0213f09>] (walk_component) from [<c021439b>] (path_lookupat+0x47/0xb0)
Mar 10 06:55:39 nari kernel: [79423.761134] [<c021439b>] (path_lookupat) from [<c02157c7>] (filename_lookup+0x57/0xc8)
Mar 10 06:55:39 nari kernel: [79423.761150] [<c02157c7>] (filename_lookup) from [<c020deff>] (vfs_fstatat+0x3f/0x78)
Mar 10 06:55:39 nari kernel: [79423.761162] [<c020deff>] (vfs_fstatat) from [<c020e391>] (SyS_lstat64+0x1d/0x2c)
Mar 10 06:55:39 nari kernel: [79423.761175] [<c020e391>] (SyS_lstat64) from [<c01060c1>] (ret_fast_syscall+0x1/0x4c)
Mar 10 06:55:39 nari kernel: [79423.761181] Mem-Info:
Mar 10 06:55:39 nari kernel: [79423.761202] active_anon:2116 inactive_anon:11199 isolated_anon:0
Mar 10 06:55:39 nari kernel: [79423.761202]  active_file:108003 inactive_file:147886 isolated_file:0
Mar 10 06:55:39 nari kernel: [79423.761202]  unevictable:0 dirty:0 writeback:0 unstable:0
Mar 10 06:55:39 nari kernel: [79423.761202]  slab_reclaimable:182236 slab_unreclaimable:2742
Mar 10 06:55:39 nari kernel: [79423.761202]  mapped:6873 shmem:4296 pagetables:349 bounce:0
Mar 10 06:55:39 nari kernel: [79423.761202]  free:55967 free_pcp:523 free_cma:0
Mar 10 06:55:39 nari kernel: [79423.761220] Node 0 active_anon:8464kB inactive_anon:44796kB active_file:432012kB inactive_file:591544kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:27492kB dirty:0kB writeback:0kB shmem:17184kB writeback_tmp:0kB unstable:0kB pages_scanned:11169818 all_unreclaimable? yes
Mar 10 06:55:39 nari kernel: [79423.761240] Normal free:3452kB min:3460kB low:4324kB high:5188kB active_anon:0kB inactive_anon:0kB active_file:76kB inactive_file:0kB unevictable:0kB writepending:0kB present:786432kB managed:753372kB mlocked:0kB slab_reclaimable:728944kB slab_unreclaimable:10968kB kernel_stack:992kB pagetables:0kB bounce:0kB free_pcp:1264kB local_pcp:548kB free_cma:0kB
Mar 10 06:55:39 nari kernel: lowmem_reserve[]: 0 10088 10088
Mar 10 06:55:39 nari kernel: [79423.761272] HighMem free:220416kB min:512kB low:2004kB high:3496kB active_anon:8464kB inactive_anon:44796kB active_file:431912kB inactive_file:591468kB unevictable:0kB writepending:0kB present:1307648kB managed:1307648kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:1396kB bounce:0kB free_pcp:828kB local_pcp:76kB free_cma:0kB
Mar 10 06:55:39 nari kernel: lowmem_reserve[]: 0 0 0
Mar 10 06:55:39 nari kernel: [79423.761285] Normal: 607*4kB (UME) 128*8kB (ME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3452kB
Mar 10 06:55:39 nari kernel: HighMem: 0*4kB 468*8kB (UM) 404*16kB (UM) 289*32kB (M) 266*64kB (UM) 109*128kB (M) 68*256kB (M) 64*512kB (UM) 19*1024kB (UM) 19*2048kB (UM) 15*4096kB (M) = 220416kB
Mar 10 06:55:39 nari kernel: 260204 total pagecache pages
Mar 10 06:55:39 nari kernel: [79423.761387] 16 pages in swap cache
Mar 10 06:55:39 nari kernel: [79423.761392] Swap cache stats: add 1980, delete 1964, find 759/1109
Mar 10 06:55:39 nari kernel: [79423.761395] Free swap  = 3276696kB
Mar 10 06:55:39 nari kernel: [79423.761398] Total swap = 3276792kB
Mar 10 06:55:39 nari kernel: [79423.761401] 523520 pages RAM
Mar 10 06:55:39 nari kernel: [79423.761404] 326912 pages HighMem/MovableOnly
Mar 10 06:55:39 nari kernel: [79423.761407] 8265 pages reserved
Mar 10 06:55:39 nari kernel: [79423.761410] 4096 pages cma reserved
Mar 10 06:55:39 nari kernel: [79423.761414] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Mar 10 06:55:39 nari kernel: [79423.761460] [  245]     0   245     2110      937       8       0        0             0 systemd-journal
Mar 10 06:55:39 nari kernel: [79423.761470] [  250]     0   250     2682      604       7       0        0         -1000 systemd-udevd
Mar 10 06:55:39 nari kernel: [79423.761481] [  596]     0   596     1798     1012       8       0        0             0 haveged
Mar 10 06:55:39 nari kernel: [79423.761490] [  598]     0   598     7860      633      10       0        0             0 rsyslogd
Mar 10 06:55:39 nari kernel: [79423.761500] [  599]     0   599      444      294       6       0        0             0 inetd
Mar 10 06:55:39 nari kernel: [79423.761509] [  600]     0   600     1635      897       9       0       12         -1000 sshd
Mar 10 06:55:39 nari kernel: [79423.761518] [  601]   110   601      856      532       6       0        0             0 avahi-daemon
Mar 10 06:55:39 nari kernel: [79423.761528] [  605]     0   605     1114      501       7       0        0             0 cron
Mar 10 06:55:39 nari kernel: [79423.761537] [  607]   110   607      791       54       5       0        0             0 avahi-daemon
Mar 10 06:55:39 nari kernel: [79423.761546] [  609]     0   609      774      473       6       0        6             0 systemd-logind
Mar 10 06:55:39 nari kernel: [79423.761556] [  611]   104   611     1146      607       7       0        3          -900 dbus-daemon
Mar 10 06:55:39 nari kernel: [79423.761565] [  653]     0   653      780      330       5       0        0             0 lircd
Mar 10 06:55:39 nari kernel: [79423.761574] [  691]     0   691     5026     1288       9       0        0             0 cupsd
Mar 10 06:55:39 nari kernel: [79423.761584] [  719]   105   719     1185      768       5       0        0             0 ntpd
Mar 10 06:55:39 nari kernel: [79423.761594] [  726]     0   726    13589     1052      14       0        5             0 bacula-fd
Mar 10 06:55:39 nari kernel: [79423.761603] [  730]   107   730     4153     2650      12       0        0             0 unbound
Mar 10 06:55:39 nari kernel: [79423.761613] [  739]   108   739     1357      551       8       0        0             0 dnsmasq
Mar 10 06:55:39 nari kernel: [79423.761623] [  747]     0   747      889      348       6       0        0             0 agetty
Mar 10 06:55:39 nari kernel: [79423.761631] [  753]     0   753      844      406       7       0        0             0 agetty
Mar 10 06:55:39 nari kernel: [79423.761640] [  775]     0   775     2229     1394       8       0        0             0 ddclient - slee
Mar 10 06:55:39 nari kernel: [79423.761650] [  797]     0   797    26741     4711      27       0        0             0 apache2
Mar 10 06:55:39 nari kernel: [79423.761659] [  801]    33   801    26751     1483      21       0        0             0 apache2
Mar 10 06:55:39 nari kernel: [79423.761669] [  802]    33   802    26751     1485      21       0        0             0 apache2
Mar 10 06:55:39 nari kernel: [79423.761678] [  803]    33   803    26751     1448      21       0        0             0 apache2
Mar 10 06:55:39 nari kernel: [79423.761687] [  804]    33   804    26751     1448      21       0        0             0 apache2
Mar 10 06:55:39 nari kernel: [79423.761697] [  805]    33   805    26751     1448      21       0        0             0 apache2
Mar 10 06:55:39 nari kernel: [79423.761709] [ 4115]     0  4115     1442      488       8       0        0             0 cron
Mar 10 06:55:39 nari kernel: [79423.761718] [ 4119]     0  4119      360      101       4       0        0             0 sh
Mar 10 06:55:39 nari kernel: [79423.761727] [ 4120]     0  4120      343       78       4       0        0             0 run-parts
Mar 10 06:55:39 nari kernel: [79423.761737] [ 4362]     0  4362      360       90       5       0        0             0 leafnode
Mar 10 06:55:39 nari kernel: [79423.761747] [ 4363]     0  4363     1461      619       8       0        0             0 su
Mar 10 06:55:39 nari kernel: [79423.761756] [ 4364]     9  4364     1135      767       7       0        0             0 systemd
Mar 10 06:55:39 nari kernel: [79423.761765] [ 4365]     9  4365     1651      485       6       0        1             0 (sd-pam)
Mar 10 06:55:39 nari kernel: [79423.761774] [ 4367]     9  4367      360      105       6       0        0             0 sh
Mar 10 06:55:39 nari kernel: [79423.761783] [ 4368]     9  4368     2004     1751      10       0        0             0 texpire
Mar 10 06:55:39 nari kernel: [79423.789501] rs:main Q:Reg invoked oom-killer: gfp_mask=0x2420848(GFP_NOFS|__GFP_NOFAIL|__GFP_HARDWALL|__GFP_MOVABLE), nodemask=0, order=0, oom_score_adj=0
Mar 10 06:55:39 nari kernel: [79423.789519] rs:main Q:Reg cpuset=/ mems_allowed=0
Mar 10 06:55:39 nari kernel: [79423.789541] CPU: 1 PID: 604 Comm: rs:main Q:Reg Not tainted 4.9.12-sunxi #4
Mar 10 06:55:39 nari kernel: [79423.789544] Hardware name: Allwinner sun7i (A20) Family
Mar 10 06:55:39 nari kernel: [79423.789583] [<c010c8dd>] (unwind_backtrace) from [<c01096a5>] (show_stack+0x11/0x14)
Mar 10 06:55:39 nari kernel: [79423.789601] [<c01096a5>] (show_stack) from [<c04fd277>] (dump_stack+0x67/0x74)
Mar 10 06:55:39 nari kernel: [79423.789618] [<c04fd277>] (dump_stack) from [<c0207e27>] (dump_header+0x63/0x174)
Mar 10 06:55:39 nari kernel: [79423.789640] [<c0207e27>] (dump_header) from [<c01c3a75>] (oom_kill_process+0x261/0x3e4)
Mar 10 06:55:39 nari kernel: [79423.789652] [<c01c3a75>] (oom_kill_process) from [<c01c3e19>] (out_of_memory+0xc5/0x224)
Mar 10 06:55:39 nari kernel: [79423.789665] [<c01c3e19>] (out_of_memory) from [<c01c758d>] (__alloc_pages_nodemask+0x8d1/0x950)
Mar 10 06:55:39 nari kernel: [79423.789678] [<c01c758d>] (__alloc_pages_nodemask) from [<c01c050f>] (pagecache_get_page+0xbb/0x218)
Mar 10 06:55:39 nari kernel: [79423.789695] [<c01c050f>] (pagecache_get_page) from [<c023196d>] (__getblk_gfp+0xcd/0x288)
Mar 10 06:55:39 nari kernel: [79423.789714] [<c023196d>] (__getblk_gfp) from [<c026b8e7>] (__ext4_get_inode_loc+0xc3/0x374)
Mar 10 06:55:39 nari kernel: [79423.789730] [<c026b8e7>] (__ext4_get_inode_loc) from [<c026ef2f>] (ext4_reserve_inode_write+0x23/0x60)
Mar 10 06:55:39 nari kernel: [79423.789740] [<c026ef2f>] (ext4_reserve_inode_write) from [<c026ef93>] (ext4_mark_inode_dirty+0x27/0x15c)
Mar 10 06:55:39 nari kernel: [79423.789751] [<c026ef93>] (ext4_mark_inode_dirty) from [<c027239f>] (ext4_dirty_inode+0x33/0x4c)
Mar 10 06:55:39 nari kernel: [79423.789763] [<c027239f>] (ext4_dirty_inode) from [<c022b55b>] (__mark_inode_dirty+0x2b/0x2d4)
Mar 10 06:55:39 nari kernel: [79423.789777] [<c022b55b>] (__mark_inode_dirty) from [<c021e16d>] (generic_update_time+0x6d/0xa0)
Mar 10 06:55:39 nari kernel: [79423.789790] [<c021e16d>] (generic_update_time) from [<c021e377>] (file_update_time+0x77/0xa4)
Mar 10 06:55:39 nari kernel: [79423.789803] [<c021e377>] (file_update_time) from [<c01c1a59>] (__generic_file_write_iter+0x6d/0x174)
Mar 10 06:55:39 nari kernel: [79423.789816] [<c01c1a59>] (__generic_file_write_iter) from [<c02681ef>] (ext4_file_write_iter+0xa7/0x2e8)
Mar 10 06:55:39 nari kernel: [79423.789832] [<c02681ef>] (ext4_file_write_iter) from [<c0209b6f>] (new_sync_write+0x7b/0xa0)
Mar 10 06:55:39 nari kernel: [79423.789844] [<c0209b6f>] (new_sync_write) from [<c020a5c7>] (vfs_write+0x77/0x140)
Mar 10 06:55:39 nari kernel: [79423.789854] [<c020a5c7>] (vfs_write) from [<c020b1b1>] (SyS_write+0x31/0x6c)
Mar 10 06:55:39 nari kernel: [79423.789868] [<c020b1b1>] (SyS_write) from [<c01060c1>] (ret_fast_syscall+0x1/0x4c)
Mar 10 06:55:39 nari kernel: [79423.789873] Mem-Info:
Mar 10 06:55:39 nari kernel: [79423.789893] active_anon:2116 inactive_anon:11194 isolated_anon:0
Mar 10 06:55:39 nari kernel: [79423.789893]  active_file:107982 inactive_file:147878 isolated_file:32
Mar 10 06:55:39 nari kernel: [79423.789893]  unevictable:0 dirty:0 writeback:0 unstable:0
Mar 10 06:55:39 nari kernel: [79423.789893]  slab_reclaimable:182227 slab_unreclaimable:2742
Mar 10 06:55:39 nari kernel: [79423.789893]  mapped:6873 shmem:4296 pagetables:349 bounce:0
Mar 10 06:55:39 nari kernel: [79423.789893]  free:55967 free_pcp:522 free_cma:0
Mar 10 06:55:39 nari kernel: [79423.789912] Node 0 active_anon:8464kB inactive_anon:44776kB active_file:431928kB inactive_file:591512kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:27492kB dirty:0kB writeback:0kB shmem:17184kB writeback_tmp:0kB unstable:0kB pages_scanned:11196848 all_unreclaimable? yes
Mar 10 06:55:39 nari kernel: [79423.789931] Normal free:3452kB min:3460kB low:4324kB high:5188kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:44kB unevictable:0kB writepending:0kB present:786432kB managed:753372kB mlocked:0kB slab_reclaimable:728908kB slab_unreclaimable:10968kB kernel_stack:992kB pagetables:0kB bounce:0kB free_pcp:1260kB local_pcp:716kB free_cma:0kB
Mar 10 06:55:39 nari kernel: lowmem_reserve[]: 0 10088 10088
Mar 10 06:55:39 nari kernel: [79423.789961] HighMem free:220416kB min:512kB low:2004kB high:3496kB active_anon:8464kB inactive_anon:44776kB active_file:431912kB inactive_file:591468kB unevictable:0kB writepending:0kB present:1307648kB managed:1307648kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:1396kB bounce:0kB free_pcp:828kB local_pcp:732kB free_cma:0kB
Mar 10 06:55:39 nari kernel: lowmem_reserve[]: 0 0 0
Mar 10 06:55:39 nari kernel: [79423.789973] Normal: 607*4kB (UME) 128*8kB (ME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3452kB
Mar 10 06:55:39 nari kernel: HighMem: 0*4kB 468*8kB (UM) 404*16kB (UM) 289*32kB (M) 266*64kB (UM) 109*128kB (M) 68*256kB (M) 64*512kB (UM) 19*1024kB (UM) 19*2048kB (UM) 15*4096kB (M) = 220416kB
:

Edited March 11, 20179 yr by Igor
logging under spoiler

March 11, 20179 yr

7 hours ago, timmr said:

Any suggestions what I should do?

In general:

- going back to old kernel, where this problem does not manifest (?),

- try new (developing) kernel to see if this is already fixed,

- try to fix the problem and provide patch,

April 1, 20179 yr

Hi,

meanwhile I am absolutely sure that this behaviour can not be observed at least with Kernel 4.8.4 from linux-image-next-sunxi_5.23_armhf.deb (and most probably all other version before). Something in 5.25 (did not ran this for a long time) and/or 5.26 may have introduced this. I did a rollback to 4.8.4 from linux-image-next-sunxi_5.23_armhf.deb about one week ago and found no issues so far. Nothing else changed during this time beside the kernel rollback. Dropping caches with a cronjob (0 */2 * * * sync && echo 1 > /proc/sys/vm/drop_caches) also did not helped for me (found this in some blog from a RedHat Guy) when running the 4.9.something kernel from package 5.26.

Unfortunately I have no spare time to dig deeper into this - just want to leave it here for others facing this.

Greetings.

Thom

April 2, 20179 yr

Hi,

thanks Igor for your answer (and of course for creating/developing armbian!) and also Thom for your add-on.

Unfortunately at the moment, I do not have the knowledge (and also probably not enough time) to do more analysis... so I will try to downgrade to 5.23... (as the problem still persists - I reboot nearly every day because of this).

Is it enough to do a "sudo apt-get install linux-image-next-sunxi=5.23" to downgrade the kernel? (or should I at least also downgrade linux-headers? Anything else?)

(Never done kernel downgrading before, sorry)

Thanks in advance...

Timm

April 3, 20179 yr

Hi,

I downloaded the following files from the repository:

linux-dtb-next-sunxi_5.23_armhf.deb
linux-headers-next-sunxi_5.23_armhf.deb
linux-jessie-root-next-cubietruck_5.23_armhf.deb
linux-firmware-image-next-sunxi_5.23_armhf.deb
linux-image-next-sunxi_5.23_armhf.deb

and installed them manually using 'dpkg -i'. Maybe you want to check with --dry-run first to see what will happen. Additionally I added the following to /etc/apt/preferences.d/50-armbian.pref in order to prevent checking for new kernels for the moment:

Package: linux-dtb-next-sunxi linux-firmware-image-next-sunxi linux-headers-next-sunxi linux-image-next-sunxi linux-jessie-root-next-cubietruck
Pin: origin "apt.armbian.com"
Pin-Priority: -10

Regards

Thom

random Oom on cubietruck

Featured Replies

Similar Content

Important Information

Account

Navigation

Search

Configure browser push notifications

Chrome (Android)

Chrome (Desktop)

Safari (iOS 16.4+)

Safari (macOS)

Edge (Android)

Edge (Desktop)

Firefox (Android)

Firefox (Desktop)