I will try and remember all the relevant details.  Going ~ in reverse chronological order here.  Earlier this evening, @lanefuwas helping me in IRC (which I greatly appreciate btw) but he had to go to bed.  Next step was getting out the UART he said.  So I did that, now I come here to report the results:

 

U-Boot SPL 2020.10-armbian (Oct 18 2020 - 23:16:40 +0200)
DRAM:Timeout initialising DRAM

resetting ...

[...repeats...]

 

Prior to that, we were trying to fix the bootloader and/or firmware.  At that time the new kernel (latest stable), initrd, etc. all appeared to be in /boot (we had thought maybe sd card was fried, but apparently not).

 

Prior to that, I had tried to do a kernel upgrade, via armbian-config.  That was when the problems started (the Cubietruck never came back up).  I was coming from a slightly older kernel before that.  I think because at some point I did not upgrade (Arm/Debian) for a long time, I somehow skipped over the naming change and never picked up the new kernel.  I also did not know it's necessary to manually upgrade kernel (on regular Debian that I am used to, it's automatic)?  Or maybe because armbian-config was never installed (I had also installed that at this time to upgrade the kernel manually)?  Anyway, at this point not sure exactly how I got into this pickle, just trying to note as many potentially relevant details as possible.

 

Result of armbianmonitor -u before the kernel upgrade: http://ix.io/2DQK  As you can see, I am on slightly older kernel (4.19.62).

 

Prior to that, I had dist-upgraded Debian / Armbian to Buster (but somehow not the kernel, apparently).  That was a while back.  And everything seemed to be working fine in the meantime.

 

Please let me know if any other info is needed.  You could look also at IRC logs from earlier this evening.

 

I am at a loss of what to do next.

Edited November 13, 20205 yr by TRS-80
clarity

@TRS-80

 

Can you post the ls -la of /boot and contents of /boot/boot.cmd

 

 

Spoiler

# ls -al /boot/
total 31124
drwxr-xr-x  3 root root    4096 Nov 11 23:45 .
drwxr-xr-x 22 root root    4096 Nov 11 21:08 ..
-rw-r--r--  1 root root    6944 Sep 14  2016 boot.bmp
-rw-r--r--  1 root root    2796 Sep 14  2016 boot.cmd
-rw-r--r--  1 root root    2868 Nov 11 23:40 boot.scr
-rw-r--r--  1 root root  189548 Oct 20 16:15 config-5.8.16-sunxi
lrwxrwxrwx  1 root root      16 Nov 11 21:11 dtb -> dtb-5.8.16-sunxi
drwxr-xr-x  3 root root   12288 Nov 11 21:10 dtb-5.8.16-sunxi
lrwxrwxrwx  1 root root      17 Jul 28  2019 dtb.old -> dtb-4.19.57-sunxi
-rw-r--r--  1 root root 9885905 Nov 11 21:11 initrd.img-5.8.16-sunxi
lrwxrwxrwx  1 root root      18 Sep 14  2016 script.bin.disabled -> bin/cubietruck.bin
-rw-r--r--  1 root root 4071205 Oct 20 16:15 System.map-5.8.16-sunxi
lrwxrwxrwx  1 root root      20 Nov 11 21:11 uInitrd -> uInitrd-5.8.16-sunxi
-rw-r--r--  1 root root 9885969 Nov 11 21:11 uInitrd-5.8.16-sunxi
-rw-r--r--  1 root root       0 May 31  2017 .verbose
-rwxr-xr-x  1 root root 7793968 Oct 20 16:15 vmlinuz-5.8.16-sunxi
lrwxrwxrwx  1 root root      20 Nov 11 21:11 zImage -> vmlinuz-5.8.16-sunxi

# cat boot/boot.cmd 
if ext4load mmc 0 0x00000000 /boot/.verbose
then
setenv verbosity 7
else
setenv verbosity 1
fi

# nonstandard monitor settings for A10, A20 and A31 based boards
# screen is initialized before this script -> saving to u-boot environment is mandatory 
#setenv video-mode sunxi:1024x768-24@60,monitor=dvi,hpd=0,edid=0,overscan_x=1,overscan_y=2
#saveenv
# nonstandard monitor settings

setenv bootargs "console=ttyS0,115200 console=tty1 root=/dev/mmcblk0p1 rootwait rootfstype=ext4 cgroup_enable=memory swapaccount=1 sunxi_ve_mem_reserve=0 sunxi_g2d_mem_reserve=0 sunxi_no_mali_mem_reserve sunxi_fb_mem_reserve=16 hdmi.audio=EDID:0 disp.screen0_output_mode=1920x1080p60 panic=10 consoleblank=0 enforcing=0 loglevel=${verbosity}"
#--------------------------------------------------------------------------------------------------------------------------------
# Boot loader script to boot with different boot methods for old and new kernel
#--------------------------------------------------------------------------------------------------------------------------------
if ext4load mmc 0 0x00000000 /boot/.next || fatload mmc 0 0x00000000 .next || ext4load mmc 0 0x00000000 .next
then
# sunxi mainline kernel
#--------------------------------------------------------------------------------------------------------------------------------
ext4load mmc 0 ${fdt_addr_r} /boot/dtb/${fdtfile} || fatload mmc 0 ${fdt_addr_r} /dtb/${fdtfile} || ext4load mmc 0 ${fdt_addr_r} /dtb/${fdtfile}
ext4load mmc 0 ${ramdisk_addr_r} /boot/uInitrd || fatload mmc 0 ${ramdisk_addr_r} uInitrd || ext4load mmc 0 ${ramdisk_addr_r} uInitrd || setenv ramdisk_addr_r "-"
ext4load mmc 0 ${kernel_addr_r} /boot/zImage || fatload mmc 0 ${kernel_addr_r} zImage || ext4load mmc 0 ${kernel_addr_r} zImage
bootz ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}
#--------------------------------------------------------------------------------------------------------------------------------
else
# sunxi android kernel
#--------------------------------------------------------------------------------------------------------------------------------
ext4load mmc 0 ${fdt_addr_r} /boot/script.bin || fatload mmc 0 ${fdt_addr_r} script.bin || ext4load mmc 0 ${fdt_addr_r} script.bin
ext4load mmc 0 ${ramdisk_addr_r} /boot/uInitrd || fatload mmc 0 ${ramdisk_addr_r} uInitrd || ext4load mmc 0 ${ramdisk_addr_r} uInitrd || setenv ramdisk_addr_r "-"
ext4load mmc 0 ${kernel_addr_r} /boot/zImage || fatload mmc 0 ${kernel_addr_r} zImage || ext4load mmc 0 ${kernel_addr_r} zImage
bootz ${kernel_addr_r} ${ramdisk_addr_r}
#--------------------------------------------------------------------------------------------------------------------------------
fi
# Recompile with:
# mkimage -C none -A arm -T script -d /boot/boot.cmd /boot/boot.scr 

 

 

I just noticed that mkimage command at the bottom.  Do i need to run that?

@lanefu,

 

OK, so that recent test image booted!

 

What now?

Someone recently reworked our outdated troubleshooting guide. Check if it helps for quick restoration: https://docs.armbian.com/User-Guide_Advanced-Features/

Thanks for the pointer, Igor, I had not seen that before.

 

I am reading how to unbrick the system (section at the provided link) and the first paragraph states:

 

Quote

With this procedure you will reinstall the u-boot, kernel and hardware settings.

 

However, the list of (4) packages provided shortly thereafter does not actually appear to contain the u-boot:

 

Quote

Root:
https://apt.armbian.com/pool/main/l/linux-focal-root-current-nanopineo2/linux-focal-root-current-nanopineo2_20.08.13_arm64.deb
Kernel:
https://apt.armbian.com/pool/main/l/linux-5.8.16-sunxi64/linux-image-current-sunxi64_20.08.13_arm64.deb
Firmware:
https://apt.armbian.com/pool/main/a/armbian-firmware/armbian-firmware_20.08.13_all.deb
DTB:
https://apt.armbian.com/pool/main/l/linux-5.8.16-sunxi64/linux-dtb-current-sunxi64_20.08.13_arm64.deb

 

 

Out of those 4 packages, I understand kernel, dtb, and firmware, but what is 'root'?

 

Further, looking in https://apt.armbian.com/pool/main/l/ I see a lot of linux-u-boot-* packages.

 

Sorry if these are dumb questions, I'm really out of my depth here and just don't want to go flashing random stuff without understanding.  I will be happy to update the docs (if appropriate) once I actually understand what is going on here.

 

Particularly as further diagnosis (late last night) seemed to indicate that my u-boot might be the problem.  I was able to boot a more recent image on a different sd card.  In fact Tony was suggesting I grab a newer u-boot and dd it onto the old card (following instruction here).

 

Which of course leaves the issue of how it got like that in the first place (sd card corruption, starting to wear out, etc.).  But first things first.

 

So I guess in the meantime I will at least do the filesystem check as outlined here.

OK, I did the file system check.

 

I tried many combinations:

 

$ sudo fsck /dev/mmcblk1 -f
$ sudo fsck /dev/mmcblk1 -f -b 8193
$ sudo fsck /dev/mmcblk1 -f -b 32768
$ sudo e2fsck -b 8193 /dev/mmcblk1
$ sudo e2fsck -b 32768 /dev/mmcblk1

 

They all return the same result:

 

fsck from util-linux 2.33.1
e2fsck 1.44.5 (15-Dec-2018)
fsck.ext2: Bad magic number in super-block while trying to open /dev/mmcblk1

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

Found a dos partition table in /dev/mmcblk1

 

What does it mean?  Corrupted or bad sd card?  Or just that I borked something somewhere along the way?

 

It's a 64GB Samsung Evo from known good source (B&H Photo & Video), which I did thorough test (according to Armbian recommended method) when it was new, however OTOH it is few years old now, been in service the whole time...

 

Besides that, someone in IRC (+rneese) answered my question from above.  The 'root' package is apparently the root filesystem?  If that is true, then my assumption from above that those instructions do not actually replace u-boot is correct?  I think this is relevant, as I suspect I need to replace u-boot.

 

I know, I know.  SD CARD.  However I am not convinced that is the problem.  Unless above (fsck results) clearly shows that without a doubt (I am not sure)?  Because everything was fine until I tried to upgrade my kernel through armbian-config.  I still think that I need to "surgically replace u-boot" as Tony so succinctly put it in IRC.  But I welcome feedback from anyone with more experience.

Edited November 13, 20205 yr by TRS-80
clafify

Ahem.  IT VEEEEEERRRRKS! 

 

In the end, I had to dd the u-boot into place (I got the location from sunxi wiki):

 

# dd if=u-boot-sunxi-with-spl.bin of=/dev/mmcblk1 bs=1024 seek=8

 

I got the file from https://apt.armbian.com/pool/main/l/linux-u-boot-cubietruck-current/ (of course, if you are reading later, yours may be different).

 

The advice here was very helpful.  But I still would have never got there without a little help in IRC.

 

Special thanks to @Igor, @lanefu, @TonyMac32, and @archetech, who all helped me in one way or another.  Cheers (I am actually having a cold one right now)! 

A couple things to follow up.

 

First, there was still something we were speculating about in IRC: Does Armbian upgrade u-boot some times upon upgrades?  Some thought maybe yes.  But I would not even know where to look for such information.

 

Also, I am still a little worried about the results of fsck from above.  I'm too tired right now, but I think tomorrow I will try it again, now that I have a working u-boot, just to see if I get same error.  As I have nagging worry in the back of my mind still about SD card.

3 hours ago, TRS-80 said:

Does Armbian upgrade u-boot some times upon upgrades?  Some thought maybe yes.  But I would not even know where to look for such information.

It only upgrade package that is containing u-boot but flashing must be done via armbian-config or manually. As you did. I think this part of the information is missing in the manual.

3 hours ago, TRS-80 said:

Also, I am still a little worried about the results of fsck from above. 

You referenced the raw device by "/dev/mmcblk1" where MBR , u-boot and partitions reside.
You need to reference a partition by "/dev/mmcblk1pX" where a suitable filesystem resides.
Replace "X" with the number of the partition you want to check.