rockpi-S crashing at startup with memory errors

There seems to be some issue with the very early bootup on the rockpi-S using the dev or current target (not the 4.4 kernel) It hangs and requires a reboot, something like every 4th bootup works. I have a bunch of logs with various errors like:

Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: add_uevent_var+0x12c/0x130
[ccff800010110d58] address between user and kernel address ranges
Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: vsnprintf+0x724/0x728

This usually happens about 2.7 - 2.8 seconds into bootup. I have been building from source with a custom config and this (or something similar) and I believe 5.4 was the last stable kernel in my testing.

Reproduction:

Start with Armbian_20.08.1_Rockpi-s_buster_current_5.8.6_minimal.img

Change add 'earlycon debug' to extrags in /boot/armbianEnv.txt

Start (observe with serial connection).

When the board doesn't boot, the blue LED never starts blinking.