Hello, everyone,

 

I found a very serious issue with Armbian build framework: build may corrupt your host's rootfs!

This has happened to my system 3 times: my rootfs has been totally destroyed without being able to recover it (I used a number tools, like testdisk, etc. All of the tools detected some deleted files but all of those files appeared to be 0-sized).

I have delayed reporting this to be 100% sure that it is Armbian build that caused those issues. Now I am, absolutely, because such a problem NEVER happened to me before (I have been using Ubuntu 16.04 for a long time).  After 2nd crash I changed my HDD to a brand new one and installed all again from beginning. 

Unfortunately, I can't give you many details about what actually happened at crash time because all files in armbian's "build" folder have been lost without being able to recover, as I said above.

Here is only a few things I can say about it:

1) This has never happened at 1st build on a clean system (always after a number of successful builds). I am not sure if this is related to the problem but I have periodically deleted some cache files (i.e. build/cache/rootfs and  output/debs) between builds.

2) The console says that build script can't remove some files in the output folder (don't remember exactly)

3) At this time no more shell commands are working:  mkdir, ls, cd, etc - all of them are failing, saying that there is no such tool avaliable

4) Ubuntu is stopping to respond to any UI actions, like open apps, or shutdown button from top panel. 

5) After hard reboot Ubuntu can't boot anymore, showing a problem with grub.

 

I use Armbian build framework intensively (nearly every day), I have done a lot of customization on top of it. I like the framework, mainly because of your outstanding efforts to support it, and it is the best option for my project now. Hope you will find some time to look at possible reasons of the issue. 3 total system breakdowns during a period like 3-4 months - that's too many. 

 

Thanks.

 

Wrong title. Your customizations caused that!

I never crashed my host rootfs. The script is running with root privileges and you have to understand what you are doing.

 

 

3 minutes ago, ssuloev said:

I have delayed reporting this to be 100% sure that it is Armbian build that caused those issues.

 

3 minutes ago, ssuloev said:

I use Armbian build framework intensively (nearly every day), I have done a lot of customization on top of it.

Until you reproduce this without any customizations (especially if you are using customize-image-host.sh) and provide a console log or any other kind of output that shows error messages we will think that this is not related to our build script. I can easily create a "customization" that will kill the build host on the first run, but it doesn't mean that our build script has issues.

Thanks for reply.

I understand your point, but I don't do anything special in the customization, only apt-get install. Do you think that may produce such an issue ??

And seems like it happens before any customization, at the time build script is packaging kernel or  board-specific packages.  But again I am not sure, because it happens unexpectedly every time. No logs are available after the crash, no way to recover them. The only thing I can do next time is to somehow remember the console output... I am not sure that clipboard will work.

In my kernel patches I am mainly modifying target DTS files.

22 minutes ago, Igor said:

Wrong title. Your customizations caused that!

I never crashed my host rootfs. The script is running with root privileges and you have to understand what you are doing.

 

 

OK, I will review again my customization. But I have no idea how simple commands like apt-get in chroot may cause such an issue... And one even more interesting question: why this happens only sometimes ??

If you are not confident about the build process executed directly on your host, you can install a Ubuntu VM on your host, and doing a backup of that VM once in a while. If your issue comes back, you can easily restore the VM.

 

... or if you are running Ubuntu already you could use Docker, though for "a lot of customizations" Virtualbox may be better.

I remember a long time ago similar issues were reported for desktop Ubuntu build host, but we are mostly using CLI/server ones (AFAIK) and server image is listed as a build environment (though not explicitly). It's possible that desktop services (like auto mounting filesystems) may interfere with the build process.

I am using Desktop one... That's sad.

I just created a separate partition to build Armbian. As soon as I get the issue again I will come back with more details.

Thanks.

2 hours ago, martinayotte said:

If you are not confident about the build process executed directly on your host, you can install a Ubuntu VM on your host, and doing a backup of that VM once in a while. If your issue comes back, you can easily restore the VM.

 

Thanks, that makes sense.

Today happened again with exactly the same scenario.(((

Ok, I finally got to understanding that this may happen due to bugs in custom code constructing file/dir paths.