One fairly easy task is to try our build tools in detail / with all possible scenarios and write down your thoughts about. This can help us reshape the tools and write better manual for it. Here we actually want less experienced point of view.

 

I'm currently trying out various features before going through the tools in detail and attempting to tackle all scenarios.

 

I'll write down some thoughts in this thread for now.

Initial thoughts:

- Why is root needed, and what will it do to my system? (Docs could give an overview of what it will install, etc). Is this why virtualbox etc. is recommended? (but not needed?) Is it conceivable to run as normal user, with a few sudo calls for anything needing root?

- Why does it update code (and sunxi tools etc) every time it is run? It seems to be set up for automating regular builds, but I imagine users who are trying to patch or hack on their system might have to compile several times and wouldn't want to get code while making changes. (There is the ".ignore_changes" option but it is handled differently from the other options. If it was documented, and perhaps echo'd a message that code updating was being skipped, that might suffice).

 

- Similarly, does it need to automatically clean everything? While experimenting with FEL mode, it takes many minutes compiling and prepping the image each time, even if no changes were made, and then unmounts the image when done, so it is difficult to continue experimenting outside of compile.sh.

- Why is root needed, and what will it do to my system? (Docs could give an overview of what it will install, etc). Is this why virtualbox etc. is recommended? (but not needed?) Is it conceivable to run as normal user, with a few sudo calls for anything needing root?

• Installing/updating build dependencies
• Mounting/unmounting filesystems and image files
• Using chroot
• Configuring NFS server exports for FEL mode

Since superuser rights are needed at different stages of build process and sudo tokens have timeout, it's won't be a good UX for unattended runs.

 

- Why does it update code (and sunxi tools etc) every time it is run? It seems to be set up for automating regular builds, but I imagine users who are trying to patch or hack on their system might have to compile several times and wouldn't want to get code while making changes.

Legacy, should be changed to recompiling sunxi-tools only after updates

 

There is the ".ignore_changes" option but it is handled differently from the other options. If it was documented, and perhaps echo'd a message that code updating was being skipped, that might suffice.

This is mainly for developers, to make testing intermediate (not commited/not pushed) changes (or branches other than master) faster.

 

 

While experimenting with FEL mode, it takes many minutes compiling and prepping the image each time, even if no changes were made, and then unmounts the image when done, so it is difficult to continue experimenting outside of compile.sh.

This is when I use

./compile.sh CLEAN_LEVEL=""

it still takes a few minutes to unpack rootfs cache and install kernel, headers and board support package, but in case previous run was interrupted (by user or error was encountered) you don't have to deal with image from previous run in unknown state, built for unknown device with unknown options. 

 

it still takes a few minutes to unpack rootfs cache and install kernel, headers and board support package, but in case previous run was interrupted (by user or error was encountered) you don't have to deal with image from previous run in unknown state, built for unknown device with unknown options. 

 

 

I got obsessed with getting FEL boot working for me, and ran compile.sh countless times while working with different network settings, UART, h3disp settings etc.

Meanwhile I kept hacking on the build scripts to get to FEL as quick as possible, and I remember the moment it became bearable, when I could get to fel-load in under 10 seconds (by loading a rootfs tgz that already had everything installed).

 

I thought about making my own script for FEL or to just source debootstrap-ng.sh and fel-load.sh, but since compile.sh sets up so many variables, build names etc, I figured it would be easier to just modify it. I love the idea of FEL at the moment and I know I'll use this, so:

 

Suggestions for FEL options

Edit: I think these are too messy. See simpler options below?...

 

1. Add options to skip getting any code, and compiling. Then once a rootfs tgz is built, you can skip clean and clock sync, and basically everything before debootstrap-ng.sh.

2. Add options FEL_SAVE_ROOTFS and ROOTFS_FILENAME_SUFFIX. Save a rootfs tgz after installing kernel headers etc. Better yet, I do this after the FEL boot has been done and quit, so I can boot into FEL system, configure it, install packages etc, and have that saved to run later.

3. Add an option to skip installing kernel headers etc, instead using the previously saved rootfs (chosen via ROOTFS_FILENAME_SUFFIX).

4. (Maybe) add an option to not use a tmpfs for FEL, and not delete output/cache/sdcard. With other options set, creating/loading the rootfs is skipped on subsequent runs, and I just get straight to the FEL stuff in about a second. I wanted this option because the memory cost of the tmpfs was too high while having multiple browser windows open etc.

 

 

I've made no changes to fel-load.sh and personally haven't had a problem so far with it doing all its configuration every time, no matter what other options I've set.

 

 

Using these options, I can do several things:

- Keep a persistent FEL rootfs on my host. Back it up when I want to a tgz.

- Change ROOTFS_FILENAME_SUFFIX to have multiple different FEL rootfs images saved. Optionally save them out after finishing FEL, so they are each optionally persistent.

 

One problem (at least the way I cobbled this together) is that I have to set the right combinations of options, both when I want to rebuild a rootfs and then after when I want to reuse it.

 

These hacks have been immensely useful for playing with FEL, and I suspect they'll be useful for experimenting with other build configurations, but I don't know if they'll be useful after that. It may require endless tweaks to do what I want (eg. should /boot be excluded or not when loading a saved rootfs?).

 

 

 

 

Also... it may be too early for me to suggest changes as I might keep changing everything as I figure out what I'm doing. For example the simplest way to achieve all of the above might be:

 

Simpler suggestions for FEL options

 

- Optionally (SKIP_BUILD) skip everything before debootstrap-ng.sh

- Optionally (FEL_PERSISTENT) don't use a tmpfs and don't delete output/cache/sdcard/

- Don't install kernel headers etc. if they're already there (or if SKIP_BUILD && FEL_PERSISTENT).

 

The use of a tmpfs and saving/switching the rootfs images can be done manually if needed.

1. Add options to skip getting any code, and compiling. Then once a rootfs tgz is built, you can skip clean and clock sync, and basically everything before debootstrap-ng.sh.

CLEAN_LEVEL="" effectively skips cleaning and rebuilding u-boot and kernel; building board support package is usually fast enough, but in the future it should be skippable with same logic as kernel/u-boot. Syncing clock may be useful when using VMs, otherwise you can put SYNC_CLOCK=no in userpatches/lib.config to skip it. As for downloading source code/checking for updates - I'll try to make this skippable in the future.

 

2. Add options FEL_SAVE_ROOTFS and ROOTFS_FILENAME_SUFFIX. Save a rootfs tgz after installing kernel headers etc. Better yet, I do this after the FEL boot has been done and quit, so I can boot into FEL system, configure it, install packages etc, and have that saved to run later.

This mode was originally designed for one-time tests, if you want to constantly use same rootfs with FEL, your best bet now would be writing a standalone script that will use fel-load.sh, it shouldn't require setting too many variables

 

4. (Maybe) add an option to not use a tmpfs for FEL, and not delete output/cache/sdcard. With other options set, creating/loading the rootfs is skipped on subsequent runs, and I just get straight to the FEL stuff in about a second. I wanted this option because the memory cost of the tmpfs was too high while having multiple browser windows open etc.

You can always set FORCE_USE_RAMDISK=no to use real filesystem instead of tmpfs.

 

As for persistent rootfs with installed headers and kernel - I'll think about the best way to do it without making too many changes and introducing too many new configuration variables. Again, if you are playing with a single board and not changing kernel, u-boot and other things in the process, writing your own script based on existing code may be the fastest way right now.

I'll think about the best way to do it without making too many changes and introducing too many new configuration variables.

 

Your suggestions sound good. Minimal changes and then maybe a "howto" in the documentation should allow non-experts to do all this. I could write that later if I get it figured out enough.

Patching

 

If I build Armbian, it will patch the source files. If I edit one of the patched files, and run compile.sh again, advanced_patch function will silently overwrite my changes.

 

I believe this is what's described here, with a work-around: http://www.orangepi.org/orangepibbsen/forum.php?mod=viewthread&tid=1707

"You need make changes to 4 files, but make changes directly to the kernel source is not allowed by default. Because compilation scripts will revert all changes  . So we need make patches:"

 

I expected being able to modify code and rebuild.

If I build Armbian, it will patch the source files. If I edit one of the patched files, and run compile.sh again, advanced_patch function will silently overwrite my changes.

 

I believe this is what's described here, with a work-around: http://www.orangepi.org/orangepibbsen/forum.php?mod=viewthread&tid=1707

"You need make changes to 4 files, but make changes directly to the kernel source is not allowed by default. Because compilation scripts will revert all changes  . So we need make patches:"

 

I expected being able to modify code and rebuild.

There are some options that control this behavior: first, FORCE_CHECKOUT=no should disable overwriting modified source files, second, DEBUG_MODE=yes (it's a really non-descriptive name for this option that should be changed) allows creating patches based on your modifications to the sources.

A few small hangups... They might be mentioned in documentation (I could in the near future), if there's nothing to fix.
 
 
If RELEASE variable is set, no dialog option for BUILD_DESKTOP is presented. They both must be set, or you get no desktop.
 
 
Toolchains:
Are these all needed? Are they all downloaded because it's easier than figuring out which are needed? Are the different versions of gcc-linaro-*-x86_64_arm-linux-gnueabihf redundant? If they're not needed, a docs section on customizing the scripts could mention commenting them out.
 
 
I had an odd situation where one of the toolchains was downloaded each time I ran compile.sh. I think the download was ending prematurely???, even though curl displayed 100.0% progress. The gpg verify failed, and there was a warning displayed, but it was quickly (half second maybe?) pushed offscreen by a full-screen option dialog. I assume this is rare enough to not justify any changes.
 
 
NFS:
The first time I try to boot FEL, it doesn't work because the nfs-kernel-server isn't started yet.
 
After the script finished, I ended up with a lot of memory being used until I stop the server. Edit: This must have been an unusual circumstance as it didn't happen upon retesting. Maybe it only happens with tmpfs, maybe only if the board isn't shut down properly?

Toolchains:

Are these all needed? Are they all downloaded because it's easier than figuring out which are needed? Are the different versions of gcc-linaro-*-x86_64_arm-linux-gnueabihf redundant? If they're not needed, a docs section on customizing the scripts could mention commenting them out.

 

I guess this part is not fully optimised and tool-chains are downloaded even they might not be needed but on the other hand it meets the philosophy - first the whole environment is prepared, than we proceed to source downloading, patching and compiling. We could rework that tool-chain is downloaded if / when source need a special one. Once, since this is not so urgent.

 

I had an odd situation where one of the toolchains was downloaded each time I ran compile.sh. I think the download was ending prematurely???, even though curl displayed 100.0% progress. The gpg verify failed, and there was a warning displayed, but it was quickly (half second maybe?) pushed offscreen by a full-screen option dialog. I assume this is rare enough to not justify any changes.

 

This could be a bug - can you reproduce?

 

If RELEASE variable is set, no dialog option for BUILD_DESKTOP is presented. They both must be set, or you get no desktop.

 

Also need an inspection.

 

In any case, feel free to edit docs repository if / when you find out that something from those findings above belong there.

 

This could be a bug - can you reproduce?

 

Yes, I've seen that too.

In a build log, I found that error :

[[[0;32m o.k. [[0m] Checking toolchain [[[0;33m gcc-linaro-arm-linux-gnueabihf-4.8-2014.04_linux [[0m]
[[[0;32m .... [[0m] Downloading
[[[0;32m .... [[0m] Verifying
[[[0;35m warn [[0m] Verification failed

If RELEASE variable is set, no dialog option for BUILD_DESKTOP is presented. They both must be set, or you get no desktop.

Should be fixed: https://github.com/igorpecovnik/lib/commit/9fbbb4448763fac50feb132f2427e87c0b89d1bc

 

Toolchains:

Are these all needed? Are they all downloaded because it's easier than figuring out which are needed? Are the different versions of gcc-linaro-*-x86_64_arm-linux-gnueabihf redundant? If they're not needed, a docs section on customizing the scripts could mention commenting them out.

Some toolchains are needed for different u-boot or kernel versions. Since they should be downloaded only once and they don't take too much disk space, it's the easiest way to solve all requirements.

 

I had an odd situation where one of the toolchains was downloaded each time I ran compile.sh. I think the download was ending prematurely???, even though curl displayed 100.0% progress. The gpg verify failed, and there was a warning displayed, but it was quickly (half second maybe?) pushed offscreen by a full-screen option dialog. I assume this is rare enough to not justify any changes.

GCC 4.8 is the only toolchain verified by GPG detached signature rather than by MD5 checksum, it's possible that check is failing due to missing GPG or some other errors. Hardcoding MD5 checksum in this case may be an option if we don't figure out what causes this error.

GCC 4.8 is the only toolchain verified by GPG detached signature rather than by MD5 checksum, it's possible that check is failing due to missing GPG or some other errors. Hardcoding MD5 checksum in this case may be an option if we don't figure out what causes this error.

 

 

I can't reproduce the toolchain download error. I should have copied the messages and directory lists.

However I remember the file gcc-linaro-arm-linux-gnueabihf-4.8-2014.04_linux.tar.xz being about 41Mb when it should have been 48 or something.

Come to think of it... I only remember *one* message showing

######################################################################## 100.0%

If that was really the case, it's likely that that was for the .asc file and I mistakenly thought it claimed to download the whole .xz file.

Unfortunately I don't recall if there was any output before that, for the .xz file, showing an incomplete download or not.

 

I figured I had network problems or there was a server problem.

curl is using the "-f" flag so it might have hidden a fail message.

 

I can repro a download error simply by unplugging the network for a minute while downloading. curl didn't resume downloading when it was plugged back in, and about 10 minutes later it failed. However, it failed with an error message:

############                                                              17.4%
curl: (56) SSL read: error:00000000:lib(0):func(0):reason(0), errno 104

The script displayed "[ warn ] Verification failed" and continued running.

The remaining outstanding issues I've seen so far involve trying to use the scripts for experimentation and kernel hacking. My impression of the scripts is that they weren't meant for this, instead for automating building known good images, but I think it's appropriate for someone who wants to hack on the build to also expect to hack on the build scripts. The scripts do maybe 95% of what I want, and it would be a nightmare to try to get started making a custom build without them.

 

 

Stuff I immediately hacked into the scripts after downloading a fresh copy:

- Option to keep the rootfs persistent for FEL, and another option to control when to (re)create the rootfs.

 

Desired things I haven't got around to:

- A separate script for cleaning various things (including build output, downloads, mounts, nfs server etc), with dialog options.

- compile_kernel when code changes instead of when deb doesn't exist. Or, figure out quickest way to compile kernel without building too much else.

- A doc to help with pulling useful commands out of the scripts to use in standalone scripts.

- Documenting suggested hacks

 

 

 

 

Docs:

- Wherever something is recommended (eg. virtualbox, DHCP server and UART console monitor for FEL), it would be helpful if it explained why, or what will be missed by anyone skipping the recommendation. 

- compile_kernel when code changes instead of when deb doesn't exist. Or, figure out quickest way to compile kernel without building too much else.

This depends on how do you define "code changes". Kernel source can be changed by maintainer provided patches, user provided patches or from upstream. it's easier to adjust CLEAN_LEVEL on command line if you know that code was changed. And I wouldn't call u-boot and board support package "too much" compared to kernel (by compilation/building time required).

 

Docs:

- Wherever something is recommended (eg. virtualbox, DHCP server and UART console monitor for FEL), it would be helpful if it explained why, or what will be missed by anyone skipping the recommendation. 

Virtualbox (or other virtualization software) is required for people who use Windows as "main" desktop OS; for people who use modern Linux distributions containerization (Docker, systemd-nspawn) will provide much better performance.

 

DHCP is required to use FEL boot with minimum additional configuration; for using static addresses you will have to provide network details (at least IP, netmask and NFS server IP)  in custom u-boot script (userpatches/fel-boot.cmd). And using this approach for multiple devices will be multiple times harder.

 

UART console IMO is just more convenient than using keyboard and monitor or SSH, where you'll have to know IP address of freshly booted board by checking leases of your DHCP server

This depends on how do you define "code changes". Kernel source can be changed by maintainer provided patches, user provided patches or from upstream. it's easier to adjust CLEAN_LEVEL on command line if you know that code was changed. And I wouldn't call u-boot and board support package "too much" compared to kernel (by compilation/building time required).

I was thinking of user edits of source files, and of wanting to treat compile.sh like make.

 

Virtualbox (or other virtualization software) is required for people who use Windows as "main" desktop OS; for people who use modern Linux distributions containerization (Docker, systemd-nspawn) will provide much better performance.

If Ubuntu is your main system, neither is needed, but is it still recommended to keep your system clean or for security?

 

UART console IMO is just more convenient than using keyboard and monitor or SSH, where you'll have to know IP address of freshly booted board by checking leases of your DHCP server

Also if the boot fails before video and networking are initialized, it may be the only way(?) to see status/failure messages.

If Ubuntu is your main system, neither is needed, but is it still recommended to keep your system clean or for security?

Small correction: If Ubuntu Xenial x64 is your main system (or you use dedicated "build server") - yes, you can run build script natively, if you don't care about installing build dependencies and know how to fix issues that may appear (like ccache owner/group issues). Regarding security - there always is a possibility that some build script feature wasn't tested properly and it may cause some issues on host system, in this case container is more secure.

nand-sata-install.sh exits if it's not run from an SD card, and later checks the root filesystem for a few supported types, it looks like.

Is there any reason this script couldn't support FEL/NFS-root? It seems like it would be the ultimate least-cost solution to failures due to bad SD cards.

 

(However, FEL is fairly advanced and I don't know how many people go for a board with emmc but don't have a good sd card to spare. Is there a place where potential features are listed and people can vote or express interest, to see if such features are worth doing?)

 

 

 

 

DEBUG_MODE runs "git add" and I unexpectedly end up with a lot of files to be committed, after the build script is done. Is this supposed to happen? Would this have happened because of some unhandled state I'm in, such as having outdated code (using FORCE_CHECKOUT="no") and the git add + git reset then not behaving as it was intended? Does this need further investigation?

 

Since it's doing a git add, and it seems to want me to edit code only(?) in the middle of running the build script, I'd assumed that DEBUG_MODE was intended to be used to patch the code using known-working fixes, and not to iteratively write/compile/test changes. Is this true?

DEBUG_MODE is intended to create a (kernel, u-boot) patch out of changes. The name might not be completely adequate to what it does. But it does the job perfectly well and force checkout is a must for proper functioning. Perhaps we could do one IF sentence to checkup this.