[Armbian build PR] - hashed-OCI-revisioned-debs: introduce "reversioning" of .deb packages

hashed-OCI-revisioned-debs: introduce "reversioning" of .deb packages

tl-dr:

• maximize OCI cache hit ratio across nightlies/releases/PRs/etc;
• publish simple Version:'s that don't include a crazy hash in repo and images
• introduce output/packages-hashed directory
• radically change the output/debs directory structure

• trap-logging: just discard logs dir when ANSI_COLOR=none
• repo-reprepro.py: set -e & pipefail
• prepare_host: handle PRE_PREPARED_HOST inside prepare_host_init() so WORKDIR etc is always available
• kernel-debs: avoid showing tree of kernel modules if they've not been built
  • for example EXT=nomod causes no modules to be built
• debs: all produced debs now get a placeholder DEBIAN/changelog and a usr/share/doc .gz changelog with hash
• artifacts: don't keep deb-tar's .tar after uploading to OCI
  • they won't ever be used after this, and just accumulate trash for no reason
• hashed-OCI-revisioned-debs: get completely rid of artifact_prefix_version
• hashed-OCI-revisioned-debs: build debs in PACKAGES_HASHED_STORAGE, not DEB_STORAGE (temp commit, will be rewritten by a later commit)
  • fakeroot_dpkg_deb_build() now only takes a single argument, the unpacked package dir
• kernel/tmpfs: remove usage of the kernel-specific tmpfs for temporary kernel .deb's, now build directly to packages-hashed dir
• uboot: drop CHOSEN_UBOOT, change deployment directory, remove uboot_name as temp_dir prefix
  • change deployment dir to not include REVISION or ARCH
  • get rid uboot's CHOSEN_UBOOT, REVISION and ARCH in the directory name.
  • no two u-boot debs can be installed in the same machine anyway
• pipeline: output-debs-to-repo-json.py: adapt to new reversioned JSON info
• hashed-OCI-revisioned-debs: introduce "reversioning" of .deb packages
  • simplify artifact's prepare_version() method for deb and deb-tar artifacts:
    • artifact_base_dir and artifact_final_file will now be auto-calculated; thus removed from each artifact (except rootfs)
    • artifact_deb_repo ("global", "jammy", "bookworm") is now required; "global" means common across all RELEASES
    • artifact_deb_arch is now required, "all" is arch-independent, otherwise use ${ARCH}
    • artifact_map_debs is now auto-calculated based on the above, and shouldn't be specified manually
    • artifact_final_version_reversioned is optional, and can force the final version of the artifact (specific for the base-files case)
    • artifacts that need special handling for reversioning can add function names to artifact_debs_reversion_functions array (base-files and bsp-cli cases)
    • artifacts prepare_version() should set artifact_version, but never include it in other variables; artifact_version is now changed by framework after prepare_version() returns
  • no longer use/refer/mention ${REVISION} when building packages. All packages should be ${REVISION}-agnostic.
  • ${REVISION} (actually, artifact_final_version_reversioned) will be automatically swapped in the control file during reversioning
  • fakeroot_dpkg_deb_build() now takes exactly two arguments: the directory to pack, and the deb ID (key of artifact_map_packages dict); add this change in all the artifact's code for this
  • obtain_complete_artifact():
    • automatically adds -Rxxxx "revisioning-hash" to artifact_version, by hashing the revisioning functions and any artifact_debs_reversion_functions set
    • calculates more complex subdirectory paths for both the output/packages-hashed and output/debs/output/debs-beta directories
      • with the new subdirectories we can be sure a re-version is already done correctly and can skip it (eg, for partial download-debs re-runs)
      • in the future we can automatically clean/remove old versions that are no longer relevant based on the dir structure
      • exports a lot more information to JSON, including the new subdirectory paths
    • comment-out code that implemented skip_unpack_if_found_in_caches, I'm very unsure why we had this in the first place
  • obtain_artifact_from_remote_cache()
    • for deb type artifacts, OCI won't preserve the subdirectory structure, so move downloaded files to the correct subdirectory manually
    • this is not needed for deb-tar, since that can preserve the dir structure itself
  • introduce artifacts-reversion.sh and its main function artifact_reversion_for_deployment()
    • this has the logic for reversioning .deb's, by ar-unpacking them, changing control.tar (and possibly data.tar), handling .xz compression, etc.
    • also handles hashing those functions, for consistency. Any changes in reversioning code actually change the artifact itself so we're not caught by surprise
    • by default, it changes control file only:
      • replace Version: (which is the hash-version originally) with artifact_final_version_reversioned (which is mostly just ${REVISION})
      • add a custom field Armbian-Original-Hash: with the original hash-version
    • artifact_reversion_for_deployment() is called by
      • new CLI wrapper cli_obtain_complete_artifact(), used for CLI building of specific artifact, but also for download-artifact
      • build_artifact_for_image() used during image build
  • armbian-bsp-cli-deb.sh: move ${REVISION} related stuff from the main package build to new reversioning functions.
  • artifact-armbian-base-files.sh: move ${REVISION} related stuff from the main package build to new reversioning functions.
  • kernel:
    • add some custom fields to DEBIAN/control:
      • Armbian-Kernel-Version: / Armbian-Kernel-Version-Family: (for future use: cleanup of usage of Source: field which should be removed)
    • declutter the Description: field, moving long description out of the first line
    • obtain IMAGE_INSTALLED_KERNEL_VERSION from the reversioned deb (this is still a hack and has not been fixed)
  • uboot:
    • declutter the Description: field, moving long description out of the first line
    • use the reversioned .deb when deploying u-boot to the image
  • main_default_build_packages() now stores reversioned values and complete paths to reversioned .deb's
  • list_installed_packages() now compares custom field Armbian-Original-Hash: , and not the Version: to make sure debs in the image are the ones we want
  • install_artifact_deb_chroot() is a new wrapper around install_deb_chroot() for easy handling of reversioned debs
    • use it everywhere install_deb_chroot() was used in distro-agnostic.sh and distro-specific.sh

Signed-Off-By: Ricardo Pardini ricardo@pardini.net

View the full article