1 Introduction

Isar can be shortly described as a set of bitbake recipes that implement main build system logic. To simplify overall Isar understanding, this document is split into two main parts:

Isar logical components
Isar internal processes

2 Isar Logical Components

In this chapter the most important Isar components are considered in details. In this text component doesn't especially mean some self-contained single entity of something, it's just an attempt to split Isar internals by various criteria.

2.1 Bitbake and Recipes

All the processes in Isar are started by bitbake, so it manages general build executing process. Recipes in Isar can be split in two categories:

System recipes and classes: they are responsible for setting up Debian-like infrastructure, manages Debian tools execution for package building and installation, generation root file system images
User recipes: custom user applications, that can be built from sources

There are two types of dependencies in Isar:

Dependency between bitbake recipes
Dependencies in Debian filesystem

NOTE: Isar doesn't manage dependencies in Debian file systems. User can only list specific Debian dependencies in recipe, so they eventually will be passed to apt-get. Dependency installation is managed by Debian tools.

2.2 Stamps

Each task managed by bitbake uses stamp to notify that it has been completed. Due to Isar supports various Debian distributions and parallel builds for multiple machines and architectures, the stamps for tasks use special suffixes that include:

Debian distro name
Architecture
Machine

Typical example, when Isar builds the following configurations:

Debian Stretch, amd64
Debian Stretch, i386
Debian Buster, arm

In this case there will be 3 different build subtrees. So the standard hello-isar demo application will be processed 3 times for each environment. Three different sets of stamps will be used to distinguish build progress between different bitbake configurations.

2.3 Buildchroot

One of the key aspect of Debian philosophy claims the fact, that everything in Debian should be built within Debian environment. To follow this rule, Isar introduces the new component - buildchroot. Bulidchroot is typical Debian filesystem that is created using standard Debian tools: debootstrap, apt. The source of packages can be either official Debian repositories or custom repositories created by user. Buildchroot could be one of the following type:

buildchroot-host with the host architecture for cross-compilation.
buildchroot-target with the target architecture for target native compilation.

Buildchroot lifecycle can be described as following:

Buildchroot starts out with a minimal base system that was generated by the isar-bootstrap-[host,target] recipe that uses debootstrap and apt.
During building custom Debian package, list of its build dependencies is installed to buildchroot.
When package has been built, it's installed to current buildchroot to satisfy further packages build dependencies.

2.4 Target Root Filesystem

Target filesystem is quite similar to buildchroot. The only difference is that it doesn't have development packages installed and always has target architecture.

Target filesystem lifecycle can be described as following:

Target filesystem starts, like the buildchroot, with the same minimal base system generated by the isar-bootstrap recipe.
According to the list of custom packages in bitbake recipes, the initial filesystem will be populated by successfully built packages from locally created Isar repository.

3 Isar Internal Processes

3.1 General Overview

Whole Isar build process can be split into the following steps:

Generation of a minimal base system by isar-bootstrap using debootstrap and apt for each configuration requested by the user (Debian distro, machine and architecture).
Generation of initial buildchroots for each configuration.
Generation of initial target filesystems for each configuration.
Building custom packages.
Populate target filesystems.
Generate bootable images.

All these steps are described in details below.

3.2 Minimal base system

A minimal base system is generated by the isar-bootstrap recipe found here: meta/recipes-core/isar-bootstrap/isar-bootstrap-[host,target].bb.

This recipe implements several tasks which are executed in this order:

do_generate_keyring: Generates a new gpg keyring from the gpg keys specified in the APTKEYFILES variable. This keyring is later used by debootstrap and apt to verify the apt repositories.
do_apt_config_prepare: Generates aggregated apt preferences and source files based on the DISTRO_APT_PREFERENCES and DISTRO_APT_SOURCES variables. That are later deployed to the minimal base system to be used by apt itself.
do_bootstrap: Bootstraps a minimal debian base system using debootstrap.
do_apt_config_install: Installs the apt configuration generated by do_apt_config_prepare to the base system as well as an apt base configuration that ensures no recommended or suggested package are installed automatically. This setting keeps the system minimal for all further steps.
do_apt_update: Performs an initial apt-get update and apt-get dist-upgrade on the minimal system. This is necessary because debootstrap itself only allows using one repository. This task ensures that more up to date packages, for instance from security repos are installed if they exist.
do_deploy: Creates a symlink to the directory containing the generated minimal base system, that can be used by the other recipes, that depend on isar-bootstrap.

3.3 Initial Buildchroot Generation

As mentioned above, initial buildchroot is generated using the minimal base system generated by isar-bootstrap. The bitbake recipe which is responsible for buildchroot can be found here: meta/recipes-devtools/buildchroot/buildchroot-[host,target].bb

Section 3.8 describes when host or target architecture is used, please refer to it for more details.

This recipe implements do_build task which performs the following:

Calls the setup_root_file_system helper function defined in meta/classes/isar-bootstrap-helper.bbclass that copies the minimal base system to the buildchroot working directory, sets up the isar-apt repository which will later contain all isar-build packages, calls update on just this repository and installs all packages given via a parameter to this system. Those are, in buildchroots case, the basic package building toolchain.
Install script for building custom Debian packages: meta/recipes-devtools/buildchroot/files/build.sh.
Installs and executes a configuration script, that further customizes the buildchroot. For instance it sets the locale settings.

The single stamp is created for each user buildchroot configuration.

3.4 Initial Target Filesystem Generation

Initial target filesystem generation process is very similar to buildchroot creating, the difference is only in initial packages list.

Target image recipes are the part of Isar core. There is a sample of typical Isar image that can be customized according to the user requirements: meta-isar/recipes-core/images/isar-image-base.bb.

Like for buildchroot, the configuration files for image can be found here: meta-isar/recipes-core/images/files, and it implements do_build task.

3.5 Building Custom Packages

Isar provides the possibility to build Debian packages from sources. This feature works with Debian-like source packages, i.e. the source code tree should contain debian folder. This build process is implemented in meta/classes/dpkg.bbclass. Moreover this process is common for both: native and cross compilation modes.

Another way of creating Debian packages is implemented in meta/classes/dpkg-raw.bbclass. This class can be used for customizations that are not based on source code that should become a package.

Both consist of the following steps:

Task do_fetch: fetch files from external links
Task do_unpack: unpack those files to ${WORKDIR}
Task do_install only for dpkg-raw: copy all you want in your debian package to ${D}, install hooks in ${D}/DEBIAN
Task do_prepare': perform any preparation steps to the unpacked/patched sources before the build. This task calls the dpkg_prepare shell function with the buildchroot mounts in place (dpkg_do_mounts')

4.1. the dpkg_prepare function of dpkg.bbclass runs /isar/deps.sh in

    the buildchroot. That performs the following:

    1. Go to `/home/build/${PN}`

    2. Get list of dependencies from debian/control and install them

4.2. the dpkg_prepare function of dpkg-raw.bbclass translate the

    recipe meta-data into a debian/control file suitable for packaging
    with dpkg-deb

Task do_build: mount folder (dpkg_do_mounts') with unpacked files to buildchroot, execute the actual build functiondpkg_runbuild, and finally umount again (dpkg_undo_mounts')

5.1. the dpkg_runbuild function of dpkg.bbclass runs build.sh in the

    buildchroot. That performs the following:

    1. Go to `/home/build/${PN}`

    2. Run dpkg-buildpackage

5.2. the dpkg_runbuild function of dpkg-raw.bbclass basically runs

    `dpkg-deb` to construct a Debian package from a folder of files,
without compiling anything

Task do_deploy_deb: install successfully built packages ${WORKDIR}/*.deb to deploy directory ${DEPLOY_DIR_DEB}

3.6 Populate Target Filesystem

Each target image can be extended by custom packages listed in IMAGE_INSTALL variable. Task do_populate performs the following:

Parse IMAGE_INSTALL variable.
Find respective packages in ${DEPLOY_DIR_DEB}.
Copy them to deb folder in dedicated target filesystem.
Execute dpkg command in chroot for all the copied packages.

3.7 Generate an Image

This process contains the following steps:

Task do_<..>_image: creates filesystem or disk images

1.1. do_ext4_image: target filesystem is packed to extfs image. 1.2. do_wic_image: a bootable disk image gets created for the platform

3.8 Cross-compilation

Isar provides possibility to cross-compile all or individual packages. There is variable ISAR_CROSS_COMPILE that could be set globaly for whole Isar build, but also it could be overwritten from the individual recipe to force specific compilation mode for it. This variable influences on which buildchroot will be used for recipe compilation. Isar is free from packages modification to perform cross-compilation, this is done by the same tools as for native compilation. Depending on ISAR_CROSS_COMPILE value, additional architecture specifiers are passed to build tools automatically, so this is absolutely transparent from the user point of view.

3.9 Additional features

3.9.1 Template files

A basic templating system implemented on top envsubst(1) is available. It allows limited access to bitbake variables for file generation.

This system is implemented in the template.bbclass and defines the do_transform_template task, the TEMPLATE_FILES and TEMPLATE_VARS variables. Tasks that use need to use files generated via templates need to be executed after the do_transform_template task.

The TEMPLATE_FILES variable contains a space separated list of template files and the TEMPLATE_VARS variable a space separated list of bitbake variable names. The do_transform_template task takes these and generates files using envsubst(1). The output files are placed in the same directory as the template files. The name of the output files are either the name of the template file with .tmpl removed from the file name end or, if the template file did not end with .tmpl, the name of the template file with .out added to the end.

Only template files from the WORKDIR are accepted. Either specify relative paths based on the recipes WORKDIR or absolute paths containing the WORKDIR in the TEMPLATE_FILES variable.

technical_overview.md 12 KB Riwayat Mentahan