# 1 Introduction

Isar can be shortly described as a set of bitbake recipes that implement main
build system logic. To simplify overall Isar understanding, this document is
split into two main parts:

 - Isar logical components

 - Isar internal processes

# 2 Isar Logical Components

In this chapter the most important Isar components are considered in details.
In this text component doesn't especially mean some self-contained single
entity of something, it's just an attempt to split Isar internals by various
criteria.

## 2.1 Bitbake and Recipes

All the processes in Isar are started by bitbake, so it manages general build
executing process. Recipes in Isar can be split in two categories:

 - System recipes and classes: they are responsible for setting up Debian-like
   infrastructure, manages Debian tools execution for package building and
   installation, generation root file system images

 - User recipes: custom user applications, that can be built from sources

There are two types of dependencies in Isar:

 - Dependency between bitbake recipes

 - Dependencies in Debian filesystem

**NOTE:** Isar doesn't manage dependencies in Debian file systems. User can
only list specific Debian dependencies in recipe, so they eventually will be
passed to apt-get. Dependency installation is managed by Debian tools.

## 2.2 Stamps

Each task managed by bitbake uses stamp to notify that it has been completed.
Due to Isar supports various Debian distributions and parallel builds for
multiple machines and architectures, the stamps for tasks use special suffixes
that include:

 - Debian distro name

 - Architecture

 - Machine

Typical example, when Isar builds the following configurations:

 - Debian Stretch, amd64

 - Debian Stretch, i386

 - Debian Buster, arm

In this case there will be 3 different build subtrees. So the standard
hello-isar demo application will be processed 3 times for each
environment. Three different sets of stamps will be used to distinguish
build progress between different bitbake configurations.

## 2.3 Buildchroot

One of the key aspect of Debian philosophy claims the fact, that everything in
Debian should be built within Debian environment. To follow this rule, Isar
introduces the new component - buildchroot. Bulidchroot is typical Debian
filesystem that is created using standard Debian tools: debootstrap, apt. The
source of packages can be either official Debian repositories or custom
repositories created by user. Buildchroot could be one of the following type:

 - `buildchroot-host` with the host architecture for cross-compilation.

 - `buildchroot-target` with the target architecture for target native
   compilation.

Buildchroot lifecycle can be described as following:

 - Buildchroot starts out with a minimal base system that was generated by the
   isar-bootstrap-[host,target] recipe that uses debootstrap and apt.

 - During building custom Debian package, list of its build dependencies is
   installed to buildchroot.

 - When package has been built, it's installed to current buildchroot to
   satisfy further packages build dependencies.

## 2.4 Target Root Filesystem

Target filesystem is quite similar to buildchroot. The only difference is that
it doesn't have development packages installed and always has target
architecture.

Target filesystem lifecycle can be described as following:

 - Target filesystem starts, like the buildchroot, with the same minimal base
   system generated by the isar-bootstrap recipe.

 - According to the list of custom packages in bitbake recipes, the initial
   filesystem will be populated by successfully built packages from locally
   created Isar repository.

# 3 Isar Internal Processes

## 3.1 General Overview

Whole Isar build process can be split into the following steps:

 - Generation of a minimal base system by isar-bootstrap using debootstrap and
   apt for each configuration requested by the user (Debian distro, machine and
   architecture).

 - Generation of initial buildchroots for each configuration.

 - Generation of initial target filesystems for each configuration.

 - Building custom packages.

 - Populate target filesystems.

 - Generate bootable images.

All these steps are described in details below.

## 3.2 Minimal base system

A minimal base system is generated by the isar-bootstrap recipe found here:
`meta/recipes-core/isar-bootstrap/isar-bootstrap-[host,target].bb`.

This recipe implements several tasks which are executed in this order:

1. `do_generate_keyring`: Generates a new gpg keyring from the gpg keys
   specified in the `APTKEYFILES` variable. This keyring is later used by
   debootstrap and apt to verify the apt repositories.

2. `do_apt_config_prepare`: Generates aggregated apt preferences and source
   files based on the `DISTRO_APT_PREFERENCES` and `DISTRO_APT_SOURCES`
   variables. That are later deployed to the minimal base system to be used by
   `apt` itself.

3. `do_bootstrap`: Bootstraps a minimal debian base system using debootstrap.

4. `do_apt_config_install`: Installs the apt configuration generated by
   `do_apt_config_prepare` to the base system as well as an apt base
   configuration that ensures no recommended or suggested package are installed
   automatically. This setting keeps the system minimal for all further steps.

5. `do_apt_update`: Performs an initial `apt-get update` and
   `apt-get dist-upgrade` on the minimal system. This is necessary because
   debootstrap itself only allows using one repository. This task ensures that
   more up to date packages, for instance from security repos are installed if
   they exist.

6. `do_deploy`: Creates a symlink to the directory containing the generated
   minimal base system, that can be used by the other recipes, that depend on
   isar-bootstrap.

## 3.3 Initial Buildchroot Generation

As mentioned above, initial buildchroot is generated using the minimal base
system generated by isar-bootstrap. The bitbake recipe which is responsible for
buildchroot can be found here:
`meta/recipes-devtools/buildchroot/buildchroot-[host,target].bb`

Section 3.8 describes when host or target architecture is used, please refer
to it for more details.

This recipe implements `do_build` task which performs the following:

1. Calls the `setup_root_file_system` helper function defined in
   `meta/classes/isar-bootstrap-helper.bbclass` that copies the minimal base
   system to the buildchroot working directory, sets up the isar-apt repository
   which will later contain all isar-build packages, calls update on just this
   repository and installs all packages given via a parameter to this system.
   Those are, in buildchroots case, the basic package building toolchain.

2. Install script for building custom Debian packages:
   `meta/recipes-devtools/buildchroot/files/build.sh`.

3. Installs and executes a configuration script, that further customizes the
   buildchroot. For instance it sets the locale settings.

The single stamp is created for each user buildchroot configuration.

## 3.4 Initial Target Filesystem Generation

Initial target filesystem generation process is very similar to buildchroot
creating, the difference is only in initial packages list.

Target image recipes are the part of Isar core. There is a sample of typical
Isar image that can be customized according to the user requirements:
`meta-isar/recipes-core/images/isar-image-base.bb`.

Like for buildchroot, the configuration files for image can be found
here: `meta-isar/recipes-core/images/files`, and it implements `do_build` task.

## 3.5 Building Custom Packages

Isar provides the possibility to build Debian packages from sources. This
feature works with Debian-like source packages, i.e. the source code tree
should contain debian folder. This build process is implemented in
`meta/classes/dpkg.bbclass`. Moreover this process is common for both: native
and cross compilation modes.

Another way of creating Debian packages is implemented in
`meta/classes/dpkg-raw.bbclass`. This class can be used for customizations that
are not based on source code that should become a package.

Both consist of the following steps:

1. Task `do_fetch`: fetch files from external links

2. Task `do_unpack`: unpack those files to `${WORKDIR}`

3. Task `do_install` _only_ for `dpkg-raw`: copy all you want in your
   debian package to `${D}`, install hooks in `${D}/DEBIAN`

4. Task `do_prepare': perform any preparation steps to the unpacked/patched
   sources before the build. This task calls the dpkg_prepare shell function
   with the buildchroot mounts in place (`dpkg_do_mounts')

   4.1. the `dpkg_prepare` function of `dpkg.bbclass` runs `/isar/deps.sh` in
        the buildchroot. That performs the following:

        1. Go to `/home/build/${PN}`

        2. Get list of dependencies from debian/control and install them

   4.2. the `dpkg_prepare` function of `dpkg-raw.bbclass` translate the
        recipe meta-data into a debian/control file suitable for packaging
        with dpkg-deb

5. Task `do_build`: mount folder (`dpkg_do_mounts') with unpacked files to buildchroot,
   execute the actual build function `dpkg_runbuild`, and finally umount again
   (`dpkg_undo_mounts')

   5.1. the `dpkg_runbuild` function of `dpkg.bbclass` runs `build.sh` in the
        buildchroot. That performs the following:

        1. Go to `/home/build/${PN}`

        2. Run dpkg-buildpackage

   5.2. the `dpkg_runbuild` function of `dpkg-raw.bbclass` basically runs
        `dpkg-deb` to construct a Debian package from a folder of files,
	without compiling anything

6. Task `do_deploy_deb`: install successfully built packages
   `${WORKDIR}/*.deb` to deploy directory `${DEPLOY_DIR_DEB}`

## 3.6 Populate Target Filesystem

Each target image can be extended by custom packages listed in IMAGE_INSTALL
variable. Task `do_populate` performs the following:

1. Parse IMAGE_INSTALL variable.

2. Find respective packages in `${DEPLOY_DIR_DEB}`.

3. Copy them to deb folder in dedicated target filesystem.

4. Execute dpkg command in chroot for all the copied packages.

## 3.7 Generate an Image

This process contains the following steps:

1. Task `do_<..>_image`: creates filesystem or disk images

   1.1. `do_ext4_image`: target filesystem is packed to extfs image.
   1.2. `do_wic_image`: a bootable disk image gets created for the platform

## 3.8 Cross-compilation

Isar provides possibility to cross-compile all or individual packages. There is
variable ISAR_CROSS_COMPILE that could be set globaly for whole Isar build,
but also it could be overwritten from the individual recipe to force specific
compilation mode for it. This variable influences on which buildchroot will be
used for recipe compilation. Isar is free from packages modification to perform
cross-compilation, this is done by the same tools as for native compilation.
Depending on ISAR_CROSS_COMPILE value, additional architecture specifiers are
passed to build tools automatically, so this is absolutely transparent from
the user point of view.

## 3.9 Additional features

### 3.9.1 Template files

A basic templating system implemented on top `envsubst(1)` is available. It
allows limited access to bitbake variables for file generation.

This system is implemented in the `template.bbclass` and defines the
`do_transform_template` task, the `TEMPLATE_FILES` and `TEMPLATE_VARS`
variables. Tasks that use need to use files generated via templates need
to be executed after the `do_transform_template` task.

The `TEMPLATE_FILES` variable contains a space separated list of template
files and the `TEMPLATE_VARS` variable a space separated list of bitbake
variable names. The `do_transform_template` task takes these and generates
files using `envsubst(1)`. The output files are placed in the same directory
as the template files. The name of the output files are either the name of the
template file with `.tmpl` removed from the file name end or, if the template
file did not end with `.tmpl`, the name of the template file with `.out` added
to the end.

Only template files from the `WORKDIR` are accepted. Either specify relative
paths based on the recipes `WORKDIR` or absolute paths containing the `WORKDIR`
in the `TEMPLATE_FILES` variable.
