Yocto: Unravel the Way Files Take When Building Software With BitBake

Introduction

Ever wondered where files go when you build software with bitbake?

Well, I have. The build directory seemed to be a mess at the beginning. While the OpenEmbedded Build System Concepts [1] section of the Yocto Project Overview and Concepts Manual [2] does a really good job of explaining the general concept, I still found it difficult to really understand all the paths in the build directory. That’s why I thought, I would do the following:

• Look at a concrete example.
• Run through the different tasks needed to build the software.
• Explain where to find the relevant bitbake variables.
• Show where different files end up in the build directory.

I hope this will shine a light on the apparent mess in the build directory.

To do so, I started out by setting up an example project directory. This was done in accordance to the directory structure I suggested in a previous post. I wanted to keep the example relatively simple, yet it should include a little more then just the “default” layers that poky supplies. Since I’m working on a project that is supposed to run on a BeagleBone Black board, I include the meta-ti and the meta-arm layers from the OpenEmbedded Layer Index [3]. The latter one was include, because meta-ti depends on it.

So I started out with an example project directory that looked like this:

$ cd <path/to/example/project>
$ tree -L 2
.
├── bitbake
│   ├── downloads
│   └── sstate-cache
├── build
│   └── conf
├── layers
│   ├── meta-arm
│   └── meta-ti
├── poky
│   ├── (...)           // output truncated
│   └── SECURITY.md
└── README.md

Note: For those interested in the exact setup: meta-arm, meta-ti, and poky were added as git submodules to the project. I used the kirkstone branch for all these submodules. The specific commits are:
meta-arm: 98cf1495dbca946b8c6a82efdfce0afb63ff41e2
meta-ti: 155218f03ee8222eeb02f11ea9bc41135cf28e38
poky: d8d6d921fad14b82167d9f031d4fca06b5e01883

Besides some other stuff that is not relevant for this post, the file ./build/conf/local.conf contains the following variables:

MACHINE ?= "beaglebone"
DL_DIR ?= "${TOPDIR}/../bitbake/downloads"
SSTATE_DIR ?= "${TOPDIR}/../bitbake/sstate-cache"
DISTRO ?= "poky"
PACKAGE_CLASSES ?= "package_rpm"

The file ./build/conf/bblayers.conf looks like this:

# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  /home/andreas/projects/active/beagle/yocto/poky/meta \
  /home/andreas/projects/active/beagle/yocto/poky/meta-poky \
  /home/andreas/projects/active/beagle/yocto/poky/meta-yocto-bsp \
  /home/andreas/projects/active/beagle/yocto/layers/meta-arm/meta-arm-toolchain \
  /home/andreas/projects/active/beagle/yocto/layers/meta-arm/meta-arm \
  /home/andreas/projects/active/beagle/yocto/layers/meta-ti/meta-ti-bsp \
  "

All right, now let’s look at an example software package and build it with bitbake. For whatever reason I decided to look at the package openssl for this post.

Build openssl with Bitbake

After setting up the project, you could just do the following to build and package openssl:

cd <path/to/example/project>
source poky/oe-init-build-env build/
bitbake openssl

But to demonstrate what’s going on during the build process, I skipped the last command. In the following, I will go through the build process in small steps by running the individual tasks that are needed to build openssl. You can run bitbake -c listtasks openssl to see all available tasks related to the openssl recipe. Not all of them are needed to build openssl. I will just go through the ones relevant to the build process.

Note: You can find the recipe that build openssl in <path/to/example/project>/poky/meta/recipes-connectivity/openssl/. The recipe file is called openssl_3.0.12.bb.

OK, let’s finally get started.

Note: All the following commands are run from within the build directory inside the example project directory. If you want to follow along make sure that you have run the first two commands from the previous command listing. The current working directory should be <path/to/example/project/build.

do_fetch

The first command you need to run is:

bitbake -c fetch openssl

This command fetches all the source files necessary to build the software. If you actually start from scratch, it will actually do a little more than just fetching the source code for the particular software (openssl in this case). I will ignore that “little more” for now. The really interesting result of this command is a new file called: openssl-3.0.12.tar.gz:

$ tree ../bitbake/downloads
../bitbake/downloads/
├── (...)    // output truncated
├── openssl-3.0.12.tar.gz
└── openssl-3.0.12.tar.gz.done

Note: Remember we are in the build directory. That’s the reason why we had to use ../bitbake/downloads.

So, how did bitbake know what to download?

Well, if you take a look into the recipe file (openssl_3.0.12.bb), you will find the following lines:

SRC_URI = "http://www.openssl.org/source/openssl-${PV}.tar.gz \
           file://run-ptest \
           file://0001-buildinfo-strip-sysroot-and-debug-prefix-map-from-co.patch \
           file://afalg.patch \
           file://0001-Configure-do-not-tweak-mips-cflags.patch \
           "

The first line tells bitbake what to download. The ${PV} variable in the path is not explicitly set in the recipe, but taken from the recipe file name (“3.0.12”).

And why does the downloaded file end up in ../bitbake/downloads/?

This is determined by the $DL_DIR variable set in conf/local.conf (see Introduction).

do_unpack

Next, the source files have to be unpacked. This can be done with:

bitbake -c unpack openssl

Where do the unpacked files end up?

If you look into the Yocto Project Overview and Concepts Manual [2] (specifically section “4.3.5.1 Source Fetching”), you will see that two directories become important here. They are set in bitbake’s variables WORKDIR and S.

Before explaining were these two variables are set, let’s look at the result of the previous command. For the openssl recipe, the WORKDIR variable is set to tmp/work/armv7at2hf-neon-poky-linux-gnueabi/openssl/2.0.12-r0 within the build directory. It is actually set as an absolute path, I just skipped the beginning to safe some space. If you look into that directory, you will find the following:

$ ls tmp/work/armv7at2hf-neon-poky-linux-gnueabi/openssl/2.0.12-r0
openssl-3.0.12         0001-buildinfo-strip-sysroot-and-debug-prefix-map-from-co.patch
recipe-sysroot         0001-Configure-do-not-tweak-mips-cflags.patch
recipe-sysroot-native  afalg.patch
source-date-epoch      run-ptest
temp

On the left hand side you can see five directories which I will not discuss for now. On the right hand side you can see the four files. These are the four files that were additionally set in the SRC_URI variable in the recipe file (see do_fetch). The do_unpack task didn’t download them. It found them under ../poky/meta/recipes-connectivity/openssl/openssl/ and copied them into the WORKDIR.

The S variable is set to the openssl-3.0.12 directory in the WORKDIR. Again, this is actually done as an absolute path, but I shortened it to safe some space. If you look into the S directory, you will find the decompressed source code that was compressed into the ../bitbake/downloads/openssl-3.0.12.tar.gz archive.

All right now let’s look into how WORKDIR and S are set. It’s a little complicated (at least for WORKDIR).

WORKDIR

Looking into the Variable Glossary [4] reveals that the WORKDIR for a recipe is defined as:

${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}

In our case, this unfolds to be the tmp/work/armv7at2hf-neon-poky-linux-gnueabi/openssl/3.0.12-r0 within the build directory.

But how do we get there?

Well, that’s a lot to unpack. Let’s do it! If not otherwise noted, the following information comes from the Variable Glossary [4]. Additionally, when trying to analyze something like such bitbake variables, it is helpful to get bitbake’s environment for a recipe. I temporarily saved it into a file called environment.txt by running:

bitbake -e openssl > environment.txt

Note: In the following, I will refer to the environment.txt file as the “environment file”.

Let’s dissect the WORKDIR variable piece by piece:

• TMPDIR

  If not specifically set in conf/local.conf, the default is ${TOPDIR}/tmp. TOPDIR is the build directory. Since I didn’t explicitly set TMPDIR it is the tmp directory inside the build directory.

• MULTIMACH_TARGET_SYS

  This variable is set to ${PACKAGE_ARCH}${TARGET_VENDOR}-${TARGET_OS}. Well, another variable to take apart:

  • PACKAGE_ARCH

    Some recipes set this variable explicitly (e.g. to MACHINE_ARCH). This is not the case here. If it is not set explicitly in the recipe, PACKAGE_ARCH is set to TUNE_PKGARCH in ../poky/meta/conf/bitbake.conf. All right, let’s find out where TUNE_PKGARCH is defined. Bear with me 💪:

    • TUNE_PKGARCH

      This is were it gets a little complicated. If you look into the environment file, you will find the following line:

      TUNE_PKGARCH="armv7at2hf-neon"
      

      Immediately above that line you can find a “pre-expansion value” that looks like this:

      ${ARMPKGARCH}${ARMPKGSFX_THUMB}${ARMPKGSFX_DSP}${ARMPKGSFX_EABI}${ARMPKGSFX_ENDIAN}${ARMPKGSFX_FPU}
      

      And above that, some options where that value might come from. Analyzing this part of the environment file reveals it was set in line 12 of ../poky/meta/conf/machine/include/arm/arch-arm.inc. I trust that you can verify that for yourself.

      I won’t go into further detail on where to find these new variables. I hope, after reading this post, you will have a good idea how to analyze that yourself. But I will explain why arch-arm.inc plays a role in setting TUNE_PKGARCH.

      So, why is arch-arm.inc relevant?

      If you remember, I set the MACHINE variable to beaglebone in conf/local.conf (see Introduction). This causes bitbake to look for a machine configuration file called beaglebone.conf and it finds it in ../layers/meta-ti/meta-ti-bsp/conf/machine/. This file has the following line in it:

      require conf/machine/include/ti33x.inc
      

      The file ti33x.inc can be found in ../layers/meta-ti/meta-ti-bsp/conf/machine/include/ and in turn requires conf/machine/include/arm/armv7a/tune-cortexa8.inc. Since bitbake searches in all layer directories that are set in the BBLAYERS variable in conf/bblayers.conf (see Introduction), it finds the required file under ../poky/meta/conf/machine/include/arm/armv7a/.

      What follows now is a trace of required files. Starting point is tune-cortexa8.inc. It requires the next file in the list and so forth until the last file will finally be arch-arm.inc´. I will just list the names of the required files. All these file can be found under ../poky/meta/conf/machine/include/arm/`.

      • tune-cortexa8.inc
      • arch-armv7a.inc
      • arch-armv6.inc
      • arch-armv5-dsp.inc
      • arch-amrv5.inc
      • arch-armv4.inc
      • arch-arm.inc

      Please verify this trace for yourself if you want. Have fun 😉!

  • TARGET_VENDOR

    Since I set the DISTRO variable to poky in conf/local.conf (see Introduction), the TARGET_VENDOR variable is set to -poky in ../poky/meta-poky/conf/distro/poky.conf. Looking into the environment file gave a first clue on where to look 😀.

  • TARGET_OS

    This variable is set in ../poky/meta/conf/bibake.conf to the value: linux${LIBCEXTENSION}${ABIEXTENSION}. Again, I started by looking into the environment file. As with TUNE_PKGARCH, I won’t go any deeper in explaining where the contained variables come from. By now, you should have a good idea on how to find them yourself.

• PN

  This variable is usually extracted from the recipe file name. In our case it is set to openssl. More details can be found in the Variable Glossary [4].

• EXTENDPE

  This variable has to do with the epoch of a recipe. In our case it is just set to an empty string. More details can be found in the Variable Glossary [4].

• PV

  This is the package/recipe version and is normally extracted from the file name of the recipe. In our case it is set to 3.0.12. More details can be found in the Variable Glossary [4].

• PR

  This is the revision of the recipe. The default value is r0 and that is the value that is used here. The Variable Glossary [4] explains how it is set.

S

Compared to the WORKDIR variable, finding out what S is set to is quite easy. Some package recipes set this variable explicitly. In such a case you would find it in the recipe file. That is not the case with openssl. As it is not set in the recipe itself, the definition from ../poky/meta/conf/bitbake.conf is used. There it is set to ${WORKDIR}/${BP}. BP on the other hand is set to ${BPN}-${PV}. And these two variables are also set in bitbake.conf by means of so called “executable meta data”. BPN is basically the package name and PV the package version (see the Variable Glossary [4]).

do_patch

After unpacking the source code, the next step is to patch the sources (if necessary). The recipe for openssl does define some patches as seen earlier. Running bitbake -c patch openssl may take longer than you might think. This is because bitbake first needs to build some tools that are needed to do the actual patching. But once the command finishes, the patches we saw earlier have been applied. More details about the do_patch task can be found in the Yocto Project Overview and Concepts Manual [2] (specifically in the section “4.3.5.2 Patching”).

do_configure

The do_configure task configures the software. This is pretty specific to the software being build. In the case of openssl, the recipe defines the whole task. One of the main steps it takes is running a perl script that can be found here: ${S}/Configure. The do_configure task is the first time the B variable becomes relevant. It contains the path to where the software is build. If the recipe does not set it, it is set to ${S}, but the openssl recipe sets is explicitely to ${WORKDIR}/build.

After running bitbake -c configure openssl, you can check that the B directory exists and mainly contains a bunch of empty directories (and a few files). Again, this task takes a little longer than you might expect. Also because bitbake has to build some auxiliary software, before it can configure openssl.

More details about the do_patch task can be found in the Yocto Project Overview and Concepts Manual [2] (specifically in the section “4.3.5.3 Configuration, Compilation, and Staging”).

do_compile

Finally lets actually build/compile openssl:

bitbake -c compile openssl

Once this task has finished, you can find all compiled object files and the linked software in the B directory. I won’t show the whole content of that directory, because this would be way to much. But two files that can be found in the B directory are:

• libcrypto.so.3
• libssl.so.3

And corresponding links called libcrypto.so and libssl.so.

More details about the do_patch task can be found in the Yocto Project Overview and Concepts Manual [2] (specifically in the section “4.3.5.3 Configuration, Compilation, and Staging”).

Originally I thought I would also write about the packaging of software as well. But this post has become quite long already. I will probably do that in a following post. Stay tuned.

Take care,
Andreas

---

References

1. The Linux Foundation, “OpenEmbedded Build System Concepts,” 07-Feb-2024. [Online]. Available at: https://docs.yoctoproject.org/overview-manual/concepts.html#openembedded-build-system-concepts. [Accessed: 12-Feb-2024].
2. The Linux Foundation, “Yocto Project Overview and Concepts Manual,” 07-Feb-2024. [Online]. Available at: https://docs.yoctoproject.org/overview-manual/index.html. [Accessed: 12-Feb-2024].
3. Openembedded.org, “OpenEmbedded Layer Index.” [Online]. Available at: http://layers.openembedded.org/layerindex/branch/master/layers/. [Accessed: 15-Jan-2024].
4. The Linux Foundation, “Variable Glossary – The Yocto Project,” 22-Jan-2024. [Online]. Available at: https://docs.yoctoproject.org/ref-manual/variables.html. [Accessed: 22-Jan-2024].