Embedded Linux Conference 2016

Buildroot vs. OpenEmbedded/Yocto Project: A Four Hands

Discussion

Alexandre Belloni, Thomas Petazzoni

Bootlin
[email protected]
[email protected]

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 1/1
 Alexandre Belloni

I The Yocto Project guy for this talk

I Embedded Linux engineer/trainer at Bootlin since 2013
I Yocto Project/OE expert
I Maintainer of several Freescale boards in meta-fsl, strong contributor to
meta-atmel
I OpenEmbedded setup for numerous customers
I Trainer for Bootlin Yocto Project / OpenEmbedded course
I And also a kernel contributor: co-maintainer of the RTC subsystem and the Atmel
ARM processor support

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 2/1
 Thomas Petazzoni

I The Buildroot guy for this talk

I CTO and Embedded Linux engineer/trainer at Bootlin since 2008
I Strong Buildroot contributor
I Interim Buildroot maintainer
I Trainer for Bootlin Buildroot course
I And also a kernel contributor: support for Marvell processors

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 3/1
 Common aspects

I Embedded Linux build systems

I Goal is to build a complete, customized, embedded Linux system
I Root filesystem, toolchain, kernel, bootloaders
I Build from scratch from source
I Using cross-compilation
I Very actively maintained and developed projects
I Widely used in the industry
I Well documented, training courses
I Free software

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 4/1
 Buildroot: general philosophy

I Strong focus on simplicity

I Simple to use, but also simple to understand/extend
I Special use cases handled via extension scripts, rather than in Buildroot itself
I Re-use of existing technologies/languages: kconfig, make.
I Learning effort worth the investment
I Minimalist: small by default
I Purpose-agnostic
I Open community, no vendor or bureaucratic/corporate management

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 5/1
 Yocto Project: general philosophy

I Support for the major architectures

I OpenEmbedded: only qemu
I Yocto Project: adds support for a few machines
I Only provides core recipes, use layers to get support for more packages and more
machines
I Custom modifications should stay in a separate layer
I Versatile build system: tries to be as flexible as possible and to handle most use
cases.
I Open community but the project is still governed by the Yocto Project Advisory
Board made up of its corporate sponsors.
I OpenEmbedded is an independent community driven project.

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 6/1
 Buildroot: output

I Buildroot main product is a root filesystem image

I But also: toolchain, kernel image, bootloaders, etc.
I Many formats supported: ext2/3/4, ubifs, iso9660, etc.
I No binary packages, no package management system
I Some people call it a firmware generator
I Updates are not possible via packages
I Updates require a full system update, like Android
I Belief that partial updates are harmful

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 7/1
 Yocto Project: output

I Builds distributions, the main output is a package feed

I the package management system is optional on the target
I it is possible to install or update only part of the system
I Also generates root filesystem images by installing those packages. Supports
the usual ext2/3/4, ubifs, iso9660, etc.. but also VM images: vmdk, vdi, qcow2
I Finally, images classes or a tool, wic, can be used to create disk images
I Also able to generate an SDK alongside the image to allow application developers
to compile and test their applications without having to integrate it in the build.
But, the SDK has to be kept in sync with the image.

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 8/1
 Buildroot: configuration

I Re-uses kconfig from the Linux kernel

I Simple {menu,x,n,g}config interface
I Entire configuration stored in a single .config/defconfig
I Defines all aspects of the system: architecture, kernel version/config, bootloaders,
user-space packages, etc.
I make menuconfig, make, profit!
I Building the same system for different machines: to be handled separately
I A tool can generate defconfig from fragments
I Doable, but not super easy
I Complete separate build for each machine

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 9/1
 Yocto Project: configuration

I The configuration is separated in multiple parts:

I Distribution configuration (general package configuration, toolchain and libc
selection,...)
I Machine configuration (defines the architecture, machine features, BSP.)
I Image recipe (what packages should be installed on the target.)
I Local configuration (distribution and default machine selection, how many threads to
use when compiling, whether to remove build artifacts,...)
I It is also necessary to gather the various layers that will be used and declare them.
I Allows to build the same image for different machines or using different
distributions or different images for one machine.

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 10/1
 Buildroot: layers

I Initially: no concept of layers

I All packages are maintained in the official repository
I Allows for very high quality, thanks to review by experts
I Addition of BR2_EXTERNAL
I Allows to store package definitions, configurations and other artefacts
I One BR2_EXTERNAL only
I Generally used for proprietary/custom packages and configurations
I Can only add packages, not override the ones in Buildroot

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 11/1
 Yocto Project: layers

I Layer mechanism allows to modify or add new package or image recipes.

I Clear separation between the core build system, the BSP and custom
modifications.
I Scales properly, third parties provide their layers with BSPs or a set of recipes
handling their dedicated applications
I Layers have to be compatible and use the same OE branch base.
I Beware of layer quality, reviews are not systematic.
I OpenEmbedded Metadata Index lists available layers, recipes, machines:
http://layers.openembedded.org/layerindex/
I Also, there is a powerful overrides mechanism allowing to adjust recipe variables
based on the machine or distribution.

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 12/1
 Buildroot/Yocto Project: toolchain

Similar capabilities
I Building their own toolchains, based on gcc, a choice of C libraries (glibc, uClibc,
musl)
I Using pre-built external toolchains
I Somewhat easier in Buildroot, since it’s built-in
I Only really properly supported with additional vendor layers in the Yocto Project

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 13/1
 Buildroot: new package

package/tmux/Config.in
config BR2_PACKAGE_TMUX
bool "tmux"
depends on BR2_USE_MMU # fork()
select BR2_PACKAGE_LIBEVENT
select BR2_PACKAGE_NCURSES
help
tmux is a terminal multiplexer, it enables a number of terminals
(or windows) to be accessed and controlled from a single terminal.
https://tmux.github.io/

package/tmux/tmux.mk
TMUX_VERSION = 2.1
TMUX_SITE = https://github.com/tmux/tmux/releases/download/$(TMUX_VERSION)
TMUX_LICENSE = ISC
TMUX_LICENSE_FILES = README
TMUX_DEPENDENCIES = libevent ncurses host-pkgconf
$(eval $(autotools-package))

package/tmux/tmux.hash
# Locally computed:
sha256 31564e7bf4bcef2de...f6176 tmux-2.1.tar.gz

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 14/1
 Yocto Project: new package

meta-oe/recipes-extended/tmux/tmux 2.1.bb
SUMMARY = "Terminal multiplexer"
HOMEPAGE = "http://tmux.sourceforge.net"
SECTION = "console/utils"

LICENSE = "ISC"
LIC_FILES_CHKSUM = "file://tmux.c;beginline=3;endline=17;md5=8685b4455...

DEPENDS = "ncurses libevent"

SRC_URI = "git://github.com/tmux/tmux.git;branch=master"
SRCREV ?= "310f0a960ca64fa3809545badc629c0c166c6cd2"

S = "${WORKDIR}/git"
B = "${WORKDIR}/build"

inherit autotools pkgconfig

PACKAGECONFIG ??= ""

PACKAGECONFIG[utempter] = \
"ac_cv_header_utempter_h=yes,ac_cv_header_utempter_h=no,libutempter,"

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 15/1
 Buildroot: complexity

I Designed to be simple
I Every proposed feature for the core is analyzed in terms of
usefulness/complexity ratio
I Core logic written entirely in make, less than 1000 lines of code including 230 lines
of comments
I Really easy to understand what is going on, why and how a package is built
I Almost as simple as a shell script downloading, extracting, building and installing
your software components one after the other
I Detailed documentation, lots of resources/talks
I One hour talk sufficient to describe all the internals (ELCE 2014)
I Typical feedback on IRC: coming over from Yocto and very pleasantly surprised
how much easier it is. This is the first thing that’s stumped me

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 16/1
 Yocto Project: complexity

I Somewhat steeper learning curve.

I The core is bitbake, a separate project written in python (60kloc).
I A set of classes define the common tasks.
I Recipes are written in a mix of bitbake specific language, python and shell script.
I Logging and debugging allows to understand what is done for each task.
I Detailed documentation but many different configuration variables.
I It is not always easy to understand the best practices (e.g. Poky should not be
used for production, distro/image modifications should not be done in local.conf,
removing tmp/)
I People are still confused by the terminology (Yocto Project, Poky,
OpenEmbedded, bitbake)

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 17/1
 Buildroot: number/variety of packages

I 1800+ packages
I Graphics: X.org, Wayland, Qt4/Qt5,
Gtk2/Gtk3, EFL
I Multimedia: Gstreamer 0.10/1.x, ffmpeg, Kodi,
OpenGL support for numerous platforms
I Languages: Python 2/3, PHP, Lua, Perl,
Erlang, Mono, Ruby, Node.js
I Networking: Apache, Samba, Dovecot, Exim,
CUPS, lots of servers/tools
I Init systems: Busybox (default), initsysv,
systemd
I No support for a toolchain on the target

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 18/1
 Yocto Project: number/variety of packages

I Several thousand recipes: around 2200 for oe-core, meta-openembedded,

meta-qt5. More than 8400 known by the Metadata Index (includes duplicates).
I Mostly the same packages as Buildroot
I Has layers for more languages: Java, Go, Rust, smalltalk
I Still has an active layer for Qt3
I meta-virtualization (Docker, KVM, LXC, Xen) and meta-openstack layers

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 19/1
 Buildroot: approach of dependencies

I Minimalistic approach
I if a feature can be disabled, it should be disabled by default
I Lots of automatic dependencies
I i.e., if you enable OpenSSL, you will automatically get SSL support in all other
packages you have enabled that can provide SSL support
I Leads to small root filesystems by default, with no effort

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 20/1
 Yocto Project: approach of dependencies

I Package configuration is done at the distribution level

I Enabling OpenSSL will enable it for all the packages
I but it is still possible to disable it for a few packages
I inversely, it is possible to enable a feature only for selected packages.
I Can be amended at the machine level but this should be avoided.
I Each recipe can define its own set of default features to lead to a sane default
configuration.

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 21/1
 Buildroot: updates/security

I Releases published every three months: two months of development, one month
stabilization.
I Every release includes updates of package versions: both security updates and
major updates.
I And also potentially changes in the core infrastructure.
I No LTS versions with only security updates so far, users are handling this on their
own, only for the packages they care about
I Script to assess the unfixed CVEs in a given Buildroot configuration is being
contributed.

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 22/1
 Yocto Project: updates/security

I Releases published every six months: one release in April, one in October.
I Planning and roadmap is available on the wiki:
https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.1_Status
I Four milestones with 3 months between M1 and the final release
I At least the previous and the current releases have appointed maintainers, they
get security and important fixes but no recipe upgrade.
I Older releases are maintained by the community, on a best effort basis.

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 23/1
 Buildroot: detection of configuration changes

I Buildroot does not try to be smart

I When doing a change in the configuration, it does not try to detect what needs to
be rebuilt
I Once a package has been built, Buildroot will not rebuild it unless you force it
I Major configuration changes require a full rebuild
I Starts to be annoying for large systems
I Minor configuration changes can often be handled without a full rebuild
I Requires understanding a bit what you’re doing
I No way to share build artefacts between different configurations: one
configuration, one build.

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 24/1
 Yocto Project: detection of configuration changes

I bitbake maintains a shared State Cache (sstate-cache) to allow incremental

builds.
I It detects changes in the inputs of task by creating a checksum of those inputs
I This cache is shared between all the builds. Building for similar machines is fast.
I It is possible to share that cache across hosts (for example a nightly build server
and a developer machine), allowing to greatly speed up a full build.

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 25/1
 Buildroot: architecture support

I Wide range of architecture support

I Big ones: ARM(64), MIPS, PowerPC(64), x86/x86-64
I But also numerous more specific architectures
I Xtensa, Blackfin, ARC, m68k, SPARC, Microblaze, NIOSII
I ARM noMMU, especially ARMv7-M
I Contributions from architecture vendors
I MIPS by Imagination Technologies
I PowerPC64 by IBM
I ARC by Synopsys
I Blackfin by Analog Devices

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 26/1
 Yocto Project: architecture support

I In the core: ARM, MIPS, PowerPC, x86 and their 64-bit variants
I In separate layers:
I Microblaze, NIOSII
I Usually silicon vendors maintain their own BSP layers:
I meta-intel
I meta-altera (both ARM and NIOSII)
I meta-atmel, meta-fsl, meta-ti, ...
I meta-xilinx (both ARM and Microblaze)
I But also community effort:
I meta-rockchip, meta-sunxi, ...

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 27/1
 Buildroot: minimal build
Minimal build (qemu_arm_versatile_defconfig) takes 15 minutes and 25 seconds,
image size is 2.2MB.
ALL

toolchain busybox initscripts skeleton linux rootfs-ext2

toolchain-buildroot host-mke2img host-fakeroot host-makedevs

host-gcc-final host-kmod host-e2fsprogs host-genext2fs

uclibc host-pkgconf

host-gcc-initial linux-headers

host-binutils host-mpc

host-mpfr

host-gmp

host-m4

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 28/1
 Yocto Project: minimal build
Minimal build (core-image-minimal for qemuarm) takes 50 minutes and 47 seconds,
image size is 4.9MB. With an existing sstate-cache, it takes 1 minute 21 seconds.
python udev

openssl sysvinit pciutils busybox sqlite3 gdbm modutils-initscripts init-ifupdown gettext run-postinsts core-image-minimal module-init-tools

Native Toolchain

perl-native-runtime opkg-native unifdef-native texinfo-native opkg-utils-native quilt-native makedevs-native kern-tools-native update-rc.d-native gzip-native ldconfig-native gcc-cross-initial-arm glibc gcc-cross-arm glibc-initial gcc-runtime binutils-cross-arm gcc-source-5.2.0 linux-yocto base-passwd libusb-compat db readline bzip2 util-linux glib-2.0 kmod expat usbutils initscripts update-rc.d qemuwrapper-cross

libxslt-native libarchive-native git-native virtual/arm-oe-linux-gnueabi-libc-initial linux-libc-headers libgcc libgcc-initial virtual/arm-oe-linux-gnueabi-libc-for-gcc virtual/arm-oe-linux-gnueabi-binutils libtool-cross packagegroup-core-boot libusb1 libffi zlib opkg-utils netbase cryptodev-linux ncurses depmodwrapper-cross virtual/gettext virtual/libiconv libusb

mklibs-native rpm-native libxml2-native virtual/arm-oe-linux-gnueabi-gcc-initial virtual/arm-oe-linux-gnueabi-g++ virtual/arm-oe-linux-gnueabi-gcc virtual/arm-oe-linux-gnueabi-compilerlibs virtual/libc virtual/update-alternatives

acl-native pseudo-native pbzip2-native file-replacement-native python-native e2fsprogs-native

prelink-native attr-native bzip2-replacement-native readline-native openssl-native util-linux-native libmpc-native kconfig-frontends-native

binutils-native elfutils-native libpcre-native file-native sqlite3-native pigz-native cryptodev-linux-native bc-native ncurses-native mpfr-native glib-2.0-native kmod-native

byacc-native db-native popt-native virtual/libintl-native beecrypt-native bzip2-native ossp-uuid-native zlib-native bison-native flex-native lzo-native libffi-native pkgconfig-native xz-native gmp-native gperf-native gtk-doc-stub-native gettext-native

gettext-minimal-native libtool-native

automake-native

autoconf-native

m4-native

texinfo-dummy-native gnu-config-native

perl-native-runtime-native

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 29/1
 License compliance

I Both are able to create a list of used licenses

I Both are able to detect a license change
I Yocto Project can be configured to exclude GPLv3

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 30/1
 Buildroot/Yocto Project: choosing

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 31/1
 Questions?

Alexandre Belloni
Thomas Petazzoni
[email protected]
[email protected]

Slides under CC-BY-SA 3.0

http://bootlin.com/pub/conferences/2016/elc/belloni-petazzoni-buildroot-oe/

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 32/1