TM

Jan/Feb 2015

JOURNAL

B R I N G I NG UP

The New 64-bit

Core Design Architecture

!
NEW

INTERACTING
• Getting Involved with the FreeBSD Project
• A Journey from Adoption to Contribution
 Table of Contents
TM

Jan/Feb 2015

Vol. 2, Issue No. 1

3 Foundation Letter
Welcome to FreeBSD Journal 2015!

32 Book Review
The Design and Implementation of the
BRINGING UP

MIPS
FreeBSD Operating System, Second
Edition. While being focused on FreeBSD sets
this book apart from other operating systems
books where the focus is Linux, that’s not the
only thing that sets it apart. By Rik Farrow

34 Conference Report
Going to any conference alone can be some-
what intimidating. But this was not the case at
meetBSD 2014 where the author was greeted
with open arms. By Wallace Barrow

36 This Month in FreeBSD

Dru Lavigne interviews Glen Barber, who has
44 Porting FreeBSD to a new CPU is a significant under-
taking. This article helps prospective porters orient
themselves before they begin the process. While the focus
been on the FreeBSD Release Engineering is on MIPS ports, the code structure in other platforms—
Team since January 2013 and lead Release
Engineer since March 2014. By Dru Lavigne
especially ARM—is quite similar. By Brooks Davis,
Robert Norton, Jonathan Woodruff, Robert N. M. Watson
40 svn update

arm64
Whether you are a FreeBSD systems adminis-
trator, application developer, or hobbyist,
you’ll be interested in some changes that can
be expected in upcoming FreeBSD releases.
By Glen Barber

42 Ports Report
In November and December the activity on
the ports tree was not high; still, the figure is
impressive with 3,778 commits! On the bug
front, however, more problem reports were
14
X Historically ARM has produced CPU core designs
for chips mainly used in embedded and mobile
devices. Until recently, these CPUs have all been 32-bit,
closed than during last period, with 1,157 but ARM has now announced a new 64-bit architecture.
problems fixed. By Frederic Culot
By Andrew Turner
44 Events Calendar
By Dru Lavigne

INTERACTING with the FreeBSD Project

22 Getting Involved This article provides a general overview of how

and why to start contributing to the FreeBSD Project. It also addresses some of the concerns and questions
commonly voiced by new contributors. By Dru Lavigne

28 A Journey from Adoption to Contribution The decision to adopt and deploy

FreeBSD globally within Verisign’s Domain Name System infrastructure was just the start of the author’s involvement.
They knew from the beginning that it was important—and necessary—to actively engage and collaborate with
the FreeBSD Foundation, the Project, and the support community. By Rick Miller and Julien Charbon
Nov/Dec
Jan/Feb2014
2015 1
 TM

LETTER
from the Board •

Editorial Board
JOURNAL
Welcome to
John Baldwin • Member of the
FreeBSD Core Team FreeBSD Journal
•
2015!
Daichi Goto Director at BSD Consulting Inc.
(Tokyo)
With this new year, we bring our FreeBSD
Joseph Kong • Author of FreeBSD Device
Drivers Journal subscribers a fresh slate of articles, as well as a couple of
Dru Lavigne • Director of the FreeBSD new developments, the first of which is the inclusion of articles
Foundation and Chair of the
BSD Certification Group about interacting with the FreeBSD Project. In this issue Dru
Michael W. Lucas • Author of Absolute FreeBSD Lavigne writes about different ways to get involved, and Rick
Miller and Julien Charbon discuss how they went from adopting
Kirk McKusick • Director of the FreeBSD
Foundation and lead author
of The Design and
FreeBSD within Verisign to being active contributors to the
Implementation book series FreeBSD source base.
George Neville-Neil • Director of the FreeBSD
One of the Journal’s goals is to serve as a storehouse of
Foundation and co-author of
The Design and Implementation knowledge about FreeBSD and associated software systems. It
of the FreeBSD Operating
System has always been the Board’s intention to release some of the
Hiroki Sato • Director of the FreeBSD Journal’s articles online—after a reasonable period of time—so
Foundation, Chair of
AsiaBSDCon, member of the that they can be indexed, referenced, and used by the general
FreeBSD Core Team and
Assistant Professor at Tokyo public. Beginning immediately, we will start that process by
Institute of Technology
releasing a few of the articles from the Jan/Feb 2014 issue on the
Robert Watson • Director of the FreeBSD
FreeBSD Foundation’s website. If you are a subscriber, know that
Foundation, Founder of the
TrustedBSD Project and
Lecturer at the University of
your subscription gets you the articles well ahead of any free
Cambridge releases plus the timely svn update and Ports Report columns,
book reviews, conference reports, calendar, and This Month in
S &W PUBLISHING LLC
P O B O X 4 0 8 , B E L F A S T, M A I N E 0 4 9 1 5
FreeBSD, all of which will not be part of the free release.
FreeBSD Journal’s circulation continues to grow each month and
Publisher • Walter Andrzejewski
we expect that to continue in 2015. You can help us by showing
[email protected]

Editor-at-Large • James Maurer other folks our work, posting pointers to articles, and making sure
[email protected]
everyone you know who should know about FreeBSD also knows
Art Director • Dianne M. Kischitz that the FreeBSD Journal is the best way to keep up with the latest
[email protected]
releases and new developments in FreeBSD.
Office Administrator • Michael Davis
[email protected] Finally, for European readers on iOS, please check your sub-
Advertising Sales • Walter Andrzejewski scriptions. An increase in VAT within Europe caused Apple to
[email protected]
Call 888/290-9469 cancel all automatic magazine subscription renewals, meaning
that you will need to re-subscribe to the FreeBSD Journal if you
FreeBSD Journal (ISBN: 978-0-615-88479-0) bought it on iTunes from one of the European stores—even if
is published 6 times a year (January/February,
March/April, May/June, July/August, you checked automatic renewal.
September/October, November/December).
Trust us, no one wants to miss even one of our issues in 2015.
Published by the FreeBSD Foundation,
PO Box 20247, Boulder, CO 80308
ph: 720/207-5142 • fax: 720/222-2350 Sincerely,
email: [email protected]
Copyright © 2015 by FreeBSD Foundation.
All rights reserved.
FreeBSD Journal Editorial Board
This magazine may not be reproduced in whole or in
part without written permission from the publisher.
Jan/Feb 2015 3
 MIPS
bringing up

Porting FreeBSD
to a new CPU, even
within a previously
supported family,
is a significant
undertaking.

Fig. 1
BERIpad with
Application Launcher.

4 FreeBSD Journal
By Brooks Davis, Robert Norton, Jonathan Woodruff & Robert N. M. Watson

W e hope this article will help prospective porters orient

themselves before they begin the process. While we
focus on MIPS ports, the code structure in other plat-
forms—especially ARM—is quite similar. From its modest origins as
a fork of 386BSD targeting Intel i386 class CPUs, FreeBSD has been
ported to a range of architectures including AArch64, AMD x86_64
(aka amd64), ARM, MIPS, and PowerPC. While the x86 systems are
fairly homogeneous targets with mechanics for detecting and
adapting to specific board and peripheral configurations, embed-
ded systems platforms such as ARM, MIPS, and PowerPC are much
more diverse. Porting to a new MIPS board often requires adding
support for a new system on chip (SoC) or CPU type with different
interrupt controllers, buses, and peripherals. Even if the CPU is sup-
ported, boot loaders and associated kernel calling conventions
generally differ significantly between boards.

We have ported FreeBSD/MIPS to BERI, Based on our experiences bringing up

an open-source MIPS R4000-style [1] FreeBSD on BERI, we have documented
FPGA-based soft-core processor that we how we boot FreeBSD from the firmware
have developed. This required a range of embedded in the CPU to userspace, to
work including boot-loader support, plat- provide a new view on the boot process
form startup code, a suite of device driv- and help porters gain a high-level under-
ers (including the PIC), and also adapting standing of the boot process.
FreeBSD’s existing FDT support to The rest of this article narrates the boot
FreeBSD/MIPS. We currently run process with a special focus on the places
FreeBSD/BERI under simulation, on the where customization was required for
Altera Stratix IV FPGA on a Terasic DE4 BERI. In the interest of brevity, many
FPGA board, on the Altera Cyclone V on a aspects of boot are omitted; others that
Terasic SoCKit board, and on a Xilinx are not platform- or architecturally-specific
Virtex-5 FPGA on the NetFPGA-10G plat- are ignored. Some platform-specific com-
form. The majority of our peripheral work ponents such as the MIPS pmap are not
has been in simulation and the DE4 plat- covered. The goal is to provide a guide to
form. FreeBSD BERI CPU support is those pieces someone would need to
derived from the MALTA port, with some know when porting to a new, but relative-
inspiration from the SiByte port. ly conventional, MIPS CPU. Porters inter-

Jan/Feb 2015 5
 model = "SRI/Cambridge BeriPad (DE4)"; sdcard@7f008000 {
compatible = "sri-cambridge,beripad-de4"; compatible = "altera,sdcard_11_2011";
cpus { reg = <0x7f008000 0x400>;
cpu@0 { };
device-type = "cpu"; flash@74000000 {
compatible = "sri-cambridge,beri"; partition@20000 {
}; reg = <0x20000 0xc00000>;
}; label = "fpga0";
soc { };
memory { partition@1820000 {
device_type = "memory"; reg = <0x1820000 0x027c0000>;
reg = <0x0 0x40000000>; label = "os";
}; };
beripic: beripic@7f804000 { };
compatible = "sri-cambridge,beri-pic"; ethernet@7f007000 {
interrupt-controller; compatible = "altera,atse";
reg = <0x7f804000 0x400 0x7f806000 0x10 reg = <0x7f007000 0x400 0x7f007500 0x8
0x7f806080 0x10 0x7f806100 0x10>; 0x7f007520 0x20 0x7f007400 0x8
} 0x7f007420 0x20>;
serial@7f002100 { };
compatible = "ns16550"; touchscreen@70400000 {
reg = <0x7f002100 0x20>; compatible = "sri-cambridge,mtl";
}; reg = <0x70400000 0x1000
serial@7f000000 { 0x70000000 0x177000 0x70177000 0x2000>;
compatible = "altera,jtag_uart-11_0"; };
reg = <0x7f000000 0x40>; };
};

Fig. 2 Excerpt from Flat Device Tree (FDT) Description of the DE4-based BERI Tablet.

ested in less conventional CPUs will want to (altera_sdcard), which implement low-level
examine the NLM and RMI ports in mips/nlm console/tty, Ethernet interface, and block stor-
and mips/rmi for examples of the more age classes. In addition, we have implemented
extensive modifications required for a complete a generic driver for Avalon-attached devices
multi-core platform. (avgen) that allows memory mapping of arbi-
For more information on the high-level boot trary bus-attached devices without interrupt
process see Chapter 15 of The Design and sources—such as the DE4 LED block, BERI con-
Implementation of the FreeBSD Operating figuration ROM, and DE4 fan-and-temperature
System, Second Edition [7]. control block.

The BERIpad Platform Flat Device Tree

We have developed BERI as a platform that can Most aspects of BERI board configuration are
enable experiments on the hardware-software described in Flat Device Trees (FDTs), which are
interface such as our ongoing work on hard- commonly used on PowerPC and ARM-based
ware-supported capabilities in the CHERI CPU systems [3]. Currently a Device Tree Blob (DTB)
[6]. Our primary hardware target has been a is built into each FreeBSD kernel, describing a
tablet based on the Terasic DE4 FPGA board specific hardware configuration. Each DTB is
with a Terasic MTL touch screen and integrated built from a device-tree syntax (DTS) file by the
battery pack. The source for the CPU and the device-tree compiler (dtc(1)) before being
design for the tablet have been released as embedded in the kernel. Figure 2 excerpts
open source at http://beri-cpu.org. The modifi- boot/fdt/dts/mips/beripad-de4.dts—
cations to FreeBSD support have been merged the DTS file—and includes the BERI CPU, 1GB
to FreeBSD. Single-processor support appeared DRAM, programmable interrupt controller (PIC),
in 10.0 and multiprocessor support will appear hardware serial port, JTAG UART, SD card read-
in future releases. The tablet (Figure 1) and the er, flash partition table, gigabit Ethernet, and
internal architecture of BERI are described in touch screen.
detail in The BERIpad Tablet [2].
We developed device drivers for three Altera The Early Boot Sequence
IP cores: the JTAG UART (altera_jtag_uart),
The common FreeBSD boot sequence begins
triple-speed MAC (atse), and SD Card
with CPU firmware arranging to run the

6 FreeBSD Journal
FreeBSD boot2 second-stage boot loader, which
in turn loads /boot/loader, which loads the
kernel and kernel modules. This leads to the
kernel boot, described later in this article in a
section called “The Path to Usermode.”

Miniboot
At power on or after reset, the CPU sets the
program counter of at least one hardware
thread to the address of a valid program. From
the programmer perspective, the process by
which this occurs is essentially magic and of no
particular importance. Typically the start address
is some form of read-only or flash-upgradable
firmware that allows for early CPU setup and
that may handle details such as resetting cache
state or pausing threads other than the primary
thread until the operating system is ready to
handle them. In many systems, this firmware is
responsible for working around CPU bugs.
On BERI this code is known as miniboot for
physical hardware and simboot for simulation.
Fig. 3 Layout of the DE4 Flash.
Miniboot is compiled into the FPGA bitfile as
a read-only BRAM. It is responsible for setting
registers to initial values, setting up an initial Support.”
stack, initializing the cache by invalidating Before miniboot enters the kernel, it clears
the contents, setting up a spin table for most registers and sets a0 to argc, a1 to
multiprocessor (MP) boot, and loading the next- argv, a2 to env, and a3 to the size of system
stage loader or kernel from flash, or waiting for memory. In practice argc is 0 and argv and
it to be loaded via the debug unit and executing env are NULL. It then assumes that an ELF64
it. With BERI, we are fortunate that we need not object is located at 0x100000, loads the entry
work around CPU bugs in firmware, because we point from the ELF header, and jumps to it.
can simply fix the hardware. We intend for miniboot to be minimal, but
Miniboot kernel loading and boot behavior also sufficiently flexible to support debugging of
is controlled by two DIP switches on the DE4. If various boot layouts, as well as loading alterna-
DIP0 is off or if miniboot is compiled with tive code such as self-contained binaries. This
-DALWAYS_WAIT, then we spin in a loop wait- allows maximum flexibility for software develop-
ing for the MIPS-ISA general-purpose register t1 ers who may not be equipped to generate new
to be set to 0 using JTAG. This allows a user to hardware images.
control when the processor starts executing, giv-
ing the user an opportunity to load a kernel boot2
directly to DRAM before boot proceeds. DIP1
On most FreeBSD systems two more boot stages
controls the relocation of a kernel from flash. If
are interposed between the architecture-
the DIP switch is set, the kernel is loaded from
dependent boot code and the kernel. The first
flash at offset of 0x2000000 to 0x100000 in
of these is boot2, the second-stage bootstrap
DRAM. Otherwise, the user must load a kernel
(boot(8)); boot2 has a mechanism for access-
via the debug unit as described in the BERI
ing local storage and has code for read-only
Software Reference [5].
access to a limited set of file systems (currently
The kernel will be loaded only on hardware
UFS or ZFS). Its primary job is to load the loader
thread 0. In other hardware threads, miniboot
and to pass arguments to it. By default it loads
enters a loop waiting for the operating system
/boot/loader, but the user can specify an
to provide a kernel entry point via the spin-
alternative disk, partition, and path. boot2 can
table. Booting with multithreading and multi-
also boot the kernel directly.
core configurations is discussed later in this arti-
We have ported boot2 to BERI, creating
cle in the section titled “Multiprocessor

Jan/Feb 2015 7
 three microdrivers—allowing JTAG UART console (http://ficl.sourceforge.net). This interpreter is used
access, and use of CFI or the SD card to load to provide the boot menu shown in Figure 4; it
/boot/loader or the kernel. These micro- parses configuration files such as
drivers substitute for boot device drivers provided /boot/loader.conf, and implements functionality
by the BIOS on x86 or OpenFirmware on SPARC. such as nextboot(8). In order to do this, the
It also supports jumping to an instance of loader also contains microdrivers to access plat-
/boot/loader loaded via JTAG. In our current form-specific devices and contains implementa-
implementation, boot2 is linked at the same tions of UFS and ZFS with read and, limited,
address as the kernel and loaded from CFI flash write support. On x86 systems that means BIOS
allowing it to be used with an unmodified or UEFI disk access, and with the pxeloader net-
miniboot. In the future, we plan to place a work access via PXE. On BERI this currently
similar version of boot2 at 0x03fe0000, a includes a basic microdriver for access to the CFI
128K area reserved for its use. This will allow a flash found on the DE4.
normal file system to be placed in CFI flash from We have ported the loader to FreeBSD/MIPS
0x1820000, which might contain the full boot and shared the SD card and CFI microdrivers with
loader, a kernel, etc. Currently, we use boot2 to boot2 to allow kernels to be loaded from CFI
relocate FDT embedded in the CPU image and to flash or SD card. We currently load the kernel
load /boot/loader from the SD card, which from the SD card. We hope to eventually add a
offers an experience more akin to conventional driver for the on-board Ethernet device to allow
desktop/server platforms than conventional us to load kernels from the network.
embedded targets. The loader’s transition to the kernel is much
Many versions of boot2 exist that are target- the same as miniboot. The kernel is loaded to
ed at different architectures. The version of the expected location in the memory, the ELF
boot2 in BERI is derived from the x86 boot2, header is parsed, arguments are loaded into reg-
isters, and the loader jumps into the kernel.

The bootinfo Structure

In order to facilitate passing information among
boot2, /boot/loader, and the kernel, a
pointer to a bootinfo structure allows informa-
tion such as memory size, boot media type, and
the locations of preloaded modules to be shared.
When present and detected by boot2, the loca-
tion in memory of a relocated copy of embedded
FDT is also present in bootinfo.

The Path to Usermode

Fig. 4 FreeBSD Loader Boot Menu This section narrates the interesting parts of
the FreeBSD boot process from a MIPS porter’s
perspective.
and is (marginally) more feature-rich than those
targeted at more space-constrained embedded Early Kernel Boot
architectures. There is currently more diversity of
The FreeBSD MIPS kernel enters at _start in
boot2 implementations than necessary, largely
the _locore function defined in
due to excessive copying and modifying of code
mips/mips/locore.S; _locore performs
during the early parts of the porting process.
some early initialization of the MIPS CP0 regis-
ters, sets up an initial stack, and calls the plat-
loader form-specific startup code in platform_start.
The third common boot stage is the loader(8). On BERI, platform_start saves the argu-
The loader is in practice a small kernel whose ment list, environment, and pointer to struct
main job is to set up the environment for the bootinfo passed by the loader. BERI kernels
kernel and then load the kernel and any config- also support an older boot interface, in which
ured modules from the disk or network. The memory size is passed as the fourth argument
loader contains a Forth interpreter based on FICL (direct from miniboot). It then calls the com-

8 FreeBSD Journal
 static void
print_caddr_t(void *data)
mon MIPS function mips_postboot_fixup, {
which provides kernel-module information for printf("%s", (char *)data);
manually loaded kernels and corrects ker- }
nel_kseg0_end (the first usable address in SYSINIT(announce, SI_SUB_COPYRIGHT,
kernel space) if required. Per-CPU storage is then SI_ORDER_FIRST, print_caddr_t,
copyright);
initialized for the boot CPU by
mips_pcpu0_init. Since BERI uses Flat Device
Tree to allow us to configure otherwise non- Fig. 5 Implementation of Copyright Message
discoverable devices, platform_start then Printing on FreeBSD Boot.
locates the DTB and initializes FDT. This is the
norm for ARM and PowerPC ports, but is cur-
rently uncommon on MIPS ports. We expect it registration mechanism called SYSINIT(9) was
to become more popular over time. The created. Any code that needs to be run at start-
platform_start function then calls up can use the SYSINIT macro to cause a
mips_timer_early_init to set system function to be called in a sorted order to boot or
timer constants, currently to a hardcoded on module load. The sysinit implementation
100MHz—although eventually this will come relies on the linker set feature, by which con-
from FDT. The console is set up by cninit and structors/destructors for kernel subsystems and
some debugging information is printed. The modules are tagged in the ELF binary so that the
number of pages of real memory is stored in the kernel linker can find them during boot, module
global variable realmem 1. The BERI-specific load, module unload, and kernel shutdown.
mips_init 2 function is then called to perform The implementation of mi_startup is sim-
the bulk of the remaining early setup. ple. It sorts the set of sysinits and then runs
BERI’s mips_init is fairly typical. First, mem- each in turn marking each as done when it is
ory-related parameters are configured including complete. If any modules are loaded by a sysinit,
laying out the physical memory range and set- it re-sorts the entire set and starts from the
ting a number of automatically tuned parame- beginning, skipping previous run entries. The
ters in the general functions init_param1 and end of mi_startup contains code to call
init_param2. The MIPS function swapper, which is never reached—as the last
mips_cpu_init performs some optional per- sysinit never returns. One implementation detail
platform setup (nothing on BERI), identifies the of note in mi_startup is the use of bubble
CPU, configures the cache, and clears the TLB. sort to sort the sysinits, due to the fact that allo-
The MIPS version of pmap_bootstrap is called cators are initialized via sysinits and thus not yet
to initialize the pmap. Thread 0 is instantiated by available.
mips_proc0_init, which also allocates space Figure 5 shows a simple example of a sysinit.
for dynamic per CPU variables. Early mutexes In this example, announce is the name of the
(including the legacy Giant lock) are initialized in individual sysinit, SI_SUB_COPYRIGHT is the
mutex_init, and the debugger is initialized in subsystem, SI_ORDER_FIRST is the order
kdb_init. If so configured, the kernel may within the subsystem, print_caddr_t is the
now drop into the debugger or, much more function to call, and copyright is an argu-
commonly, return and continue booting. ment to be passed to the function. A complete
Finally mips_timer_init_params is called list of subsystems and orders within subsystems
to finish setting up the timer infrastructure can be found in sys/kernel.h. As of this
before platform_start returns to _locore, writing there are more than 80 of them. Most
which switches to the now configured thread0 have little or no architecturally-specific function,
stack and calls mi_startup, never to return. and thus are beyond the scope of this aticle.
We highlight sysinits with significant port-
Calling All SYSINITS specific content.
The job of mi_startup is to initialize all the The first sysinit of interest is
kernel’s subsystems in the right order. SI_SUB_COPYRIGHT. It does not require port-
Historically, mi_startup was called main and ing specifically, but reaching it and seeing the
the order of initialization was hard-coded. This output is a sign of an architectural port nearing
was obviously not scalable, so a more dynamic completion since it means low-level consoles
work, and the initial boot described above is
1 The btoc macro converts bytes to clicks, which in FreeBSD are single pages. Mach allowed multiple pages to be managed as a virtual page.
2 Most ports have one of these, but it seems to be misnamed, as it is not MIPS generic code.

Jan/Feb 2015 9
 Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.

Fig. 6 Copyright and Trademark Messages in FreeBSD 10.

complete. The MIPS port has some debugging identical.

output earlier in boot, but on mature platforms The bus hierarchy is established and device
the copyright message is the first output from probing is performed at the
the kernel. Figure 6 shows the three messages SI_SUB_CONFIGURE stage (aka autoconfigura-
printed at SI_SUB_COPYRIGHT. tion). The platform-specific portions of this
The next sysinit of interest to porters is stage are the configure_first function
SI_SUB_VM. The MIPS bus_dma(9) implemen- called at SI_ORDER_FIRST, which attaches the
tation starts with a set of statically allocated nexus bus to the root of the device tree;
maps to allow it to be used early in boot. The configure, which runs at SI_ORDER_THIRD
function mips_dmamap_freelist_init adds and calls root_bus_configure to probe
the static maps to the free list at SI_SUB_VM. and attach all devices; and configure_final,
The ARM platform does similar work, but does which runs at SI_ORDER_ANY to finish setting
require malloc and thus runs busdma_init at up the console with cninit_finish, and clear
SI_SUB_KMEM instead. the cold flag. On MIPS and some other plat-
Further bus_dma(9) initialization takes place forms configure also calls intr_enable to
at SI_SUB_LOCK in the platform-specific, but enable interrupts. A number of console drivers
often identical, init_bounce_pages function. complete their setup with explicit sysinits at
It initializes some counters, lists, and the bounce- SI_SUB_CONFIGURE, and many subsystems
page lock. such as CAM and acpi(4) perform their initial-
All ports call a platform-specific ization there.
cpu_startup function at SI_SUB_CPU, set up Each platform registers the binary types it sup-
the kernel address space, and perform some ini- ports at SI_SUB_EXEC. This primarily consists of
tial buffer setup. Many ports also perform board-, registering the expected ELF header values. On a
SoC-, or CPU-specific setup such as initializing uniprocessor MIPS, this is the last platform-
integrated USB controllers. Ports typically print specific sysinit.
details of physical and virtual memory, initialize The final sysinit is an invocation of the
the kernel virtual address space with scheduler function at
vm_ksubmap_init, the VFS buffer system with SI_SUB_RUN_SCHEDULER, which attempts
bufinit, and the swap buffer list with to swap in processes. Since init(8)
vm_pager_bufferinit. On MIPS the plat- was previously created by create_init at
form-specific cpu_init_interrupts is also SI_SUB_CREATE_INIT and made runnable
called to initialize interrupt counters. by kick_init at SI_SUB_KTHREAD_INIT,
Most platforms have their own starting the scheduler results in entering
sf_buf_init routine to allocate userland.
sendfile(2) buffers and initialize related
locks. Most of these implementations are Multiprocessor Support
Multiprocessor systems follow the same boot
struct spin_entry { process as uniprocessor systems, with a few
uint64_t entry_addr;
uint64_t a0; added sysinits to enable and start scheduling the
uint32_t rsvd1; other hardware threads. These threads are
uint32_t pir; known as application processors (APs).
uint64_t rsvd2; The first MP-specific sysinit is a call to
}; mp_setmaxid at SI_SUB_TUNABLES to initial-
ize the mp_ncpus and mp_maxid variables. The
generic mp_setmaxid function calls the plat-
Fig. 7 Definition of a spin_entry with Explicit
form-specific cpu_mp_setmaxid. On MIPS,
Padding and the Argument Variables Renamed to cpu_mp_setmaxid calls the architecturally-spe-
Match MIPS Conventions. cific platform_cpu_mask to fill a cpuset_t

10 FreeBSDJournal
with a mask of all available cores or threads. implement the spin-table method described in
BERI’s implementation extracts a list of cores the ePAPR 1.0 specification [3] because it is
from the DTB and verifies that they support the extremely simple. The spin-table method
spin-table enable method. It further verifies that requires that each AP have an associated
the spin-table entry is properly initialized or that spin_entry structure located somewhere in
the thread is ignored. the address space and for that address to be
The initialization of APs is accomplished by recorded in the DTB. The BERI-specific definition
the mp_start function called at SI_SUB_CPU of struct spin_entry can be found in
after cpu_startup. If there are multiple CPUs, Figure 7. At boot the entry_addr member of
it calls the platform-specific cpu_mp_start each AP is initialized to 1, and the AP waits for
and upon return prints some information about the LSB to be set to 0—at which time it jumps
the CPUs. The MIPS implementation of to the address loaded in entry_addr passing
cpu_mp_start iterates through the list of valid a0 in register a0. We implement waiting for
CPU IDs as reported by platform_cpu_mask, entry_addr to change with a loop in mini-
and attempts to start each one except itself as boot. In BERI’s platform_cpu_mask we look
determined by platform_processor_id 3 up the spin_entry associated with the
with the platform-specific start_ap. The port- requested AP, set the pir member to the CPU
specific platform_start_ap’s job is to cause ID and then assign the address of mpentry to
the AP to run the platform-specific mpentry. the entry_addr member.
When run successfully, it increments the The MIPS implementation of mpentry is
mp_naps variable, and start_ap waits up to assembly in mips/mips/mpboot.S. It disables
five seconds per AP for this to happen before interrupts, sets up a stack, and calls the port-
giving up on it. specific platform_init_ap to set up the AP
Various mechanisms have been implemented before entering the MIPS-specific
to instruct a CPU to start running a particular smp_init_secondary to complete per-CPU
piece of code. On BERI we have chosen to setup and await the end of the boot process. A
3 Implemented in mips/beri/beri_asm.S on BERI.

I S I L O N The industry leader in Scale-Out Network Attached Storage (NAS)

Isilon is deeply invested in advancing FreeBSD

performance and scalability. We are looking
to hire and develop FreeBSD committers for
kernel product development and to improve
the Open Source Community.

H i r i n g!
We’r e
With offices around the world,
we likely have a job for you!
Please visit our website at
http://www.emc.com/careers
or send direct inquiries to
[email protected].

Jan/Feb 2015 11
 typical MIPS implementation of means atomically setting the aps_ready flag to 1
platform_init_ap sets up interrupts on the and spinning until smp_started is non-zero.
AP and enables the clock and IPI interrupts. On
BERI we defer IPI setup until after device probe, A Word on IPIs
because our programmable interrupt controller
In multiprocessor (MP) systems, CPUs signal each
(PIC) is configured as an ordinary device and
other via Inter-Processor Interrupts (IPIs). Various
thus cannot be configured until after
IPI mechanisms exist, with FreeBSD MIPS using
SI_SUB_CONFIGURE.
the simplest model, a per-CPU integer bitmask
The MIPS-specific smp_init_secondary
of pending IPIs and a port-specific mechanism
function initializes the TLB, sets up the cache,
for sending an interrupt—almost always to hard-
and initializes per-CPU areas before incrementing
ware interrupt 4. This is implemented by the
mp_naps to let start_ap know that it has fin-
ipi_send, which is used by the public
ished initialization. It then spins waiting for the
ips_all_but_self, ipi_selected, and
flag aps_ready to be incremented, indicating
ipi_cpu functions. MIPS IPIs are handled by
that the boot CPU has reached SI_SUB_SMP as
mips_ipi_handler, which clears the inter-
described below. On BERI it then calls plat-
rupt with a call to platform_ipi_clear,
form_init_secondary to route IPIs to the
reads the set of pending IPIs, and handles each
AP and set up the IPI handler. The AP then sets
of them.
its thread to the per-CPU idle thread, increments
On BERI IPIs are implemented using the BERI
smp_cpus, announces itself on the console,
PIC’s soft interrupt sources. IPIs are routed by
and if it is the last AP to boot, sets
beripic_setup_ipi, sent by
smp_started to inform release_aps that all
beripic_send_ipi, and cleared by
APs have booted, and sets the smp_active
beripic_clear_ipi. These functions are
flag to inform a few subsystems that it is run-
accessed via kobj(9) through the FDT_IC
ning with multiple CPUs. Unless it was the last
interface defined in dev/fdt/fdt_ic_if.m.
AP to boot, it spins waiting for smp_started
The internals of BERI PIC are described in the
before starting per-CPU event timers and enter-
BERI Hardware Reference [4].
ing the scheduler.
The final platform-specific sysinit subsystem is
SI_SUB_SMP, which platform-specific Acknowledgments
release_aps functions are called to enable IPIs We thank our colleagues—especially Jonathan
on the boot CPU, inform previously initialized APs Anderson, Ruslan Bukin, David Chisnall, Nirav
• that they can start operating, and spin until they
do so as described above. In the MIPS case this
Dave, Alexandre Joannou, Wojciech Koszek, Ben
Laurie, A. Theodore Markettos, Ed Maste, Simon

REFERENCES
[1] Heinrich, J. MIPS R4000 Microprocessor User’s Manual, Second Edition. (1994)
[2] Markettos, A. T.; Woodruff, J.; Watson, R. N. M.; Zeeb, B. A.; Davis, B.; and Moore, S. W. The BERIpad tablet:
open-source construction, CPU, OS and applications (http://www.cl.cam.ac.uk/research/security/ctsrd/pdfs/
2013terasic-beri-submitted.pdf), Proceedings of 2013 FPGA Workshop and Design Contest, Southeast University,
Nanjing, China. (November 1–3, 2013) [3] Power.org, Power.org Standard for Embedded Power Architecture
Platform Requirements (ePAPR). (2008)
[4] Watson, R. N. M.; Woodruff, J.; Chisnall, D.; Davis, B.; Koszek, W.; Markettos, A. T.; Moore, S. W.; Murdoch,
S. J.; Neumann, P. G.; Norton, R.; and Roe, M. Bluespec Extensible RISC Implementation: BERI Hardware Reference
(http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-852.pdf) , Technical Report UCAM-CL-TR-852, University of
Cambridge, Computer Laboratory. (April 2014)
[5] Watson, R. N. M.; Chisnall, D.; Davis, B.; Koszek, W.; Moore, S. W.; Murdoch, S. J.; Neumann, P. G.; and
Woodruff, J. Bluespec Extensible RISC Implementation: BERI Software Reference (http://www.cl.cam.ac.uk/
techreports/UCAM-CL-TR-853.pdf) , Technical Report UCAM-CL-TR-853, University of Cambridge, Computer
Laboratory. (April 2014)
[6] Woodruff, J.; Watson, R. N. M.; Chisnall, D.; Moore, S. W.; Anderson, J.; Davis, B.; Laurie, B.; Neumann,
P. G.; Norton, R.; and Roe, M. “The CHERI Capability Model: Revisiting RISC in an Age of Risk,” in Proceedings of
the 41st International Symposium on Computer Architecture (ISCA 2014). (June 2014)
[7] McKusick, M. K.; Neville-Neil, G. V.; Watson, R. N. M. The Design and Implementation of the FreeBSD Operating
System, Second Edition, Boston, Massachusetts: Pearson Education. (September 2014)

12 FreeBSDJournal
W. Moore, Alan Mujumdar, Steven J. Murdoch, Projects Agency (DARPA) under Contract
Peter G. Neumann, Philip Paeps, Michael Roe, FA8750-10-C-0237. The views, opinions, and/or
Colin Rothwell, Hans Petter Selasky, Stacey Son, findings contained in this paper are those of the
and Bjoern A. Zeeb. authors and should not be interpreted as repre-
This paper is approved for public release; dis- senting the official views or policies of the
tribution is unlimited. It was developed with Department of Defense or the U.S.
funding from the Defense Advanced Research Government. •

BROOKS DAVIS is a Senior Software Engineer in the Computer Science Laboratory at SRI International
and a Visiting Research Fellow at the University of Cambridge Computer Laboratory. He has been a
FreeBSD user since 1994, a FreeBSD committer since 2001, and was a core team member from 2006 to
2012. His computing interests include security, operating systems, networking, high performance comput-
ing, and, of course, finding ways to use FreeBSD in all these areas.
ROBERT NORTON is a PhD student at the University of Cambridge. He is working on the CHERI CPU,
a project to improve application security using hardware support for fine-grained memory protection and
compartmentalization. He is particularly focusing on aspects of sandboxing and transitions between secu-
rity domains.
JONATHAN WOODRUFF is a researcher at the University of Cambridge. He is a core developer of
the BERI/CHERI open-source CPU and has helped develop the CHERI ISA for memory safety and
isolation for practical hardware implementation. He has also worked on large processor architec-
ture simulation on FPGA. He supports open-source processor research enabled by FPGAs for
reproducible, full-system research that reaches to customized hardware.
DR. ROBERT N. M. WATSON is a University Lecturer in Systems, Security, and Architecture at the
University of Cambridge Computer Laboratory; FreeBSD developer and core team member; and member
of the FreeBSD Foundation board of directors. He leads a number of cross-layer research projects span-
ning computer architecture, compilers, program analysis, program transformation, operating systems, net-
working, and security. Recent work includes the Capsicum security model, MAC Framework used for
sandboxing in systems such as Junos and Apple iOS, and multithreading in the FreeBSD network stack. He
is a coauthor of The Design and Implementation of the FreeBSD Operating Systems (Second Edition).

Premier VPS Hosting

RootBSD has multiple datacenter locations,
and offers friendly, knowledgeable support staff.
Starting at just $20/mo you are granted access to the latest
FreeBSD, full Root Access, and Private Cloud options.

www.rootbsd.net
Jan/Feb 2015 13
64 arm
By Andrew Tur ner
Historically ARM has produced CPU core
designs for chips mainly used in a large
number of embedded and mobile devices.
Users of FreeBSD will most likely be famil-
iar with the development boards in which
ARM chips are used, for example, the
Raspberry Pi and PandaBoard. Until recent-
ly these CPUs have all been 32-bit; howev-
er, in the last few years, ARM has
announced their new 64-bit architecture.
This new architecture is known as AArch64
and includes a new instruction set A64,
with the first AArch64 processors designed
to follow the ARMv8 specification.
Developers familiar with the 32-bit ARM
and Thumb instruction sets should feel
right at home with A64.

14 FreeBSD Journal
The ARMv8 Specification pany that has taken this route is Marvell,
which implements an ARMv7 compatible core
The first change developers will notice is the of their own design. With ARMv8, ARM has
increase in size and large number of the gen- endeavored to have a larger number of inde-
eral purpose registers—there are 31, 64-bit, pendent designs.
general-purpose registers. In addition, the The last level is the implementation of the
program counter and stack pointer are no core—for example, the Cortex-A7. This is an
longer part of the general purpose register set ARMv7 design with LPAE, and, from the soft-
and can only be accessed through a few ware point-of-view, is architecturally identical
instructions. to the Cortex-A15. This is important, for it
Another large change is removing the abili- allows pairs of cores to be used together in an
ty for most instructions to be conditionally SMP system—for instance, having four power-
executed. Now, only the branch instruction is efficient Cortex-A7 cores and two high-per-
able to be conditionally executed. This condi- formance Cortex-A15 cores. The software can
tional execution was one of the features of turn these on and off as needed. This configu-
earlier chips; however, they used 4 bits from ration is known as big.LITTLE, and is designed
each instruction, leaving only 28 bits to to provide improved battery life while offering
encode the instruction. performance when required.
The ARM naming scheme can be confusing. The ARMv8 architecture follows this by
ARMv8, in this case, describes the instruction introducing the Cortex-A53 and Cortex-A57
sets available along with other important cores from ARM as well as a number of third-
architectural areas like how the cache is party designs. One example is Project Denver
expected to work, or how to program the from Nvidia, a design that takes the ARM
MMU. The architecture will be backwards instructions and translates them into an inter-
compatible with the previous revision, but may nal instruction set that may, at a later stage,
add a new flag or an incompatible option. An be optimized. This is a continuation of the
example of the latter is that some ARMv7 Transmeta x86 chips, just with a new input
designs have added support for larger physical instruction set.
addresses in the MMU, known as the Large ARMv8 also changes the operation of
Physical Address Extension, or LPAE. With exception states. These exception states can
these, the existing page table format will still be thought of as different privilege levels.
work, but the operating system can use the These were rather complex in the 32-bit chips,
new format to access more than 4GB of physi- where interrupt and system-call handling are
cal memory. different states and where they both have the
Not only are there ARMv names, but there same privilege. With ARMv8 there are four
are also design families. For ARMv7, ARM has exception levels; the lower three can be
its Cortex-A designs. These designs are all thought of as the userland, kernel, and hyper-
ARMv7 designs, but have some different char- visor states. The most privileged level is nor-
acteristics—e.g., they may have different mally programmed by the chip vendor and is
pipeline lengths or may implement LPAE sup- used to abstract some of the chip’s functionali-
port. There is also an option for a company to ty, for example, to provide a power-manage-
design its own ARM core, known as an ment interface. These states are known as EL0
Architecture License, where the licensees to EL3 where the larger number is at higher
design their own compatible core. One com- privilege. These could be used as EL0 for user-

Jan/Feb 2015 15
 land, EL1 for the kernel, and EL2 for bhyve there were no issues involved in modifying it to
of xen. be used as a starting point for experimentation.
Initially, these experimentations were simple,
The arm64 Project as there was no documentation—just an
overview of the instruction set. But, as is too
I have been working on this project in my spare
often the case with FreeBSD, this required infer-
time for the last two years, and, as of November
ring how the hardware worked by looking at
2014, as part of a FreeBSD Foundation project
Linux and finding the similarities with ARMv7.
to port FreeBSD to the AArch64. Initially, this
Some things—for example, the MMU page
involved writing simple code to bootstrap the
tables—are very similar and can be worked on,
hardware and to allow jumping to C code. Later,
whereas others, like the system registers, have
this was brought into the FreeBSD tree in a proj-
been changed and so they may or may not be
ect branch. Finally, for the FreeBSD Foundation
the same. Even without this documentation, I
project, this has moved to GitHub to allow col-
progressed to the point where I could move
laboration among a number of developers. This
from pure assembly to executing C code. The
new architecture will be known in FreeBSD as
MMU was still turned off, but this allowed the
arm64.
rest of the code to be tested.
The project has two goals: The first, and
In September 2013, ARM released the ARMv8
where I am focusing my time, is to lay the foun-
Architecture Reference Manual, known as the
dation for FreeBSD to run on a number of differ-
ARM ARM. This allowed the last parts to be
ent chips, which includes the internal kernel
written for enabling the MMU and for any
infrastructure and userland. There is a second
magic numbers to be correctly documented.
goal, which is to have FreeBSD run on the
With this, the code could be moved from
Cavium ThunderX processor. This is a server chip
GitHub into a FreeBSD project branch. As part of
with up to 48 ARMv8 cores that can operate in
this, llvm and clang in the base could be used,
a dual-chip configuration allowing for 96 cores.
as they are new enough to generate AArch64
Working on porting an operating system to a
code, and with very little work can generate
new platform before hardware is available can
code for FreeBSD. A copy of binutils is still need-
be difficult. Luckily, ARM released a version of
ed to provide a few missing FreeBSD tools, as
their simulator—known as the Foundation
the copy in base is too old.
Model—so that developers could test their code.
This has been where all of my work has taken
place, including getting to the point where user- The Boot Environment
land can be run. It is expected that the boot environment on
Most of the code written for any new port is AArch64 servers will be UEFI based, and, to sim-
within the machine-dependent parts of the ker- plify booting, having a UEFI-enabled FreeBSD
nel, and the arm64 port is no exception. There loader is an important step in the process. When
are a number of steps required to get a new working on the port, the loader can be thought
port with the first being a toolchain. When the of as a simple, single-threaded kernel.
project was started, there was only a single tool- Unfortunately, the UEFI boot environment needs
chain available—gcc with binutils—which the loader to be an EFI binary. The process is a
required me to port both to build FreeBSD bina- little more difficult with AArch64 than with
ries. (This was mostly copying the existing Linux other UEFI platforms, as the AArch64 binutils is
and modifying as needed.) Having previously unable to convert from an ELF file to the needed
worked on the ARM EABI, I realized this PE+. To work around this, a combination of
required a similar approach, and so I was famil- assembly, linker scripts, and the use of objcopy
iar enough with both projects to get them is needed to create the image. A PE+ header is
working. created in an assembly file, a linker script pro-
But a toolchain is just the first step. One also vides the needed information as to the size of
needs either hardware or a simulator to test the binary, and objcopy will copy from the ELF
booting the hardware. This is where the image to a binary image, leaving the PE+ header
Foundation Model was useful, as it allowed for at the beginning of the binary.
experimentation a number of years before the Even with the loader being in the correct for-
hardware would arrive. And ARM released a mat, more work is required for it to run and
simple boot wrapper to initialize the simulator to load the kernel. First the loader may need to be
be used by Linux. As this is under a BSD license, relocated in a similar process to loading a

16 FreeBSD Journal
dynamic library. A few functions also need to be ed. Bootstrapping the pmap layer can be a little
implemented, including functions to copy data tricky. The existing mappings need to be taken
between physical memory and what will be the into account and new mappings added.
kernel’s virtual memory. Most of these, along Working with the page tables can often be
with the main function, can be copied and problematic, as configuring an entry without per-
adjusted as needed from an existing port. forming the correct TLB invalidation can lead to
unexpected results. To assist with this, ARM has
The Kernel added a useful instruction to see how the hard-
ware is performing the address translation. This
For AArch64 I decided to use the semihosting file
instruction, the AT instruction, performs an
system to load the loader and kernel. Semi-
address translation and reports back the result
hosting is a method where software running on
for the developer to inspect. This, along with
an ARM processor may access the host environ-
software-based page table walking, has proved
ment. With physical hardware, this works
to be useful in tracking down problems. The
through the debug adapter, while the simulator
address translation instruction can report back a
allows for any file to be accessed in the directory
number of useful bits of information, including
from which it was run. The UEFI environment
whether or not the translation was successful
exports this as a file system available to any EFI
and either the physical address or a flag to
applications. To access this, a new file system
describe why it failed. It is also able to attempt
handler was written that included working
the translation on both userland and kernel
around bugs in the UEFI implementation. With
addresses. It is common for this instruction to be
this, the kernel can be loaded without creating a
the first place to check when the kernel exhibits
disk image, thus allowing faster development.
odd behavior, as was the case recently where the
Having gotten the loader to the point where it
shell was crashing just after forking to execute a
was capable of running and loading a kernel, the
command. With the help of this feature, the
next step was to build the kernel for it to load
problems with the context switching and pmap
and run. The kernel needs a number of machine-
code were found and fixed.
dependent functions to be written. To begin
Having finished this early platform initializa-
with, these can be stub functions where each
tion, the kernel enters into the machine inde-
called panic printing its name. With the kernel
pendent code. Yet even at this stage it will still
building, it needs to be loaded and have initial
call back into the machine dependent code. One
code to run. As I had written my earlier experi-
place the kernel performs this is for device enu-
mentations with a plan to import them into
meration. On FreeBSD we have a tree of devices,
FreeBSD, this simplified the process, as I already
the root of which is the nexus device. The job of
had code to start execution of the kernel.
nexus is to handle any bus drivers the architec-
But even with this code in the kernel, we are
ture may need, and for this it needs to imple-
still only able to get into the kernel’s C code.
ment a number of resource handling functions.
From here, work is needed to continue the early
These involve allocating and releasing device
bootstrap. This includes parsing any kernel meta-
memory and interrupts, and configuring the
data passed to it by the loader, bootstrapping
interrupt. This configuration is needed, as there
pmap, and initializing the kernel. The kernel
may be a few different interrupt controllers—
metadata included data describing the hardware,
which is the case on 32-bit ARM. Depending on
which could be either a Flattened Device Tree
the configuration, there may be the standard
Blob, or ACPI data. By having the loader pass in
ARM Generic Interrupt Controller; however, this
this data, we are able to keep the kernel generic.
is only common on chips that support SMP. On
FreeBSD has a layer named pmap to handle
older designs, designers used their own interrupt
the machine dependent parts of the virtual mem-
controllers.
ory subsystem, for example, updating the MMU
Nexus also needs to handle device memory
to adjust which physical address a virtual address
allocation. For this it configures the base case for
points to. This will be different for each architec-
memory resource in the resource management
ture, and, as is the case with 32-bit ARM and
abstraction. The driver needs to handle any phys-
PowerPC, may need different implementations
ical addresses that are to be allocated from.
within a single architecture. A decision was made
FreeBSD will then restrict the size of the physical
to include a single, large, direct mapped region
memory to manage the use of Open Firmware
on arm64. This simplifies the code when a physi-
and Flattened Device Tree buses. It is expected
cal address to virtual address calculation is need-

Jan/Feb 2015 17
 that ACPI will work in a similar way; however, As the internal ARM device bus is memory
work has yet to start on this, as most of the mapped, this requires bus space to allocate a
ARMv8 focus thus far has been to use Flattened block of virtual addresses and then map them to
Device Tree, even on Linux. the correct physical address. For this, the existing
Having nexus and the Open Firmware and 32-bit ARM code can be reused with minimal
Flattened Device Tree devices is not enough to changes.
boot a system. We also need, at a minimum, an There are a few other parts of the kernel that
interrupt controller, a timer, and some form of need to be implemented for early booting. The
console. On the Foundation Model, these are the kernel expects to create kernel threads. For this,
Generic Interrupt Controller version 2, the Global the kernel has a few functions to configure or
Timer, and an ARM pl011 UART, respectively. All needs to duplicate new threads and processes
of these are already supported by FreeBSD; how- and to switch between them. Some of these
ever, work was needed for them to work on need to duplicate an existing process’s data or
ARMv8. The simplest was the UART. Due to how create it from scratch, whereas others handle
ARM specified the device tree for the Foundation writing the state of the current process to mem-
Model, the base address of the parent bus need- ory and loading a new process’s state along with
ed to be hard-coded, but this was later fixed to any cache handling requirements. It can be tricky
correctly parse the data. to get these correct and one good way to test
The interrupt controller and timer drivers them is to get the userland working, as it will
needed more work. The interrupt controller was quickly show when, for example, one process is
changed to allow support for multiple con- accessing the wrong memory and changing
trollers. This is based on the PowerPC design, but another process’s stack.
it is expected this will be replaced with the It is unlikely to get to this point without hit-
updated ARM design when this separate project ting a hardware exception. The main cause will
is integrated into CURRENT. With the timer, the be some sort of memory fault—for example,
method for accessing timer values on 32-bit when the kernel is accessing an invalid virtual
ARM is through a system co-processor. This is address. To handle this, I wrote a simple excep-
accessed using instructions for moving data tion handler. It needed to store the register state,
between this co-processor and ARM registers. call the appropriate handling function, then
On AArch64 these have moved to special regis- restore the registers and return to where it was
ters. Along with this, there are two groups of executing from. Early on, the handler could be
these registers, one for the physical timer and simple, as it only needed to handle the kernel
another for the virtual timer. This virtual timer is map where the address would have the top bit
set up to be the physical timer plus a random set. If we found an invalid address, the kernel
delta. Normally the kernel will only have access could dump the saved registers to help track
to the virtual timer; however, it may get access down the issue.
to the virtual timer in some cases, for example, At this stage, configuring interrupts worked
as a Hypervisor. To handle this, the driver’s inter- along with allocating and accessing device mem-
nal API was updated to allow access to either of ory, process creation and switching, and simple
these modes depending on the situation. exception handling. This allowed the kernel to
The kernel won’t get very far with just these reach its first major milestone, getting to the
three devices. The first problem is that it needs mountroot prompt. This is the point where
to read and write to device memory. This enough of the system is running that the kernel
abstraction is known as bus space and allows a tries to mount the root file system and fails. The
single driver to talk to a bus without knowing scheduler is running off the timer interrupt, the
the details of the bus. Bus space provides a num- system UART is working, and any triggered
ber of functions to read and write to a device, exceptions are at least handled, even if it is to
and, eventually, all of these need to be written. panic.
During the early work only a few are needed, Having gotten this far with the project, there
the main group being the reading and writing of is just a kernel running. The userland is still not
a single item, be it 1, 2, 4 or 8 bytes. Without working, and it needs some sort of file system.
this, no drivers are able to access the devices FreeBSD has the ability to embed a small file sys-
they are controlling. tem image into the kernel, but to build this, bits
The bus space abstraction also handles any of userland need to be ported, and the system
mapping and unmapping of these bus resources. call interface needs to be defined.

18
 AArch64 handles system calls in the
kernel as hardware exceptions, with
the kernel deciding how to decode
which system call userland is accessing.
For this, I have decided to follow the
example of Linux and store the excep-
tion value in a register. Both libc and
the kernel need to be updated to
encode or handle these respectively.
Along with this libc, functions such as
setjmp and longjmp need to be writ-
ten. Initially, only a static library is
needed, which can simplify the early
work. As more of userland is ported,
the required libc functions can be
added. On arm64, only a very minimal
set of functions has been implemented,
but more are being worked on as
needed.
To start executing an application,
there is a small piece of machine-
dependent userland code named csu.
This has to load argc and argv from
memory along with the environment
and a few other pointers set by the
kernel. There may also be code that
needs to be run before main, for exam-
ple, to create static C++ objects. This is
all handled by csu.
Most of the userland libraries needed
to get a simple file system can be built
with this. The minimum requirements for
a usable file system are init and sh.
Having built both of these, I used the
makefs command. This takes a directory
structure and generates a UFS file sys-
tem image. If the size of this image is binaries on the Foundation Model. Work has
small enough, it can be embedded into the kernel; started on loading dynamic executables; however,
a 4-MB image is large enough to hold a static and this is still only running from single user mode.
stripped init, sh and ls. Further work is progressing toward a stable
Having implemented all the required kernel stub port. In the short term, work is being done to get
functions, and with a little luck, init should start FreeBSD running on the Cavium ThunderX hard-
executing. To begin with, only indirect evidence ware, along with support for dynamic binaries
will exist, for example, by observing the state in and multi-user mode. The port is still only run-
exception handlers. As more bugs are fixed, init ning on a single core, and getting SMP working is
should get further along until if init is set up to a requirement due to the large number of cores
boot to single-user mode, it will print out a mes- in the ThunderX.
sage to select the shell. There may be further fixes Looking further ahead, there will need to be
required at this point, but the kernel should be stability work to make sure the code will work
mostly ready for userland, and init will try to run when used in a production environment. One
the shell, which can be used to run any other way to perform this is to attempt to build ports
applications in the file system. on the hardware. This will also give us a chance
This is the point to which the arm64 port has to see the state of the ports tree on this new
progressed as of the time of this writing. FreeBSD platform. There is also interest in other items like
can boot with a kernel file system and run static

Jan/Feb 2015 19
 4
DTrace and hwpmc. Along with this there will to their chips. Further sponsorship will also help

6
always be work to port FreeBSD to any new in getting FreeBSD to a stable and production-
hardware that turns up, as, unfortunately, a ready state on ARMv8. •
large number of devices are specific to each
hardware vendor. Andrew Turner started with FreeBSD on ARM
by porting it to the Samsung CPU within the
Acknowledgments OpenMoko phones and is responsible for
bringing in ARM EABI support to FreeBSD. He
I would like to thank the FreeBSD Foundation, has worked as an embedded software engi-
along with ARM Ltd and Cavium for sponsoring neer on projects from deeply embedded
the project. As this is just the start, further work ARM devices with a few kB of RAM, to multi-
will be needed. The chip vendors need to release core ARM boards with many gabytes of mem-
their documentation so FreeBSD can be ported ory. He also works as a contractor porting
FreeBSD to ARMv8 chips.

arm64 ”ESSENTIALLY READY FOR USERLAND”

FURTHER READING
The FreeBSD wiki – https://wiki.freebsd.org/arm64
Technology Preview: The ARMv8 Architecture – http://www.arm.com/files/downloads/ARMv8_white_paper_v5.pdf
ARM Architecture Reference Manual –
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0406c/index.html

TM

TM

JOURNAL

DYNAMIC EDITION!
$19.9SU9B The Dynamic Edition format
YEAR

$ 6.99 offers subscribers the same

COPY
SINGLE features as the App, but
permits viewing the Journal
through your favorite browser.
The DE, like the App, is an individual product.
You will get an email notification when an issue is released.

Read It Today! www.freebsdfoundation.com

20 FreeBSDJournal
 NEW EDITION – Now Available!

The most complete, authoritative technical guide to the FreeBSD

kernel’s internal structure has now been extensively updated to
cover all major improvements between Versions 5 and 11.

SAVE 35%
When you order from informit.com/freebsd
Use discount code FREEBSD35 during checkout
EBOOK FORMATS INCLUDE EPUB, MOBI, AND PDF – ALL FOR ONE PRICE  FREE SHIPPING WITHIN THE U.S.

Terms & Conditions: Discount code FREEBSD35 is applied to list price of Design and Implementation of FreeBSD, Second Edition
print or eBook and cannot be combined with any other offers. Offer is only good at informit.com.

Jan/Feb 2015 21
 I N T E R A C T I N G with the FreeBSD Project
by Dru Lavigne

Getting
Involved
As an open-source project, FreeBSD relies on the ongoing contributions
and diverse skills of a vibrant community in order to create and maintain
software that meets the needs of its users. Perhaps you have thought
about contributing to the FreeBSD Project but are not sure how to get
started. Or, perhaps you already contribute to the Project, but would like
to encourage others to get involved as well.
‘

T
his article provides a general overview of
how and why to start contributing to the
FreeBSD Project. It also addresses some of
the concerns and questions commonly voiced by
new contributors.

But I Don’t Write Code!

It’s true: the FreeBSD Project creates an operat-
ing system that is primarily written in the C pro-
gramming language. While developers are always
needed, this does not mean that the only contri-
butions that the Project accepts are in the form
of C code. Examples of non-code contributions
include, but are not limited to:
Documentation: The FreeBSD Project pro-
vides man pages, which include working exam-
ples, and a comprehensive Handbook for using
FreeBSD. This documentation is an ever-changing
target as new features are added to software
releases and users come up with new use cases
for using the software. It takes the contributions
of writers, who create new documentation; edi-
tors, who review and fine-tune documentation
changes; and users, who report outdated or
incorrect documentation, to keep the documen-
tation resources useful and up-to-date.
Translations: FreeBSD is an international com-
munity and English is not the first language for
many users. Having documentation available in
one’s native language significantly reduces the
learning curve for new users. If English is not

22 FreeBSD Journal
your first language, consider assisting with the Advocacy: Do you attend any technical confer-
translation of the Handbook into your native lan- ences or have a local technical user group?
guage and the impact that translation can have for Consider giving a presentation, lightning talk, or
users in your geographic region. BoF (Birds of a Feather) session about what you are
Events and News Items: The main page of the doing with FreeBSD. If your presentation is accept-
FreeBSD website is typically one’s first exposure to ed, send a note to the freebsd-doc mailing list so
FreeBSD. Submitting news items and upcoming that the information about the event can be listed
events is a quick and effective way to make sure on the FreeBSD website.
this content reflects the dynamic and far-reaching Development: While this section has concen-
nature of the Project. trated on non-code contributions, it is worth not-
Ported Applications: The extensive collection ing that contributions from developers are still
of ports, or software that has been tested to run on needed. As an entire operating system, FreeBSD
FreeBSD, is one of the compelling features of using offers many coding opportunities: device drivers,
FreeBSD. At over 24,400 software applications and networking protocols, file systems, APIs, and hard-
growing, it’s clear that it takes the contributions of ware subsystems, to name a few.
many users to test and keep applications up-to-date
as new versions become available. Maintaining an But I’m Not an Expert!
existing port or creating a new port for an applica- Hopefully, some of the contribution areas caught
tion that currently isn’t available for FreeBSD helps your interest. But you may be thinking “I'm still
to keep the software collection available to other very new at FreeBSD and have so much to learn!
users while also providing a way to improve your Shouldn’t I wait until I’m an expert before I start
own development or scripting skills. contributing?” In short, the answer is “no, don’t
Testimonials and Case Studies: Does your wait.” You can start contributing using what you
company use FreeBSD or do you use FreeBSD to
know now and the type and depth of your contri-
make your livelihood? The Project is always inter-
butions can grow as you gain experience.
ested in showcasing where, how, and why
The FreeBSD Project does have a lot of contribu-
FreeBSD is used.
tors who are very good at what they do and who
User Support: The FreeBSD Project provides
are known as experts in their field. This can be
many resources so that members of the communi-
pretty intimidating to new or casual users who are
ty can assist each other, including forums, mailing
interested in contributing, but fear that their con-
lists, and IRC channels. Taking the time to answer
tributions are too small or too basic to make any
unanswered questions that you know the answer
difference. If you feel this way, don’t forget that
to or to point new users to the resources they
the community is much, much larger than the
need may not seem like much, but it helps to build
names that you are familiar with and that a large
a strong community.
portion of the community is learning as they go
Social Media: Do you live on Twitter? Consider
and relies on the contributions of other users.
adding Tweets of interest to FreeBSD users using
#freebsd. Are you doing interesting things with
FreeBSD or trying out new things as you learn to Why Get Involved?
use FreeBSD? Consider blogging your efforts, as This is a very good question as it sometimes seems
they will be of interest to other FreeBSD users that contributing to an open-source project is a
while adding to the collective knowledge of “yes, one-way street. Sure, it is nice to give back when
you can do xyz on FreeBSD and here’s how.” Or, if you're using something for free, but we all have
you prefer to talk instead of write, contact the only so much time and energy to spare. In reality,
folks at bsdnow.tv or bsdtalk.org to arrange for a there are many compelling (and even downright
short interview. self-serving) reasons for contributing back to any
Monetary Support: Is life just too busy for you open-source project. The actual reasons will vary
right now to contribute time, but you still want to depending upon the people and their goals, but
assist the Project? Consider making a monetary here are some examples that apply to contributing
donation to the FreeBSD Foundation, knowing that to open source in general:
the funds will be used to assist the Project. Before Students, the unemployed or underemployed,
doing so, check to see if your employer has a and those individuals looking to improve their job
donation matching program or if they would be prospects need a way to differentiate themselves
interested in becoming a financial sponsor of the in the job market. Open-source contributions,
Foundation. especially over time, can be used to show initiative

Jan/Feb 2015 23
 (I did this in my spare time), to learn from senior- excels at to make sure that it matches your inter-
level technologists, to improve technical skills ests and goals. FreeBSD excels in many technical
(here is an online list of my contributions), to areas, including:
improve soft skills (I’ve worked with others and Networking: FreeBSD inherited an excellent net-
within processes as part of a large, organized working heritage from the original Berkeley
project), and to cultivate contacts with experts in Software Distribution and continues to innovate in
the field. the networking field. For example, did you know
It’s fun! Where else do you get to interact with that BSD sockets provided the original API for
people from all over the world who happen to TCP/IP? Or that FreeBSD provided the first IPv6-
share your passion for that bit of the open-source only networking stack? Recent innovations include
world that has caught your interest? This is espe- being the research platform for modular TCP con-
cially true if you are the only person in your family, gestion control algorithms and the development
workplace, or town that has even heard of that of netmap for testing high performance direct-to-
bit of open-source technology. It can be a relief to hardware packet I/O.
work with others without having to first figure Security: FreeBSD is well known for being a
out which terms to use in order to explain what it secure operating system and for working closely
is you do. with security researchers as a reference platform for
It’s free! Where else can you learn about the new security frameworks. Jails have provided light-
internal components of an operating system and weight isolation and operating system virtualization
how these evolve over time, with nothing more since 2000 and continue to see improvements such
than an Internet connection and your spare time? as fine-tuned resource limits. The Capsicum refer-
FreeBSD is particularly well suited for this purpose. ence platform is defining new approaches for
Not only is it well documented, its code and com- extending operating system security.
mit history have been publicly available since the File Systems: File system innovations began with
Project’s inception in 1993. BSD (the FFS introduced cylinder groups) and con-
There’s no glass ceiling. Since open-source con- tinue to this day. For example, SUJ adds journaled
tributions come in over the Internet, no one needs soft updates to UFS, and HAST provides synchro-
to know your age, gender, race, economic status, nous, block-level replication over TCP/IP. Several
or sexual orientation. Contributors become known FreeBSD developers are active in the OpenZFS
by the quality of their contributions and the areas community and this modern file system is built
in which they contribute. Stick around for a while into FreeBSD.
and you’ll find that you have turned into a Development Tools: The FreeBSD Project was an
respected expert in your niche. early adopter of Coverity Prevent and an early
Organizations that build a product or service adopter of the CLANG/LLVM toolchain. FreeBSD
using an open-source technology quickly learn to provides built-in DTrace for system performance
appreciate the pain that comes with maintaining analysis.
their own fork of that project. Even if contributing In addition, the FreeBSD Project excels in sever-
back patches and improvements is not required by al nontechnical areas:
the open-source license, the amount of develop- License: The two-clause BSD license encourages
ment resources that can be saved when doing so the reuse of code. This allows the producers of
quickly adds up. In addition, organizations that commercial products to concentrate on their
are looking for a competitive advantage do well “secret sauce” and encourages the wide adoption
to cultivate a working relationship with the project of new standards.
they use: Being an active participant increases the Documentation: The Project provides many
likelihood that your voice is heard when the sources of documentation to help new users and
Project is considering new features and its devel- new contributors to quickly get up to speed.
opment roadmap. These include the FreeBSD Handbook, FAQ,
Developer’s Handbook, Porter’s Handbook, the
Why FreeBSD? Documentation Project Primer, and the built-in
man pages.
The previous section outlined some advantages to
Mentoring: The Project’s processes and culture
contributing to open source in general. Why con-
revolve around a mentoring environment. New
tribute to FreeBSD in particular, given that there
code, documentation, and ports contributors are
are literally thousands of open-source projects
assigned several mentors who review and commit
looking for contributors?
their patches as well as provide assistance for
Before contributing to any open-source project,
learning the Project’s coding/documentation stan-
it is worthwhile to research what that project
dards and best practices.

24 FreeBSD Journal
Community: As open-source projects go, the As an example, let’s say you have found a typo
FreeBSD community is a friendly, encouraging, in a man page or you have a patch that allows a
well-informed space. There is a wide range of broken port to successfully build. Go to
expertise and experience levels and a culture of bugs.freebsd.org, create a new account, and con-
learning and professionalism. firm the account-creation email. You can then use
Processes: As a mature (21 years) community, the the quick links to create a bug report and attach
FreeBSD Project is very organized and has well- your patch. More details on how to create a useful
documented processes. There are several organiza- bug report can be found at
tional teams, including an elected core team to freebsd.org/support/bugreports.html. In theory,
help set direction and resolve disputes, a security someone will review your patch, possibly ask for
officer and security team to manage security advi- more information, and commit the change for you.
sories, a release engineer and release engineering Once committed, your contribution is now “live”
team to manage software releases, a documenta- and able to benefit other FreeBSD users. Quick and
tion engineering team to manage the documenta- painless!
tion infrastructure, a port management team to What if you have grander ambitions than just a
manage the Ports Collection, and various infra- one-off patch that fixes some little thing you hap-
structure teams to manage the Project’s websites, pened to stumble across? In this case, you want to
repositories, and other infrastructure. start integrating yourself into the community.
Typically this is done by subscribing to the mailing
How Do I Get Started? list(s) that match your interest(s). Depending upon
the technical nature of the list and your comfort
Contributing code or documentation to FreeBSD
level, you might introduce yourself and your inter-
can be as simple as finding something that inter-
ests or just lurk for a while in order to get a good
ests/bugs you and submitting a patch that makes it
feel for the tone of the conversations and a better
better. This section elaborates on that statement
idea of who does what.
by providing some helpful tips for interacting with
As you submit patches, don’t get frustrated if
the Project.

FreeBSD Journal
The NEW publication for FreeBSD buffs, is OLD enough now to have

6 BACK ISSUES!
JO UR NA
Jan/Feb
2014

L
TM

March/April 2014

JOU RNA L
Vol. 1 • Issue No. 2
May/June 2014 Vol. 1 • Issue No. 3
JOURNAL
V I R T U A L I Z A T I O N
TM

July/August 2014

JOURNAL
Vol. 1 • Issue No. 4

Vol. 1 •
Issue No.
1

FreeBSD in the Amazon

Networking EC2
pkg(8)
DTrace
BeagleBon

svn
e Black
Started
Getting

update
Source Tree
Changes

10
CLANG in& Library
NEW PROBES
KQUEUE
MADNESS
TCP
SCALING
IPFW
Cloud XEN
piler
Technology
A New Com OVERVIEW
DNSSEC DATA DISTRIBUTION USING bhyve
GEME NT SYSTE M
A MODE RN PACK AGE MANA Hypervisor

FreeBSD10 Poudriere Journaled Soft-updates

Format hwpmc(4)
The Evolution of the PBI

D
FO R WA R
M O V IN G

read all you missed!

Order as desktop editions at www.freebsdfoundation.org
or purchase at your favorite app store.

Jan/Feb 2015 25
 they are not reviewed right away. There may be an area that interests you, submit good patches,
several reasons for this. If the patch is fairly long interact with your reviewer, and learn from their
or complex, a developer needs to take the time suggestions.
to digest and understand the patch and how it Be aware that the community notices people
fits into the larger scheme of things. If the patch who stick around, respect others, seem willing
is fairly simple, double-check that you have to learn, and continue to contribute. Don’t be
included all of the necessary information in your surprised if someone at some point asks if you
submission and that the utility of the patch is would like to be mentored. This is a good thing.
obvious and not too esoteric. If a week or so has
gone by without any feedback, send a polite Additional Resources
email to a related mailing list to inquire if any- In summary, these resources cover a variety of
thing else is needed to assist patch reviewers. Be types of contributions:
sure to promptly respond with any information • Submit a code, port, or doc patch or report a
that is requested to help expedite the patch bug: bugs.freebsd.org
review process. Remember that your patches are • Submit event listings and news items:
being submitted to a community that consists of lists.freebsd.org/mailman/listinfo/freebsd-doc
real people with busy lives and limited free time.
• Assist new users: forums.freebsd.org
They do share your passion for FreeBSD and
• Discuss the creation of a testimonial/case study
want to work with others who respect their time
or request company introductions or a
and abilities.
Foundation visit by sending an email to
Another contribution example is a company
[email protected]
that would like to contribute back their patches.
• Donate to the Foundation:
In this instance, the best case scenario is having
freebsdfoundation.org/donate
at least one employee who either has a relation-
• Include @freebsdblogs when Tweeting new
ship with the FreeBSD community or who hap-
FreeBSD-related blog posts
pens to be a FreeBSD developer. The worst case
• Find additional documentation at
scenario is to dump a large chunk of code into a
freebsd.org/docs.html
bug report—as any developer knows, it is very
difficult and time-consuming to test and inte-
grate large chunks of someone else’s code into Dru Lavigne has been using FreeBSD as her
an existing codebase. If the organization does primary platform since 1997 and is the lead
not have an existing relationship with the com- documentation writer for the FreeBSD-
munity and is interested in establishing one, the derived PC-BSD and FreeNAS projects. She is
Foundation can assist in making introductions to author of BSD Hacks, The Best of FreeBSD
the appropriate developers and possibly in Basics, and The Definitive Guide to PC-BSD.
arranging a visit with the organization’s engi- She is founder and current Chair of the BSD
neering and management teams to further dis- Certification Group Inc., a nonprofit organiza-
cuss their collaboration goals. tion with a mission to create the standard for
As an individual, keep the following points in certifying BSD system administrators, and
mind when interacting with the FreeBSD community: serves on the Board of the FreeBSD
Don’t be shy. Your contributions matter. Find Foundation.

We Can’t Do This
Without
YOU!
Your contribution makes a real difference! Help the PLEASE DO
FreeBSD Foundation Support: •Project Development YOUR PART
& DONATE
•FreeBSD Advocacy •Growth of the FreeBSD Journal
TODAY
•And More! h t t p s / / w w w. f r e e b s d f o u n d a t i o n . o r g
26 FreeBSD Journal
 TM
,

# WE WANT YOU!#

Testers, Systems Administrators,

Authors, Advocates, and of course
Programmers to join any of our diverse teams.

# DOWNLOAD OUR SOFTWARE #

http://www.freebsd.org/where.html

# JOIN OUR MAILING LISTS #

http://www.freebsd.org/community/mailinglists.html?

# ATTEND A CONFERENCE #
• http://www.bsdcan.org • http://www.asiabsdcon.org

The FreeBSD Project

# #

Jan/Feb 2015 27
 INTERACTING with the FreeBSD Project
by Rick Miller and Julien Charbon

A Journey
• From Adoption
to Contribution
Selecting the right platform on which to
deploy a product is fairly straightforward,
in that it requires evaluating available options
against a product’s key metrics. And even deploy-
ing the platform is relatively trivial, barring the
complexity of integrating it with systems like provi-
sioning and installation frameworks. FreeBSD was
able to meet our requirements, and thus was
adopted and deployed as a diversity platform in
the Domain Name System infrastructure operated
by Verisign. But we didn’t stop there.

T
o encourage further development of FreeBSD, we became—
and remain—an active participant and collaborator within
the community, which now includes one of our own engi-
neers who has become a committer to the project.
The decision to adopt and deploy FreeBSD globally within
Verisign’s Domain Name System infrastructure was just the start
of our involvement. We knew from the beginning that it was
important—and necessary—to actively engage and collabo-
rate with the FreeBSD Foundation, the project, and the sup-
port community. But before engaging with the communi-
ty, it was important for us to understand the project
and who operates it and to comprehend the
ecosystem—how the components relate to one
another and how they operate together.
The FreeBSD website (http://freebsd.org)
contains valuable resources that describe
the FreeBSD ecosystem in terms of phi-
losophy, policy, processes, and prac-
tices. The functions of FreeBSD’s
core, release, and security
teams are also well docu-
mented with detailed descrip-
tions of the practices and
processes for each. Additionally,

28 FreeBSD Journal
the policies for communicating with, submitting the San Francisco metropolitan area and Ottawa,
patches to, and becoming a committer are found Canada, respectively. The benefits of participating in
here. Understanding these functions and processes these events are numerous. They range from a
provides deeper insight into how the various com- casual conversation with a committer with whom
ponents interact with one another. one has been conversing via email or mailing list, to
something as complex as sitting in a workgroup at
Electronic Communication a conference-hosted hacker lounge and collabora-
Our communication with the FreeBSD project tively completing a project.
At MeetBSD 2012, our engineers experienced
began via email on the various project mailing lists.
these benefits firsthand as they had the opportuni-
The mailing list archives contain examples of col-
ty of meeting, in person, with various individuals
laborative efforts and include technical interactions
they had become familiar with through mailing
like customizing and building release media and
lists and email. The rapport between iXsystems
ports/packages, integration of FreeBSD into instal-
and Verisign culminated in the establishment of
lation frameworks, driver functionality,
Verisign’s first-ever vBSDcon in 2013—a great way
network performance, and more.
to actively give back to the community.
An unexpected mode of commu-
BSDCan is, undoubtedly, the “must-attend”
nication arose early on while eval-
BSD-related conference on the North American
uating vendor hardware against
continent, and it was in 2013 that Verisign sent its
FreeBSD. Although this mode
first small delegation to attend the event. The
was exercised only once, it
experiences gained at MeetBSD were built upon at
provided results. A network
BSDCan, as it included invitations to attend the
card in the hardware failed
Vendor and Developer Summits, which precede
to function as expected,
and so we communicated some BSD-related conferences. Additionally, it put
our engineers in direct contact with FreeBSD engi-
the problem to the vendor,
neers who were leading projects in which Verisign
who subsequently put us
had a specific interest—such as networking per-
in touch with an organiza-
formance.
tion that was a heavy
user of FreeBSD. This
organization, which had Verisign’s Experience
committers on staff, From a development perspective, porting the
assisted us in identifying Verisign core DNS infrastructure software stack
an existing upstream onto FreeBSD was seamless. In large part, this was
patch, which mitigated due to much of the prerequisite software being
the functional issue that supported by default and its high level of POSIX
was subsequently merged compliance. The last step for FreeBSD validation as
into internal development a diversity operating system was to run stability
branches. While this proved and performance tests. However, Verisign’s work-
helpful, we also realize it’s load is specific enough to drive unique issues with-
unlikely to be exercised in the operating system like kernel panics, driver
again due to the rarity of the crashes, scalability issues, and more.
scenario and also because we A kernel panic within udp_input() was one of
now have internal talent to the first interesting issues encountered. Engineers
address such issues on our own. designed and executed experiments implementing
code to reliably reproduce the problem in order to
Technical Networking devise a proper solution which was later submitted
to the project through the bug tracking system [1].
Electronic communication is valuable,
It was imperative that we debug the problem and
but in an increasingly well-connected world,
have confidence in its scope before opening the
technical networking still provides invaluable bene-
bug report. This expedited technical discussions
fits. Networking occurs at specially targeted events
with appropriate committers and engendered con-
such as local and/or regional user groups and con-
fidence in the abilities of Verisign engineers to
ferences like the regional BSD user group, CapBUG,
diagnose and resolve the issues. Traits like thor-
based in the Washington D.C./Baltimore region, and
oughness, patience, and perseverance in bug
conferences like MeetBSD and BSDCan, based in
reports and mailing list communications attracted

Jan/Feb 2015 29
 the attention of FreeBSD developers working in the From an email on freebsd-questions asking
same fields. about customizing installation media to filing the
Continued development facilitated discovery of first bug report to attending and organizing con-
more interesting edge cases [2][3][4] and encour- ferences to obtaining commit bit privileges, this
aged tighter collaboration with the FreeBSD proj- journey illustrates the level of work and dedication
ect. The Developer’s Summit prior to BSDCan 2013 that it takes to become involved in meaningful
was the ideal venue to describe and discuss our ways. It takes a great deal of time by dedicated
findings with those working on related technolo- engineers—from both organizations—sometimes
gies as well as with the broader community. The performing work not directly related to their jobs.
in-person collaboration at the Developer Summit But it is done from a sense of contributing to a
and BSDCan enabled specifically targeted coopera- project that aims to provide the community a plat-
tion that resulted in an increased pace of the sub- form on which to achieve their own goals. We
mission/review/acceptance life cycle for code and thank the FreeBSD project, its community, and the
patches. Foundation for their continued support, and we
git, a well-designed tool that enables collabora- look forward to future collaborations, perhaps at
tive management of the full FreeBSD stack, was the upcoming vBSDcon 2015 (to be formally
utilized to keep track of internal changes to announced at a later date). •
FreeBSD sources using the official FreeBSD git mir-
ror. This enables Verisign to build FreeBSD with Julien Charbon is a software development engi-
patches for testing under real scenarios—prior to neer at Verisign Inc. Julien has worked on the
submitting them back to the project. Additionally, company’s high-scale network service ATLAS
it helps maintain the relevancy of patches, particu- platform and related high-scale network servic-
larly when review and patch improvement time- es. Julien has worked with FreeBSD to perform
lines don’t line up. Furthermore, our relevant tasks including porting software, developing
changes are tracked and published, making them kernel fixes and patches, and network stack
available to the project for inclusion in their own performance improvements.
development branches [5].
The associations continue to evolve, resulting in Rick Miller is a UNIX systems engineer at
broader exposure of our work in the community Verisign Inc., where he builds infrastructure
through such avenues as the BSDnow podcast, a systems supporting global DNS resolution plat-
renewed invitation to the BSDCan 2014 DevSummit, forms. Rick’s focus over the previous five years
and our article on a TCP network stack performance has been integrating and deploying FreeBSD
scalability issue and Verisign’s proposed strategy to into these platforms. This includes building
solve it in FreeBSD Journal’s May/June 2014 issue development/operational support systems and
[6]. And, last but not least, one of Verisign’s engi- managing operating system source code and
neers [Julien Charbon] was welcomed into the com- image builds.
munity of FreeBSD committers.

References
[1] Kernel panic in udp_input()
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=172963

[2] Concurrency in ixgbe driving out-of-order packet process and spurious RST
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=176446

[3] TCP stack lock contention with short-lived connections

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=183659

[4] Kernel panic: Sleeping thread owns a non-sleepable lock from

netinet/in_multi.c
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=185043

[5] Verisign’s FreeBSD public repository

https://github.com/verisign/freebsd

[6] FreeBSD Journal May/June 2014 issue

https://mydigitalpublication.com/publication/?i=217642\

30 FreeBSD Journal
 BOOK review by Rik Farrow
•
McKUSICK
McKUSICK

The Design and

NEVILLE-NEIL
WATSON
T
WATSON
IMPLEMENT
IMPLEMENTATION

Implementation of the
THE DESIGN AND

FreeBSD Operating System,

TAT
TIO OF THE
TION

THE
DESIGN AND
IMPLEMENTATION
Second Edition
®
OF THE
Marshall Kirk McKusick, George V. Neville-Neil, and Robert N. M. Watson
®
OPERATING
OPERA

® 2015 Pearson Education, ISBN 978-0-321-96897-5 859 pages

OPERATING SYSTEM
ATING
T S
SYSTEM

SECOND EDITION

T
his book comes how they need to work. Approaches that log
®

SECOND
EDITION
MARSHALL KIRKMcKUSICK out of a lineage metadata updates are considered in the follow-
GEORGE V. NEVILLE-NEIL
of books about ing section (the approach used in Linux’s ext3).
ROBERT N.M. WATSON

the BSD operating‘ sys- Like operating systems books in general, the
tem, starting with the book begins with a history of UNIX written by
Design of 4.3BSD UNIX one of its participants, followed by an overview
in 1989. While being focused on FreeBSD sets of the kernel. Process management follows,
this book apart from other operating systems then a completely rewritten chapter on security.
books where the focus is Linux, that’s not the If you are seriously interested in operating sys-
only thing that sets it apart. tem security features, this chapter provides an
Kirk McKusick has been involved in key excellent overview of the many mechanisms that
design decisions that still have bearing on UNIX- have appeared, and been implemented, over the
related systems since he was a graduate student past 25 years. While the Linux Security Module
sharing an office with Bill Joy. And this book and the related SELinux and Type Enforcement
reflects not only McKusick’s influence on the get only brief mention, there are thorough dis-
designs of file systems and virtual-memory sys- cussions of access control lists, mandatory access
tems, but also that of its two other authors. control, the new NFSv4 ACLs, security event
Where a book like Robert Love’s Linux Kernel auditing, cryptographic services, random number
Development dives into getting, building, and generator, jails, and the Capsicum capabilities
examining kernel code, Design and model, a recent addition to FreeBSD.
Implementation stays at a higher level. The next chapter, on memory management, is
Algorithms and data structures are explained, just as long as the security chapter, and just as
but so are the design decisions behind why a detailed. The next part of the book covers the
particular algorithm‘ or design was chosen. I/O system, starting with overview, then devices
Soft updates provide a particularly con- in general, moving to FFS, then a new chapter
tentious example. Early Linux file systems could on the Zettabyte File System. Again, this chapter
create and delete files much faster than the 4.3 would be useful to anyone who wants a deep
BSD fast file system (FFS), because the authors understanding of ZFS, whether you are using
of ext2 had decided to do away with ordered, FreeBSD, or Solaris descendants like Illumos. The
synchronous writes of file system metadata. The I/O section ends with a chapter on NFS, includ-
FreeBSD developers’ response, led by McKusick, ing NFSv4.
was to create a process called soft updates, Part four covers Interprocess Communication,
which allows metadata updates to occur asyn- which begins with IPC and continues with chap-
chronously, but still in an ordered manner. Soft ters on network layer protocols, like IPv4 and
updates are explained in clear and concise text, IPv6, followed by transport layer protocols. The
both why they are considered necessary and book concludes with a chapter on system start-

32 FreeBSD Journal
up and shutdown, and a glossary.
FreeBSD
n
Each chapter ends with exercises and one or
more pages of references. The exercises cover
ideas from each chapter, as well, to help the dedi-

S T I G
O
cated reader think about potential solutions that
go beyond what’s covered in each chapter.
I did what I usually do with large technical

Vi P S H u d
books: I jumped around, after reading all of the
introductory material, focusing on the parts I
found most interesting. The writing makes this
easy to do, in that I rarely found myself referred to n the C l o
another section in the book.
If you want or need to have a better under-
standing of how the FreeBSD operating system
works, you should get this book. You can read all
of it, or just the parts you need (once you’ve read
• Deploy in 30 Seconds
“If you want or need to • 24/7/365 Live Support
have a better understand-
ing of how the FreeBSD
• No Contract,
operating system works,
Cancel Anytime
you should get this book. • Free Setup, Free DNS
You can read all of it, or
just the parts you need.” • 100% Enterprise SSD
• 1TB Outbound Transfer
the first section), and expect answers to your
questions about why something is done in a par-
ticular way, such as the use of paging, IPv6 imple-
• 1 Gbps Port
mentation, or virtualization with FreeBSD. And
while each version of BSD is different, the versions
do borrow a lot from each other, and roughly have
the same design philosophy. What makes this

Plans star t at onl y

book special is that there is a wealth of experi-
ence, a record of different approaches taken, writ-
ten by three FreeBSD committers, all with stellar
records. •

Rik Farrow is editor of ;login:, the USENIX mem-

99 c
per month
bership magazine. Rik began working with
Unix in 1983 by writing systems documentation
for microcomputer-based UNIX systems, then
went on to write several books about Unix. His
research into Unix security led him into teach-
ing and consulting about UNIX and Internet
security for many years. Along the way, he con-
sulted as technical editor for UnixWorld maga-
zine. Rik is also tutorial manager for USENIX.

GeT STarTeD TODaY!

Jan/Feb 2015 33
 conference REPORT
by Wallace Barrow
•

meetBSD
oing to any conference alone can be somewhat intimidating. Scheduling,
G traveling, and knowing no one can put you on edge. This was not the case
for me going to meetBSD 2014. I could not wait to go to my first meetBSD con-
ference, and I was greeted with open arms. Getting involved in a new community
is always fun, as I discovered when I first started using FreeBSD three years ago.

meetBSD California 2014 was awesome! The David Maxwell showed us a new Unix com-
night before the conference started I got to meet mand called pipecut, with which users can create
a handful of great people, and many more during pipeline commands quickly and easily. We can
the conference, including Dru Lavigne, Sean create hotkeys stitching many Unix commands
Bruno, Jim Maurer, Michael Dexter, and Glen together to manipulate data and see each step
Barber, just to name a few. Putting names to of the process. I got to sit down with David at
faces is a surreal experience. the social mixer and discuss how I will use this in
I greatly enjoyed the breakout sessions and my own environment, which was a very cool
wished there was more time to attend them all. experience.
However, of all the discussions, I was most inter- Kirk McKusick told us the story of IPWars. In
ested in bhyve. As a server administrator, I use summary: “If Kirk McKusick had a time machine,
Xen Server on a day-to-day basis, and learning he'd put 48 bits on Ipv4” @Brendan Gregg on
about another hypervisor I could use in produc- Twitter. I also got my new book, The Design and
tion is very intriguing. Talking to the developers Implementation of the FreeBSD Operating System,
directly and learning how this Second Edition, signed by Kirk.
hypervisor works was the Brendan Gregg works for Netflix
most beneficial thing a tech and showed us five major areas of
could ask for. You have direct performance analysis in FreeBSD: the
access to the experts’ knowl- tools, methodologies, benchmark-
edge, and that goes for any ings, tracing, and counters. FreeBSD
topic you are interested in at provides 34.2% of Internet traffic
meetBSD. I hope to start some today because of Netflix.
benchmarking of Bhyve in a If you have never been to a meetBSD
lab I have set up very soon. conference or any BSD conference,
start planning to do so and making
Presenters arrangements now. From the people,
We had a great two days of the presenters, the breakout sessions,
presenters. Rick Reed from WhatsApp showed us you get all you ever wanted from it. I am very
a great outline of a half a billion FreeBSD users. excited to attend the next conference in Canada.
Currently WhatsApp holds over 600M monthly See you all then! •
users, 140M concurrent connections, 440K con-
nections per second, and 1.1M messages sent per With a major in computer science and a
second. minor in security, Wallace Barrow worked for
Jordan Hubbard gave us an overview of the University of Wisconsin administrating a
“FreeBSD: The Next 10 Years” in light of cele- Windows Domain for an agriculture depart-
brating FreeBSD’s 21st birthday on November 2, ment. He began working with FreeBSD three
2014. Jordan explained how we need to become years ago when he took a job with a small,
more Lego-like in the architecture we create and local ISP in Madison, Wisconsin. Wallace
choose hardware platforms that have mass loves his Packers, Badgers, and open source!
appeal in the industry.

34 FreeBSD Journal
 TM
TM

®
FreeBSDJ O U R N A L

LEAKTHIS!
• BLOG IT • TWEET IT • READ IT

TM

FreeBSD Journal is available as a

mobile app for iPad, iPhone, Kindle,
and all Android devices. Also available
as a desktop Edition at www.freebsd
foundation.org. $19.99 per year
(6 issues), or $6.99 per single copy.
TM

FreeBSDTM Journal is published by The FreeBSD Foundation

 this month
In FreeBSD BY DRU LAVIGNE

As a mature open-source project, the FreeBSD Project has a well-

defined and well-documented process for managing releases, as well
as a Release Engineering Team and Lead Release Engineer. This month,
we had a chance to interview Glen Barber, who has been on the
FreeBSD Release Engineering Team since January 2013 and lead
Release Engineer since March 2014.
••••••••••••••••••

Q Tell us a bit about your-

self. What is your tech-
on a spare desktop-style com-
puter. There were a number of
things I liked about FreeBSD
A The FreeBSD release cycle
actually starts several
months before any publicly-visi-
nology background and how (and some that I did not like), ble changes take place, such as
did you get started with but overall, something about it the code freeze or release candi-
FreeBSD? just felt right. In particular, I date builds.
liked the separation between the Lately, we’ve tried to have the
A I’m 33 years old and cur-
rently live in Pennsylvania,
USA. I've been using FreeBSD
base system and third-party soft-
ware, which was one of the
schedule for the next release
written and agreed upon within
things I did not like about the Release Engineering Team
since around 2006, after reading
Linux—each distribution did three months prior to the start
about it in a Linux magazine. I
their “base install” differently, of the cycle, after which we’ll
remember sitting in the waiting
and, of course, each with a dif- send several reminder emails to
room of a doctor’s office and
ferent utility. The ability to run a the FreeBSD developers.
came across a small article about
few commands and end up with While this may seem trivial,
this thing called FreeBSD, which
just the base installation was it’s quite important to communi-
had just released 6.1-RC1. I fold-
one of the things I found most cate the target dates with
ed over the corner of the page
attractive about FreeBSD. FreeBSD developers very early
as a reminder to try FreeBSD
Once I was comfortable because we typically get a num-
when I got home from the
enough with FreeBSD, I replaced ber of replies informing of work
appointment, mainly because I
the Linux installation on my lap- that is in progress that is target-
was looking for a different oper-
top with 6.1-RELEASE, and since ed to be included in the release,
ating system to run on my Dell
then, every computer I’ve owned such as new features or non-
Inspiron b120 laptop.
(laptops, desktops, and servers) trivial bug fixes. In some cases,
At the time, I had just
has all been FreeBSD systems. we will grant blanket approvals
resumed going to college. I had
In hindsight, I’m often to developers, so pre-commit
been using Linux on my laptop
amused at the almost accidental approval is not required for each
for maybe one or two years,
chain of events that led to my change when the code freeze is
because it was easy to find dif-
involvement in FreeBSD, espe- in effect. Granting blanket
ferent software and tools. Back
cially considering my roles within approval depends on several fac-
then, I had no idea what I want-
the Project. tors, such as the scale of the tar-
ed to do for a career, but I knew
I wanted to work with comput- geted changes and how self-
•••••••••••••••••• contained the changes are
ers. This was when I first
Most readers will already expected to be. For example,
became aware of open-source
software and started to discover
Q have some idea of how
the release engineering
something like a new network
device driver would likely be a
all of the great projects and
process works. What is the candidate for granting a blanket
communities that existed.
typical workday like for a approval, while a major overhaul
So when I got home after the
Release Engineer? How does of the virtual memory subsystem
doctor’s appointment, I down-
that work day evolve as a would not.
loaded the 6.1-RC1 installer, and
release date draws near? Writing the schedule for an
installed my first FreeBSD system

36 FreeBSD Journal
operating system release is a bit Each change during the code x86 architectures have been
more complex than one might freeze requires explicit review cross-built on amd64 hardware,
imagine, because there are so and approval from a member of which makes the order of com-
many different moving pieces the Release Engineering Team, pletion predictable. Once the
that need to all be coordinated unless a blanket approval is builds are started around 00:00
and timed perfectly for every- granted. There is a lot of email UTC (about 20:00 local time), I
thing to fit together. For exam- during the release cycle, but can be fairly certain that the
ple, we need to make sure we luckily there are a number of FreeBSD/amd64 and
have binary packages built for highly active FreeBSD develop- FreeBSD/i386 builds will finish
the release, which cannot be ers on the Release Engineering around midnight, after which I
done until we get close to the Team, so the turnaround time copy the installation medium to
end of the release cycle. The for the review/approval process my local test machines. This
FreeBSD Documentation Project is quite low. Depending on the usually takes a few hours, so I’ll
sources need to be tagged for change in question, we’ll occa- get a few hours sleep while the
the release, so the documenta- sionally defer to someone with- remaining builds finish running
tion packages are available for in Release Engineering who has in serial, and the x86 builds fin-
the release, and this needs to expertise in the part of the sys- ish copying locally.
happen before the package tem for the change review, or Of course, this is assuming
builds. Since the FreeBSD Ports request review from someone there aren’t any problems.
Collection started branching the outside of Release Engineering, When things do go wrong, they
tree quarterly, it also makes which may delay a response to need to be addressed. The
sense to align the package the approval request, but it is problems can vary and do not
builds with the quarterly not usually too much of a delay. necessarily indicate a problem
branch, which is also the basis About a day before the start with the FreeBSD source tree.
of the release tag. of a BETA or RC build, I will There are human factors as
Then, each phase of the send an email to the Release well. On more than one occa-
release cycle, such as BETA and Engineering Team to inform sion (more times than I'd like to
RC builds, needs to be timed that the builds will begin as admit), I have had to restart
properly. For 10.1-RELEASE, planned, and any incoming BETA, RC, and, yes, even final
each BETA and RC build took commit approval requests RELEASE builds because I’ve
around nine hours, from start should not be answered until made a mistake. These mistakes
to finish for all of the supported after the builds are completed can range from setting the
architectures. If anything goes and have been tested. I’ll also incorrect Subversion revision for
wrong during the builds (either send a similar email to the the tree in the release.conf con-
from human error or a bug in FreeBSD developers to inform figuration file for the build, to a
the build process), the builds them that any approval subtle typographical error in a
need to be restarted from the
beginning. Then it takes rough-
ly another eight hours for the
images to propagate to the FTP
mirrors. Then there needs to be
enough time allocated for test-
“ The evolution of any given FreeBSD release is
interesting because there are so many
different factors involved.
”
ing, both by the Release requests will be answered after configuration file. Mistakes
Engineers and FreeBSD con- the builds have finished, so happen, and when they do,
sumers, and a sufficient amount developers do not think their they guarantee a sleepless
of time for bug fixes before the request was not received. night.
next builds begin. This all needs Lately, I’ve been trying to Once the builds finish, I copy
to be considered months in start the builds just after 00:00 the installation images to a
advance when writing the UTC, but this isn’t a defined public web server, and send a
schedule, and as you can see, part of our work flow or poli- PGP-signed email to the
makes it very easy to fall behind cies. The main reason for this is FreeBSD Release Engineering
schedule. It’s surprisingly com- the length of time the builds Team and FreeBSD Security
plex. It is really like putting take, and the timing based on Team, providing the URL for the
together a puzzle, in a way. the order in which the builds images where others on the
Once the code freeze starts, complete. Since FreeBSD 9.2- Release Engineering Team can
things begin to pick up quickly. RELEASE, release builds for non- download and help with test-

Jan/Feb 2015 37
 ing. The Security Team then
starts building the various sets “ Being the Release Engineer for an operating
system is tough, and probably not
of patches for freebsd-
update(8) consumers.
Once the images are tested,
they are uploaded to the pri-
mary FTP mirror, pending prop-
comparable to any other software project.

always the potential of intro-

ducing a regression, which
could make the last few weeks
may seem like an obvious
”
be done is testing. While this

answer, there are a few things

agation to the various world- of the release cycle particularly that cannot be tested until a
wide mirrors. I’ll then send an stressful. prerequisite stage has hap-
email to the Release Engineers As an example, imagine pened, so each stage of the
and Security Team to give an there is a problem report about release cycle may differ slightly.
approximate time frame of a disk controller driver that For example, when BETA1
when the announcement email scribbles garbage to the drive builds are complete, I tend to
will be sent to the mailing lists. every 300th write. It’s a serious focus my attention on the
After the announcement email problem, and certainly behavior installer and other “setup” class
is sent, we start the process that is far from desirable. As software in the base system. A
over again by replying to the FreeBSD Release Engineers, the problem in bsdinstall(8) can
commit approval requests that solution is much less simple quickly change my focus for
have been pending since the than “just fix it,” especially BETA2 if a serious problem is
builds started, and keeping a when we have already found. My primary testing envi-
close eye on the mailing lists announced what was, accord- ronment consists of a relatively
and problem report system for ing to the schedule, the final modern, powerful machine
issues that may arise. release candidate. At that point with a significant amount of
The evolution of any given of the cycle, every solution is disk space (4.2 TB free space
FreeBSD release is interesting non-ideal. right now) running FreeBSD-
• because there are so many dif-
ferent factors involved. The
A driver update will make it
necessary to add another
CURRENT. I use VirtualBox to
test the installation images and
influx of commit approval release candidate to the sched- installed system because creat-
requests at the start of the ule. But what if it isn’t such an ing machines with varying con-
release cycle is fairly steady, but easy fix as a driver update? figurations can be scripted.
as we progress through each of What if the fix requires an I start with three provisioned
the various stages of the release update to the cam(4) subsys- virtual machines, one for
cycle, we need to be increasing- tem? Will the fix adversely FreeBSD/amd64, one for
ly more cautious about what affect other drivers using FreeBSD/i386, and recently
changes we will allow. Con- cam(4)? Are we even sure the added to the setup, one for
versely, as the cycle progresses problem is not due to malfunc- FreeBSD/amd64 with UEFI. I
and BETA and RC builds are tioning hardware and it isn’t a take these provisioned
released, there seems to be a bug at all? These are all ques- machines, each already config-
slow, but steady increase in tions that need to be answered, ured with virtual hard drives,
adoption of what will be the and their answers each play a allocated RAM, and virtual
next release. significant role in what, ulti- CPUs, and clone them for vari-
I should note that we don’t mately, will be a difficult deci- ous different installation paths.
have any hard evidence to sup- sion to the question “What do I’ll create a “FreeBSD amd64
port this other than my own we do about this?” bootonly” virtual machine to
observations relating to the boot-test the bootonly.iso
stage of the release cycle and •••••••••••••••• image, but since the distribu-
problem reports. As we get What is the checklist of tion sets do not yet exist on the
closer to the end of the release
cycle, we need to be extremely
Q items that needs to hap-
pen before a release can be
FTP mirrors, I cannot do much
more with those images once
cautious as to what change released? they are verified to boot. We do
requests are approved. This not publish the images on the
becomes particularly challeng-
ing at the end of the release
A In addition to what was
mentioned earlier regard-
ing the order in which events
FTP mirrors until they are test-
ed, in case we find a critical
cycle, because we do want to need to take place when problem that makes it necessary
fix outstanding issues with the preparing the release schedule, to redo the build, in which case
release; however, there is the primary thing that needs to it would be more difficult to

38 FreeBSD Journal
pull the problematic builds release. The caveat here is if munication channels open. The
from the mirror sites. there is a change that affects Release Engineering Team and
For the disc1.iso installer either the ABI (Application Security Team need to be in
image, I create four independ- Binary Interface) or API sync in various other parts of
ent virtual machines. One for (Application Programming the process, such as determin-
testing the installation, using Interface) in FreeBSD after the ing the end-of-life date for the
the default for all options; one package builds have finished, release, and following up on
for testing the installation they must be rebuilt. Unfor- any outstanding errata items
using a root-on-ZFS setup; one tunately, this does happen. after the release.
for testing root-on-ZFS with We generally have the final-
GELI encrypted disks; the last ized package set available for ••••••••••••••••
one is a clone of the first. RC2, and at that point I start You have been the
When the installation is com-
plete, I install a GNOME desk-
to focus the testing more on
using the system, such as test-
Q Lead Release Engineer
for almost a year now. What
top on the first virtual machine ing various parts of GNOME have you learned? Any sur-
and a KDE on the second, and and KDE. I’ll also install various prises along the way?
do basic testing. Since the server software, such as
finalized package sets are far
off in the future, it does not
Apache, Nginx, and
PostgreSQL, to make sure
A It has been a learning
experience, that’s for sure.
I certainly learned how difficult
make sense to spend too much there are no glaringly-obvious the job of being the Release
time doing in-depth testing of problems. Engineer is. I truly cannot
the desktop machines just yet, The benefit, I think, of doing express my respect and grati-
given how quickly the ports testing this way is that I can tude to all of the FreeBSD
tree changes. What does not worry less about installation Release Engineers enough.
work today may work tomor- issues near the end of the Being the Release Engineer
row, and vice-versa. release cycle and focus atten- for an operating system is
Then I create two virtual tion on what we expect to be tough, and probably not com-
machines for the dvd.iso the released operating system. parable to any other software
installer, one for GNOME and project. And, to be clear, I
one for KDE. Since I have •••••••••••••••• don’t at all mean that other
already tested the installation The tasks performed by
paths using the disc1.iso, I
don’t need to spend too much
Q a Release Engineer are
just one part of a very large
software projects are not as
important as an operating sys-
tem. But, when releasing an
time on the installer at this project. Who else does the operating system, there really
point, and can focus on testing Release Engineering Team is great potential for disaster.
package installation from the work with? It is, without a doubt, the
packages included on the DVD
itself.
When the BETA2 builds are
A We work very closely with
the FreeBSD developers,
mainly through the
most difficult job I’ve ever had.
It is also the most rewarding
job I’ve ever had, and an expe-
ready in the next phase of the review/approval process, and rience unlike any other. It’s one
release cycle, I use the change the Ports Management and of the reasons I’m truly grate-
list from the subversion logs as Documentation Engineering ful for the FreeBSD developers
the focus of the testing. Teams to coordinate tagging and FreeBSD community. We
Assuming there were not any the various source trees and have amazing people in the
changes to the installer since package builds. FreeBSD Project.
BETA1, I can focus less on the However, I would have to You can learn more about
details of the various installa- say that we work most closely the FreeBSD Release
tion paths, and spend more with the FreeBSD Security Engineering Team and its
time testing (as much as possi- Team, and for a number of release engineering process
ble, given hardware limitations) reasons. The FreeBSD Security at https://www.freebsd.org/
what has changed since Officer and Deputy Security releng/.
BETA1. Officer are the two people that
When the first release candi- have access to the freebsd- Dru Lavigne is a Director of
date is ready, the FreeBSD update(8) build machines, so the FreeBSD Foundation and
Ports Management Team can during the BETA and RC phas- Chair of the BSD Certification
do the package builds for the es we need to keep our com- Group.

Jan/Feb 2015 39
 svn UPDATE by Glen Barber

IT’S THAT TIME AGAIN—a new year exists. LOCALBASE, in FreeBSD nomenclature, is
the root of the non-base installation path,
and a great time to take a look at some of the /usr/local.
new features and enhancements that FreeBSD The rc(8)Subsystem
developers have been working on. Sit back, http://svnweb.freebsd.org/changeset/base/276918
relax, and enjoy the highlights.
Whether you are a FreeBSD systems
T he rc(8) subsystem is one of the most essen-
tial subsystems in FreeBSD used to determine
whether or not a service is started at boot, and
administrator, application developer, or also for command-line arguments that should be
hobbyist, changes to the system, in par- used in conjunction when the service is started.
ticular utilities and configurations, are Although the rc(8) subsystem is arguably one
some of the most important things to of the most complex pieces of FreeBSD, its con-
keep an eye on. Here are some figuration has remained remarkably simple for
administrators, and its simplicity and extensibility
changes that can be expected in
has had another update. The rc(8) subsystem
upcoming FreeBSD releases. now has support for allowing service startup
configuration files in both /etc/rc.conf.d and
The mailwrapper(8) Utility LOCALBASE/etc/rc.conf.d (which expands to
http://svnweb.freebsd.org/changeset/base/270675 /usr/local/etc/rc.conf.d by default), allowing
administrators to create smaller configuration
T he mailwrapper(8) utility is used to invoke
the appropriate MTA (Mail Transfer Agent)
such as Sendmail, Postfix, or qmail based on
files for system configuration.
Each file within the rc.conf.d directory could
what is specified by the configuration file contain, for example, the type of service which
mailer.conf(5). The mailer.conf(5) file is used to then would contain the configuration parame-
map certain commands used by the mail subsys- ters for the service. One popular example of this
tem to the absolute path of the program that is creating a /etc/rc.conf.d/jail file to contain all
should be used for the command. jail(8) startup parameters, and a
For example, mailer.conf(5) may contain com- /usr/local/etc/rc.conf.d/postfix file, which con-
mands such as 'sendmail', 'mailq', 'newaliases', tains startup parameters for the Postfix mail
and 'hoststat'. On FreeBSD, these commands server.
map to the /usr/libexec/sendmail/sendmail pro-
gram, which is based on the command name The linux(4) ABI Layer
that is used to invoke it, and will behave differ- http://svnweb.freebsd.org/changeset/base/271982
ently to modify parts of the email system.
newaliases(8) will rebuild the aliases(5) database
file aliases.db; hoststat(8) will print the host sta-
A fter much work, much testing, and even
more necessity, the linux(4) ABI (Application
Binary Interface) layer has been updated from
tus database used during SMTP transactions, etc.
Fedora Core 10 support to CentOS 6 support,
Historically, mailer.conf(5) would exist in the
allowing a number of recent, Linux-only applica-
base system configuration directory, /etc/mail,
tions to run on FreeBSD. To check the Linux ABI
but as of revision 276917 it is now possible to
compatibility version on a system, examine the
put the mailwrapper(8) configuration file else-
output of the compat.linux.osrelease sysctl(8):
where on the system, avoiding conflicts and/or
- 2.6.16: Fedora Core 10
inadvertent reversal of local changes. The mail-
- 2.6.18: CentOS 6
wrapper(8) utility now respects the LOCALBASE
If it is necessary to continue using the Fedora
environment variable, and will search there first
Core 10 ABI layer, the default can be rolled back
for its configuration file, which overrides the
by adding 'compat.linux.osrelease=2.6.16' to
FreeBSD base system default mailer.conf(5) if it
sysctl.conf(5).

40 FreeBSD Journal
 •
The bsdinstall(8) Installer The crypto(4) Driver
http://svnweb.freebsd.org/changeset/base/272274 http://svnweb.freebsd.org/changeset/base/275732

T he bsdinstall(8) installer is the default installa-

tion utility since the FreeBSD 9.0-RELEASE.
Since the 9.0-RELEASE, bsdinstall(8) has received
T he crypto(4) driver, part of the OpenCrypto
framework, supports two new cryptographic
modes—AES-ICM and AES-GCM. The crypto(4)
various updates and enhancements to supplant driver provides userspace access to hardware-
its successor, sysinstall(8). accelerated cryptography devices and is used in
The change in revision 272274 is important conjunction with various other cryptography
for both systems administrators and developers devices, such as aesni(4), ipsec(4), padlock(4),
alike, especially when using root-on-ZFS installa- and random(4).
tions—the default /var dataset now has the 'can- The OpenCrypto framework updates were
mount=off' ZFS property set, so although /var is sponsored by The FreeBSD Foundation.
created by default, it is not automatically mount-
ed at boot. This allows the system to have multi- The vxlan(4) Device
ple /var datasets, which, when used with multi- http://svnweb.freebsd.org/changeset/base/273331
boot environments such as the sysutils/beadm
port, prevent the /var directory from numerous
individual environments from conflicting with
T he vxlan(4) device was recently added to
FreeBSD which, analogous to vlan(4), creates
a virtual tunnel endpoint. The vxlan(4) segment is
each other.
a virtual Layer 2 network overlaid in a Layer 3
There are a large number of very important
endpoint, which, compared to vlan(4), is better
files within /var. Everything from the pkg(8) data-
suited for multi-tenant data center environments.
base, to backup files generated from periodic(8)
runs, to crontab(5) files, to system crash dumps
exist there. This makes it very important to The gre(4) Driver
ensure that numerous independent environments http://svnweb.freebsd.org/changeset/base/274246
that share nothing more than the physical hard-
ware on which they run (and the underlying utili-
ty to control which environment boots by
T he gre(4) driver, which provides a generic
routing encapsulation interface, has been
split into two separate modules—gre(4) and
default) do not share this data, so one environ- me(4). The gre(4) module itself has been over-
ment does not collide with the other, such as hauled significantly, with its counterpart, me(4),
installed packages. providing an interface for more minimal encap-
sulation within Layer 3.

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •
THANK YOU, READERS, for supporting the FreeBSD community, FreeBSD Journal,
and, of course, The FreeBSD Foundation.
Don’t forget, development ISO and preinstalled virtual machine images (in VHD, VMDK,
QCOW2, and RAW formats) of the FreeBSD-CURRENT and FreeBSD-STABLE branches can be
found on the FTP mirrors, and are built weekly: ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/.
As always, development snapshots are not intended for production use; however, we do
encourage regular testing so we can make the next FreeBSD releases as great as you expect
them to be.

As a hobbyist, Glen Barber became heavily involved with the FreeBSD project around
2007. Since then, he has been involved with various functions, and his latest roles have
allowed him to focus on systems administration and release engineering in the Project.
Glen lives in Pennsylvania, USA.

Jan/Feb 2015 41
 PORTSreport
by Frederic Culot
•
In November and December the activity on the ports tree was not high, mainly due to
end-of-year celebrations. Still, the figure is impressive with 3,778 commits on the ports
tree! On the bug front, however, more problem reports were closed than during last peri-
od, with 1,157 problems fixed. Thanks to all of you who contributed feedback and took
the time to make FreeBSD ports better!
YEARLY figures
IMPORTANT PORTS UPDATES 2014 was the year that saw the highest
everal exp-runs were performed (22 actually) number of commits in all of our ports
S to check whether major ports updates are safe
or not. Among those important updates, we men-
tree’s history! Whereas we never reached
more than 30,000 commits before, 2014
tion the following highlights: saw almost 37,500 commits applied to our
ports [https://people.freebsd.org/~eadler/
• llvm and clang updated to 3.5.0
datum/ports/commits_by_year.png]. So we
• xorg-server updated to 1.14
take this opportunity to thank all our
• gnome updated to 3.14.2
developers and contributors, and we hope
• enlightenment updated to 0.19.2
to see such dedication in 2015 as well!
• pkg updated to 1.4
• default Perl version set to 5.18
• default PostgreSQL version set to 9.3
As is usual, if manual steps are needed to update
mits to the ports tree, with more than 20,000 com-
mits since 2006!
specific ports, then those steps are clearly men-
tioned in the /usr/ports/UPDATING file. It is highly NEW portmgr LURKERS
advised to check this file before performing any
update on the ports tree!
Also, please note that default versions of ports
E very four months the port management
team welcomes a new pair of lurkers, i.e.,
two ports committers who are given an opportuni-
having multiple versions available are set in the ty to contribute at a higher level, learn the inner
/usr/ports/Mk/bsd.default-versions.mk file. If for workings of portmgr@, and to share in the work-
some reason the default version of a port does not load. Those two lurkers were added to the portm-
suit your needs, then you can simply override it by gr@ mailing list, and will have access to confiden-
adding the DEFAULT_VERSIONS variable in tial correspondence. They are encouraged to partic-
make.conf as follows: ipate in all discussions and add their voice to the
• DEFAULT_VERSIONS= perl5=5.16 ruby=1.9 outcome of decisions taken by portmgr@.
A new term began in November, and our two
NEW PORTS COMMITTERS lurkers are now ak@ and sunpoet@. As a tradition,
AND SAFEKEEPING they were asked to answer a questionnaire to get
uring the last two months, several developers to know them a bit better. Here are links to the
D who already owned an src commit bit decided
to join our community and were granted a ports bit,
questionnaires answered by Alex
[http://blogs.freebsdish.org/portmgr/2014/11/04/ge
too. Those are: jmg@, jmmv@, and truckman@. Also, tting-to-know-your-portmgr-lurker-ak/] and Po-
Muhammad Rahman was granted a ports commit bit Chuan [http://blogs.freebsdish.org/portm-
and will be mentored by marino@ and bapt@. gr/2014/12/03/getting-to-know-your-portmgr-lurk-
Only one commit bit was taken in for safekeep- er-sunpoet/]
ing during the last two months (motoyuki@), but
we also received sad news from miwi@, who Frederic Culot has worked in the IT industry for the
decided to step down from duties at FreeBSD to past 10 years. During his spare time he studies busi-
focus more on his family and professional life. ness and management and just completed an MBA.
Unless you have been hiding under a rock for the Frederic joined FreeBSD in 2010 as a ports commit-
past 10 years, you have likely heard about Martin ter, and since then has made around 2,000 commits,
(miwi@) Wilke. Simply put, Martin is the developer mentored six new committers, and now assumes the
who has contributed the largest number of com- responsibilities of portmgr-secretary.

42 FreeBSD Journal
 BSDCAN 2015
2 COME JOIN US AT THE 12TH ANNUAL BSDCAN!
Ottawa, Canada
WHERE .........................................................Ottawa, Canada
THE TECHNICAL University of Ottawa
BSD CONFERENCE.
The BSD Conference held in Ottawa,
WHEN ......................Thurs. & Fri. June 10-11 (tutorials)
Canada, has quickly established Fri. & Sat. June 12-14 (conference)
itself as the technical conference for
people working on and with 4.4BSD
BSD Certification • Courseware DVD • Register for an Exam
Join the growing ranks of people taking the BSDA exam.
based operating systems and related If you missed it in the past, now is your chance to catch up.
projects. The organizers have found
a fantastic formula that appeals to a GO ONLINE FOR MORE DETAILS
wide range of people from extreme
novices to advanced developers. h t t p : / / w w w. b s d c a n . o r g

WELCOME
to AsiaBSDCon 2015!
AsiaBSDCo Date
n2015

March 12-15, 2015

LOCATION
Tokyo University of Science,
Tokyo, Japan
www.http://asiabsdcon.org

CONTACT [email protected]
 THROUGH JUNE BY DRU LAVIGNE

Events Calendar
The following BSD-related conferences are scheduled
during the first half of 2015. More information about these events,
as well as local user group meetings, can be found at www.bsdevents.org.

SCALE • Feb. 19 – 22 • Los Angeles, CA

http://www.socallinuxexpo.org/scale13x/ • The 13th annual Southern California
Linux Expo will once again provide several FreeBSD-related presentations, a FreeBSD
booth in the expo area, and an opportunity to take the BSDA certification exam. This
event requires registration at a nominal fee.

AsiaBSDCon • March 12 – 15 • Tokyo, Japan

http://2015.asiabsdcon.org/ • This is the annual BSD technical conference for
users and developers on BSD-based systems. It provides several days of workshops,
presentations, a Developer Summit, and an opportunity to take the BSDA
certification exam in either English or Japanese. Registration is required for this event.

LinuxFest Northwest • April 25 & 26 • Bellingham, WA

http://linuxfestnorthwest.org/2015 • This is the 16th year for this annual, community-
based conference. This event is free to attend and always has at least one BSD booth in the
expo area and at least one BSD-related presentation. It also provides an opportunity to take
the BSDA certification exam.

BSDCan • June 10 – 13 • Ottawa, Canada

http://www.bsdcan.org/2015/ • The 12th annual BSDCan will take place in Ottawa, Canada.
This popular conference appeals to a wide range of people from extreme novices to advanced
developers of BSD operating systems. The conference includes a Developer Summit, Vendor
Summit, Doc Sprints, tutorials, and presentations. The BSDA certification exam and
the beta exam for the BSDP will also be available during this event.

Are you aware of a conference, event, or happening that

might be of interest to FreeBSD Journal readers?
Submit calendar entries to [email protected].

44 FreeBSD Journal
THE INTERNET
NEEDS YOU
GET CERTIFIED AND GET IN THERE!
Go to the next level with
Getting the most out of
BSD
CERTIFICATION

. ... .
BSD operating systems requires a
serious level of knowledge
and expertise . . NEED
AN EDGE?
..
SHOW
YOUR STUFF!
Your commitment and
. . BSD Certification can
make all the difference.
Today's Internet is complex.
Companies need individuals with
proven skills to work on some of
the most advanced systems on
dedication to achieving the the Net. With BSD Certification
BSD ASSOCIATE CERTIFICATION YOU’LL HAVE
can bring you to the WHAT IT TAKES!
attention of companies
that need your skills.

BSDCERTIFICATION.ORG
Providing psychometrically valid, globally affordable exams in BSD Systems Administration
FreeBSDTM Journal is published by The FreeBSD
Foundation