The L4Ka Vision

Uwe Dannowski Kevin Elphinstone Jochen Liedtke Gerd Liefländer Espen Skoglund
Volkmar Uhlig Christian Ceelen Andreas Haeberlen Marcus Völp

University of Karlsruhe
System Architecture Group
76128 Karlsruhe, Germany
[email protected]

Abstract client projects that have strongly influenced preparatory L4Ka

work are SawMill, Drops, and Mungi.
Microkernels are minimal but highly flexible kernels. Both con- SawMill is a joint project between the IBM T. J.Watson Re-
ventional and non-classical operating systems can be built on top search Center in Yorktown Heights, NY, and the University of
or adapted to run on top of them. Microkernel-based architectures Karlsruhe. SawMill is mainly funded through IBM Research.
should particularly support extensibility and customizability, ro- SawMill aims at a general methodology for constructing highly-
bustness including reliability and fault tolerance, protection and configurable component-based multi-server operating systems.
security. It complements the current L4Ka project: SawMill focuses on
After desastrous results in the early 90’s, the microkernel ap- how to construct efficient servers and components on top of a mi-
proach now seems to be promising, although it still bears a lot crokernel. It does not only deal with construction from scratch
of research risks. Currently, University of Karlsruhe’s system ar- but also tries to find principles how to take existing systems apart
chitecture group is defining the state of the art in microkernel re- into components. (The first subproject is a multi-server Linux.)
search. The project requires a robust, highly flexible, and extremely per-
The L4Ka research project aims at substantiating and establish- formant microkernel technology.
ing a new methodology for system construction that helps to man- As such, it is a prominent customer of L4Ka. SawMill imposes
age ever-increasing OS complexity and minimizes legacy depen- practically relevant requirements on the L4Ka microkernels, and it
dence. Our vision is a microkernel technology that can be and is gives us an ideal test case for our technology.
used advantageously for constructing any general or customized
operating system including pervasive systems, deep-computing “The Dresden Real-Time Operating Systems project Drops is
systems, and huge servers. a research project aiming at the support of applications with
The technology should help to manage ever-increasing OS Quality-of-Service requirements. Although much research
complexity, enable stepwise innovations in OS technology while has been done on networking support for continuous-media
preserving legacy compatibility, and lead to a widely-accepted applications, very few projects tackle related operating system
foundation of system architecture. issues, such as scheduling and file system support for bounded
response time. The Drops project attempts to find design tech-
niques for the construction of distributed real time operating
1 Status Quo systems whose every component guarantees a certain level of
service to applications.
A key component is L Linux, the Linux server on top of
Managing OS complexity is the challenge for OS technology.
the L4 microkernel; it services standard Linux applications.
The complexity results from combining system requirements such
In addition, separate real-time components, designed from
as security, reliability, configurability, customizability, quality-of- scratch, provide deterministic service to real-time applica-
service guarantees, and performance. The challenge becomes tions. At the moment, an ATM-based real-time protocol and
even harder due to (a) the ever-growing number of services and a real-time file system are being developed.” [Dre]
applications that increasingly interact with each other and (b) the
increasing hardware complexity and parallelism. “Mungi is an operating system based on the idea of a single
OS research focuses therefore on techniques to structure sys- address space, shared by all processes and processors in the
tems dynamically, to isolate a system’s components from each system. The aim of the project is to show that such a single-
other, and to control interaction/communication between those address-space operating system (SASOS) can work on stan-
components. Microkernels, extensible kernels, and sandboxing dard hardware, can be made as secure as traditional systems,
(e.g. through virtual machines) are relevant approaches in this is not inherently less efficient than traditional systems, and
area. Microkernels might have the broadest and highest impact delivers performance advantages over traditional systems on
on systems technology — if they fly. some class of important applications.
In addition, we are trying to build Mungi as a very pure
SASOS, that is all data (even the systems’) is in the single
1.1 Client Projects address space, and no other IPC mechanisms are supported
by the OS. A 64-bit Mungi kernel has been developed. It is
Some projects already use L4Ka technology. Such client projects implemented on top of the L4 microkernel and currently runs
are extremely important and useful for L4Ka. As our “cus- on MIPS R4x00 platforms. Critical to efficient operation of
tomers”, they constantly evaluate our microkernels and also in- Mungi is the implementation of its capability-based protection
fluence our research through their requirements. Closely related system, in particular the protection domain extension (PDX)
 mechanism used to perform protected (cross-domain) proce- IDL . Interface-definition-languages (IDL) are well-known tools
dure calls. Extensive caching of validation information is used from Corba’s world of middleware.

However, IDL compilers,, e.g.
to ensure that good performance.” [Syd] Utah’s Flick compiler [EFF 97], are also beneficial for intra-node
communication,particularly client/server interaction.
In a component-based system, an IDL makes IPC (and RPC)
1.2 L4Ka Status Quo mechanisms conveniently available for application-, component-
, and server-writers. Through interface specifications, e.g. in
The L4 microkernel for Intel’s x86 processors is currently avail- CORBA [Obj] or DCOM [EE98] IDL, remote methods can be
able in two independently developed versions: called as easy as local methods. The IDL compiler generates all
necessary code for marshalling/unmarshalling parameters, method
selection, etc. Surprisingly, we found that with increasing IPC per-
“Lemon Pip” and “Lime Pip” Kernels. The original L4 as- formance, the IDL-generated stub code became a serious bottle-
sembler version for x86 (“Lemon Pip”) has been developed 1995- neck. We had to invest in an experiment on generating sufficiently
98 at IBM Research and GMD. Based on our Joint Study Agree- optimized stub code:
ment with IBM Research, it was maintained and improved in Karl-
sruhe. The current version is highly efficient and stable. Most “As IPC mechanisms become faster, stub-code efficiency be-
microkernel-based research and teaching activities are based on comes a performance issue for local client/server RPCs and
this version. We expect open-source availability in fall 2000. inter-component communication. Inefficient and unnecessary
A derivative of Lemon Pip, the “Lime Pip” kernel has been de- complex marshalling code can almost double communication
veloped in Karlsruhe and currently serves as an experimental ker- costs. We have developed an experimental new IDL com-
nel in the SawMill project. piler that produces near-optimal stub code for gcc and the
L4 microkernel. The current experimental IDL compiler co-
operates with the gcc compiler and its x86 code generator.
“Hazelnut” Kernels. For years, we were convinced that Other compilers or target machines would require different
assembler-based microkernel implementations were necessary for optimizations. In most cases, the generated stub code is ap-
achieving ultimate performance. High costs for maintaining, port- proximately 3 times faster (and shorter) than the code gen-
ing and adapting such microkernels are obvious disadvantages. A erated by a commonly used portable IDL compiler. Bench-
first C++ implementation of L4, Fiasco [Hoh], corroborated the marks have shown that efficient stubs can increase application
claim that higher-level-language implementations offer substan- performance by more than 10 percent. The results are applied
tially less performance. However, we started a new approach that within IBM’s SawMill project that aims at technology

for con-
might result a microkernel-implementation technology that com- structing multi-server operating systems.” [HLP 00]
bines higher-level-language benefits and performance:
Hazelnut kernel have been developed from scratch at Karl- Currently, IDL is an experimental compiler. We plan to improve
sruhe University over the past 10 months. Implementation lan- its flexibility and make it widely usable as a standard tool for the
guages are C++ and assembler. Currently, prototypes are available L4 API.
for x86 and ARM processors as Open Source under GPL since
09/00). Furthermore, IBM Research has started to adapt Hazelnut
Real Time. Early work of the L4 group evaluated principle
to Power PC 750.
techniques of OS-based cache-partitioning to increase the pre-
Early performance measurements of Hazelnut/x86 look very dictability of real-time components in systems that run a mixture
promising. Exactly comparable microbenchmarks show IPC per- of real-time and non-real-time applications [LHH97]. Although
formance within a 10%-range of the original Lime Pip kernel. most real-time dedicated work is done with our partner group at
However, Hazelnut still lacks some important features, most no- TU Dresden, the L4Ka group conducted some further research on
tably the emulation of a tagged TLB on x86 which can improve real-time specific architectures, e.g. how multiprocessor systems
IPC performance by a factor of 3 or more. (For sake of fair- could be used for real-time applications. The memory bus in such
ness, the above mentioned IPC comparison did not use that fea- a system is a common resource for all processors and makes pre-
ture on the Lime Pip.) Implementing that necessary optimization dictability and scheduling even harder than caches do:
in Hazelnut might thus lead to worse results. Another potential
source of surprise may be the cache. Indeed, cache-miss cycles
“Assume that we have 4 job mixes that can be correctly
today dominate in many cases instruction-execution cycles. Mi-
scheduled on 4 independent uniprocessors. What happens if
crokernel costs are thus determined by both the raw execution
we put those 4 job mixes on a 4-processor system with a sin-
time and the cache working set that is required for IPC or other gle memory bus? Without any additional scheduling provi-
primitives. The old Lime Pip kernel has an extremely small cache sions, the shared memory bus can, in the worst case, stretch
working set, about 2% of the L1 cache for an IPC (short mes- each schedule by a factor of 4. This is clearly unacceptable.
sage). We still do not know how small cache working sets can be In general, it would mean that the real-time capacity of an  -
achieved for Hazelnut. It depends on the additional features still processor system is only  of the capacity of a uniprocessor
to be implemented in Hazelnut and on to be explored code/data system. Multiprocessors would be unusable for real-time ap-
optimization techniques that can be combined with gcc. plications.
If it turns out that we succeed in making the full Hazelnut/x86 Therefore, memory-bus scheduling is desirable. It should
perform comparable to the Lime Pip kernel, we will probably drop enable us to give soft and perhaps even hard guarantees in re-
the Pip line and completely switch to Hazelnut. Otherwise, we lation to memory bandwidth and latency to real-time applica-
have to include a very deep performance analysis comparing both tions. For non real-time applications, it should help optimize
versions in detail. Either we can thereafter fix the Hazelnut prob- a system’s overall throughput and/or latency.
lems or have to keep the Pip line in parallel, and adapt it to all Work in this area seems to be rare. The author’s are only
future kernel developments. aware of Frank Bellosa’s work [Bel97].” [LVE00]
Persistence. Making data and perhaps even active programs technology that can be and is used advantageously for constructing

persistent might largely simplify program construction. Data and any general or customized operating system.
programs should automatically survive shutdowns and crashes The technology should help to manage ever-increasing OS
without that the programs explicitly write all their internal data complexity, enable stepwise innovations in OS technology while
into files all the time. Such operating

systems have been around preserving legacy compatibility, and lead to a widely-accepted
since decades [Lan92, Lie93, DdBF 94, SSF99]. The point that is foundation of system architecture.
extremely interesting to us is whether orthogonal persistence can
be implemented easily and efficiently on top of the microkernel.
In other words: is the microkernel sufficiently general and flexible
Strategic Goals
that persistence does not require special kernel integration?
At first, we describe the strategic goals why we want to substanti-
“Orthogonal persistence opens up the possibility for a num-
ate and to establish a microkernel technology.
ber of applications. We present an approach for easily en-
abling transparent orthogonal persistence, basically on top of
a modern  -kernel. Not only are all data objects made per- “A F OUNDATION TO M ANAGE OS C OMPLEXITY.” Ever-
sistent. Threads and tasks are also treated as normal data ob- increasing complexity is a serious problem in OS construction.
jects, making the threads and tasks persistent between system It might be the key challenge for the next decade.
restarts. As such, the system is fault surviving. Persistence Improved complexity management could have strong research
is achieved by the means of a transparent checkpoint server and practical consequences. Research and industry could deal
running in user-level. The checkpoint server takes regular with more complex systems and/or improve system qualities. Fur-
snapshots of all user-level memory in the system, and also thermore, industry might reduce development and maintenance
of the thread control blocks inside the kernel. The execu- costs.
tion of the checkpointing itself is completely transparent to Our primary strategic goal, finding a usable way to manage
the  -kernel, and only a few recovery mechanisms need to be complexity, dominates all further reasoning. Currently, the micro-
implemented inside the kernel in order to support checkpoint- kernel road looks most promising from this point of view. Never-
ing. During system recovery (after a crash or a controlled
theless, our ideas of microkernels might substantially change on
shutdown), the consistency of threads is assured by the fact
that all their user-level state (user memory) and kernel-level
that path, perhaps even be replaced by new concepts beyond mi-
state (thread control blocks) will reside in stable storage. All crokernels.
other kernel state in the system can be reconstructed either
upon initial recovery, or by standard page fault mechanisms “W EAKENING T HE L EGACY D EPENDENCE .” Legacy soft-
during runtime.” [SL00] ware and legacy systems always form a “bed of Prokrustes” for
research, development, and even customers. Innovative and in-
Security. For a microkernel-based system that is built on inter- compatible new systems are hard to establish as research topics
process communication, IPC control, i.e. communication inter- since such systems can not coexist with the “standard” (which will
ception, is a fundamental concept required to implement security sooner or later become legacy). Far too often, innovative ideas
servers and security policies. can also not be explored because they would require to build an
Over the past years, we found that L4’s original concept, Clans entirely new OS with full functionality more or less from scratch.
& Chiefs [Lie92], was not sufficiently flexible and efficient. The The story continues at the industrial development and at the cus-


new Elphinstone-Jaeger redirection model [JEL 99] avoids most tomer level: Legacy requirements make it impossible to develop or
of the old model’s problem (while being upward compatible to the even use incompatible (sub)systems and/or services concurrently
Clans & Chiefs model). to legacy systems and services.
Further research activities dealt with how to build security ar- Our vision is a technology that substantially weakens this
chitectures, models, and policies based on our security primitives. legacy dependence. OS research should more easily be able to
build principally new systems and services while still using exist-
ing OS services. Typically, the new system/service would extend
IA-64 In order to follow trends in hardware architecture devel- an existing OS by adding services, or modify it by replacing ser-
opment and broaden the impact area of microkernel systems, the vices, or coexist with the old OS. Microkernel architecture is not
L4 microkernel must support upcoming 64-bit architectures such a sufficient precondition but probably a necessary one for such
as Intel’s IA-64 and IBM’s Power 4. IA-64 has been chosen as the models.
main vehicle for this research, and current work has been purely
theoretical, focusing on how to provide the right abstractions for
dealing with EPIC, huge register sets, large amounts of memory, “A S YSTEM -A RCHITECTURE F OUNDATION .” In contrast to
and multi-level caches. hardware architecture and programming languages, we still do not
Work is now entering a state where more experimental studies have a general set of low-level basic concepts and paradigms in
are needed to propel research any further. Awaiting the availabil- system architecture. Most existing concepts are either only avail-


ity of real hardware, Intel’s SoftSDV environment [UFG 99] is able in theory, or they are of a too high level, or they are too much
being used to create a preliminary version of the kernel—enabling specialized. In particular, we are lacking concepts that orthogo-
accurate hardware timing measurements to be made without the nally fit together and complement each other.
underlying hardware. The bare availability of such a set of general basic paradigms
would boost systems in science and engineering — provided the
paradigms were implemented, performed nicely, would be flexible
2 Goals and powerful, and would be convenient to use.
Currently, the microkernel approach is one of very few ap-
The L4Ka project aims at substantiating and establishing a new proaches that might have the potential to lead to such a set of
methodology for system construction. Our vision is a microkernel orthogonal, general, and practically available paradigms.
 Tactical Goals “E VALUATING THE U PPER B OUNDS OF P ERFORMANCE .”
For all classes of systems and applications, performance must be
“close to infinity”. Ideally, no adequately written program should
“W IDE A PPLICABILITY.” To substantiate and establish micro- be slower when running on a microkernel than when running on
kernel technology as a general basis for system construction, we any other system architecture. Although this requirement can
must evaluate microkernels for all relevant classes of systems, ap- never be fulfilled in entirety and with full generality, we have to
plications, and hardware architectures. The goal is a technology drive performance as far as possible. We operate on the architec-
that is as generally applicable as, e.g., processors, i.e. for almost all ture level and aim at wide applicability. We must therefore under-
different types of systems and applications, from pervasive com- stand that performance of the primitives does not only influence
puting to deep computing (including classical applications and current applications but also determines the applicability range,
systems). i.e. flexibility and generality, of the architecture. Primitives like
Consequently, we need to make microkernel technology avail- IPC can in practice only be used as long as their relative costs are
able for (i) conventional workstations and PCs, (ii) pervasive sys- reasonable or, even better, negligible. Reducing costs from 200
tems including real-time systems, (iii) huge servers, (iv) and deep- to 20 cycles might then enable a whole new class of applications
computing systems including massively parallel systems and clus- using the mentioned primitive.
ter systems. From application/system perspective, the “same” mi- Therefore, finding and evaluating the upper bounds of perfor-
crokernel (cum grano salis), i.e. the same API, should be used mance is the most crucial part to make the technology widely ap-
in all cases. Having also a common code base would be benefi- plicable.
cial. However, its is still an open question to which degree a uni- Evaluating the upper bounds of performance is by far more than
form implementation approach is possible. Different architectures simply try to make a “fast” implementation. It requires perfor-
might require different implementation methods. mance analyses (a priori and a posteriori) that let us understand
Microkernels should be available for all relevant hardware ar- the reasons for the achieved performance, its limitations, and its
chitectures, including not only the basic 32-bit processor fami- dependencies from hardware and software architectural features
lies but also upcoming 64-bit processor architectures, parallel ar- and details. Sometimes, such analyses trigger inventive ideas that
chitectures, and clusters. Only such a broad availability can en- improve performance and invalidate the original analysis. Ideally,
sure that microkernels are sufficiently general and offer a single, this road has to be followed as far as possible, practically only as
hardware-independent API. Also, availability on all major hard- far as reasonable. (Concluding when it becomes unreasonable is
ware platforms is necessary to convince a larger number of exter- the hardest part.)
nal research projects to use microkernel technology as a platform Since performance typically depends on many architectural fea-
or a tool for their own research. tures and even details of hardware, performance analyses and op-
timizations are necessary for all relevant classes of hardware plat-
“O PEN BACKBONE P ROJECT.” To get broad acceptance in forms. This wide approach will most likely result in good syner-
scientific and engineering communities and to come to a suffi- getic effects.
ciently broad evaluation, we need customers for our technology X 86. Intel’s x86 family (and compatible processors from other
and we need to foster and support external experiments and re- vendors) form the de facto standard for PCs and workstations. An
search participation. L4 implementation for x86 is a precondition for microkernel re-
C LIENT P ROJECTS . We have to ensure that we deal with prac- search: Many (if not most) OS research and development projects
tically relevant problems and optimizations and do not lose our fo- use this architecture, and it is absolutely dominant for classical
cus. We must constantly evaluate L4Ka microkernels in the con- systems and applications. L4Ka’s success relies to a a large de-
text of “real” systems and applications running on top. Closely gree on the existence and on the qualities of an x86 microkernel,
related client projects such as SawMill and Drops are very effec- particularly, because many external projects/parties will first eval-
tive in this context. They are “customers” of our technology and uate the microkernel concepts on that basis. High performance
give us steady and quick feedback. More loosely related external and good stability of this implementation are essential.
research users should extend the set of client projects. ARM. Many pervasive systems are built on low-power archi-
E ASY ACCESS AND PARTICIPATION . All L4Ka technology tectures such as ARM processors. A corresponding microkernel
must be easily accessible for everyone who wants to use it (open is thus required to evaluate microkernel concepts for cell-phone
source). Also, the project must enable other research groups to OSes or other thin pervasive systems. Since the mentioned low-
participate in microkernel research. power platforms are all based on conventional 32-bit architectures
with page-based memory management chances are high that the
S TEADY S TREAMLINING AND M AINTENANCE . Continuous x86, ARM, and . . . kernels can all be specializations of the same
streamlining is required to achieve our strategic goals and to meet 32-bit microkernel.
our optimality requirements, L4Ka has to be steadily improved
and extended. This includes ongoing efforts in validating, evalu- IA-64 AND P OWER 4. Upcoming 64-bit architectures require a
ating, improving, and minimizing fundamental kernel abstractions new internal kernel design to efficiently handle new processor ar-
and concepts as well as simple maintenance. Correctness, perfor- chitectures, such as EPIC and huge register sets, to handle caches
mance, robustness, support for application and OSes have to be of 3+ levels and Gigabyte capacity, and to ensure scalability of
improved whenever possible. Maintenance complements stream- both physical and virtual memory. These architectures challenge
lining and is necessary to support client projects and to improve microkernel API and implementation methodology. Correspond-
global impact. ing microkernels are essential for evaluating the future firmness of
microkernel technology.
M ASSIVELY PARALLEL A RCHITECTURES . Clusters will
probably not require special microkernel features or implemen-
Technical Goals tations. Massively parallel NUMA machines however will need
a special kernel implementation. For example, it has to include 3 Work Plan
efficient handling of the interconnect, NUMA memory classes,
and special cross-processor IPC implementations. In collaboration Research explores the unknown. Any workplan is thus more a
with the vendor of an MPP platform, such a microkernel should be snapshot of our current thinking than a plan that is likely to cor-
developed, if possible based on an already existing microkernel for rectly predict the project future. Unexpected research results and
uniprocessors and SMPs. insights will most likely modify the plan substantially. Neverthe-
less, it is useful to structure our research activities from our current
point of view and from our current understanding. (However, no-
“M AXIMIZING F LEXIBILITY AND G ENERALITY.” Effective body should read it in 2003.)
widely applicable performance requires general concepts and To structure the project, we group research activities under 5
primitives that are extremely flexible and support wide ranges of headlines.
policies and customizations. For any system/application class, the
microkernel mechanisms should enable adequate and efficient so-
lutions. “G ENERAL C ONCEPTS AND A BSTRACTIONS ” Basically,
From that perspective, flexibility and generality are higher-level this group comprises all central and common scientific and en-
performance principles ensuring performance of policies, algo- gineering topics of L4Ka, such as general concepts, abstractions,
rithms, and applications based on the microkernel mechanisms. mechanisms, methods, etc. It drives and triggers all other subpro-
They have to be evaluated in a similar process as described for jects through the general concepts and API specifications it pro-
the performance-evaluation goal. Instead of analyzing mechanism duces.
performance depending on hardware architectures, we have to an- Key is to ensure that the concepts and APIs are orthogonal and
alyze policy performance depending on operating-system and ap- complementary to each other, that they are general and flexible,
plication architectures. and that they have an architectural performance1 potential as high
Another dimension of flexibility is ease of modification, e.g., as possible.
when extending, customizing, or configuring a system. Evaluat-
ing this dimension can only be done in collaboration with client
“P ERVASIVE K ERNEL A RCHITECTURES ” The goal is to de-
projects such as SawMill.
liver microkernel technology for pervasive systems. An impor-
C ONVENTIONAL S YSTEMS . This is an obvious point, and we tant point is to design and construct microkernels and microkernel
include it only for reasons of completeness. L Linux has shown families for most relevant processors in this field, e.g. x86, ARM,
the principle usability of the microkernel approach for classical SHx. Kernels for classical PCs and workstations — although not
operating systems. Of course, any new microkernel API and im- pervasive — are also included because pervasive kernels will most
plementation has to be checked against L Linux. likely also be best-suited for PCs and workstations. (It will not pay
to develop workstation-specialized kernels.)
P ERVASIVE S YSTEMS . Ubiquitous or pervasive computing
will be one of the key topics of the current decade. From the OS Key criteria are performance and adaptability/portability. The
perspective, pervasive systems are to a certain degree evolutions tradeoff between both criteria should be minimized. Ideally, the
from classical systems. For example, building systems highly con- highest possible performance should be combined with reasonable
figurable and based on components is crucial for pervasive sys- adaptability.
tems (and nice to have for classical systems). Based on L4 micro- Furthermore, problems specific to pervasive systems are in-
kernels, such fundamental research is, e.g., done in the SawMill cluded, e.g. power control, real-time scheduling and cache control,
project. However, pervasive systems also include new functional- SMP real-time support (bus scheduling), memory footprint, sup-
ity that does not evolve from classical systems. Pervasive systems port for nomadic applications, wireless high-performance commu-
are, e.g., real-time systems, embedded systems, low-power sys- nication, etc.
tems, nomadic systems, wireless-connected systems. These func-
tionality is partially evaluated in our client projects SawMill and “64- BIT K ERNEL A RCHITECTURES ” The focus are micro-
Drops. Further external clients are necessary. kernels for upcoming 64-bit high-performance processor architec-
H UGE S ERVERS . Huge servers impose very hard robustness tures, such as Intel’s IA-64 and IBM’s Power 4. We hope that
requirements, e.g. 7x24 availability, on the underlying hardware the microkernel API will not need to be modified. However, to
and microkernel architecture. Performance problems are basically achieve the highest possible performance, the kernel implementa-
focused on memory management, cache management, and proces- tion needs to be completely redesigned for those processors be-
sor management. So far, we do not yet have an according client cause of fundamentally new hardware features such as EPIC. Fur-
project. thermore, activities dealing with problems specific to large 64-bit
systems are included, e.g., multi-level cache control, memory-
S INGLE -A DDRESS -S PACE OS. Systems based on the para- failure handling, virtualized processors, persistence, SASOS sup-
digm of a single global address space and multiple protection do- port, high 7x24 reliability.
mains impose untypical requirements on the virtual-memory sys-
tem, e.g. efficient handling of sparse address spaces. On IA-64
processors, they could also benefit from MMU mechanisms that “PARALLEL AND C LUSTER K ERNEL A RCHITECTURES ”
especially support protection domains. It is still an open question The challenge is to make microkernel technology available for
whether this MMU mechanisms can be efficiently used without ex- deep computing, i.e. for massively parallel systems and cluster
tending the microkernel address-space paradigm and API. Client systems. NUMA architectures, fast interconnects, and low-latency
project in this context is UNSW’s Mungi. 1 We use the term architecturalperformance as opposed to implementa-
D EEP C OMPUTING . Massively-parallel machines and clusters tion performance to express the performance that the specified architecture
run very special operating systems and applications. For clusters, permits for the best possible implementation. Architectural performance
Karlsruhe’s Resh will be our client project. limits implementation performance.
cluster networks will require special kernel implementations. Per- mated time the activity needs. The figure also shows connections
haps, even conceptual API extensions might be needed. General between activities and their primary client projects. Note that is
goal is to achieve optimal performance with minimal conceptual relationship is active for the entire activity (although the connect-
kernel extensions (but probably very specialized kernel implemen- ing arrow starts from the end of the activity).
tations).
“G ENERAL C ONCEPTS AND A BSTRACTIONS ”
“E VALUATION AND A NALYSIS ” This headline comprises all
activities that evaluate and analyze the systems that result from Version 4 will be the next major revision and redesign of L4’s
our L4Ka research activities. The evaluation is mainly based on API. Version 4 will incorporate research results and lessons we
macro and micro benchmarks; the analysis aims at understanding have learned in the past years. Besides purely technical changes,
the benchmark results. Version 4 will probably include relevant conceptual modifications.
Currently, Version 4 is a moving target. We expect to have a first
experimental definition at the end of this year and the final API
definition in 2001.
 Since “Version 4” is basically the synthesis and integration of
many research results, engineering ideas, hardware-related tech-
" #! $ nical solutions, and lessons learned over the past years, and since
%     
Version 4 is currently a moving target, we will not describe it in
detail here. The interested reader can find detailed technical in-
formation in the current version of the L4 Version-4 Reference
  Manual which is attached to this application. Unfortunately, this
document focuses on specification and does not contain much rea-
  ! soning about “why” and “how to use”. Those questions will be
dealt with more deeply in the (to-be-written) System Program-
mer’s Manual.
&'
Version-5 Kernel API (Gen/A) In 2002 or 2003, lessons
learned and new insights will result in an API revision. A sen-
sitive topic for this revision may be to introduce threads as virtual
objects (see bubble activity ??. The address space, would then also
time include a task’s view of the external tasks and threads; i.e., map-
ping would be the single controlling primitive in the system and
also control IPC. Work on Version 4 will probably foster further
(*)#+-,/. 0 0
sensitive topics for Version 5.
1&2354%6
,/789.
:;6< IDL (Gen/B) The preliminary IDL performance results are
= ,?>>A@ promising (see page 2). Remaining activities are —
B
Extend the compiler to handle all data types.
B
Influence from design and spec Experimental Add IDL code generation for ARM processors.
Influence from implementation and practical use Production B
Stabilize the compiler so that it can be used as a standard
Influences from and to client project Conditional
tool.
Maintenance
B
Make the compiler Version-4 compliant.
B
Extend the IDL compiler to generate also pure IPC code.
This feature will enable to combine non-standard communi-
Figure 1: Main line activities. cation protocols, e.g. on the server side, with IDL interfaces.
B
Evaluate whether and how IDL code generation can be in-
cluded in the Flick distribution. Include IDL into Flick if
possible with reasonable effort and without loss of perfor-
Line Activities mance.
B
Line activities for a longer time and typically result in a usable Support code-generator modifications for further processor
“research product”, i.e. an implementation. Most line activities architectures and target compilers. (Current target compiler
are very likely to happen and to be completed successfully. Their is gcc.)
intended general outcome (target), e.g. a microkernel for IA-64
processors, is usually relative clear from the beginning, but spe-
cific properties and details will evolve over time. Typically, line “P ERVASIVE K ERNEL A RCHITECTURES ”
activities are heavily influenced by bubble activities and incorpo-
rate their research results. The Hazelnut kernels have been developed from scratch at
Figure 1 shows the main line activities and their interaction. Karlsruhe University over the past 10 months. Implementation
The time axis expresses “before” and “after” relationships. How- languages are C++ and assembler. Currently, prototypes are avail-
ever, the length of an activity bar is not proportional to the esti- able under GPL for x86 and ARM processors (see page 2).
 So far, the Hazelnut prototypes do not yet include the em- “64- BIT K ERNEL A RCHITECTURES ”
C
ulation of tagged TLBs. Both x86 and ARM processors use
untagged TLBs (Translation Lookaside Buffers). Therfore, any Cashew kernels aim at upcoming high-performance 64-bit pro-
cross-address-space IPC has to flush the TLB which imposes sub- cessor architectures such as IA-64 and Power4. They will evolve
stantial TLB-refill overhead on the subsequently executed code. as much as possible from Pistachio technology but will need sub-
To avoid that costs, the Pip kernels can emulate tagged TLBs for stantially different internal algorithms. So far, we have identified
a limited number of address spaces. The mechanism is described the following basic new Cashew-related problems:
in detail in [Lie95b] and [Lie95a]. This optimization is crucial Intel’s IA-64 platform is the first target for Cashew. Specific
for performance. For example, the TCP throughput of SawMill’s IPC-performance problems result from Intel’s EPIC architecture
network stack increases by 25% on a Pentium 166 MHz through with its large number of registers and its register stack engine.
this optimization. Evidently, the mentioned technique must be in- In short, the large number of registers contributes to a potentially
cluded into Hazelnut to make it competitive, even though substan- massive context (more than 2KB) to be stored on each thread con-
tial internal modifications of the kernel are required. text switch. This added context switch overhead may prove fatal to
A second point yet to be done is measuring and minimizing microkernel systems. A combined hardware/software solution is
Hazelnut’s cache working sets. Effectively, a microkernel’s cache therefore required to reduce the amount of information stored. An-
working set is even more important for performance than its exe- other solution might be to weaken the trust-relationship between
cution cycles. The goal for all performance-critical Hazelnut op- certain threads, so that a thread might be allowed to read, but not
erations are cache working sets which are roughly comparable to modify, the register contents of other threads.
those of Lime Pip, e.g. 2–3% of the L1 cache for short IPC. Other problems of the IA-64 architecture relate to providing
Pistachio is the code name for a Version-4 compliant kernel policy free abstractions of the memory management hardware, so
that is based on the Hazelnut technology. It should replace the that an OS personality may harness the special hardware mecha-
Hazelnut kernels on all processors. nisms which enables, e.g., a SASOS to be implemented efficiently.
Based on our past experiences and on the ideas that lead to
Version 4, the Pistachio kernel will internally largely differ from “PARALLEL AND C LUSTER K ERNEL
Hazelnut. New internal structures and methods include partly A RCHITECTURES ”
user-accessible kernel thread-control blocks, superfast IPC (??),
fine-granular timeouts (??), and improved real-time-scheduling Version-4 kernel specialized for massively-parallel systems and
support (??). clusters. Currently, we discuss two alternative approaches:
B
a single microkernel that runs as one distributed kernel on all
Orange Pip Kernel (Per/A) Orange Pip is the code name for nodes, or
a Version-4 kernel for x86 processors. The kernel will be based B
on the existing Pip technology, i.e. developed processor specific, a microkernel per node, complemented by a set of basic sys-
mainly in assembler. tem servers.
The experimental Orange Pip kernel will serve as a vehicle So far, we favor the second approach. The basic servers then man-
for experimental implementation of new version-4 concepts and age inter-node communication, inter-node scheduling and load
mechanisms. The activity is scheduled to run in parallel of the balancing, cross-node address spaces, and cross-node memory ac-
corresponding Pistachio/x86 implementation for better and easier cess. Furthermore, they comprise special low-latency drivers for
exploration of new mechanisms and concepts. controlling the interconnect and/or specialized multi-gigabit inter-
Although it seems to be surprising at a first glance, such low- node networks.
level architectural experiments are sometimes easier in an environ- We aim at building Grapestone kernels based on their corre-
ment that is not restricted by a compiler, its coding conventions, sponding Cashew kernels. Ideally, Grapestone/IA-64, e.g., should
and code-generator properties. Architecturally relevant items such consist of the Cashew/IA-64 kernel complemented by a Grape-
as cache-line usage, instruction parallelism, kernel-stack reduc- stone package. Once the first Grapestone kernel has been con-
tion, and controlling special hardware features strongly influence structed successfully this way, we should be able to build Grape-
the achievable performance. Ease of modification is crucial in stone kernels at relatively low costs for any Cashew hardware plat-
this context to find the best performing methods. Correspond- form.
ing experimental implementations and evaluations often require We hope that we can basically use the same Grapestone kernel
less effort in an unrestricted assembler environment. Furthermore, for cluster systems as well as for MPPs. Only some drivers, in-
avoiding to be biased by unknown or unwanted compiler/code- terconnect, etc., will be different. However, this hope is not yet
generator influences/optimizations is easier. substantiated.
For the described exploration method, experimental and some-
times only partial implementations are sufficient. Once the opti-
mal solution is identified, a complete and stable solution can be References
implemented in Pistachio. The experimental assembler solution
serves as a guideline and defines the performance goal. [Bel97] F. Bellosa. Process cruise control: Throttling mem-
ory access in a soft real-time environment. Techni-
cal report, Dept. of Comp. Sci., University of Erlangen-
Fully-functional Orange Pip (Per/A.1) A fully functional and Nürnberg, 1997. Available: http://www4.informatik.uni-
stable Orange Pip kernel will only be developed if the Hazelnut erlangen.de/ELiTE/pub.html.


technology turns out not to perform sufficiently well or if fixing [DdBF 94] A. Dearle, R. di Bona, J. Farrow, F. Henskens, A. Lindström,
the Hazelnut problems would take too much time and thus delay J. Rosenberg, and F. Vaughan. Grasshopper: an orthogonally
the availability of a Version-4 kernel. We hope that a fully func- persistent operating system. Computing Systems, 7(3):289–
tional Orange Pip kernel will not be needed. 312, Summer 1994.
[Dre] TU Dresden. Drops. http://os.inf.tu-dresden.de/drops.
D
[EE98] G. Eddon and H. Eddon. Inside Distributed COM. Microsoft
Press, 1998.


[EFF 97] Eric Eide, Kevin Frei, Bryan Ford, Jay Lepreau, and Gary
Lindstorm. Flick: A flexible, optimizing idl compiler. Pro-
ceedings of the ACM SIGPLAN ’97 Conference on Program-
ming Language Design and Implementation (PLDI), pages
44–56, June 1997.


[HLP 00] A. Haeberlen, J. Liedtke, Y. Park, V. Uhlig, and L. Reuther.
Stub code performance is becoming important. In First
Workshop on Industrial Experiences with Systems Software
(WIESS), San Diego, CA, October 2000. To appear.
[Hoh] M. Hohmuth. Fiasco. http: //os.inf.tu-dresden.de /fiasco
/doc.html.


[JEL 99] T. Jaeger, K. Elphinstone, J. Liedtke, V. Panteleenko, and
Y. Park. Flexible access control using ipc redirection. In
Hot Topics in Operating Systems (HotOS VII), Rio Rico, AZ,
March 1999.
[Lan92] C. R. Landau. The checkpoint mechanism in KeyKOS. In
Proceedings of the 2nd International Workshop on Persistent
Object Systmes (POS2), pages 24–25, Paris, France, Septem-
ber 1992.
[LHH97] J. Liedtke, H. Härtig, and M. Hohmuth. OS-controlled cache
predictability for real-time systems (RTAS). In 3rd IEEE
Real-time Technology and Applications Symposium, pages
213–223, Montreal, May 1997.
[Lie92] J. Liedtke. Clans & chiefs. In 12. GI/ITG-Fachtagung Ar-
chitektur von Rechensystemen, pages 294–305, Kiel, March
1992. Springer.
[Lie93] J. Liedtke. A persistent system in real use – experiences
of the first 13 years –. In 3 EF International Workshop on
Object Orientation in Operating Systems (IWOOOS), pages
2–11, Asheville, NC, December 1993.
[Lie95a] J. Liedtke. Improved address-space switching on Pen-
tium processors by transparently multiplexing user address
spaces. Arbeitspapiere der GMD No. 933, GMD — Ger-
man National Research Center for Information Technology,
Sankt Augustin, September 1995.
[Lie95b] J. Liedtke. On  -kernel construction. In 15GH ACM Sym-
posium on Operating System Principles (SOSP), pages 237–
250, Copper Mountain Resort, CO, December 1995.
[LVE00] J. Liedtke, M. Völp, and K. Elphinstone. Preliminary
thoughts on memory-bus scheduling. In 9 GH SIGOPS
European Workshop, pages 207–210, Kolding, Denmark,
September 2000.
[Obj] The Object Management Group (OMG). The Complete
CORBA Services Book. http: //www.omg.org /library
/csindx.html.
[SL00] E. Skoglund and J. Liedtke. Transparent orthogonal check-
pointing through user-level pagers. In I GH International
Workshop on Persistent Object systems (POS9), Lilleham-
mer, Norway, September 2000. Springer LNCS. To appear.
[SSF99] J. S. Shapiro, J. M. Smith, and D. J. Farber. EROS: a fast
capability system. In 17th ACM Symposium on Operating
System Pronciples (SOSP), pages 12–15, Kiawah Island Re-
sort, SC, December 1999.
[Syd] UNSW Sydney. Mungi. http://www.cse.unsw.edu.au/ J disy
/Mungi/.


[UFG 99] R. Uhlig, R. Fishtein, O. Gershon, I. Hirsh, and H. Wang.
SoftSDV: A presilicon software development environment
for the IA-64 architecture. Intel Technology Journal, Q4,
1999.