196 IEEE TRANSACTIONS ON COMPUTERS, VOL. 72, NO.

1, JANUARY 2023

Operating System Noise in the Linux Kernel

Daniel Bristot de Oliveira , Daniel Casini , Member, IEEE, and Tommaso Cucinotta , Member, IEEE

Abstract—As modern network infrastructure moves from hardware-based to software-based using Network Function Virtualization,
a new set of requirements is raised for operating system developers. By using the real-time kernel options and advanced CPU isolation
features common to the HPC use-cases, Linux is becoming a central building block for this new architecture that aims to enable a new
set of low latency networked services. Tuning Linux for these applications is not an easy task, as it requires a deep understanding of the
Linux execution model and the mix of user-space tooling and tracing features. This paper discusses the internal aspects of Linux that
influence the Operating System Noise from a timing perspective. It also presents Linux’s osnoise tracer, an in-kernel tracer that
enables the measurement of the Operating System Noise as observed by a workload, and the tracing of the sources of the noise, in
an integrated manner, facilitating the analysis and debugging of the system. Finally, this paper presents a series of experiments
demonstrating both Linux’s ability to deliver low OS noise (in the single-digit ms order), and the ability of the proposed tool to provide
precise information about root-cause of timing-related OS noise problems.

Index Terms—Linux kernel, operating system noise, high-performance computing, soft real-time systems

1 INTRODUCTION The 5 G network stack is built upon this paradigm, and it

is enabling a new set of services characterized by strict tim-
Linux Operating System (OS) has proved to be a via-
T HE
ble option for a wide range of very niched applications,
despite its general-purpose nature. For example, Linux can
ing requirements [6]. These were generally satisfied using
physical appliances in traditional networks. However, in
the new 5 G stack, these requirements need to be achieved
be found in the High-Performance Computing (HPC)
by software-based appliances, requiring the support of a
domain, running on all the top 500 supercomputers.1 It can
real-time operating system. For example, in Virtualized
also be found in the embedded real-time systems domain,
Radio Access Network (vRAN), latencies are in the order of
not only in the area of industrial automation and robot con-
tens of microseconds [4], [7]. Such a need made time and
trol but even reaching out the space [1]. These achievements
processing latency one of the main metrics for vendors in
are possible thanks to the great flexibility in the configura-
this market [8], [9], [10].
tion options of Linux, and specifically its kernel.
To meet these tight timing requirements, both hard-
Another remarkable domain where Linux plays a central
ware and Linux are configured according to standard
role is the one of developing core services supporting modern
practices from both the HPC and the real-time domains.
networking infrastructures and the Internet. With Network
To this end, the hardware is configured to achieve the
Function Virtualization (NFV) [2] and Software-Defined Net-
best trade-off between performance and determinism.
working (SDN) [3], this domain is shifting from the traditional
This setup includes adjusting the processor speed and
paradigm of hardware appliances sized for the peak-hour, to
power savings setup while disabling features that could
the new one of flexible software-based and programmable
cause hardware-induced latencies, such as system man-
networking services with horizontal elasticity abilities to
agement interrupts (SMIs).
adapt dynamically to the workload conditions. These new
Regarding the Linux configuration, the system is usually
architectures often rely on general-purpose hardware [4] and
partitioned into a set of isolated and housekeeping CPUs, which
software stacks based on Linux [5].
is a typical setup for HPC systems. The housekeeping CPUs
are those where the tasks necessary for the regular system
usage will run. This includes kernel threads responsible for
1. By Nov. 2021, according to https://www.top500.org/.
in-kernel mechanisms, such as RCU (read-copy-update) call-
back threads [11], kernel threads that perform deferred work

Daniel Bristot de Oliveira is with Real-time Scheduling Team, Red Hat, such as kworkers and threads dispatched by daemons and
Inc., 56025 Pontedera, PI, Italy, and also with TeCIP Institute, Scuola
Superiore Sant’Anna, 56127 Pisa, Italy. E-mail: [email protected]. users. General system’s IRQs (Interrupt ReQuests) are also

Daniel Casini is with the TeCIP Institute, Department of Excellence in routed to housekeeping CPUs. This way, the isolated CPUs are
Robotics & AI, Scuola Superiore Sant’Anna, 56127 Pisa, Italy. E-mail: d. then dedicated to the NFV work. However, despite the high-
[email protected].

Tommaso Cucinotta is with TeCIP Institute, Scuola Superiore Sant’Anna, grade CPU isolation level currently available on Linux, some
56127 Pisa, Italy. E-mail: [email protected]. housekeeping work is still necessary on all CPUs. For exam-
Manuscript received 29 January 2022; revised 24 May 2022; accepted 26 June ple, the timer IRQ still needs to happen under certain condi-
2022. Date of publication 30 June 2022; date of current version 13 December tions, and some kernel activities need to dispatch a kworker
2022. for all online CPUs. Drawing from real-time setups, NFV
(Corresponding author: Daniel Bristot de Oliveira.)
Recommended for acceptance by Daniel Mosse, Tam Chantem, Enrico Bini. threads are often configured with real-time priorities, and
Digital Object Identifier no. 10.1109/TC.2022.3187351 the kernel is generally configured with the fully preemptive
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
DE OLIVEIRA ET AL.: OPERATING SYSTEM NOISE IN THE LINUX KERNEL 197

mode (using the PREEMPT_RT patch-set [12]) to provide 2 BACKGROUND

bounded wakeup latencies.
We start presenting the needed background. First, Sec-
In order to debug and evaluate the system setup, Linux
tion 2.1 presents the Linux execution contexts and their rela-
practitioners use syntectic workloads that mimic the behav-
tion. Then, Section 2.2 summarizes the Linux schedulers
ior of these complex scenarios. NFV applications run both
hierarchy and the most commonly used tracers.
triggered by an interrupt or by polling the network device
while waiting for packets, running non-stop. While the
Linux wakeup latency has been extensively studied from 2.1 Linux Execution Contexts and Their Relation
the real-time perspective [13], [14], this is not the case for In Linux, there are four main execution contexts: non-mask-
the interference suffered by threads. This subject however able interrupts (NMIs), maskable interrupts (IRQs), softirqs
was extensively covered by another community: the HPC (deferred IRQ activities), ( note that in the PREEMPT_RT,
one, in a metric named OS noise [15], [16]. In this paper, we the softirq context is moved from its own execution context
focus on the practicalities of OS noise measurement and to run as a regular thread), and threads [12]. When there is
analysis on Linux, from a real-time viewpoint. no explicit reason to distinguish among them, we hereafter
Why Yet Another Tool? Several tools have been proposed refer to all of them as tasks. Interrupts are managed by the
over the years to measure OS noise, and they can be classi- interrupt controller, which queues and dispatches multiple
fied into two categories: workload and trace-based methods. IRQs and one NMI for each CPU. The NMI handler is the
Both of these have advantages and disadvantages, exten- highest-priority activity on each CPU, it is non-maskable,
sively discussed later in the paper. In summary, workload and hence it is capable of preempting IRQs and threads.
methods simulate a workload, being capable of accounting IRQs, in turn, are able to preempt threads and softirqs,
for the OS noise measurement as a metric reported by the unless they have been temporarily disabled within critical
workload. For instance, by detecting a large amount of time sections of the kernel. Ssoftirq is a software abstraction, and
elapsed between two consecutive reads of the time or by the in the standard kernel configuration, runs after IRQ execu-
number of finished operations. The limitation of workload- tion, preempting threads. Finally, threads are the task
based tools is that they do not provide any insight into the abstraction managed by the Linux schedulers.
root cause of the noise. Conversely, trace-based methods Linux’s execution contexts are characterized by the fol-
show potential causes of latency spikes, but they cannot lowing rules:
account for how the workload perceives the noise.
R1 The per-CPU NMI preempts IRQs, softirqs and
Differently from previous work, we cover both worlds by
threads;
designing and implementing a comprehensive kernel tracer
R2 The per-CPU NMI, once started, runs to completion.
to deal with the OS noise on Linux, called osnoise. It uses
R3 IRQs can preempt softirqs and threads.
a hybrid approach, leveraging both the workload and a trac-
R4 Once an IRQ is started, it is not preempted by
ing mechanism synchronized together to account for the
another IRQ.
operating system noise while still providing detailed infor-
R5 Softirqs can preempt threads.
mation on the root causes of OS noise spikes, and also
R6 Once a softirq is started, it is not preempted by any
reducing the tracing overhead using in-kernel tracing fea-
other softirq.
tures. While the tool was developed with extreme isolation
R7 Threads cannot preempt the NMI, IRQs and sofirqs.
cases in mind, targeting the detection of single-digit ms
The rule-set is derived from the automata-based Thread
noise occurrences, it is not limited to this use case. Indeed, it
Synchronization Model of Linux [17], which showed to
can be applied to any HPC system setup.
model the Linux synchronization behavior faithfully, and
The osnoise tracer is officially part of the Linux kernel
by industrial expertise.
since version 5.14, passing by the thorough kernel revision
process, including experts in real-time, scheduling, and
tracing, evidencing the agreement on the abstractions and 2.2 Linux Schedulers and Tracing
technologies used by osnoise. Since Linux kernel version Next, we proceed introducing the Linux’s scheduler and
5.17, the tracer can be used as an user-space tool available tracing mechanisms.
via the rtla (Real-Time Linux Analysis) toolset, becoming Schedulers. Linux has a hierarchy of five schedulers, which
easily accessible both by practitioners to test their systems handle all the threads irrespectively of their memory contexts
and developers to extend it. (e.g., kernel threads, process context in user-space). The five
Paper Contributions. The contributions of this paper are schedulers are queried in a fixed order to determine the next
three-fold: (I) propose a precise definition of the causes of thread to run. The first one is thestop-machine, a pseudo sched-
OS noise in Linux, from the real-time perspective; (II) pres- uler used to execute kernel facilities. The second one
ent a kernel tracer that is able to measure the OS noise using isSCHED_DEADLINE [18], a deadline-based real-time sched-
the workload approach, while also providing tracing infor- uler based on Earliest Deadline First (EDF). The third one is a
mation essential to pinpoint the tasks suffering of OS noise, POSIX-compliant fixed-priority real-time scheduler. A thread
not only caused by the OS, but also from the hardware or using this scheduler can be either aSCHED_RR or aSCHED_-
the virtualization layer; (III) report on empirical measure- FIFO thread. The difference between the two is only for
ments of the OS noise from different configurations of threads at the same priority: in this case,SCHED_RR threads
Linux, commonly found in NFV setups, showing how the are scheduled in a round-robin fashion with a given time
tool can be used to find the root causes of high latency slice, whileSCHED_FIFO threads release the CPU only on sus-
spikes, thus enabling finer-grained tuning of the system. pension, termination or preemption. The fifth scheduler, is
198 IEEE TRANSACTIONS ON COMPUTERS, VOL. 72, NO. 1, JANUARY 2023

set of CPUs isolated from most of the operating noise that

users or the OS could cause.
In NFV, to achieve high throughput, the generic network
stack of the operating system is often bypassed, with all the
network packet processing done in user-space by a specific
process that handles the network flows. Similar to the HPC
case, these processes receive dedicated resources, including
dedicated isolated CPUs.
To reduce even more the latency for handling new pack-
ets, some of these network applications poll the network in
a busy-wait fashion, most notably using the DPDK’s Poll
Mode Driver (PMD).2 In this use case, the Linux setup fol-
lows the same script as HPC. The difference between this use-
case and the HPC one is the real-time constraints, for example in
the order of tens of microseconds for vRAN.
Although some turn-key options to provide CPU isola-
Fig. 1. The single-program multiple-data (SPMD) model used for HPC tion are available, such as moving all threads and IRQs to a
workloads, and the effects of the OS noise (adapted from [19]). reduced set of house-keeping CPUs, the fine-tuning of the
configuration for these time-sensitive use-cases is not a sim-
the general-purpose scheduler, the completely fair scheduler ple task. The reason is that the OS still requires some per-
(CFS), also calledSCHED_OTHER. Finally, when no ready CPU actions, such as scheduler tick, virtual memory stat
threads are available from these schedulers, theIDLE scheduler operations, legacy network packet processing, etc. While
returns the idle thread. some of these noise sources can be mitigated via fine-tuning
Tracers. Linux has a rich set of tracing features. For of the configuration, like enabling NOHZ_FULL that reduces
instance, it is possible to trace specific events such as sched- the scheduler tick frequency, others might even require
uling decisions and many functions called in kernel context. reworking the currently existing kernel algorithms to either
These features are ready to use in most Linux distributions, remove the cause of noise or add a method to mitigate the
mainly because they do not add overhead to the system if issue.
they are not in use. Ftrace is an in-kernel set of tracing fea- Despite of the appealing use-case, Linux is a general-pur-
tures designed to aid practitioners in observing in-kernel pose operating system, with the main focus on developing
operations. general-purpose features. Specialized communities, such as
the real-time and HPC ones, need to constantly monitor the
evolution of the OS to adapt possible non-HPC and non-RT
3 MOTIVATIONS AND PROPOSED APPROACH aware functionality for these specific use-cases. It is imprac-
Next, we start introducing the problem and the motivations tical to force Linux developers to test their new algorithms
for this work. to all the specific use-cases’ metrics when there is no simple
The OS noise, sometimes named OS jitter, is a well- way to observe and debug them.
known problem for the HPC field [15], [16]. To measure the OS noise, a practitioner generally starts
Generally, HPC workloads follow the single-program by spawning a synthetic workload. An example of work-
multiple-data (SPMD) model, shown in Fig. 1. In this model, load is sysjitter, and its clone oslat. These tools loop
a system is composed of M processors, and a parallel job reading the time using architecture-specific instructions.
consists of one process per processor [19]. All processes are They define a jitter when two consecutive readings of the
dispatched simultaneously at the beginning of the execu- time have a gap larger than a given threshold. These tools do
tion. At the end of the execution, the process synchronizes not attempt to correlate a jitter to a root cause.
to compose the final work, and repeat cyclically. To find a root cause, practitioners need to observe the sys-
Ideally, the parallel job process should be the only work- tem. The most efficient way to observe the system is using
load assigned to the processor. However, some operating tracing. The challenge of using tracing is in the tradeoff
system-specific jobs need to run on all processors for the between information and overhead. With a workload and
correct operation of the system, like the periodic scheduler tracing features in place, the user must identify a correlation
tick, critical kernel threads or others. In this scenario, the between the noise and tracing information. This correlation
scheduler decisions of each local processor significantly is not always possible, mainly because the workload and the
impact in response time of a parallel job. These delays tracing features are unaware of each other.
caused to a parallel workload by OS activities running on It is essential to notice that at the dozens of microseconds
the same processor(s) is named Operating System Noise. figure, hardware-induced noise is also noticeable by the
One of the main reasons that led Linux to dominate the workload. Hardware noise can be a side effect of hardware
top 500 super-computers list is the flexibility of the system stalls caused by shared resources, as happening in hyper-
configuration. These systems’ setup involve selecting a threading enabled processors or execution contexts with a
small set of CPUs to be in charge of all tasks necessary for higher priority than the OS, like SMIs. Because these actions
the system execution and operation, such as running system
daemons, periodic maintenance tasks, managing user access
to the system for monitoring activities, etc., leaving a large 2. https://doc.dpdk.org/guides/prog_guide/poll_mode_drv.html
DE OLIVEIRA ET AL.: OPERATING SYSTEM NOISE IN THE LINUX KERNEL 199

are not a side effect of the operating system, it is impossible (for example, a constrained-deadline sporadic task [20]) will
to observe the events via trace. This noise observed by the complete within the deadline, under a partitioned schedul-
workload but not by the trace creates a grey area that often ing setting where workloads are considered for each proces-
misleads the analysis. sor separately. To this end, the classical worst-case response
time equation can be leveraged
3.1 Proposed Approach X
In this paper, we propose an integrated tracing and syn- Ri ¼ ei þ hh ðRi Þ  eh ; (1)
thetic workload solution that aims to join workload- and th 2hpi

tracing-based approaches benefits while minimizing the where eh is the worst-case execution time (WCET) of t h ,
drawbacks of each solution. hh ðDÞ is its arrival curve [21] bounding the maximum num-
The steps taken for such an approach include: ber of release events of t h in a time window3 of length D,

Define the composition of the OS Noise on Linux and the set hpi contains the higher-priority activities that
from the real-time HPC point of view; can interfere with the thread t i under analysis.

Define the minimum set of tracing events to provide While using Equation (1) at design time is in principle
evidence of the root cause of each noise, at a limited possible, it is quite often hard. Indeed, in modern heteroge-
overhead; neous computing platforms, many design principles used

Create a synthetic workload aware of tracing, to increase average-case performance (e.g., complex cache
enabling an unambiguous correlation of the trace hierarchies [22], un-revealed memory controller poli-
and the noise; cies [23], out-of-order execution, etc.) are making it hard to

Make the approach production-ready, with a stan- obtain reliable WCET estimates for user threads. This is
dard and easy-to-use interface. even harder for OS threads and interrupt service routines,
for which also the arrival pattern is unknown, and therefore
3.2 OS Noise Composition for Real-Time HPC it is difficult to obtain an arrival curve.
Workload Adopting such computing platforms in a small subset of
This paper adopts the following generalized definition of strictly hard real-time systems, e.g., avionics, calls for com-
OS noise: prehensive solutions allowing to know all the parameters
involved in Equation (1), e.g., by leveraging static analysis
Definition 1 (Generalized (OS)-Noise). The OS noise is tools for WCET estimation [24].
defined as all the time spent by a CPU executing instructions However, most real-time systems are robust enough to
not belonging to a given application task assigned to that CPU tolerate small uncertainties in the estimation of the parame-
while the task is ready to run. ters, and they can tolerate a small amount of deadline
misses [25] (e.g., in multimedia [26]).
The definition generalizes the usual interpretation of OS
In these cases, osnoise can be used to empirically mea-
noise, which typically only includes OS-related activities
sure the high-priority interference in Equation (1). For
and overheads, by accounting also for the time used by any
example, to estimate the high-priority interference faced by
interfering computational activity, not limited to the OS but
a NFV workload running at a given priority under
also from regular user-space threads. This makes a differ-
SCHED_FIFO (a common use-case), the system engineer
ence when multiple user threads can run in the same CPU,
can setup osnoise to run under SCHED_FIFO at the same
as any computational activity that can interfere with the
priority, thus exposing the measurement thread to the same
measurement thread would also interfere with any user
sources of noise.
thread running with the same scheduler and scheduler set-
tings (e.g., priority), irrespectively to whether it belongs to
the OS or not. Therefore, it would constitute an actual 4 RELATED WORK
source of noise which the fine-grained tuning of the system The adverse effects to workload performance due to the
needs to account for. operating system noise have been known for a long
This extended definition gives room for an interesting time [27]. One of the first works addressing the problem of
link between the OS noise, a metric from the HPC domain, detecting the OS noise is due to Petrini et al. [15], which
and the high-priority interference commonly considered in identified and eliminated some source of noise for an HPC
real-time systems theory. application running on the ASCI Q supercomputer. The
This generalizes the approach beyond the HPC and NFV study has been extended in a later paper [28]. Ferreira et al.
use cases, allowing to practically profile all the sources of [29] provided a characterization of application sensitivity to
interference that can affect a task running with a given con- the noise by injecting interference at the OS level.
figuration of the scheduler: for example, a thread running at As discussed at the beginning of the paper, Linux tools
a given priority under the fixed-priority scheduler of Linux. for detecting OS noise are divided into two categories:
Indeed, thanks to this generalized definition, the workload and trace-based methods.
osnoise tracer can be used not only to monitor the noise Some workload-based methods run micro-benchmarks
strictly-related to the operating system but all high-priority with a known duration, and they measure the difference
interference in a broader sense. between the expected duration of the microbenchmark and
Generalized (OS)-Noise Under Fixed-Priority Scheduling. As
a highly-relevant example, we consider the case in which a 3. e.g., if th is a sporadic thread with minimum inter-arrival time Ti ,
designer needs to determine whether a thread of interest t i it holds hh ðDÞ ¼ dD=Th e).
200 IEEE TRANSACTIONS ON COMPUTERS, VOL. 72, NO. 1, JANUARY 2023

Fig. 2. osnoise summary output from ftrace interface.4

the actual time needed to process it. For this category, one required by each interfering activity from its measured run-
relevant example is due to Sottile and Minnich [30], who time. The tool is not limited to a specific preemption model
proposed the Finite-Time Quantum (FTQ) benchmark. FTQ of Linux, and it can work with any of its preemption mod-
measured the number of basic operations done within a els, from the non-preemptive kernel to PREEMPT_RT.
fixed time quantum. Another work from Tsafrir et al. [31] Before discussing the internals, we present the tool at a
used microbenchmarks in conjunction with a mechanism high level. As mentioned, osnoise has two components:
based on “smart timers” to measure OS noise. the workload and the tracing components.
A widely used by practitioners workload-based tool is
sysjitter [32]. It measures OS noise by running a thread
on each CPU and keeping track of the duration of each time 5.1 The osnoise Workload Threads
interval in which the thread is not running, e.g., due to OS The osnoise workload threads used for measurements
activities. Other similar tools are oslat [33], jHicckup [34], work on a per-CPU basis.
and MicroJitterSampler [35]. However, all these tools are By default, osnoise creates a periodic kernel thread on each
not capable of pointing to the root causes of the OS noise. CPU. The kernel thread can be assigned to any Linux sched-
This problem can be solved with other tools, which use a uler, such as SCHED_DEADLINE, SCHED_FIFO, SCHED_RR, or
trace-based approach. For example, De et al. [36] presented CFS.
a tool to identify the sources of the OS jitter using kernel Each thread runs for a pre-determined amount of run-
instrumentation by leveraging OProfile. The authors time. The primary purpose of the workload thread is to
obtained statistics on preemptions and interruptions. Later, detect the time stolen from its execution, which is consid-
Morari et al. [16] proposed a different tool to measure the ered OS noise. Each osnoise thread works by reading the
OS noise but using a similar (but more extended) technique time in a loop. When it detects a gap between two consecu-
based on kernel instrumentation and building on the tive readings higher than a given tolerance threshold, a new
LTTng [37] tracer. With respect to [36], the work by Morari noise sample is collected. The time is read using the trace_-
et al. allows for capturing additional causes of the OS noise, local_clock() function. This architecture-specific non-
e.g., softirqs. Nataraj et al. [38] proposed another approach blocking function provides a lightweight CPU-level coher-
to instrument the Linux kernel and measure the OS noise ent timestamp, at the nanoseconds granularity, at the same
during application execution by using KTAU [39]. More accuracy used by other ftrace tracing mechanisms.
recent work is due to Gonzalez et al. [40], which presents Jit- The thread runs with preemption and IRQs enabled. This
ter-Trace, a tool that uses the information provided by the way, it can be preempted at any time by any task abstraction
perf tracer (see Section 5.3). However, all these trace-based present in Linux.
methods do not account for how workloads perceive the After runtime microseconds are elapsed since the first
noise. time read of the current period, the workload reports a sum-
Different from previous work, the osnoise tool pro- mary of the OS noise faced by the current activation. This
posed in this paper takes the best of both workload-based summary is reported using tracing features of Linux, as in
and trace-based methods, allowing to point to the root Fig. 2.
causes of the operating system noise while also accounting The osnoise summary reports:
for how the workload perceives the noise.
RUNTIME IN US, i.e., the amount of time in ms in
And finally, but most relevantly, osnoise is the only which osnoise looped reading the timestamp.
tool for directly monitoring the OS noise that has been just
NOISE IN US, i.e., the overall amount of noise in ms
recently made available in the mainline Linux kernel [41], observed in the associated runtime.
and hence it is ready-to-use on billions of devices.
PERCENTAGE OF CPU AVAILABLE, i.e., the percent-
age of CPU available to the osnoise thread in the
measuring period.
5 THE OSNOISE TRACER
MAX SINGLE NOISE IN US, i.e., the longest observed
This section presents the osnoise tracer, which leverages occurrence of noise in ms during the runtime.
the rules presented in Section 2.1 to correctly profile the exe-
The interference counters: for each type of interference
cution time of each task by correctly subtracting the time among the classes NMI, IRQs, softirqs, and threads,
DE OLIVEIRA ET AL.: OPERATING SYSTEM NOISE IN THE LINUX KERNEL 201

Fig. 3. Example of tracepoints: IRQ and thread context switch events read from ftrace interface4.

osnoise maintains an interference counter that is possible to pre-process data in the tracepoints in such a way
increased in correspondence of an entry event of as to minimize the amount of data written to the trace
activity of that type. buffer. This method has shown good results, reducing the
It is worth noting that Fig. 2 shows a high number of tracing overhead when the trace processing presents lower
hardware noise samples: this is because osnoise was run- overhead than writing trace to the buffer [42].
ning on a virtual machine, and the interference due to virtu- The osnoise tracer leverages the current tracing infra-
alization is detected as hardware noise. structure in two ways. It adds probes to existing tracepoints
to collect information and adds a new set of tracepoints
5.2 The osnoise Parameters with pre-processed information.
The osnoise tracer has a set of parameters. These options Linux already has tracepoints that intercept the entry and
are accessible via ftrace’s interface, and they are: exit of IRQs, softirqs, and threads. osnoise attaches a
probe to all entry and exit events and uses it to: 1) account

osnoise/cpus: CPUs on which a osnoise thread for the number of times each of these classes of tasks added
s will execute. noise to the workload; 2) to compute the value of the inter-

osnoise/period_us: the period ( in ms) of the ference counter used by the workload to identify how many
osnoise thread s. interferences occurred between two consecutive reads of

osnoise/runtime_us: how long ( in ms) an the time;5 3) to compute the execution time of the current
osnoise thread s will look for noise occurrences. interfering task; 4) to subtract the noise occurrence duration

osnoise/stop_tracing_us: stop the system of a preempted noise occurrence by leveraging the rules dis-
tracing if a single noise occurrence higher than the cussed in Section 2.1.
configured value in ms happens. Writing 0 disables At the exit probe of each of these interference sources, a
this option. single tracepoint from osnoise is generated, reporting the

osnoise/stop_tracing_total_us: stop the noise-free execution time of the task’s noise observed via trace.
system tracing if total noise occurrence higher than In addition to the tracepoints and the summary at the
the configured value in ms happens. Writing 0 dis- end of the period, the osnoise workload emits a tracepoint
ables this option. anytime a noise is identified. This tracepoint informs about

tracing_threshold: the minimum delta between the noise observed via workload, and the amount of interfer-
two time reads to be considered as a noise occur- ence that happened between the two consecutive time
rence, in ms. When set to 0, the default value will will reads. The interference counters are fundamental to unam-
be used, which is currently five ms. biguously defining the root cause for a given noise.
For example, in Fig. 4, the first four lines represent the
noise as identified by the trace, while the last line is the
5.3 The osnoise Tracing Features tracepoint generated by the workload, mentioning the pre-
The tracepoints are one of the key pillars of the Linux kernel vious four interferences.
tracing. The tracepoints are points in the kernel code where Both Figs. 3 and 4 were extracted from the same trace file.
it is possible to attach a probe to run a function. They are The difference is that the former contains the previous exist-
most commonly used to collect trace information. For exam- ing tracepoints, while the latter includes the new tracepoints
ple, ftrace register a callback function to the tracepoints. added to the kernel with osnoise. With these two exam-
These callback functions collect the data, saving it to a trace ples, it is possible to notice that the amount of information
buffer. The data in the trace buffer can then be accessed by a reported by the osnoise tracepoints is reduced and more
tracing interface. Fig. 3 shows an example of tracepoint out- intuitive.
put via ftrace interface. Regarding the noise reported in Fig. 4, it is important to
The usage of tracepoints is not limited to saving data to notice that the duration reported by the irq_noise and
the buffer. They have been leveraged for many other use- thread_noise are free of interference. For example, the
cases. For instance, patch the kernel at runtime or transform local_timer:236 has a start time later than the sleep-5843.
network packets [42]. Tracepoints can also be used to opti- This means that local_timer:236 preempted sleep-5843,
mize tracing itself. While saving data to the trace buffers in a case of nested noise. The local_timer:236, however,
has been optimized to the minimum overhead, it is also
5. The single per-CPU NMI is a particular case without tracepoints;
4 All ftrace excerpts share the same column description, as in the in this case, a special function was added to collect the same
header in Fig. 2. information.
202 IEEE TRANSACTIONS ON COMPUTERS, VOL. 72, NO. 1, JANUARY 2023

Fig. 4. Example of tracepoints: osnoise events read from ftrace interface with equivalent data highlighted4.

discounted its own duration from the duration of sleep-5843 The rtla Interface. Since version 5.17, Linux includes the real-
This facilitates the debugging of the system by removing the fas- time Linux analysis tool, named rtla. This is a meta-tool that
tidious work of computing these values manually or via a script aims at analyzing the real-time properties of Linux by exploit-
in user-space. This also reduces the amount of data saved in the ing its tracing features to provide information on the causes
trace buffer, reducing resource usage and overhead. of measurements. rtla includes an user-space tool named
Another important thing to notice is that the total noise rtla osnoise [44], transforming the tracer into a bench-
observed via trace accounts for 1409532 ns,6 but the noise mark-like tool. This tool works by configuring, dispatching
obseved via workload reports 5092 ns more (1414624 ns), as and collecting data from the osnoise tracer via ftrace interface.
illustrated in Fig. 5. The reasons behind are multiple. For The rtla osnoise can either collect the periodic workload
example, the overhead added by the tracepoints enabled in summary, creating a long run summary, or create a histogram
Figs. 3 and 4; the delays added by the hardware to manage with data from the sample_threshold tracepoint.
context switch and the dispatch of IRQs handlers; delays
caused by cache inlocality after an interrupt [43]; low level 5.5 The osnoise Internals
the code that enables the tracing at IRQ context, like making
The osnoise tracer aims to measure possible sources of
the RCU aware of the current context;7 and the scheduler
noise at the single-digit ms scale, and this represents a chal-
call caused by the thread noise.
lenge when dealing with parallel and re-entrant code as in
This justifies the dual approach and motivates the nov-
the Linux kernel. This section presents some of these chal-
elty to prior work that used only one of the main distin-
lenges and how they have been tackled.
guishing factors with respect to prior work (as extensively
discussed in Section 4): using both the measuring thread
and the tracing. Indeed, the trace cannot be used as the only 5.5.1 Task and Memory Model
source of information because it cannot account for the osnoise aims to simulate applications built using the
overheads occurring outside the scope of the tracing. Simi- SPMD model, presented in Section 2. When dispatched, the
larly, the measurement thread alone cannot capture the rea- tracer creates a per-CPU kernel thread to run the osnoise
sons for the OS noise, and hence it does not provide workload component. Each per-CPU thread has its affinity
essential information to understand and reduce the OS- configured to run on the target CPU only. All threads share
related interference. the same configuration that the user can update as the
Hardware-Induced Noise. To identify hardware-induced workload runs. This data is only accessed outside the main
noise, introduced in Section 2, Linux includes a tracer workload loop, thus not representing a problem for the
named hwlat. It works running a workload in the kernel, measurement phase. A mutex protects the configuration.
with preemption and IRQs disabled, avoiding all the sour- The runtime data utilized during the measurement phase is
ces of interference except the hardware and NMIs noise, organized on per-CPU structures, and the thread only
which cannot be masked. While running a busy-loop read- accesses the one related to the CPU where it is executing.
ing the time, when hwlat detects a gap in two subsequent osnoise aims to simulate a user-space workload that
reads, it reports a hardware-induced noise. follows the rules reported in Section 2.1. Specifically, a user-
The resemblance of hwlat and osnoise is not a coinci- space workload on Linux can be preempted by all types of
dence because the latter was indeed inspired to the former OS tasks mentioned in Section 2. While there are methods to
tool. osnoise is also able to detect hardware noise. Because temporarily disable thread preemption, softirqs, and inter-
it tracks all the tasks execution, when a sample noise is rupts, they have some undesired drawbacks. For example,
detected without a respective increase in any of the interfer- disabling interrupts is a costly operation, and it should be
ence counters, it is safe to assume that a layer below the avoided in cases where overhead needs to be minimized,
operating system generated the noise. like on the Linux tracing subsystem. Moreover, it is not

5.4 The osnoise Interface

The Ftrace Interface. The osnoise was integrated into the
Linux kernel as a tracer part of ftrace in version 5.14. This
is the basic interface that allows users to start and stop the
workload, and set the parameters and tracing options.

6. The quantity 1409532 ns is the sum of the four individual noise

contribution detected by the tracer and reported in Fig. 4, i.e.,
195472 þ 415172 þ 5627 þ 793261.
7. See irq_enter() and irq_exit() functions of Linux. Fig. 5. Graphical representation of Fig. 4.
DE OLIVEIRA ET AL.: OPERATING SYSTEM NOISE IN THE LINUX KERNEL 203

with an AMD Ryzen 9 5900 processor, with 12 cores and 24

threads. The system is configured with Fedora Linux 35
server and runs the kernel 5.15 patched with the PRE-
EMPT_RT patchset. osnoise has been executed via rtla
osnoise tool, both to collect a summary of the OS noise
and a histogram of each noise occurrence.
Fig. 6. Code reentrancy problem when incrementing the interference The first considered configuration of the system has no
counter. tuning applied and is so-called As-is. The system is said to
be Tuned when the best practices for CPU isolation are
applied. In this case, CPUs f0; 1g are reserved for house-
possible to mask non-maskable interrupts (NMIs). Finally, keeping work of users and the operating system tasks.
with such an operation, osnoise would influence the CPUs f2; . . . ; 23g are reserved for the workload execution,
behavior of the noise tasks. Therefore, osnoise has been and osnoise is set to run on these CPUs. The system tun-
developed with the constraint of not using methods that ing includes 1) set the performance CPUFreq governor;8 2)
prevent preemption of any class of tasks while measuring using cpu isolation features; 3) enabling RCU callbacks off-
the OS noise. If the kernel is configured as non-preemptive, load and nohz_full configuration; and 4) moving all possible
the osnoise measurement thread would be non-preemp- kernel threads and IRQs to the CPUs f0; 1g. By default,
tive too. In this particular case, a fully-preemptive behavior osnoise workload threads runs with the default’s task pri-
is simulated by adding a function to check for the need for ority (SCHED_OTHER with 0 nice). However, for the NFV
re-scheduling between two reads of the timer. This opera- use case, it is common for the users to set a real-time priority
tion does not represent a significant source of overhead for the workload. To also evaluate this specific scenario,
because Linux uses a single variable that informs whether additional experiments have been performed. In the experi-
there is a higher-priority thread waiting to run. ments marked as FIFO:1, the osnoise workload has been
set to run with priority 1 under SCHED_FIFO.
5.5.2 Dealing With Code Reentrance For the regular case (i.e., SCHED_OTHER with 0 nice) the
One of the main benefits of osnoise is the ability to associ- workload has been configured with default parameters, so
it runs with one second runtime and period, attempting to
ate the number of interferences that lead to an observed
monopolize the CPU.
noise by the workload. This feature is essential to avoid
The FIFO:1 case required some additional system and
speculation concerning the events that caused a noise, lead-
workload configuration. The system configuration includes
ing the debugging to wrong directions and causing delays
disabling the runtime throttling mechanism of Linux, allowing
in troubleshooting critical and costly systems. However,
reading the current time coherently with the interference the real-time threads to use more than 95% of CPU time,
counter is not a straightforward operation when considering and setting the ksoftirqd (thread responsible for softirq
the possibility of preemption. For example, take into consid- processing when using PREEMPT_RT) and RCU per-CPU
eration the pseudo-code in Fig. 6. The interference counter threads with FIFO 2 priority to avoid RCU [11] stalls. The
would account for one interrupt more than the number at workload has been configured with a 10 seconds period,
the time in which the timer was read. Even simple opera- and the runtime has been set to allow a 100 ms between
each period, allowing any starving thread to have a chance
tions such as incrementing a counter in a variable are not
to run. Finally, the tolerance threshold was configured to 1 ms.
guaranteed to be atomic on all architectures supported by
Linux.
In osnoise, these problems have been solved using
mainly two solutions: local atomic variables and compiler bar-
6.1 Percentage of OS Noise
riers. An excerpt from the current code is shown in Fig. 7. A six-hours experiment has been conducted for all tune/
The interference counter is defined as a local atomic integer, FIFO priority cases collecting the OS noise summary. A sum-
a atomic type of integer coherent in the local CPU. To ensure mary of the percentage OS noise is shown in Fig. 8.
that the current time is consistent with the interference counter, The maximum observed total noise was 0:5484% with the
the current time is read inside two reads of interference system As-is , while the minimum was 0:00001% for both
counter. Additionally, compiler barriers are utilized to avoid Tuned cases. It is also possible to notice the main factor for
compiler optimizations that could rearrange the interference OS noise reduction is provided by the CPU isolation fea-
counter read, moving the current time outside of protection. tures. The reason is that most of the work left for isolated
A similar approach is used for other non-atomic operations, CPUs is essential for the OS, mostly in the IRQ context.
such as computing the interference free noise for the It worth to highlight that these experiments have been
osnoise tracepoints shown in Fig. 4. In this way, osnoise conducted with the PREEMPT_RT kernel, which incurs
is able to provide coherent information while avoiding additional overheads on the average case to bound the
more costly methods, enabling measurements within one worst-case scheduling latencies. In this setup, Linux is able
ms of granularity required by NFV use-cases. to provide both low latency for interrupt-based workloads
and low OS noise for polling-based workloads in a
single solution. This flexibility is fundamental for NFV
6 EXPERIMENTAL RESULTS
This section reports on the osnoise usage for the measure- 8. Linux CPUFreq: https://www.kernel.org/doc/Documentation/
ment and trace of a system. The system is a Dell workstation cpu-freq/governors.txt.
204 IEEE TRANSACTIONS ON COMPUTERS, VOL. 72, NO. 1, JANUARY 2023

Fig. 7. Code excerpt of set_int_safe_time(): how osnoise deals with reentracy problems.

deployments with dynamic and diverse workloads in the The Tuned kernel was able to deliver consistent results,
same host. with the kernel Tuned using FIFO:1 was able to provide
below 5 ms maximum single noise occurrence. That is
because background OS activities that run as threads are
6.2 OS Noise Occurrence Analysis deferred by the real-time scheduler, without creating a fault
A six-hours experiment has been conducted for all tune/ in the system. For example, jobs dispatched on all CPUs
FIFO priority cases collecting a histogram of each detected viakworkers threads that execute deferrable work [45].
noise occurrence. This experiment is important for the NFV However, these still need to have the possibility to run to
use-case because a single long noise occurrence might cause avoid major problems. Thus, a wise choice for the develop-
the overflow of queues in the network packets processing. ment of high-performance applications with low latency
The results are presented in Fig. 9. requirements is to be aware of this property and yield some
With this experiment, it is possible to see the main prob- CPU time when it would not cause performance issues (for
lem of using the system As-is in Fig. 9a. Theosnoise instance, when network buffers are empty), even for a small
workload detected 230 out-of-scale noise samples, with amount of time, like 100 ms every 10 seconds.
the maximum value as long as 13045 ms. Fig. 9b also shows These experiments also serve to show the low impact that
that usingFIFO:1 in the system As-is represents an easy- the osnoise internals imposes in the evaluation, allowing
to-use option to reduce the maximum single noise occur- the user to receive information in the ms granularity used
rence value. The reason being is that because the workload by practitioners on other tools like cyclictest.
causes starvation of non-real-time threads, these threads It is important to highlight that the results presented in
are migrated to the CPUs with time available for them to this section are only valid for this specific scenario. Different
run. hardware, CPU count, auxiliary operating system services,
As-is using FIFO:1 however has two major drawbacks and conditions will likely provide different results. Thus
when compared against the Tuned options with or without the importance of such a tool, providing an integrated OS
using FIFO:1 in Figs. 9c and 9d. The first is the high count of noise benchmark and guidance for the fine tune of the
noise occurrences. The Tuned experiment includes the system.
nohz_full option that reduces the occurrence of the
scheduler tick, reducing the execution of the ksoftirqd kernel
6.3 Using osnoise to Trace Sources of Latency
thread that checks for expired timers and activities that fol-
low. Another difference is the tail latency, which is lower on The experiment of the system As-is with FIFO:1 pre-
the Tuned cases. This difference is explored in Section 6.3. sented an interesting result with regard to the tail latency,
The results with system Tuned in Figs. 9c and 9d show as only few samples passing the 30 ms mark. To understand
that the tune dramatically changes the entries and duration the reasons behind these cases that go over 30 ms, the
of each noise occurrence when compared with the osnoise tracer was set to trace the osnoise events, stop-
system As-is. Figs. 9e and 9f have been added to better visu- ping the tracer when a noise occurrence over 30 ms was
alize the Tuned cases. detected. The trace with sole osnoise events is shown in
Fig. 10. It shows an interrupt noise, caused by the interrupt
62, responsible for the eno1 ethernet driver. Right after the
ksoftirqd is scheduled, causing a long -duration noise
occurrence. The ksoftirqd thread is responsible for run-
ning the softirq jobs context in the PREEMPT_RT kernel
(recall from Section 2.1 that the softirq context does not exist
under PREEMPT_RT, and softirq jobs run in the thread
context).
Following the evidence that the problem is caused by
softirqs, the sole events that provide information about
softirqs were enabled, as selecting a small amount of tracing
events helps to avoid influencing too much in the timing
Fig. 8. Average percentage of OS noise observed by the workload on dif- behavior of the system due to overhead. The trace again
ferent scenarios. Error bars represent the range between minimum and reported a similar output in Fig. 11. This trace shows that
maximum percentage. the network receive (NET_RX) softirq was the reason for
DE OLIVEIRA ET AL.: OPERATING SYSTEM NOISE IN THE LINUX KERNEL 205

Fig. 9. osnoise noise occurrence per-cpu histogram under different system setup, mixing CPU isolation tune and real-time priority for the workload
(less noise occurrence and less occurrence count is better).

Fig. 10. osnoise tracer finding source of latencies4.

Fig. 11. osnoise tracer finding source of latencies augmented with other events4.

the ksoftirqd activation. The NET_RX softirq is acti- was configured to fire in the CPUs ½0 : 1. With this configu-
vated by the network driver that is running in the same ration applied, the experiment in Fig. 9b was re-executed
CPU. This it is an side effect of the eno1 ethernet driver for six hours, the results is shown in Fig. 12. This configura-
causing the interrupt. Following this evidence, the IRQ 62 tion alone was reposible to reduce the the tail latency to
206 IEEE TRANSACTIONS ON COMPUTERS, VOL. 72, NO. 1, JANUARY 2023

REFERENCES
[1] L. Tung, “SpaceX: We’ve launched 32,000 linux computers
into space for starlink internet,” Jun. 2011. [Online]. Available:
https://www.zdnet.com/article/spacex-weve-launched-32000-
linux-computers-into-space-for-starlink-internet/
[2] Network Functions Virtualisation – Introductory White Paper – An
Introduction, Benefits, Enablers, Challenges & Call for Action,”
SDN and OpenFlow World Congress. Oct. 2012. [Online]. Avail-
able: https://portal.etsi.org/NFV/NFV_White_Paper.pdf
[3] D. Kreutz, F. M. V. Ramos, P. E. Verıssimo, C. E. Rothenberg, S. Azo-
dolmolky, and S. Uhlig, “Software-defined networking: A. compre-
hensive survey,” Proc. IEEE, vol. 103, no. 1, pp. 14–76, Jan. 2015.
[4] Z. Li, L. Ruan, C. Yao, Y. Dong, and N. Cui, “A best practice of 5G
layer 2 network on Intel architecture,” in Proc. IEEE Globecom
Fig. 12. As-is using FIFO:1 after moving the network IRQ as suggest Workshops, 2019, pp. 1–5.
from trace in Fig. 11. [5] T. Cucinotta, L. Abeni, M. Marinoni, R. Mancini, and C. Vitucci,
“Strong temporal isolation among containers in OpenStack for
figures like the system Tuned, with the debug facilitated by NFV services,” IEEE Trans. Cloud Comput., to be published,
doi: 10.1109/TCC.2021.3116183.
osnoise. [6] GSM Association, “Cloud infrastructure reference model version
1.0,” [Online]. Available: https://www.gsma.com/newsroom/
7 CONCLUSION AND FUTURE WORK wp-content/uploads//NG.126-v1.0-2.pdf
[7] N. Bhushan et al., “Industry perspective,” IEEE Wireless Commun.,
Network function virtualization and modern low latency vol. 24, no. 5, pp. 6–8, Oct. 2017.
communications are creating the need for Linux systems [8] L. Mandyam and S. Hoenisch, “RAN workload performance is
equivalent on bare metal and vSphere,” [Online]. Available:
with low latency for both scheduling latency and OS noise. https://blogs.vmware.com/telco/ran-workload-performance-
These real-time HPC workloads require noise to be in the tests-on-vmware-vsphere/
order of a few tens of microseconds. [9] Intel, “FlexRAN,” [Online]. Available: https://github.com/intel/
FlexRAN
However, debugging these cases is not an easy task. [10] Red Hat, “What is NFV?” [Online]. Available: https://www.
Workload-based tools are precise for measurements but do redhat.com/en/topics/virtualization/what-is-nfv
not point to a root cause. Trace-based measurements pro- [11] P. E. McKenney, J. Fernandes, S. Boyd-Wickizer, and J. Walpole,
vide information about the cause but without an accurate “RCU usage in the linux kernel: Eighteen years later,” SIGOPS
Oper. Syst. Rev., vol. 54, no. 1, pp. 47–63, Aug. 2020. [Online].
picture of the actual noise observed by the thread.
Available: https://doi.org/10.1145/3421473.3421481
Practitioners use both methods together, but this requires [12] D. B. de Oliveira and R. S. de Oliveira, “Timing analysis of the
advanced knowledge of the tracing features, and it can often PREEMPT_RT linux kernel,” Softw., Pract. Exper., vol. 46, no. 6,
mislead the investigation because the trace is not synchro- pp. 789–819, 2016.
[13] D. B. de Oliveira, D. Casini, R. S. de Oliveira, and T. Cucinotta,
nized with the workload or adds too much overhead. “Demystifying the real-time linux scheduling latency,” in Proc.
The osnoise tool puts together the tracing and the 32nd Euromicro Conf. Real-Time Syst., 2020, pp. 9:1–9:23.
workload, providing precise information at low overhead [14] F. Cerqueira and B. Brandenburg, “A comparison of scheduling
by processing and exporting only the necessary information latency in linux, PREEMPT-RT, and LITMUS RT,” in Proc. 9th
Annu. Workshop Operating Syst. Platforms Embedded Real-Time
for pointing to the root causes of the latency, serving as a Appl., 2013, pp. 19–29.
good starting point for the investigation. [15] F. Petrini, D. Kerbyson, and S. Pakin, “The case of the missing
The experimental results show that the tool is able to supercomputer performance: Achieving optimal performance on
the 8,192 processors of ASCI Q,” in Proc. ACM/IEEE Conf. Super-
serve both as a tracer and benchmark tool, facilitated by the computing, 2003, pp. 55–55.
usage of rtla osnoise interface to collect data. The exper- [16] A. Morari, R. Gioiosa, R. W. Wisniewski, F. J. Cazorla, and
iment shows that Linux can deliver extremely low OS noise, M. Valero, “A quantitative analysis of OS noise,” in Proc. IEEE Int.
achieving maximum sample noises as low as less than 5 ms. Parallel Distrib. Process. Symp., 2011, pp. 852–863.
[17] D. B. de Oliveira, R. S. de Oliveira, and T. Cucinotta, “A thread
But more importantly, the tool is able to follow the kernel, synchronization model for the PREEMPT_RT linux kernel,”
delivering results in the desired scale. J. Syst. Archit., vol. 107, 2020, Art. no. 101729.
Both the osnoise tool and rtla osnoise interfaces are [18] J. Lelli, C. Scordino, L. Abeni, and D. Faggioli, “Deadline scheduling
an integral part of the Linux kernel, thus accessible for the in the linux kernel,” Softw.: Pract. Exp., vol. 46, no. 6, pp. 821–839,
2016.
entire Linux user base. [19] A. C. Dusseau, R. H. Arpaci, and D. E. Culler, “Effective distrib-
Because the osnoise tracer uses the most basic building uted scheduling of parallel workloads,” in Proc. ACM SIGMET-
blocks of the Linux tracing sub-system, it can be combined RICS Int. Conf. Meas. Model. Comput. Syst., 1996, pp. 25–36.
[20] J. Lehoczky, L. Sha, and Y. Ding, “The rate monotonic scheduling
with many other existing tracing tools, such as performance
algorithm: Exact characterization and average case behavior,” in
counters provided via perf tool, or to be used with graphi- Proc. Real-Time Syst. Symp., 1989, pp. 166–171.
cal interfaces provided by LTTng and KernelShark . This [21] R. Henia, A. Hamann, M. Jersak, R. Racu, K. Richter, and R. Ernst,
creates an end-less set of possibilities for future work, “System level performance analysis - the SymTA/S approach,” in
IEEE Proc. - Comput. Digit. Techn., 2005.
extending the osnoise measurements to include data from [22] G. Gracioli, A. Alhammad, R. Mancuso, A. A. Fr€ ohlich, and R. Pel-
the memory/cache, to include workload-dependent meth- lizzoni, “A survey on cache management mechanisms for real-
ods, other clock sources, and energy-aware methods, for time embedded systems,” ACM Comput. Surv., vol. 48, no. 2, Nov.
example. Extending the analysis with a more formal 2015, Art. no. 32.
[23] D. Casini, A. Biondi, G. Nelissen, and G. Buttazzo, “A holistic
approach is another possibility, as well as conducting exper- memory contention analysis for parallel real-time tasks under par-
imental evaluations based on other real-time schedulers of titioned scheduling,” in Proc. IEEE Real-Time Embedded Technol.
Linux, e.g.,SCHED_DEADLINE. Appl. Symp., 2020, pp. 239–252.
DE OLIVEIRA ET AL.: OPERATING SYSTEM NOISE IN THE LINUX KERNEL 207

[24] D. Hardy, B. Rouxel, and I. Puaut, “The heptane static worst-case [44] D. B. de Oliveira, “rlta-osnoise: Measure the operating system
execution time estimation tool,” in Proc. 17th Int. Workshop Worst- noise,” 2022. [Online]. Available: https://www.kernel.org/doc/
Case Execution Time Anal., 2017, pp. 8:1–8:12. html/latest/tools/rtla/rtla-osnoise.html
[25] B. Brandenburg, “The case for an opinionated, theory-oriented [45] J. Corbet, A. Rubini, and G. Kroah-Hartman, Linux Device Driver,
real-time operating system,” in Proc. 1st Int. Workshop Next- Gener. 3rd ed. Sebastopol, CA, USA: O’Reilly Media, 2005.
Oper. Syst. Cyber- Phys. Syst., 2019.
[26] L. Abeni and G. Buttazzo, “Adaptive bandwidth reservation for
multimedia computing,” in Proc. 6th Int. Conf. Real-Time Comput. Daniel Bristot de Oliveira received the joint PhD
Syst. Appl., 1999, pp. 70–77. degree in automation engineering from UFSC (BR)
[27] R. Mraz, “Reducing the variance of point to point transfers in the
and embedded real-time systems from Scuola
IBM 9076 parallel computer,” in Proc. Conf. Supercomputing, 1994,
Superiore Sant’Anna (IT). Currently, he is senior
pp. 620–629. principal software engineer with Red Hat, working
[28] R. Gioiosa, F. Petrini, K. Davis, and F. Lebaillif-Delamare, on developing the real-time features of the Linux ker-
“Analysis of system overhead on parallel computers,” in Proc. 4th nel. He helps in the maintenance of real-time related
IEEE Int. Symp. Signal Process. Inf. Technol., 2004, pp. 387–390. tracers and toolings for the Linux kernel and the
[29] K. B. Ferreira, P. Bridges, and R. Brightwell, “Characterizing SCHED_DEADLINE. He is an affiliate researcher
application sensitivity to OS interference using kernel-level
with the Retis Lab, and researches real-time and for-
noise injection,” in Proc. ACM/IEEE Conf. Supercomputing, 2008,
mal methods. He is an active member of the real-
pp. 1–12. time academic community, participating in the technical program committee
[30] M. Sottile and R. Minnich, “Analysis of microbenchmarks for per- of academic conferences, such as RTSS, RTAS, and ECRTS.
formance tuning of clusters,” in Proc. IEEE Int. Conf. Cluster Com-
put., 2004, pp. 371–377.
[31] D. Tsafrir, Y. Etsion, D. G. Feitelson, and S. Kirkpatrick, “System
noise, OS clock ticks, and fine-grained parallel applications,” in Daniel Casini (Member, IEEE) received the grad-
uate (cum laude) degree in embedded computing
Proc. 19th Annu. Int. Conf. Supercomputing, 2005, pp. 303–312.
systems engineering, the master’s degree jointly
[Online]. Available: https://doi.org/10.1145/1088149.1088190
[32] D. Riddoch, “sysjitter v1.4,” [Online]. Available: https://github. offered by the Scuola Superiore Sant’Anna of
com/alexeiz/sysjitter Pisa and University of Pisa, and the PhD degree
[33] RT-Tests. [Online]. Available: https://git.kernel.org/pub/scm/ in computer engineering from the Scuola Superi-
utils/rt-tests/rt-tests.git ore Sant’Anna of Pisa (with honors), working
[34] G. Tene, “jHicckup,” [Online]. Available: http://www.azulsystems. under the supervision of Prof. Alessandro Biondi
and Prof. Giorgio Buttazzo. He is an assistant
com/jHiccup
[35] P. Lawrey, “MicroJitterSampler,” [Online]. Available: http:// professor with the Real-Time Systems (ReTiS)
blog.vanillajava.blog/2013/07/micro-jitter-busy-waiting-and- Laboratory of the Scuola Superiore Sant’Anna of
binding.html Pisa. In 2019, he has been visiting scholar with the Max Planck Institute
[36] P. De, R. Kothari, and V. Mann, “Identifying sources of operating for Software Systems Germany. His research interests include software
system jitter through fine-grained kernel instrumentation,” in predictability in multi-processor systems, schedulability analysis, syn-
chronization protocols, and the design and implementation of real-time
Proc. IEEE Int. Conf. Cluster Comput., 2007, pp. 331–340.
operating systems and hypervisors.
[37] The LTTng Project, “LTTng,” [Online]. Available: https://lttng.
org/
[38] A. Nataraj, A. Morris, A. D. Malony, M. Sottile, and P. Beckman,
“The ghost in the machine: Observing the effects of kernel opera- Tommaso Cucinotta (Member, IEEE) received
tion on parallel application performance,” in Proc. ACM/IEEE the MSc degree in computer engineering from
Conf. Supercomputing, 2007, pp. 1–12. the University of Pisa, Italy, and the PhD degree
[39] A. Nataraj, A. D. Malony, S. Shende, and A. Morris, “Kernel-level in computer engineering from Scuola Superiore
measurement for integrated parallel performance views: The Sant’Anna (SSSA), in Pisa, where he has been
KTAU project,” in Proc. IEEE Int. Conf. Cluster Comput., 2006, investigating on real-time scheduling for soft real-
pp. 1–12. time and multimedia applications, and predictabil-
[40] N. M. Gonzalez, A. Morari, and F. Checconi, “Jitter-trace: A low- ity in infrastructures for cloud computing and NFV.
overhead OS noise tracing tool based on linux perf,” in Proc. 7th He has been MTS in Bell Labs in Dublin, Ireland,
Int. Workshop Runtime Oper. Syst. Supercomputers, 2017, Art. no. 2. investigating on security and real-time perfor-
[41] D. B. de Oliveira, “Osnoise tracer,” [Online]. Available: https:// mance of cloud services. He has been a software
www.kernel.org/doc/html/latest/trace/osnoise-tracer.html, engineer in Amazon Web Services in Dublin, Ireland, where he worked
2021. on improving the performance and scalability of DynamoDB. Since
[42] D. B. de Oliveira, T. Cucinotta, and R. S. de Oliveira, “Efficient for- 2016, he is associate professor with SSSA and head of the Real-Time
mal verification for the linux kernel,” in Proc. Int. Conf. Softw. Eng. Systems Lab (RETIS) since 2019.
Formal Methods, 2019, pp. 315–332.
[43] L. Soares and M. Stumm, “FlexSC: Flexible system call scheduling
" For more information on this or any other computing topic,
with exception-less system calls,” in Proc. 9th USENIX Conf. Oper.
Syst. Des. Implementation, 2010, pp. 33–46. please visit our Digital Library at www.computer.org/csdl.

Open Access funding provided by ‘Scuola Superiore ‘S.Anna’ di Studi Universitari e di Perfezionamento’
within the CRUI CARE Agreement