REDBEND User Manual

REDBEND Device Virtualization for Connected

Vehicles

Hypervisor Description

6$0681*&RQILGHQWLDO
Software Version 11.6
Doc. Rev. 1.8 / None
Reference: DV-0018
&2$6,$1(;(//&2/7'PFMLQRDW
Date: 12/10/2021
REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description

Table of Contents
1 Introduction....................................................................................................................... 5
1.1 This Document ....................................................................................................................5
1.2 Related Documentation ......................................................................................................5
2 Prerequisite ....................................................................................................................... 6
3 Overview of the Hypervisor Services ................................................................................... 6
3.1 Hardware Resource Partitioning, Sharing, and Virtualization ............................................6
3.2 Guest OS Management Services .........................................................................................7
3.3 Summary of Hypervisor Services ........................................................................................7
4 Memory Management and Spatial Isolation ....................................................................... 8
4.1 RAM ....................................................................................................................................8
4.1.1 Virtual Machine Memory Partition .....................................................................8
4.1.2 Hypervisor Memory Partition ..............................................................................9
4.1.3 Memory Granting Between VM ....................................................................... 10
4.2 I/O Memory ..................................................................................................................... 10
4.3 Interrupt Management .................................................................................................... 11
4.3.1 Interrupt Processing ......................................................................................... 12

6$0681*&RQILGHQWLDO
5 Scheduling and Temporal Isolation ................................................................................... 13
5.1 Exclusive CPU Binding ...................................................................................................... 13
5.2 Hypervisor Scheduling ..................................................................................................... 14
&2$6,$1(;(//&2/7'PFMLQRDW
5.3 CPU Affinity ...................................................................................................................... 14
5.4 CPU handover .................................................................................................................. 15
6 Communication Channels ................................................................................................. 15
6.1 VM - Hardware Communication Channels ...................................................................... 15
6.2 VM - SMC Communication Channels ............................................................................... 16
6.3 VM - Hypervisor Communication Channels ..................................................................... 17
6.4 Inter-VM Communication Channels ................................................................................ 18
6.4.1 Vlink .................................................................................................................. 18
7 Hypervisor Public Interfaces ............................................................................................. 19
7.1 Example: Console Output Management.......................................................................... 20
7.2 Peripheral Emulation example ........................................................................................ 21
8 Monitoring ...................................................................................................................... 22
9 Hypervisor BSP................................................................................................................. 23
9.1 Typical BSP ....................................................................................................................... 23
9.2 BSP Source Tree Layout ................................................................................................... 24
10 Initialization..................................................................................................................... 24
11 Build Overview ................................................................................................................ 25
11.1 Build Tool Chains.............................................................................................................. 26

© 2021 HARMAN. All Rights Reserved [2 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description
11.2 Multi-VM Boot Image ...................................................................................................... 26

Table of Figures
Figure 1: Example Architecture Diagram .............................................................................................6
Figure 2: RAM partition ........................................................................................................................9
Figure 3:Hypervisor Memory Partition ............................................................................................. 10
Figure 4: Interrupts Mapping ............................................................................................................ 12
Figure 5: Interrupt Processing ........................................................................................................... 13
Figure 7: CPU Affinity ........................................................................................................................ 14
Figure 8: Hardware Access ............................................................................................................... 16
Figure 9: VM SMC Communication ................................................................................................... 17
Figure 10: VM - Hypervisor Communication .................................................................................... 18
Figure 11 : Vlink Architecture ............................................................................................................ 19
Figure 12: Hypervisor Public Interfaces............................................................................................. 20
Figure 13: Hypercall Example, Write to the Console ........................................................................ 21
Figure 14: Peripheral Emulation........................................................................................................ 22

6$0681*&RQILGHQWLDO
Figure 15: BSP Source Tree Layout .................................................................................................... 24
Figure 16: Build Overview ................................................................................................................. 25
&2$6,$1(;(//&2/7'PFMLQRDW
Figure 17: Detailed Hypervisor Image Generation ............................................................................ 26
Figure 18: Multi-VM Boot Image ....................................................................................................... 27

© 2021 HARMAN. All Rights Reserved [3 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description

6$0681*&RQILGHQWLDO
&2$6,$1(;(//&2/7'PFMLQRDW

© 2021 HARMAN. All Rights Reserved [4 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description

1 Introduction
1.1 This Document
This document describes the Hypervisor provided with the REDBEND Device Virtualization for
Connected Vehicles product. As described in the "Product Description" document, the REDBEND
Device Virtualization for Connected Vehicles is more than a mere Hypervisor, as it includes tools
and virtual drivers to be plugged in guest OSes.
The Hypervisor is the core component of the product providing virtualization services which
enable to run simultaneously multiple guest OSes on the same hardware platform. It is a Type-1
virtualization solution.
This document covers the generic virtualization services provided by the Hypervisor: memory
management and spatial isolation, scheduling and temporal isolation, inter-virtual machines
communication, Hypervisor interfaces, initialization.
The Hypervisor needs to deal with different hardware platforms (SoCs and boards), and possibly
different processor architectures. Hence it is architected so that platform dependent services are
implemented in a BSP (Board Support Package), enabling to port the Hypervisor to a new SoC
and/or board within a supported processor architecture (e.g. ARMv8).

1.2 Related Documentation

6$0681*&RQILGHQWLDO
The Device Virtualization for Connected Vehicles product includes the following documentation:
• Architecture:
&2$6,$1(;(//&2/7'PFMLQRDW
° Product Description
° System Architecture and Overview
° Hypervisor Description
• Configuration guides:
° Hypervisor First Steps
° Configuring Guest OS and Applications
• User guides:
° Inter-VM Communication
° How to Move a Pass-Through Device from System Domain to User Domain
° Inter-VM Communication for Linux User Space Applications
° Virtual Driver Writer's Guide
° Virtual ION Memory Allocator
° Console Multiplexer
° Debugging with GDB
° Hypervisor Awareness in Debugging Environment
° Suspend to RAM technical report
° Hypervisor Trace Visualization
° CPU handover
• Reference manuals:
° Device Virtualization Reference Manual (Public and/or BSP editions)
° Virtual Device Driver Reference Manual
• Release Notes

© 2021 HARMAN. All Rights Reserved [5 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description

2 Prerequisite
The reader should be familiar with virtualization concepts.

3 Overview of the Hypervisor Services

Note: This is a copy of a chapter of "Virtualization Overview and Architecture" document

3.1 Hardware Resource Partitioning, Sharing, and

Virtualization
The Hypervisor partitions hardware platform resources (RAM and IO devices) among multiple
Virtual Machines. Access to the CPU between the different Virtual Machines is achieved by the
scheduling provided by the Hypervisor. In addition, the Hypervisor virtualizes some instructions of
the processor (especially for I/O accesses) and the MMU for each guest OS.
The Hypervisor does the following:
• Coordinates initialization of the various guest OSes
• Assigns memory (RAM) and IO (devices) resources to each guest OS, and isolates those
resources using Stage-2 MMU and System MMU hardware
• Schedules guest OSes on physical processors, running one or more of their virtual CPUs

6$0681*&RQILGHQWLDO
(vCPUs) on the available physical CPUs
• Transmits cross-interrupts from one guest OS to another
• Provides low-level inter guest OS communications based on shared memory and cross-
interrupts (bus-based communication paradigm)
&2$6,$1(;(//&2/7'PFMLQRDW
This architecture allows all guest OSes to run independently and to be readily stopped and
restarted. The REDBEND Virtual Device Driver Framework allows (re)synchronizing guest OSes that
share access to the same peripheral devices.
The following figure shows an example of three virtual machines: one virtual machine for an RTOS
and two virtual machines for two general purpose guest OSes.

Applications Applications
Applications
Guest OS Guest OS
Highest priority OS
(RTOS or other)

vCPU 1 vCPU 2 vCPU 3 vCPU 4

REDBEND Hypervisor

CPU 1 CPU 2 Hardware platform

(CPU, memory, peripheral devices)

Figure 1: Example Architecture Diagram

© 2021 HARMAN. All Rights Reserved [6 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description
The physical memory is partitioned between the guest OSes. The processors are shared by the
guest OSes under the control of the Hypervisor scheduler.
All hardware resources available are split into two basic categories: static and dynamic.
Typically hardware resources such as RAM and IO devices are statically assigned to each given
partition at build or boot time. Their owner is, therefore, rarely changed. The assignment of static
resources among guest OSes is quite straightforward and depends on the overall system
configuration. The Hypervisor provides a global static resource repository and a common
framework to manage resource allocation.
The main dynamic resources available on a hardware platform are the CPU together with CPU
auxiliary devices such as FPU, Interrupt Controller, Timer or MMU. The main role of the Hypervisor
is to dynamically share the CPU resource between virtual machines using some predefined
scheduling policy. The Hypervisor provides one or more virtual CPUs (together with auxiliary
devices) to each virtual machine and schedules them dynamically to the available physical CPU. At
guest OS switch time, the Hypervisor saves and restores the state of the CPU and auxiliary devices
making such a partition switch transparent to guest OSes.

3.2 Guest OS Management Services

Some Virtualization configurations provide API enabling one guest OS to manage another. These
services provide:
• Temporary suspension of the execution of a guest OS

6$0681*&RQILGHQWLDO
• Resumption of the execution of a previously suspended guest OS
• Halt and reboot of a guest OS
• Modification of the priority of a guest OS
&2$6,$1(;(//&2/7'PFMLQRDW
• Acquisition of usage statistics
Available services vary per guest OS and can be masked. In Linux, for example, masking can be
done by entering the relevant information in a control file located in the /proc subtree.

3.3 Summary of Hypervisor Services

The Hypervisor provides API to:
• Control the guest OSes :
° Stop, start or restart a guest OS
° Change the guest OS scheduling policy including priority
° Get the state (running, suspended, stopped) of VM.
• Perform guest OS power management:
° Hot plug/unplug a virtual CPU
° Suspend a guest OS to RAM
• Set or get the value of Hypervisor properties
In addition a mechanism of notification allows a guest OS to know when a property has been
modified (Property API implements a publish/subscribe mechanism)
• Introspect a VM from another VM:
° Get/Set register context of the virtual CPU
° Read/Write the memory of VM
• A rich Inter-VM communication API, including a

© 2021 HARMAN. All Rights Reserved [7 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description
° API for browsing the virtual device database
° API for handling resources (Cross interrupt, device and shared memory) associated to
virtual devices
° Control and police inter-VM communication by using “Bridge drivers”
° API for sending cross-interrupt
• A logging mechanism (with data persistent to machine reboot)
° Allow a VM to log a system message
° Allow a VM to retrieve the log of messages
• A rich monitoring framework :
° Gathering usage statistics
° Tracing events such as interrupts, exceptions and hyper-calls
• Create / control virtual watchdog.
Note that all API are controlled by a credential mechanism which prevents non-critical guest OSes
to access critical information.

4 Memory Management and Spatial Isolation

One platform, one Hypervisor and multiple guest OSes!
The Hypervisor and the guest OSes have various memory needs. Each of them requires some
amount of physical memory, access to some specific IO memory and needs to deal with a subset of

6$0681*&RQILGHQWLDO
interrupt requests. This is achieved by statically partitioning these physical resources between the
Hypervisor and the guest OSes. As the target systems are embedded static systems, there is no
need to provide dynamic partition mechanisms.
All partitions are defined up-front as part of the configuration of the overall system. The
&2$6,$1(;(//&2/7'PFMLQRDW
configuration is described in a Device Tree provided at system build time. During initialization, the
Hypervisor parses the Device Tree and instantiates the corresponding partitions.
As a result, configuration errors if any are found during early initialization. If the system passes the
initialization step, no more errors such as lack of memory may be encountered once initialization is
over.

4.1 RAM
The physical memory is partitioned by the Hypervisor according to the memory requirements
defined for each VM in the Device Tree. No memory overcommit is allowed. If there is not enough
physical memory to match the configuration defined in the Device Tree, the system stops with a
"configuration error".

4.1.1 Virtual Machine Memory Partition

Partitioning the physical memory is not enough. The Hypervisor ensures that a guest OS running in
its Virtual Machine can only access its memory partition. As a result, a guest OS cannot read or
write the physical memory of others Virtual Machines or the physical memory used by the
Hypervisor.
To achieve this isolation, the Hypervisor relies on the two stages MMU (Memory Management
Unit) featured by the ARM processor. For each memory partition (for each VM), the Hypervisor
establishes a mapping: Intermediate Physical Addresses (IPA) are converted by the stage-2 MMU
into Physical Addresses (PA). The IPA are the addresses presented to the guest OS. The guest OS

© 2021 HARMAN. All Rights Reserved [8 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description
uses these addresses as if they were "physical addresses" and can use the stage-1 MMU to
establish any mapping it wishes to present Virtual Addresses (VA) to its own processes.
The stage-2 MMU can only be configured by the Hypervisor. A guest OS can only control the stage-
1 MMU and has thus no way to escape from the memory partition defined by the stage-2
mapping.

Stage-1 MMU Stage-1 MMU

VM 1 VM 2
PA PA
IPA IPA

Stage-2 MMU Stage-2 MMU

PA Hypervisor PA

VM1 memory partition VM2 memory partition

6$0681*&RQILGHQWLDO
Hypervisor memory partition

Figure 2: RAM partition

&2$6,$1(;(//&2/7'PFMLQRDW
The Hypervisor runs in its own memory partition. The Hypervisor memory partition is disjoint from
the partitions allocated to the Virtual Machines.
The stage-2 MMU has two roles: it isolates each Virtual Machine from the others and it performs a
translation to make Virtual Machines independent from the addresses of the physical memory it
uses. The mapping provided by stage-2 MMU can be either identical or non-identical. When an
identical mapping is used, Intermediate Physical Addresses (IPA) are identical to Physical
Addresses (PA). In a non-identical mapping, IPA and PA differ. A Virtual Machine may use an
identical mapping while another one may use a non-identical mapping. The mapping is defined by
the Hypervisor Device Tree (See Device Virtualization Reference Manual > Configuration > VM
Virtual Platform Bindings > Virtual Platform Memory Node - "memory").

4.1.2 Hypervisor Memory Partition

The Hypervisor memory partition is described in the Device Tree as follows:
&hyp_memory {
reg = <0 0xd2000000 0x01000000 >;
no-heap-alloc-after-boot;
hyp-region {
reg = <0 0x00080000 VDT_AUTO_SIZE>;
};
}

© 2021 HARMAN. All Rights Reserved [9 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description
The reg property defines the address and size of the Hypervisor physical memory partition. The
hyp-region/reg defines the offset within the partition and the size of the area where the
Hypervisor executable binary is located. The no-heap-alloc-after-boot flag tells the
Hypervisor to reject dynamic memory allocation requests once its initialization is over.
During initialization, dynamic memory allocation is required. Such requests are satisfied from three
different heap areas:
• A small "Early Heap" area part of the Hypervisor binary BSS, used for some very early needs
of the Hypervisor itself.
• A Lower (Early) Heap is used for MMU translation tables allocation
• A Higher Heap is used for all other allocations. It is also used for allocation of translation
tables when the Lower Heap is exhausted. This Higher Heap is available only after the Multi-
VM Boot Image parsing.

0xd2000000
Lower Heap
0xd2080000
Hypervisor
Binary

Early Heap

6$0681*&RQILGHQWLDO 0xd3000000
Higher Heap

&2$6,$1(;(//&2/7'PFMLQRDW
Figure 3:Hypervisor Memory Partition

4.1.3 Memory Granting Between VM

Memory partitions of VMs are isolated from each other, hence exchanging data between VMs for
communication purposes need to use dedicated communication channels (Inter-VM
Communication Channels). This leads to copy information from the VM memory to the
communication channel and from the communication channel to memory. While this is acceptable
for small amounts of data (meta data), the cost is far too large for large amount of data.
To overcome this performance issue, the Hypervisor provides ways for a VM to grant another VM
access to some of its memory. Access can be granted for dynamically defined regions (page aligned
areas of memory). The access may be granted for reading, writing, and for DMA access.
This may be used by a client VM to provide access to some of its memory to a server VM to
perform a disk device operation. As the server VM may now read/write the client memory, no data
copy is required, hence I/O overhead is kept to a minimum needed to set up the memory access
for the server VM.

4.2 I/O Memory

Based on the configuration defined by the Hypervisor Device Tree, the Hypervisor grants access to
I/O memory regions or registers to a Virtual Machine. Again, the Hypervisor uses the stage-2 MMU
to provide such access to the Virtual Machine. Hence, the guest OS can then directly interact with

© 2021 HARMAN. All Rights Reserved [10 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description
the device controlled by the I/O memory. Such an interaction is achieved without any further
control of the Hypervisor. This is known as a "pass-through" device configuration.
As the access to the physical device is granted to the Virtual Machine without any further control
of the Hypervisor, it is possible to use legacy device driver in the guest OS to manage the device. In
addition, as there is no Hypervisor imposed overhead after configuration, performance of I/O
operations are not impacted.
Most of the time, an identical mapping is used so that Intermediate Physical Addresses seen by the
Virtual Machine correspond to the Physical Addresses. Whether the mapping is identical or not is
controlled by properties in the Hypervisor Device Tree (See Device Virtualization Reference
Manual > Configuration > VM Virtual Platform Bindings > Virtual Platform I/O Memory Node -
"vl,io-memory").
By default I/O memory mapping is page-aligned: addresses and sizes are multiples of the page size
(4K on ARM processors). However, it is possible to control access at a finer granularity than the
page size, or even to disable access to a range of addresses. Memory ranges must be aligned on
power of two values. When such finer control is established, there are two possibilities: either the
access is denied in which case an exception is raised or the access is granted in which case it is
transparently performed by the Hypervisor on behalf of the guest OS.
As for RAM, I/O memory mapping is maintained by the Hypervisor on a per Virtual Machine basis.

4.3 Interrupt Management

6$0681*&RQILGHQWLDO
All physical interrupts are virtualized by the Hypervisor. When a physical interrupt occurs, the
interrupt virtualization component of the Hypervisor routes that interrupt to the appropriate
Virtual Machine where it triggers the Interrupt Service Routine of the guest OS.

&2$6,$1(;(//&2/7'PFMLQRDW
From the Hypervisor point of view, an interrupt is known as an eXtended interrupt, also called
XIRQ, and is part of the range [0, 1024), which is composed of two disjoint ranges, e.g:
• [0, 5121): physical interrupts
• [512, 1024): software interrupts, also called cross-interrupts, typically used for
communication between virtual drivers
Interrupt mapping configuration defines, for each XIRQ, if it is forwarded to a given VM, and under
which virtual Interrupt Controller IRQ ID. Below picture provides an example of interrupt mapping
configuration of two VMs:

1
512 is the typical default value, but it depends on the SoC

© 2021 HARMAN. All Rights Reserved [11 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description

Virtual IC Space VM1 space Hardware IC Space VM2 Space Virtual IC Space
(vGIC Space) (GIC Space) (vGIC Space)
1019 1023 1024
1019

768

639
512 512 512
511
495
332 384
315
300
284

32
0 0 0 0 0

Software Interrupt IDs Hardware Interrupt IDs

Figure 4: Interrupts Mapping

6$0681*&RQILGHQWLDO
As seen in the above picture, Figure 4: Interrupts Mapping, hardware interrupts [0, 512) are
partitioned between different guest OSes: one hardware interrupt ID may be assigned only to one
VM.

&2$6,$1(;(//&2/7'PFMLQRDW
Software interrupts, on the other hand, obey to a different rule: each VM is provided with an
independent space of Software Interrupts. Thus, the same software interrupt ID may be reused in
multiple VMs.
In a large majority of use cases, an identical mapping will be defined by each VM for hardware and
software interrupts. One reason of a non-identical mapping might be a limitation of the number of
supported interrupt lines in the guest OS kernel. In above picture, VM3 only supports interrupts up
to 892, so we use a non-identical mapping.

4.3.1 Interrupt Processing

The following picture, Figure 5: Interrupt Processing, shows how XIRQ, extended Interrupts, are
processed by the Hypervisor.

© 2021 HARMAN. All Rights Reserved [12 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description
vdriver GuestOS
IRQ GuestOS IRQ
or Interrupt
GIC driver
‘regular’ managem
driver
IRQ
EL1
EL2

Hypervisor Hypervisor Hypervisor

hw IRQ hw IRQ hw IRQ Hypervisor
Interrupt BSP Interrupt
vGIC driver
Management GIC driver Virtualization sw IRQ

hw IRQ Hypervisor IRQ Emulated IRQ

Figure 5: Interrupt Processing

As explained previously, there are two kinds of interrupts:

• Hardware ones, denoted 'hw IRQ', triggered by the hardware, and
• Software ones, denoted 'sw IRQ', triggered by guest OSes or by the Hypervisor.
A hardware interrupt is first handled by the "Interrupt Management" module of the Hypervisor,
and then passed to the GIC driver. Such a driver is provided by the Hypervisor BSP. The GIC driver
determines whether the interrupt is directed toward the Hypervisor or toward one of the Virtual

6$0681*&RQILGHQWLDO
Machines.
Hardware interrupts directed to a VM is virtualized by the Hypervisor Interrupt Virtualization
component. This component also takes care of software interrupts. Both are transmitted to a

&2$6,$1(;(//&2/7'PFMLQRDW
virtual GIC driver provided by the Hypervisor.
This virtual GIC driver sends the virtual interrupt toward the appropriate Virtual Machine where it
is processed by a guest OS GIC driver and then by the appropriate device driver.

5 Scheduling and Temporal Isolation

The Hypervisor provides mechanisms and services to ensure that the behavior of a Virtual
Machine cannot disturb the behavior of another Virtual Machine from a temporal stand point. The
configuration of a Virtual Machine defines its number of virtual CPUs (vCPU). One of the runtime
job of the Hypervisor is to schedule these vCPUs on the physical CPUs of the board.

5.1 Exclusive CPU Binding

As part of the configuration, defined by the Device Tree, it is possible to bind a given vCPU on a
physical CPU and to make sure that only that vCPU uses that physical CPU. This is known as

GuestOS1 GuestOS2
vCPU0 vCPU0 vCPU1 vCPU2

CPU0 CPU1 CPU2 CPU3

© 2021 HARMAN. All Rights Reserved [13 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description
exclusive CPU binding. In such a case, the Hypervisor has no dynamic scheduling to perform. The
only remaining job of the Hypervisor is to dispatch interrupts when needed. Therefore the impact
imposed by the Hypervisor is a small and bounded time overhead for interrupt latency from the
guest OS point of view. As the binding is exclusive, no other vCPU can steal the physical CPU.

Error! Bookmark not defined.

In the above figure, Error! Bookmark not defined., each VCPU is exclusively bound to a physical
CPU. There is no scheduling overhead imposed by the Hypervisor, and the two Virtual Machines
are isolated from each other from a temporal standpoint.

5.2 Hypervisor Scheduling

The Hypervisor provides more flexible ways to share the physical CPUs than the static partitioning
mechanism described above. It actually provides a fixed priority preemptive scheduler to
dynamically assign vCPUs to physical CPUs. The scheduler supports static priorities defined at
configuration time on a per Virtual Machine basis. Priorities range from 0, the highest priority, to
253, the lowest priority. All vCPUs of a Virtual Machine inherit the priority of the VM. A vCPU
whose priority is higher than the priority of a running vCPU always preempts the currently running
low-priority vCPU.
The Hypervisor provides the following guarantees:
• The vCPU with the highest priority is running.

6$0681*&RQILGHQWLDO
•

•
The time to preempt a low priority vCPU and assign the physical CPU to a high priority vCPU is
small and bounded.
The overhead induced for interrupt virtualization is small and bounded.
&2$6,$1(;(//&2/7'PFMLQRDW
• The different vCPUs of a given VM are equally scheduled on physical CPUs. No vCPU shall lag
behind other vCPUs of the same VM.
The Hypervisor scheduler also provides a fair share scheduling policy. It is currently not described
in this document.

5.3 CPU Affinity

At configuration time, it is possible to define the affinity of vCPUs: it is possible to define the set of
physical CPUs where vCPUs of a given VM are allowed to execute.
It is also possible to pin a vCPU on a physical CPU: the vCPU executes only on that physical CPU
and never "migrates" to a different physical CPU.

GuestOS1 GuestOS2
vCPU0 vCPU0 vCPU1 vCPU2 vCPU3

CPU0 CPU1 CPU2 CPU3

Figure 7: CPU Affinity

© 2021 HARMAN. All Rights Reserved [14 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description
In the above figure:
• vCPU0 of guest OS1 can only execute on physical CPU0
• vCPUs of guest OS2 can execute on any physical CPU.

5.4 CPU handover

The CPU handover mechanism permits to dynamically assign a physical CPU to a VM. As a result,
the "receiving" VM has more processing power available. Usually, the physical CPU is hot
unplugged from another VM, which gets less processing power.
This is a way to dynamically assign processing power to VMs which need it. Even though the
mechanism is dynamic, it relies on appropriate VCPU configuration properties.
When a CPU handover is triggered, the Hypervisor signals the VM using the physical CPU that the
physical CPU is being unplugged. It the signals the receiving VM that a new physical CPU is being
plugged. When the VM are configured with pinned VCPUs, the interaction between the Hypervisor
and the VM is synchronous, otherwise the interaction is asynchronous.
A CPU handover is managed as a transaction by the Hypervisor.
Usually a user space application running in one of the VM acts as a cpu handover governor and
implements the policy which rules the CPU handover operations.

6 Communication Channels
6$0681*&RQILGHQWLDO
6.1 VM - Hardware Communication Channels
&2$6,$1(;(//&2/7'PFMLQRDW
Based on the Virtual Machine configuration defined in the Device Tree, the Hypervisor grants the
Virtual Machine access to a subset of the hardware resources (3.1).
During initialization, the Hypervisor populates the Virtual Machine I/O space with a subset of
hardware I/O resources granted to the Virtual Machine as specified by a its static configuration. As
a result, the guest OS can access such resources directly without any Hypervisor assistance. No
runtime overhead is imposed on such I/O operations.
The Hypervisor transparently sends to the guest OS a subset of the hardware IRQs granted to the
Virtual Machine as specified by its static configuration. Interrupts are virtualized. There is minimal
bounded overhead introduced by virtualization between the hardware interrupt is triggered and
the time the virtualized interrupt is delivered to the guest OS IRQ routine.

© 2021 HARMAN. All Rights Reserved [15 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description

GuestOS

I/O access IRQ receive

VM I/O Space Interrupt

Controller

Hypervisor

I/O Space Interrupt

SoC Controller

Figure 8: Hardware Access

6$0681*&RQILGHQWLDO
6.2 VM - SMC Communication Channels
&2$6,$1(;(//&2/7'PFMLQRDW
The Hypervisor traps Secure Monitor Calls (SMC) instructions issued by the guest OS as specified
by its static Virtual Machine configuration.
The Hypervisor BSP implements a customizable SMC policy combining the following mechanisms
provided by the Hypervisor frameworks:
• SMC emulation
• Execution of this or other SMCs
• SMC rejection

© 2021 HARMAN. All Rights Reserved [16 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description

GuestOS GuestOS
SMC instr. SMC instr.

VM SMC VM SMC
Trap Trap

SMC
Hypervisor Policy

SMC
Secure World

Figure 9: VM SMC Communication

6$0681*&RQILGHQWLDO
6.3 VM - Hypervisor Communication Channels
Guest OS and Hypervisor communicate via
&2$6,$1(;(//&2/7'PFMLQRDW
• Hypercalls
The Hypervisor polices hypercalls according to the access list defined as part of the
configuration of the Virtual Machine. Whether the guest OS is entitled to invoke a given
operation is specified by such an access list.
• Virtual I/O peripherals
The Hypervisor populates the Virtual Machine I/O Space with virtual I/O resources as
specified by the static Virtual Machine configuration. The Hypervisor intercepts any guest OS
access to such a virtual resource and emulates the peripheral logic transparently for the guest
OS.
• Virtual interrupts
All IRQs delivered to a guest OS are statically defined in the Virtual Machine configuration.

© 2021 HARMAN. All Rights Reserved [17 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description

GuestOS
Hypercall I/O access IRQ receive

Access I/O Space Interrupt

List Controller
VM

Hypervisor

Figure 10: VM - Hypervisor Communication

6.4 Inter-VM Communication Channels

The Hypervisor provides vlinks as a communication mechanism enabling two Virtual Machines to
efficiently exchange data. Such communication channels are based on a few simple mechanisms

6$0681*&RQILGHQWLDO
provided by a communication framework, on top of which more complex and specialized
communication protocols can be built and then used to connect "communication virtual drivers",
running in different Virtual Machines:
• Memory regions to support data exchanges between different Virtual Machines:
&2$6,$1(;(//&2/7'PFMLQRDW
° PDEV: Persistent Device memory to handle device metadata. Such a memory region is
accessed only by one Virtual Machine and the Hypervisor and is not shared between two
Virtual Machines. The metadata contains the description of the virtual link end-points
and data descriptors. Access to this metadata by virtual drivers running in the guest OS
is done by simply dereferencing pointers to such data. The content of PDEV memory
regions is preserved across guest OS reboot.
° PMEM: Persistent Memory to handle data to be exchanged between Virtual Machines.
Such memory is allocated by the Hypervisor and can be mapped by multiple Virtual
Machines.
• Cross-interrupts: virtual interrupts which can be sent from one Virtual Machine to another to
signal it that something has to be done. See 4.3 Interrupt Management.
All of these mechanisms are used by virtual drivers running in guest OSes.

6.4.1 Vlink
A vlink is an inter-VM communication channel between two Virtual Machines. A virtual
communication link is a peer-to-peer, uni-directional communication channel between a server
side end-point and a client side end-point. Usually the vlink server and client sides are located in 2
different Virtual Machines but can be located within the same Virtual Machine.
A vlink encapsulates channel’s persistent resources:

© 2021 HARMAN. All Rights Reserved [18 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description
• XIRQs – software cross-interrupts
• PMEM – shared memory
• PDEV – private memory shared with Hypervisor only
Allocated persistent resources survive Virtual Machine reboot.
For a more detailed description of vlinks, please refer to the "Virtual Driver Writer's Guide"
document.

VM1 VM2

XIRQs XIRQs

PMEM

PDEV PDEV

vlink

6$0681*&RQILGHQWLDO
HYPERVISOR

&2$6,$1(;(//&2/7'PFMLQRDW
Figure 11 : Vlink Architecture

7 Hypervisor Public Interfaces

The Hypervisor interacts with the rest of the word, guest OSes, Hardware, Configuration, through
different mechanisms:
• Hypercalls used by guest OSes to explicitly request a service from the Hypervisor
• Synchronous Exceptions triggered by guest OSes when executing an instruction trapped by
the Hypervisor
• Asynchronous Exceptions, virtual interrupts, delivered by the Hypervisor to the guest OSes
• Physical Asynchronous Exceptions, physical interrupts delivered by the hardware to the
Hypervisor
• Configuration data:
° Hypervisor Device Tree
° Physical platform Device tree
° Per Virtual Machine Device Tree
The following picture, Error! Reference source not found., illustrates these different interfaces.

© 2021 HARMAN. All Rights Reserved [19 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description

Guest OS … Guest OS

Asynchronous
Hyper- Synchronous
Exceptions
calls Exceptions
(Interrupts)
Configuration

VM
Hypervisor VM
VM

Hypervisor
Hardware Asynchronous Exceptions
(Interrupts)

6$0681*&RQILGHQWLDO Figure 12: Hypervisor Public Interfaces

7.1 &2$6,$1(;(//&2/7'PFMLQRDW
Example: Console Output Management
This section describes an example of hypercall used by guest OSes to print message on the
console.
A guest OS may print messages on the console using the following hypercalls:
• hyp_call_putchar(‘H’): prints one character
• hyp_call_putstr(“Hello world”) : prints a character string
• hyp_call_cons_write(“Hello world”, 11) : prints a number of characters
Hypervisor aggregates all guest OS console output in a single circular buffer called Hypervisor
Console History.
The Hypervisor prints the Console History on the UART console, so that guest OS outputs appear
on the console. Output of guest OSes are differentiated by using different colors for each guest OS.
The Hypervisor provides a hypercall to retrieve the Console History buffer content:
• char ch = hyp_call_cons_hist_getchar (&ordinal);
It is also possible to retrieve the console content via:
• cat /dev/vlx-history from a shell of a Linux guest OS,
• adb shell cat /dev/vlx-history from a shell of the Host OS.

© 2021 HARMAN. All Rights Reserved [20 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description
/dev/vlx-history
GuestOS
GuestOS

hyp_call_cons_write() hyp_call_cons_hist_getchar()

Console
History
Buffer

UART

Figure 13: Hypercall Example, Write to the Console

6$0681*&RQILGHQWLDO
7.2 Peripheral Emulation example
Instead of using hypercalls as shown in the previous section, a guest OS may print messages on the
console using a specific (virtual) UART peripheral (e.g. S4210).
&2$6,$1(;(//&2/7'PFMLQRDW
A Hypervisor Emulation Driver (e.g. vs5pv210-uart.c) intercepts all guest OS accesses to the
UART registers, emulates the UART logic and aggregates the console output in the Console History
buffer.
This requires that the VM configuration (vplatform) specifies the emulated (virtual) UART
peripheral, e.g.:
/ {
serial@14C30000 {
compatible = "samsung,exynos4210-uart";
reg = <0 0x14C30000 0x100>;
interrupts = <0 433 0>;
vl,vfifo = "console";
};
};

© 2021 HARMAN. All Rights Reserved [21 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description

GuestOS /dev/vlx-history
GuestOS

(virtual)
UART

Synchronous hyp_call_cons_hist_getchar()
Exceptions

Emulation
Driver
Console
History
Buffer

UART

Figure 14: Peripheral Emulation

6$0681*&RQILGHQWLDO
8 Monitoring
The Hypervisor offers services to help monitor, tune and debug the overall software system.
&2$6,$1(;(//&2/7'PFMLQRDW
• Monitoring and diagnostic
It collects data about Hypervisor events such as VM restarts, context switches, interrupts,
exceptions. The Hypervisor provides APIs to access these collected data. Collecting data is
dynamically enabled and disabled using Hypervisor properties.
• Hypervisor event logging
Hypervisor events are logged in per physical CPU circular log buffer rings. Oldest events can
be overwritten if they are not periodically retrieved by a Guest OS. The content of these log
buffers persist across machine reboot, allowing to inspect data even after failure.
• Hypervisor event statistics
The Hypervisor counts the occurrences of Hypervisor events in per CPU/vCPU counter
matrices. Events may be: event counters e.g. interrupts, exceptions, hypercalls counts, event
timing counters.
• CPU snapshots
In case of panic, the Hypervisor dumps the system registers in a persistent memory area. The
content of this area is preserved across warm reset of the platform, hence the content of the
registers can be examined after reboot of the platform to identify the cause of the failure.
The event logs and statistics are stored in areas of memory shared between the Hypervisor and
the Guest OSes, making it easy to retrieve this information from the Guest OSes. Virtual device
drivers are provided for Linux Guest OSes to make it even easier.
CPU usage statistics and CPU snaphots may be retrieved by Guest OSes through dedicated
hypercalls. Again, virtual device drivers are provided for Linux Guest OSes.

© 2021 HARMAN. All Rights Reserved [22 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description
The event log mechanism generates traces complying with the CTF (Common Trace Format)
definition, enabling to use tools such as Trace Compass.

9 Hypervisor BSP
The Hypervisor is structured in two pieces:
• The core Hypervisor which is generic and portable. It remains unchanged when adapting the
Hypervisor to a new board. The core Hypervisor is usually delivered in binary form.
• The Hypervisor BSP which interfaces the underlying hardware and the core Hypervisor. It
includes all drivers, initialization routines and configuration files that implement
CPU/SoC/Board/Use-case specific integration logic. The Hypervisor BSP also includes the
Hardware Abstraction Layer (HAL) used by the core Hypervisor.
The Hypervisor BSP is seen by the core Hypervisor as a collection of API. Therefore, developing a
new Hypervisor BSP consists in providing the implementation of these API on a specific board.
The overall configuration of a system is defined by a Device Tree which includes three kinds of
subtrees:
• The Hypervisor Device Tree defines the configuration of the Hypervisor,
• One Device Tree per Virtual Machine, each describing the configuration of one Virtual
Machine such as memory, VCPUs, devices…

6$0681*&RQILGHQWLDO
• A physical platform Device Tree which describes the actual hardware. This Device Tree is
generally provided by the bootloader. The Hypervisor BSP mainly deals with this Device Tree.
The Hypervisor BSP has to provide the set of API expected by the core Hypervisor. However, it can
&2$6,$1(;(//&2/7'PFMLQRDW
uses some services defined by the core Hypervisor. These services are defined by the "Component
Interfaces" chapter of the "Device Virtualization Reference Manual, BSP Edition".
The Hypervisor BSP is composed of the following sets of components:
• Generic software components
• CPU dependent components
• SoC dependent components
• Board dependent components
This structure is designed to re-use the existing software components, and therefore to accelerate
the time needed to port on a new SoC or Board.

9.1 Typical BSP

A typical BSP contains:
• Interrupt controller driver, typically GICv2 or v3 on ARMv8
• SoC vendor specific UART driver
• Clock driver (time count) (Architected Timer on ARMv8)
• Timer driver (interval timer) (Architected Timer on ARMv8)
• SoC vendor specific System MMU driver (optional)
• Power-management driver (PSCI on ARMv8)
• SoC vendor specific second-level (cascaded) interrupt controller (optional)

© 2021 HARMAN. All Rights Reserved [23 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description
• SoC vendor specific early initialization logic
• CPU and MM drivers (ARMv8)

9.2 BSP Source Tree Layout

The BSP source tree layout is as follows:

soc

board

cpu
bsp
framework

6$0681*&RQILGHQWLDO
uart

vdt
&2$6,$1(;(//&2/7'PFMLQRDW
Figure 15: BSP Source Tree Layout

• Soc: contains system on chip software and resources

• Board: board specific resources
• Cpu: architecture specific software
• Framework : generic software components
• UART: generic UART drivers
• vdt: Generic parts of the Hypervisor Device Tree and generic Virtual Platforms Device Trees

10 Initialization
The hardware boot loader loads the Multi-VM boot image2 which includes the Hypervisor image. It
then jumps into the Hypervisor entry-point.
The Hypervisor entry routine (crt0) is written in assembly. It performs low-level initializations
(e.g. sets-up an execution stack) and then continues in C-written initialization code.

2
See the description of the build mechanism further in this document.

© 2021 HARMAN. All Rights Reserved [24 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description
BSP initializes itself, exports the BSP drivers via HAL, and finally invokes the entry point of the core
Hypervisor (aka nano-kernel).
The generic Hypervisor uses the BSP drivers via the HAL.

11 Build Overview
The overall build process is depicted by the following picture.

Figure 16: Build Overview

Core
Hypervisor Core Hypervisor Hypervisor
Hypervisor
Build Bin. Distribution Hypervisor Image
Source Build Hypervisor
Hypervisor BSP Hypervisor BSP Build
Source REDBEND Source

Hypervisor device tree

Configuration Multi-VM
Multi-VM
Boot
Virtual platform device trees Build Configuration
Image

Guest OSes device trees

6$0681*&RQILGHQWLDO
GuestO
GuestOS
GuestO
GuestOS SS
Image
Build(s)
Hypervisor Image and
&2$6,$1(;(//&2/7'PFMLQRDW
Configuration Build Process
Multi-VM Boot Image
Build Process

The core Hypervisor source is usually not delivered. It is used to generate a binary distribution
which is delivered along with the sources of a Hypervisor BSP. Customers may use the delivered
BSP or develop their own BSP.
From the core Hypervisor binary and the Hypervisor BSP source, one builds a Hypervisor image.
The configuration of the Hypervisor and the Virtual Machines (Virtual Platform configuration) are
built separately (Examples of “Virtual platform” device trees and hypervisor device tree are
provided in the hypervisor distribution).
This has to be completed by the binary guest OS images to run within the Virtual Machines. The
Hypervisor image, the configuration file (Device Tree Blob) and the guest OS images are then
packaged into a so-called multi-VM boot image.
The hardware bootloader is in charge of loading such an image into the main memory and to pass
control to the entry point of the Hypervisor.

© 2021 HARMAN. All Rights Reserved [25 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description

•Hypervisor source •Generated headers

•Build infrastructure Transitory •Objects
•Hypervisor headers Build Tree
•Hypervisor BSP source
•Build infrastructure
Hypervisor Build from source •Hypervisor headers
Source Tree •Hypervisor BSP source
•Generated headers
Binary Distribution
BSP source Tree •Objects

BSP Source Tree

Product
Object/Image
build from distribution
Tree
•Generated headers
Vendor-specific
•Hypervisor Image
Hypervisor BSP
Source Tree •Device Tree Blobs

Hypervisor
Image

Figure 17: Detailed Hypervisor Image Generation

6$0681*&RQILGHQWLDO
11.1 Build Tool Chains
Building a Multi-VM boot image is therefore quite complex. Guest OSes images must be generated
&2$6,$1(;(//&2/7'PFMLQRDW
with their own tool chain. Integrators may have specific build requirements, depending upon their
board and the applications they are developing.
All Hypervisor build mechanisms rely on Linux common tools such as:
• GNU make
• GCC x-chain
• dtc
• sh, cp, ln, rm, mkdir,
• Python3
Internally, a set of tools (called mkvlm) perform the various steps of the build process. However, as
the basic build mechanisms rely on well-known tools, it is possible to adapt the tool chain to
specific needs. As an example, Yocto has been used to generate Multi-VM boot images. Other
build chain integration are possible as well.

11.2 Multi-VM Boot Image

The following picture describes the layout of the boot image including the Hypervisor, the
configuration and the guest OSes to be launched in the Virtual Machines. The boot image is
actually built by the build-hbi.py Python script. The boot image is made of the different files
generated by the build tools: Hypervisor Image, configuration files which are Device Tree Blobs
and guest OS images and their initrd, for Linux guest OSes. Headers are inserted prior each
Virtual Machine section. The end of the boot image is marked with an "End of Image" trailer.

© 2021 HARMAN. All Rights Reserved [26 of 27] Confidential

REDBEND User Manual REDBEND Device Virtualization for Connected Vehicles
Hypervisor Description

Multi-VM Boot Image(HBI)

Hypervisor Hypervisor
Hypervisor
Image Image
Hypervisor
Configuration DTB Hypervisor
Configuration DTB
VM vplatform
DTB Header
VM vplatform Hypervisor
GuestOS DTB Configuration
Another Image + Initrd
Block
GuestOS VM vplatform
Storage VM vplatform DTB
DTB
Header
GuestOS GuestOS
GuestOS
Image + Initrd Image + Initrd
VM1 Image
Image + Initrd
GuestOS-specific Build Hypervisor Build
GuestOS
Image + Initrd VM2 Image

6$0681*&RQILGHQWLDO
End of Image

&2$6,$1(;(//&2/7'PFMLQRDW

Figure 18: Multi-VM Boot Image

© 2021 HARMAN. All Rights Reserved [27 of 27] Confidential