Scribd
Upload
403 views

Introduction To Bhyve

Bhyve is a hypervisor for FreeBSD that allows virtual machines to run on the host operating system. It utilizes hardware-assisted virtualization features of Intel VT-x and AMD-V to directly execute guest instructions, while trapping sensitive operations and emulating hardware as needed. Bhyve loads guest operating systems using bhyveload, runs virtual machines using bhyve, and provides APIs to manage virtual machines through libvmmapi and the /dev/vmm device nodes in the kernel. This allows bhyve to efficiently virtualize CPUs, memory, and I/O using hardware features while providing an interface to control virtual machines from userspace.

Uploaded by

wanna_ac
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
403 views

Introduction To Bhyve

Bhyve is a hypervisor for FreeBSD that allows virtual machines to run on the host operating system. It utilizes hardware-assisted virtualization features of Intel VT-x and AMD-V to directly execute guest instructions, while trapping sensitive operations and emulating hardware as needed. Bhyve loads guest operating systems using bhyveload, runs virtual machines using bhyve, and provides APIs to manage virtual machines through libvmmapi and the /dev/vmm device nodes in the kernel. This allows bhyve to efficiently virtualize CPUs, memory, and I/O using hardware features while providing an interface to control virtual machines from userspace.

Uploaded by

wanna_ac
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 34

Introduction to bhyve

Takuya ASADA / @syuu1228

What is bhyve?

What is bhyve?

bhyve is a hypervisor introduced in FreeBSD


Similar to Linux KVM, runs on host OS


BSD License

Developed by Peter Grehan and Neel Natu

bhyve features

Required Intel VT-x and EPT (Nehalem or later)


AMD support in progress

Does not support BIOS/UEFI for now


UEFI support in progress

Minimal device emulation support:


virtio-blk, virtio-net, COM port +

Supported guest OS:


FreeBSD/amd64, i386, Linux/x86_64, OpenBSD/amd64

How to use it?


kldload vmm.ko

/usr/sbin/bhyveload -m ${mem} -d ${disk} ${name}

/usr/sbin/bhyve -c ${cpus} -m ${mem} \
-s 0,hostbridge -s 2,virtio-blk,${disk} \
-s 3,virtio-net,${tap} -s 31,lpc -l com1,stdio vm0

How to run Linux?

bhyve OS Loader(/usr/sbin/bhyveload) only supports


FreeBSD
You need another OS Loader to support other OSs

grub2-bhyve is the solution


Its modified version grub2, runs on host OS (FreeBSD)


Can load Linux and OpenBSD


Available in ports & pkg!

Virtualization in general

Difference between container


and hypervisor

Jail is container

bhyve is hypervisor

Its virtualize OS environment on kernel level


It virtualizes whole machine


Totally different approach

Container
jail1

proc
ess

proc
ess

Process in jail is just a normal


process for the kernel

The kernel do some tricks to


isolate environments between jails

Lightweight, less-overhead

Share one kernel with all jails


If the kernel panics, all jails
will die

You cannot install another OS


(No Windows, No Linux!)

jail2

proc
ess

Kernel

NIC
Disk

Hypervisor
vm1

proc
ess

proc
ess

Kernel

Kernel

NI
C

Hypervisor virtualizes a machine


From guest OS, it looks like real


hardware

Virtual machine is a normal


process for host OS

Does not share kernel, it is


completely isolated

You can run Full OS inside of


the VM Windows! Linux!

vm2

proc
ess

Disk

Disk

Hypervisor

NI
C

Kernel

NIC
Disk

How hypervisor virtualize


machine?

To make complete virtual machine, you need to


virtualize following things:

CPU

Memory (Address Space)


I/O

CPU Virtualization:
Emulate entire CPU?
QEMU

CPU
emula
virtual
device IO tor

run

mov
mov
out

dx,3FBh
al,128
dx,al

OS
physica
l device

physic
al
CPU

Like QEMU

You can emulate the entire CPU operation on a normal process


Very slow, not a really useful choice for virtualization

CPU Virtualization:

Direct execution?

You want run guest instructions directly on a real


CPU since you are virtualizing x86 on x86

You need to avoid executing some instructions which


modify system global state, or perform I/O (called
sensitive instructions)

If you execute these instructions on a real CPU, it


may break host OS state such as directly accessing
a HW device

Perform I/O on VM
GuestOS
Real Display
Virtual CPU

outb

You need to avoid access to real HW from VM


Need to prevent execution of the instruction

Perform IO on VM
Virtual Display

GuestOS

trap!

Virtual CPU

outb

You can trap them by executing in lower privileged mode


However, on x86, there are some instructions which are impossible to trap
because these are nonprivileged instructions

Software techniques to
virtualize x86

Binary translation (old VMware): interpret & modify


guest OSs instructions on-the-fly
Runs fast, but implementation is very complex

Paravirtualization (old Xen): Modify guest OS for


the hypervisor
Runs fast, but is impossible to run unmodified OSs

We want an easier & better solution


HW assisted virtualization!

Hardware assisted
virtualization(Intel VT-x)
VMX
root mode

VMX
non-root
mode

User
(Ring 3)

User
(Ring 3)

Kernel
(Ring 0)

VMEntry
VMExit

Kernel
(Ring 0)

New CPU mode:

VMX root mode (hypervisor) / VMX non-root mode (guest)


If some event needs to emulate in the hypervisor,


CPU stops guest, exit to hypervisor VMExit

You dont need complex software techniques


You dont have to modify the guest OS

Memory Virtualization
Process A
1
Process B
1
2

Process A
1

Guest A
Page table A
1
2

Page table B
1

1
2

1
2

Host physical memory

3
4

1
2
3
4

Guest B
Page table A
1

2
Process B

Guest physical memory

Page table B
1

5
Guest physical memory

6
7

2
3
4

If you run guest OS natively, memory address translation become problematic


If GuestB loads Page table A, virtual page 1 translate to Host physical page 1
but you meant Host physical page 5

Shadow Paging
Page table A'
1

Host physical memory

2
Page table B'

Process A
1

1
2

3
4

Guest A
Page table A
1
2
2

Process B

1
2

Page table B
1

5
6
7
1

2
3
4
Guest physical memory

Trap page table loading/modifying, create Shadow Page Table,


tell physical page number to the MMU

A software trick that works well, but is slow

Nested Paging (Intel EPT)


Host physical memory
Process A
1

Guest A
Page table A
1

2
Process B
1
2

Page table B
1
2

3
4

EPT A
1
2
3
4

2
3
4

6
7
8

Guest physical memory

1
2
3
4
5
6
7
8

HW assisted memory virtualization!


You will have Guest physical : Host physical translation table


MMU translates address by two step (Nested)

I/O Virtualization

To run unmodified OSs, youll need to emulate all


devices what you have on the real hardware

SATA, NIC(e1000), USB(ehci),VGA(Cirrus),


Interrupt controller(LAPIC, IO-APIC),
Clock(HPET), COM port

Emulating real devices is not very fast because it


causes lot of VMExits, not ideal for for virtualization

Paravirtual I/O

Virtual I/O device is designed for VM use


Much faster than emulating real devices


Required device driver on guest OS


De-facto standard: virtio-blk, virtio-net

PCI Device passthrough


Process A
1

Guest A
Pagetable A
1

2
Process B
1
2

Physical memory

Pagetable B
1

EPT A
1
2
3
4
Guest physical
memory

2
3

6
7

1
2
3
4
5
6

IOMMU
translation table
5
1
6
7

2
3

DMA!
PCI
Devices

7
8

If you attach a real HW device on a VM, you will have a problem with DMA

Because the device requires physical address for DMA but the guest OS
doesnt know the Host physical address

Address translator for the devices: IOMMU(Intel VT-d)

Translates guest physical to host physical using a translation table

bhyve internals

How bhyve virtualize machine?

CPU: HW-assisted virtualization (Intel VT-x)


Memory: HW-assisted memory virtualization (Intel


EPT)

IO: virtio, PCI passthrough, +


Uses HW assisted features

bhyve overview
2. Run VM instace
Disk image
tap device
stdin/stdout

1. Create VM instance,
load guest kernel
Guest

bhyveload

H
D

N
I
C

Console

kernel

bhyve

3. Destroy VM
instance
bhyvectl

libvmmapi
mmap/ioctl
/dev/vmm/${vm_name} (vmm.ko)
FreeBSD kernel

bhyveload: loads
guest OS

bhyve: userland part of


Hypervisor, emulates
devices

bhyvectl: a management
tool

libvmmapi: userland API



vmm.ko: kernel part of
Hypervisor

vmm.ko
All VT-x features only accessible in kernel
mode, vmm.ko handles it

Most important work of vmm.ko is CPU

mode switching between hypervisor/guest


Provides interface for userland via /dev/


vmm/${vmname}

Each vmm device file contains each VM


instance state

/dev/vmm/${vmname}
interfaces

create/destroy

Can create/destroy device file via sysctl


hw.vmm.create, hw.vmm.destroy

read/write/mmap

Can access guest memory area by standard


syscall (Which means you even can dump
guest memory by dd command)

ioctl

Provides various operations to VM

/dev/vmm/${vmname}
ioctls
VM_MAP_MEMORY: Maps guest memory
area at requested size

VM_SET/GET_REGISTER: Access registers



VM_RUN: Run guest machine, until virtual
devices accessed (or some other trap
happened)

libvmmapi

wrapper library of /dev/vmm operations


vm_create(name) sysctl(hw.vmm.create, name)


vm_set_register(reg, val)
ioctl(VM_SET_REGISTER, reg, val)

bhyveload

bhyve uses OS loader instead of BIOS/UEFI, to load guest OS



FreeBSD bootloader ported to userland: userboot

bhyveload runs host OS, to initialize guest OS

Once it called, it does following things:

Parse UFS on diskimage, find kernel



Load kernel to guest memory area

Initialize Page Table

Create GDT, IDT, LDT

Initialize special registers to get ready for 64bit mode

Guest machine can starts from kernel entry point, with 64bit mode

bhyve

bhyve command is the userland part of the hypervisor


It invokes ioctl(VM_RUN) to run GuestOS


Emulates virtual devices


Provides user interface(no GUI for now)

main loop in bhyve


while (1) {

ioctl(VM_RUN, &vmexit);

switch (vmexit.exit_code) {

case IOPORT_ACCESS:


emulate_device(vmexit.ioport);

}

}

Q&A?

You might also like