History of Virtualization:

Virtualization was first developed in the 1960s to partition large, mainframe hardware for better hardware
utilization. Today, computers based on x86 architecture are faced with the same problems of rigidity and
underutilization that mainframes faced in the 1960s. VMware invented virtualization for the x86 platform
in the 1990s to address underutilization and other issues, overcoming many challenges in the process.
Today, VMware is the global leader in x86 virtualization, with over 300,000 customers, including 100%
of the Fortune 100.
The term virtualization has many meanings, and aspects of virtualization
permeate all aspects of computing. Virtual machines are one instance of this
trend. Generally, with a virtual machine, guest operating systems and applications
run in an environment that appears to them to be native hardware and
that behaves toward them as native hardware would but that also protects,
manages, and limits them.

In the Beginning: Mainframe Virtualization

Virtualization was first implemented more than 30 years ago by IBM as a way to logically partition
mainframe computers into separate virtual machines. These partitions allowed mainframes to
“multitask”: run multiple applications and processes at the same time. Since mainframes were expensive
resources at the time, they were designed for partitioning as a way to fully leverage the investment.

The Need for x86 Virtualization

Virtualization was effectively abandoned during the 1980s and 1990s when client-server applications and
inexpensive x86 servers and desktops led to distributed computing. The broad adoption of Windows and
the emergence of Linux as server operating systems in the 1990s established x86 servers as the industry
standard. The growth in x86 server and desktop deployments led to new IT infrastructure and operational
challenges. These challenges include:
•Low Infrastructure Utilization. Typical x86 server deployments achieve an average utilization of only
10% to 15% of total capacity, according to International Data Corporation (IDC), a market research firm.
Organizations typically run one application per server to avoid the risk of vulnerabilities in one
application affecting the availability of another application on the same server.
•Increasing Physical Infrastructure Costs. The operational costs to support growing physical
infrastructure have steadily increased. Most computing infrastructure must remain operational at all
times, resulting in power consumption, cooling and facilities costs that do not vary with utilization levels.
•Increasing IT Management Costs. As computing environments become more complex, the level of
specialized education and experience required for infrastructure management personnel and the
associated costs of such personnel have increased. Organizations spend disproportionate time and
resources on manual tasks associated with server maintenance, and thus require more personnel to
complete these tasks.
•Insufficient Failover and Disaster Protection. Organizations are increasingly affected by the downtime
of critical server applications and inaccessibility of critical end user desktops. The threat of security
attacks, natural disasters, health pandemics and terrorism has elevated the importance of business
continuity planning for both desktops and servers.
•High Maintenance end-user desktops. Managing and securing enterprise desktops present numerous
challenges. Controlling a distributed desktop environment and enforcing management, access and
security policies without impairing users’ ability to work effectively is complex and expensive.
Numerous patches and upgrades must be continually applied to desktop environments to eliminate
security vulnerabilities.

The VMware Solution: Full Virtualization of x86 Hardware

In 1999, VMware introduced virtualization to x86 systems to address many of these challenges and
transform x86 systems into a general purpose, shared hardware infrastructure that offers full isolation,
mobility and operating system choice for application environments.

Challenges & Obstacles to x86 Virtualization

Unlike mainframes, x86 machines were not designed to support full virtualization, and VMware had to
overcome formidable challenges to create virtual machines out of x86 computers.

The basic function of most CPUs, both in mainframes and in PCs, is to execute a sequence of stored
instructions (ie, a software program). In x86 processors, there are 17 specific instructions that create
problems when virtualized, causing the operating system to display a warning, terminate the application,
or simply crash altogether. As a result, these 17 instructions were a significant obstacle to the initial
implementation of virtualization on x86 computers.

To handle the problematic instructions in the x86 architecture, VMware developed an adaptive
virtualization technique that “traps” these instructions as they are generated and converts them into safe
instructions that can be virtualized, while allowing all other instructions to be executed without
intervention. The result is a high-performance virtual machine that matches the host hardware and
maintains total software compatibility. VMware pioneered this technique and is today the undisputed
leader in virtualization technology.

What is Virtualization?

Today’s x86 computer hardware was designed to run a single operating system and a single application,
leaving most machines vastly underutilized. Virtualization lets you run multiple virtual machines on a
single physical machine, with each virtual machine sharing the resources of that one physical computer
across multiple environments. Different virtual machines can run different operating systems and
multiple applications on the same physical computer. While others are leaping aboard the virtualization
bandwagon now, VMware is the market leader in virtualization. Our technology is production-proven,
used by more than 170,000 customers, including 100% of the Fortune 100.
Improve the efficiency and availability of IT resources and applications through virtualization. Start by
eliminating the old “one server, one application” model and run multiple virtual machines on each
physical machine. Free your IT admins from spending so much time managing servers rather than
innovating. About 70% of a typical IT budget in a non-virtualized datacenter goes towards just
maintaining the existing infrastructure, with little left for innovation.
An automated datacenter built on the production-proven VMware virtualization platform lets you
respond to market dynamics faster and more efficiently than ever before. VMware vSphere delivers
resources, applications—even servers—when and where they’re needed. VMware customers typically
save 50-70% on overall IT costs by consolidating their resource pools and delivering highly available
machines with VMware vSphere.
•Run multiple operating systems on a single computer including Windows, Linux and more.
•Let your Mac run Windows creating a virtual PC environment for all your Windows applications.
•Reduce capital costs by increasing energy efficiency and requiring less hardware while increasing your
server to admin ratio
•Ensure your enterprise applications perform with the highest availability and performance
•Build up business continuity through improved disaster recovery solutions and deliver high availability
throughout the datacenter
•Improve enterprise desktop management & control with faster deployment of desktops and fewer
support calls due to application conflicts
Virtualization and Operating-System Components

Thus far, we have explored the building blocks of virtualization and the various types of virtualization.
In this section, we take a deeper dive into the operating-system aspects of virtualization, including how
the VMM provides core operating-system functions like scheduling, I/O, and memory management.
Here, we answer questions such as these: How do VMMs schedule CPU use when guest operating
systems believe they have dedicated CPUs? How can memory management work when many guests
require large amounts of
memory?

Process and system VMs

To understand what a virtual machine is, it is first necessary to consider the meaning of “machine”
from both a process and system perspective.
From the perspective of a process executing a user program, the machine consists of a logical memory
address space assigned to the process along with user-level instructions and registers that allow the
execution of code belonging to the process. The machine’s I/O is visible only through the operating
system, and the only way the process can interact with the I/O system is through operating system
calls. Thus the ABI defines the machine as seen by a process. Similarly, the API specifies the machine
characteristics as seen by an application’s HLL program.
From the perspective of the operating system and the applications it supports, the entire system runs on
an underlying machine. A system is a full execution environment that can support numerous processes
simultaneously. These processes share a file system and other I/O resources. The system environment
persists over time as processes come and go. The system allocates real memory and I/O resources to the
processes, and allows the processes to interact with their resources. From the system perspective,
therefore,the underlying hardware’s characteristics alone define the machine; it is the ISA that provides
the interface between the system and machine.

Just as there are process and system perspectives of “machine,” there are process and system virtual
machines. A process VM is a virtual platform that executes an individual process. This type of VM
exists solely to support the process; it is created when the process is created and terminates when
the process terminates. In contrast, a system VM provides a complete, persistent system environment
that supports an operating system along with its many user processes. It provides the guest operating
system with access to virtual hardware resources, including networking, I/O, and perhaps
a graphical user interface along with a processor and memory.
The process or system that runs on a VM is the guest, while the underlying platform that supports
the VM is the host. The virtualizing software that implements a process VM is often termed the runtime,
short for “runtime software.” The virtualizing software in a system VM is typically referred
to as the virtual machine monitor (VMM).
Figure 3 depicts process and system VMs, with compatible interfaces illustrated graphically as
meshing boundaries. In a process VM, the virtualizing
software is at the ABI or API level, atop the OS/HW combination. The runtime emulates both
user-level instructions and either operating system or library calls. In a system VM, the virtualizing
software is between the host hardware machine and the guest software. The VMM emulates the
hardware ISA so that the guest software can potentially execute a different ISA from the one
implemented on the host. However, in many system VM applications, the VMM does not perform
instruction emulation; rather, its primary role is to provide virtualized hardware resources.
PROCESS VIRTUAL MACHINE
In essence, a process VM presents an ABI to an application process, translates a set of OS and user-level
instructions composing one platform to those of another ( Figure 2.14a ). A process VM is a virtual
platform for executing a single process. As such, the process VM is created when the process is
created and terminated when the process is terminated.
In order to provide cross-platform portability, a common implementation of the process VM architecture
is as part of an overall HLL application environment.
The resulting ABI does not correspond to any specific machine. Instead, the ABI specification is
designed to easily support a given HLL or set of HLLs and to be easily portable to a variety of ISAs. The
HLL VM includes a front-end compiler that generates a virtual binary code for execution or
interpretation. This code can then be executed on any machine that has the process VM implemented.
Two widely used examples of this approach are the Java VM architecture and
the Microsoft Common Language Infrastructure, which is the foundation of the .NET framework.
CPU Scheduling

A system with virtualization, even a single-CPU system, frequently acts like a multiprocessor system.
The virtualization software presents one or more virtual CPUs to each of the virtual machines running on
the system and then schedules the use of the physical CPUs among the virtual machines.
The significant variations among virtualization technologies make it difficult to summarize the effect of
virtualization on scheduling. First, let’s consider the general case of VMM scheduling. The VMM has a
number of physical CPUs available and a number of threads to run on those CPUs. The threads can be
VMM threads or guest threads. Guests are configured with a certain number of virtual CPUs at creation
time, and that number can be adjusted throughout the life of the VM. When there are enough CPUs to
allocate the requested number to each guest, the VMM can treat the CPUs as dedicated and schedule only
a given guest’s threads on that guest’s CPUs. In this situation, the guests act much like native operating
systems running on native CPUs.
Of course, in other situations, there may not be enough CPUs to go around. The VMM itself needs some
CPU cycles for guest management and I/O management and can steal cycles from the guests by
scheduling its threads across all of the system CPUs, but the impact of this action is relatively minor.
More difficult is the case of overcommitment, in which the guests are configured for more CPUs than
exist in the system. Here, a VMM can use standard scheduling algorithms to make progress on each
thread but can also add a fairness aspect to
those algorithms. For example, if there are six hardware CPUs and twelve guest allocated
CPUs, the VMM can allocate CPU resources proportionally, giving each
guest half of the CPU resources it believes it has. The VMM can still present all
twelve virtual CPUs to the guests, but in mapping them onto physical CPUs, the
VMM can use its scheduler to distribute them appropriately.
Even given a scheduler that provides fairness, any guest operating-system
scheduling algorithm that assumes a certain amount of progress in a given
amount of time will most likely be negatively affected by virtualization. Consider
a time-sharing operating system that tries to allot 100 milliseconds to each
time slice to give users a reasonable response time. Within a virtual machine,
this operating system receives only what CPU resources the virtualization system
gives it. A 100-millisecond time slice may take much more than 100 milliseconds
of virtual CPU time. Depending on how busy the system is, the time
slice may take a second or more, resulting in very poor response times for users
logged into that virtual machine. The effect on a real-time operating system can
be even more serious.
The net outcome of such scheduling is that individual virtualized operating
systems receive only a portion of the available CPU cycles, even though they
believe they are receiving all of the cycles and indeed are scheduling all of
the cycles. Commonly, the time-of-day clocks in virtual machines are incorrect
because timers take longer to trigger than they would on dedicated CPUs.
Virtualization can thus undo the scheduling-algorithm efforts of the operating
systems within virtual machines.
To correct for this, the VMM makes an application available for each type of
operating system that the system administrator can install into the guests. This
application corrects clock drift and can have other functions, such as virtual
device management.

Memory Management

Efficient memory use in general-purpose operating systems is a major key to

performance. In virtualized environments, there aremore users of memory (the
guests and their applications, as well as the VMM), leading to more pressure on
memory use. Further adding to this pressure is the fact that VMMs typically
overcommit memory, so that the total memory allocated to guests exceeds
the amount that physically exists in the system. The extra need for efficient
memory use is not lost on the implementers of VMMs, who take extensive
measures to ensure the optimal use of memory.
For example, VMware ESX uses several methods of memory management.
Before memory optimization can occur, the VMM must establish how much
real memory each guest should use. To do that, the VMM first evaluates each
guest’s maximum memory size. General-purpose operating systems do not
expect the amount of memory in the system to change, so VMMs must maintain
the illusion that the guest has that amount ofmemory. Next, theVMMcomputes
a target real-memory allocation for each guest based on the configured memory
for that guest and other factors, such as overcommitment and system load. It
then uses the three low-level mechanisms listed below to reclaimmemory from
the guests
1. Recall that a guest believes it controls memory allocation via its pagetable
management, whereas in reality the VMM maintains a nested page
table that translates the guest page table to the real page table. The VMM
can use this extra level of indirection to optimize the guest’s use of memory
without the guest’s knowledge or help. One approach is to provide
double paging. Here, the VMM has its own page-replacement algorithms
and loads pages into a backing store that the guest believes is physical
memory.Of course, the VMM knows less about the guest’s memory access
patterns than the guest does, so its paging is less efficient, creating performance
problems. VMMs do use this method when other methods are
not available or are not providing enough free memory.However, it is not
the preferred approach.
2. A common solution is for the VMM to install in each guest a pseudo–
device driver or kernel module that the VMM controls. (A pseudo–device
driver uses device-driver interfaces, appearing to the kernel to be a device
driver, but does not actually control a device. Rather, it is an easy way
to add kernel-mode code without directly modifying the kernel.) This
balloon memory manager communicates with the VMM and is told to
allocate or deallocate memory. If told to allocate, it allocates memory
and tells the operating system to pin the allocated pages into physical
memory. Recall that pinning locks a page into physical memory so that it
cannot be moved or paged out. To the guest, these pinned pages appear
to decrease the amount of physical memory it has available, creating
memory pressure. The guest then may free up other physical memory
to be sure it has enough free memory. Meanwhile, the VMM, knowing
that the pages pinned by the balloon process will never be used, removes
those physical pages from the guest and allocates them to another guest.
At the same time, the guest is using its own memory-management and
paging algorithms to manage the available memory, which is the most efficient option. If memory
pressure within the entire system decreases,
the VMM will tell the balloon process within the guest to unpin and free
some or all of the memory, allowing the guest more pages for its use.
3. Another common method for reducing memory pressure is for the VMM
to determine if the same page has been loaded more than once. If this
is the case, the VMM reduces the number of copies of the page to one
and maps the other users of the page to that one copy. VMware, for
example, randomly samples guest memory and creates a hash for each
page sampled. That hash value is a “thumbprint” of the page. The hash
of every page examined is compared with other hashes stored in a hash
table. If there isamatch, the pages are comparedbyte bybyte to see if they
really are identical. If they are, one page is freed, and its logical address
is mapped to the other’s physical address. This technique might seem
at first to be ineffective, but consider that guests run operating systems.
If multiple guests run the same operating system, then only one copy of
the active operating-system pages need be in memory. Similarly, multiple
guests could be running the same set of applications, again a likely source
of memory sharing.
The overall effect of these mechanisms is to enable guests to behave and
perform as if they had the full amount of memory requested, although in reality
they have less.

I/O

In the area of I/O, hypervisors have some leeway and can be less concerned
with how they represent the underlying hardware to their guests. Because
of the wide variation in I/O devices, operating systems are used to dealing
with varying and flexible I/O mechanisms. For example, an operating system’s
device-driver mechanism provides a uniform interface to the operating system
whatever the I/O device. Device-driver interfaces are designed to allow
third-party hardware manufacturers to provide device drivers connecting their
devices to the operating system. Usually, device drivers can be dynamically
loaded and unloaded. Virtualization takes advantage of this built-in flexibility
by providing specific virtualized devices to guest operating systems.
As described in Section 18.5, VMMs vary greatly in how they provide I/O
to their guests. I/O devices may be dedicated to guests, for example, or the
VMM may have device drivers onto which it maps guest I/O. The VMM may
also provide idealized device drivers to guests. In this case, the guest sees an
easy-to-control device, but in reality that simple device driver communicates to
the VMM, which sends the requests to a more complicated real device through
a more complex real device driver. I/O in virtual environments is complicated
and requires careful VMM design and implementation.
Consider the case of a hypervisor and hardware combination that allows
devices to be dedicated to a guest and allows the guest to access those devices
directly. Of course, a device dedicated to one guest is not available to any
other guests, but this direct access can still be useful in some circumstances.
The reason to allow direct access is to improve I/O performance. The less the
hypervisor has to do to enable I/O for its guests, the faster the I/O can occur.
With type 0 hypervisors that provide direct device access, guests can often run at the same speed as
native operating systems. When type 0 hypervisors
instead provide shared devices, performance may suffer.
With direct device access in type 1 and 2 hypervisors, performance can
be similar to that of native operating systems if certain hardware support
is present. The hardware needs to provide DMA pass-through with facilities
like VT-d, as well as direct interrupt delivery (interrupts going directly to the
guests). Given how frequently interrupts occur, it should be no surprise that
the guests on hardware without these features have worse performance than
if they were running natively.
In addition to direct access, VMMs provide shared access to devices. Consider
a disk drive towhich multiple guests have access. The VMM must provide
protection while the device is being shared, assuring that a guest can access
only the blocks specified in the guest’s configuration. In such instances, the
VMM must be part of every I/O, checking it for correctness as well as routing
the data to and from the appropriate devices and guests.
In the area of networking, VMMs also have work to do. General-purpose
operating systems typically have one Internet protocol (IP) address, although
they sometimes have more than one—for example, to connect to a management
network, backup network, and production network.With virtualization,
each guest needs at least one IP address, because that is the guest’s main mode
of communication. Therefore, a server running a VMM may have dozens of
addresses, and the VMM acts as a virtual switch to route the network packets
to the addressed guests.
The guests can be “directly” connected to the network by an IP address
that is seen by the broader network (this is known as bridging). Alternatively,
the VMM can provide a network address translation (NAT) address. The NAT
address is local to the server on which the guest is running, and the VMM
provides routing between the broader network and the guest. The VMM also
provides firewalling to guard connections between guests within the system
and between guests and external systems.

Storage Management

An important question in determining how virtualization works is this: If

multiple operating systems have been installed, what and where is the boot
disk? Clearly, virtualized environments need to approach storagemanagement
differently than do native operating systems. Even the standard multiboot
method of slicing the boot disk into partitions, installing a boot manager in one
partition, and installing each other operating system in another partition is not
sufficient, because partitioning has limits that would prevent it from working
for tens or hundreds of virtual machines.
Once again, the solution to this problemdepends on the type of hypervisor.
Type 0 hypervisors often allow root disk partitioning, partly because these
systems tend to run fewer guests than other systems. Alternatively, a disk
manager may be part of the control partition, and that disk manager may
provide disk space (including boot disks) to the other partitions.
Type 1 hypervisors store the guest root disk (and configuration information)
in one or more files in the file systems provided by the VMM. Type 2
hypervisors store the same information in the host operating system’s file systems.
In essence, a disk image, containing all of the contents of the root disk of the guest, is contained in one
file in the VMM. Aside from the potential performance
problems that causes, this is a clever solution, because it simplifies
copying and moving guests. If the administratorwants a duplicate of the guest
(for testing, for example), she simply copies the associated disk image of the
guest and tells the VMM about the new copy. Booting the new virtual machine
brings up an identical guest. Moving a virtual machine from one system to
another that runs the same VMM is as simple as halting the guest, copying the
image to the other system, and starting the guest there.
Guests sometimes need more disk space than is available in their root
disk image. For example, a nonvirtualized database server might use several
file systems spread across many disks to store various parts of the database.
Virtualizing such a database usually involves creating several files and having
the VMM present those to the guest as disks. The guest then executes as usual,
with the VMM translating the disk I/O requests coming from the guest into file
I/O commands to the correct files.
Frequently, VMMs provide a mechanism to capture a physical system as
it is currently configured and convert it to a guest that the VMM can manage
and run. This physical-to-virtual (P-to-V) conversion reads the disk blocks of
the physical system’s disks and stores them in files on the VMM’s system or
on shared storage that the VMM can access. VMMs also provide a virtual-tophysical
(V-to-P) procedure for converting a guest to a physical system. This
procedure is sometimes needed for debugging: a problem could be caused by
the VMM or associated components, and the administrator could attempt to
solve the problem by removing virtualization fromthe problem variables.V-to-
P conversion can take the files containing all of the guest data and generate disk
blocks on a physical disk, recreating the guest as a native operating system and
applications. Once the testing is concluded, the original system can be reused
for other purposes when the virtual machine returns to service, or the virtual
machine can be deleted and the original system can continue to run.