White Paper

IOS XE Virtual Service Development Guide

The Cisco Enterprise routing portfolio consists of multiple products targeted at

different segments of the network. Even though these appear to be very different
products at first glance, they all share a common architecture and operating system.
Cisco IOS XE is the common architecture unifying the 4000 Series Integrated
Services Routers, 1000 Series Aggregation Services Routers and the 1000 Series
Cloud Services Router.

All of these network elements share a common architecture with a platform specific
data plane and a common control plane. At their core these are based on an open-
source Linux architecture. One common aspect to Linux environments is their ability to
host both virtual machines (KVM) and Linux Containers (LXC) guest applications in a
protected environment. This capability has always been available for Cisco developed
applications in IOS XE
platforms. These
applications were required
to carry a digital signature
from Cisco identifying them
as genuine before they
could be installed.

That all changes after IOS

XE release 3.17 in
November 2015. With that
release Cisco introduces a
configuration option

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 1 of 26
 allowing customers to bypass the required digital signature when installing new KVM
applications1.

IOS-XE Virtual Service Architecture

IOS-XE routing platforms all share a common software architecture. This means that
the architecture and code base is the same for the 4000 Series ISR in the branch,
1000 Series ASR at the headend, and the Cloud Services Router in the cloud.

Features and applications are consistent across all three.

Where these platforms differ is in the Data Plane. Platforms implement IOS-XE Data
Plane software in the way that makes the most sense for their deployment. An ASR1K
will use a Quantum Flow Processor ASIC, an ISR4K implements the same
functionality in an off-the-shelf CPU, and a CSR uses multiple threads in a virtual
environment. Packet forwarding and features are consistent across the portfolio, but
isn’t important for virtual service hosting.
When it comes to virtual service hosting, all IOS-XE platforms share exactly the same
architecture. IOS-XE runs a Linux environment outside of the Data Plane. Within this
environment reside the control plane processes as well as all other processes
necessary for the proper functioning of the box. However, even in the busiest router, it
is very rare that the control plane of the system is very active at all. There might be
brief periods of activity, such as initially building a large routing table, but for the most
part the control plane is idle while the data plane does the work of moving packets
through the system.

That leaves a significant amount of CPU time available to do other things. Since IOS-
XE is a Linux environment, hosting applications in containers or virtual machines is a
very straightforward concept. Using standard open-source tools like vman, libvirt and

1
At the time of this writing, only KVM applications are allowed without a Cisco digital signature. In the future
there may be options for LXC or even Docker container support based on customer interest.

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 2 of 26
 qemu, IOS-XE provides a hosting environment familiar to developers in the Linux
world.

APPLICATION H OSTING COMPARISON

Today there exists a spectrum of choices in the application hosting world. These
range from very tight coupling and dependencies between the guest application and
the host and very loose coupling in the form of virtualization. Many of these are
options in networking platforms today including from Cisco.

 Native Process: This option consists of running an application natively within

the Linux operating system. Applications can be added with standard package
management tools like RPM and YUM. While some platforms do support this
option, the potential security concerns and the requirements for the application
developer to maintain compatibility with any IOS-XE changes make this
unattractive. It is not an option on IOS-XE platforms.

 Linux Containers (LXC): LXC is a lightweight virtualization technology that

provides some separation and protection between the guest application and the
host system. However, common resources like the kernel and core
components are shared between the two. This means that component
compatibility must always be maintained between the two. This requirement
makes LXC unattractive for application developers even though it does offer
some performance advantages over other virtualization options. LXC hosting is
available to signed Cisco applications on IOS-XE platforms, but is not available
to non-Cisco applications.

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 3 of 26
  Docker: Docker is an emerging virtualization technology which solves some of
the compatibility issues associated with LXC. While Docker containers are not
currently supported in IOS-XE, it is being looked at as a possible addition if
there is sufficient interest.

 Kernel Virtual Machine (KVM): KVM is the standard virtualization technology

within Linux. Guest applications maintain their own kernel and completely
separate resources from the host. This separation provides additional levels of
security while increasing flexibility for application developers. This is the only
hosting option available to non-Cisco applications on IOS-XE platforms.

 VMWare: Technically this category could include other commercial hypervisors

such as HyperV or Xen. These are generally heavier weight and targeted at
data center applications. These are supported on server blades such as the
UCS E-Series, but are generally too heavyweight for hosting applications within
IOS-XE platforms. Support for Linux hosts is generally lacking while some also
include licensing fees.

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 4 of 26
 Architectural Overview

IOS-XE VIRTUALIZATION COMPONENTS

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 5 of 26
 Table 1. Platform Capabilities

Platform Intel X86 CPU for System Memory for

Processor KVM Storage for KVM2
Memory KVM
Intel Gladden 3 cores NIM-SSD(200GB, 400GB),
ISR4451 4-16GB 0-12GB
4 core 2GHz (equivalent) NIM-HD(500GB, 1TB)
Intel Gladden 3 cores NIM-SSD(200GB, 400GB),
ISR4431 4 core 1GHz (equivalent) 4-16GB 0-12GB
NIM-HD(500GB, 1TB)

ISR 4351 Intel Rangeley 3 cores

8 core 2.4GHz (equivalent) 4-16GB 0-12GB MSATA(50GB, 200GB), NIM-SSD, NIM-HD

ISR 4331 Intel Rangeley 3 cores

8 core 2.0GHz (equivalent) 4-16GB 0-12GB MSATA(50GB, 200GB), NIM-SSD, NIM-HD

Intel Rangeley 1 core

ISR 4321 4 core 2.4GHz (equivalent) 4-12GB 0-8GB MSATA(50GB, 200GB), NIM-SSD, NIM-HD

ASR 1001-X
ASR 1002-X
ASR 1000 RP2
CSR 1000v Various

2
NIM-SSD, NIM-HD and MSATA capacities subject to change. Refer to Cisco.com for the latest options.

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 6 of 26
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 7 of 26
 Building Your First Service Container Application
Building applications is hard. Really hard.

Fortunately there’s almost no additional work to building a Service Container

application beyond what it takes to build the application itself. If you’re taking
advantage of open-source or previously developed applications built for other KVM
deployments building a Service Container application is a simple matter of packaging.

The industry standard package for a virtual machine is known as an Open Virtual
Appliance or OVA. In reality OVA is a packaging standard that says nothing about the
contents of the package. For those familiar with common Unix tools, an OVA file is
actually a TAR archive with all of the files necessary to deploy a virtual machine on a
target hypervisor.

Because hypervisors vary greatly, the contents of an OVA file need to be customized
to provide exactly what the hypervisor requires. An OVA designed for a VMWare
system is going to include different contents than an OVA intended for HyperV or Xen.
Even within KVM the requirements for an OVA package can vary from one system to
another.
All OVA files will have some common requirements. At a minimum they will need a
binary file or files for any disks, hard drives or CD/DVD drives, available to the virtual
machine. The OVA will also need to include a description of what the virtual machine
hardware configuration looks like. This is often an XML or formatted text file that
describes the number of CPUs, amount of DRAM and interfaces such as Ethernet or
serial ports available to the virtual machine.

Cisco Service Container OVA packages are relatively simple. There are only a few
required files listed in the table below. Most of these are simple text files that can be
created in a few minutes. The most complex file is the virtual disk image. This
represents the binary read-only (ISO) or read-write (QCOW2 or RAW) disk image for
the virtual machine.
For reference, this OVA package is the same format used for Cisco IOX applications.
To create the actual OVA file, simply use the Linux tar application, or a compatible
packaging program on other operating systems, to generate a TAR archive with the
.ova file name extension.
CRITICAL OVA CONTENTS

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 8 of 26
 Category Description Usage Origin

Virtual Machine definition Used by Virtualization Manager to Provided by s/w developer for
package.yaml
defined in YAML format. provision the virtual service virtual service.

Manifest file that contains Used by Virtualization Manager to Automatically generated by

*.mf SHA1 hash for each file in the verify the integrity of the files in script or created using tools
OVA OVA such as openssl.

Simplified compatibility check Used by Virtualization Manager to

Simple text file provided by s/w
*.ver with Virtualization perform simple compatibility
developer.
Infrastructure check.

Used to package pre-installed

images or pre-allocated empty
*.img HDD image files (qcow2, raw) Provided by s/w developer.
storage for usage by virtual
machine.

Used to pass CDROM images or

*.ISO ISO image files Provided by s/w developer.
root file systems

VIRTUAL SERVICE DEFINITION FILE

When a virtual service is installed into an IOS XE system, the host needs to know
exactly what the service requires. In the KVM world, this is typically provided by a
specially formatted XML file which passes parameters to the libvirt process. This file
describes things such as CPU, memory and storage requirements, network interfaces,
pointers to disk images and any serial console options. The libvirt process is
extremely sensitive to the formatting of this XML file and is constantly evolving the
capabilities of this file making the XML file very specific to the version of libvirt used
and very sensitive to typos in the XML formatting. This is great for the open source
community, but it’s lousy for predictibality when writing applications for wide
distribution in network devices.

For that reason, Cisco developed a YAML formated definition file. YAML is a human-
friendly standard format that is significantly easier to deal with than libvirt format XML.
Behind the scenes, the IOS XE processes responsible for installing an application are
actually parsing this YAML file into a libvirt XML file taking care of all of the formatting
and context specific to the version used in the system. The YAML format used for IOS
XE virtual services is identical to that used for IOX applications as well as future open
virtualization services from Cisco.

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 9 of 26
 YAML Virtual Service Definition File

manifest-version: <manifest version>

info:
name: <application name>
description: <application description string>
version: <application version>
author-name: <application author/vendor>
author-link: <application author/vendor
website>

app:
apptype: <vm/app type>

resources:
cpu: <cpu share %>
memory: <memory in megs>
vcpu: <no. of vcpus>

disk:
- target dev: <disk name>
file: <image name>
upgrade-model: <ha-sync | local>
share-model: <core>
capacity: <disk capacity in megs>
- ...

interfaces:
- target-dev: <interface name>
type: <management>
- ...

serial:
- serial
- console
- syslog
- tracelog

startup:
runtime: <kvm>
boot-dev: <boot device>

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 10 of 26
 Description of Fields in the YAML Virtual Service Definition File

manifest-version: The version of the descriptor file which may be different from the
version specified in the package version file.

info section
This section contains the fields that define the "application" running in the virtual
service. The application a is program or group of programs that are designed for the
end user and runs within the virtual service. Current supported version is "1.0".
name: Application name
version: Application version
description: A string describing the application
author-name: Application author name
author-link: Application author website for reference

app section
apptype: Only 'vm' is supported.

resources section
This section contains all the fields that describe the virtual service. Use "show virtual-
service" in IOS-XE exec mode to determine the current resource quota on the system.

memory: Amount of RAM in KB allocated to the virtual service. The sum of all
memory of active virtual services cannot exceed the amount specified in the
quota.

vcpu: Number of vcpus assigned to the virtual service. This field is only valid
for KVM. Defaults to 1 if not specified.

cpu: Percent of system CPU share assigned to the virtual service (minimum
guarantee). The sum of all cpu shares of active virtual services cannot exceed
the amount specified in the quota.

disk section

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 11 of 26
 This section is used to specify the disk resources for the virtual service. Up to three
disk devices can be specified. For each disk, either a disk image file or the capacity
has to be specified.
target-dev: Name used for the disk device.
file: Disk image file could either be iso, qcow2 or raw format.
capacity: Size of disk to allocate in MB.

upgrade-model: ha-sync - Sync this disk with the standby route processors.
This is not supported for iso or boot devices.
local - Does not get synced to the standby route processor.
(defaults to local if upgrade model not specified)

interface section
Two type of interfaces are supported, virtualPortGroup and the management interface.
The virtualPortGroup interface will act as the default gateway to the guest's interface.
The management interface will need to be in the same subnet as the guest interface,
but will not act as the gateway. The order in which these interfaces are defined is also
maintained in the guest.
target-dev: Name used for interface device.
alias: Alias name of interface device (optional).

type: The Interface type is set to 'management' to specify that management

interface. If not specified, the virtualPortGroup will be used (optional).

serial section
console: Guest ttyS0 will be the console (optional).
aux: Guest ttyS1 will be the aux port (optional).
syslog: Guest ttyS2 will be used for syslog events (optional).
logger: Guest ttyS3 will be used for debug log events (optional).

The guest tty serial port numbers will always remain in sequential order,
regardless of which serial port is configured in the YAML descriptor file.

startup section

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 12 of 26
 runtime: Only 'kvm' is currently supported.

boot-dev: Reference to the disk boot device. Can be 'cdrom' or 'hd'. Only one
boot device is currently allowed.
Example YAML Virtual Service Definition Files

Bootable harddisk image in OVA, one Virtual Port Group interface and one
management interface:

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 13 of 26
 # KVM Linux test distribution descriptor file
manifest-version: 1.0

info:
name: kvm_linux
description: "KVM Linux Test Distro"
version: 1.0
author-name: Cisco Systems, Inc.
author-link: "http://www.cisco.com"

app:
apptype: vm

resources:
cpu: 6
memory: 262144
vcpu: 1

disk:
- target-dev: hda
file: linux.img

interfaces:
- target-dev: net1
- target-dev: net2
type: management

serial:
- console
- aux
- syslog
- tracelog

startup:
runtime: kvm
boot-dev: hd

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 14 of 26
 Bootable iso image and harddisk image in OVA, carve additional 2GB harddisk, two
VirtualPortGroup interfaces:
manifest-version: 1.0

info:
name: kvm_linux_2
description: "KVM Linux Test Distro"
version: 1.0
author-name: Cisco Systems, Inc.
author-link: "http://www.cisco.com"

app:
# Indicate app type
apptype: vm

resources:
cpu: 6
memory: 262144
vcpu: 1

disk:
- target dev: hdc
file: linux.iso
- target dev: sda
file: kvm_storage_4000MB.img
upgrade-model: ha-sync
- target dev: sdb
capacity: 2000

interfaces:
- target-dev: net1
- target-dev: net2

serial:
- serial
- console
- syslog
- tracelog

# Specify runtime and startup

startup:
runtime: kvm
boot-dev: cdrom

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 15 of 26
 DEVELOPER WORKFLOW
Working with Virtual Services is almost identical to working with KVM virtual
machines. There are just a few additional components needed in the packaging
process. The development workflow and common development tools used for KVM
can still be used when developing, or modifying existing applications, for the Virtual
Service Environment.

Several open source tools exist that make the final packaging process easier. If this is
your first time developing a KVM application, these will come in very handy. If you
have experience working in the VMWare world, some of these concepts will be
familiar but different when moving to a Linux environment. There is no requirement to
use these tools when developing Virtual Services for IOS XE platforms. They’re
mentioned here simply to give new developers a head start.

Tool Uses Example

virt-manager GUI Linux VM Manager for crafting GUI
KVMs.
qemu-img Converting disk image formats. qemu-img convert -p -c -f raw -O qcow2
<raw.img> <qcow2.img>
openssl Generates the manifest file. qemu-img convert -p -c -f raw -O qcow2
<raw.img> <qcow2.img>
tar Packages the files into an OVA. tar -cvf VM.ova vm.qcow2 *.yaml vm.mf
create_ova.sh Cisco script for packing an OVA in one create_ova.sh [<options>] <directory>
step.

Working with the Cisco Packaging Script

Cisco provides a simple script (create_ova.sh) to help developers with packaging a

virtual service compatible OVA. Use of this script is entirely optional and will generally
be more useful for high-volume developers. This script takes the raw files for the
application, generates the manifest file and builds the OVA tar package.

Usage:

create_ova.sh [<options>] <directory>

Options:

 -mts or -max_total_size - (default 600) Specify the maximum directory size before compression is
considered.

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 16 of 26
  -mfs or -max_file_size - (default 10) If max_total_size is exceeded, compress each file larger than
this threshold.

Together -mts and -mfs provide a heuristic to calculate whether or not to compress large files. If
total directory size exceeds -max_total_size, compress files greater than -max_file_size.
Specifying '-max_total_size 0' will force compression on all files greater than -max_file_size Setting
-max_total_size to a very high value will avoid compression

Examples:

 create_ova.sh Test - Create unsigned OVA package, compress files if necessary using Directory
'Test'
 create_ova.sh -mts 500 -mfs 20 Test - Create unsigned OVA package, compress files greater
than 20M if total directory size of 'Test' is greater than 500M

The OVA package will be created inside the specified directory. Example procedure for creating
OVA package shown below:
mkdir test
cp linux.img test
echo 1.7 > test/test.ver
# Note: edit test/package.yaml and add required yaml parameters for virtual ser
create_ova.sh test

If an existing ova is untared and repackaged, be sure none of the files are compressed before
running the packing script or the resulting manifest file will prevent the ova package from being
used on the router.

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 17 of 26
 Service Container Deployment
Phase Trigger Actions Virtual Service Instance State

Pre-Installation 1. Gather and prepare system resources  Host is ready to accept new
virtual service.
2. Establish internal communication
infrastructures.

Installation VMan received a 1. Unzip and unpack the virtual service  Validated that package is
request to install a definition from its OVA package. Cisco signed.
virtual service
package. 2. For signed application, perform SHA2 code  Validated integrity of OVA
signing check using the artifacts in the OVA content.
(.cert, .mf) and a hidden Cisco public key.
 Validated and parsed machine
3. Validates the machine definition specified in definition and binds it to a
the OVA and performs preliminary resource virtual service “instance name”
check (for warnings).

4. Parses the machine definition and creates

internal objects for manageability.

5. Process tiered resource profiles requests.

Configuration VMan received a 1. Perform validation and necessary network  Virtual service is configured
request to configure provisioning for configured guest IP address
instance of the virtual (if applicable)
service.
2. Perform resource check and reservation for
selected profile.

Activation VMan received a 1. Carves our storage resource from host  Virtual service is activated.
request to activate system as per need.

2. Commit CPU, Memory, Storage and

Networking resources as needed.

3. Update the machine definition XML and

request libvirt to start virtual machine.

4. Service to console, aux, logging and tracing

ports as needed.

Post Activation 1. Perform monitoring services

2. Process lifecycle control services

DEPLOYING A KVM VIRTUAL SERVICE IN AN IOS-XE ROUTER

In this document we’re going to cover the end-user interactions for installing and managing
virtual services using the command line interface. There are other graphical management tools
available today and in the future that will take advantage of APIs provided by the system.
These are generally outside the scope of this document targeted at developers. Just be aware

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 18 of 26
 that options such as Prime Infrastructure and Cisco Fog Director are options for dealing with
virtual services in addition to the command line.

virtual-service install name

<name> package <uri:.ova>

(conf)virtual-service <name>
Install (conf)
activate
Service interface virtualportgroup 0
virtual-service name
(package) ip address 10.0.0.1 255.255.255.0
<name> uninstall
virtual-service <name>
Deactivate and Configure vnic gateway virtualportgroup 0
Service guest ip address 10.0.0.2
Un-Install
Service (VM instance)

Monitor Start Service

Service (VM instance)
(conf)virtual-service <name>
show virtual-service list
activate
show virtual-service detail name <name>

This section uses an example KVM virtual machine to demonstrates bringing up of a generic guest OS
within the host OS.

Step #1: Copy the package to target

Step #2 : Install the package

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 19 of 26
 Step #3 : Configure the service

Step #4: Start the service

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 20 of 26
 Step #5: Monitor the service

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 21 of 26
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 22 of 26
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 23 of 26
 Step #6: Console into the service

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 24 of 26
© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 25 of 26
 For More Information
Read more about the Cisco IP Phone 7905G, or contact your local account representative.
Read more about the Cisco End-of-Life Policy.
Subscribe to receive end-of-life/end-of-sale information.

Printed in USA CXX-XXXXXX-XX 10/11

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 26 of 26