Network Functions Virtualization: Testing Best Practices

Network Functions Virtualization (NFV) is a network architecture concept that proposes using IT virtualization
related technologies to virtualize entire classes of network node functions into building blocks that may be
connected, or chained, together to create communication services.

1. Scope
The document provides a list of best practices to be used for ensuring smooth migration of Network Elements and Services to NFV
environment.

It is recognized that a certain portion of the best practices and recommendation are not required in all cases of NFV
implementations.

Some part of testing, such as security and usability, are not addressed in this document, but still should be addressed when defining
the right strategy for NFV testing.

This document may apply to:

 ISVs developing VNF and NFV based network services and functions
 Telco operators in a process of implementing NFV

2. References
This document is based on ETSI ISG NFV Standards.

The following referenced documents are served as a baseline for this document:

GS NFV 001 Network Functions Virtualisation (NFV); Use Cases

GS NFV 003 Network Functions Virtualisation (NFV); Terminology for Main Concepts

NFV GS NFV 004 Network Functions Virtualisation (NFV); Virtualisation Requirements

GS NFV-PER 001 Network Functions Virtualisation (NFV); NFV Performance & Portability
Best Practises

3. Abbreviations
BHTR Busy Hour Traffic Rates

BIOS Basic Input/Output System

CDN Content Delivery Network

CIFS Common Internet File System

CPU Central Processing Unit

DPI Deep Packet Inspection

DUT Device Under Test

FTP File Transfer Protocol

HTML HyperText Markup Language

HTTP HyperText Transfer Protocol

HW Hardware

IMS IP Multimedia Subsystem

IP Internet Protocol

KPI Key Performance Indicator

MANO MANagement and Orchestration

MOS Mean Opinion Score

MOS-AV MOS-Audio & Video

N/A Not Applicable

NF Network Function

NFV Network Functions Virtualisation

NFVI Network Functions Virtualisation Infrastructure

NIC Network Interface Card

NID Network Interface Device

OS Operating System

P2P Peer-to-Peer

PPP Point-to-Point Protocol

PPPoE Point-to-Point Protocol over Ethernet

PSNR Peak Signal-to-Noise Ratio

QoE Quality of Experience

QoS Quality of Service

SLA Service Level Agreement

TC Test Case

TCP Transmission Control Protocol

VF Virtual Function

VM Virtual Machine

VNF Virtualised Network Function

VQM Video Quality Metric

VTA Virtual Test Appliance

DNS Domain Name System

NTP Network Time Protocol

SSH Secure SHell

NFS Network File System

TA Test Agent

ISV Independent Software Vendor

4. Introduction
Network Functions Virtualization (NFV) is a concept introduced by network operators in 2012 [Network Functions Virtualisation –
Introductory White Paper].

Network Functions Virtualization (NFV) is a network architecture concept that proposes using IT virtualization related technologies to
virtualize entire classes of network node functions into building blocks that may be connected, or chained, together to create
communication services.

NFV relies upon, but differs from traditional server virtualization techniques such as those used in enterprise IT. A virtualized network
function, or VNF, may consist of one or more virtual machines running different software and processes, on top of industry standard
high volume servers, switches and storage, or even cloud computing infrastructure, instead of having custom hardware appliances
for each network function.

The European Telecommunications Standards Institute (ETSI) has formed an Industry Specification Group on Network Function
Virtualization (ISG NFV).

The contributors of the NFV Introductory white paper as well as the ETSI ISG have identified Testing and QoE monitoring as one of
the main use cases and subjects to address when implementing NFV environment.

NFV Framework

The NFV framework consists of three main components.

1. Virtualized Network Functions (VNF) are software implementations of network functions that can be deployed on a Network
Function Virtualization Infrastructure (NFVI).

2. NFV Infrastructure (NFVI) is the totality of all hardware and software components which build up the environment in which
VNFs are deployed. The NFV-Infrastructure can span across several locations. The network providing connectivity between
these locations is regarded to be part of the NFV-Infrastructure.

3. Network Functions Virtualization Management and Orchestration Architectural Framework (NFV-MANO Architectural
Framework) is the collection of all functional blocks, data repositories used by these functional blocks, and reference points
and interfaces through which these functional blocks exchange information for the purpose of managing and orchestrating
NFVI and VNFs.
 High level NFV framework

The building block for both the NFVI and the NFV-MANO is the NFV platform. In the NFVI role, it consists of both virtual and physical
compute and storage resources, and virtualization software. In its NFV-MANO role it consists of VNF and NFVI managers and
virtualization software operating on a hardware controller. The NFV platform implements carrier-grade features used to manage and
monitor the platform components, recover from failures and provide effective security – all required for the public carrier network.

NFV Use Cases

The first standard issued by ETSI identified 9 use cases for NFV in GS NFV 001:

Use Case #1: NFV Infrastructure (NFVI) as a Service

Use Case #2: Virtual Network Functions as a Service (VNFaaS)

Use Case #3: Virtual Network Platform as a Service (VNPaaS)

Use Case #4: VNF Forwarding Graphs

Use Case #5: Virtualisation of Mobile Core Network and IMS

Use Case #6: Virtualisation of Mobile Base Station

Use Case #7: Virtualisation of the Home Environment

Use Case #8: Virtualisation of CDNs (vCDN)

Use Case #9: Fixed Access Network Functions Virtualisation

Each of those use cases requires different level of techniques and has different set of QoS and QoE KPIs. Nevertheless, from
System Test point of view, same workload scenarios may apply for all use cases.

5. Assuring compliance and conformance to ETSI ISG Standards

Assuring compliance aim to find the deviations from the ETSI ISG Standards. It determines whether the NFV is meeting the defined
standards.

Compliance is performed on two levels:

 MANO
 VNF/NE

When performing compliance testing on NFV layer, each component is analyzed independently (MANO, VNFM, VIM, NFVI and
interfaces).

VNF/NE compliance is archived mainly by comparing the VNF and NE descriptor to the main

Test Procedure

Compliance tests are done using the following steps:

Step 1: Analyze ISG current standards

Step 2: Prepare compliance requirements

Step 3: Analyze each NFV component and interfaces

Step 4: Define gap points and analyze

Step 5: Prepare correction plan

Step 6: Monitor correction plan execution

Step 7: Repeat steps 1-6 for each VNF and NE

MANO Compliance Criteria

Following criteria are evaluated:

 VNF Descriptor format

 VNF redundancy model
 VNF state transitions
 Host affinity and anti-affinity rules for deployment of VNFC instances
 Authorization of the lifecycle management request
 Validation of the lifecycle management request
 VNF Package
 Support multiple SWA-1 interfaces
 Support multiple SWA-5 interfaces
 Allocation of addresses
 Dynamic allocation and deallocation of addresses
 Scaling event types and format
 Auto-scaling
 Maintain records on VNF Packages

VNF Compliance Criteria

Following criteria are evaluated:

 VNF Design Patterns

 VNF Internal Structure

 VNF Instantiation
 VNFC States
 VNF Load Balancing Models
 VNF Scaling Models
 VNF Component Re-Use
 VNF Update and Upgrade

 Automatic procedure
 Control Update and Upgrade process
 Requesting virtual resources
 Roll-back
 VNF’s Properties

 Hardware Independence
 Virtualization and Container Awareness
 Elasticity
 VOID
 VNF Policy Management
 Migration operations
 VNF State
 VNF Internal Structure
 Reliability
  Location Awareness
 Application Management
 Diversity and Evolution of VNF Properties
 VNF Topological Characteristics

 Deployment Behaviour
 Virtualisation containers
 NFVI Resources
 Components and Relationship
 Location
 VNF States and Transitions

 States and Transitions as Architectural Patterns

 The VNF Descriptor
 VNF Instantiation
 VNFC Instantiation
 VNFC Instance Termination
 VNF Instance Termination
 VNF Instance Scaling
 Start and Stop VNF
 VNF Instance Configuration
 VNF Fault Management

 Virtualised resource faults

 VNF faults

6. NFV Test Environment

Build NFV Test Environment

NFV provides great flexibility when building Test Environment. Using NFV, the end user can:

 Build parallel test environments to address different needs

 Scale up and down compute resources allocated based on type of testing (performance, functional etc.)
 Copy any existing environment, either from development environment (once testing starts) or from production environment (to
investigate production problems)

When building Test Environment for NFV, the following guidelines should be kept:

 Management and Orchestration should be similar to the production, including all elements of NFV (MANO, VNFM, VIM, NFVI)
 NS and VNF Instantiation should be similar to production environment (hypervisors, computing, storage and network
resources)
 If possible, based on the available resources, same level of resources should be allocated. However, in case not enough free
resources are available, a load environment should be established when running performance and scalability testing

Test Appliances

Test Appliances are required in order to:

 Simulate valid workload traffic on the client and server

 Simulate both data plane and control plane traffic
 Measure key metrics, both data plane metrics and control plane metrics
  Assuring the appropriate compute, storage and network resources are allocated
 Active and passive monitoring of a single function, a set of functions or the entire service chain to ensure services adhere to
established QoS and QoE metrics
 Monitoring production SLA (not covered in this document)

Virtual Test Appliances vs. Physical Test Appliances

Existing physical test appliances may be used, but most test equipment vendors have developed virtual test appliances that work on
separated VMs in the NFV Test Environment. Virtual test appliances offer equivalent capabilities for almost all scenarios and meets
the flexibility required when testing NFV on multiple, geographically disparate. Virtual test appliances are usually much more cost
effective.

Physical test appliances are mainly recommended for the testing virtual environments that require the highest levels of data-plane
performance (line-rate) or microsecond-level timing accuracy.

Simulating real workload traffic

When simulating real workload traffic, consider the following steps:

Step 1 – measure production busy hours traffic rates (BHTR)

Step 2 – Simulate a traffic mix on the client and server including:

 HTTP
 FTP
 DNS
 Streaming video
 NTP
 SSH
 Syslog
 NFS
 Messaging
 VoIP
 Social Networks
 Unicast Video
 P2P
 CIFS
 Background Traffic

Step 3 – create different workload scenarios based on percentage of BHTR. The following traffic rates are recommended:

Level % of BHTR Used for

Low 5% Functional Testing
Average 50% Scalability, on-going, fail-over
Busy 100% Performance
Stress 150% Stress Testing, auto-scaling

Metrics

NFV Service Metrics

 VM Provisioning Latency, instantiation latency. Time between VM instantiation and first available packet
 VM Stall (event duration and frequency)
  VM Scheduling Latency

QoS and Data-plane metrics:

 Latency on each of tens of thousands of data streams

 Throughput and forwarding rate
 Frame loss rate
 Packet-delay variation and short-term average latency
 Dropped frames and errored frames
 Service Disruption Time for Fail-over Convergence

QoE and Control-plane metrics:

 HTTP: page load time, load time variance

 Video: MOS-AV score, range = 2-5 with 5 being the best
 HTML5 video – AS score, 100 % score as the maximum
 Direct metrics:

 Peak Signal to Noise Ratio (PSNR)

 Structural Similarity (SSIM) – compare the original image with the received image
 Video Quality Metric (VQM)
 Mean Opinion Score (MOS) – This metric combines delays, perceived jitter at application layer, codec used for
communication and packet loss at application layer
 Indirect metrics:

 Startup time: Time difference between sending the request for content and the time when the user actually received the
content
 Delivery synchronization – In a multicast many-to-many scenario it is important that the content is received by all
participants at the same time. Consider online gaming or video conferencing
 Freshness: The time difference between the time when the content is actually generated and the time when the users
receives it, e.g. celebrating a goal with friends while watching a sports event
 Blocking: When the buffers on the receiver are empty and the user has to wait for content.
 Connections establishment rate, and transactions per second
 Total number of connections, round trip time and goodput

7. Test Automation in NFV

Disclaimer: The solution described in this document is based on CloudShell for SDN/NFV by QualiSystems. Similar solution may be
built using a different tool.

The Need for Heterogeneous SDN/NFV Self-Service Test Infrastructure Automation

Software Defined Networking introduces the concept of network programmability from applications that interact with centralized SDN
controllers via northbound APIs. This API-driven network paradigm opens the way for agile SDN application development, but also
creates the need for networking organizations to deliver access to end-to-end network environments to application delivery
stakeholders for supporting a DevOPS process. Network Function Virtualization (NFV) adds to the complexity of this picture by
allowing agile service chaining of network functions hosted on virtual machines rather than hardware appliances. Automation of
access to network sandboxes is complicated by the fact that SDN and NFV will gradually phase into networks, leaving significant
portions of the network operating in a legacy mode, potentially for years. Networking teams need a self-service automation platform
that can handle both SDN/NFV and legacy networks in a unified manner.

Network DevOps Orchestration & Automation

Testing NFV requires a comprehensive automation platform including resource management, provisioning, test automation and
integrated reporting and business intelligence that is ideal for delivering self-service automation and continuous network certification
processes for SDN and NFV.

The following capabilities should be included:

 Centralized inventory management of all legacy and SDN/NFV network resources allowing engineers to gain visibility to any
components needed to design and publish network topologies required by developers and testers
 Integration with all the existing and future infrastructure, including legacy network devices, SDN-enabled switches, SDN
controllers and virtualized network functions
 Visual network topology and service chain design and publishing
 Visual workflow and test automation creation to build continuous integration
 Integrated reporting and business intelligence
 Sustainable object-based automation architecture
 Easy to use web-based self-service portal

NFV Test infrastructure automation framework based on CloudShell

Object-Based Test Automation Approach with TestShell

TestShell allows all automation elements to be captured as small-scope objects to enable high reusability for test workflow
construction.

Objects are divided into the following categories:

 NFV-Mano – an automation object for each MANO operation, such as instantiation of Network Service, Disable VNF package
etc.
 NFV Infrastructure – an automation object for each NFVI operation, such as create, shutdown, destroy and update virtual
machine (VM)
 VNF – Each VNF shall have dedicated objects based on its internal workflow.
 Legacy Network Services – each legacy network service shall have dedicated objects
 Test Appliances – each Test Appliance should have a list of Test Automation objects, such as simulate control plane traffic and
retrieve QoE parameters

8. Testing MANO
The first step is to test NFV Management and Orchestration Architectural Framework.
 NFV Management and Orchestration Architecture

NFV Orchestrator
 On-boarding of new Network Service (NS) and VNF Packages
 Instantiate Network Service i.e. create a Network Service using the NS on-boarding artefacts
 Query VNF – retrieve VNF instance state and attributes
 Check VNF instantiation feasibility
 Scale Network Service, i.e. grow or reduce the capacity of the Network Service
 Update Network Service by supporting Network Service configuration changes of various complexity such as changing inter-
VNF connectivity or the constituent VNF instances
 Disable, enable, update, query, delete VNF package
 Create, delete, query, and update of VNF Forwarding Graphs (VNF-FG) associated to a Network Service
 Create, delete, query, and update of Virtual Links (VL)
 Query Network Service and assure all attributes retrieved properly
 Terminate Network Services, i.e. request the termination of constituent VNF instances, request the release of NFVI resources
associated to NSs, and return them to NFVI resource pool if applicable
 Get VNF performance metrics and notification
 Simulate Notification and different fault information:

 physical infrastructure (compute, storage, and networking related faults)

 virtualised infrastructure (e.g., VM-related faults)
 application logic (i.e. VNF instance related faults)
 Monitoring and collection of information related to resource usage, including mapping of usage
 Scheduled request regarding VNF instances
 Global resource management, validation and authorization of NFVI resource requests
 Resources sharing between VNFs
 Create, update, delete, query, activate and de-activate policy (e.g., policies related with affinity/anti-affinity, management of
VNF or NS scaling operations, access control, resource management, fault management, NS topology etc.)
 Constraints management
 SLA parameters
  Network capacity adaptation to load
 Coexistence with legacy network equipment
 Controlling and managing Inventory of versions, releases and patches of all units of hardware and software
 Maintenance, hardware and software exchange, SW upgrades, Firmware upgrades, repair
 Manage log of all changes to inventory, unexpected events and maintenance activities
 Administration domains and permissions

VNF Manager:
 VNF instantiation, including VNF configuration if required by the VNF deployment template (e.g., VNF initial configuration with
IP addresses before completion of the VNF instantiation operation)
 VNF instantiation feasibility checking
 VNF instance software update/upgrade
 VNF instance modification
 VNF instance scaling out/in and up/down
 VNF instance-related collection of NFVI performance measurements and faults/events information, and correlation to VNF
instance-related events/faults
 VNF instance assisted or automated healing
 VNF instance termination
 VNF lifecycle management change notifications
 Configuration and event reporting between NFVI and the E/NMS
 Liveness checking of an VNF, e.g. watchdog timer or keepalive
 Failure detection
 Fault remediation of each VNF resiliency category

Virtualised Infrastructure Manager (VIM) and NFV Infrastructure (NFVI):

 Resource catalog management – controlling and managing Inventory of software (hypervisors), computing, storage and
network resources
 Collection and forwarding of performance measurements and faults/events
 NFV Infrastructure faults collection and remediation
 Create, shutdown, destroy and update virtual machine (VM)
 Create list of virtual VMs, query, reboot, suspend, resume, save and restore VM
 Create, modify, list, delete and query storage pool
 Create, delete, list, and query VM storage
 Allocate, query, update scale, migrate, operate, release of NFVI resources, and managing the association of the virtualised
resources to the compute, storage, networking resources, e.g. increase resource to VMs, improve energy efficiency and
resource reclamation
 Create, query, update and release resource reservation
 Create, query, update, delete and notify of VNF Forwarding Graphs, e.g., by creating and maintaining Virtual Links, virtual
networks, sub-nets, and ports
 Add, delete, update, query and copy of software images
 Root cause analysis of performance issues from the NFV infrastructure perspective
 Mechanism for time-stamping of hardware (e.g. network interface cards, NICs and NIDs)
 Create, update, list, query and delete hypervisor policies
 Create, delete, update, list and query virtual network
 Create, update, list, query, and delete subnet
  Create, update, list, query and delete port

Orchestrator – VNF Manager (Or-Vnfm)

 Resource related requests, e.g. authorization, validation, reservation, allocation by VNF Manager(s)
 Sending configuration information to the VNF Manager, so that the VNF can be configured appropriately to function within the
VNF Forwarding Graph in the NS
 Collecting state information

Virtualised Infrastructure Manager – VNF Manager (Vi-Vnfm)

 Resource allocation requests
 Virtualised hardware resource configuration and state information (e,g. events) exchange

Orchestrator – Virtualised Infrastructure Manager (Or-Vi)

 Resource reservation and allocation
 Virtualised hardware resource configuration and state information (e,g. events)

OSS/BSS – NFV MANO (Os-Nfvo)

 Requests for network service lifecycle management
 Requests for VNF lifecycle management
 Forwarding of NFV related state information
 Policy management exchanges
 Data analytics exchanges
 Forwarding of NFV related accounting and usage records
 NFVI capacity and inventory information exchanges

VNF – VNF Manager

 Requests for VNF lifecycle management
 Exchanging configuration information
 Exchanging state information

9. Scalability and Performance Testing

Scalability refers to the maximum number of control plane sessions that can be established. Examples include the number of PPPoX
sessions or number of routing peers. For routers, number of routes per session and total number of routes in the routing table are
also a measure of scalability.

In NFV environments, VNF should have auto-scale feature, so resources (compute, memory, network etc.) should be scaled
automatically in response to the varying network function performance needs.

The purpose of this test is:

 Ensure auto-scale works properly

 Ensure resources consumption is efficient
 Measure performance meets SLA in high performance workloads

Scalability
 Disable auto-scale and run low (5%), Average (50%) and busy (100%) traffic. Measure metrics for each traffic level.
 Manually allocate resources for average traffic to meet SLA. Increase traffic to busy and manually scale resources until SLA is
met. Compare resources allocations and analyze results
  Enable auto-scale and measure SLA in average and busy traffic. Analyze resources allocation results
 Run a dynamic test to shift between different traffic rates each 5 minutes randomly. Analyze results

Performance
 Run tests with low traffic. Increase traffic by 5% every 1 hour and analyze metrics
 Run on-going busy traffic rate for at least 5 days and measure metrics over time
 Run on-going busy traffic rate and execute all MANO tests

Stress testing
 Run stress traffic rate (150%) over 5 hours and measure metrics
 Define resource limit to support full traffic and run stress traffic rate (150%) over 5 hours and measure metrics

10. Failover Convergence Testing

Convergence time is one of the key metrics for validating SLAs and high availability in service provider networks. The requirements
for failover convergence times can be in the order of milliseconds depending on the services and applications that are under
consideration. These constraints are equally valid in an NFV environment as well.

In NFV deployments, there is an added factor of variability where failover convergence time for a VNF can be impacted by the
number of VNFs on the physical server that is converging to alternate routes. Convergence measurement involves the measurement
of processing time of the trigger event in the control plane and the traffic switchover time. It is important in a multiple VNF
deployment scenario that the convergence time of any VNF is not impacted by the other VNFs on the same physical server, so that
the VNF continues to satisfy the SLAs for which it was provisioned.

Test Setup

Fail-over Convergence Test Topology (source: ETSI GS NFV-PER 001)

 Provision a VNF for DUT with routing protocol functionality.

 Provision two virtual test endpoints (VTA) connected to the virtual DUT. In order to avoid the performance impact of resource
sharing between DUT and virtual test boxes, install the virtual test endpoints on a separate physical server.
 Configure the pre-determined routing protocol on the two virtual test appliances and advertise same set of routes from both
virtual test appliances. Use a preferred metric in the routes advertised from one of the virtual test appliances (primary next
hop/path).
 Configure L3 traffic between endpoints advertised by VTA 3 and endpoints advertised by VTA 1
 Configure liveness detection mechanisms such as BFD protocol on the VNF as well as VTA1 and VTA2.

11. VNF Migration Testing

Each VNF deployed in the NFV environment requires independent testing.

VNF Testing shall cover:

  Static testing: static testing involves reviewing requirements and specifications to ensure completeness or appropriateness for
the VNF
 Conformance testing: see VNF Compliance Criteria above.
 VNF-Mano Integration:

 VNF Instantiation
 VNFC States
 VNF Scaling Models
 VNF Component Re-Use
 VNF Update and Upgrade
 Virtualization and Container Awareness
 Elasticity
 VNF Policy Management
 Migration operations
 VNF State
 The VNF Descriptor
 Start and Stop VNF
 VNF Instance Configuration
 Virtualised resource faults
 VNF faults
 VNF Internal functionality – testing specific functions for VNF
 Scalability/Performance – run Performance testing and validate resources allocation
 End-to-end/system test: Testing VNF performance together with other VNFs and validate the introduction of the new VNF
doesn’t affect other VNFs functionality and performance

Solutions

Your Technologies

Your Initiatives

Industries

Insights

White Papers

The Qualitest Blog

Podcast: The Testing Show

Webinars

The “How To” Knowledge Center

About Us

Company Overview

Leadership Team

News

Careers

Diversity & Inclusion

Why Qualitest

Testimonials

Analyst Coverage

Case Studies

Talent Induction Process

Request a quote
Global Offices

Work with us

All rights reserved ©2020 | Privacy Policy | Modern Slavery Act | Terms & Conditions | Site Map | Contact Us