U.S.

General Services Administration

Networx PMO

Networx OSS Verification Testing

Test Plan Guidelines1

Version 2.1
March 19, 2007

1
This document is a supplement to the Networx OSS Verification Testing Concept of Operations (CONOPS)
 Networx OSS Verification Testing – Test Plan Guidelines

Document History

Status Version Date Comments Audience

Draft various 3/15/07 Drafts for comment and review Limited

Final 2.0 3/19/07 Updated based upon GSA comments Full

- ii -
 Networx OSS Verification Testing – Test Plan Guidelines

Table of Contents
1 OVERVIEW.......................................................................................................................................................1
1.1 OSS VERIFICATION TESTING – DESCRIPTION ..............................................................................................1
1.2 NETWORX OSS VERIFICATION TEST PLAN .................................................................................................2
1.3 NETWORX UNIVERSAL SERVICES OVERVIEW..............................................................................................3
1.4 NETWORX ENTERPRISE SERVICES OVERVIEW .............................................................................................5
1.5 OPERATIONS SUPPORT SYSTEMS – OVERVIEW ............................................................................................8
2 REFERENCED DOCUMENTS .......................................................................................................................9
3 NETWORX OSS TEST PLANS.....................................................................................................................10
3.1 TEST PROCESS ...........................................................................................................................................10
3.1.1 Process Description.............................................................................................................................10
3.1.2 Testing Documentation ........................................................................................................................11
3.2 TEST ENVIRONMENT .................................................................................................................................11
3.2.1 Testing Locations.................................................................................................................................12
3.2.2 Test Environment Hardware and Networking .....................................................................................12
3.2.3 Test Environment Software..................................................................................................................12
3.2.4 Data Flow and Processing ..................................................................................................................12
3.3 TEST SCENARIOS .......................................................................................................................................12
3.3.1 Order Flow Related Scenarios.............................................................................................................13
3.3.2 Billing Scenarios..................................................................................................................................13
3.3.3 Inventory Impact Scenarios .................................................................................................................13
3.4 INDIVIDUAL VERIFICATION TESTS.............................................................................................................13
3.5 OSS VERIFICATION TESTING WEBSITE .....................................................................................................14
3.6 TEST RESULTS ...........................................................................................................................................14
4 NETWORX OSS VERIFICATION TESTS..................................................................................................15
4.1 GSA-PROVIDED TEST DATA .....................................................................................................................15
4.2 COMPARE DATA TO ORDER .......................................................................................................................15
4.3 TEST CASE 1: ORDER ACCEPTANCE .........................................................................................................16
4.3.1 Test Case 1: Direct Data Entry ...........................................................................................................16
4.3.2 Test Case 1: Single Orders in Electronic format .................................................................................16
4.4 TEST CASE 2: INFORMATION EXCHANGE ..................................................................................................17
4.5 TEST CASE 3: BULK ORDERS ....................................................................................................................17
4.6 TEST CASE 4: INVENTORY DATABASE MANAGEMENT .............................................................................18
4.7 TEST CASE 5: INVOICE FILES/DETAILED BILLING FILES ...........................................................................19
4.8 TEST CASE 6: OSS INFORMATION SECURITY REQUIREMENTS ASSESSMENT ............................................20

- iii -
 Networx OSS Verification Testing – Test Plan Guidelines

1 Overview
The purpose of the Networx OSS Verification Testing Project is to support the Networx Program
Management Office (PMO) in monitoring and reporting on the results of the required Operations Support
System (OSS) Verification Testing for Networx Universal and Networx Enterprise contract awardees.
This Networx OSS Verification Testing Test Plan Guidelines document is intended to provide guidance
to Networx contractors in finalizing their Networx OSS Test Plans as described in the following:
• Networx Universal RFP, Section C.3.9.2.2 Step 2--Verification Testing
• Networx Enterprise RFP, Section C.3.9.2.2 Step 2--Verification Testing.
The Networx PMO will use these guidelines in the evaluation of awarded contractors’ Networx OSS Test
Plans submitted for review and approval. To expedite the OSS Verification Testing process in general
and the acceptance of contractor OSS Verification Test Plans in particular, it is recommended that
Networx OSS Contractors ensure that their OSS Test Plans be consistent with these guidelines.
It is the government’s objective to work effectively and efficiently with our industry partners in
completing the required OSS Verification Testing. By providing this guideline in advance, it is
anticipated that the elapsed time required for OSS Verification Testing will be minimized.
The Networx PMO seeks, by this effort, to optimize the operational environment that must include GSA,
our Agency customers and our industry partners. By identifying and correcting OSS shortcomings during
testing, it should be possible to avoid most ordering, inventory and billing-related difficulties that would
otherwise occur after Networx operations have commenced. This document is intended to initiate and
facilitate the on-going dialog required to maximize the use of the Networx contracts.

1.1 OSS Verification Testing – Description

The Networx Universal and Enterprise RFPs require successful completion of six OSS Verification
Test Cases (Section E.3). They are:
TEST
CASE TEST CASE
#
1 Accept an order for each of the services the contractor is contracted to provide

2 Generate required acknowledgements for each order entered in Test Case #1

3 Accept a bulk order for each type of order entered in Test Case #1

Demonstrate that the system manages the inventory of Networx services delivered by the contractor
4
produces output that is consistent with the orders that were entered.
Demonstrate that the billing system produces an Invoice File and a Detail Billing File that correspond to
5
Test Case #1 and Test Case #3 and generates invoices that are accurate.
6 Demonstrate that the OSS meets Networx Security Requirements

For each type of service for these test cases, a variety of test scenarios must be defined. Test scenarios
specify the business condition to be tested. Figure 1 shows the relationship of test scenarios to services
for the various Networx Test Cases.

-1-
 Networx OSS Verification Testing – Test Plan Guidelines

NETWORX
OSS
VERIFICATION
TESTING

TEST TEST TEST TEST TEST TEST

CASE 1 CASE 2 CASE 3 CASE 4 CASE 5 CASE 6

SERVICES SERVICES SERVICES SERVICES SERVICES

SCENARIOS

SCENARIOS

SCENARIOS

SCENARIOS

SCENARIOS

SCENARIOS
TEST
TEST

TEST

TEST

TEST

TEST
Access
Voice Voice Voice Voice Voice
Security

SCENARIOS
SCENARIOS

SCENARIOS

SCENARIOS

SCENARIOS
Circuit Circuit Circuit Circuit Circuit

TEST
TEST

TEST

TEST

TEST
Switched Switched Switched Switched Switched
Data Data Data Data Data

SCENARIOS
SCENARIOS

SCENARIOS

SCENARIOS

SCENARIOS

TEST
TEST

TEST

TEST

TEST
Toll Free Toll Free Toll Free Toll Free Toll Free
...

...

...

...

...
SCENARIOS

SCENARIOS

SCENARIOS

SCENARIOS
SCENARIOS
TEST

TEST
TEST

TEST

TEST
Paging Paging Paging Paging Paging

Figure 1 - OSS Verification Testing Approach 2

For Networx OSS Verification Testing, specific test scenarios must be defined by the contractor. As
indicated in Figure 1, these test scenarios must cover all services offered for which the test is valid.
They must include all valid order types and include business rules for commonly encountered error
conditions (including those that interrupt processing and those that allow processing to continue). Test
scenarios must also include commonly encountered service, feature, circuit, location, provisioning and
billing conditions.
Verification tests must be conducted to exercise and validate the various test scenarios. A single set of
test data is used to exercise one or more test scenario. For each test the contractor must document the
test being performed, test data used, the OSS systems or system components affected, manual
procedures required and expected results including verification of database contents,
acknowledgements, generation of billing records, etc. OSS Verification Testing for a service is
complete when all test scenarios for that service have been validated.

1.2 Networx OSS Verification Test Plan

To comply with these guidelines, an acceptable Networx OSS Test Plan must, at a minimum, include
the following:

2
Services for Networx Universal are shown in the Figure.

-2-
 Networx OSS Verification Testing – Test Plan Guidelines

• A description of the proposed Networx OSS verification testing process

• A description of the proposed Networx OSS verification testing environment
• A description of test scenarios that will demonstrate the GSA-required Networx OSS capabilities
• A description of how individual verification tests will be conducted and documented
• These individual verification tests must use the contractor’s own data (previously submitted with
their proposal and/or data submitted at time of award) and data supplied by GSA.
Each of these requirements is described in detail in Sections 3 and 4 below.

1.3 Networx Universal Services Overview

Networx Universal contractors must provide 36 mandatory services and may chose to offer one or
more of 12 optional services. The 48 mandatory and optional services specified in the Networx
Universal RFP are:
• 20 Telecommunications Services
o Communications Transport Based
o Internet Protocol Based
o Optical Based
• 13 Management and Applications Services
• 8 Security Services
• 3 Special Services
o Includes 2 satellite services
o Land Mobile Radio
• 4 Wireless Services
A detailed listing of these mandatory and optional services is provided in Table 1. Networx Enterprise
Services are described in Section 1.4 below.
In the Networx Universal RFP, originally issued in May 2005 with subsequent amendments, each service
is described in functional terms, including the industry standards to be met and the connectivity to be
provided. Other information contained in the RFP relates to connectivity to Government Furnished
Property (GFP), the Public Switched Telephone Network (PSTN), and other Networx Universal and
Networx Enterprise contractor networks and associated technical capabilities required.

-3-
 Networx OSS Verification Testing – Test Plan Guidelines

Networx Universal Services

RFP Reference Service Type Service Category Service
Telecommunications Services
Communications Transport
2.2.1 Voice
2.2.2 Circuit Switched Data
2.2.3 Toll-Free
2.3.1 Frame Relay
2.3.2 Asynchronous Transfer Mode
2.5.1 Private Line
2.6.1 Combined
2.7.1 Ethernet
IP-Based
2.4.1 Internet Protocol
2.4.6 Content Delivery Network
2.7.2 Premises-Based IP VPN
2.7.3 Network-Based IP VPN
2.7.8 Voice over IP Transport
2.7.9 IP Video Transport
2.7.10 IP Telephony
2.7.11 Converged IP
2.7.12 Layer 2 VPN
Optical
2.5.2 Synchronous Optical Network
2.5.3 Dark Fiber
2.5.4 Optical Wavelength
Management & Application Services
2.4.2 Dedicated Hosting
2.4.3 Collocated Hosting
2.4.5 Internet Facsimile
2.8.1 Video Teleconferencing
2.8.2 Audio Conferencing
2.8.3 Web Conferencing
2.9.1 Managed Network
2.11.2 Call Center/Customer Contact Center
2.11.9 Customer Specific Design and Engineering
2.11.10 Storage
2.11.11 Unified Messaging
2.11.12 Collaboration Support
2.12.1 Teleworking Solutions
Security Services
2.7.4 Managed Tiered Security
2.10.1 Managed Firewall
2.10.2 Intrusion Detection and Prevention
2.10.3 Vulnerability Scanning
2.10.4 Anti-Virus Management
2.10.5 Incident Response
2.10.6 Managed E-Authentication
2.10.8 Secure Managed Email
Special Services
2.14.6 Land Mobile Radio
2.15.1 Mobile Satellite
2.15.2 Fixed Satellite
Wireless Services
2.14.1 Cellular/PCS
2.14.2 Cellular Digital Packet Data
2.14.3 Multimode Wireless
2.14.5 Paging
Key: Mandatory Service
Optional Service

Table 1 - Networx Universal Services

-4-
 Networx OSS Verification Testing – Test Plan Guidelines

1.4 Networx Enterprise Services Overview

Networx Enterprise contractors must provide either:
(a) IP based service sets or
(b) Wireless based service sets
If IP-based service sets are offered, the contractor must provide 9 mandatory services and may chose to
offer one or more of 41 optional services.
If Wireless-based service sets are offered, the contractor must provide (1) Cellular/PCS service and
may chose to offer one or more of 49 optional services.
The 50 mandatory and optional services specified in the Networx Enterprise RFP are:
• 20 Telecommunications Services
o Communications Transport Based
o Internet Protocol Based
o Optical Based
• 13 Management and Applications Services
• 8 Security Services Type
• 1 Special Service (Land Mobile Radio)
• 4 Wireless Services
• 4 Access Services
A complete listing of these mandatory and optional services is provided in Tables 2a and 2b.
In the Networx Enterprise RFP, issued in May 2005 with subsequent amendments, each service is
described in functional terms, including the industry standards to be met and the connectivity to be
provided. This includes connectivity to Government Furnished Property (GFP), the Public Switched
Telephone Network (PSTN) and other Networx Universal and Networx Enterprise contractor networks
and associated technical capabilities required.

-5-
 Networx OSS Verification Testing – Test Plan Guidelines

Networx Enterprise Services: IP-Based Services Set

RFP Reference Service Type Service Category Service
Telecommunications Services
Communications Transport
2.2.1 Voice
2.2.2 Circuit Switched Data
2.2.3 Toll-Free
2.3.1 Frame Relay
2.3.2 Asynchronous Transfer Mode
2.5.1 Private Line
2.6.1 Combined
2.7.1 Ethernet
IP-Based
2.4.1 Internet Protocol
2.4.6 Content Delivery Network
2.7.2 Premises-Based IP VPN
2.7.3 Network-Based IP VPN
2.7.8 Voice over IP Transport
2.7.9 IP Video Transport
2.7.10 IP Telephony
2.7.11 Converged IP
2.7.12 Layer 2 VPN
Optical
2.5.2 Synchronous Optical Network
2.5.3 Dark Fiber
2.5.4 Optical Wavelength
Management & Application Services
2.4.2 Dedicated Hosting
2.4.3 Collocated Hosting
2.4.5 Internet Facsimile
2.8.1 Video Teleconferencing
2.8.2 Audio Conferencing
2.8.3 Web Conferencing
2.9.1 Managed Network
2.11.2 Call Center/Customer Contact Center
2.11.9 Customer Specific Design and Engineering
2.11.10 Storage
2.11.11 Unified Messaging
2.11.12 Collaboration Support
2.12.1 Teleworking Solutions
Security Services
2.7.4 Managed Tiered Security
2.10.1 Managed Firewall
2.10.2 Intrusion Detection and Prevention
2.10.3 Vulnerability Scanning
2.10.4 Anti-Virus Management
2.10.5 Incident Response
2.10.6 Managed E-Authentication
2.10.8 Secure Managed Email
Special Services
2.14.6 Land Mobile Radio
Wireless Services
2.14.1 Cellular/PCS
2.14.2 Cellular Digital Packet Data
2.14.3 Multimode Wireless
2.14.5 Paging
Access Services
2.13.2 Wireline Access
2.13.3 Broadband Access
2.13.4 Wireless Access
2.13.5 Satellite Access
Key: Mandatory Service (IP-Based Services Set)
Optional Service (IP-Based Services Set)

Table 2a – Networx Enterprise Services: IP-Based Services Set

-6-
 Networx OSS Verification Testing – Test Plan Guidelines

Networx Enterprise Services: Wireless Services Set

RFP Reference Service Type Service Category Service
Telecommunications Services
Communications Transport
2.2.1 Voice
2.2.2 Circuit Switched Data
2.2.3 Toll-Free
2.3.1 Frame Relay
2.3.2 Asynchronous Transfer Mode
2.5.1 Private Line
2.6.1 Combined
2.7.1 Ethernet
IP-Based
2.4.1 Internet Protocol
2.4.6 Content Delivery Network
2.7.2 Premises-Based IP VPN
2.7.3 Network-Based IP VPN
2.7.8 Voice over IP Transport
2.7.9 IP Video Transport
2.7.10 IP Telephony
2.7.11 Converged IP
2.7.12 Layer 2 VPN
Optical
2.5.2 Synchronous Optical Network
2.5.3 Dark Fiber
2.5.4 Optical Wavelength
Management & Application Services
2.4.2 Dedicated Hosting
2.4.3 Collocated Hosting
2.4.5 Internet Facsimile
2.8.1 Video Teleconferencing
2.8.2 Audio Conferencing
2.8.3 Web Conferencing
2.9.1 Managed Network
2.11.2 Call Center/Customer Contact Center
2.11.9 Customer Specific Design and Engineering
2.11.10 Storage
2.11.11 Unified Messaging
2.11.12 Collaboration Support
2.12.1 Teleworking Solutions
Security Services
2.7.4 Managed Tiered Security
2.10.1 Managed Firewall
2.10.2 Intrusion Detection and Prevention
2.10.3 Vulnerability Scanning
2.10.4 Anti-Virus Management
2.10.5 Incident Response
2.10.6 Managed E-Authentication
2.10.8 Secure Managed Email
Special Services
2.14.6 Land Mobile Radio
Wireless Services
2.14.1 Cellular/PCS
2.14.2 Cellular Digital Packet Data
2.14.3 Multimode Wireless
2.14.5 Paging
Access Services
2.13.2 Wireline Access
2.13.3 Broadband Access
2.13.4 Wireless Access
2.13.5 Satellite Access
Key: Mandatory Service (Wireless Services Set)
Optional Service (Wireless Services Set)

Table 2b – Networx Enterprise Services: Wireless Services Set

-7-
 Networx OSS Verification Testing – Test Plan Guidelines

1.5 Operations Support Systems – Overview

Operations Support Systems (OSS) provide the framework for the day-to-day control and management
of a telecommunications service provider’s network. These systems include the following:
• Service Order Entry
• Provisioning
• Inventory Management
• Billing
• Network Management
A carrier’s service order entry systems provide for the timely and accurate capture of all information
pertaining to ordering (adding, changing, disconnecting) a communications circuit or service of any
type or size. Accuracy within ordering systems is critical to both the carrier and the carrier’s customers
since the order information dictates how circuits and other communications services are created and
billed to customers. An inaccurate order or ordering system usually leads to customer frustration and
may cause loss of customers and revenue.
Provisioning systems are used to physically assign network elements and other equipment to the
service ordered. Additionally, provisioning systems track the progress of service implementation and
may trigger updates to order information or confirmations and notifications. As part of this process, an
inventory of equipment and equipment assignments is created. The execution of order provisioning
may generate non-recurring charges (NRC) associated with service installation as well as monthly
recurring charges (MRC) that are passed to the billing system. Accuracy is important since errors can
result in service problems or implementation delays while the configuration errors are corrected.
Telecommunications billing systems must produce complete and accurate invoices, including recurring,
non-recurring and usage charges, fees and taxes. These invoices must also accurately reflect the
customer’s desired accounting structure. If desired by the customer, a single invoice must be available
for all types of services.
Network Management Systems are essential for the maintenance of network services. Through
mediation networks, they are connected to a carrier’s network elements. They monitor network
elements and identify temporary errors, equipment failures and network outages. Increasingly, they
play a central role in connecting provisioning and other OSS to network elements to download circuit
configuration data on a “flow-through” basis, avoiding the need for local technicians to re-key circuit-
specific information.
Networx OSS Verification Testing encompasses the following OSS processes:
• Service order entry and order management process
• Service inventory management
• Billing/Invoicing resulting from the orders entered and sample usage data.
A test of the contractor’s access controls to their Internet-based ordering portal is also included.

-8-
 Networx OSS Verification Testing – Test Plan Guidelines

2 Referenced Documents
The following documents were used in preparing this Test Plan Guidelines document:
• NIST Special Publication 800-14, Generally Accepted Principles and Practices for Securing
Information Technology Systems
• NIST Special Publication 800-53, Recommended Security Controls for Federal Information Systems
• NIST Special Publication 800-44, Guidelines on Securing Public Web Servers
• NIST Special Publication 800-42, Guideline on Network Security Testing
• Networx Universal RFP Section C, Amendment 0010, 11/15/2006.
• Networx Universal RFP Section E, Amendment 0010, 11/15/2006.
• Networx Universal RFP Section J, Amendment 0010, 11/15/2006.
• Networx Universal RFP Section L33-L34, Amendment 0010, 11/15/2006.
• Networx Enterprise RFP Section C, Amendment 08, 12/29/2006
• Networx Enterprise RFP Section E, Amendment 08, 12/29/2006.
• Networx Enterprise RFP Section J, Amendment 08, 12/29/2006
• Networx Enterprise RFP Section L33-L34, Amendment 08, 12/29/2006
• Networx OSS Verification Testing Concept of Operations (CONOPS)

-9-
 Networx OSS Verification Testing – Test Plan Guidelines

3 Networx OSS Test Plans

A review of the proposed OSS Verification Test Plans submitted after Networx Universal and Networx
Enterprise contract awards will be conducted prior to the execution of OSS verification tests. As
specified in both RFPs, if the Test Plans submitted after contract award are not satisfactory, awardees will
be required to revise them.
The test plan review process will include the tasks identified below.

3.1 Test Process

Networx contractor OSS test plans must identify the test process tasks that the contractor proposes to
use in addressing OSS verification testing requirements. Tasks should completely describe specific
actions that will be followed during testing. This should include a description of formal and informal
testing schedules, communications, and manual actions along with system process tasks and interfaces.
Any assumptions that the contractor has made about the OSS Verification Testing process should also
be described. Merely restating the OSS Verification Testing requirements described in the RFP is not
an acceptable test process description.
In addition to describing the specifics of their test plan, Networx contractors must also include the
names and contact details of the individuals responsible for each phase of their testing process.
3.1.1 Process Description
The test process description must include:
• Process flow diagrams
• Process descriptions
• Task descriptions and interfaces.
Testing tasks should include all manual and system-performed testing activities. Interfaces should
include all interactions between testing personnel and the OSSs, all interactions between individual
OSSs, and all interactions between testing personnel.
The descriptions should identify all anticipated test-related communications within the contractor test
team, including notifications of service orders submitted, statuses modified, system processing cycles
completed, issues encountered, and other pertinent information. The process description must also
identify individuals responsible for each test task and/or system along with their contact information.
The process description should identify the anticipated elapsed time for each process task and interface.
It must also identify the specific OSS system or system component that will perform each task in the
test process flow for each type of order and service, each type of response/message and each type of
output (e.g., invoice, report). Examples are described below.
Testing Tasks may include:
• Order submission
• Order detection/receipt
• Order acknowledgement
• Order processing
• Order confirmation
• Order completion process/notification
• Inventory update
• Usage processing

- 10 -
 Networx OSS Verification Testing – Test Plan Guidelines

• Rating
• Invoice generation
To accomplish these tasks, the following may be required:
• Modifications of the testing environment to simulate passage of time (e.g., changing the system
date)
• Manual notifications between groups of testing personnel regarding items to be processed, tasks
to be initiated, etc.
• Output review and assessment
• Test reporting
3.1.2 Testing Documentation
The test process documentation must describe how the contractor will document individual verification
test results. This should include:
• Test name, number or description
• Test scenario(s) covered
• Test data set used
• OSS component(s) affected
• Manual actions required
• Expected and actual outcomes
• Notes
− Analysis of individual test results
− Information on testing/task activities that differ from the production environment
− Other information
An example of a potential test documentation format is shown in Table 2. Contractors must describe
the testing documentation approach that they will use.
Table 2 - Example of Test Documentation Format

Test Name Test OSS Manual

Test Data Expected Actual
or Scenario Component Actions Notes
Set Results Results
Number Reference Affected Required

Testing archives must be maintained as prescribed in the Networx Universal and Enterprise RFPs
(Sections C.3.9.2.2 Step 2--Verification Testing).

3.2 Test Environment

The government’s objective in testing the contractor’s Operations Support Systems is to identify any
ordering, inventory, or billing problems before starting Networx operations. In order for the OSS
Verification Testing to be effective, the contractor test system environment must, as closely as possible,
mimic the conditions of the contractor’s production system. Contractor OSS test plans must specify

- 11 -
 Networx OSS Verification Testing – Test Plan Guidelines

the test environment that will be used for the Networx OSS verification tests. The environment
information should include the configuration of the test systems or test partition of operational systems,
how the test configuration differs from the operational configuration, and any test tools to be used in
the conduct of the tests.
3.2.1 Testing Locations
The location or locations (address, city, state) where OSS Verification Testing will be conducted must
be indicated in the contractor’s OSS Verification Test Plan.
3.2.2 Test Environment Hardware and Networking
The contractor’s OSS test plan should specify the OSS test processing hardware (PCs/workstations,
server/mainframe type and sizing information), storage (e.g., local data storage versus network
attached storage (NAS), storage area networks (SAN) and networking configuration and bandwidth.)
Where test partition hardware and networking is different from operational hardware and networking,
this must be identified.
3.2.3 Test Environment Software
The Contractor OSS test plan should specify the OSS test software including operating system(s),
DBMS, commercial middleware, COTS and custom-developed applications and integration software,
and any other elements of the software environment. Any differences between test partition software
and operational software (e.g., configuration settings, versions) must be described.
3.2.4 Data Flow and Processing
Any differences between data flow and processing in the test environment and in the contractor’s
normal operational environment must be identified. Any actions unique to the test environment (e.g.,
changing system dates to test provisioning interval validity or to allow multiple billing cycles) must
also be clearly identified with reasons for those actions and information on controls related to those
actions.

3.3 Test Scenarios

As described in paragraph 1.1, test scenarios for system testing are commonly used to describe the
business condition to be tested. Specific test scenarios must be defined by the contractor.
These test scenarios must cover all services offered by the contractor, all valid order types and all
business rules for error handling (both those conditions that interrupt processing and those that do not).
They must also demonstrate the ability to handle various service, feature, circuit and location
conditions and the associated provisioning and billing scenarios.
GSA will review the contractor-proposed test scenarios to verify their adequacy and to confirm that
they are consistent with current industry best practices for end-to-end OSS test scenarios.
If the proposed test scenarios submitted in the OSS Verification Test Plans are not sufficient and/or are
not consistent with industry best practices for end-to-end OSS test scenarios, GSA will provide a
written description of test plan deficiencies in accordance with the time frames provided in the RFP.
The Networx Universal and Networx Enterprise RFPs 3 allow the government 15 business days to
perform this review following receipt of contractor-submitted OSS Verification Test Plans. Also, the
Networx Universal and Networx Enterprise RFPs state 4 that “the contractor shall provide updates to
the OSS Verification Test Plan within 10 business days of receiving government comments.”

3
Section C.3.9.2.2 Step 2--Verification Testing.
4
Ibid.

- 12 -
 Networx OSS Verification Testing – Test Plan Guidelines

Since it is the objective of this guidance to support a timely transit through this testing phase of the
contract, GSA will attempt to provide its review and feedback in three business days.
Likewise, contractors are encouraged to provide their updates as soon as possible following receipt of
GSA’s documentation of plan deficiencies.

Order Flow Related Scenarios

Contractor test scenarios must, at a minimum, include the order flow conditions indicated below.
These must be performed for each type of service.
• Establish new service
• Migrate existing service from another provider with coordination
• Migrate existing service to newer technology service (e.g., POTS to VoIP) with coordination
• Modify an order after acknowledgement but before a Firm Order Commitment notice has been
issued
• Modify an order after a Firm Order Commitment notice has been issued
• Modify an order in response to an order rejection notice
• Change an established service – administrative (e.g., contact, billing information)
• Change an established service – technical (e.g., modify features, add legs)
• Disconnect an established service
3.3.1 Billing Scenarios
For both Direct Billing 5 and Centralized Billing 6 , test scenarios must, at a minimum, include the
following billing conditions for each type of service:
• Initial invoice – full month and partial month
• Subsequent monthly invoices including MRC and usage
• Final invoice (following service cancellation) – full month, partial month and MRC paid in
advance
3.3.2 Inventory Impact Scenarios
Test scenarios must at a minimum include the following inventory-modifying conditions for each
service:
• Reflect service installation in inventory
• Add features to existing service in inventory
• Add equipment to existing service in inventory
• Delete service inventory as a result of disconnect activity

3.4 Individual Verification Tests

As described in paragraph 1.1, individual verification tests exercise and validate one or more test
scenario. The results of each individual test must be documented using the contractor’s approach

5
Test data provided by GSA will include account billing instructions for a direct-billed agency, based upon Agency
Hierarchy Codes, as described in the Networx Universal and Networx Enterprise RFPs, Section C.3.6.1.
6
The Contractor will be expected to provide data records for centralized billing as described in the Networx
Universal and Networx Enterprise RFPs, Section C.3.6.2.

- 13 -
 Networx OSS Verification Testing – Test Plan Guidelines

described in their OSS Verification Test Plan. This documentation must include an indication of the
OSS systems or system components affected by the test, manual procedures required and a description
of expected versus actual results including verification of database contents, acknowledgements,
generation of billing records, etc. This must also indicate the test scenario(s) being validated.
GSA will provide three sets of test data to each contractor to be used in verifying their test scenarios.
These are:
• The contractor’s own test data previously submitted.
• Service order data submitted via test data sets prepared by GSA. This data will reflect real-world
ordering conditions as described in paragraph 4.1, below. It will be identical for all contractors.
• Ad-hoc order data to be entered on site into the contractor’s portal.

The documentation for all completed individual verification tests will be reviewed by GSA to ensure
that they are complete and internally consistent.

3.5 OSS Verification Testing Website

During OSS Verification Testing, the government will establish and maintain an OSS Verification
Testing website to publish test results 7 . During testing activities, this website will be updated on a
daily basis. Each Networx contractor will have secure access to the website. However, they will be
restricted to viewing information related to their own testing activities.
To facilitate the establishment of this website, the contractor’s OSS verification test plan must include
the following:
• The names and contact details of individuals who are authorized to access contractor test results
on this website.
• The names and contact details of the individuals who are authorized to add or delete names from
the authorized access list. This must include a primary and no more than two alternate contacts
for this role.
Failure to include the above information in the contractor’s OSS Verification Test plans submitted after
contract award may result in delays.

3.6 Test Results

Individual verification test results will be sent to the OSS Verification Testing project office from the
contractor’s testing location(s) as testing progresses. These results will be reviewed and a
determination made of whether the individual verification tests have passed or failed.
When individual verification tests fail to satisfy the test scenarios for a service, modifications to the
OSS systems or supporting procedures must be made and the tests repeated. When all test scenarios
for a service have been satisfied by individual verification tests, that service will have passed the
Networx test case.
As shown in Figure 1, multiple test scenarios will be used to verify that each service has successfully
met the acceptance criteria for each Networx Test Case. When the acceptance criteria has been met for
all services within a Test Case, a “pass” will be assigned to that Test Case and the results posted on the
OSS Verification Testing website. A similar process will be followed for all Networx Test Cases.

7
The URL for the OSS Verification Testing is www.networxossvt.fas.gsa.gov.

- 14 -
 Networx OSS Verification Testing – Test Plan Guidelines

4 Networx OSS Verification Tests

GSA will provide OSS verification test data sets to contractors for use in OSS verification testing.
Contractor test plans must incorporate the use of these data sets, as described below.

4.1 GSA-Provided Test Data

As part of the preparation for OSS testing, GSA will use actual service order data to construct tests for
the various types of services. The use of actual data is critical as accurate information for service
locations, serving wire centers and associated sites will provide the most realistic tests. The data will
also include actual service configurations, options and special arrangements.
Most of the test data will be valid order information required for successfully installing the desired
service. However, some intentional errors and cases of missing information will be included to ensure
that the errors are identified and receive proper handling by the appropriate OSS. This is important to
avoid problems where errors made on initial orders are time-consuming or nearly impossible to correct
once the order has been placed in service.
The types of errors in GSA-provided test data may include:
• Invalid addresses
• Missing required information
• Invalid DARs
• Invalid services
• Invalid CLINs
• Invalid AHCs
• Inconsistent order information
• Other typical error conditions identified by Agencies.

Three types of order data will be prepared. These are:

• Orders for direct data entry
• Single order data
• Bulk order data
Each of these scenarios is described below with information on how they will be used.

4.2 Compare Data to Order

In addition to reviews performed by the Networx contractor, GSA will compare the expected results of
order data to the corresponding results for each service offered by that contractor. A history will be
retained for each testing iteration, including those where failures or errors occurred during testing and
have to be repeated. This will provide an audit trail of all testing activities to support additional testing
iterations and for future reference. Separate processes will be used for each of the Test Cases as
described below.

- 15 -
 Networx OSS Verification Testing – Test Plan Guidelines

4.3 Test Case 1: Order Acceptance

Test Case 1 in both the Networx Universal and Enterprise RFPs requires the following:

Test
Description Acceptance Criteria
Case #

1 Accept an order for each of the Demonstrate that an authorized Government user can place an order
services specified in Section using Internet secure access, electronic mail, or electronic file
C.2, Technical Requirements, transfer and the order populates the fields in the contractor’s
that the contractor is ordering system in a way that meets the requirements of J.12.1,
contracted to provide Ordering Data Elements

4.3.1 Test Case 1: Direct Data Entry

If chosen as the method of entering order data, orders for each service must be entered into the
contractor’s portal using Internet secure access. The entry of orders will be a multi-stage process and
must include the following:
• New account creation
• Entry of new orders
• Modification of an order placed but not provisioned
• Modification of service location
• Order cancellation
• Other modifications
• Modification of an order placed, provisioned and in service (service modification)
• Move of service location
• Disconnect/cancellation order
Since the tests will be conducted prior to placing orders in service, the modification and cancellations
described above must be entered after initial order entry and/or provisioning activities.
Test data for Direct Data Entry will be provided by GSA in a format that is readable by testers to
minimize data entry errors and facilitate testing.
4.3.2 Test Case 1: Single Orders in Electronic format
Test orders similar to those described in Section 4.3.1 will also be available to contractors
electronically for automated entry into their order entry systems.
For the purposes of OSS testing, service order records in the Networx OSS Testing Order format (a
modified ASR 8 format) will be available for download on the OSS Verification Testing website. The
Networx OSS Testing Order format specifications will also be published on the OSS Verification
Testing website and will be available to prospective contractors prior to contract award.

8
Access Service Request (ASR) formats are a standard produced by the Alliance for Telecommunications Industry
Solutions (ATIS), 1200 G Street, NW, Washington, DC 20005.

- 16 -
 Networx OSS Verification Testing – Test Plan Guidelines

4.4 Test Case 2: Information Exchange

Test Case 2 in both the Networx Universal and Enterprise RFPs requires the following:

Test
Description Acceptance Criteria
Case #

2 Generate required Using Internet secure access, electronic mail, or electronic file transfer, the
acknowledgements contractor demonstrates that its ordering system can provide Order Receipt
for each order Acknowledgements and Service Order Completion Notices (SOCNs)
entered in Test containing the data elements specified in Attachments J.12.2.1, Order
Case #1 Receipt Acknowledgement, and J.12.2.5, Service Order Completion Notice
(SOCN)

The Networx Universal and Enterprise RFPs, Section L.34.2.3.9.3, specifies that the following types of
information be exchanged between the government and the contractor for each service:
• Order
• Order Receipt Acknowledgement
• Service Order Confirmation (SOC)
• Order Rejection Notice
• Firm Order Commitment Notice
• Service Order Completion Notice (SOCN).
In Test Case 2, OSS Validation Testing only examines Order Receipt Acknowledgements and Service
Order Completion Notices generated in response to test orders. Thus, for each order for each service,
these specific responses to order data submitted will be verified by the government to determine:
• They are consistent with the order placed
• They include all Networx-required data (both government and contractor-supplied)
• All data is accurate
• Erroneous data has been correctly detected.
Results of Test Case 2 will be posted on the OSS Verification Testing website

4.5 Test Case 3: Bulk Orders

Test Case 3 in both the Networx Universal and Enterprise RFPs requires the following:

Test
Description Acceptance Criteria
Case #

3 Accept a bulk Using Internet secure access, electronic mail, or electronic file transfer, the
order for each contractor demonstrates that its ordering system can accept an order for multiple
type of order instances of the same service (bulk order) under a single Agency Service Request
entered in Test Number and provide a single instance of each of the following: Order Receipt
Case #1 Acknowledgement, Service Order Confirmation, Firm Order Commitment Notice,
and SOCN and service order number in accordance with Section C.3.5.1.2.2,
Agency Places Order And Contractor Provides Agency With Acknowledgement

- 17 -
 Networx OSS Verification Testing – Test Plan Guidelines

Bulk orders must be for the same service for the same location. They can be placed in a variety of
different scenarios.
Examples include:
• Large numbers of cellular phones
• Large number of calling cards
• Multiple voice and/or data orders for the same location
For testing purposes, interface records in the Networx OSS Testing Order format described in
paragraph 4.3.2 of this document will be used to send bulk orders to contractors. These records will be
available for downloading from the OSS Verification Testing website.
In addition to the initial order, the following will also be submitted:
• New account creation
• Entry of new orders
• Modification of an order placed but not provisioned
• Modification of service location
• Order cancellation
• Other modifications
• Modification of an order placed, provisioned and in service (service modification)
• Move of service location
• Disconnect/cancellation order
Results of Test Case 3 will be placed on the OSS Verification Testing website.

4.6 Test Case 4: Inventory Database Management

Test Case 4 in both the Networx Universal and Enterprise RFPs requires the following:

Test
Description Acceptance Criteria
Case #

4 Demonstrate that the system The contractor demonstrates that its service inventory management
that manages the inventory system contains a database that maintains a complete and accurate
of Networx services inventory of Networx service orders that were established in Test Cases
delivered by the contractor #1 and #3. The contractor demonstrates that authorized Government
produces output that is users can access this database using secure Web queries in accordance
consistent with the orders with Section C.3.8.2.4, Contractor Makes Networx Inventory Data
that were entered. Available to Government.

Networx Universal and Enterprise RFP Sections C.3.8.2.4 describe, among other things, alternative
methods for the government to download inventory data from the contractor’s secure website. To
facilitate OSS Verification Testing, the contractor’s OSS Verification Test Plan submitted after
contract award should include the default file type and format that will be used to download inventory
data to the government to validate Test Case 4 results 9 . The inventory files must include, at a

9
Although indicated as an acceptable file exchange option in the RFPs, the use of comma separated values (CSVs)
in OSS Verification Testing is discouraged since commas sometimes legitimately appear as data in name fields,

- 18 -
 Networx OSS Verification Testing – Test Plan Guidelines

minimum, all inventory management data elements specified in Networx Universal and Networx
Enterprise Section C.3.8.1.2. This information is needed by the Networx PMO to prepare for
conducting OSS Verification Tests. Failure to include this format in the contractor’s OSS Verification
Test plans submitted after contract award may result in delays.
When actual inventory data is received, it will be compared with the inventory prior to and following
processing of the order. Any differences from the expected results will be identified. Results will be
placed on the OSS Verification Testing website.

4.7 Test Case 5: Invoice Files/Detailed Billing Files

Test Case 5 in both the Networx Universal and Enterprise RFPs requires the following:

Test
Description Acceptance Criteria
Case #

5 Demonstrate that the billing The contractor demonstrates that the output of its billing system is
system produces an Invoice consistent with the orders entered into its ordering system in Test
File and a Detail Billing File Cases #1 and #3, that the billing data elements meet the requirements
that correspond to Test Case of Attachments J.12.4, Billing Invoice and Detail, J.12.4.1, Invoice
#1 and Test Case #3 and File, and J.12.4.2, Detail Billing File, and that the charges are
generates invoices that are accurate and assigned in a manner that is consistent with the Agency
accurate Hierarchy Code

The Invoice File and Detail Billing File requirements are specified in Networx Universal and
Enterprise RFP Sections J.12.4.1 and J.12.4.2 respectively. The OSS Verification Testing will focus
only on contractor systems and records produced.
Prior to the beginning of testing activities, Networx OSS Testing Usage Records (a modified EMI 10
format) will be specified to provide contractors with detailed call and other usage information to be
used in creating invoices and the exchange of invoice and detailed billing information. Record
specifications and sample call information records will be published in advance on the OSS
Verification Testing website.
The government will compare pricing and other information provided in the contractor’s detailed
billing and invoice files to the corresponding contractor test orders placed in Test Cases 1 and 3. All
prices reflected in both the invoice and detailed billing files must agree with the contract pricing.
Usage charges for usage-sensitive services provided in the Networx OSS Testing Usage Records must
also be included. Any discrepancies between anticipated and actual results will be noted.
The results of this testing of the detailed billing and invoice files (i.e., pass or fail) will be posted on the
OSS Verification Testing website.
These comparisons must be performed at least twice for each set of contractor orders. The first set of
billing and invoice data received immediately after order placement and activation will be reviewed to
determine whether NRC and any prorated MRC charges have been calculated correctly. A second
review will be made of the billing and invoice data for the subsequent billing period to verify that
ongoing charges have been calculated and invoiced correctly.

foreign addresses and other fields. See RFC 4180 and www.creativyst.com/Doc/Articles/CSV/CSV01.htm for more
information about difficulties with CSV formats.
10
Electronic Message Interchange (EMI) records are a standard produced by the Alliance for Telecommunications
Industry Solutions (ATIS), 1200 G Street, NW, Washington, DC 20005.

- 19 -
 Networx OSS Verification Testing – Test Plan Guidelines

4.8 Test Case 6: OSS Information Security Requirements Assessment

Test Case 6 in both the Networx Universal and Enterprise RFPs requires the following:

Test
Description Acceptance Criteria
Case #

6 Demonstrate that the OSS The contractor demonstrates that its OSS identifies and authenticates
meets Networx security each user and that each user can only access information that they are
requirements authorized to access

The purpose of Test Case 6 is to assess the effectiveness of security controls of the contractor’s OSS
and supporting information systems with respect to publicly-accessible network-based (e.g., Internet)
access of ordering, inventory management and billing information.
The primary scope of Test Case 6 is an assessment of functionality of the security controls (e.g.,
authentication, authorization,
and accounting controls) TEST CASE 6
C.3.9.2 OSS Functional Requirements
implemented by the C.3.9.2.1 Step 1 – Security and
E.3 Verification Testing of
contractor. The specific Contractor’s OSS Performance

Networx Universal and • Table E.3-1 refers to C.3.9.2.1 • ID Number 1 refers to C.3.3.2
Enterprise RFP reference is
Sections C.3.9.2
Operational Support
Systems Functional C.3.3.2 Security Management
C.3.3.2.2.1 Step 1 – Plan
Requirements. Also, the
overall security • ID Number 4 refers to C.2.1.11
requirements as specified in
RFP Sections 3.3.2 Security
Management apply.
C.2.1.11 Security Requirements for Networx
Any security controls C.2.1.11.2 Security Guidance
implemented by the • Must comply with C.3.3.2 and OMB A-130
contractor on its OSS and • Shall comply with NRIC Focus Group 1A, ANSI T1.276-2003
and Telcordia security standards
OSS-supporting information • Mandatory guidance from FISMA, NIST FIPS 199,
systems must be in NIST FIPS 140-2, HIPPA, NS/EP directives in C.5 and C.2.1.12
accordance with the
contractor’s (complete)
security management process Figure 2 - Security References
and the contractor’s Security Plan (to be submitted upon award by the contractor for review by the
GSA CIO). Refer to Networx Universal and Enterprise RFP Sections 3.3.2 Security Management for
more information. The Security Plan review, Security Risks Assessment Report, and certification and
accreditation processes, as specified in Networx Universal and Enterprise RFP Sections 3.3.2, will be
comprehensive and inclusive of the OSS and OSS-supporting information systems. Therefore, the
security controls assessment requirements as part of OSS Verification Testing process focus on the
functionality of the security controls implemented by the contractor. Figure 2 contains a cross
reference of the various security-related sections of the RFPs and other documents.
In this context, testing scenarios proposed by the contractor for Test Case 6 are required to be
sufficiently comprehensive to enable an assessment of the security controls in place to control user

- 20 -
 Networx OSS Verification Testing – Test Plan Guidelines

access to information available via the OSS and/or OSS-supporting information systems via the
Internet. Testing will focus on security control functionality.
Security controls must adhere to Federal Government-accepted security principles and practices, or
better, as defined in NIST Special Publications (SP) and Federal Information Processing Standards
(FIPS) such as NIST SP 800-14 and NIST SP 800-53.
If access to ordering, billing and inventory management information is provided via a publicly
accessible Web page (or Web-based portal), the security controls, as assessed by Test Case 6, must
minimally comply with security principles and practices as defined in NIST SP 800-44 Guidelines on
Securing Public Web Servers.
Test scenarios must demonstrate:
• Functionality of controls on how information is accessed by authorized users (e.g., networks,
applications)
• Functionality of controls on who can access information (authentication of users)
• Functionality of controls on user-hierarchy based access to information (authorizing of user
access on a default-to-deny basis)
• Adequate procedures are in place that will prevent an unauthorized user from receiving
information as a result of a verbally submitted request
If all of the security controls assessed by Test Case 6 are rated as satisfactory, no further action will be
needed. If improvements are needed, the security testing will be repeated after the contractor has made
improvements.
The results will be placed on the OSS Verification Testing website.

- 21 -