Admin Program

Case Study
Architecting Continuous Testing for
Large-Scale Systems Transformation

February 15, 2017

Presenters’ Bio
Vidya Iyer | [email protected]
Consultant
 Vidya is an ISTQB certified testing expert and a Consultant with Deloitte Consulting’s Global Testing Practice
focused on functional testing
 She has over six years of experience that spans across all phases of testing lifecycle including test strategy
development, planning, estimation, design and execution

Cyril Gonsalves | [email protected]

Senior Consultant
 Cyril is a Senior Consultant with Systems Integration practice focused on System Integration and Interfaces
testing
 He has over eight years of strong testing experience in Database, Data warehouse, Service Oriented
Architecture and BI reports

Mradul Kapoor |[email protected]

Senior Consultant
 Mradul is a Certified Scrum Master and an ISTQB Certified Testing expert working as a Senior Consultant with
Deloitte Consulting
 He brings over eight years of experience. His testing skills demonstrated across all test levels, including
specialized testing such as automation, performance and accessibility

Bharat Jadwani | [email protected]

Manager
 Bharat is a Manager in Deloitte Consulting’s Insurance Technology practice with over 14 years of focusing in
managing functional requirements and testing efforts for financial services industry
 He has extensive experience of working with FSI clients leading technology transformation and
2
modernization programs for new business and underwriting, policy administration and claims management
Agenda and Key Takeaways

Agenda Key Takeaways

 Discuss testing capabilities assessment Testing assessment framework aligned to agile

framework and enhancement roadmap methodology

 Outline agile testing framework for Agile testing capabilities and frameworks that
functional, integration testing other project can use

 Demonstrate automation frameworks for Frameworks for automated functional and

functional and integration testing integration testing

We delivered 143 Agile testing artifacts including frameworks, tools and

templates that we hope other SI projects will find useful
3
Project Background
For Policy Admin and Billing Systems transformations Life Insurance
Company #1 partnered with Deloitte to provide PMO and Testing
services
Issue Solution Impact
 Life Insurance Company #1  Deloitte was selected to implement  Created and implemented Agile
embarked on a multi-year PAS / PMO and testing processes to help testing processes to address
Billing systems transformation promote PAS and Billing in testing capability gaps
 Selected supplier had no prior production  Implemented Modular and
experience in Group Insurance;  Deloitte’s testing assessment Business process driven
platform was not ready for Group framework was used to determine functional framework for multi-
business testing capability gaps insurance application system
 Life Insurance Company #1 desired  Defined agile testing processes and  Implemented end to end
to implement Scaled Agile methods for functional and automation framework consisting
methodology, however foundational integration testing of integration and functional
capabilities were missing automations supporting CI/CD
 Defined capabilities for functional,
 Capability needed to execute integration and automation testing  Implemented Integration testing
common testing types such as  Deloitte defined quality methods framework for real-time/near
functional, integration, automation real-time interfaces
and processes for the below
were absent  Achieved 94% reduction by
 Agile testing capabilities
 Advanced testing capabilities were automating 76% of end to end
 Functional Testing
desired for efficient test execution test cases
such as test data management,  Integration Testing
 Clear definition of requirements
automation, service virtualization  Automation Framework and requirements management
 Requirement management process  Advised client on how to manage roles
was not defined thus rendering it requirements in a multi vendor set-
not useful for any program up
workstream

4
Testing Capability
Assessment

5
Testing Capability Assessment Approach
The assessment contained three phases including Current State
Assessment, Gap Analysis, and Recommendations

Key Inputs Admin Program Testing Capability Assessment Approach

Current State Gap Go-forward

Assessment Analysis Recommendations

 Study current state testing  Compare current state findings to  Develop recommendations to
Current capabilities known industry best practices improve current testing
Operating capabilities
Model  Analyze Admin Program testing  Identify capability gaps in
artifacts and tools to identify program test strategy and  Align Admin Program test
Key Activities

opportunities and constraints develop recommendations strategy with overall program

outlook, objectives and
Admin  Study program objectives, key  Conduct sessions with leadership assessment results
Program priorities and risks to discuss desired future state
Objectives testing capabilities and  Determine appropriate next
and  Deep dive current state testing considerations steps to ensure testing
Priorities processes and capabilities excellence for Admin Program
 Conduct follow-up sessions with
 Interview key program select stakeholders to review
Industry stakeholders to further gaps and recommendations
Insights and
understand priorities and goals
Leading
Practices
 Current State Findings  Preliminary Gap Analysis and  Final Recommendations and
Deliverable

Recommendations Roadmap

Deloitte
s

Experience

6
Testing Capabilities Assessment Summary
The following highlights the current, recommended, and final future
state maturity levels of each Admin Program testing capability

Key Observations
Capability Maturity Assessment  Most capabilities did not exist, key
processes were undefined
0 1 2 3 4 5
 Program test strategy was created
Test Governance but was outdated

Requirements Traceability  Roles, responsibilities and

ownerships were not clearly
defined; efforts were fragmented
Test Planning and Strategy
across teams

Testing Types  A holistic automation plan was

missing to drive program goals and
Test Preparation objectives
 Enterprise test environments and
Test Execution release management capabilities
existed but needed to be clearly
Defect Management defined for 3rd party builds
 No test data management approach
Testing Tools existed
Test Environment and  Testing tools existed but they were
Release underutilized
Advanced Testing Capabilities

Key
7
At the time of assessment Recommended State Final Future State
Testing Capability Enhancement Deliverables and Outcomes
The following capabilities were developed / enhanced as part of 2016
Admin Program functional and integration testing

Capability Deliverables Outcomes

Admin Program RACI, Communication Structure, Testing Clearly defined roles / expectations for multi-vendor
Test Governance Metrics and Reports projects, testing metrics / reports definition

Defect management tools and processes defined and

Defect Management Defect Management Approach, Guidelines, Tools executed for Life Insurance Company #1 Integrations and
Functional

Requirements Approach defined and implemented for Life Insurance

Requirements Traceability Approach, Guidelines, Tools
Traceability Company #1 Integrations and Functional

Automated regression tests for Life Insurance Company #1

Assessment and POC, Functional and Integration
Test Automation Frameworks, regression tests, CI / CD Integration
integrations (over 80% automation, cycle-time reduced by
94%)

Test Data Management Test Data Management Strategy and Framework Test data management strategy and tools defined

Admin Program Nonfunctional Test Types Definition and Test types and ownerships defined and executed with Life
Nonfunctional Testing Approach Insurance Company #1 and 3 rd party vendor

Execution plans, prioritization, steps and tasks defined and

Test Execution Agile Test Execution and Closure Guidelines executed for Life Insurance Company #1 Integrations and
functional

Approach to test scenarios and cases, test readiness

Test Preparation Agile Test Design and Preparation Guidelines
checklist and criteria defined

Admin Program Functional Test Types Definition and Functional test types, acceptance criteria, testing
Testing Types Approach objectives and ownership defined

Testing tools usage and frameworks for test management,

Testing Tools Testing Tools Configuration and Standardization
defect management and automation

Test Planning and Test approach for functional and integration testing,
Revised Admin Program Test Strategy, testing templates
Strategy templates for test scripts, defects, reports, metrics etc.

Test Environment and Test Environment and Release Management Processes and Processes defined for 3rd party builds / acceptance;
Release Management Framework implemented continuous builds testing
8
Deliverables Index Illustrative
A number of Agile testing tools and frameworks are available that we
will be happy to share with other testing project teams

9
Functional Testing
Capabilities

10
Functional Test Framework
This depicts the architectural view of functional test case development
framework

Test Case 1
5
System
Requirement Test Case 2 Test Suite 1
Life
s
Insurance
Company #1
L1/L2/L3/L4 Test Case 3
Requirement
s 6
Business Function Test Case 4
2 4
Library
1 (TFS / MTM Test Suite 2
Modules) Test Case 5
Business
Function
Catalogue
3
Test Data
Matrix
Test Suite n
Test Case n

Notes
1 Life Insurance Company #1 requirements were used to 4 Test cases were created by grouping reusable test modules
develop high level business functions and data

2 Business functions and system requirements were used to 5 Test cases were grouped together to create test suites. For
develop reusable test module in TFS / MTM example – End to End test suite

3 Reusable test data matrix was defined for each business 6 Test cases / suites were further used as input for automated
module testing
11

Platform independent
Functional Test Case Framework – Benefits
The following highlights the key benefits of functional testing approach

Reusable System independent

Business functions and data matrix can Business process catalog and test data is
be used to develop any number of test based on Life Insurance Company #1
cases for various products / business requirements rather than system
requirements requirements

Data driven Efficient and scalable

Test cases are data driven, hence same Reduced test development time and
test cases can be executed for different maintenance efforts; can be enhanced
set of data to achieve various testing to include new requirements /
objectives capabilities

Automation support Traceability support

Business functions are used for both Test components are directly mapped to
manual and automation test case design; business requirements, hence easily
automation can be used at component or traceable / maintainable as requirements
end to end level change

12
Business Function Catalog
The business function catalog is created using business requirements.
The modules are further fragmented into reusable test modules
Life Insurance
Company #1
Modular business function Test module id High level test
Business
used to define test steps MTM for data requirements
Requirements
and data traceability

ILLUSTRATIVE

Functional areas Brief description MTM module

for logical of business name to be used
grouping function in tests 13
Test Data Matrix
The test data matrix is a template which includes the master data
requirements by the test team during the test execution phase

Business function Test data needed

from the catalog for test cases

ILLUSTRATIVE

Grouped as per Test data

the functional parameters
areas
14
Business Functions
This slide shows the creation of Business Function Library and how test
cases will be linked in Microsoft Test Manager
Library Structure
1
MTM Modules
2

Test Steps

Test Data 5
Matrix

Notes
1 Reusable test components aligned with business function 4 Test steps written for a test module / business function in TFS /
catalog MTM

2 System level components (PAS, Billing etc.) further broken down 5 Test modules / steps contain the input parameters needed to
into functional areas execute a business function
15
3 MTM test modules for each business function
Test Case creation using Business Function Library
After high level scenarios are identified, test case development involves
grouping the business functions in MTM

PAS QPR Creation Test Case

16
Functional Testing Deliverables
Following artifacts were delivered as part of 2016 functional testing for
Life Insurance Company #1 Admin Program

Deliverable Description

 Overall test approach highlighting scope, test types, environments, tools,

Test Approach reporting, communication and roles and responsibilities across Life
Insurance Company #1, Majesco and Deloitte
 Reusable estimation models for functional testing types
Testing Estimation Models  Baseline estimates for end to end testing by Life Insurance Company #1
and Deloitte testers
 Detailed plan for Life Insurance Company #1 and Deloitte testers covering
Training Plan
Group Insurance, PAS and Billing systems, testing process and tools

Requirements Analysis  Gap Tracker and Query Tracker

Deliverables
 High level test scenarios to cover functional testing scope, including
Test Scenarios
business scenarios by Life Insurance Company #1 BAs

Requirements Traceability  Mapping of functional test scenarios and cases to PAS and Billing
Matrix capabilities

 Detailed test cases, execution steps, expected results and test data
Test Cases
requirements

 Execution plan covering batch schedule, test case ownerships and testing
Test Execution Plan
calendar

 Log of all the defects found during testing including their severity, priority,
Defect Logs
root cause and state at the time of reporting
 Daily and test completion report encompassing testing highlights and take-
Test Reports – Daily, Test
aways, metrics, defects, risks / impediments, feedback on system
Completion 17
improvements
Integration Testing
Capabilities

18
Life Insurance Company #1 Integration Testing Approach
Ensure an appropriate data flow through each interface integration and
to third party

Testing Types in Scope

Responsible
Testing Type Description
Group
Testing of each PBI development component, point-to-point,
Unit Developers
and IT integrated testing
Testing of each interface component and the integration
Deloitte Testing
Interface / Integration between them including error / exception handling and
Team
rollback
Deloitte Testing
Functional Testing of functional test scenarios triggered from Salesforce
Team
Deloitte Testing Testing of a subset of interface and functional scenarios, and
Regression / End-to-end
Team end-to-end data flow using automated tools (SoapUI)
Limited testing to confirm that the developed interfaces meet
Internal Client
Small-scale Performance expected performance expectations individually and end-to-
team
end

19
Integration Test Approach
Integration testing approach provides capabilities to test real-time, near
real-time and batch integrations

1. Test / Data 2. Automated 3. Output 4. Analysis

Preparation Execution and reviews and report

 Data input systems e.g.  Single batch / interface  Database insert / update  Utilities for automation
Salesforce 
 or workflow session UI systems /app validation and analysis
Flat file, XMLs   
 Real-time, Near real- Flat files, XMLs Defect management and
Database 
 time, Batch 3rd party file format reporting engine
3rd party file format

Real /
Pre- Post- Reporting
Near-Real
Scripts Scripts Engine
/ Batch

 Batch jobs and

 Production data copy
workflows  Automation using open  Updated test results
 Gold copy from  Scheduled / on-demand source tools, XML parser,  Defect logs
automation frameworks 
 execution for varied SQL scripts Test reports
XSD, XML schemas
testing needs

Automated tests and Batch scheduler / Processing and data Report triggers and
data creation automated workflows validation defect management

Key Accomplishments
 Automated 76% End-to-end SIT test cases  Reusable framework for web services and database
 94% reduction in End-to-end test execution effort type of validations
(108 hours manual hours vs 6 hours using automation)
 Matured automated test suite that is being executed 20

in new Sprints for regression purposes

Integration Testing Deliverables
Following artifacts were delivered as part of 2016 integration testing for
Life Insurance Company #1 Admin Program

Deliverable Description

 Overall test approach highlighting scope, test types, environments, tools,

Test Approach
reporting, communication and roles and responsibilities across scrum team

Testing Estimation  Reusable estimation models for integration testing types

Models  Baseline testing estimates pipeline / interface development

 High level test scenarios to cover integration testing scope, including scenarios
Test Scenarios
by Life Insurance Company #1 BAs and developers

Requirements  Mapping of functional test scenarios and cases to integration system / data
Traceability Matrix requirements

 Detailed test cases, execution steps, expected results and test data requirements
Test Cases
including automated regression scripts

 Log of all the defects found during testing including their severity, priority, root
Defect Logs
cause and state at the time of reporting

Test Reports – Sprint  Sprint completion reports are reviewed during Sprint demos
Completion, Test  Test completion to summarize the overall Life Insurance Company #1 Integration
Completion testing

21
Test Automation
Capabilities

22
Test Automation Capability Assessment Summary
The following was the outcome of the automation testing assessment
that we did for each test automation capability

Key Assessment Findings

Maturity
Capability  Holistic automation test plan was undefined
Level
 Client automation framework could be used
Usability by non-programmers but a robust
documentation was required
Maintenance  End to end testing, integrations, and service
virtualization tools were undefined
Reusability and Scalability  PAS automation POC was limited to basic tests
and framework features
Multi Application Testing  Regression approach and testbed did not exist

 Client automation framework had not been

Tools Integration
integrated with TFS
 Integration with CI / CD tools was undefined
Reporting and Troubleshooting
 iTAF only supported sequential execution on
Execution machines where the framework was set up
 Configuration management was undefined
Configuration Management

Key
Modular and Data-Driven
No Capability Capability Partially Capability Fully
Exists Implemented Implemented

Test Type Support and Testbed Maturity

23
Test Automation Capabilities
Key capabilities needed for Life Insurance Company #1’s test
automation were prioritized and developed

Capability Description
 Ability and ease of developing automated tests including by non-programmer users
Usability  User guides to help setup and install the frameworks, execution and development manuals

 Ability to make changes in the framework and scripts with minimum rework
Maintenance  Ability to support frequent releases /sprint

 Ability to reuse the framework, tests scripts and data across testing lifecycle
Reusability and Scalability  Ability to scale up and accommodate new requirements, tests, tools and applications

 Ability to run tests on multiple applications and technologies

System Agnostic  End to end automation testing across multiple integrated systems

 Integration with requirements, test and defect management tools

Tools Support  Ability to integrate with enterprise build and deployment tools (CI/CD)

Reporting and  Ability to generate execution reports in a user-friendly, consumable format

Troubleshooting  Ability to monitor, troubleshoot, and drill-down execution / application issues and defects

 Ability to execute test scripts on multiple physical and virtual machines

Execution Options  Support for concurrent, scheduled, on-demand test executions

 Ability to track and control changes in the scripts and framework

Configuration Management  Alignment with system development and source code management

 Ability to define framework and test modules that encapsulate logical use cases
Modular and Data-Driven  Ability to run tests with a varied range of test data

Test Type Support and  Ability to support various testing types such as User Story, integration, end to end, regression, etc.
Testbed Maturity  Clear definition of regression suite and availability of automated test scripts for various releases
24
Functional Test Automation Framework
The diagram depicts our automation framework for functional testing

Test Type
Coverage Framework Components Execution / Monitoring

Functional testing Systems Under Test (PAS, Billing, Etc.)

6 Manual Test Execution
Web UI testing 1 Manual Test Design

Mobile and cross 5

Presentation

Common Object
browser testing Repository TDD Approach
Utilities
Layer

UAT/ Model office

testing Web Utilities
2 7 8
CI / CD SCM

Exception Handling / Logging

System testing
Business Test automation triggers
Driver
Workflow
Business

Incremental
Layer

Integration testing
9 Parallel
Business Execution

Smoke / E2E Regression Function Library

testing
3
10 Reporting
Data validation Input / Engine
Data Access
testing Output
Components
Helpers
Verify test results
Layer
Data

Database validation Excel / XML

testing Utilities
11 Update Results /
DB Data Excel Data Defect Management
Forms Validation 4 Sources Sources

Key
1 Manual Test Design 4 Test data components 7 Integration with CI / CD 10 Test execution reports

2 UI components and utilities 5 Exception handling and logging 8 TFS for source code management 11 Defect management
25

3 Business / functional components 6 Manual Test Execution 9 Multi-machine execution System-dependent components
Integration Test Automation Framework
The diagram depicts Integration testing capabilities

Execute Test Type

Integration Test Scripts
and Monitor Coverage
1 Service Provider

9
Request Response /
WSDL / Source Unit testing
Template Input

Batch Testing

2 Request Generator
10 Response /
Near real-time Testing
Result
5 7 Analysis
Payload Response
Real Time testing
Driver
Script
6 8 11 E2E Integration
Logging Results Defect testing
Management

Regression Testing
3 Test Generation

Batch Performance
Testing
Team Foundation Server
4
Test Case
Test Data
Selection

Key
1 Service provider for Web Service 4 Data generation tools 7 Response / interface output 10 Result analysis

2 Request XML / files generation 5 Request / response payloads 8 Execution results 11 Defect management

3 Test case ready to execute 6 Execution logs for troubleshooting 9 Test input
26
 Automation Development and Execution flow
The automation test development and execution flow is shown below

Test Development Test Execution

Run deployment scripts on hub

Business CI Servers CI server compiles and builds
Requirement scripts and sends it to the hub
4 5
s machine Hub
machine

Send test results

ish
bl ts The hub sends the consolidated
Business Pu sul result back to the CI server
Test Data re Execute
Function automated
Matrix
Catalog scripts
6

Stakeholders

Reporting
Manual Test Cases The automation test execution
results are sent to QAs and
Stakeholders Execution on node machines
1 2
Select test Hub sends the tests to
Green 3
case for each node and when it is
scripts
automation Source code executed, the test results
TDD
Script verification on UI: repository are sent back to the hub
Automation engineer
verifies the script and
Testers submits it to the code
repository
Automation team collaborates with development
to get the ‘page source’ and mock-ups Smoke test will be executed after every build using Continuous Integration

Key
1 Test case selection for automation 3 Automation code repository 5 Machine that initiates test execution

2 Test-driven automation development 4 CI Server facilitates CI / CD 6 Machine that executes automated scripts
27
Test Automation Deliverables
Following artifacts were delivered as part of test automation for Life
Insurance Company #1 Admin Program

Deliverable Description

Test Automation  Detailed report of Admin Program’s current state test automation capabilities
Assessment Report  Recommendations and a roadmap to enhance test automation capabilities

Test Automation Proof of  A proof of concept encompassing review of supplier’s automation framework
Concept  An executive summary of framework capabilities; recommended next steps

Functional Test  Test automation using Selenium framework for functional testing
Automation Framework  Integration with TFS and Jenkins, select test cases for Salesforce and PAS

Integration Test  Test automation using SoapUI framework for data pipeline testing
Automation Framework  Supports near real-time, real-time and batch integration testing

 End to end automated tests developed for Prospect and Opportunity data
Automated Regression
pipeline
Test Suite  76% of manual end to end test cases have been automated

 Framework architecture, developers’ guide, maintenance guide, setup guide

User Guides  Troubleshooting guide, API catalog and Business Function library

28
Test Automation
Demo

29
About Deloitte
Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee (“DTTL”), its network of member firms, and their related
entities. DTTL and each of its member firms are legally separate and independent entities. DTTL (also referred to as “Deloitte Global”) does not provide services to
clients. Please see www.deloitte.com/about for a detailed description of DTTL and its member firms. Please see www.deloitte.com/us/about for a detailed description of
the legal structure of Deloitte LLP and its subsidiaries. Certain services may not be available to attest clients under the rules and regulations of public accounting.

Copyright © 2017 Deloitte Development LLC. All rights reserved.

36 USC 220506