Evaluation Process

Step 1

Step 2

Step 3

Step 4

Step 5

Step6

Step 7
 Evaluation Process
Line items has been divided into following sections and weightages :
Functional Compliance (32%)
Technical Compliance (15%)
Analytics Capabilities (3%)
Regulatory Requirements (2%)
Service and Maintenance (10%)
Project Management (5%)
References (3%)
The above line items were further incorporated within the following 3 sections, Funtional, Technical and Others
In the VED analysis column, classify the requirement as Vital, Essential or Desirable based on its importance to the business
The weightage column will be automatically populated by a numerical figure depending on the category chosen in VED anal
VITAL --- 3
ESSENTIAL --- 2
DESIRABLE --- 1
Next step is to evaluate the vendor meeting the requirement. Based on the output, rate each of the the vendor on the corre
Highest score --- 5
Lowest score --- 1
Score can be anything from 1 to 5
Based on the VED weightage value and the score assigned to the vendor, Solution Capability Weighted Score will be calcula
Solution Capability Weighted Score = Capability score * VED Weightage
Overall Score and Section Weighted score will be calculated automatically for each section based on the section weightage(
Section Weighted Score = Section Score * Category weightage
The % target achieved by vendor would be calculated against the max score possible
Sr No Evaluation Matrix - Broad areas Weightage
1 Functional Compliance 32%
2 Technical Compliance 48%
3 Non-functional/Others 21%
100%

1 Functional Specifications

2 Non-functional Specifications
Functional requirements describe elements of the solution, and what the solution can and must do. Overall understanding of
help determine all the necessary components, features, and functionality of the solution.

Criteria that will be used to judge the operational aspect of the said system
 Advanced Analytics and

Sr No Heading
1 Functional Compliance
2 Technical Compliance
3 Non-functional/Others
152

Sr no Key Parameters
Functional Compliance
1 Managed ingestion
2 Metadata Creation
3 Data Governance

4 Data usage

5 ML model development

6 Data Visualization

7 Overall Solution capability

Functional Compliance

Technical Compliance
Data Ingestion
Machine learning capability

Data Security and auditing

Overall Weightage
Technical Compliance

Non-functional/Others
Vendor competency to deliver, explain and training

Monitoring Strategy
Incident & Problem Management (Level of logs
available for debugging)
Audit Trails (Debugging, security etc.)
Backup and disaster recovery
Performance Parameters
Storage, Archival and Purging
Project completion timelines
Resource Allocations
Publicly available referenciable big data and data science use cases
Overall Weightage
India market share
Global market share
Others
 Advanced Analytics and Machine Learning solution Evaluation M

Weightage
32%
48%
20%
100%

Capability of the solution to create managed data pipeline

Capability of the solution to create/attach metadata as transformation logic is applied
Capability of the solution to show data lineage
Capability of the solution for maintaining data quality
Capability of the solution for maintaining data privacy
Capability of the platform for maintaining data access audit log
Capability of the solution for automated data lifecycle management
Capability of the solution for automatic data discovery and cataloguing
Capability of the solution for self service data extraction and usage
Capability of the solution for data search ability - Based on field name
Capability of the solution for providing data science work bench
Capability of the solution for statistical analysis
Capability of the solution to develop data science model development/deployment
Capability of the solution for building interactive visualizations
Capability of the solution for statistical analytical visualizations
Capability of the solution for creating dynamic reports
Overall functional completeness of the solution

Workflow driven managed data ingestion pipeline

Mechanism for data ingestion using both pull and push based mechanism
Mechanisms for failure and back pressure handling
Data reconciliation mechanism
Data wrangling and data preparation using visual UI
Data transformation capability for streaming data (e.g. transformed, derived and aggregated
data columns)
Capability of the solution for change data capture
Out of the box adapters for data ingestion from various sources
 Has a data science workbench for ML model development
Workbench sharing b/t developers
Workbench capability for statistical analysis
Open source ML library support
Multiple ML model deployment and monitoring
Ensemble model deployment capability
Model Version control
Capability for Champion / Challenger model deployment
Capability for feed backing loop and parameter auto tuning - Based on live data feed
Model deployment - Phased rollout
Automatic identification of out of sync models
Visualization - Interactive dashboards
Support for development of DNN/RNN algorithms
Out of the box telecom specific data science models
Data security considerations for the data in motion - From VIL DC to Cloud DC
Policy driven access control (file, table, column, data field masking capability)
Unauthorized access and audit log
Encryption key management and automated key rotation
Overall technical completeness of the solution

and data science use cases

ning solution Evaluation Matrix
Max Score % Req Met by % Req Met by
AWS Azure Google Achievable AWS Azure
0.00 0.00 0.00 1.61 0% 0%
0.00 0.00 0.00 2.40 0% 0%
0.00 0.00 0.00 0.99 0% 0%
0.00 0.00 0.00 5.00 0% 0%

Vital/Essential/
Desirable AWS ( max 5) Azure (max 5) Google (max 5) Max Score

Vital 3 15
Essential 2 10
Essential 2 10
Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
49 245

Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
Essential 2 10

Vital 3 15
Vital 3 15
Vital 3
Vital 3 15
Desirable 1 5
Essential 2 10
Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
Desirable 1 5
Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
Desirable 1 5
Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
Vital 3 15
73 350

Essential 2 10
Vital 3 15

Vital 3 15
Vital 3 15
Vital 3 15
Essential 2 10
Vital 3 15
Vital 3 15
Vital 3 15
Essential 2 10
Vital 3 15
Vital 3 15
Essential 2 10
30 150
% Req Met
by Google
0%
0%
0%
0%

Remarks
AWS Azure Google AWS Azure

0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0

0 0 0
0 0 0
0 0 0
0 0 0
0 0 0

0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0

0 0 0
0 0 0

0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
Google