Tech Support Team Handbook for Techonline

9/23/2019

© 2019 EdgeVerve Systems Limited. Strictly private and confidential. No part of this
document should be reproduced or distributed without the prior permission of EdgeVerve
Systems Limited.
Tech Support Team Handbook For Techonline call lodgment - FINCRM

Confidential Information
This document is confidential to EdgeVerve Systems Limited ("EdgeVerve") and the Bank. This
document contains information and data that EdgeVerve considers confidential and proprietary
(“Confidential Information”).
Confidential Information includes, but is not limited to, the following:
 EdgeVerve’ project management, support and quality processes
Any disclosure of Confidential Information to, or use of it by a third party (i.e., a party other than the
Bank), will be damaging to EdgeVerve. Ownership of all Confidential Information, no matter in what
media it resides, remains with EdgeVerve.

Page 2 of 13
Tech Support Team Handbook For Techonline call lodgment - FINCRM

Version Control

Version Author Reviewer Date Changes

1.0 Divya Gupta Sushma I 30-Sep-2019 Techonline call lodgment
guideline and help document

2.0 Divya Gupta Sushma I 03-Oct-2019 Changes in Error logs, FI and

performance issues

3.0

Page 3 of 13
Tech Support Team Handbook For Techonline call lodgment - FINCRM

Table of Contents
1. What this document is about............................................................................................... 5
2. Required Details....................................................................................................................6
2.1 Basic Information:..................................................................................................................6
2.2 Standard Definition of Severity Levels:...............................................................................7
2.3 Detailed components of ‘L2 Initial Analysis’.......................................................................8
2.4 Check list while lodging the call:..........................................................................................9

3. Logs to be collected............................................................................................................ 10
3.1 Application logs....................................................................................................................10
3.2 Server Logs..........................................................................................................................11
3.3 Database Logs.....................................................................................................................12
3.4 Logs to be collected for CRM-FI integration issues........................................................12
3.5 Logs to be collected for CRM Performance issues.........................................................13

Page 4 of 13
Tech Support Team Handbook For Techonline call lodgment - FINCRM

1. What this document is about

This document is intended for IT technical team of the bank. IT team of the bank should refer to this
document to understand the guidelines for lodging a call in techonline for 10.x version.
This document will assist those who lodge a call in Techonline – the Ticket Tracking system for
Finacle. This document explains the information that is expected to be captured while lodging a
Techonline call.

Page 5 of 13
Tech Support Team Handbook For Techonline call lodgment - FINCRM

2. Required Details
This section explains on the details that have to be included in TechOnline when a request is raised
by bank. The type of information is broadly classified into:
 Basic Information
 Detailed Information
Attachments
1.1 Basic Information:
These are the very basic type of information regarding the environment in which the issue is
observed.

Information Details
Product* Product in which the issue is observed [e.g. FINCRM]
Product Version* Finacle version in which the issue is observed. [e.g. 10.2.09]
Module name* Module in which the issue is reported [e.g. SALES]
Menu option involved* The menu options involved while the issue occurred. [e.g. CIF ] or batch
job or services where this issue is occurring

Issue Classification* The type of Issue [e.g. Clarification/ Customization / Product]

CFO* Customer Impact / Financial Impact / Operational Impact

Location Status* The Onsite area or environment where the issue Occurred [e.g.
Production/UAT]

Category* Classification of the type of issue. [e.g. Customization, Clarification,

Performance, Report etc.]
Severity* When an issue is logged the correct severity has to be assigned to the
issue. When the bank users logs the call in defect tracking system,
request to follow the guidelines provided under ‘Definition of Severity
Levels’ below.
Subject* Describe the problem in brief
Problem Description* Describe the problem in detail
Initial L2 Analysis* Details of first level of analysis captured by the Bank’s IT team. For
details, please check section 2.3

Page 6 of 13
Tech Support Team Handbook For Techonline call lodgment - FINCRM

1.2 Standard Definition of Severity Levels:

Severity Finacle Availability Business Impact

1. Directly reducing Bank’s ability to conduct
Severity – 1 More than 25% of the
business.
Baseline bank users affected.
2. Essential function not working.
Major Impact 3. Critical program aborts or stops.
4. Critical application unavailable.

1. Directly causing partial loss of Bank’s ability to

Severity – 2 5% to 25% of the Baseline
conduct business.
bank users affected
2. Essential function severely restricted.
Moderate 3. Severe service degradation.
Impact
1. Problem encountered has low impact on
Severity – 3 Multiple users but less than
business.
5% of the Baseline Bank
2. A non-critical problem in the System where the
Users affected
Minimal Impact bank will be able to continue to run the
application or a workaround is available.
1. Causing only inconvenience.
Severity – 4 Single Bank User affected.
2. No business impact.
No impact or impairment of
3. No Bank regulatory or revenue impact but
service.
No Impact alternate mechanism available.

Note: Above severities are descriptive and subject to change as per SLA agreement with the
respective customer.

Page 7 of 13
Tech Support Team Handbook For Techonline call lodgment - FINCRM

1.3 Detailed components of ‘L2 Initial Analysis’

This section mentions detailed information that is required as part of first level analysis of the issue /
requirement referred by bank. This will help for faster understanding and quicker resolution of the
issue.

Information Details
Simulation Steps If the issue is simulatable at bank, mention the exact steps with screen
shots. All the menu operations and the entity type details also should
be provided. Also clearly mention Area/Environment where the
simulation is done (production / UAT / MIS etc.)
Impact How many entities / accounts impacted

Frequency How frequently the error is occurring in production and the time when
this issue is first reported.

Recent Deployment Is the issue observed after any recent deployment? If yes, please share
details of the deployment

Error logs Systemout, Systemerr logs, FinacleCRM logs, error logs, fatal / fatal-
info logs, dumps as applicable. For details, please check the logs
section

Miscellaneous Here bank can provide any other details bank team has noticed /
suspecting for causing the error, as per their analysis. Any recent
changes in the system level or bank level settings / set-up or resources
can be mentioned here.
Data Details Related table data, reports, batch job details and transaction details to
be shared

Page 8 of 13
Tech Support Team Handbook For Techonline call lodgment - FINCRM

1.4 Check list while lodging the call:

1 Where is the reported issue occurring? Production / Test / UAT
2 If in Production, is it possible to simulate the same on Test/UAT Setup (Yes/No/NA)? .
If it is simulatable, request to share Detailed simulation steps along with screen snapshots .
(Simulation performed in UAQ/UAT/TEST)

If not simulatable, request to provide bank’s initial observations on the issue reported.

3 The time when the issue occurred (Is it during EOD/BOD or during the day)

4 Can you recall any such similar problem faced earlier?

5 Was any solution suggested to the problem that you had faced earlier. If yes, mention the
same.

6 Is the issue occurring after any recent patch movements, customized script in production? If
yes, mention the same (Call id , Exe name , Exe size etc )

7 Any recent changes done at Application level (E.g. Account level, product level, scheme
level, SCFM level)

8 Any recent changes done at Environment level (E.g. Environmental variable, configuration
files, property files, jsp , dll/jar etc.,)

9 Any recent changes done at OS/DB level (E.g. Memory tuning , Network related,
Permissions at table level ,Installation of OS/DB patches)

10 Reason for the performing the changes(at Application / Environment /OS/DB level)(Call id or
reference)

11 Request to specify if the issue is specific to a particular User/sol/Role/Account?

12 Is the issue occurring always or intermittently

13 Any pattern seen if the issue is intermittent.

14 Kindly zip the following files and upload the same in Techonline as attachment while lodging
the call.
1. Relevant Screen shots
Contents of ‘log’ folder (logs like systemout, systemerr, FinacleCRM, dblayer) (NOTE:
logs formed during the issue occurrence)
2. Issue relevant table details. Request to share the records for the affected/problematic
cases.
As a standard practice test server must be refreshed periodically to be in sync with production
server. Whenever any simulation is done in test server, request to ensure appropriate backups are
taken.

Page 9 of 13
Tech Support Team Handbook For Techonline call lodgment - FINCRM

3. Logs to be collected
There are various types of logs present in the application server, Web server, and Database
server. These logs are to be shared along with the issue description and initial analysis to
identify the cause of the defect based on the type of the issue being raised. There are 3
varieties of logs
1) Application logs
2) Server Logs
3) Database Logs
Separate logs need to be collected for CRM-FI integration and performance related issues.

1.5 Application logs

 Location of the Log: FinacleCRM.ear/FinacleCRM.war/log
 FinacleCRM_Log1.xml
 FinacleCRM_Log2.xml

 Enabling user level logs:

1- Go to Administration -> User Log Generator

2- Search for Login ID for which the logs have to be enabled.

3- Click on start logger
4- Click on Go
5- The Agent ID will get loaded in the Logger Status
6- Click on Start.
7- You will get a pop up with the Path of the logs
8- Simulate the issue.
9- Go to same tab and click on stop logger
10- Go to backend path - FinacleCRM.ear/FinacleCRM.wat/log/<folder named with AgentID>
11- Collect FinacleCRM_Log1.xml and FinacleCRM_Log2.xml

Page 10 of 13
Tech Support Team Handbook For Techonline call lodgment - FINCRM

1.6 Server Logs

WAS SERVER
 Location of Logs: Appserver/Profiles/ProfileName/logs. Location may change based on the
configuration.
 server1/Systemerr.log
 server1/Systemout.log
 FFDC logs

Enabling logs if they are not getting generated:

WAS Level Log Enabling.

1. Go to Logging and Tracing> Server name>JVM Logs, set the configuration as per your need.

WEBLOGIC SERVER
 Location of Logs: Weblogic Home/servers/server1/logs or Weblogic Home/bin. Location
may change based on the configuration.
 nohup.log
 Instrumentation.log

Page 11 of 13
Tech Support Team Handbook For Techonline call lodgment - FINCRM

JBOSS SERVER
 Location of Logs: jboss-as/server/bin/or jboss-as/server/. Location may change based on
the configuration.
 nohup.log

1.7 Database Logs

Bank need to send the correct logs based on the database that is deployed in the bank
location.

Database Type Log Definition

Oracle  AWR Report (4 reports, each collected for 30minutes -1-hour
duration
Note:

 Check for availability of license (Diagnostic and Tuning Task)

to generate AWR report.
 In case of RAC environment, AWR should be collected from
each instance instead of Global report.
 In case, if issue is only for small duration (in few
seconds/minutes), Share ASH report (1min each) during the
issue period.

 Output of table_stats.sql
Note: Table_stats.sql output should be collected for tables which are
identified to be problematic (based on query). Please input Full Table
Name (not synonym) when prompted.
SQL Server  SQLProfiler
 Utilization details from each server (both CPU and memory)
DB2 DSC logs, DB2Master logs and WLM report,

Procedure for collecting AWR Report/SQL Profiler is explained in the attached document.

1.8 Logs to be collected for CRM-FI integration issues

 Location of Logs: <FI_APP_PATH>/common/log/finint/fcrmmb
 ciData.log
 ciDebug.log
 ciMessage.log
 ciEvent.log

Page 12 of 13
Tech Support Team Handbook For Techonline call lodgment - FINCRM

 Location of Logs: <FI_APP_PATH>/common/log/fsb/fsbhttp

 instrumentationLog.log

 Enabling logs:
Logs should be set as Y in FIMASTER.FI_CONFIG_DATA table.

1.9 Logs to be collected for CRM Performance issues

1. Five sets of Thread dumps which should be taken at 5-10 sec interval at application server
level. Thread dumps need to be taken before restart.
2. Heap dump or thread dumps that got generated at the working folder automatically by the
WAS. (Heap dumps will be big in size, hence bank should have heap analyzer in any
machine along with live meeting setup so that our Archie team will analyze and provide
suitable suggestion. Getting reviewed from IBM by raising PMR request for heap analysis
can also be followed by bank)
Procedure for dumps generation is covered in the below document.

3. Utilization details from each server (both CPU and memory)

4. AWR Report (4 reports, each collected for 30minutes -1-hour duration)

 Screen loading/process related performance issue

Collect the file structure being used while loading the screen or during process completion

Page 13 of 13