Page |1

Table of Contents
GIR Recording Call flow...........................................................................................................................2
GIA Call flow ...........................................................................................................................................3
SIP Call Flow between different components .......................................................................................4
Troubleshooting Flow-chart...................................................................................................................5
GIR Health Checks ..................................................................................................................................6
1. Check RWS Status ………………......................................................................................................6
2. Check Cassandra Status..............................................................................................................6
3. Check ES Status ..........................................................................................................................8
4. Check RCS Status ........................................................................................................................9
5. Check RPS Status .......................................................................................................................10
6. Check Webdav Status ...............................................................................................................10
7. Check Interactionreceiver Status ............................................................................................ 11
8. Validate certificate matching ...................................................................................................11
9. Check Loadbalancing Status .....................................................................................................12
GIR Common Issues and its Required logs: .........................................................................................12
1. SM/GIR GUI showing up “Loading Error” while playing back Recording files ........................12
2. SM/GIR GUI not showing up new inbound calls .....................................................................13
3. Screen Recording not working..................................................................................................13
4. Agent endpoint not recording calls .........................................................................................14
5. MCP failing to post recordings to IR ........................................................................................15
6. MCP failing to post metadata to Recording Processor ...........................................................15
7. Recording Processor not receiving metadata from ICON .......................................................16
8. Recording Processor failing to submit recordings to RWS .....................................................16
9. RP failing to submit recordings to SpeechMiner IR.................................................................16
Other Failure Scenarios........................................................................................................................17
1. MCP failing Scenarios................................................................................................................17
2. RPS failing Scenarios ................................................................................................................18
Typical GIR Log Snippets ......................................................................................................................21
1. MCP log snippets showing PUT/POST .....................................................................................18
2. RPS log snippets showing the record processing and POSTs to RWS & SM............................22
3. RCS log snippet showing the request/playback.......................................................................23
4. RWS log snippet showing processing of POST ........................................................................24
5. RWS log snippet showing Playback .........................................................................................25
6. Successful log snippet for Screen Recording ...........................................................................25
7. Tracing Playback Issue .............................................................................................................26
GIR Monitoring ....................................................................................................................................23
Comprehensive list of all GIR Alarms ..................................................................................................23
Stopping and Starting Recording Components ...................................................................................24

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 Page |2

GIR Recording Call flow

1. When a call enters the contact center, SIP Server sends the call to MCP (via RM) to process the
recording.

2. MCP sends the mp3 recording file to WebDAV to store.

3. When MCP receives a successful notification that the file is stored, it sends the recording’s metadata
information to the Recording Processor Script.
4. The Recording Processor Script(RPS) parses the metadata received from MCP
and retrieves the corresponding metadata from the ICON database that was
previously provided by the SIP Servers.
5. Once the metadata is retrieved from the ICON database successfully, RPS sends the information
about the recording to Web Services.

6. Recording Web Services(RWS/GWS) stores the recording information to the Cassandra database.
7. Recording Web Services(RWS/GWS) tells the RPS that the recording information
is stored.
8. The RPS sends the recording file to the SpeechMiner.

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 Page |3

GIA Call flow

1. Fetcher: Takes unprocessed interaction data and metadata from the input folder (where the
Interaction Receiver / UConnector placed it after retrieving it from the recording system),
inserts it into the GIA database, prepares the data files for processing by GIA, and places it in the
store folder to await processing.
2. Call Recognizer: Processes call audio according to the requirements of the Program to which
the call belongs, by transcribing the text and identifying Topics and other events in it.
3. Indexer: Maintains an index of calls, metadata, and events, so it can be searched quickly.
4. Report caching: Runs reports that are included in active users' Views pages overnight so that
they can be displayed quickly in their widgets when the users open their Views pages; the
amount of time to store cached results is configured in the Reports.
5. Exploration: Performs the data analysis required for the Exploration feature of the web-
based interface.
6. Recategorizer: Assigns Categories to the processed interactions in accordance with the
Category definitions defined in the system.
7. Text Recognizer: Processes written interaction input data and identifies Topics and other
events in it.
8. Web server: Runs the GIA web-based interface.

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 Page |4

SIP Call Flow between different components

After Party A and Party B are connected and a recording request is made to the SIP Server, two sessions
are initiated; one session for each party to MCP. SIP Server first INVITEs with the Session Description
Protocol (SDP) offer from the connected parties to MCP, and a second reINVITE to MCP to get an SDP
offer from MCP. The offer from SIP Server is sent to the connected parties to proxy the media through
Media Control Platform. Once the media is established, Media Control Platform bridges the media
between the parties and writes the recording to a file on the disk. The Recording Server fetches and
indexes the recording after the call completes.

The following diagram describes the call flow (Media Server refers to Media Control Platform):

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 Page |5

Troubleshooting Flow-chart

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 Page |6

GIR Health Checks

1. Check RWS Status.

[root@HOSTD etc]# service gir status
Interaction Recording Web Services is running pid=10010
Version: "8.5.202.70"
[root@HOSTD etc]#

Verify version from command line or from a browser. Add -v for verbose.
curl -v http://<rws host>:<rws port>/api/v2/diagnostics/version
curl -v http://<LB host>/api/v2/diagnostics/version

[root@HOSTD ~]# curl https://HOSTD:8443/api/v2/diagnostics/version

://xx.xx.xx.x3/api/v2/diagnostics/version
{"statusCode":0,"version":"8.5.202.70"}
[root@HOSTD ~]#

[root@HOSTD ~]# curl http://xx.xx.xx.x3/api/v2/diagnostics/version

{"statusCode":0,"version":"8.5.202.70"}
[root@HOSTD ~]#

Access via browser:

No returned version means RWS is not started. Verify logs for errors, check that Cassandra and ES nodes were
running.

2. Check Cassandra Status.

Depending on your nodes and replication factor configuration, “Token” count and “Owns (effective)” will be
different from screenshot below. Check this link for different cassandra nodes and replication results.
https://www.ecyrd.com/cassandracalculator/

On each of the Cassandra nodes, navigate to its install bin directory and execute below to see if nodes are Up
and running.

./nodetool ring sipfs

./nodetool status

[root@HOSTD bin]# ./nodetool ring sipfs

Datacenter: DC1
==========
Address Rack Status State Load Owns Token
3074457345618258602
xx.xx.xx.x4 RAC1 Up Normal 2.53 MB 100.00% -9223372036854775808
xx.xx.xx.x3 RAC1 Up Normal 2.06 MB 100.00% -3074457345618258603
xx.xx.xx.x2 RAC1 Down Normal 2.65 MB 100.00% 3074457345618258602

[root@HOSTD bin]#

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 Page |7
[root@HOSTD bin]# ./nodetool status
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN xx.xx.xx.x3 2.06 MB 1 100.0% 67e418c3-e37b-4790-baa5-de06d17d159b RAC1
UN xx.xx.xx.x4 2.53 MB 1 100.0% 0ce25692-89c0-421f-9686-9b08ae336826 RAC1
DN xx.xx.xx.x2 2.65 MB 1 100.0% bbb35bf5-1640-4e09-a0df-ab49056b51f0 RAC1

[root@HOSTD bin]#

Above output result with below configuration. RWS is still running even if one node is down.
Cluster size = 3
Replication factor = 3
write_consistency_level = CL_QUORUM
read_consistency_level = CL_QUORUM

Cassandra DEBUG logging can be set inside the file /<Cassandra_install_directory/conf/logback.xml. Modify as
needed.

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 Page |8

3. Check Elasticsearch Status.

Verify status from command line or from a browser. Status should be green.
curl http://localhost:9200/_cluster/health?pretty=true
curl http://localhost:9200/_cat/indices?v

Indices used by GIR:

resourcesv2-<contact-center id>
call-recordingv2-<region>-<contact-center id>
screen-recordingv2-<region>-<contact-center id>

[root@HOSTD logs]# curl -XGET http://localhost:9200/_cluster/health?pretty=true

{
"cluster_name" : "escluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 30,
"active_shards" : 60,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}
[root@HOSTD logs]#

[root@HOSTD logs]# curl http://localhost:9200/_cat/indices?v

health status index pri rep docs.count docs.deleted store.size pri.store.size
green open statistics-2018-02-20 5 1 0 0 7.8kb 3.9kb
green open resourcesv2-e9d59b88-xxxx-4242-8f80-a374fda96a12 5 1 56 139 171.1kb 85.5kb
green open statistics-2018-02-21 5 1 0 0 7.8kb 3.9kb

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 Page |9
green open statistics-2018-02-22 5 1 0 0 7.7kb 3.9kb
green open call-recordingv2-us-e9d59b88-xxxx-4242-8f80-a374fda96a12 5 1 271 0 285.4kb 142.7kb
green open screen-recordingv2-us-e9d59b88-xxxx-4242-8f80-a374fda96a12 5 1 22 0 86.7kb 43.3kb
[root@HOSTD logs]#

If cluster status is yellow, verify if indices used by GIR is affected. So long as (resourcesv2-<contact-center id>,
call-recordingv2-<region>-<contact-center id>, screen-recordingv2-<region>-<contact-center id>) are green in
status means GIR ES is not affected.

4. Check RCS Status.

Verify status from command line or from a browser by querying its version. You may append -v to show verbose
result.

curl -v https://<rcs ip>:<rcs port>/rcs/version

curl -v http://<LB host>/rcs/version

[root@HOSTD logs]# curl https://hostc:8008/rcs/version; echo

8.5.075.71
[root@HOSTD logs]

[root@HOSTD logs]# curl http://xx.xx.xx.x3/rcs/version -v; echo

8.5.075.71
[root@HOSTD logs]#

If response to version is quite long, append date to curl query to capture how long RCS responds to version
query. Call rcs version from LB and RCS, then check further where the delay is coming from.

date; curl http://<LB host>/rcs/version -v; date

[root@HOSTD logs]# date; curl http://xx.xx.xx.x3/rcs/version -v ;date

Sun Mar 4 00:34:44 PST 2018
* About to connect() to xx.xx.xx.x3 port 80 (#0)
* Trying xx.xx.xx.x3...
* Connected to xx.xx.xx.x3 (xx.xx.xx.x3) port 80 (#0)
> GET /rcs/version HTTP/1.1
> User-Agent: curl/7.29.0
> Host: xx.xx.xx.x3
> Accept: */*
>

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 10
< HTTP/1.1 200 OK
< Date: Sun, 04 Mar 2018 08:34:45 GMT
< Server: Jetty(7.x.y-SNAPSHOT)
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Content-Length: 10
< Content-Type: text/plain;charset=ISO-8859-1
< Set-Cookie: JSESSIONID=RCS_2_8_5_075_718mcfyymvmwdc1k1bgz7xqwn7f.RCS_2_8_5_075_7;Path=/rcs
< Set-Cookie: ROUTEID=.RCS_2_8.5.075.7; path=/
<
* Connection #0 to host xx.xx.xx.x3 left intact
8.5.075.71Sun Mar 4 00:34:45 PST 2018
[root@HOSTD logs]#

5. Check RPS Status.

Open command prompt and check that RP ports are open
telnet <RP1_IP_Address> <RP1_Port>
telnet <RP2_IP_Address> <RP2_Port>
telnet <RP3_IP_Address> <RP3_Port>

Check if RPS is shown started in SCI, GA/GAX

Check below URL returns success page showing RPS version (say 8.x version)

http://<RPS_host:port>/api/version/ (note the trailing slash is required)

https://<LB host>/rp/api/version/

## LB Configuration, the reason api is called as ../rp/api/.. .

<Proxy balancer://rp>
BalancerMember https://hostg:8889
BalancerMember https://hostd:8889
</Proxy>
ProxyPass /rp/api balancer://rp/api

[root@HOSTD logs]# curl https://hostg:8889/api/version/; echo

Recording Processor Version 8.5
[root@HOSTD logs]#
[root@HOSTD logs]# curl http://xx.xx.xx.x3/rp/api/version/ >> LB IP
Recording Processor Version 8.5[root@HOSTD logs]#
[root@HOSTD logs]#

6. Check Webdav Status.

a. Open a putty ssh session to WebDAV Server and execute the following command: "service httpd
status"
b. Verify the service is running as shown below:

If it’s not running, you can use below commands to start/stop the service
“service httpd start”
“service httpd stop”

c. Open a browser with the below URL

http://<WebDAV_IP_Address/recordings/
• Login with <WebDAVusername>/<WebDAVpassword>
• Ensure above links shows up all recording files as shown below:

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 11

7. Check Interactionreceiver Status.

Navigate to the following Interaction Receiver URL and make sure that the page displays a ‘’Hello’’:
Example:

Also browse with Load Balancer VIP (if any) as below:

http://<load balancer vip>/interactionreceiver
You should see “Hello” again
a. Also, Interaction Receiver runs as an IIS and hence you should also check if IIS and its SM/IR
application pools are running on IR hosts
b. SpeechMiner logs can be accessed via Event Viewer - > Applications and Services Logs ->
SpeechMiner
c. It can also be accessed from launching the ULogger utility on these servers.

8. Validate certificate matching.

a. Ca_cert.pem -> CA cert host
b. XX_Priv_key.pem - > Host certificate key/ private key
c. XX_Cert.pem - > Host public certificate key
d. All clients should have CA cert
e. Hosts acting as a server must have all three (ca cert, host certificate key, host public key)

To validate CA against public key, issue below: Output of OK means match.

openssl verify -verbose -CAfile <cacert.pem> <public key>

To validate public and private key match, issue below: Both results should yield the same value:
openssl x509 -noout -modulus -in <public key> | openssl md5
openssl rsa -noout -modulus -in <private key> | openssl md5

Since public key is validated on first openssl command, private key is valid to CA as well after having the same
MD5 output from second validation.

[root@HOSTC conf.d]# openssl verify -verbose -CAfile /opt/GCTI/SecurityPack/gcti-ca/ca_conf/ca_cert.pem

/opt/GCTI/SecurityPack/gcti-ca/repository/06_HOSTB_cert.pem
/opt/GCTI/SecurityPack/gcti-ca/repository/06_HOSTB_cert.pem: OK
[root@HOSTC conf.d]#
[root@HOSTC conf.d]#
[root@HOSTC conf.d]#
[root@HOSTC conf.d]# openssl x509 -noout -modulus -in /opt/GCTI/SecurityPack/gcti-
ca/repository/06_HOSTB_cert.pem | openssl md5
(stdin)= 5328065dcea5d0384e887d9829c9a005
[root@HOSTC conf.d]#
[root@HOSTC conf.d]#
[root@HOSTC conf.d]# openssl rsa -noout -modulus -in /opt/GCTI/SecurityPack/gcti-
ca/repository/06_HOSTB_priv_key.pem | openssl md5
(stdin)= 5328065dcea5d0384e887d9829c9a005
[root@HOSTC conf.d]#

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 12
Certificate serial number matches certificate filename. Sample certificate loaded in Windows.

Regenerate certificate if results do not match.

9. Check Loadbalancer status

Service httpd status should be running
root@HOSTC ~]# service httpd status
Redirecting to /bin/systemctl status httpd.service
httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
Active: active (running) since Sat 2018-03-03 20:36:36 PST; 4 days ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 8066 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
Main PID: 31017 (httpd)
Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec"
CGroup: /system.slice/httpd.service
8069 /usr/sbin/httpd -DFOREGROUND
8070 /usr/sbin/httpd -DFOREGROUND
8071 /usr/sbin/httpd -DFOREGROUND
8072 /usr/sbin/httpd -DFOREGROUND
8073 /usr/sbin/httpd -DFOREGROUND
31017 /usr/sbin/httpd -DFOREGROUND

Mar 03 20:36:35 HOSTC systemd[1]: Starting The Apache HTTP Server...

Mar 03 20:36:36 HOSTC systemd[1]: Started The Apache HTTP Server.
Mar 04 03:33:01 HOSTC systemd[1]: Reloaded The Apache HTTP Server.
[root@HOSTC ~]#

GIR Common Issues and its Required logs:

1. SM/GIR GUI showing up “Loading Error” while playing back Recording
Files
Logs Required:
a. RWSlogs (Debug cloud.log preferred) and its application.yaml file.
b. RCS logs (Debug log preferred) and RCS config export
c. SM Web logs and UTOPY_LOG export (Windows event viewer, under Applications and
Services)
d. SpeechMiner SQL query output for “Select * from dbo.configServer and dbo.cmrsParams
tables”, IVR profile export and screenshot of SMConfig->Recording Tab.
e. Chrome Console logs enabled through “Settings->More tools->Developer tools”, click
preserve logs and either capture screenshot as shown below or right click and save to flat
file.
f. Issue Call UUID/Timestamp of call to trace the call in the above logs.

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 13
g. Load Balancer logs (access_log/error_log)

Additional Traces in case of connection issues between RWS, RCS and LB hosts are as below:

h. Fiddler traces on RWS/RCS/Load Balancer hosts

2. SM/GIR GUI not showing up new inbound calls

Logs Required:
a. Sip Server/RM/MCP Logs (Preferred in DEBUG level)
b. CallIDs (of the problematic call)/Timestamp of call/DN’s involved in call
c. Export of Recording IVR Profile config
d. Export of MCP config
e. Export of SIP Server config object
f. RPS Logs and rpconfig.cfg
g. RWS logs (cloud.log)
h. ICON DB Server logs
i. Load Balancer logs (access_log/error_log)

Additional Traces in case of connection issues between RWS, RCS and LB hosts are as below:
j. Fiddler traces on RWS/RPS/InteractionReceiver/Load Balancer hosts

3. Screen Recording not working

Below logs required from client where screen recording not working
a. Recorder.log, GSR.log, AuthenticationServer.log (Usually found in C:\Program Files
(x86)\GCTI\Genesys SR Service\Logs)
b. config.json (Usually found in C:\Program Files (x86)\GCTI\Genesys SR Service)
c. cloud.log (RWS log)
d. RMS logs

Kindly do initial verification on the following if not already set:

a. Verify config. json per agent workstation
[
{"name":"server","value":"http://<rws ip>:<rws port>"},
{"name":"logLevel","value":"debug"}
]
b. Check AuthenticationServer.log if agent upon login was authenticated on its loopback address
(127.0.0.1)
INFO : 2018/03/03 01:34:07Z.717 : AuthServer: Starting HTTP Authentication server on 127.0.0.1:8080
INFO : 2018/03/03 01:34:45Z.608 : AuthServer: New Authorization Header Received "Agent001"
INFO : 2018/03/03 01:34:46Z.592 : AuthServer: Logout request received for user None cookie
IQ6xHP6kAdaxySP2k5eyF42fS1o

c. Verify WDE/WWE user has below privilege

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 14
d. WDE/WWE application object was configured for screen recording options.

e. Verify screen-recording is enabled on tenant level

Go to Web Services Cluster object > Options > screen-recording-client > recordingWhen
Go to Web Services Cluster object > Options > screen-recording-client > server

4. Agent endpoint not recording calls

a. Check if the agent’s DN has record=true.
b. Check SIP application option > msml-record-support = true
c. Check RM and MCP logs at the time of the call for any errors. For example, did RM correctly find
an MCP resource to handle the recording?
d. Are there any Alarms showing in SCI related to recording failure?

e. Check SIP, RM and MCP logs to confirm if recording is being triggered.

Logs Required:

a. Sip Server/RM/MCP Logs (Preferred in DEBUG level)

b. CallIDs (of the problematic call)/Timestamp of call/DN’s involved in call
c. Export of Recording IVR Profile config
d. Export of MCP config
e. Export of SIP Server config object

5. MCP failing to post recordings to IR

a. Check the failure folder on both MCP servers to see if files are building up.
The location is: \\<MCP host>\e$\GCTI\Media_Server_8505040\cache\record\failed

b. Files building up in these locations indicate that the MCP is either unable to post audio files to the
SpeechMinerIR, or metadata files to the Recording Processor.
c. There are two types of files in this location – audio files (.wav) and metadata files (.json) for each
recording.

d. If these post operations are via the Load Balancer VIP address, the first check would be
whether the Load Balancer VIP is responding.

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 15
e. Try browsing to the load balanced URL for SpeechMiner, and also for Recording Processor from the
MCP host.

f. If the Load Balancer is available but recordings are still failing to Post from MCP, then check MCP logs
to see whether the failure is IR or Recording Processor.

g. You can also verify that audio files are appearing in the IR input folder or WebDav location by direct
observation:
• If MCP is failing to post audio files to the IR location/webdav location, and the Load Balancer is
unavailable – then it could indicate that the Load Balancer is failing to post the recordings to
IR – for example a communication or firewall problem between Load Balancer and IR
hosts.

h. Check Load Balancer logs to verify this.

Logs Required:
• MCP logs, IR logs (both preferably in DEBUG level)
• Call UUID/Timestamp of call to trace the call in the above logs.
• IVR profile export and screenshot of SMConfig->Recording Tab.
• Load Balancer logs (access_log/error_log)
• SM Web logs and EventViewer export (Windows event viewer, under Applications and
Services.

6. MCP failing to post metadata to Recording Processor

a. If MCP is failing to post metadata to Recording Processor, and the Load Balancer is available, then
check that the Recording Processor is showing as Started in SCI. All RPS in the cluster should be
checked.
b. Restart each Recording Processor from SCI and verify that RPS version can be queried.
c. Check RP logs after restart for any
d. Once MCP is successfully posting to RP – you should see the recording is received in the RP log.

Logs Required:
• Sip Server/RM/MCP Logs (Preferred in DEBUG level)
• CallIDs (of the problematic call)/Timestamp of call/DN’s involved in call
• Export of Recording IVR Profile config
• Export of MCP config
• Export of SIP Server config object
• RPS Logs and rpconfig.cfg
• RWS logs (cloud.log)
• ICON DB Server logs
• Load Balancer logs (access_log/error_log)

7. Recording Processor not receiving metadata from ICON

a. Check the RP logs for successful connection to ICON databases
b. If any errors, check if ICON DB Servers are also up, and also check their logs
Logs Required:

• Debug ICON DB Server logs

• Debug RPS logs and config export of RPS

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 16

8. Recording Processor failing to submit recordings to RWS

a. Follow RPS Health Checks as detailed above
b. Follow RWS Health Checks as detailed above
c. Follow Cassandra Health Checks as detailed above
Logs Required:
• Sip Server/RM/MCP Logs (Preferred in DEBUG level)
• CallIDs (of the problematic call)/Timestamp of call/DN’s involved in call
• Export of Recording IVR Profile config
• RPS Logs and rpconfig.cfg
• RWS logs (cloud.log)
• ICON DB Server logs
• Load Balancer logs (access_log/error_log)

9. RP failing to submit recordings to SpeechMiner IR

a. Check the RP log for any errors relating to post failures to SM to confirm whether this is the
problem.
b. Check if SM Interaction Receiver is running, by using the following URLs:
http://<IR host>/interactionreceiver (Should see "hello")
via Load Balancer:
http://<load balancer vip>/interactionreceiver (Should see "hello")

c. Interaction Receiver runs as an IIS website – so you should also check if IIS webserver and “IR”
application pool is running on IR host.
d. SpeechMiner logs can be viewed in Event Viewer - > Applications and Services Logs -
SpeechMiner
e. They can also be accessed from launching the ULogger utility on those servers.
f. Access IR on web host as http://localhost/interactionreceiver to rule out connectivity issue.
Logs required:
• RPS Debug logs
• IR Debug logs
• Load Balancer logs if any (access_log/error_log)

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 17

Other Failure Scenarios

1. MCP failing Scenarios
What if MCP fails to PUT the recording to WebDAV or POST to RPS?

MCP will first attempt to PUT to WebDAV. After a successful attempt to WebDAV, MCP will send the metadata to
RPS. Until both WebDAV and RPS are successfully completed, the metadata json and recording file stay in the
cache directory. MCP keeps track of retry attempts and when all the attempts have exhausted MCP moves the
file to the failed directory.

In case of failure on MCP, would recovery be possible?

Yes. Added in release 8.5.216.01 is the LVR script to automate MCP handling for failure. Use MCP LVR script and
change necessary parameters on MCP_premise.properties.
Connection to Configuration Server
MCP connection to webdav
MCP connection to RPS

mcp.minimumRecordingAge=1

This property is the minimum age, in hours, of a recording that will be processed. Any recordings newer than this
time will not be processed.
The default value is 24

Set TLS certificate if secure transport is enabled.

lvrrecovery.webDAV.<tenantname>.recoverable.trustedCA
lvrrecovery.webDAV.<tenantname>.unrecoverable.trustedCA
mcp.rpTrustedCA

Handling Failed Recordings

For handling MCP post failures in the cache directory:
a. Find the MCP's $InstallationRoot$/cache/record/failed directory for the following file types:
[audiofilename]-mp3.bin-RWSmetadata.json
[audiofilename].mp3.bin
[audiofilename].pem (provided only when encryption is enabled)

The audio file name can have extensions of .mp3 or .mp3.bin. (.bin when recordings are encrypted)

b. LVR will attempt to recover and upload to recording storage, to RPS and will then delete records on MCP
cache failed directory.

Sample:
2018-03-06 17:33:04 INFO - Processing metadata file C:\Program
Files\GCTI\MCP_2\cache\record\failed\4O2J4BKS2L29HAATFT19NRQ20K00005S_2018-03-07_00-24-36-008B01A9-
10005CA9-00000001.mp3.bin-RWSmetadata.json
2018-03-06 17:33:04 INFO - Processing metadata file: C:\Program
Files\GCTI\MCP_2\cache\record\failed\4O2J4BKS2L29HAATFT19NRQ20K00005S_2018-03-07_00-24-36-008B01A9-
10005CA9-00000001.mp3.bin-RWSmetadata.json
2018-03-06 17:33:04 INFO - C:\Program
Files\GCTI\MCP_2\cache\record\failed\4O2J4BKS2L29HAATFT19NRQ20K00005S_2018-03-07_00-24-36-008B01A9-
10005CA9-00000001.mp3.bin is encrypted
2018-03-06 17:33:04 INFO - Updated metadata: {"mediaFiles":[{"certAlia……
……….
-36","ani":"733686868","callUuid":"4O2J4BKS2L29HAATFT19NRQ20K00005S"},"tenant":"Environment"}]}
2018-03-06 17:33:04 INFO - Updating metadata with webDAV path
https://HOSTD/recordings/4O2J4BKS2L29HAATFT19NRQ20K00005S_2018-03-07_00-24-36-008B01A9-10005CA9-
00000001.mp3.bin
2018-03-06 17:33:04 INFO - Updated mediaDescriptor:
{"path":"https://HOSTD/recordings/4O2J4BKS2L29HAATFT19NRQ20K00005S_2018-03-07_00-24-36-008B01A9-10005CA9-
00000001.mp3.bin","storage":"webDAV"}

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 18
2018-03-06 17:33:04 INFO - start uploading to WebDAV
2018-03-06 17:33:04 INFO - Attempting to recover recording up to 6 times
2018-03-06 17:33:04 INFO - Attempting to upload C:\Program
Files\GCTI\MCP_2\cache\record\failed\4O2J4BKS2L29HAATFT19NRQ20K00005S_2018-03-07_00-24-36-008B01A9-
10005CA9-00000001.mp3.bin to https://HOSTD/recordings/4O2J4BKS2L29HAATFT19NRQ20K00005S_2018-03-07_00-24-
36-008B01A9-10005CA9-00000001.mp3.bin on WebDAV
2018-03-06 17:33:04 INFO - TLS certificate will be validated with trusted ca at C:\Program Files\GCTI\ca_conf2\ca_cert.pem
2018-03-06 17:33:05 INFO - Successfully uploaded C:\Program
Files\GCTI\MCP_2\cache\record\failed\4O2J4BKS2L29HAATFT19NRQ20K00005S_2018-03-07_00-24-36-008B01A9-
10005CA9-00000001.mp3.bin to https://HOSTD/recordings/4O2J4BKS2L29HAATFT19NRQ20K00005S_2018-03-07_00-24-
36-008B01A9-10005CA9-00000001.mp3.bin
2018-03-06 17:33:05 INFO - Uploading metadata to https://HOSTG:8889/api/contact-centers//recordings/
2018-03-06 17:33:05 INFO - TLS certificate will be validated with trusted ca at C:\Program Files\GCTI\ca_conf2\ca_cert.pem
2018-03-06 17:33:05 INFO - Adding authorization to request
2018-03-06 17:33:07 INFO - Status code from upload attempt is 200
2018-03-06 17:33:07 INFO - Finished trying to post metadata to https://HOSTG:8889/api/contact-centers//recordings/
2018-03-06 17:33:07 INFO - Successfully recovered recording after 1 tries
2018-03-06 17:33:07 INFO - Successfully recovered C:\Program
Files\GCTI\MCP_2\cache\record\failed\4O2J4BKS2L29HAATFT19NRQ20K00005S_2018-03-07_00-24-36-008B01A9-
10005CA9-00000001.mp3.bin-RWSmetadata.json and its associated media files to RP and webDAV
2018-03-06 17:33:07 INFO - Deleting C:\Program
Files\GCTI\MCP_2\cache\record\failed\4O2J4BKS2L29HAATFT19NRQ20K00005S_2018-03-07_00-24-36-008B01A9-
10005CA9-00000001.mp3.bin-RWSmetadata.json
2018-03-06 17:33:07 INFO - Deleting C:\Program
Files\GCTI\MCP_2\cache\record\failed\4O2J4BKS2L29HAATFT19NRQ20K00005S_2018-03-07_00-24-36-008B01A9-
10005CA9-00000001.mp3.bin
2018-03-06 17:33:07 INFO - C:\Program
Files\GCTI\MCP_2\cache\record\failed\4O2J4BKS2L29HAATFT19NRQ20K00005S_2018-03-07_00-24-36-008B01A9-
10005CA9-00000001.mp3.bin is encrypted
2018-03-06 17:33:07 INFO - Deleting C:\Program
Files\GCTI\MCP_2\cache\record\failed\4O2J4BKS2L29HAATFT19NRQ20K00005S_2018-03-07_00-24-36-008B01A9-
10005CA9-00000001.mp3.pem
2018-03-06 17:33:07 INFO - 1 recordings were initially detected in C:\Program Files\GCTI\MCP_2\cache\record\failed
2018-03-06 17:33:07 INFO - 0 additional recording artifacts were found
2018-03-06 17:33:07 INFO - 1 recordings were successfully recovered
2018-03-06 17:33:07 INFO - 0 recordings seemed recoverable, but were not recovered
2018-03-06 17:33:07 INFO - 0 unrecoverable files were found
2018-03-06 17:33:07 INFO - 0 metadata files and artifacts were skipped
2018-03-06 17:33:07 INFO - Failed to upload to S3 0 times
2018-03-06 17:33:07 INFO - Failed to upload to WebDAV a total of 0 times
2018-03-06 17:33:07 INFO - Failed to upload to recoverable WebDAV 0 times for tenant: Environment
2018-03-06 17:33:07 INFO - 0 recordings are still in C:\Program Files\GCTI\MCP_2\cache\record\failed

2. RPS failing Scenarios

What if RPS fails to retrieve metadata from ICON or POST to RWS or SpeechMiner?
We know that RPS keeps the metadata that still needs to be processed in the rpqueue.db (sqlite3
database), but:

a. What would be the recover procedure for those scenarios where RPS failed to retrieve metadata
from ICON and the icon_retries limit has been reached? Is there a way to reset the
icon_retries counter and force RPS to try to retrieve metadata from ICON again?
In this case RP considers this call with no metadata and will simply process the recording as is
and POST to RWS and SM. This covers the case for trunk-to-trunk calls where there is no ICON
data.

b. What would be the recover procedure for those scenarios where RPS was able to POST metadata
to RWS, but failed to post metadata to SM and the "post_retries" limit has been reached?
There will be an error level log in rp log file. The json information should be there as info log. Run
RPS LVR Recovery Script to automatically POSTs failed .json to interactionreceiver.

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 19
Sample Snippet:
2018-03-07 21:11:44 INFO - Processing metadata file /opt/GCTI/Applications/RPS_Aus/failed/e9d59b88-xxxx-4242-8f80-
a374fda96a12/4O2J4BKS2L29HAATFT19NRQ20K000063_20180307-210301_1.json
2018-03-07 21:11:44 INFO - Attempting to recover recording up to 6 times
2018-03-07 21:11:44 INFO - Uploading metadata to https://HOSTD:8443/internal-api/contact-centers/e9d59b88-xxxx-4242-
8f80-a374fda96a12/recordings
2018-03-07 21:11:44 INFO - TLS certificate will be validated with trusted ca at /opt/GCTI/Certs/ca_cert.pem
2018-03-07 21:11:44 INFO - Ops credentials specified. Adding Authorization header to request
2018-03-07 21:11:45 INFO - initial attempt to upload metadata to RWS status code is 200
2018-03-07 21:11:45 INFO - Successfully posted metadata to RWS after 1 tries
2018-03-07 21:11:45 INFO - rp.SpeechMiner_uri parameter found in /opt/GCTI/Applications/RPS_Aus/failed/e9d59b88-xxxx-
4242-8f80-a374fda96a12/4O2J4BKS2L29HAATFT19NRQ20K000063_20180307-210301_1.json. Uploading to SpeechMiner IR
at https://HOSTB/interactionreceiver/
2018-03-07 21:11:45 INFO - Uploading metadata to https://HOSTB/interactionreceiver/
2018-03-07 21:11:45 INFO - TLS certificate will be validated with trusted ca at /opt/GCTI/Certs/ca_cert.pem
2018-03-07 21:11:45 INFO - Adding authorization to request
2018-03-07 21:11:46 INFO - Status code from upload attempt is 200
2018-03-07 21:11:46 INFO - Finished trying to post metadata to https://HOSTB/interactionreceiver/
2018-03-07 21:11:46 INFO - Successfully posted metadata to SpeechMiner after 1 tries
2018-03-07 21:11:46 INFO - SpeechMiner IR Status: 200
2018-03-07 21:11:46 INFO - Successfully uploaded /opt/GCTI/Applications/RPS_Aus/failed/e9d59b88-xxxx-4242-8f80-
a374fda96a12/4O2J4BKS2L29HAATFT19NRQ20K000063_20180307-210301_1.json to all destinations
2018-03-07 21:11:46 INFO - Deleting /opt/GCTI/Applications/RPS_Aus/failed/e9d59b88-xxxx-4242-8f80-
a374fda96a12/4O2J4BKS2L29HAATFT19NRQ20K000063_20180307-210301_1.json

c. What would be the recovery procedure for those scenarios where RPS failed to POST metadata to
RWS, but was able to post metadata to SM and the "post_retries" limit has been reached?
RP won’t post to SM until metadata is sent to RWS.

d. What Happens if Recording Processor fails to send recordings to SpeechMiner or Web Services?
If Recording Processor fails to send recordings to SpeechMiner or Web Services for any reason,
Recording Processor will continue trying to send these recordings until it exhausts the number
of retries configured. When the number of retries have been exhausted, Recording Processor
logs an INFO message to the log file with the corresponding JSON content that is not submitted
to SpeechMiner or Web Services. The following steps show how to search for the specific
messages in the log file to recover the post failure to SpeechMiner and Web Services.

Prerequisites:
Make sure that the Recording Processor logging is set to the INFO level.
Important: Metadata for the same call could be sent multiple times to Web Services or
SpeechMiner if the call is transferred. Therefore, even if a given metadata record is not sent
upstream, the call may still exist inthe Web Services or SpeechMiner database.

Sample Snippet:
RPS Failed to RWS
2018-03-07 22:04:31,094 RWS_client DEBUG Failed to GET record for CSRF token
https://HOSTD:8443/api/v2/diagnostics/version [Errno 111] Connection refused
2018-03-07 22:04:31,094 process_metadata INFO Response: 0. Could not POST metadata to RWS:
4O2J4BKS2L29HAATFT19NRQ20K000068
2018-03-07 22:04:31,325 metadata_client DEBUG Failed to GET record from https://HOSTD:8443/api/v2/ops/contact-
centers/%s/recordings [Errno 111] Connection refused
2018-03-07 22:04:31,325 process_metadata INFO POST to SM skipped because of incomplete GWS information. Will retry
POST again later: 4O2J4BKS2L29HAATFT19NRQ20K000068

Running RPS LVR Script

2018-03-07 22:12:18 INFO - Processing metadata file /opt/GCTI/Applications/RPS_Aus/failed/e9d59b88-xxxx-4242-8f80-
a374fda96a12/4O2J4BKS2L29HAATFT19NRQ20K000068_20180307-220431_1.json
2018-03-07 22:12:18 INFO - Attempting to recover recording up to 6 times
2018-03-07 22:12:18 INFO - Uploading metadata to https://hostd:8443/internal-api/contact-centers/e9d59b88-xxxx-4242-
8f80-a374fda96a12/recordings

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 20
2018-03-07 22:12:18 INFO - TLS certificate will be validated with trusted ca at /opt/GCTI/Certs/ca_cert.pem
2018-03-07 22:12:18 INFO - Ops credentials specified. Adding Authorization header to request
2018-03-07 22:12:20 INFO - initial attempt to upload metadata to RWS status code is 200
2018-03-07 22:12:20 INFO - Successfully posted metadata to RWS after 1 tries
2018-03-07 22:12:20 INFO - rp.SpeechMiner_uri parameter found in /opt/GCTI/Applications/RPS_Aus/failed/e9d59b88-xxxx-
4242-8f80-a374fda96a12/4O2J4BKS2L29HAATFT19NRQ20K000068_20180307-220431_1.json. Uploading to SpeechMiner IR
at https://HOSTB/interactionreceiver/
2018-03-07 22:12:20 INFO - Uploading metadata to https://HOSTB/interactionreceiver/
2018-03-07 22:12:20 INFO - TLS certificate will be validated with trusted ca at /opt/GCTI/Certs/ca_cert.pem
2018-03-07 22:12:20 INFO - Adding authorization to request
2018-03-07 22:12:21 INFO - Status code from upload attempt is 200
2018-03-07 22:12:21 INFO - Finished trying to post metadata to https:// HOSTB /interactionreceiver/
2018-03-07 22:12:21 INFO - Successfully posted metadata to SpeechMiner after 1 tries
2018-03-07 22:12:21 INFO - SpeechMiner IR Status: 200
2018-03-07 22:12:21 INFO - Successfully uploaded /opt/GCTI/Applications/RPS_Aus/failed/e9d59b88-xxxx-4242-8f80-
a374fda96a12/4O2J4BKS2L29HAATFT19NRQ20K000068_20180307-220431_1.json to all destinations
2018-03-07 22:12:21 INFO - Deleting /opt/GCTI/Applications/RPS_Aus/failed/e9d59b88-xxxx-4242-8f80-
a374fda96a12/4O2J4BKS2L29HAATFT19NRQ20K000068_20180307-220431_1.json
2018-03-07 22:12:21 INFO - 0 additional recording artifacts were found
2018-03-07 22:12:21 INFO - 2 recordings were successfully recovered

Check RPS LVR logs should running the script failed to recover the metadata. Verify configuration
made on RPS_premise.properties.

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 21

Typical GIR Log Snippets

MCP log snippets showing PUT/POST

MCP PUT to Webdav

2018-03-07 19:29:32.180 DBUG 008B01AA-10005CDD-0000006E-008B01AA-10005CDD 1632 01F00000
FMThreadQueue.cxx:1273 Processing PUT request, URL:
https://HOSTD/recordings/4O2J4BKS2L29HAATFT19NRQ20K000069_2018-03-08_03-30-21-008B01AA-10005CDD-
00000001.mp3
2018-03-07 19:29:32.181 DBUG 008B01AA-10005CDD 5304 0B100000 MPCConference.cxx:2172 MPCConference::RecordDone
nConfID=3006C118 this=0000000024E148E0 : end of recording
….
2018-03-07 19:29:32.832 DBUG 00000000-00000000 1632 01F00000 FMResponseData.cxx:341 HTTP/1.1 201 Created
2018-03-07 19:29:32.832 DBUG 00000000-00000000 1632 01F00000 FMResponseData.cxx:360 Date: Thu, 08 Mar 2018
03:29:36 GMT
2018-03-07 19:29:32.832 DBUG 00000000-00000000 1632 01F00000 FMResponseData.cxx:360 Server: Apache/2.4.6 (CentOS)
OpenSSL/1.0.2k-fips
2018-03-07 19:29:32.832 DBUG 00000000-00000000 1632 01F00000 FMResponseData.cxx:360 Location:
https://hostd/recordings/2018/03/08/03/4O2J4BKS2L29HAATFT19NRQ20K000069_2018-03-08_03-30-21-008B01AA-
10005CDD-00000001.mp3
2018-03-07 19:29:32.832 DBUG 00000000-00000000 1632 01F00000 FMResponseData.cxx:360 Content-Length: 281
2018-03-07 19:29:32.832 DBUG 00000000-00000000 1632 01F00000 FMResponseData.cxx:360 Content-Type: text/html;
charset=ISO-8859-1
2018-03-07 19:29:32.843 DBUG 008B01AA-10005CDD-0000006E-008B01AA-10005CDD 1632 01F00000
FMThreadQueue.cxx:461 Connection #70 to host hostd left intact

2018-03-07 19:29:32.843 DBUG 008B01AA-10005CDD-0000006E-008B01AA-10005CDD 1632 01F00000

FMThreadQueue.cxx:727 Request successful: uri=https://hostd/recordings/4O2J4BKS2L29HAATFT19NRQ20K000069_2018-03-
08_03-30-21-008B01AA-10005CDD-00000001.mp3
2018-03-07 19:29:32.843 DBUG 008B01AA-10005CDD-0000006E-008B01AA-10005CDD 1632 01F00000
FMThreadQueue.cxx:891 HTTP request completed
2018-03-07 19:29:32.843 DBUG 00000000-00000000 1632 01F00000 FMResponseData.cxx:433 Received fetch completed,
response code=201

MCP PUT to IR
2018-03-07 19:29:32.181 DBUG 008B01AA-10005CDD-0000006F-008B01AA-10005CDD 1632 01F00000
FMThreadQueue.cxx:1273 Processing PUT request, URL:
https://HOSTB/interactionreceiver/4O2J4BKS2L29HAATFT19NRQ20K000069_2018-03-08_03-30-21-008B01AA-10005CDD-
00000002.wav
…..
2018-03-07 19:29:32.208 DBUG 008B01AA-10005CDD-0000006F-008B01AA-10005CDD 1632 01F00000 FMUserRequest.cxx:684
PUT /interactionreceiver/4O2J4BKS2L29HAATFT19NRQ20K000069_2018-03-08_03-30-21-008B01AA-10005CDD-00000002.wav
HTTP/1.1
2018-03-07 19:29:32.208 DBUG 008B01AA-10005CDD-0000006F-008B01AA-10005CDD 1632 01F00000 FMUserRequest.cxx:579
Host: HOSTB
2018-03-07 19:29:32.208 DBUG 008B01AA-10005CDD-0000006F-008B01AA-10005CDD 1632 01F00000 FMUserRequest.cxx:579
Accept: */*
2018-03-07 19:29:32.208 DBUG 008B01AA-10005CDD-0000006F-008B01AA-10005CDD 1632 01F00000 FMUserRequest.cxx:579
Content-Length: 548284
2018-03-07 19:29:32.208 DBUG 008B01AA-10005CDD-0000006F-008B01AA-10005CDD 1632 01F00000 FMUserRequest.cxx:579
Date: Thu, 08 Mar 2018 03:29:32 +0000
2018-03-07 19:29:32.208 DBUG 008B01AA-10005CDD-0000006F-008B01AA-10005CDD 1632 01F00000 FMUserRequest.cxx:579
Authorization: Basic QWRtaW5pc3RyYXRvcjpBYmJhMTIxOTg3
…
2018-03-07 19:29:32.788 DBUG 008B01AA-10005CDD-0000006F-008B01AA-10005CDD 1632 01F00000
FMThreadQueue.cxx:727 Request successful:
uri=https://HOSTB/interactionreceiver/4O2J4BKS2L29HAATFT19NRQ20K000069_2018-03-08_03-30-21-008B01AA-10005CDD-
00000002.wav
2018-03-07 19:29:32.788 DBUG 008B01AA-10005CDD-0000006F-008B01AA-10005CDD 1632 01F00000
FMThreadQueue.cxx:891 HTTP request completed
2018-03-07 19:29:32.788 DBUG 00000000-00000000 1632 01F00000 FMResponseData.cxx:433 Received fetch
completed, response code=200

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 22
MCP POST to RPS
2018-03-07 19:29:32.843 DBUG 008B01AA-10005CDD-00000070-008B01AA-10005CDD 1632 01F00000
FMThreadQueue.cxx:1273 Processing POST request, URL: https://HOSTG:8889/api/contact-centers//recordings/
2018-03-07 19:29:32.843 DBUG 008B01AA-10005CDD-0000006E-008B01AA-10005CDD 1632 01F00000
FMThreadQueue.cxx:1385 Processing ReleaseRequest

2018-03-07 19:29:33.086 DBUG 008B01AA-10005CDD-00000070-008B01AA-10005CDD 1632 01F00000

FMThreadQueue.cxx:461 Connected to HOSTG (192.168.62.104) port 8889 (#72)

2018-03-07 19:29:33.086 DBUG 008B01AA-10005CDD-00000070-008B01AA-10005CDD 1632 01F00000

FMThreadQueue.cxx:461 ALPN, offering http/1.1

2018-03-07 19:29:33.086 DBUG 008B01AA-10005CDD-00000070-008B01AA-10005CDD 1632 01F00000

FMThreadQueue.cxx:461 Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH

2018-03-07 19:29:33.086 DBUG 008B01AA-10005CDD-00000070-008B01AA-10005CDD 1632 01F00000

FMThreadQueue.cxx:461 successfully set certificate verify locations:

2018-03-07 19:29:33.086 DBUG 008B01AA-10005CDD-00000070-008B01AA-10005CDD 1632 01F00000

FMThreadQueue.cxx:461 CAfile: C:\Program Files\GCTI\ca_conf2\ca_cert.pem
CApath: none
..
2018-03-07 19:29:33.568 DBUG 008B01AA-10005CDD-00000070-008B01AA-10005CDD 1632 01F00000 FMUserRequest.cxx:684
POST /api/contact-centers//recordings/ HTTP/1.1
2018-03-07 19:29:33.568 DBUG 008B01AA-10005CDD-00000070-008B01AA-10005CDD 1632 01F00000 FMUserRequest.cxx:579
Host: HOSTG:8889
2018-03-07 19:29:33.568 DBUG 008B01AA-10005CDD-00000070-008B01AA-10005CDD 1632 01F00000 FMUserRequest.cxx:579
Accept: */*
2018-03-07 19:29:33.568 DBUG 008B01AA-10005CDD-00000070-008B01AA-10005CDD 1632 01F00000 FMUserRequest.cxx:579
Content-Length: 830
2018-03-07 19:29:33.568 DBUG 008B01AA-10005CDD-00000070-008B01AA-10005CDD 1632 01F00000 FMUserRequest.cxx:579
Date: Thu, 08 Mar 2018 03:29:32 +0000
2018-03-07 19:29:33.568 DBUG 008B01AA-10005CDD-00000070-008B01AA-10005CDD 1632 01F00000 FMUserRequest.cxx:579
Authorization: Basic dXNlcjpwYXNzd29yZA==
2018-03-07 19:29:33.568 DBUG 008B01AA-10005CDD-00000070-008B01AA-10005CDD 1632 01F00000 FMUserRequest.cxx:579
Content-Type: application/json
2018-03-07 19:29:33.568 DBUG 008B01AA-10005CDD-00000070-008B01AA-10005CDD 1632 01F00000 FMUserRequest.cxx:579
X-Genesys-FMSessionID: 008B01AA-10005CDD-00000070-008B01AA-10005CDD
2018-03-07 19:29:33.568 DBUG 008B01AA-10005CDD-00000070-008B01AA-10005CDD 1632 01F00000
FMThreadQueue.cxx:461 upload completely sent off: 830 out of 830 bytes
2018-03-07 19:29:33.820 DBUG 00000000-00000000 1632 01F00000 FMResponseData.cxx:341 HTTP/1.1 200 OK

2. RPS log snippets showing the record processing and POSTs to RWS & SM

RPS time between “Received metadata” and “Processing record” should not be greater than 1 min as this is
considered too long. Verify what/ where is causing delay in processing.

2018-03-07 22:29:33,193 recording_process DEBUG Received metadata: 4O2J4BKS2L29HAATFT19NRQ20K000069

2018-03-07 22:29:33,194 recording_process DEBUG Storing metadata: 4O2J4BKS2L29HAATFT19NRQ20K000069
2018-03-07 22:29:43,210 process_metadata DEBUG processing record: 4O2J4BKS2L29HAATFT19NRQ20K000069
..
2018-03-07 22:29:43,211 event_collector DEBUG Collecting call segment information for
4O2J4BKS2L29HAATFT19NRQ20K000069 from switch CanadaSwitch

2018-03-07 22:29:44,406 event_collector DEBUG Call events metadata: [{'calluuid': '4O2J4BKS2L29HAATFT19NRQ20K000069', ..

…..
2018-03-07 22:29:44,651 process_metadata DEBUG Call ACW status has been collected --
4O2J4BKS2L29HAATFT19NRQ20K000069...
2018-03-07 22:29:44,651 process_metadata DEBUG Sending to RWS: 4O2J4BKS2L29HAATFT19NRQ20K000069
2018-03-07 22:29:47,411 metadata_client DEBUG POST: https://hostd:8443/internal-api/contact-centers/e9d59b88-xxxx-4242-
8f80-a374fda96a12/recordings: Response Status: 200

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 23
2018-03-07 22:34:59,349 metadata_client DEBUG GET: https://hostd:8443/api/v2/ops/contact-centers/e9d59b88-xxxx-4242-
8f80-a374fda96a12/recordings/4O2J4BKS2L29HAATFT19NRQ20K000069: Response Status: 200
2018-03-07 22:34:59,350 metadatarecord DEBUG merging records: 4O2J4BKS2L29HAATFT19NRQ20K000069 &
4O2J4BKS2L29HAATFT19NRQ20K000069
...
2018-03-07 22:34:59,350 process_metadata DEBUG Sending to SM: 4O2J4BKS2L29HAATFT19NRQ20K000069
2018-03-07 22:35:00,478 metadata_client DEBUG POST: https://HOSTB/interactionreceiver/: Response Status: 200

3. RCS log snippet showing the request/playback

2018-03-07T21:09:12.217 Dbg 09900 org.eclipse.jetty.server.Server REQUEST

/rcs/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/4O2J4BKS2L29HAATFT19NRQ20K00006A_2018-03-08_05-02-
08-008B01AA-10005CE1-00000001.mp3.bin on BlockingHttpConnection@31e62172,g=HttpGenerator{s=0,h=-1,b=-1,c=-
1},p=HttpParser{s=-5,l=10,c=0},r=1
2018-03-07T21:09:12.217 Dbg 09900 org.eclipse.jetty.server.session Got Session ID
RCS_2_8_5_075_7b8pa7or2dj7b1ibv62plb8h62.RCS_2_8_5_075_7 from cookie

Request received for

'/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/4O2J4BKS2L29HAATFT19NRQ20K00006A_2018-03-08_05-02-08-
008B01AA-10005CE1-00000001.mp3.bin':
(GET /rcs/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/4O2J4BKS2L29HAATFT19NRQ20K00006A_2018-03-08_05-
02-08-008B01AA-10005CE1-00000001.mp3.bin)@1517297691 org.eclipse.jetty.server.Request@5a70201b
servletPath:/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/4O2J4BKS2L29HAATFT19NRQ20K00006A_2018-03-
08_05-02-08-008B01AA-10005CE1-00000001.mp3.bin

2018-03-07T21:09:12.219 Dbg 09900 org.springframework.security.web.access.intercept.FilterSecurityInterceptor

Authorization successful

2018-03-07T21:09:12.221 Dbg 09900 com.genesyslab.rcs.controller.HomeController recordingId:

4O2J4BKS2L29HAATFT19NRQ20K00006A, mediaId: 4O2J4BKS2L29HAATFT19NRQ20K00006A_2018-03-08_05-02-08-008B01AA-
10005CE1-00000001.mp3.bin, entering playbackRecording
2018-03-07T21:09:12.221 Dbg 09900 com.genesyslab.rcs.controller.HomeController Calling RWS GET url:
https://HOSTD:8443/api/v2/ops/contact-centers/e9d59b88-xxxx-4242-8f80-
a374fda96a12/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A

2018-03-07T21:09:12.303 Dbg 09900 org.apache.http.impl.conn.DefaultClientConnection Sending request: GET

/api/v2/ops/contact-centers/e9d59b88-xxxx-4242-8f80-a374fda96a12/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A
HTTP/1.1
2018-03-07T21:09:12.303 Dbg 09900 org.apache.http.headers >> GET /api/v2/ops/contact-centers/e9d59b88-xxxx-4242-8f80-
a374fda96a12/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A HTTP/1.1

2018-03-07T21:09:12.430 Dbg 09900 com.genesyslab.rcs.media.MediaFile URL=https://HOSTD:8443/api/v2/ops/contact-

centers/e9d59b88-xxxx-4242-8f80-a374fda96a12/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/06521f55-7d52-
4f91-bca5-c1d53b7fa422.mp3, fetching media file

2018-03-07T21:09:12.729 Dbg 09900 org.eclipse.jetty.server.Server RESPONSE

/rcs/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/4O2J4BKS2L29HAATFT19NRQ20K00006A_2018-03-08_05-02-
08-008B01AA-10005CE1-00000001.mp3.bin 200

4. RWS log snippet showing processing of POST

03/16/2018 01:20:37.692 DEBUG [ops] [PxLvG0SZoXkTTIo/QSZ/jQ11el01csvsjoni2cmu4r7c9n4] [] [qtp1716918369-24] /internal-
api/contact-centers/e9d59b88-6155-4242-8f80-a374fda96a12/recordings c.g.g.v.a.c.c.CreateCallRecordingOpsController
Processing POST of call recording [{"status": 0, "retries": 0, "mediaFiles": [{"mediaId":
"ESECG0EV656234LL46RCGJLFVK00000M_2018-03-16_01-21-09-008B01AA-1000618B-00000001.mp3", "parameters":
{"username": "RSnippet", "agentId": "1601", "recordDN": "6001", "ani": "733686868", "dateTime": "2018-03-16T01:21:09Z",
"sipsAppName": "SIPServer", "connId": "007002ba36b3e016", "callUuid": "ESECG0EV656234LL46RCGJLFVK00000M", "dnis":
"6001", "rp.SpeechMiner_auth": "MPaladar:sa11", "id": "ESECG0EV656234LL46RCGJLFVK00000M_2018-03-16_01-21-09",

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 24
"rp.SpeechMiner_uri": "http://xx.xx.xx.xx.2/interactionreceiver/"}, "accessgroups": ["/KPaladar_Hierarchy"], "channels": "2",
"duration": "31922", "stopTime": "2018-03-16T01:20:16Z", "startTime": "2018-03-16T01:19:44Z", "mediaDescriptor": {"path":
"https://HOSTD/recordings/ESECG0EV656234LL46RCGJLFVK00000M_2018-03-16_01-21-09-008B01AA-1000618B-
00000001.mp3", "storage": "webDAV"},
......
{"deleted": {"WrapUpTime": "0"}}}, {"eventId": "2018-03-16
01:21:41.830_1521163301_ESECG0EV656234LL46RCGJLFVK00000M", "calluuid": "ESECG0EV656234LL46RCGJLFVK00000M",
"occurredAt": "2018-03-16T01:21:41Z", "event": "Data", "data": {"deleted": {"GSIP_RECORD": "ON"}}}], "callerPhoneNumber":
"733686868", "id": "ESECG0EV656234LL46RCGJLFVK00000M", "ccid": "e9d59b88-6155-4242-8f80-a374fda96a12"}]

03/16/2018 01:20:37.861 DEBUG [ops] [PxLvG0SZoXkTTIo/QSZ/jQ11el01csvsjoni2cmu4r7c9n4] [] [hystrix-ApiCreatePool-22]

/internal-api/contact-centers/e9d59b88-6155-4242-8f80-a374fda96a12/recordings c.g.g.v.c.r.c.CallRecordingRepository
putting record CallRecording(id=ESECG0EV656234LL46RCGJLFVK00000M, ccid=e9d59b88-6155-4242-8f80-a374fda96a12,
callerPhoneNumber=733686868, dialedPhoneNumber=6001, startTime=Thu Mar 15 18:19:44 PDT 2018,
mediaFiles=[CRMediaFile(mediaDescriptor=CRStorageMediaDescriptor(storage=webDAV, storageVersion=null,
path=https://HOSTD/recordings/ESECG0EV656234LL46RCGJLFVK00000M_2018-03-16_01-21-09-008B01AA-1000618B-
00000001.mp3, data=null, auth=null), folder=null, startTime=Thu Mar 15 18:19:44 PDT 2018, stopTime=Thu Mar 15 18:20:16 PDT
2018, callUUID=ESECG0EV656234LL46RCGJLFVK00000M, mfUUID=761c58e1-cc99-4348-986d-b2106b80c5d5,
mediaId=ESECG0EV656234LL46RCGJLFVK00000M_2018-03-16_01-21-09-008B01AA-1000618B-00000001.mp3,
type=audio/mp3, duration=31922, tenant=Environment, ivrprofile=NoEncryptionHTTPtoSM, size=126864,
parameters={dateTime=2018-03-16T01:21:09Z, agentId=1601, rp.SpeechMiner_auth=MPaladar:sa11,
rp.SpeechMiner_uri=http://xx.xx.xx.xx.2/interactionreceiver/, sipsAppName=SIPServer, recordDN=6001,
connId=007002ba36b3e016, dnis=6001, id=ESECG0EV656234LL46RCGJLFVK00000M_2018-03-16_01-21-09, ani=733686868,
......
data={added={GSRS_STATE=SRSScreenRecordingStateStarted}}), EventDescriptor(occurredAt=Thu Mar 15 18:21:10 PDT 2018,
calluuid=ESECG0EV656234LL46RCGJLFVK00000M, eventId=2018-03-16
01:21:10.840_1521163270_ESECG0EV656234LL46RCGJLFVK00000M, contact=null, event=Data,
data={added={IW_CaseUid=3b55eb2b-c26e-4371-9a8c-4998d323e836, IW_BundleUid=2c371d8c-1417-40e8-9794-
dfc0c04ad0db}}), EventDescriptor(occurredAt=Thu Mar 15 18:21:41 PDT 2018, calluuid=ESECG0EV656234LL46RCGJLFVK00000M,
eventId=2018-03-16 01:21:41.827_1521163301_ESECG0EV656234LL46RCGJLFVK00000M, contact=null, event=Data,
data={deleted={WrapUpTime=0}}), EventDescriptor(occurredAt=Thu Mar 15 18:21:41 PDT 2018,
calluuid=ESECG0EV656234LL46RCGJLFVK00000M, eventId=2018-03-16
01:21:41.830_1521163301_ESECG0EV656234LL46RCGJLFVK00000M, contact=null, event=Data,
data={deleted={GSIP_RECORD=ON}})], labels=[], nonDelete=false, callType=Inbound, region=us, deleteCandidate=false,
screenRecording=null, callRecordingId=null) with timestamp 1521163237861

03/16/2018 01:20:37.909 DEBUG [ops] [PxLvG0SZoXkTTIo/QSZ/jQ11el01csvsjoni2cmu4r7c9n4] [] [hystrix-ApiCreatePool-22]

/internal-api/contact-centers/e9d59b88-6155-4242-8f80-a374fda96a12/recordings c.g.g.v.c.r.c.CallRecordingRepository
putting call recording ESECG0EV656234LL46RCGJLFVK00000M to columnFamily cr_83b0510c2c2844058e976f713e861e05

03/16/2018 01:20:38.210 DEBUG [ops] [PxLvG0SZoXkTTIo/QSZ/jQ11el01csvsjoni2cmu4r7c9n4] [] [hystrix-ApiCreatePool-22]

/internal-api/contact-centers/e9d59b88-6155-4242-8f80-a374fda96a12/recordings c.g.g.d.AbstractRecordingSearchService
Saving call recording id ESECG0EV656234LL46RCGJLFVK00000M to contactCenter e9d59b88-6155-4242-8f80-a374fda96a12

03/16/2018 01:20:38.815 INFO [ops] [PxLvG0SZoXkTTIo/QSZ/jQ11el01csvsjoni2cmu4r7c9n4] [] [hystrix-ApiCreatePool-22]

/internal-api/contact-centers/e9d59b88-6155-4242-8f80-a374fda96a12/recordings c.g.g.v.a.t.c.CreateCallRecordingApiTaskV2
Create call-recording ID [ESECG0EV656234LL46RCGJLFVK00000M] contactCenterId [e9d59b88-6155-4242-8f80-a374fda96a12]
succeeded

03/16/2018 01:20:38.820 DEBUG [ops] [PxLvG0SZoXkTTIo/QSZ/jQ11el01csvsjoni2cmu4r7c9n4] [] [qtp1716918369-24] /internal-

api/contact-centers/e9d59b88-6155-4242-8f80-a374fda96a12/recordings c.g.g.v.a.c.c.CreateCallRecordingOpsController
Successfully processed POST of call recording

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 25

5. RWS log snippet showing Playback

03/08/2018 05:09:10.638 INFO [ops] [PxLvG0SZoXkTTIo/QSZ/jQ1jobf2hv4bipjgn7yzsiy59cv] [] [qtp663457961-26]

/api/v2/ops/contact-centers/e9d59b88-xxxx-4242-8f80-
a374fda96a12/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/06521f55-7d52-4f91-bca5-c1d53b7fa422.mp3
c.g.g.v.a.c.c.b.GetCallRecordingMediaControllerBase Play media [06521f55-7d52-4f91-bca5-c1d53b7fa422] of recording
[4O2J4BKS2L29HAATFT19NRQ20K00006A] from contact center [e9d59b88-xxxx-4242-8f80-a374fda96a12] requested

03/08/2018 05:09:10.642 DEBUG [ops] [PxLvG0SZoXkTTIo/QSZ/jQ1jobf2hv4bipjgn7yzsiy59cv] [] [hystrix-ApiGetPool-13]

/api/v2/ops/contact-centers/e9d59b88-xxxx-4242-8f80-
a374fda96a12/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/06521f55-7d52-4f91-bca5-c1d53b7fa422.mp3
c.g.g.v.a.t.c.GetCallRecordingMediaApiTaskV2 fetching media resource e9d59b88-xxxx-4242-8f80-
a374fda96a12:4O2J4BKS2L29HAATFT19NRQ20K00006A:06521f55-7d52-4f91-bca5-c1d53b7fa422

03/08/2018 05:09:10.692 INFO [ops] [PxLvG0SZoXkTTIo/QSZ/jQ1jobf2hv4bipjgn7yzsiy59cv] [] [hystrix-ApiGetPool-13]

/api/v2/ops/contact-centers/e9d59b88-xxxx-4242-8f80-
a374fda96a12/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/06521f55-7d52-4f91-bca5-c1d53b7fa422.mp3
c.g.g.v.a.t.c.GetCallRecordingMediaApiTaskV2 Streaming media [06521f55-7d52-4f91-bca5-c1d53b7fa422-
CRStorageMediaDescriptor(storage=webDAV, storageVersion=null,
path=https://hostd/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A_2018-03-08_05-02-08-008B01AA-10005CE1-
00000001.mp3.bin, data=null, auth=null)] of recording [4O2J4BKS2L29HAATFT19NRQ20K00006A] from contact center
[e9d59b88-xxxx-4242-8f80-a374fda96a12] requested

03/08/2018 05:09:10.768 INFO [ops] [PxLvG0SZoXkTTIo/QSZ/jQ1jobf2hv4bipjgn7yzsiy59cv] [] [qtp663457961-26]

/api/v2/ops/contact-centers/e9d59b88-xxxx-4242-8f80-
a374fda96a12/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/06521f55-7d52-4f91-bca5-c1d53b7fa422.mp3
c.g.g.v.a.c.c.b.GetCallRecordingMediaControllerBase Play media [06521f55-7d52-4f91-bca5-c1d53b7fa422] of recording
[4O2J4BKS2L29HAATFT19NRQ20K00006A] from contact center [e9d59b88-xxxx-4242-8f80-a374fda96a12] succeeded

6. Successful log snippet for Screen Recording

On Authentication Log where agent was successfully authticated on localhost.
INFO : 2018/03/07 05:47:03Z.838 : AuthServer: Starting HTTP Authentication server on 127.0.0.1:8080
INFO : 2018/03/07 05:47:39Z.236 : AuthServer: New Authorization Header Received "Agent1"
INFO : 2018/03/07 05:47:39Z.825 : AuthServer: Logout request received for user None cookie
0ZmNuoEKDsjOgQNCDsZTmkZSznM
INFO : 2018/03/07 05:47:40Z.874 : AuthServer: New Authorization Header Received "Agent1"

• Check IRWS Cluster configuration should agent is not authenticated upon login.

On RWS cloud.log
Upon Agent Login
03/08/2018 05:48:40.552 DEBUG [] [] [] [qtp663457961-26] c.g.c.w.s.CloudWebBasicAuthenticationFilter Authentication
success: org.springframework.security.authentication.UsernamePasswordAuthenticationToken@4abfef2b: Principal:
CloudWebUserDetails(super=com.genesyslab.cloud.web.security.CloudWebUserDetails@f5d71a22: Username: Agent1;
Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true;
Granted Authorities: UserRole(name=ROLE_AGENT), contactCenterId=e9d59b88-xxxx-xxxx-8f80-a374fda96a12,
genesysEnvironmentId=ON_PREMISE_ENVIRONMENT, userId=9c817aa3e97341a89be86b357a09d3bc, samlUser=false,
serviceUser=false); Credentials: [PROTECTED]; Authenticated: true; Details:
org.springframework.security.web.authentication.WebAuthenticationDetails@ffffc434: RemoteIpAddress: 192.168.102.1;
SessionId: null; Granted Authorities: UserRole(name=ROLE_AGENT)

03/08/2018 05:48:44.755 DEBUG [Agent1] [PxLvG0SZoXkTTIo/QSZ/jQ1szcf5ct04vtmxo3832gv7ma] [e9d59b88-xxxx-xxxx-

8f80-a374fda96a12] [qtp663457961-21] /api/v2/me/settings/screen-recording-client
c.g.g.v.a.c.u.s.GetUserScreenRecordingClientSettingsController Get user screen recording client settings for current user:
groupName: screen-recording-client, userId: 9c817aa3e97341a89be86b357a09d3bc
.. /api/v2/me/settings/screen-recording-client c.g.c.v.c.ContactCenterContentProvider Reading application settings
.. /api/v2/me/settings/screen-recording-client c.g.c.v.c.ContactCenterContentProvider Reading contact center settings
.. /api/v2/me/settings/screen-recording-client c.g.c.v.a.t.s.SettingsHelperV2 Applying annex values for user
.. /api/v2/me/settings/screen-recording-client c.g.c.v.a.t.u.s.GetUserSettingsApiTaskV2 Reading user settings
.. /api/v2/me/settings/screen-recording-client c.g.c.v.a.t.u.s.GetUserSettingsApiTaskV2 Settings evaluation complete
GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA
 P a g e | 26

When voice call is received, RWS sends PUBLISH start to SRClient.

03/08/2018 06:00:58.076 DEBUG [] [] []
[com.genesyslab.PCT.invoker.EventBrokerEx:TServer:ON_PREMISE_ENVIRONMENT:be19fc4a-c464-4958-ae4b-
863b21d14e22] c.g.g.v.a.e.s.ScreenRecordingVoiceSession Current session 4O2J4BKS2L29HAATFT19NRQ20K00006C_2018-
03-08_06-02-17, connId: 007002b6949860c7 state: STATUS_INIT, next session state: Pair(first=STATUS_RECORDING,
second=START)
03/08/2018 06:00:58.076 DEBUG [] [] []
[com.genesyslab.PCT.invoker.EventBrokerEx:TServer:ON_PREMISE_ENVIRONMENT:be19fc4a-c464-4958-ae4b-
863b21d14e22] c.g.g.v.a.e.s.ScreenRecordingTelephonyEventHandlerV2 Publish Start Screen Recording to DN 6001

... c.g.g.v.g.t.t.AttachSRSScreenRecordingDataTServerTaskV2 Sent 'RequestAttachUserData' (30) attributes:

AttributeConnID [long] = 007002b6949860c7
AttributeReferenceID [int] = 8
AttributeUserData [bstr] = KVList:
'GSRS_STATE' [str] = "SRSScreenRecordingStateStarted"
AttributeThisDN [str] = "6001"

Voice call ended

03/08/2018 06:01:12.969 DEBUG ... c.g.g.v.a.e.s.ScreenRecordingVoiceSession Current session
4O2J4BKS2L29HAATFT19NRQ20K00006C_2018-03-08_06-02-17, connId: 007002b6949860c7 state: STATUS_RECORDING,
next session state: Pair(first=STATUS_STOPPED, second=STOP)
03/08/2018 06:01:12.969 DEBUG ... c.g.g.v.a.e.s.ScreenRecordingTelephonyEventHandlerV2 Publish Stop Screen Recording
to DN 6001

On Recorder_Agent.log
INFO : 2018/03/08 05:48:41Z.457 : Agent1.Comet: Subscribe to channel /internal-api/me/screen-recording
INFO : 2018/03/08 06:00:54Z.262 : Agent1.Connection: Start recording
INFO : 2018/03/08 06:01:09Z.132 : Agent1.Connecti

7. Tracing Playback Issue

Get metadata on RWS for details of the call.

curl -v -u opsuser:opspassword https://<rws user: rws port>/api/v2/ops/contact-centers/<contact-center
ID>/recordings/<calluuid of the call> | python -m json.tool

[root@hostd ~]# curl -v -u ops:ops https://hostd:8443/api/v2/ops/contact-centers/e9d59b88-xxxx-4242-8f80-

a374fda96a12/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A | python -m json.tool
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4271 0 4271 0 0 8572 0 --:--:-- --:--:-- --:--:-- 8576
{
"recording": {
"callType": "Inbound",
"callerPhoneNumber": "733686868",
"dialedPhoneNumber": "6001",
"eventHistory": [
…..
…..
"calluuid": "4O2J4BKS2L29HAATFT19NRQ20K00006A",
"data": {
"deleted": {
"GSIP_RECORD": "ON"
{
"mediaFiles": [
{
"accessgroups": [
"/CustomerCare_Hierarchy"
],
"callUUID": "4O2J4BKS2L29HAATFT19NRQ20K00006A",
"rcs_Environment:1:CN=HOSTC,[email protected],OU=VSS,O=GCC,L=QC,C=PH:9"
],

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 27
"duration": "20015",
"ivrprofile": "DalyIVRProfile",
"mediaId": "4O2J4BKS2L29HAATFT19NRQ20K00006A_2018-03-08_05-02-08-008B01AA-10005CE1-
00000001.mp3.bin",
"mediaPath": "/ops/contact-centers/e9d59b88-6155-4242-8f80-
a374fda96a12/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/06521f55-7d52-4f91-bca5-c1d53b7fa422.mp3",
"mediaUri": "https://hostd:8443/api/v2/ops/contact-centers/e9d59b88-xxxx-4242-8f80-
a374fda96a12/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/06521f55-7d52-4f91-bca5-c1d53b7fa422.mp3",
"originalMediaDescriptor": {
"path": "https://hostd/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A_2018-03-08_05-02-08-008B01AA-
10005CE1-00000001.mp3.bin",
"storage": "webDAV"
},
"startTime": "2018-03-08T05:00:45.000+0000",
"stopTime": "2018-03-08T05:01:05.000+0000"
….
….
"statusCode": 0
}

Check if file exists on webdav.

a. Access thru browser and attempt if you can play it.
b. Access thru curl and attempt to play it. See storage path from metadata output above (under
"path":) and issue curl to webdav to play the recording.
curl -u webdavuser:webdavpassword https://<webdav
host>/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A_2018-03-08_05-02-08-008B01AA-
10005CE1-00000001.mp3

Output is continuous random keys which means file is being played. To stop display/ playing, press [Ctrl key] + c
[root@hostd ~]# curl -u user:password https://hostd/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A_2018-03-08_05-
02-08-008B01AA-10005CE1-00000001.mp3
,@0e>
ch@>FacX@r(
g6b#_H}lFg
o"9Vdw\TR2L6eo1US5c>aFKP7$hj:Iz4nOt $$1@9CC'Z:n5o3AUH_Ru\ ,!Z>ar}a>ZHj
i&hr`+K2EyB
mFZ#&HJ):ugP}:_t~3E7F$TO,4l+.7N(m<Q>wTRki`p!d&Aqc
f%ao3)\?
3.cXrr +C`'[email protected]|C};$vB%i=odXA44 !Pgt-Te',de

If recordings cannot play from any of the above (browser or curl), then issue is with the actual recording. Worth to
check file size.

Check if RWS able to play the file.

Issue command below. To stop display/ playing, press [Ctrl key] + c
curl -u opsuser:opspassword <"mediaUri" Result>

[root@HOSTD ~]# curl -u ops:ops https://hostd:8443/api/v2/ops/contact-centers/e9d59b88-xxxx-4242-8f80-

a374fda96a12/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/06521f55-7d52-4f91-bca5-c1d53b7fa422.mp3
,@0e>
ch@>FacX@r(
g6b#_H}lFg

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 28
o"9Vdw\TR2L6eo1US5c>aFKP7$hj:Iz4nOt $$1@9CC'Z:n5o3AUH_Ru\ ,!Z>ar}a>ZHj
i&hr`+K2EyB
mFZ#&HJ):ugP}:_t~3E7F$TO,4l+.7N(m<Q>wTRki`p!d&Aqc
f%ao3)\?
3.cXrr +C`'[email protected]|C};$vB%i=odXA44 !Pgt-Te',de

sNNifp
Y.$Jf.%%pqGvUPE./Qb4qbpm'TyK
8+"}T:Am"{woU3hjw>R1QRFYE*
f$bvG9<|{&\;X-zw:=p)e{7z`m,óCSxR>Q5ըUC?T]] "ae2vF(<6i3SBUEt
VAZN` 39(GS~lu]"Sm^u 6l)YsqX0/l3Po.r]|&HtG9)n@|zjBfJALz|s1xycU3a#]MVV

Should RWS cannot play it, check cloud.log for warning and errors. Append -v to curl command to get other details
for failure.

Check if RCS was able to play the file.

Issue command below. To stop display/ playing, press [Ctrl key] + c
curl -u rcsuser:rcspassword https://<rws host>:<rws port>/rcs/<GET parameters from RCS log>

ON RCS LOG to get the value for GET:

Request received for
'/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/4O2J4BKS2L29HAATFT19NRQ20K00006A_2018-03-08_05-02-08-
008B01AA-10005CE1-00000001.mp3.bin':

(GET /rcs/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/4O2J4BKS2L29HAATFT19NRQ20K00006A_2018-03-
08_05-02-08-008B01AA-10005CE1-00000001.mp3.bin)@1115682139 org.eclipse.jetty.server.Request@427ff55b

servletPath:/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/4O2J4BKS2L29HAATFT19NRQ20K00006A_2018-03-
08_05-02-08-008B01AA-10005CE1-00000001.mp3.bin

[root@HOSTD ~]# curl -u giruser:girpassword

https://hostc:8008/rcs/recordings/4O2J4BKS2L29HAATFT19NRQ20K00006A/play/4O2J4BKS2L29HAATFT19NRQ
20K00006A_2018-03-08_05-02-08-008B01AA-10005CE1-00000001.mp3.bin
j@$5-r'e%0H?e l=H$ow&hsGj2,J);2P3KoZlP36]0r# `B !
UUUUUUD`wX*>@pJ@J@l@@@@@@@@@@2
LFda=~?H2t:lh%p2)lAm@hEiV$
m~EDpB&?rZz3:8DeO]2B-l73Lcll~.~'{EXL +H@O!m-'=$)$GBvdq
K#e%yj*foS:q@J$$u![P@glLF5_R@/R$LoT[7Hk&%SNA3R6i/,#L`Hb2zk@: RM$9ka]x+Ckl~}n8&"\FBEpFZ'fd
"ht#8VxwQy
hwY%Jpx`?j9mqGHaS)yV*fQ~IY$s(!QQ"2Tlz(,n'A*bFk@e

If recording is failing on RCS, check RCS logs for errors and warnings. Append -v to curl command to get other
details for failure.

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 29

GIR Monitoring
The following table outlines the various ways in which the components in the recording solution can be
monitored:

HEALTH CHECKS VALIDATION

Look for the following alarm:
T-Server Record Failure
Are there any Alarms in SCI, GA
MCP Record Failure
related to recording?
MCP Record Post Failure
MCP Record Post Backlog Exceeded

Are failed recording files present in

\\MCP Installed Directory\cache\record\failed
MCP failed folders

https://<load balancer host name>/

Is the Load Balancer VIP hostname
In case of Linux, this should display the Red Hat Linux
available?
welcome page

Check if RPs are started in GA, SCI

Does the below URL responds:
Is Recording Processor available? http://<RPHost:RPPort>/api/version/
This should return the RPS version. (See GIR Health
check above)

Are failed json files present in RP

\\RP Installed Directory\failed
failed folders
https://<RWS IP:Port>/api/v2/diagnostics/version
Is RWS available?
http://<RWS IP:Port>/api/v2/diagnostics/version

https://<SM IP>/interactionreceiver
Is Interaction Receiver available?
http://<SM IP>/interactionreceiver

Comprehensive list of all GIR Alarms

Fully comprehensive list of all GIR Alarms and its resolution is described in the below documentation link:
https://docs.genesys.com/Documentation/CR/8.5.2/Solution/alarms

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 30

Stopping and Starting Recording Components

The following table outlines the process for stopping and starting recording components

COMPONENT STARTING/ STOPPING

On each Linux host:

find the running cassandra process:
ps -ef | grep cassandra
And then kill that process:
Cassandra
Kill -9 <process Id>

To Start:
/GCTI/apache-cassandra-1.2.19/bin/cassandra

On each Linux host:

find the running gir process:
ps -ef | grep gir
RWS (Recording And then kill that process:
Web Service) Kill -9 <process Id>

To Start:
Service gir start

Apache Load Stop: Service httpd stop

Balancer Start: Service httpd start

Controlled via GA, SCI

Recording
Can kill the process when in Linux
Processor
Can kill the process on Task Manager

Controlled via GA, SCI

Recording Crypto
Can kill the process when in Linux
Server
Can kill the process on Task Manager

Stopping all nodes in a cluster: pcs cluster stop –all

Stopping individual Node: pcs stop
Pacemaker
Starting all nodes in a cluster: pcs cluster start –all
Starting individual Node: pcs start

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA

 P a g e | 31

GIR TROUBLESHOOTING GUIDE SHIVAKUMAR GURUSIDDAPPA