Cisco CallManager: Troubleshooting RTMT Issues

Document ID: 109377

Contents
Introduction
Prerequisites

Requirements
Components Used
Conventions
Problem: RTMT Java Exception Error
Solution 1
Solution 2
Problem: Unable to Pull QRT reports using RTMT
Solution
Error: [RTMT−ALERT−StandAloneCluster] RouteListExhausted
Solution
Error: [RTMT−ALERT−StandAloneCluster] CriticalServiceDown
Solution
Error: RTMT−ALERT: DBChangeNotify queue delay over 2 minutes
Solution
Unable to Download RTMT Plug−in
Solution
Unable to Monitor the Informix Database
Solution
Unable to Download PDF Files
Solution
Error: Exceeds max limit for trace file download requests
Solution
No Option Available when the QRT Softkey is Pressed
Solution
Error: CDRFileDeliveryFailed BillingServerAddress
Solution
Excessive mixer calls
Solution
How to Get Usage Information on Hardware Conference
Collect Raw CDR Data using RTMT
Pull the Report for the PRIs using the RTMT
Remove Recipient from Email Notification of RTMT
RTMT stuck at "loading please wait...."
No line/device available
Solution
RTMT alerts sent via email fail to deliver
Solution
CUCM User is unable to Access the RTMT
Solution
Getting CTI RTMT alerts
Problem
Solution
Related Information
Introduction
This document addresses how to troubleshoot the Real−Time Monitoring Tool (RTMT) issues with Cisco
Unified Communications Manager.

Prerequisites
Requirements
There are no specific requirements for this document.

Components Used
The information in this document is based on Cisco Unified CallManager 4.x/5.x/6.x/7.x.

The information in this document was created from the devices in a specific lab environment. All of the
devices used in this document started with a cleared (default) configuration. If your network is live, make sure
that you understand the potential impact of any command.

Conventions
Refer to Cisco Technical Tips Conventions for more information on document conventions.

Problem: RTMT Java Exception Error

The RTMT does not work, and these error messages appear:
java.util.MissingResourceException: Can't find bundle for base name
org.apache.xerces.msg.XMLMessages,. The error message appears when a user attempts to access
the RTMT, or after upgrading the operating system of the CallManager server.

Solution 1
Complete these steps in order to resolve this issue:

1. Go to the Windows Start menu and choose Programs > Administrative Tools > Internet Services
Manager.
2. Expand the CallManager server name from the Internet Information Services window and click
Default Web Site.
3. Right click the AST and select the Properties option from the left side of the window.
4. Select the Virtual Directory tab from the AST Properties window.
5. Verify that the Execute Permissions field under the Application Settings section is set to Script and
Executables.
6. Restart the IIS through command prompt. Use this command: C:\utils>iisreset.

Solution 2
RTMT did not work after upgrading the Windows operating system and displays the same error.
Complete these steps in order to resolve this issue:

1. Open the urlscan.ini file from the C:\Winnt\system32\inetsrv\URLscan\ location.

2. Change the RemoveServerHeader parameter value from 0 to 1.

Problem: Unable to Pull QRT reports using RTMT

With Cisco Unified Communications Manager 6.x, pulling the Quality Report Tool (QRT) reports using
RTMT fails even though QRT is enabled on Cisco IP Phones.

Solution
Complete these steps in order to resolve this issue:

1. In the Cisco Unified Communications Manager server, open the Cisco Unified Serviceability page.
2. Go to Tools > Control Center − Feature Services.
3. Check if the Cisco Extended Functions service under Voice Quality Reporter Services is activated
and running. If not, start the service. If it indicates that the service is running, restart it.

Error: [RTMT−ALERT−StandAloneCluster]
RouteListExhausted
This error message is reported by RTMT:

[RTMT−ALERT−StandAloneCluster] RouteListExhausted
Explanation: An available route could not be found in the indicated route list

Solution
The RouteListExhausted messages indicate that CallManager was unable to send a call to those Route
Groups in a Route List. This is usually as a result of no B channels or ports being available on a gateway for
routing a call. If you have a Route List with an H.323 gateway as the first Route Group, and a second Route
Group that should take any overflow, then sometimes the first H.323 gateway will send a User Busy back to
the CallManager if all of its channels or ports are busy. If this occurs, then the call is not forwarded to the
second Route Group in the Route List.

If this is how your own Route List is configured, then go to Cisco CallManager Administration > Service >
Service Parameters > Select a server > Select Cisco CallManager and configure the Stop Routing on
User Busy Flag service parameter to False.

This will mean that the CallManager will try other gateways to re−route the call on a User Busy before
reaching the RouteListExhausted state. However, if there are no RouteListExhausted events in
the Application Log, then the RouteListExhausted messages reported by RTMT can be a result of an
Unallocated number or User Busy messages from the PSTN. In this case, this message can be ignored.

Error: [RTMT−ALERT−StandAloneCluster]
CriticalServiceDown
This error message is reported by RTMT:

[RTMT−ALERT−StandAloneCluster] CriticalServiceDown
 Service status is DOWN. Connection Database Proxy.

Solution
This error can occur when the value of the Database Proxy: Service Shutdown Timer is set as the default
value, which is zero. In order to remove this error, set the value of Database Proxy Service Shutdown Timer to
a value between 0 and 999. Perform these steps:

In the Cisco Unity Connection Administration page, go to System Settings > Advanced > Connection
Administration. Enter a value between 0 and 999.

Note: If you restart the server, the remote database proxy service will remain shut off. After a system restart,
turn on the service manually.

To activate the Remote Database Proxy Service, perform these steps:

1. Go to the Cisco Unity Connection Serviceability web admin page.

2. On the Tools menu, choose the Service Management page.
3. Click the Activate button next to the Connection Database Proxy item under Optional Services.

Error: RTMT−ALERT: DBChangeNotify queue delay over 2

minutes
This error message is reported by RTMT:

RTMT alert:
DBChangeNotify queue delay over 2 minutes. Current DB ChangeNotify queue
delay (150) is over 120−sec threshold.

Solution
DBChangeNotifyFailureThis alert occurs when the Cisco Database Notification Service experiences
problems and might need to be halted. This condition indicates that Change Notification Requests queued in
the database are "stuck" and system changes are not taking effect. If you use the default alert properties, the
alert gets triggered when the DBChangeNotify queue delay is over two minutes. One alert is sent every 30
minutes.

Also, the DBL service is responsible for relaying these DBChangeNotify messages. In order to resolve the
issue, you need to restart the Cisco Database Layer Monitor service on all servers in the cluster. The command
for restarting the Cisco Database Layer Monitor service is shown here:

admin:utils service restart Cisco Database Layer Monitor

Service Manager is running
Cisco Database Layer Monitor[STOPPING]
Cisco Database Layer Monitor[STOPPING]
Cisco Database Layer Monitor[STOPPING]
Cisco Database Layer Monitor[STOPPING]
Commanded Out of Service
Cisco Database Layer Monitor[NOTRUNNIG]
Service Manager is running
Cisco Database Layer Monitor[STARTED]
admin:

Note: Restarting the Cisco DBL service will not reset any phones or devices. It just reloads the database cache
and forces the CallManager service to get a fresh copy of the database.
Unable to Download RTMT Plug−in
User is unable to download the RTMT plug−in.

Solution
If you are using third−party download software, this issue can occur. Uninstalling the third−party download
software will resolve the issue.

Unable to Monitor the Informix Database

Solution
If you already have an SNMP notification destination configured on CallManager with Write and Notify set,
you can enable traps to be sent based on local syslog messages:

1. You need to enable clogNotificationsEnabled (1.3.6.1.4.1.9.9.41.1.1.2). You can use this command
to enable it:

snmpset −c <community_string> −v 2c 14.48.46.2 .1.3.6.1.4.1.9.9.41.1.1.2.0 i 1

2. Now, in CallManager, go to the serviceability page > Configuration, choose your server, and then
Service Group > Database and Admin Services > Database Layer Monitor. Make sure local
syslogs are Enabled and set to error.
3. To check what alerts are at which levels (all of them are at error or alert), go to Alarm >
Definitions. Choose the System Alarm Catalog, and then the DBAlarmCatalog. Provided there are
all the database alerts we have and can trap on.

Unable to Download PDF Files

With CUCM 6.x, PDF files cannot be downloaded from the Serviceability Reports archive. This error
message is received:

Page cannot be displayed

Solution
In order to resolve the issue, restart Cisco Serviceability Reporter from Tools > Control Center −Feature
Services.

Error: Exceeds max limit for trace file download requests

This error message appears in RTMT:

Exceeds max limit for trace file download requests.

Current limit is 2

Solution
Restart these system services in order to resolve the issue:

1. Cisco Trace Collection Servlet

 2. Cisco Trace Collection Service

No Option Available when the QRT Softkey is Pressed

When the user presses the QRT softkey, nothing happens.

Solution
Restart the CTIManager service in order to resolve the issue.

Error: CDRFileDeliveryFailed BillingServerAddress

This alert is received on RTMT:

CDRFileDeliveryFailed BillingServerAddress : 10.1.11.1 AppID : Cisco CDR

Repository Manager ClusterID : NodeID : PUBLISHER . The alarm is
generated on Wed Jul 07 17:13:05 IST 2010.

Solution
Restart these services in order to resolve the issue:

• CDR Repository Manager

This service maintains and moves the generated CDRs that are obtained from the Cisco CDR Agent
service. In a system that supports clusters (Cisco Unified Communications Manager only), the service
exists on the first server.
• Cisco SOAP − CDRonDemand

This is a SOAP/HTTPS−based service that runs on the CDR Repository server. It receives SOAP
requests for CDR file name lists that are based on a user−specified time interval (up to a maximum of
1 hour) and returns a list of file names that fit the time duration that is specified in the request. This
service also receives requests for delivery of a specific CDR/CMR file with the file name and the
transfer method (SFTP/FTP, server name, login info, directory) that is specified in the request.

Excessive mixer calls

With Cisco Unified Communications Manager 7.0, the IP Voice Media Streaming Application service driver
shows this system log error: Excessive mixer calls.

Solution
Because this appears to be a cosmetic error, you can disable the RTMT alert related to the error message. Or if
you want to get the issue permanently resolved, you can upgrade your CallManager server to any of the
releases mentioned in Cisco bug ID CSCte19629 (registered customers only) . This bug ID contains the fix for
this defect.

How to Get Usage Information on Hardware Conference

In order to monitor usage information on Hardware Conference from the Communication Manager, perform
these steps:
 1. Log in to RTMT.
2. Go to System > Performance > Open Performance Monitoring.
3. Choose Cisco HW Conference Brige Device.

Collect Raw CDR Data using RTMT

In order to collect raw Call Detail Records (CDR) data using RTMT, perform these steps:

1. Log in to RTMT.
2. Go to System > Tools > Trace > Trace & Log Central and select the following:

Cisco CDR files on CM server

Cisco CDR files on Publisher Processed
3. Check All Servers for both the above services.
4. Click Next, and again click Next. Give the appropriate time range for which you want to collect raw
CDR data.
5. Specify a Download File Directory and zip the files.
6. Click Finish. Collect the files.

Pull the Report for the PRIs using the RTMT

Perform these steps in order to pull the Report for the Primary Rate Interface (PRI) using the RTMT:

1. Go to RTMT > System > Performance > Cisco MGCP PRI, Calls Active or Channel XX status.
2. Drag the object (Calls Active) to the right panel (if possible, select the last PRIs on the Route
Groups).
3. Right click and select Alert > Severity: Warning, Absolute Value 20.
4. If you already have the email alert enabled, choose the profile for Action email. Click Save.
5. If you do not have the email alert enabled, you need to configure this alarm on the RTMT. Perform
these steps:

a. Login to the RTMT and choose Alert Center.

b. Right click on the XXXXXX alert, and choose Set Alert/Properties.... Configure the
following and click Next:

◊ The alert is enabled.

◊ Select the severity of the alert.
◊ Select the server that you want to make the report.
c. From this window, choose Duration: Trigger alert immediately and click Next.
d. From this window, configure the following and click Next:

◊ Frequency: trigger alert on every poll

◊ Schedule: trigger alert when it occurs
e. Configure the following and click OK:

◊ Check the Enable Email check box.

◊ Go to Configuration > Add > Enter name and description.
◊ Under recipients, click Add and insert the email to where you want to send the alert
information.
Remove Recipient from Email Notification of RTMT
In order to remove a recipient(s) from email notification of RTMT, perform these steps:

1. Go to RTMT > System > Tools > Alert > Alert Central.

2. Choose the Alert Name to modify.

3. Go to RTMT > System > Tools > Alert > Config Alert Action.

4. Choose the action profile for email. Click Edit and select the Recipient(s) to be removed.

5. Click Delete. Then, click OK and Close.

RTMT stuck at "loading please wait...."

The RTMT hangs because of an UnsupportedClass Exception as the JRE used in compiling the code and
launching the RTMT is different. In order to remove this error, complete these steps:

1. Install RTMT. By default, RTMT is installed at C:\Program Files\Cisco\CallManager

Servicability.
2. Copy the jre1.5.0_14 folder from C:\Program Files\Java to C:\Program Files\Cisco\CallManager
Servicability\JRE.
3. Go to C:\Program Files\Cisco\CallManager Servicability\jrtmt and open run.bat in a notepad.
Then, replace the content with ..\JRE\jre1.5.0_14\bin\java
−Dsun.java2d.d3d=false−Xbootclasspath/a:.;.\Lib\xerces.jar;.\Lib\log4j.jar;.\Lib\jcchart45
0J.jar;.\Lib\collections.jar −jar JRtmt.jar.
4. Double click run.bat.
5. If the double click does not work, go to Start > Run and type cmd.
6. Type cd C:\Program Files\Cisco\CallManager Serviceability\jrtmt and then type run.bat. This
opens the RTMT login screen.

Note: Perform run.bat whenever you want to use RTMT. A shortcut created for run.bat on the
desktop does not work.

Refer to Cisco bug ID CSCsu86458 (registered customers only) for more information.

No line/device available
After pressing dial on the WebDialer pop up, the "No line / device available" error is received.
This error message is reported by RTMT:

CTIManager: 1055: LEHIUCMPUB.ad.local: Apr 14 2011 22:28:26.545

UTC :%UC_CTI−3−CtiDeviceOpenFailure:%[CTIconnectionId=3][DeviceName=SEPA40CC395368C]
UNKNOWN_PARAMTYPE:ReasonCode=−1932787499]
[AppID=Cisco CTIManager][ClusterID=StandAloneCluster][NodeID=LEHIUCMPUB][CTIAppID=Cisco
WebDialer][ProcessID=30024][ProcessName=CtiHandler]: Device Open failed

Solution
In order to resolve this issue, deactivate/reactivate the WebDialer service.
RTMT alerts sent via email fail to deliver
The RTMT email alerts fail to deliver if the DNS is not configured on the Cisco Unified Communications
Manager server. If DNS is not configured on the Cisco Unified Communications Manager servers during or
post install, the fully qualified domain name (FQDN) will not append to the email sent by RTMT. This causes
the email to be undeliverable.

This issue is documented by Cisco bug ID CSCsk71330 (registered customers only) .

Solution
Configure DNS from the Command Line Interface (CLI) using these commands:

set network dns primary xxx.xxx.xxx.xxx

set network dns secondary xxx.xxx.xxx.xxx

set network domain yourdomain.com

After you enter the commands, reboot the node for the changes to take effect.

CUCM User is unable to Access the RTMT

After creating a new user in Cisco Unified Communications Manager, you cannot make this user access the
RTMT.

Solution
In order to resolve this issue, complete these steps:

1. Find the Standard Packet Sniffer Users group and click Add Users to Group.
2. Add the specific end user who should access the RTMT to this group. Click Save.

Getting CTI RTMT alerts

Problem
This error is received continuously on the subscriber:

%CCM_CTI−CTI−3−kCtiEnvProcDevListRegTimeout: Directory change notification

request time out. Cluster ID:StandAloneCluster Node ID:PMC−BMI−CM04

Note: When researching the error message, it states to check network connectivity with that node. All nodes
are alive and working, but this error message is persistent.

Solution
In order to resolve this issue, restart the CTI Manager service on that node.
Related Information
• Real−Time Monitoring Tool
• Voice Technology Support
• Voice and Unified Communications Product Support
• Troubleshooting Cisco IP Telephony
• Technical Support & Documentation − Cisco Systems

Contacts & Feedback | Help | Site Map

© 2014 − 2015 Cisco Systems, Inc. All rights reserved. Terms & Conditions | Privacy Statement | Cookie Policy | Trademarks of
Cisco Systems, Inc.

Updated: Aug 25, 2011 Document ID: 109377