IBM Troubleshooting connection pooling (J2C)

problems in WebSphere Application Server

Troubleshooting for connection pooling problems in IBM® WebSphere® Application

Server. This should help address common issues with this component before calling IBM
support and save you time.

Note: The troubleshooting steps in this document are applicable to JDBC connection
problems using a standard data source in WebSphere Application Server. This document is
not applicable to Version 4 data sources. You can also use this document for
troubleshooting JMS connection problems and enterprise information system (EIS)
connection problems.

Steps to help resolve connection pooling problems

1. Are you having a problem using a data source to establish a JDBC connection to a
database? Most exceptions that occur when a data source is used to connect to a
database will have the com.ibm.ws.rsadapter package in the stack trace of the
exception. This includes java.sql.SQLExceptions and WebSphere Application Server
messages that begin with DSRA.
Yes, go to the Troubleshooting JDBC connection problems section.
No, continue to question 2.
2. Are you having a problem using a JMS connection factory to establish a JMS
connection to WebSphere MQ or another messaging system? Most exceptions that
occur when a connection factory is used to connect to a messaging system will have
the com.ibm.ejs.jms package in the stack trace of the exception. This includes
javax.jms.JMSExceptions and WebSphere Application Server messages that begin
with WMSG.
Yes, go to the Troubleshooting JMS connection problems section.
No, continue to question 3.
3. Are you seeing J2CA0045E errors with ConnectionWaitTimeoutExceptions or slow
performance in getting a connection?
Yes, go to the Troubleshooting connection wait issues, connection
leaks, and performance problems section.
No, continue to question 4.
4. Are you seeing StaleConnectionExceptions or having problems recovering from
invalid, or stale, connections in the pool?
Yes, go to the Troubleshooting stale connection problems section.
No, continue to question 5.
5. Go to the Miscellaneous connection pooling problems section. If your problem
is not covered in that section or addressed elsewhere in this document, review the
WebSphere Application Server V8.0, V8.5.5, or V9.0 product documentation, or
the WebSphere Application Server Support site for additional information that
might help you to resolve the problem, or continue to the MustGather:
Connection pooling problems for WebSphere Application Server for
database connection and connection pooling problems.

Troubleshooting JDBC connection problems

1. This section will help you to troubleshoot problems that occur when using a data
source defined in WebSphere Application Server to establish a JDBC connection to a
database. JDBC connection problems usually occur at runtime when an application
attempts to connect to the database and perform SQL queries. This section does not
include ConnectionWaitTimeoutExceptions or StaleConnectionExceptions, which are
covered in subsequent sections.

To troubleshoot this type of problem, you should have access to the administrative
console and the SystemOut.log for your application server. When diagnosing a
database connection problem, the first step is to use the Test Connection button for
the data source in the administrative console to test the connection to the database.
You can find the Test Connection button in the data source configuration panel.

2. Was the attempt to connect to the database using the Test Connection button
successful?
Yes, continue to question 3.
No, continue to question 4.
3. If you can use the Test Connection button to successfully connect to the database,
but a failure occurs when your application tries to get a connection, the problem is
likely caused by a failed JNDI lookup. Check the SystemOut.log to see if a
NameNotFoundException occurs when the application tries to use the data source. Does
a NameNotFoundException occur?
Yes, the root cause of the problem is that the JNDI lookup of the data source by
the application fails. Check the data source JNDI name and ensure that it is
bound to the JNDI namespace when the application server starts. If the JNDI
name is jdbc/ds, you should see this entry in the SystemOut.log:
WSVR0049I: Binding ds as jdbc/ds
Also, check the application code and ensure that it is looking up the correct
JNDI name. If you are doing an indirect JNDI lookup (i.e. java:comp/env
/jdbc/ds), ensure that the binding is correct in the data source resource
reference. The resource reference is configured in the web, EJB, or application
client deployment descriptor. Checking for these things can help you to resolve
many JNDI problems. However, if you can't determine the cause of the problem,
continue to the Troubleshooting JNDI and naming problems.
No, continue to question 8.
4. The error message shown in the administrative console when the Test Connection
button fails is important to understanding the cause of the problem. You can also
check the SystemOut.log to get more information about the error and see the stack
trace.

Does the error message and the accompanying SQLException indicate a problem with
the user ID and password that are used to connect to the database? The actual error
and exception will differ depending on the JDBC driver and database that you're
using.

Yes, the root cause of the problem is either that no userid and password are
passed to the database, or the userid or password is not correct. Ensure that a
J2C authentication alias, containing the correct userid and password, is
specified on the data source. Also check with your DBA to make sure that the
userid and password that you are using are correct for connecting to the
database.
No, continue to question 5.
5. If you are using a Type 2 driver or Type 3 JDBC driver, does a
java.lang.UnsatisfiedLinkError occur when you use the Test Connection button? A
java.lang.UnsatisfiedLinkError occurs when the JVM is unable to load a native
library that is needed by the JDBC driver.
Yes, the root cause of the problem is that the application server JVM is not
properly configured to load the native libraries. For a DB2 database, see
documents java.lang.UnsatisfiedLinkError: SQLConnect error
connecting to DB2 type 2 datasource, Failure in loading native
library db2jcct2, UnsatisfiedLinkError: ERRORCODE=-4472, and
WebSphere database connection exception
java.lang.UnsatisfiedLinkError. If you are trying to connect to an Oracle
database, ensure that the ORACLE_HOME and the LIBPATH (on AIX®),
LD_LIBRARY_PATH (on Sun Solaris or Linux®), or SHLIB_PATH (on HP-UX) JVM
environment entries are set. The required JVM environment entries might be
different for other databases.

You should also make sure that the user that is being used to run the application
server has the proper permissions to access the native libraries. Incorrect
permissions are another common cause of the java.lang.UnsatisfiedLinkError.

Finally, remember that 32-bit native libraries cannot be used with a 64-bit
application server JVM and vice versa. Mismatches between the two can also
cause the java.lang.UnsatisfiedLinkError.

No, continue to question 6.

6. If you are testing a connection to a Microsoft® SQL Server database, does the
following error occur?
java.sql.SQLException: [IBM][SQLServer JDBC Driver][SQLServer]Could not find
stored procedure 'master..xp_jdbc_open'.
Yes, the root cause of the problem is that the stored procedures for JTA are
not installed on the SQL Server database. These stored procedures are required
for XA connections and for using the Test Connection button. Review Test
connection button in WebSphere Application Server fails when
trying to connect to Microsoft SQL Server datasource (XA) technote
to resolve the problem.
No, continue to question 7.
7. If the error that you are getting is not one of the common errors described in the
previous steps, review the error message and SQLException that occur when using the
Test Connection button. Check the properties that you have specified on the data
source to ensure that they are correct. Refer to your database documentation or your
DBA for more information about why a particular error or exception might have
occurred. An example of a typical error message that might occur is:
DSRA8040I: Failed to connect to the DataSource. Encountered :
java.lang.Exception: java.sql.SQLException: Io exception: The
Network Adapter could not establish the connectionDSRA0010E: SQL State =
null, Error Code = 17,002

If you still cannot resolve the problem, continue to question 8.

8. Finally, if none of the previous troubleshooting steps helped to resolve the problem,
try to connect to the database server using a stand-alone java program to determine if
you are able to successfully connect to the database using the same properties that
you have set in your data source outside of WebSphere Application Server. This will
 allow you to determine whether the problem is specific to WebSphere Application
Server or if the problem occurs with the JDBC driver independently of WebSphere
Application Server..
If a direct JDBC connection fails with the same error that occurs within
WebSphere Application Server, the problem is not specific to your WebSphere
Application Server environment. Work with your DBA to troubleshoot the
problem.
If the direct JDBC connection is successful, continue to the MustGather:
Connection pooling problems for WebSphere Application Server for
database connection and connection pooling problems.

Troubleshooting JMS connection problems

1. This section will help you to troubleshoot problems that occur when using a JMS
connection factory to establish a connection to a messaging system such as
WebSphere MQ. Only JMS connection problems are covered here.

To troubleshoot this type of problem, you should have access to the SystemOut.log for
your application server. When troubleshooting a JMS connection problem to MQ, it is
useful to determine the MQ reason code associated with the JMSException. In the
SystemOut.log, find the JMSException and review the stack trace. You should see one
or more linked exceptions. Find the last linked exception. You should see a line like
this:

---- Begin backtrace for Nested Throwables

com.ibm.mq.MQException: MQJE001: Completion Code 2, Reason 2009

The four-digit number is the reason code. You can review this Messages document
to find out more about the reason code that you see. The reason code helps you to
understand the root cause of the problem.

2. Are you receiving a JMSException with MQ reason code 2009?

Yes, the root cause of the problem is that the connection to the MQ queue
manager is broken or invalid. Review technote Resolving JMSException
due to com.ibm.mq.MQException: MQJE001: Completion Code 2,
Reason 2009 to resolve the problem.
No, continue to question 3.
3. Are you receiving a JMSException with MQ reason code 2019?
Yes, the root cause of the problem could be that the connection pool Purge
Policy is not set correctly. Review technote JMS connections fail with
Reason Code 2019 to resolve the problem.
No, continue to question 4.
4. Are you receiving the following exception?
J2CA0020E: The Connection Pool Manager could not allocate a Managed
Connection: java.lang.IllegalStateException: Internal Error: cannot find the
PoolManager Reference

Yes, the root cause of the problem is that the application caches or fails to close
JMS session objects. Review technote J2CA0020E: The Connection Pool
Manager could not allocate a Managed Connection:
java.lang.IllegalStateException: Internal Error: cannot find the
 PoolManager Reference to resolve the problem.
No, continue to question 5.
5. Are either of the following authentication errors occurring when trying to establish a
JMS connection?
MSGS0508E: The JMS Server security service was unable to authenticate userid:
myuser

javax.jms.JMSSecurityException: MQJMS2013: invalid security authentication

supplied for MQQueueManager

Yes, the root cause of the problem is either that no userid and password are
passed to the queue manager, or the userid or password is not correct. Ensure
that a J2C authentication alias is specified on your JMS connection factory.
No, continue to question 6.
6. Is the problem that you observe (using a tool such as netstat) more TCP/IP
connections than you expect between your application server process and an MQ
queue manager?
Yes, this is not necessarily a problem. This condition is explained in great detail
in technote Explanation of connection pool and session pool settings
for JMS connection factories. You should also review the technote if you
want to gain an understanding of how JMS connection pooling in WebSphere
Application Server works.
No, continue to question 7.
7. Finally, if none of the previous troubleshooting steps helped to resolve the problem,
continue to the MustGather: Connection pooling problems for WebSphere
Application Server for connection pooling problems.

Troubleshooting connection wait issues, connection leaks, and performance

problems

1. This section will help you to troubleshoot connection wait issues, including those that
are caused by application connection leaks, and general performance issues that are
related to connection pooling. These types of problems could occur for any type of
backend system, including databases, messaging systems, and enterprise information
systems.

To troubleshoot this type of problem, you should have access to the administrative
console and the SystemOut.log for your application server. It is also preferable for you
to have access to the source code for your application(s). Additionally, obtaining
javacores (also known as thread dumps) may help you to resolve the problem.

The following documents can be used to diagnose the J2CA0045E error:

How to troubleshoot J2CA0045E connection pooling problems.

Webcast Reply: Understanding and Resolving ConnectionWaitTimeout
Exceptions in WAS
What does J2CA0045E mean for WebSphere Application Server?
Using Connection information in WebSphere trace files to troubleshoot
J2CA0045E and J2CA0020E or connection wait time-out problems
Application hangs with J2CA0045E errors and with J2CA0086W warning
messages

1. Are you experiencing a connection wait problem? Symptoms might include

ConnectionWaitTimeoutExceptions in the SystemOut.log file, slower performance or
delays when the application tries to get a connection, or threads hanging while
waiting for a free connection (this can be seen in a javacore). An example stack trace
for a thread that is waiting for a free connection is:
"Servlet.Engine.Transports : 2833" (TID:0x31680590, sys_thread_t:0x78C96AA0,
state:CW, native ID:0x86C7) prio=5
at java.lang.Object.wait(Native Method)
at
com.ibm.ejs.j2c.poolmanager.FreePool.queueRequest(FreePool.java(Compiled
Code))
at
com.ibm.ejs.j2c.poolmanager.FreePool.createOrWaitForConnection(FreePool.java(Compiled
Code))
at
com.ibm.ejs.j2c.poolmanager.PoolManager.reserve(PoolManager.java(Compiled
Code))
...

Yes, continue to question 4.

No, continue to question 9.
2. This situation would occur when the connection pool is at its maximum size
(defined by the Maximum Connections property on the connection pool), all of the
connections are in use, and the application requests more connections. The
application must wait for a free connection to become available.

Does the problem eventually clear up or does it persist until you restart the
application server?

If the problem eventually clears up, the root cause of the problem is likely that
Maximum Connections is not set high enough for the amount of load on the
application. You should increase Maximum Connections for the connection pool.
You should conduct thorough load testing to find the optimal value for Maximum
Connections. You can enable PMI and monitor the connection pool
counters in the Tivoli Performance Viewer to help you tune this.
If the problem does not clear up until you restart the application server,
continue to question 5.
3. Is the application calling close() on every connection object that it obtains from a
WebSphere Application Server connection pool? When an application gets a
connection from the pool, it is considered "in use" until the application calls close()
on the connection, which then returns the connection to the free pool. If the
application does not call close(), the connection is leaked and never returns to the
free pool. Eventually, the pool might become filled with leaked connections, causing
connection wait problems.

To determine if an application is leaking connections, you should review the

application source code. Note that the application should close connections in a
finally block to ensure that the connections are closed even in error conditions. In
addition, you can use WebSphere Application Server tracing to identify connection
leaks. The trace output can show you that a connection leak exists, and it can also
show exactly what application code is leaking connections. Finally, the PMI
connection pool counters might also be useful in determining whether or not
there is a connection leak.

If the application is failing to call close() on all connection objects, that is the
root cause of the problem. You should fix the application code to resolve the
 problem.
If you determine that the application server is not leaking connections, continue
to question 6.
4. Connection wait problems and slow application performance when using
connection pooling can occur when the web container is not configured properly. If
the web container thread pool size is set too high relative to the Maximum
Connections setting for the connection pool, resource contention for the available
connections could occur. It is also strongly recommended not to check the isGrowable
checkbox for the web container thread pool.

Is the web container thread pool size set too high or is the isGrowable checkbox
checked in your configuration?
Yes, then this configuration is likely the root cause of the problem. Correct the
issue and then test to see if the problem is resolved.
No, continue to question 7.
5. Other issues might cause connection wait problems. Applications should follow the
"get/use/close" pattern and close all connections immediately after finishing using
them. If an application takes too long to close connections, or caches connections, the
connections will be in use longer, which could lead to connection wait issues. Even if
the "get/use/close" pattern is followed, connections will not return to the free pool
until the transaction in which the connection is obtained is committed. If the
transaction remains active for a long time after a connection is closed, connection wait
issues could occur. This frequently occurs in servlets when a shareable connection is
obtained in a local transaction containment (LTC). More details about this scenario
and solutions for it are documented in this Default behavior of managed
connections in WebSphere Application Server article.

Is your application following the "get/use/close" pattern and not caching connections,
without long periods of time elapsing before the transaction is committed?

Yes, continue to question 8.

No, then the root cause of the problem is that connections are "in use" for too
long before returning to the free pool. Correct the issue and then test to see if the
problem is resolved.
6. Are there any network or database issues at the time that the problem occurs?
Connection wait problems and performance problems might occur when network
traffic is slow or when the database server is busy or performing slowly.
Yes, then the root cause of the problem is that connections are "in use" for long
periods of time due to the network or database slowdown. Correct the problem
with the network or the database server.
No, continue to question 9.
7. If you are seeing slow performance related to connection pooling without a
connection wait issue, check the SystemOut.log. Are WSVR0605W warnings appearing in
the SystemOut.log file?
Yes, this means that there is a thread that has been running as long as the
threshold defined in the hang detection policy. Generate javacores to analyze
why the thread is hung. This should reveal the root cause of the problem.
No, continue to question 10.
8. If none of the conditions described in this section are occurring, but the application
is performing poorly or the application server appears to be unresponsive or hung,
then use the appropriate MustGather document for an application server hang to
analyze the threads of the application server to determine the cause of the problem.
 9. Finally, if none of the previous troubleshooting steps helped to resolve the problem,
continue to theMustGather: Connection pooling problems for WebSphere
Application Server for database connection and connection pooling problems.

Troubleshooting stale connection problems

1. This section will help you to troubleshoot problems with stale or invalid connections
in the connection pool. WebSphere Application Server can determine that a
connection is stale based on the exception that is returned by the backend when trying
to establish a connection. When there is a stale database connection, WebSphere
Application Server issues a StaleConnectionException. For other backends, a fatal
connection error occurs. As a result, WebSphere Application Server purges the
connection pool based on the Purge Policy setting and applications are able to
recover from the problem. The following two links have more information on this
issue: WebSphere Application Server StaleConnectionExceptions and
Demystifying the WebSphere StaleConnectionException.

To troubleshoot this type of problem, you should have access to the administrative
console and the SystemOut.log for your application server. It is also preferable for you
to have access to the source code for your application(s).

2. Is the backend system that you are connecting to a database, a messaging system, or
an enterprise information system?
If it is a database, continue to question 3.
If it is a messaging system or an enterprise information system, continue to
question 6.
3. If you are seeing StaleConnectionExceptions in the SystemOut.log, do they occur for
every connection attempt or do they occur intermittently or only after a certain time
period has elapsed?
If it occurs for every connection attempt, the root cause of the problem is likely
that the data source properties are misconfigured. Follow the steps in the
Troubleshooting JDBC connections section.
If it occurs intermittently or only after a certain time period has elapsed,
continue to question 4.
4. Is the application handling the StaleConnectionException properly by catching the
exception and then retrying the connection?
Yes, continue to question 5.
No, then the root cause of the problem is that the application is not catching the
exception and retrying, so it cannot recover from the problem. Review the
WebSphere Application Server StaleConnectionException document
on how to handle the StaleConnectionException and implement this in your
application code.
5. A data source can be configured to pretest connections to ensure that they are valid
before they are allocated to the application. Here is the document for pretest
connections. Have you configured your data source to do this?
Yes, continue to question 6.
No, the problem can probably be resolved by configuring the data source to
pretest connections before they are allocated to the application.
6. Is the Purge Policy set to EntirePool for the connection pool? This means that when a
StaleConnectionException occurs, every free connection in the pool will be purged,
which makes it easier for the application to recover.
Yes, continue to question 7.
No, the problem symptoms can be alleviated by changing the Purge Policy to
 EntirePool.
7. Isthe Minimum Connections property in the connection pool set to 0? Setting it to 0
enables the pool maintenance thread to clean up all of the connections in the pool
after they are unused for more than the Unused Timeout number of seconds. If Minimum
Connections is set to a value greater than 0, WebSphere Application Server must keep
at least that number of connections in the pool indefinitely. The longer that a
connection remains in the pool, the more susceptible it is to becoming stale.
Yes, continue to question 8.
No, the problem symptoms can be alleviated by changing the value of Minimum
Connections to 0.
8. Does a firewall exist between the application server and the backend system that it is
connecting to?
Yes, the root cause of the problem is likely that the firewall is timing out and
dropping connections between the application server and the backend. To avoid
this possibility, set the Unused Timeout to half the value of the timeout setting on
the firewall. This way, WebSphere Application Server can clean up its unused
connections before the firewall drops them.
No, continue to question 9.
9. Finally, if none of the previous troubleshooting steps helped to resolve the problem,
continue to the MustGather: Connection pooling problems for WebSphere
Application Server for database connection and connection pooling problems.
Also, see the following documents for additional information on the StaleConnection
Exception:

Exceptions pertaining to data access (see sections "Stale Connections"

and "Example: Handling data access exception - stale connection")
WebSphere Application Server StaleConnectionException

Miscellaneous connection pooling problems

1. Are you seeing J2CA0075W warnings in the SystemOut.log?

J2CA0075W: An active transaction should be present while processing method
allocateMCWrapper.
Yes, the root cause of the problem is that the application is spawning its own
threads and obtaining connections on those threads. Review technote
J2CA0075W warning messages are written to the SystemOut.log file
during runtime to resolve the problem.
No, continue to question 2.
2. Are you seeing J2CA0106E errors in the SystemOut.log?
J2CA0106E: A 5.0 DataSource was attempted to be used in a WebModule that was
not level 2.3
Yes, the root cause of the problem is that a servlet at the 2.2 specification level
is trying to use a standard data source, which is not allowed. Review technote
J2CA0106E: A 5.0 DataSource was attempted to be used in a
WebModule that was not level 2.3 to resolve the problem.
No, continue to question 3.
3. Are you running WebSphere Application Server V7, V8.5.5 or V9 and seeing
J2CA0294W warnings in the SystemOut.log?
J2CA0294W: Deprecated usage of direct JNDI lookup of resource jdbc/ds. The
following default values are used: [Resource-ref settings]
res-auth: 1 (APPLICATION)
res-isolation-level: 0 (TRANSACTION_NONE)
res-sharing-scope: true (SHAREABLE)
loginConfigurationName: null
loginConfigProperties: null
 [Other attributes]

res-resolution-control: 999 (undefined)

isCMP1_x: false (not CMP1.x)

isJMS: false (not JMS)
Yes, it is not necessarily a problem and it can be prevented by creating a
resource reference and doing an indirect JNDI lookup of your data source or
connection factory. Review this J2CA0294W: Deprecated usage of direct
JNDI lookup of resource technote to resolve the problem.
No, continue to question 5.
4. Are you seeing the following error in the SystemOut.log?
J2CA0079E: Method getManagedConnection has detected an internal illegal state
and is throwing an IllegalStateException. The exception is:
java.lang.IllegalStateException: setManagedConnection: illegal state
exception. State = STATE_INACTIVE
Yes, the root cause of the problem is a WebSphere Application Server defect
that only occurs when tracing is enabled. Review this
Java.lang.IllegalStateException occurs when WAS.j2c tracing is
enabled technote for further details and to resolve the problem.
No, continue to question 6.
5. Are you seeing J2CA0086W warnings in the SystemOut.log?
J2CA0086W: Shareable connection MCWrapper id 686bbdf9 Managed connection
WSRdbManagedConnectionImpl@6156bdf9 State:STATE_TRAN_WRAPPER_INUSE from
resource jdbc/ds was used within a local transaction containment boundary.
Yes, this means that two or more connections were obtained within one local
transaction containment (LTC). This doesn't necessarily indicate a problem, but
you should review this Application hangs with J2CA0045E errors and
with J2CA0086W warning messages technote for more details.
No, continue to question 7.
6. Are you seeing high memory usage or java.lang.OutOfMemoryErrors due to a high
statement cache size?
Yes, the problem can probably be resolved by tuning the statement cache size.
Review this Memory utilization considerations when using prepared
statement caching technote for the details.
No, continue to question 8.
7. Are you seeing java.lang.ClassCastExceptions or other errors when using vendor
specific, non-standard JDBC methods in your application?
Yes, the problem can probably be resolved by using the JDBC 4.0 Wrapper
Pattern. This provides a standard way to access native connections and non-
standard JDBC methods.
No, continue to question 9.
8. Finally, if none of the previous troubleshooting steps helped to resolve the problem,
continue to the MustGather: Connection pooling problems for WebSphere
Application Server for database connection and connection pooling problems.

What to do next If the preceding troubleshooting steps did not solve your problem, see
the MustGather: Connection pooling problems for WebSphere Application
Server for Connection Pooling problems to continue investigation.

Other related documents

How to change from Shareable to Unshareable Connections
Configuration DB2 datasource in WebSphere Application Server V8.5 on windows
J2CA0044E occurs on getting a database connection
J2CA0046E java.lang.NullPointerException error on WebSphere Liberty 8.5.5.7
When click test connection, the older JDBC driver version shows in the SystemOut.log
file