Web Logic
Web Logic
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
Weblogic Overview
Managed Server:
Managed server would connect to admin server via http connection and gets a
copy of config.xml file and creates the related Mbeans. These Mbeans exists in memory of
server for the life time.
Once managed server has started up successfully, we do not need Admin server. The
application hosted on managed servers can be accessed irrespective whether or not
Admin server is up and running. Admin server would be needed for deployments, any
administrative activity like accessing the console etc.
Admin server is just another server instance but with added functionality.
Admin server is equipped with the functionality to drive the deployment (master deployer),
monitor the managed servers etc. All the weblogic.Admin utility commands are initiated
from Admin server. Admin server would act as central point for managing the domain.
Admin server is just another server instance but with added functionality.
Admin server is equipped with the functionality to drive the deployment (master deployer),
monitor the managed servers etc. All the weblogic.Admin utility commands are initiated
from Admin server. Admin server would act as central point for managing the domain.
Agenda
WebLogic Classloading
WebLogic JDBC,JMS,JMX,JNDI
WebLogic Troubleshooting
Enhancements in WLS9.0
Domain
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
1/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
Server startup:
Admin server: The server as part of its startup process, would parse the
config.xml and create Mbeans.
For eg: <Server ListenPort="7001" Name="myserver"
Name="myserver"/>
From the above entries, myserver instance would create a ConfigMbean of
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
2/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
Admin server is just another server instance but with added functionality.
Admin server is equipped with the functionality to drive the deployment (master deployer),
monitor the managed servers etc. All the weblogic.Admin utility commands are initiated
from Admin server. Admin server would act as central point for managing the domain.
Managed server would connect to admin server via http connection and gets a
copy of config.xml file and creates the related Mbeans. These Mbeans exists in memory of
server for the life time.
Once managed server has started up successfully, we do not need Admin server. The
application hosted on managed servers can be accessed irrespective whether or not
Admin server is up and running. Admin server would be needed for deployments, any
administrative activity like accessing the console etc.
Managed server can also start without Admin server if MSI mode is enabled. In this mode,
managed server would look for file msi-config.xml under its root directory. It creates
Mbeans out
of msi-config.xml to startup successfully.
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
3/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
1) Using scripts
2) Node manager
3) Weblogic.Admin start command
4) On windows weblogic server can be started as windows service (beasvc.exe)
Using Scripts
WebLogic installation creates the startup scripts. We can modify the created scripts to
suit our environment.
All scripts would have :
The main class is weblogic.Server that initiates the server startup sequence.
As part of server startup sequence (irrespective of how it is started) there are various
services that get initialized. RMI service, cluster service, JNDI service etc
Node Manager is a Java utility that runs as separate process from WebLogic Server
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
4/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
and allows you to perform common operations tasks for a Managed Server, regardless of
its location with respect to its Administration Server
Run the script (edit the classpath and JVM arguments if needed) to create the service which
will update the registry too. Run the service through
control panel.
5/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
The following are high level activities that occur during server startup:
1) Obtains the configuration data. While Admin server gets this from config.xml, managed
server get this from Admin server.
2) Starts logging and timer services & loads license.bea file.
3) Initializes internal component services like RMI service, Cluster service, IIOP service,
Deployment Manager service etc
4) JDBC connection Pools are created.
5) Startup classes (Eg Wily introscope) are executed.
6) Applications are deployed.
The following phases are involved in server graceful shutdown cycle (When
weblogic.Admin utlity is used or stopped from console)
RUNNING>SUSPENDING>STANDBY>SHUTTING DOWN>SHUTDOWN
Graceful shutdown of server allows all the inflight work to be finished before the server
shuts down.
The first services to shutdown are RMI service and web container, so that no new
requests can come through.
Server would wait for Graceful Shutdown Timeout before shutting down.
All the existing httpsessions are destroyed or server would wait till they expire depending
on Ignore Sessions During Shutdown.
6/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
If it finds a different version of application in the directory, then weblogic would redeploy the
app automatically.
Obviously there is a little bit of overhead involved, but this is justified for dev environment.
This is an optional feature available that would help in segregating the admin traffic with
application related traffic.
A different port has to be configured and all admin tasks (including accessing Admin
console) are picked up from the configured port.
1. The Administration Server and all Managed Servers in your domain must be
configured with support for the SSL protocol.
2. After enabling the administration port, all Administration Console traffic must
connect via the administration port.
Replace these files in the new domain and start the Admin server.
The new Admin server will automatically reconnect with running managed servers
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
7/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
Also.,
Managed Servers within the domain can be at different Service Pack levels as long as
the Admin Server is at the same Service Pack Level or higher than its Managed Servers.
All servers that are part of cluster should be of same service pack level.
WebLogic has some internal applications deployed by default to perform some of internal
activities.
FiledistributionServlet: This servlet takes care of communication activities between Admin and
managed server. (Deployment, managed
server startup etc) Eg: During deployment, this
servlet takes care of copying the application on to all managed servers staging
directory.
WebLogic Classloading
A classloader is part of JVM that loads classes into memory. Classloader is responsible for
finding the classes at runtime.
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
8/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
Points to remember:
Java classloader would always search for a class from parent classloader. If two
classes with same name exist within an EAR/WAR file and System ClassPath, the class
from system classpath is picked up always because system classpath is parent classloader
of application classloader (from the flow diagram above)
Classes in WAR file can see/load EJB classes but not viceversa
because WAR classloader is child of EJB classloader
Threads are created out of thread pools. Each thread pool is meant for specific
purpose.
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
9/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
Defaults to 3 on Unix systems and 2 on Windows.Threads from this group are used for
socket reading purpose.
Threads from this group are used for communication between Admin and managed
servers.
Eg: Deployment of application.
Threads from this group are used for weblogic internal activities like RJVM heartbeats,
getting http state dump for JNDI updates in cluster.
10/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
thread dumps.
There are 15 threads by default. The number of execute threads can be modified through
Admin console.
For eg:
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
11/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
Weblogic Admin console (console.war) is a webapplication that has custom thread queue. If
we see weblogic.xml of console.war we can see the custom queue definition as:
<wl-dispatch-policy>
weblogic.admin.HTTP
</wl-dispatch-policy>
<wl-dispatch-policy>
my-sample-queue
</wl-dispatch-policy>
Deployment modes
Deployment can be done through Admin console OR command line using webogic.Deployer utility.
More about the deployer utility can be found at:
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
12/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
http://e-docs.bea.com/wls/docs81/deployment/tools.html#wldeployer
No stage
Stage
External_stage
Prepare : This phase makes sure that the application is in a state in which it can be
deployed reliably. During this phase, Admin server copies files to target (managed servers)
Activate: The activate phase involves the actual deployment i.e the classes are loaded
into memory. This takes place only if prepare phase is successful.
No Stage
In this mode, weblogic will deploy the application from the
Path= tag that is specified in the <Application> tag in config.xml.
weblogic will not make a copy of the app to any staging directory, nor will it make any
copy of the app to distribute to managed servers. All servers should have to share a
common file system so that all the servers would reference the application from one
physical location.
Stage
Weblogic will make a copy of all of the applications into the staging directory, and will
deploy the app from there. weblogic will do all the file transfer of the application files to the
managed servers. As part of deployment process weblogic would copy the application
(WAR/EAR) on to managed servers and deploy from there. This is default deployment mode
unless changed.
External_Stage
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
13/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
weblogic will deploy from the staging directory, but we (user) are responsible for
getting the application (EAR/EAR) to staging directory.Weblogic will do no file
transfer to the managed servers.
The difference between nostage and external stage is that nostage is intended for people who
have one shared disk, so the Path= attribute is meaningful for every server in the domain.
External stage is for people who don't use a shared disk, so Path= only has meaning only the
admin server. Each managed server will have a stagedPath that makes sense for that server.
WebLogic JDBC
Connection pool:
Connection pool is a named group of identical JDBC connections to a database that are
created when the connection pool is deployed.
Connection pooling is for
efficiency purposes to reduce the overhead of creating connection objects at runtime.
When server startup, depending on the initial JDBC pool capacity, connections are created.
Connections are created based on the information provided in pool attributes.
At runtime, Client asks for a connection, weblogic would take an existing connection from pool
and do a ping (query is user defined).
If ping is successful, weblogic would give connection to client, else it will recreate and give
the connection.
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
14/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
JDBC configuration
WebLogic would also include some of the frequently used drivers. These drivers come with
installation. These drivers are shipped for convenience purpose and it is advisable to use
the latest drivers available from vendors website.
The following are the that drivers come with weblogic installation: (Apart from weblogic type 4
integrated drivers)
15/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
Any driver that implements JDBC spec can be used with weblogic. The driver should be thread
safe too. (Suitable for multithreaded apps)
WebLogic would support type2 OCI drivers too. However thin driver is recommended because it
is 100% java and not prone to jvm crash or native memory leak. In case of Oracle, the
performance between thin driver and OCI driver has been pretty much narrowed down with
JDK1.4 (As per Oracle)
MultiPool
This is a pool of connection Pools. The pools can be connected to different DBMS or
specific setup like Oracle RAC.
You would need to choose the algorithm based on application requirements (Only one
algorithm can be applied)
1. High Availability: All connection requests are served from first pool in the list. *Only*
if the pool is not available, (connection fails and cannot be refreshed) the connection
is routed to the next pool in the list.
Loadbalacing:
Connection requests to the pool are roundrobined across the list of pools. However if a
connection request to a connection pool
was not successful and cannot be recreated (such as when DB is down or pool is suspended)
then the request is routed to next connection pool in the list.
Oracle RAC:
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
16/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
The multipool concept can be applied to Oracle RAC where each connection pool is connected
to one of the oracle instance and we can choose the algorithm.
There are quite a few limitations to multipool when global transactions are involved (XA).
WebLogic JMX
<JDBCConnectionPool
DriverName="oracle.jdbc.xa.client.OracleXADataSource"
Name=Pool" PasswordEncrypted="{3DES}DpbT3SFF8mI="
InitialCapacity="5" MaxCapacity="35"
URL="jdbc:oracle:thin:@localhost:1521:Mysql"/>
For JDBCConnectionPoolMBean, there are lot of other attributes with default values.
The values that we add through console override the default values.(like
MinCapacity,MaxCapacity etc)
An Mbean is just a concrete java class that is coded as per JMX specification and it
provides a set of setters and getters of the attributes.
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
17/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
As the name implies, this is read only and used to monitor/get the runtime values.
Extending our same JDBC pool example, the related runtime Mbean is :
JDBCConnectionPoolRuntimeMBean.
If we see the API, we can see methods like:
getActiveConnectionsCurrentCount()
These are runtime values at the particular instant.
General rule of thumb is, Mbeans for which the change involves persisting data to config.xml
are termed as configuration Mbeans.
(JDBC pool capacity, driver name, timeout value etc).
Mbeans that involve read-only data is termed a runtime Mbean.
The API documentation will give a better picture of all the methods
weblogic
This provides an easy way for other vendors like vignette to monitor and administer weblogic
resources.
WebLogic JNDI
Applications use naming services to locate objects in data sources, EJBs, JMS, MailSessions,
and so on in the network.
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
18/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
JNDI provides abstraction where the client need not worry of any changes on server side.
WebLogic JMS
JMS enables applications to communicate with one another through the exchange of
messages. All J2EE servers implement JMS specification.
In JMS we have Message Producer which creates/sends message and Message Consumer
who actually consumes the message and probably do some action further.
1) Publish-Subscribe Messaging
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
19/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
Publisher may send messages to a Topic, and all applications that are subscribed to the
topic will receive the message.
Eg: Consider a stock quote application. The publisher would broadcast the updated stock
quote (topic) and all the subscribers that are interested
(subscribed) in the particular stock quote will receive it.
A queue sender (producer) sends a message to a specific queue. A queue receiver (consumer) receives
messages from a specific queue
WebLogic Clustering
Basics
WebApplications
EJB and RMI applications
JNDI
LoadBalancing
Failover
Underlying mechanism/protocol
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
20/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
So this is a value add on feature that most application server vendors provide.
Since the spec does not talk about it, each vendor would implement this in their own way.
However the ultimate objectives are loadbalancing and failover.
Replication
WebLogic Server provides clustering support for servlets and JSPs by replicating the HTTP
session state of clients that access clustered servlets and
JSPs. WebLogic Server can
maintain HTTP session states in memory, a filesystem, or a database.
Replication involves, copying all the session information on to another server (called
secondary server). In case of weblogic this is a RMI (t3) call.
In-memory replication (Session info stored both in primary and secondary servers
memory. Most commonly used in cluster setup)
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
21/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
Request comes through hardware loadbalancer and gets forwarded to one of the Apache
server.
WebLogic Apache plugin gets invoked, delegates the http request to one of the servers in
cluster. (It gets the information of weblogic instances from httpd.conf file)
WebLogic server would create a session cookie and send back the session cookie as http
response header.
As long as browser is alive, the session cookie is passed (as http header) and Apache plugin
would parse the session cookie and route the request to same server.
Performance tuning
LAN Capacity
Run the application with xaprof and do the load test. After the load test kill the server
process.
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
22/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
Upon the process exit, JVM would create a summary which includes the max amount of
memory that has been utilized.
This serves as base point and max heap can be set more than this value.
There are other commercial tools available to record the heap usage like integration of
LoadRunner into wily introscope so that memory usage can be
monitored and recorded.
Its good to have the JVM not spend more than 5 to 10 secs for full GC because FullGC pauses
the application threads per cpu. (Some of the latest
JVMs have settings that would reduce the pause time but not 100%).
In those cases, try other GC settings (parallel GC or concurrent GC or incremental GC) and see
if it reduces the GC times.
OS has to schedule the threads so that all the threads get fair share.
Too little threads will cause CPU under utilization and request waiting.
This can be done by taking periodic thread dumps during load test
to see if threads
JDBC tuning
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
23/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
Use large enough pool to service largest number of concurrent database users
It is recommended to have the Max pool capacity equal to Execute thread count. (This
can be application dependant)
Process most of the data inside the database to minimize network traffic
On Solaris, set the TCP timewait interval to 60secs Suggested TCP values from BEA is
at:
http://e-docs.bea.com/wls/docs81/perform/HWTuning.html#1121083
NativeIO is fast compared to java IO because in java, the socket needs to be polled.
1) Server log
WebLogic server creates server log file by default under:
/<domain-name>/<server name>/<server name>.log
The location is configurable.
2) JDBC log
All SQL statements and DB related exceptions/errors.
This file is created under /<server name>/jdbc.log
Domain log
All domain level information is logged into this file.
This is subset of server log file.
<domain name>/<domain name>.log
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
24/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
5) Access log
All http requests are recorded in this log file
/<server name>/access.log
6) Transaction log
All servers record transaction in the tlog file
/<server name>/<server name>.tlog
Server Crash
Determine all potential sources of native code used by the WebLogic Server.
nativeIO.
Type2 jdbc driver.(We generally use thin driver for most applications)
Sometimes the JVM will produce a small log file that may contain useful information as to which
library the crash has originated from. (hs_err_pid*.log)
When a JVM is crashed, a core file(binary image of the process) is created. Run pmap and
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
25/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
pstack against the core file to get the library that caused the crash.
Demo to figure out offending library using existing pmap & pstack out files.
Check list:
3) Using debugger (gdb,dbx,adb) (if above two steps does not provide any information)
Server Hang
Multiple thread dumps are necessary to conclude that the threads are stuck and not
progressing.
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
26/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
Unix:
Open shell window and issue the command kill -3 <PID> where PID is java processID of
weblogic. Thread dumps are logged on to STDout file.
Windows:
Windows Service:
Open a command prompt and issue the command(Make sure beasvc.exe is in the PATH)
as:
-log:"d:\bea\domains\mydomain\myserver-stdout.txt
Before we analyze thread dumps, it is important to know the common thread states:
This state indicates that the thread is either running currently or is ready to run the
time the OS thread scheduler schedules it.
next
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
27/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
These threads are something to watch out because there is lock contention here. Thread
is waiting for a lock on object and some other thread is holding the lock.
In case of weblogic, the main worker threads are from group weblogic.kernel.defalt:
"ExecuteThread: '1' for queue: 'weblogic.kernel.Default'.
This is the set of threads we need to look for hang/slow performance issues.
This is a snapshot of idle thread waiting for some work to be assigned.
On an idle system you would see lot of threads in the below state:
As for thread dump analysis & conclusion, lets see a sample thread dump and drill into
it further
Demo of thread dump (Thread stuck issue on UAT)
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
28/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
If java heap is full, server process appears to be hung and not accepting any requests
because each request needs heap for allocating objects.
So if heap is full, none of the requests get served, all the requests fail with
java.lang.OutOfMemory
OutOfMemory Analysis:
OutOfMemory can occur because of real memory crunch or a memory leak causing the heap to
fill with orphaned objects.
First step is to enable GC and run the server again.
(-XX:printGCDetails).
The STDout file would show the garbage collection details.
If the error is because of memory leak, then we would need to use profilers like Introscope or
optmizeIT to figure out the source of leak.
Process size = java heap + native memory + memory occupied by the executables and libraries.
On 32 bit operating systems, the virtual address space of a process can go up to 4 GB. This is
data bit limitation (2 pow 32)
Out of this 4 GB, the OS kernel reserves some part for itself (typically 1 2 GB).
This is not a limitation on 64 bit machines like solaris(sparc) or windows running on Itanium (64
bit)
29/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
OOM can occur due to fragmentation. In this situation, we can see free memory available but still
get OutOfMemory errors.
Over a period of time, memory allocation is becomes scattered and there might not be enough
contiguous memory available.
For eg: The verbose:gc output might look like the following if there was a fragmentation of heap.
There is free memory available, but still JVM throws OOM error.
(Most of the fragmentation bugs are resolved in Sun JDK1.4.2_xx)
Best approach is to try the latest minor version of JVM and if does not work out, we need to
work with vendor to get it fixed.
vmstat :
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
30/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
sar:
An OS utility that is termed as system activity reporter
If the application uses SSL, then the server performs slow compared to non SSL.
SSL reduces the capacity of the server by about 33 to 50 percent depending upon the
strength of encryption used in the SSL connections.
Process running out of File descriptors. Server cannot accept further requests
because sockets cannot be created. (Each socket created consumes a FileDescriptor)
OR
In the above case, the lsof utility would help. lsof utility shows the list of all open
filedescriptors. From the list of open files, we ( application owner) can easily figure out if it
is a bug or expected behavior. If it is expected behavior, then the number of FDs needs to
be increased. (default number is 1024)
Before knowing about high CPU analysis, it would be helpful if we know about solaris
threading Model.
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
31/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
Process using high CPU is not always bad. This is a common misconception. Infact we
want our application to use the CPU efficiently.
What we do not want is, a single thread or couple of threads consuming all the CPU
forever(In an infinite loop) and not allowing other threads to get the CPU share(timeslice).
2) Run prstat L p <PID> where PID is processID that is consuming the highest CPU.
From the above command you would see the lwps that are consuming the high CPU.
32/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
If the application is huge(contains more than 100 JSPs), we might encounter this problem with
default JVM settings.
The reason for this is, the MaxPermSpace getting filled up.
This space is used by JVM to store its internal datastructures as well
as class definitions. JSP generated class definitions are also stored in here.
MaxPermSpace is outside java heap and cannot expand dynamically.
So fix is to increase it by passing the argument in startup script of the server:
XX:MaxPermSize=128m (default is 64m)
If there is any issue between Apache and Weblogic and the cause is not obvious, enable debug
at Apache layer. In http.conf file add:
Debug ALL
This would create file called wlproxy.log under /tmp of Apache machine. The log would contain
all the request/response headers between Apache and WebLogic.
Most of the plug-in issues in WLS8.1 were centered around the attribute KeepAliveEnabled.
For most of the socket related errors, it worth trying turning off
KeepAliveEnabled and redo the test.
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
33/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
The classpath thats being loaded by the process is logged in weblogic log file as part of
startup sequence. There is other system related info too:
There is another classpath definiton and that is in manifest.mf file within the weblogic.jar.
Deployment
Side by side deployment. A new version of application can be deployed with Zero downtime.
Old version will be retired gracefully after all pending requests have been served.
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
34/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
Clustering
supported)
A new life cycle state, ADMIN, facilitates application redeployment, maintenance, and
troubleshooting. In the ADMIN state, WebLogic Server is running, but available only for
administration operations, allowing you to perform server and application-level
administration tasks without risk to running applications.
There is a single thread pool, where all types (internal work as well as application
requests) of work is executed. WebLogic Server prioritizes work based on rules you define
(can be defined in config.xml,weblogic-application.xml,web.xml etc), and run-time metrics,
including the actual time it takes to execute a request and the rate at which requests are
entering and leaving the pool.
The common thread pool changes its size automatically to maximize throughput. The
queue monitors throughput over time and based on history, determines whether to adjust
the thread count.
Work Manager
Diagnostic Framework
Server can be configured to create, collect, analyze, archive, and access diagnostic data
generated by a running server and the applications deployed within its containers.
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
35/36
1/30/2014
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
http://e-docs.bea.com/wls/docs90/wldf_configuring/intro.html#1062080
https://sites.google.com/site/sureshsvnsite/weblogic-intro?tmpl=%2Fsystem%2Fapp%2Ftemplates%2Fprint%2F&showPrintDialog=1
36/36