SAP Startup Troubleshooting Guide for Netweaver

Application Server
wiki.scn.sap.com/wiki/display/SI/SAP+Startup+Troubleshooting+Guide+for+Netweaver+Application+Server

Purpose
This document was designed to help SAP customers to troubleshoot and resolve the
majority of SAP system startup problems own their own, minimizing the business
impacts and costs attached to a standard support ticket life cycle. As time goes by, new
topics and startup error scenarios will be added to this guide. Stay tuned!

NEWS from December 2016: if SAP is running on AIX 7.1 or 7.2 and you are unable to
start SAP after updating the AIX, read the SAP note 2402207.

1. Preliminary Analysis
This initial checklist will approach mandatory settings that have proven to be root cause
of startup issues in several occasions. With this in mind, please *DO NOT* skip any
phases.

1.1. IS DATABASE UP AND RUNNING?

This is absolute prerequisite for SAP SYSTEM to start. In order to test this, operating
system command R3trans -d can be executed (as SIDadm user). Any return code
different than 0 means that DB cannot be reached by DIALOG work processes, and so
SAP system will not initiate. The following return indicates that database is up, running
and reachable by SAP work processes:

<hostname>:SIDadm > R3trans -d

This is R3trans version X.xx (release 742 - 18.11.14 - 20:14:09).
unicode enabled version
R3trans finished (0000).

There is NO POINT in moving the analysis further while the return code
is different than zero.
In case the return code is different than zero, please involve your DBA/Network/OS
teams.

1.2. IS SAP STARTUP SERVICE CORRECTLY DEFINED?

UNIX:
File "/usr/sap/sapservices" must contain the correct service definitions.

LD_LIBRARY_PATH=/usr/sap/<SID>/exe:$LD_LIBRARY_PATH;

export LD_LIBRARY_PATH;
1/16
/usr/sap/<SID>/<instance_nr>/exe/sapstartsrv
pf=/usr/sap/<SID>/SYS/profile/<SID>_<instance_nr>_<hostname> -D -u <SIDadm>

WINDOWS:

1. Run "services.msc";
2. Look for the relevant service SAP<SID>_<instance_nr>;
3. Check the command line assigned to it:

<DRIVE>:\usr\sap\<SID>\<instance>\exe\sapstartsrv pf=<DRIVE>:\usr\sap\
<SID>\SYS\profile\<START_PROFILE>

1.3. START PROFILES:

In order to correctly start the required processes such as message server, enqueue
server and the disp+work, the SAP startup agent (sapstartsrv) needs to find a correct
scenario for that. This scenario consists on a list of "Start_Program_<xx>" commands that
should be present an the "START" profile (for all kernel releases up to 740) or at the
"INSTANCE" profile (kernel release 740 onwards).

These start entries are mandatory in the profile being used by sapstartsrv process. You
can use the process list on UNIX and the service definition at WINDOWS to find out what
is the exactly profile currently assigned to sapstartsrv process.

Examples:
Entries needed for starting the dispatcher + work processes:

_DW=dw.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)
Execute_0X=local rm -f $(_DW)
Execute_0X=local ln -s -f $(DIR_EXECUTABLE)/disp+work$(FT_EXE) $(_DW)
Start_Program_0X=local $(_DW) pf=$(_PF)

Example of message server start entries:

_MS = ms.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)
Execute_0X = local rm -f $(_MS)
Execute_0X = local ln -s -f $(DIR_EXECUTABLE)/msg_server$(FT_EXE) $(_MS)
Restart_Program_0X = local $(_MS) pf=$(_PF)

Enqueue server related start commands:

_EN = en.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)
Execute_0X = local rm -f $(_EN)
Execute_0X = local ln -s -f $(DIR_EXECUTABLE)/enserver$(FT_EXE) $(_EN)
Restart_Program_0X = local $(_EN) pf=$(_PF)

1.4. LOGGING IN OS LEVEL DIRECTLY AS <SIDadm> USER:

2/16
It is mandatory that SIDadm user should be able to log on at operating system level
*WITHOUT ANY* other prompts such as:

1. Company policies;
2. User policies;
3. Next scheduled downtimes/System news;
4. Logging in as another user and SUdoing to SIDadm;

This can cause several problems throughout different scenarios, but mainly on upgrade
procedures. Further details on SAP note 1301712.

1301712 - UNIX sapstartsrv: Determining the environment

1.5: IS SAP STARTUP AGENT (SAPSTARTSRV) RUNNING?

There must be ONLY ONE sapstartsrv process running for each instance. Examples for
each platform below:

Examples
As an example, we will use a centralized system with the following 03 instances running
on the same host (ASCS, PAS and DI). The following output should return for the
operating system level command:

ps -ef | grep <SID> | grep sapstartsrv

(...) /usr/sap/<SID>/ASCS00/exe/sapstartsrv
pf=/usr/sap/<SID>/SYS/profile/<SID>_ASCS01_<hostname> -D
(...) /usr/sap/<SID>/DVEBMGS01/exe/sapstartsrv
pf=/usr/sap/<SID>/SYS/profile/<SID>_DVEBMGS00_<hostname> -D
(...) /usr/sap/<SID>/D02/exe/sapstartsrv
pf=/usr/sap/<SID>/SYS/profile/<SID>_D02_<hostname> -D

Profile pattern names returned with the command can be such as

"START_<INSTANCE>_<hostname>" in older releases.
There can be another sapstartsrv process (as below) but it will play no parts in the
startup of the instance.

(...) /usr/sap/hostctrl/exe/sapstartsrv pf=/usr/sap/hostctrl/exe/host_profile -D

When sapstartsrv service is not running you could run manually the above
command for the desired instance
If the service doesn't start and nothing it's written in "sapstartsrv.log" it might
happen the binaries are damaged in the local exe folder. In this case has to be
replaced

WINDOWS:
1. You shoud first configure task manager output so it displays the command line
related to the selected process:
3/16
 2. Run task manager → Switch to tab "Processes" → Menu "view" → "columns" → 'PID'
and 'COMMAND LINE';
3. Then using task manager tab "Processes", look for the instance sapstartsrv process
with similar:

<DRIVE>:\usr\sap\<SID>\<instance>\exe\sapstartsrv pf=<DRIVE>:\usr\sap\
<SID>\SYS\profile\<START_PROFILE>

1.6. IS THERE ANY SAP PROCESS ALREADY RUNNING?

After trying to start the instance (either through the SAP MMC, the "startsap" command
or using "sapcontrol ... -function Start"), no error is returned. However, the SAP
processes (Dispatcher - ABAP; jstart - Java) do not start. No trace file is updated either.

Verify whether any process is already in "GREEN" or "YELLOW" status (e.g., the IGS).

hostname:sidadm 40> sapcontrol -nr 33 -function GetProcessList

12.10.2015 20:15:45
GetProcessList
OK

name, description, dispstatus, textstatus, starttime, elapsedtime, pid

msg_server, MessageServer, GRAY, Stopped, , , 1234
disp+work, Dispatcher, GRAY, Stopped, , , 1111
igswd_mt, IGS Watchdog, GREEN, Running, 2015 07 31 11:59:15, 1:21:09, 2323
hostname:sidadm 41>

All processes must have the "GRAY" status. Otherwise, the SAP Startup Agent
(sapstartsrv) will "ignore" the start command. Stop the instance first, so all processes are
stopped. Then, start the instance again.

1.7. PERMISSIONS:
UNIX:
"/TMP" DIRECTORY:
"/tmp" directory's permissions must match the following. The final 't' for indicates sticky
bit:

drwxrwxrwt 22 root root 4096 Apr 6 20:16 tmp

927637 - Web service authentication in sapstartsrv as of Release 7.00

SAPSTARTSRV.LOG:
This file located on "/usr/sap/<SID>/<INSTANCE>/work/" should be owned by <SIDadm>
user. For specific cases when owned by root, sapstartsrv (running as <SIDadm> user) will
not be able to access the file and thus the system will fail to start. Following entry is seen
in "/var/messages":

4/16
Unable to open trace file sapstartsrv.log. (Error 13 Permission denied)

SAPUXUSERCHECK FILE
The permissions for this file should look like this:

-rwsr-x 1 root sapsys 1293660 Apr 06 2015 sapuxuserchk

There is one SAPUXUSERCHECK file per instance (/usr/sap/<SID>/<instance>/exe) and the

permissions and ownership should look alike for all files. The following command can
help tracking the files:

find /usr/sap/<SID>/* -name sapuxuserchk

1.8: IP / HOSTNAME RESOLUTION:

LOCALHOST
startup commands from sapcontrol will be using "localhost" implicitly (when no "-host"
option is explicitly informed) and for such the loopback IP "127.0.0.1" should be resolved
for it. What is the output for "niping -v -H localhost"? The loopback IP should be the only
one returned. If not, please involve your network team.

Lookup of hostname: localhost (NiHostToAddr) → IP-Addr.: 127.0.0.1

Lookup of IP-Addr.: 127.0.0.1 (NiAddrToHost) → Hostname: localhost

1476386 - Wrong localhost name resolution setting

HOSTNAME
What IP is resolved for the local hostname (output of OS level command niping -v -H
<hostname>)? There is a known error described in SAP note 1054467 where the
loopback IP 127.0.0.1 is wrongly resolved for it.

1054467 - Local host name refers to loopback address

2. IDENTIFYING AFFECTED STAGE

This troubleshooting (starting from here) *considers TRUE* that all points from the initial
checklist are done. Please keep in mind that all of above steps were taken from real
incidents and they were, in several of those cases, the required fix the for the issue
resolution. The very first question that needs to be answered is: which
component/process has failed? Looking at the image below, it is possible to find out the
sequence is which the logs will be written and therefore where the specific component
information can be found.

5/16
The main approach is to begin for the latest updated files, based on this image's timeline.
As an example, in case the work processes traces (dev_w##) are not updated after the
latest failed start attempt, then the dispatcher trace (dev_disp) needs to be verified and
so on.

3. DISPATCHER (dev_disp):
1. Dispatcher is usually the first and most important trace file analyzed for startup
issues
2. Most of the times the reason for the startup failure will be here.

In the dispatcher trace, we'll initially look for critical error situations, which directly
contributed to the startup fail. If such straightforward occurrences cannot be found, then
we look for any situation which would, indirectly, lead to the same situation. As an
example, another component has crashed, therefore dispatcher has to shut itself down.

Example of error where it's possible to see the cause right away:
***LOG Q0K=> DpMsAttach, mscon ( <server_name>) [dpMessageSer 1652]
*** DP_FATAL_ERROR => DpMsAttach: local hostname '<server_name>' is resolved to
loopback address (cf. SAP note 1054467 for details)
*** DISPATCHER EMERGENCY SHUTDOWN ***

Example of dispatcher startup error caused by the crash of all work processes. The very
next step in this case is to check the "dev_w#" traces and find out what is killing the
disp+work processes and, ultimately, what is the cause of the instance failure:

*** ERROR => DpHdlDeadWp: W64 (pid 1898) died (severity=0, status=65280) [dpxxwp.c
1739]
DpTraceWpStatus: child (pid=1898) exited with exit code 255
6/16
*** DP_FATAL_ERROR => DpWPCheck: no more work processes
*** DISPATCHER EMERGENCY SHUTDOWN ***

*** IMPORTANT:

--> Whenever suspicious about a dispatcher crash and the following entries are found:

DpSigInt: caught signal 2

DpHalt: shutdown server ><server>_<SID>_<instance_number> < (normal)
DpModState: change server state from ACTIVE to SHUTDOWN
DpHalt: stop work processes

Please keep in mind that a manual shut down happened. This can be proved by the
"signal 2" entry and also by the "< (normal)" sign, next to the shut-down sign.

3.1: MOST COMMON DISPATCHER CRASH SITUATIONS:

In this list we bring the most common causes for a "dispatcher crash", along with its
resolution:

I. "SI_EPORT_INUSE":

These error entries are preventing the dispatcher from initialize and they are not caused
by SAP but network inconsistencies. At the time dispatcher is trying to start and bind its
default port 32<instance_number> it finds that there is already a process binding this
port, thus, it aborts logging below entries:

***LOG Q0I=> NiIBindSocket: bind (10013: WSAEACCES: Permission denied) [nixxi.cpp 3740] <
Problem...
*** ERROR => NiIBindSocket: SiBind failed for hdl 9/sock 1292
(SI_EPORT_INUSE/10013; I4; ST; 0.0.0.0:3202) [nixxi.cpp 3740]
*** ERROR => DpCommInit: NiBufListen (rc=-4) [dpxxdisp.c 10223]
*** DP_FATAL_ERROR => DpSapEnvInit: DpCommInit
*** DISPATCHER EMERGENCY SHUTDOWN ***

RESOLUTION:

You must involve your network team and find out what process is listening on port
32<instance_number> and ensure that the dispatcher port is free at the time it is
starting up. As an example, in UNIX you can try the OS command "netstat -an | grep
32<instance_number>" when searching for a process listening on that port.

II. PLATFORM PATCHNO... BUT MY PATCHNO IS:

7/16
Following error is seen for scenarios where the path release of dispatcher and message
server is not the same. This can happen for couple of reasons but the major number of
occurrences is related to issues on SAPCPE which failed to copy newest files from the
GLOBAL EXE folder to the INSTANCE EXE folder, resulting in mixed releases.

*** ERROR => platform patchno is xxx [dpxxdisp.c 17170]

*** ERROR => but my patchno is yyy [dpxxdisp.c 17171]
*** DP_FATAL_ERROR => Release inconsistent with running system
*** DISPATCHER EMERGENCY SHUTDOWN ***

RESOLUTION:

You must ensure that all instances are running with the same kernel release. As a
potential fix, you can copy manually the files from the global "EXE" folder to the instance
"EXE" folder.

III. BAD MEMORY PARAMETRIZATION:

*** ERROR => SHM2_EsCreateShared: esSHMAddSharedSegment(,userID=268641024) returned 2

[esuxshm2.c 2451]
*** ERROR => DpEmInit: EmInit (1) [dpxxdisp.c 10864]
*** ERROR => DpMemInit: DpEmInit (-1) [dpxxdisp.c 10778]
*** DP_FATAL_ERROR => DpSapEnvInit: DpMemInit
*** DISPATCHER EMERGENCY SHUTDOWN ***

OR

*** ERROR => SHM2_EsInit: invalid argument ES/SHM_MAX_SHARED_SEGS=2. Valid range is 1 ... 1d
[esuxshm2.c 1381]
*** ERROR => DpEmInit: EmInit failed (1) [dpInit.c 1475]
*** ERROR => DpMemInit: DpEmInit (-1) [dpInit.c 1339]
*** DP_FATAL_ERROR => DpSapEnvInit: DpMemInit
*** DISPATCHER EMERGENCY SHUTDOWN ***

RESOLUTION:

There are a few interconnected memory parameters that must reach each other
requisites. Mainly, following SAP notes have to be consulted:

# 1998233 - IBM i: SHM2_EsInit: invalid argument ES/SHM_MAX_SHARED_SEGS

# 1014010 - EG memory up to 32 GB (em/global_area_MB up to 32000)

And mainly those two rules should be reviewed:

8/16
ES/SHM_MAX_SHARED_SEGS <= ES/SHM_PROC_SEG_COUNT - ES/SHM_MAX_PRIV_SEGS
ES/SHM_PROC_SEG_COUNT >= ES/SHM_MAX_SHARED_SEGS + ES/SHM_MAX_PRIV_SEGS

IV. SI_EADDR_NAVAIL:

This error is commonly seen when "localhost" cannot be resolved as "127.0.0.1".

***LOG Q0I=> NiIBindSocket: bind (68: Can't assign requested address) [nixxi.cpp 3239]
*** ERROR => NiIBindSocket: SiBind failed for hdl 0 / sock 7
(SI_EADDR_NAVAIL/68; I4; DG; xx.y.zzz.ww:3200) [nixxi.cpp 3239]
*** ERROR => DpCommInit: NiDgHdlBindName failed: -16 [dpxxdisp.c 10263]
*** DP_FATAL_ERROR => DpSapEnvInit: DpCommInit
*** DISPATCHER EMERGENCY SHUTDOWN ***

RESOLUTION:

See PRELIMINARY CHECKLIST above, topic [1.7] and ensure that "localhost" is resolved
as loopback IP "127.0.0.1" and vice-versa.

V. NO MORE WORK PROCESSES:

Dispatcher is shutting itself down because there are no more work processes available
for processing of requests.
In order to pinpoint the cause, work processes traces have to be analyzed. Just follow
the initialization entries and soon the ERROR entries with the root cause will be seen.

*** DP_FATAL_ERROR => DpWPCheck: no more work processes

*** DISPATCHER EMERGENCY SHUTDOWN ***

RESOLUTION:

Whenever this error is the root cause for a dispatcher crash, it means that in this very
same trace you'll see lots of entries related to the crash of every single work process.
These entries look like this:

------------------------------
*** ERROR => DpHdlDeadWp: W0 (pid 21104) died (severity=0, status=65280)
[dpxxwp.c 1727]
DpTraceWpStatus: child (pid=21104) exited with exit code 255
------------------------------

We will then search in the dispatcher trace for the very first occurrence of work
process that crashed with entries similar to the one above.
There we see that the work process 0 (zero) died. Then we need to change the
9/16
investigation to the work process trace. In this case it will be the "dev_w0", located at the
work directory of the instance.

Check the "usr/sap/<SID>/<INSTANCE>/work/dev_w#" trace to find out why work

processes are being killed. Most common cause are problems to reach/connect to
database.

(FAIL TO SPOT TOPIC [1.1] FROM PRELIMINARY CHECKLIST)

VI: DISPATCHER NOT ALLOWED TO CONNECT TO MESSAGE SERVER:

*** ERROR => not allowed to connect to message server via port XXXX
[dpxxdisp.c 12175]
*** ERROR => Please check your configuration (profile parameter
rdisp/msserv_internal) [dpxxdisp.c 12176]
DpHalt: shutdown server >PTSAPT01_QS1_03< (normal)

RESOLUTION:

SAP note # 1285956 - Not allowed to connect to message server

VII: "GATEWAY COULD NOT BE STARTED":

Straightforward. Dispatcher shut down as the gateway reader could not be started.

*** ERROR => gateway (pid 29081) died [dpxxdisp.c 16872]

DpGwCheck: appc_adm.stat = 2
*** DP_FATAL_ERROR => Gateway could not be started - I better exit now

RESOLUTION:

Gateway trace file "dev_rd" should be checked for the rootcause. A very common cause
is the unavailability of the gateway's default port 33<instance_NR> during instance
startup.
If this is the case, the following entries are seen in "dev_rd":

***LOG Q0I=> NiIBindSocket: bind (226: Address already in use) [nixxi.cpp 3738]
*** ERROR => NiIBindSocket: SiBind failed for hdl 1/sock 11
(SI_EPORT_INUSE/226; I4; ST; 0.0.0.0:3310) [nixxi.cpp 3738]

*** In this case the instance number was 10, hence "0.0.0.0:3310".
If you face the same situation, you have to involve your network/OS teams and find
out what process has already bound the port.
10/16
VIII. Es2IResCreateShm: shmget failed:
This issue is mainly seen on SAP systems based on HP-UX servers.

*** ERROR => Es2IResCreateShm: shmget failed (size= 17980981248. Check OS kernel parameter
shmmax. (22: Invalid argument) [es2ux.c 219]
*** ERROR => Es2ResCreateFiles: Es2IResCreate failed [es2xx.c 1324]
*** ERROR => Es2ResCreate: Es2ResCreateFiles failed [es2xx.c 1507]
*** ERROR => DpEmInit: Es2ResCreate (1) [dpInit.c 1642]
*** ERROR => DpMemInit: DpEmInit (-1) [dpInit.c 1417]
*** DP_FATAL_ERROR => DpSapEnvInit: DpMemInit
*** DISPATCHER EMERGENCY SHUTDOWN ***

RESOLUTION:

You must correctly set the OS kernel parameters as per below SAP note:

# 172747 - SAP on HP-UX: OS kernel parameter recommendations

Below SAP notes can provide additional info as well:

# 180251 - HP-UX 64 Bit Kernel, Startup schlaegt fehlt

# 1827388 - System cannot start due to Es2IResCreate - shmget failed - File exists error

IX. MULTICAST ERROR:

This error is present only as of kernel 740. The SAP Kernel release 740 uses a multicast
address at the network 224.0.0.0/24 to dispatch tasks to the work processes.
When required settings related to multicast are not in place, the instance will not start
and in the "dev_disp.new" trace file, the following entries are logged:

*** ERROR => NiISetSockOpt: SiSetSockOpt failed for hdl 1/sock 6

(SI_EADDR_NAVAIL/19; I4; DG; IPPROTO_IP-IP_ADD_MEMBERSHIP) [nixxi.cpp 9154]
*** ERROR => NiIDgHdlGroupAdm: failed to join 224.0.0.129 on interface 0.0.0.0 [nixxi.cpp 9572]
*** ERROR => RqQInit: NiDgHdlJoinGroup failed (rc=-10) [dpQueue.c 1559]
*** ERROR => DpIPCInit2: RqQInit [dpxxtool2.c 514]
*** DP_FATAL_ERROR => DpSapEnvInit: DpIPCInit2
*** DISPATCHER EMERGENCY SHUTDOWN ***

RESOLUTION:

--> For customers with kernel 740, the following notes guide you on how to set up
multicast in a correct way:

# 1932555 - Dispatcher startup issue with SAP Kernel 740

11/16
 # 1931675 - IP multicast configuration for dispatcher wakeup mechanism

--> For customers with kernel equal of higher than 741 PL 47, SAP does reccommend
the usage of a new implementation based on events.
This feature is enabled with the following parameter (which only exists as of kernel
741 PL 47):

rdisp/queue/use_events_for_dispatching=on

--> This parameter's default value is ON starting from SAP kernel 742.

X. APPLICATION SERVER'S SHARED MEMORY CORRUPTION:

*** ERROR => e=22 semop(3145913,(0,-1,6144),1) (22: Invalid argument) [semux.c 577]
*** ERROR => DpITmSlotAllocate: SemRq [dpxxtool2.c 4333]
*** ERROR => e=22 semop(3145913,(0,-1,6144),1) (22: Invalid argument) [semux.c 577]
*** DP_FATAL_ERROR => DpTmDisconnect: SemRq
*** DISPATCHER EMERGENCY SHUTDOWN ***

RESOLUTION:

As these entries are related to a crash other than failed startup attempts, the
resolution would be to start the instance once again.
The easiest way to reproduce this scenario (and most common cause of the
crash) is by removing the instance's IPC keys on runtime wich, by obvious reasons,
*SHOULD NOT BE DONE*.

# 1641668 - Various semop errors in workprocess traces

XI. WRONG EG (EXTENDED GLOBAL MEMORY) CONFIGURATION:

*** ERROR => EgInit: requested segment size (105327362048 bytes) too large.Max size =
34359738367 bytes [egxx.c 244]
*** ERROR => DpEmInit: EmInit (1) [dpxxdisp.c 10757]
*** ERROR => DpMemInit: DpEmInit (-1) [dpxxdisp.c 10671]
*** DP_FATAL_ERROR => DpSapEnvInit: DpMemInit
*** DISPATCHER EMERGENCY SHUTDOWN ***

RESOLUTION:

The instance will fail to start and log the above entries in case the EG configuration is
not maintained as per below SAP notes:

# 785365 - Extended Global Memory (EG) increased from 2 GB to 8 GB

# 1014010 - EG memory up to 32 GB (em/global_area_MB up to 32000)
Hint
12/16
By following this logic, any dispatcher related probem can be handled, even for
different logged errors.

Hint

There are SAP documentation for most of the possible cases. Research on them.

04. SAPCONTROL'S INSTANCE MANAGEMENT

COMMANDS

SAPSTARTSRV (through SAPCONTROL interface) offers a variety of web methods that can
be used to monitor/manage the instances of the system. They are specially useful for
handling startup error scenarios. The "must-know" ones are listed below and also
the general pattern for executing them:

GetSystemInstanceList
OK
hostname, instanceNr, httpPort, httpsPort, startPriority, features, dispstatus
testserver001, 0, 50013, 50014, 3, ABAP|GATEWAY|ICMAN|IGS, GREEN
testserver001, 1, 50113, 50114, 1, MESSAGESERVER|ENQUE, GREEN
testserver001, 2, 50213, 50214, 3, ABAP|GATEWAY|ICMAN|IGS, GREEN

sapcontrol -nr <instance number> -function <web method name>

4.1: "GetSystemInstanceList":
This one returns the list of active instances for the system, along with component list
and the current status.

GetSystemInstanceList
OK
hostname, instanceNr, httpPort, httpsPort, startPriority, features, dispstatus
testserver001, 0, 50013, 50014, 3, ABAP|GATEWAY|ICMAN|IGS, GREEN
testserver001, 1, 50113, 50114, 1, MESSAGESERVER|ENQUE, GREEN
testserver001, 2, 50213, 50214, 3, ABAP|GATEWAY|ICMAN|IGS, GREEN

EXPECTED OUTPUT:

- One entry per instance of the system, including ASCS, SCS INSTANCES;
- GREEN values for all entries on column "dispstatus".
- For all instances running on the same host (column hostname), there must be
a differente value on "instanceNr".

13/16
4.2: "GetProcessList":

GetProcessList web method will bring the component list currently in place on any of the
instances seen in the first command. It will come in handy as soon as the web method
"GetSystemInstanceList" returns one instance reporting as YELLOW. This will happen
when at least one component is presenting some kind of probem, so now it is possible to
check that affected process with more details:

GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
disp+work, Dispatcher, GREEN, Running, 2015 03 23 23:51:35, 263:27:26, 25549
igswd_mt, IGS Watchdog, GREEN, Running, 2015 03 23 23:51:35, 263:27:26, 25550
gwrd, Gateway, GREEN, Running, 2015 03 23 23:51:36, 263:27:25, 25590

icman, ICM, GREEN, Running, 2015 03 23 23:51:36, 263:27:25, 25591

4.3: "GetVersionInfo":

Great for kernel consistency checks, this output will return the current kernel release +
patch for each and every of its components within the instance informed as argument in
the command line.
Excluding the DB* libraries, all the remainning components of the instance must
always share the same kernel and patch release on a consistent scenario.

14/16
 GetVersionInfo
OK
Filename, VersionInfo, Time
/usr/sap/WSO/DVEBMGS00/exe/sapstartsrv, 742, patch 28, changelist 1540128, RKS compatibility
level 0, optU (Nov 26 2014, 19:45:41), linuxx86_64, 2014 11 26 19:29:07
/usr/sap/WSO/DVEBMGS00/exe/disp+work, 742, patch 28, changelist 1540128, RKS compatibility
level 0, optU (Nov 26 2014, 19:45:41), linuxx86_64, 2014 11 26 19:36:32
/usr/sap/WSO/DVEBMGS00/exe/gwrd, 742, patch 28, changelist 1540128, RKS compatibility level 0,
optU (Nov 26 2014, 19:45:41), linuxx86_64, 2014 11 26 18:48:09
/usr/sap/WSO/DVEBMGS00/exe/msg_server, 742, patch 28, changelist 1540128, RKS compatibility
level 0, optU (Nov 26 2014, 19:45:41), linuxx86_64, 2014 11 26 18:48:09
/usr/sap/WSO/DVEBMGS00/exe/dboraslib.so, 742, patch 5, changelist 1503525, RKS compatibility
level 0, dbgU (Jun 13 2014, 23:23:51), linuxx86_64, 2014 12 05 17:53:41
/usr/sap/WSO/DVEBMGS00/exe/dbmssslib.so, 742, patch 28, changelist 1540128, RKS compatibility
level 0, optU (Nov 26 2014, 19:45:41), linuxx86_64, 2014 11 26 18:47:06
/usr/sap/WSO/DVEBMGS00/exe/dbdb2slib.so, 742, patch 28, changelist 1540128, RKS compatibility
level 0, optU (Nov 26 2014, 19:45:41), linuxx86_64, 2014 11 26 18:47:03
/usr/sap/WSO/DVEBMGS00/exe/dbdb4slib.so, 742, patch 28, changelist 1540128, RKS compatibility
level 0, optU (Nov 26 2014, 19:45:41), linuxx86_64, 2014 11 26 19:29:51
/usr/sap/WSO/DVEBMGS00/exe/dbdb6slib.so, 742, patch 28, changelist 1540128, RKS compatibility
level 0, optU (Nov 26 2014, 19:45:41), linuxx86_64, 2014 11 26 18:47:06
/usr/sap/WSO/DVEBMGS00/exe/dbsybslib.so, 742, patch 28, changelist 1540128, RKS compatibility
level 0, optU (Nov 26 2014, 19:45:41), linuxx86_64, 2014 11 26 18:47:08
/usr/sap/WSO/DVEBMGS00/exe/enserver, 742, patch 28, changelist 1540128, RKS compatibility
level 0, optU (Nov 26 2014, 19:45:41), linuxx86_64, 2014 11 26 18:47:48
/usr/sap/WSO/DVEBMGS00/exe/icman, 742, patch 28, changelist 1540128, RKS compatibility level
0, optU (Nov 26 2014, 19:45:41), linuxx86_64, 2014 11 26 18:48:12
/usr/sap/WSO/DVEBMGS00/exe/sapwebdisp, 742, patch 28, changelist 1540128, RKS compatibility
level 0, optU (Nov 26 2014, 19:45:41), linuxx86_64, 2014 11 26 18:48:07
/usr/sap/WSO/DVEBMGS00/exe/jcontrol, 742, patch 28, changelist 1540128, RKS compatibility level
0, optU (Nov 26 2014, 19:45:41), linuxx86_64, 2014 11 26 18:49:16
/usr/sap/WSO/DVEBMGS00/exe/jlaunch, 742, patch 28, changelist 1540128, RKS compatibility level
0, optU (Nov 26 2014, 19:45:41), linuxx86_64, 2014 11 26 19:08:36
/usr/sap/WSO/DVEBMGS00/exe/jstart, 742, patch 28, changelist 1540128, RKS compatibility level 0,
optU (Nov 26 2014, 19:45:41), linuxx86_64, 2014 11 26 19:28:59

EXPECTED OUTPUT:

- All entries must share the same kernel release and patch. In this case 742 patch 28
(with the exception of the ones starting with db*).
Hint

There are hundreds of web methods available for the most diverse cases. They
can be listed by typing just "sapcontrol" command as SIDadm user.

Hint

These instance management commands can help to quickly find out version and
specific component details.

15/16
 CONCERNING THIS DOCUMENT...

The original concept of this document is that it'll be never finished. Instead, constantly
improved, based on the incoming feedback. For this reason, you are invited to share
whether or not the information present here helped you to solve a startup problem and,
even more importantly, in case it didn't. In that case you are welcome to share your own
scenario and inform what was the solution applied, constraints, specifics.

16/16