Determine Symptoms

Try to establish (as quickly as possible) the surface area of the problem:

Connectivity? (Telnet is good; if you get an error page returned in the browser, something's
obviously working - eliminate connectivity first)
General App Pool failure, or specific to a content type? (Do ASPX files work/not work,
but .HTM work? Do you have canary files for each app and content type?)
Specific in-app failure, hang, or crash? (Most of this is for hangs and app failures; crashes
dictate their own methodology: get a crash dump, debug it)

Collect Data
1. Grab whatever time-sensitive/timely data you will need to resolve the issue later. Don't
worry about persistent stuff - Event Logs and IIS logs stick around, unless you're a
compulsive clearer, in which case: stop it. (Those that don't have an Event Log of last week
are doomed to repeat it)
2. Determine the affected worker process
o

APPCMD LIST WP

can help with this, or the Worker Processes GUI at the Server

level.
o

If using the GUI, don't forget to look at the Current Requests by right-clicking the
worker process - that'll show you which module the requests are jammed in.

Determine the scope (just one App Pool, multiple App Pools, two with dependencies
- this depends on your app and website layout)

Grab a memory dump of the worker process - once you've identified which App
Pool has the problem, identify the relevant Worker Process, and use Task manager to
create a memory dump by right-clicking that process. Note the filename for later.

On Task Manager: You need to use the same bitness of Task Manager as the Worker
Process you're attacking with it - if you dump a 32-bit WP (w3wp*32) with 64-bit
Task Manager, it's not going to be interpretable. If dumping a 32-bit process on 64-bit
Windows,
you
need
to
exit
Task
Manager,
run
%WINDIR
%\SYSWOW64\TaskMgr.exe to get the 32-bit version, then dump with the same
bitness. (a ten second detour, but you must do it at the time).

Restore Service
1. Recycle the minimum number of Worker Processes in order to restore service.
o Don't bother stopping and starting Websites, you generally need the App Pool to be
refreshed in order to get the site working again, and that's what a Recycle does.
o

Recycling the App Pool is 9/10 times enough.

Note that recycling appears to happen on the next request to come in (even though
the existing WP has been told to go away), so a worker process may not immediately
reappear. That doesn't mean it hasn't worked, just that no requests are waiting.

IISReset is usually a tool used by people that don't know better. Don't use it unless
you need every website to terminate and restart all at once. (It's like trying to
hammer a nail into a wall with a brick. It might work, but you look like an idiot, and
there's going to be collateral damage)

Determine Cause i.e. look at and think about the data you've collected.
1. Take the logs and the memory dump, look for commonalities, engage the app developers,
debug the dump with DebugDiag 1.2, and so on.
Set up for next time
1. Don't assume it's the last occurrence - develop a plan for what you'll need to collect next
time, based on this time.
o For example, if the requests are all for the same URL, implement some additional
instrumentation or logging, or a Failed Request Tracing rule that'll help identify the
spot on the page that experiences a problem.
o

Performance monitor logs are helpful (if in doubt, get a perfmon log too).

General causes of kernel panics

Kernel panics are often caused by one or more of the following issues.

Defective or incompatible RAM often causes of kernel panics. Despite being a highlyreliable product, RAM can fail. Modern operating systems, like Mac OS X, are sensitive to
RAM. Purchase additional RAM from either Apple or third parties who guarantee their RAM
is compatible with Mac OS X, offer a liberal exchange policy, and provide a lifetime
warranty should the RAM become defective or a later version of Mac OS X introduce
incompatibilities.
Incompatible, obsolete, or corrupted kernel extensions. If a third-party kernel extension or
one of its dependencies is incompatible or obsolete with respect to the version of Mac OS X
you are using, kernel panics may occur when the kernel executes such extensions. Likewise,
if a kernel extension or one of its dependencies is corrupted, such as the result of hard disk
corruption, kernel panics are likely to occur when the kernel attempts to load or execute
such.

Incompatible, obsolete, or corrupted drivers. Similar to kernel extensions, drivers for thirdparty hardware which are incompatible with the version of Mac OS X you are using, or
which have become corrupted, will cause in kernel panics.

Hard disk corruption, including bad sectors, directory corruption, and other hard-disk ills.

Incorrect permissions on System-related files or folders.

Insufficient RAM and available hard disk space.

Improperly installed hardware or software.

Defective hardware or software. Hardware failures, including a defective CPU, or

programming errors can result in kernel panics.

Incompatible hardware. While rare, this is generally the result of a third-party hardware
vendors product failing to properly respond to the kernel or a kernel extension in an
expected way.

Redhat Enterprise Linux Troubleshooting Kernel

Panic issues Part 2
September 30, 2013
By Ramdev

In this post we will be Discussing about : [hide]

How do we analyse System Crash and Kernel Panic Issues

What is Kdump?

How do we configure the kdump ?

How do we analyse the VMCORE file collected using kdump?

How to Use Crash Utility ?

Sample Scenario 1 :
o

Observations:

Sample Scenario 2

Sample Scenario 3

Sample Scenario 4

This is continuation post to our earlier kernel panic reference post ( Redhat Enterprise
Linux 6 Kernel Panic and System Crash Troubleshooting Quick Reference ) where we
have discussed several types of kernel panic issues and their reasons. And in this post I will
be talking about the procedures and some guidelines to diagnosis and troubleshoot the
common kernel panic issues in redhat linux.
Btw, please note that these are just guidelines and purely your knowledge purpose, but
they doesnt guarantee any solutions to your environment specific issues. You need to take
extreme care and precaution while troubleshooting issues which are very specific to your

hardware and software environment.

How do we analyse System Crash and Kernel Panic Issues

Normally we expect we have already configured Kdump to gather necessary vmcore to
analyse these kind of issues. If we have vmcore ready we could use the crash utility to do
some of analyse of some of the stuff as given below:

What is Kdump?
Starting in Red Hat Enterprise Linux 5, kernel crash dumps are captured using the kdump
mechanism. Kexec is used to start another complete copy of the Linux kernel in a reserved
area of memory. This secondary kernel takes over and copies the memory pages to the
crash dump location.

How do we configure the kdump ?

In

brief

it

has

following

steps:

Step 1. Install kexec-tools

Step 2. Edit /etc/grub.conF, and add the crashkernel=<reservered-memory-setting> at
the end of kernel line
Example for RHEL 5:
title Red Hat Enterprise Linux Client (2.6.17-1.2519.4.21.el5)
root (hd0,0)
kernel /boot/vmlinuz-2.6.17-1.2519.4.21.el5 ro root=LABEL=/ rhgb
quiet crashkernel=128M@16M
initrd /boot/initrd-2.6.17-1.2519.4.21.el5.img
crashkernel=memory@offset
++
| RAM | crashkernel | crashkernel |
| size | memory | offset |
|+-+-|

| 0 2G | 128M | 16 |
| 2G 6G | 256M | 24 |
| 6G 8G | 512M | 16 |
| 8G 24G | 768M | 32 |
++
Example for RHEL 6:
title Red Hat Enterprise Linux Server (2.6.32-71.7.1.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-71.7.1.el6.x86_64 ro
root=/dev/mapper/vg_example-lv_root rd_LVM_LV=vg_example/lv_root
rd_LVM_LV=vg_example/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM
LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc
KEYTABLE=us crashkernel=128M rhgb quiet
initrd /initramfs-2.6.32-71.7.1.el6.x86_64.img
Guidelines for Crash Kernel Reserved Memory Settings:
crashkernel=0M to 2G: 128M, 2G-6G:256M, 6G-8G:512M,8G-:768M
Ram Size CrashKernel
>0GB 128MB
>2GB 256MB
>6GB 512MB
>8GB 768MB
Step 3. configure /etc/kdump.conf
3A) to specify the destination to send the output of kexec,i.e. vmcore. Following destinations can be
used
raw device : raw /dev/sda4
file : ext3 /dev/sda3 , that will dump vmcore to /dev/sda3:/var/crash
NFS share : net nfs.example.com:/export/vmcores
Another system via SSH : net [email protected]
3B) Configure Core Collector, to discard unnecessary memory pages and compress the only needed
ones
Option
1
2
4
8

Discard
Zero pages
Cache pages
Cache private
User pages

16

Free pages

To discard all optional pages:

core_collector makedumpfile -d 31 -c 4. reboot the server with crashkernel=<reserved-memorysetting>5. Start kdump service6. Enable the kdump for autostart on
Important Notes about Redhat Linux on HP ASR ( AUTOMATED SERVER RECOVERY )
FEATURE
According to the Automatic System Recovery definition on HP website, If the internal health LED
on a server is amber, the server might shutdown or reboot as soon as the hp Insight Manager agents
are installed. This feature is called Automatic Server Recovery, and is enabled to ensure that the
server can be recovered even though there is a major hardware problem.
The Automatic System Recovery generally occurs during low server utilization for an extended
period of time.
The Automatic System Recovery feature is a 10 minute timer. If the OS stops communicating the
server reboots. It is implemented using a heartbeat timer that continually counts down. The hpasm
driver frequently reloads the counter to prevent it from counting down to zero. If the Automatic
System Recovery timer counts down to 0, it is assumed that the operating system is locked up and
the system automatically attempts to reboot.
The logs of system lockup, the Automatic System Recovery and hardware failure are logged in
IML . You can check the IML through the ILO interface.
To check whether ASR is enabled in the BIOS or not, we can use the below command.
# hpasmcli -s show asr
We have to make sure, that ASR timeout value should not interrupt the complete core collection. So
it is often required to
Disable ASR using the command :
# hpasmcli -s DISABLE ASR
Or set longer timeout using the command:
# hpasmcli -s SET ASR 30

How do we analyse the VMCORE file collected using kdump?

If we assume that we configured that /var/crash as destination for vmcore in /etc/kdump.conf, we

will see the file as below

# ls -l /var/crash/127.0.0.1-2013-09-21-19:45:17/vmcore
-rw-. 1 root root 490958682 Sep 21 18:46 /var/crash/127.0.0.12013-09-21-19:45:17/vmcore
To Analyse the vmcore we need the crash utility. And we need to install following packages to get
started with vmcore analysis
# yum install crash
# yum install kernel-debuginfo-2.6.32-220.23.1.el6.x86_64
Purpose of Debuginfo package : Debugging symbols are stripped out of the standard kernel for
performance and size reasons. To analyze the vmcore, separate debugging information needs to be
provided. This is specific to the exact revision of the kernel which crashed. Debuginfo package will
help us to analyse these symbols from the vmcore.

How to Use Crash Utility ?

At this level We will only discuss about basic usage of crash utility from the linux administration
point of view.

log Display the kernel ring buffer log. On a running system, dmesg also
displays the kernel ring buffer log.

Often times this can capture log messages that were not written to disk due to the crash.
crash>
log

snip

SysRq
:
Trigger
a
crash
BUG: unable to handle kernel NULL pointer dereference at (null)
IP:
[<ffffffff8130e126>]
sysrq_handle_crash+0x16/0x20
PGD
7a602067
PUD
376ff067
PMD
0
Oops:
0002
[#1]
SMP

kmem -i Show available memory at time of crash

ps Show running processes at time of crash. Useful with grep

net Show configured network interfaces at time of crash

bt Backtraces are read upside-down, from bottom to top

crash>

bt

PID:
6875
TASK:
ffff88007a3aaa70
CPU:
0
COMMAND:
bash
#0
[ffff88005f0f5de8]
sysrq_handle_crash
at
ffffffff8130e126
#1
[ffff88005f0f5e20]
__handle_sysrq
at
ffffffff8130e3e2
#2
[ffff88005f0f5e70]
write_sysrq_trigger
at
ffffffff8130e49e
#3
[ffff88005f0f5ea0]
proc_reg_write
at
ffffffff811cfdce
#4
[ffff88005f0f5ef0]
vfs_write
at
ffffffff8116d2e8
#5
[ffff88005f0f5f30]
sys_write
at
ffffffff8116dd21
#6 [ffff88005f0f5f80] system_call_fastpath at ffffffff81013172
RIP: 00000037702d4230 RSP: 00007fff85b95f40 RFLAGS: 00010206
RAX: 0000000000000001 RBX: ffffffff81013172 RCX: 0000000001066300
RDX: 0000000000000002 RSI: 00007f04ae8d2000 RDI: 0000000000000001
RBP: 00007f04ae8d2000 R8: 000000000000000a R9: 00007f04ae8c4700
R10: 00000000ffffffff R11: 0000000000000246 R12: 0000000000000002
R13: 0000003770579780 R14: 0000000000000002 R15: 0000003770579780
ORIG_RAX:
0000000000000001
CS:
0033
SS:
002b

sys Displays system data same information displayed when crash starts

crash>
sys
DUMPFILE:
/tmp/vmcore
[PARTIAL
DUMP]
CPUS:
2
DATE:
Thu
May
5
14:32:50
2011
UPTIME:
00:01:15
LOAD
AVERAGE:
1.19,
0.34,
0.12
TASKS:
252
NODENAME:
rhel6-desktop
RELEASE:
2.6.32-220.23.1.el6.x86_64
VERSION:
#1
SMP
Mon
Oct
29
19:45:17
EDT
2012
MACHINE:
x86_64
(3214
Mhz)
MEMORY:
2
GB
PANIC:
Oops:
0002
[#1]
SMP

(check
log
for
details)
PID:
6875
COMMAND:
bash
TASK:
ffff88007a3aaa70
[THREAD_INFO:
ffff88005f0f4000]
CPU:
0
STATE: TASK_RUNNING (PANIC)

dmesg To check the kernel log from vmcore output

crash>

snip
CPU
0:
Machine
Check
Exception:
Kernel
panic

not
syncing:
Unable
Redirect Crash output to Regular Commands

dmesg

0000000000000004
to
continue

Example
Example

1:
2:

crash>

ps

crash>
|
fgrep

log
bash

>
|

log.txt
wc
-l

Sample Scenario 1 :
System hangs or kernel panics with MCE (Machine Check Exception) in /var/log/messages file.
System was not responding. Checked the messages in netdump server. Found the following
messages
Kernel
panic

not
syncing:
Machine
check.
System
crashes
under
load.
System
crashed
and
rebooted.
Machine Check Exception panic

Observations:
Look for the phrase Machine Check Exception in the log just before the panic message. If this
message occurs, the rest of the panic message is of no interest.
Analyze vmcore
$crash /path/to/2.6.18-128.1.6.el5/vmlinux vmcore
KERNEL: ./usr/lib/debug/lib/modules/2.6.18-128.1.6.el5/vmlinux
DUMPFILE: 563523_vmcore [PARTIAL DUMP]
CPUS: 4
DATE: Thu Feb 21 00:32:46 2011
UPTIME: 14 days, 17:46:38
LOAD AVERAGE: 1.14, 1.20, 1.18
TASKS: 220
NODENAME: gurkulnode1
RELEASE: 2.6.18-128.1.6.el5
VERSION: #1 SMP Tue Mar 24 12:05:57 EDT 2009
MACHINE: x86_64 (2599 Mhz)
MEMORY: 7.7 GB
PANIC: Kernel panic not syncing: Uncorrected machine check
PID: 0
COMMAND: swapper
TASK: ffffffff802eeae0 (1 of 4) [THREAD_INFO: ffffffff803dc000]
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash> log

CPU 0: Machine Check Exception:7 Bank 4: b40000000005001b

RIP 10:<ffffffff8006b2b0> {default_idle+0x29/0x50}

TSC bc34c6f78de8f ADDR 17fe30000
This is not a software problem!
Run through mcelog ascii to decode and contact your hardware
vendor
Kernel panic not syncing: Uncorrected machine check
Process error through mcelog ascii, specify k8 for events that are for AMD processors and p4 for
a Pentium 4 or Xeon. This resulting information might be helpful to your hardware vendor.
$ cat > mcelog.txt
CPU 0: Machine Check Exception:7 Bank 4: b40000000005001b
RIP 10:<ffffffff8006b2b0> {default_idle+0x29/0x50}
TSC bc34c6f78de8f ADDR 17fe30000
[ctrl]+[d]
$ mcelog ascii k8 < mcelog.txt
HARDWARE ERROR. This is *NOT* a software problem!
Please contact your hardware vendor
CPU 0 4 northbridge TSC bc34c6f78de8f
RIP 10:ffffffff8006b2b0
Northbridge GART error
bit61 = error uncorrected
TLB error generic transaction, level generic
STATUS b40000000005001b MCGSTATUS 7
RIP: default_idle+0x29/0x50}
Observations and Recommended Solution : The information printed by the kernel (the line
printed immediately before the panic message) comes from the hardware and should be provided to
the hardware support person for analysis. This information should resemble the following:
CPU 10: Machine Check Exception: 4 Bank 0: b200000410000800
Machine Check Exception (MCE) is an error that occurs when a computers CPU detects a hardware
problem. Typically, the impending hardware failure will cause the kernel to panic in order to protect
against data corruption.
Normally at this level we do engage the hardware vendor for further troubleshooting and diagnosis;
the message present in the logs just before the kernel panic should be given to hardware support.
How to run a memory test ?
Red Hat Enterprise Linux ships a memory test tool called memtest86+. It is a bootable utility that
tests physical memory by writing various patterns to it and reading them back. Since memtest86+
runs directly off the hardware it does not require any operating system support for execution.

This tool is available as an RPM package from Red Hat Network (RHN) as well as a boot option
from the Red Hat Enterprise Linux rescue disk.
To boot memtest86+ from the rescue disk, you will need to boot your system from CD 1 of the Red
Hat Enterprise Linux installation media, and type the following at the boot prompt (before the Linux
kernel is started):
boot: memtest86
If you would rather install memtest86+ on the system, here is an example of how to do it on a Red
Hat Enterprise Linux 5 machine registered to RHN:
# yum install memtest86+
For the Red Hat Enterprise Linux version 4, perform the following command to install memtest86+.
Make sure current system has been registered to RHN:
# up2date -i memtest86+
Then you will have to configure it to run on next reboot:
# memtest-setup
After reboot, the GRUB menu will list memtest. Select this item and it will start testing the memory.
Please note that once memtest86+ is running it will never stop unless you interrupt it by pressing the
Esc key. It is usually a good idea to let it run for a few hours so it has time to test each block of
memory several times.
memtest86+ may not always find all memory problems. It is possible that the system memory can
have a fault that memtest86+ does not detect.

Sample Scenario 2
Console

Screen

having

the

messages

as

below

Northbridge
Error,
node
1,
core:
-1
K8
ECC
error.
EDAC
amd64
MC1:
CE
ERROR_ADDRESS=
0x101a793400
EDAC
MC1:
INTERNAL
ERROR:
row
out
of
range
(-22
>=
8)
EDAC
MC1:
CE

no
information
available:
INTERNAL
ERROR
EDAC MC1: CE no information available: amd64_edacError Overflow
Observations :

check for any other related Kernel Errors from /var/log/messages

kernel: [Hardware Error]: Northbridge Error (node 1): DRAM ECC error detected on the NB.
kernel: [Hardware Error]: cache level: L3/GEN, mem/io: MEM, mem-tx: RD, part-proc: RES (no
timeout)
kernel: [Hardware Error]: CPU:2 MC4_STATUS[Over|CE|MiscV|-|AddrV|CECC]:
0xdc2c410000000a13
kernel: [Hardware Error]: MC4_ADDR: 0x0000000210b67d90
When the EDAC core monitoring module and various supported chipset drivers are loaded,
whenever an error has been detected, an error message will get logged in the syslog message file.
Based on the Keywords used in the Error message we can decode the criticality of the error, below
are the guideline
Non-Fatal Error Recoverable error
DRAM Controller In memory controller module
MC0 Memory controller 0
CE Correctable Error Page in memory page
0xc6397 Offset offset into that page at 0x0
Grain accuracy of reporting
Syndrome error bits from controller (specific)
Channel which memory channel (often only channel 0 on machines)
Row which DIMM row. How that maps to a chip is vendor specific but often as simple as row
0/1 -> DIMM0 row 2/3 > DIMM1 etc
label description of this DIMM (NULL string in U3)
e752x chip type (eg e7501, AMD76x)
Memory error checking on a memory module used to be accomplished with a parity checking bit
that was attached to each byte of memory. The parity bit was calculated when each byte of memory
was written, and then verified when each byte of memory was read. If the stored parity bit didnt
match the calculated parity bit on a read, that byte of memory was known to have changed. Parity
checking is known to be a reasonably effective method for detecting a single bit change in a byte of
memory.
Recommended Solution
EDAC messages, errors, and warnings are often indicative of a hardware problem such as a
memory module failure or memory controller failure.
If EDAC errors are encountered running a hardware diagnostic and contacting your hardware
vendor are advised.
In some cases EDAC errors can be thrown due to a bug in the EDAC kernel module, the kernel,
or due to an incompatibility between the systems chipset and the

Sample Scenario 3

The following error message appearing in /var/log/messages

kernel: Dazed and confused, but trying to continue
kernel: Do you have a strange power saving mode enabled?
kernel: Uhhuh. NMI received for unknown reason 21 on CPU 0
kernel: Dazed and confused, but trying to continue
kernel: Do you have a strange power saving mode enabled?
kernel: Uhhuh. NMI received for unknown reason 31 on CPU 0.
Observations :
The above message is typically output when system hardware has generated a non-maskable
interrupt not recognized by the kernel. If there are any codes associated with the fault these may be
trapped by hpasm software or equivalent HW Vendor monitoring software and logged there, but the
fact that the NMI is not known to the kernel suggests the problem is a fundamental hardware issue.
NMI 21 and 31 events typically indicate faulty RAM or perhaps CPU. However, hardware issues
relating to the motherboard cannot be ruled out.
Schedule some downtime in order to run hardware diagnostics. A standard memtest86 could be run,
but this is not guaranteed to find all possible memory issues. memtest86 can be initiated on a system
by booting from a RHEL 5 DVD. Following this, any available manufacturer specific hardware
diagnostics tools should be run.
Recommeded Solution:
If the issue happens frequenly on a production machine and that causes a system crash or panic, then
first to with a BIOS firmware upgrade for the hardware. Should that not resolve the issue, have the
support vendor replace faulty hardware, such as memory, CPU or motherboard.

Sample Scenario 4
Console Shows following Error Message
NMI: IOCK error (debug interrupt?)
CPU 0
Modules linked in: ipt_MASQUERADE iptable_nat ip_nat xt_state
ip_conntrack nfnetlink ipt_REJECT xt_tcpudp iptable_filter
ip_tables x_tables bridge mptctl mptbase bonding be2iscsi ib_iser
rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i
cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i cxgb3 8021q libiscsi_tcp
libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin
dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec
i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac
parport_pc lp parport joydev sr_mod cdrom hpilo bnx2 serio_raw
shpchp pcspkr sg dm_raid45 dm_message dm_region_hash dm_mem_cache

dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage qla2xxx

scsi_transport_fc ata_piix libata cciss sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd
Pid: 0, comm: swapper Not tainted 2.6.18-194.17.4.el5 #1
RIP: 0010:[<ffffffff8019d550>] [<ffffffff8019d550>]
acpi_processor_idle_simple+0x14c/0x30e
RSP: 0018:ffffffff803fbf58 EFLAGS: 00000046
RAX: 0000000000d4d87e RBX: ffff81061e10a160 RCX: 0000000000000908
RDX: 0000000000000915 RSI: 0000000000000003 RDI: 0000000000000000
RBP: 0000000000d4d87e R08: ffffffff803fa000 R09: 0000000000000039
R10: ffff810001005710 R11: 0000000000000000 R12: 0000000000000000
R13: ffff81061e10a000 R14: 0000000000000000 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffffffff803ca000(0000)
knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000009013954 CR3: 000000060799d000 CR4: 00000000000006e0
Process swapper (pid: 0, threadinfo ffffffff803fa000, task
ffffffff80308b60)
Stack: ffff81061e10a000 ffffffff8019d404 0000000000000000
ffffffff8019d404
0000000000090000 0000000000000000 0000000000000000 ffffffff8004923a
0000000000200800 ffffffff80405807 0000000000090000 0000000000000000
Call Trace:
[<ffffffff8019d404>] acpi_processor_idle_simple+00/0x30e
[<ffffffff8019d404>] acpi_processor_idle_simple+00/0x30e
[<ffffffff8004923a>] cpu_idle+095/0xb8
[<ffffffff80405807>] start_kernel+0220/0225
[<ffffffff8040522f>] _sinittext+0x22f/0236
Code: 89 ca ed ed 41 89 c4 41 8a 45 1c 83 e0 30 3c 30 75 15 f0 ff
Observations
In these scenarios we normally check dmesg and lspci output to figure out who the culprit might be
Parity and uncorrectable hardware errors are examples of why an IOCHK error could be raised.
Most hardware errors should however be reported through the MCE (Machine Check Exception)
mechanism. An MCE indicates that the CPU detected an internal machine error or a bus error, or
that an external agent detected a bus error. Normally the hardware manufacturer will be able to
provide further details.
Recommended Solution
Use vendor hardware diagnostics software to analyse system health.
Contact the hardware manufacturer for further assistance.

Under RHEL6, the kernel.panic_on_io_nmi = 1 sysctl can be set to have the system panic when an
I/O NMI is received.

Linux Admin Troubleshooting Reference Kernel

Panic and System Crash Redhat Enterprise Linux
(RHEL6)
September 23, 2013
By Ramdev

In this post we will be Discussing about : [hide]

What is the meaning of a Linux System Crash?

Hardware: Machine Check Exceptions

Error Detection and Correction (EDAC)

Non-Maskable Interrupts (NMIs)

Software: The BUG() macro

Software: Bad pointer handling

Software: Pseudo-hangs

Software: Out-of-Memory killer

What is the meaning of a Linux System

Crash?
Crash is a generic term used usually to say that the system has come to halt and no progress
is observed. The system seems unresponsive or has already rebooted.

Kernel Panic A voluntary halt to all system activity when an abnormal situation is
detected by the kernel. A Kernel panic is an action taken by an operating system upon
detecting an Internal fatal error from which it cannot safely recover. And in Linux these
Kernel Panics can be caused by different reasons

o
o

Hardware: Machine Check Exceptions

Error Detection and Correction (EDAC)

Non-Maskable Interrupts (NMIs)

Hardware NMI Button

NMI Watch Dog

unknown_nmi_panic

panic_on_unrecovered_nmi

panic_on_io_nmi

Software related BUG() macro

Software related Bad pointer handling

Software related Pseudo-hangs

Software related Out-of-Memory killer

Hardware: Machine Check Exceptions

Hardware Machine Check Exceptions normally caused by the the Component failures detected and
reported by the hardware via an exception, and they typically looks like:
kernel:
Bank 0:
kernel:
kernel:

CPU 0: Machine Check Exception: 4

b278c00000000175
TSC 4d9eab664a9a60
Kernel panic not syncing: Machine check

Sample Scenario 1 :
System hangs or kernel panics with MCE (Machine Check Exception) in /var/log/messages file.

System was not responding. Checked the messages in netdump server.

Found the following messages Kernel panic not syncing: Machine
check.
System crashes under load.
System crashed and rebooted.
Machine Check Exception panic
Troubleshooting Procedure Posted here Redhat Enterprise Linux Troubleshooting Kernel
Panic issues Part 2

Error Detection and Correction (EDAC)

Normally, EDAC errors caused by Hardware mechanism to detect and report memory chip and PCI
transfer errors, and reported in /sys/devices/system/edac/{mc/,pci} and logged by the kernel as:
EDAC MC0: CE page 0x283, offset 0xce0, grain 8,
syndrome 0x6ec3, row 0, channel 1 DIMM_B1:
amd76x_edac
All the Informational EDAC messages (such as a corrected ECC error) are printed to the system
log, where as critical EDAC messages (such as exceeding a hardware-defined temperature
threshold) trigger a kernel panic.

Sample Scenario 2 :
Console Screen having the messages as below
Northbridge Error, node 1, core: -1
K8 ECC error.
EDAC amd64 MC1: CE ERROR_ADDRESS= 0x101a793400
EDAC MC1: INTERNAL ERROR: row out of range (-22 >= 8)
EDAC MC1: CE no information available: INTERNAL ERROR
EDAC MC1: CE no information available: amd64_edacError Overflow
Troubleshooting Procedure Posted here Redhat Enterprise Linux Troubleshooting Kernel
Panic issues Part 2

Non-Maskable Interrupts (NMIs)

A Non maskable interrupt (NMI) is an interrupt that is unable to be ignored/masked out by standard
operating system mechanisms. A non-maskable interrupt (NMI) cannot be ignored, and is generally
used only for critical hardware errors however recent changes in behavior has added additional
functionality of:
1) NMI button.
The NMI This can be used to signal the operating system when other standard input mechanisms
(keyboard, ssh, network) have ceased to function.
It can be used to create an intentional panic for additional debugging. It may not always be a
physical button.
It may be presented through an iLO or Drac Interface.
Unknown NMIs The kernel has mechanisms to handle certain known NMIs appropriately,
unknown ones typically result in kernel log warnings such as:
Uhhuh. NMI received.
Dazed and confused, but trying to continue
You probably have a hardware problem with your RAM chips
Uhhuh. NMI received for unknown reason 32.
Dazed and confused, but trying to continue.
Do you have a strange power saving mode enabled?
These unknown NMI messages can be produced by ECC and other hardware problems. The kernel
can be configured to panic when these are received
though this sysctl:
kernel.unknown_nmi_panic=1
This is generally only enabled for troubleshooting

Sample Scenario 3:

The following error message appearing in /var/log/messages

kernel:
kernel:
kernel:
kernel:
kernel:
kernel:

Dazed and confused, but trying to continue

Do you have a strange power saving mode enabled?
Uhhuh. NMI received for unknown reason 21 on CPU 0
Dazed and confused, but trying to continue
Do you have a strange power saving mode enabled?
Uhhuh. NMI received for unknown reason 31 on CPU 0.

Troubleshooting Procedure Posted here Redhat Enterprise Linux Troubleshooting Kernel

Panic issues Part 2

2) A Watchdog-like software on the system that monitors for perceived system hangs
The NMI watchdog monitors system interrupts and sends an NMI if the system appears to have
hung.
On a normal system hundreds of device and timer interrupts are received per second. If there are no
interrupts in a 30 second interval*,
the NMI watchdog assumes that the system has hung and sends an NMI to the system to trigger a
kernel panic or restart.

How an NMI watchdog works

A standard system level watchdog waits for regular events to fire and reboots the machine if no
event is received within a designated timeframe. The NMI watchdog is no different. When using the
NMI watchdog the system generates periodic NMI interrupts, and the kernel can monitor whether
any CPU has locked up and print out debugging messages if so.
Enabling NMI Watchdog
The Red Hat Enterprise Linux 6 kernel is built with NMI watchdog support on currently supported
x86 and x86-64 platforms.
Ensure NMI is being used:
For SMP machines and Single processor systems with an IO-APIC use nmi_watchdog=1.
For Single processor systems without an IO-APIC use

nmi_watchdog=2.

Verification to check NMI watchdog working

Boot the system with the the parameter as stated above and check the /proc/interrupts file for the
NMI count line. This value should be non zero and increase over time. If the value is zero and
does not increase over time the wrong NMI watchdog parameter has been used, change
If it is still zero then log a problem, you probably have a processor that needs to be added to the nmi
code.
Here is an example from /etc/grub.conf for systems which utilize the GRUB boot loader:
title Red Hat Enterprise Linux Server (2.6.32-358.6.1.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-358.6.1.el6.x86_64 ro root=/dev/mapper/vg_worklaptop-lv_root
crashkernel=auto rd_LVM_LV=vg_worklaptop/lv_root rhgb quiet nmi_watchdog=1
initrd /initramfs-2.6.32-358.6.1.el6.x86_64.img
To determine if the NMI watchdog was properly activated, check the /proc/interrupts file. The NMI
interrupt should display a non-zero value. If the NMI interrupt displays a zero, alter the
nmi_watchdog value, restart the system, and examine this file again. If a zero is still displayed, then
the processor in the test system is not supported by the NMI watchdog code.
The output, when functioning correctly, should look similar to the following:
[root@work-laptop wmealing]# cat /proc/interrupts | grep ^NMI
NMI: 861 636 377 357 Non-maskable interrupts
Each processor core has an NMI count. These should all be increasing over time. The above
example is a quad core system.
System wide NMI settings
The NMI settings can be configured at runtime by using the sysctl interface.
In the /etc/sysctl.conf, to enable, set:
kernel.nmi_watchdog = 1
To disable, set:
kernel.nmi_watchdog = 0
Note that this does not enable the functionality, the kernel parameter is required to correctly enable
the NMI watchdog.

unknown_nmi_panic
A feature was introduced in kernel 2.6.9 which helps to make easier the process of diagnosing
system hangs on specific hardware.
The feature utilizes the kernels behavior when dealing with unknown NMI sources. The behavior is
to allow it to panic, rather than handle the unknown nmi source. This feature cannot be utilized on
systems that also use the NMI Watchdog or some oprofile (and other tools that use performance
metric features as both of these also make use of the undefined NMI interrupt. If
unknown_nmi_panic is activated with one of these features present, it will not work.
Note that this is a user-initiated interrupt which is really most useful for helping to diagnose a
system that is experiencing system hangs for unknown reasons.
To enable this feature, set the following system control parameter in the /etc/sysctl.conf file as
follows:
kernel.unknown_nmi_panic = 1
To disable, set:
kernel.unknown_nmi_panic = 0
Once this change has taken effect, a panic can be forced by pushing the systems NMI switch.
Systems that do not have an NMI switch can still use the NMI Watchdog feature which will
automatically generate an NMI if a system hang is detected.
panic_on_unrecovered_nmi
Some systems may generate an NMI based on vendor configuration, such as power management,
low battery etc. It may be important to set this if your system is generating NMIs in a knownworking environment.
To enable this feature, set the following system control parameter in the /etc/sysctl.conf file as
follows:
kernel.panic_on_unrecovered_nmi = 1
To disable, set:
kernel.panic_on_unrecovered_nmi = 0
panic_on_io_nmi
This setting was only available in Red Hat Enterprise Linux 6. When set, this will cause a kernel

panic when the kernel receives an NMI caused by an Input/Output error.

Sample Scenario 4 :
Console Shows following Error Message
NMI: IOCK error (debug interrupt?)
CPU 0
Modules linked in: ipt_MASQUERADE iptable_nat ip_nat xt_state
ip_conntrack nfnetlink ipt_REJECT xt_tcpudp iptable_filter
ip_tables x_tables bridge mptctl mptbase bonding be2iscsi ib_iser
rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i
cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i cxgb3 8021q libiscsi_tcp
libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin
dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec
i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac
parport_pc lp parport joydev sr_mod cdrom hpilo bnx2 serio_raw
shpchp pcspkr sg dm_raid45 dm_message dm_region_hash dm_mem_cache
dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage qla2xxx
scsi_transport_fc ata_piix libata cciss sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd
Pid: 0, comm: swapper Not tainted 2.6.18-194.17.4.el5 #1
RIP: 0010:[<ffffffff8019d550>] [<ffffffff8019d550>]
acpi_processor_idle_simple+0x14c/0x30e
RSP: 0018:ffffffff803fbf58 EFLAGS: 00000046
RAX: 0000000000d4d87e RBX: ffff81061e10a160 RCX: 0000000000000908
RDX: 0000000000000915 RSI: 0000000000000003 RDI: 0000000000000000
RBP: 0000000000d4d87e R08: ffffffff803fa000 R09: 0000000000000039
R10: ffff810001005710 R11: 0000000000000000 R12: 0000000000000000
R13: ffff81061e10a000 R14: 0000000000000000 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffffffff803ca000(0000)
knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000009013954 CR3: 000000060799d000 CR4: 00000000000006e0
Process swapper (pid: 0, threadinfo ffffffff803fa000, task
ffffffff80308b60)
Stack: ffff81061e10a000 ffffffff8019d404 0000000000000000
ffffffff8019d404
0000000000090000 0000000000000000 0000000000000000 ffffffff8004923a
0000000000200800 ffffffff80405807 0000000000090000 0000000000000000
Call Trace:
[<ffffffff8019d404>] acpi_processor_idle_simple+0x0/0x30e
[<ffffffff8019d404>] acpi_processor_idle_simple+0x0/0x30e
[<ffffffff8004923a>] cpu_idle+0x95/0xb8
[<ffffffff80405807>] start_kernel+0x220/0x225
[<ffffffff8040522f>] _sinittext+0x22f/0x236

Code: 89 ca ed ed 41 89 c4 41 8a 45 1c 83 e0 30 3c 30 75 15 f0 ff

Troubleshooting Procedure Posted here Redhat Enterprise Linux Troubleshooting Kernel

Panic issues Part 2

Software: The BUG() macro

This kind of kernel panic normally caused by the kernel code when an abnormal situation is seen ,
that indicates a programming error . And normally the Output looks like:
Kernel BUG at spinlock:118
invalid operand: 0000 [1] SMP
CPU 0

Sample Scenario 5:
NFS client kernel crash because async task already queued
hitting BUG_ON(RPC_IS_QUEUED(task)); in __rpc_execute
kernel BUG at net/sunrpc/sched.c:616!
invalid opcode: 0000 [#1] SMP
last sysfs file:
/sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
CPU 8
Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss
pcc_cpufreq sunrpc power_meter hpilo
hpwdt igb mlx4_ib(U) mlx4_en(U) raid0 mlx4_core(U) sg microcode
serio_raw iTCO_wdt
iTCO_vendor_support ioatdma dca shpchp ext4 mbcache jbd2 raid1
sd_mod crc_t10dif mpt2sas
scsi_transport_sas raid_class ahci dm_mirror dm_region_hash dm_log
dm_mod
[last unloaded: scsi_wait_scan]
Pid: 2256, comm: rpciod/8 Not tainted 2.6.32-220.el6.x86_64 #1 HP
ProLiant SL250s Gen8/
RIP: 0010:[<ffffffffa01fe458>] [<ffffffffa01fe458>]
__rpc_execute+0x278/0x2a0 [sunrpc]


Process rpciod/8 (pid: 2256, threadinfo ffff882016152000, task
ffff8820162e80c0)

Call Trace:
[<ffffffffa01fe4d0>] ? rpc_async_schedule+0x0/0x20 [sunrpc]
[<ffffffffa01fe4e5>] rpc_async_schedule+0x15/0x20 [sunrpc]
[<ffffffff8108b2b0>] worker_thread+0x170/0x2a0
[<ffffffff81090bf0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8108b140>] ? worker_thread+0x0/0x2a0
[<ffffffff81090886>] kthread+0x96/0xa0
[<ffffffff8100c14a>] child_rip+0xa/0x20
Code: db df 2e e1 f6 05 e0 26 02 00 40 0f 84 48 fe ff ff 0f b7 b3
d4 00 00 00 48 c7
c7 94 39 21 a0 31 c0 e8 b9 df 2e e1 e9 2e fe ff ff <0f> 0b eb fe 0f
b7 b7 d4 00 00 00
31 c0 48 c7 c7 60 63 21 a0 e8
RIP [<ffffffffa01fe458>] __rpc_execute+0x278/0x2a0 [sunrpc]

Troubleshooting Procedure Posted here Redhat Enterprise Linux Troubleshooting Kernel

Panic issues Part 2

Software: Bad pointer handling

This kind of kernel panics typically indicates a programming error and normally appear as below:
NULL pointer dereference at 0x1122334455667788 ..
or
Unable to handle kernel paging request at virtual address 0x11223344
One of the most common reason for this kind of error is possible memory corruption

Sample Scenario 6 :

NFS client kernel panics when doing an ls in the directory of a snapshot that has
already been removed.
NFS client kernel panics under certain conditions when connected to NFS server

either NetApp or Solaris ZFS

Kernel crashes with message

BUG: unable to handle kernel NULL pointer dereference at

0000000000000018
IP: [<ffffffff81192957>] commit_tree+0x77/0x100
PGD 7ff2e69067 PUD 7feaf59067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file:
/sys/devices/pci0000:00/0000:00:03.0/0000:07:00.0/vendor
CPU 64
Modules linked in: nls_utf8 fuse mptctl mptbase autofs4 nfs lockd
fscache(T) nfs_acl auth_rpcgss bnx2fc cnic uio fcoe libfcoe libfc
scsi_transport_fc scsi_tgt 8021q garp stp llc smbus(U) ipmi_devintf
ipmi_si ipmi_msghandler sunrpc cpufreq_ondemand acpi_cpufreq
freq_table nf_conntrack_ftp ipt_REJECT ipt_LOG iptable_filter
ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4
ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
nf_conntrack ip6table_filter ip6_tables ipv6 vfat fat dm_mirror
dm_region_hash dm_log microcode sg i2c_i801 i2c_core iTCO_wdt
iTCO_vendor_support ioatdma i7core_edac edac_core ixgbe mdio igb
dca ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif ata_generic
pata_acpi ata_piix megaraid_sas dm_mod [last unloaded:
scsi_wait_scan]
Modules linked in: nls_utf8 fuse mptctl mptbase autofs4 nfs lockd
fscache(T) nfs_acl auth_rpcgss bnx2fc cnic uio fcoe libfcoe libfc
scsi_transport_fc scsi_tgt 8021q garp stp llc smbus(U) ipmi_devintf
ipmi_si ipmi_msghandler sunrpc cpufreq_ondemand acpi_cpufreq
freq_table nf_conntrack_ftp ipt_REJECT ipt_LOG iptable_filter
ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4
ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
nf_conntrack ip6table_filter ip6_tables ipv6 vfat fat dm_mirror
dm_region_hash dm_log microcode sg i2c_i801 i2c_core iTCO_wdt
iTCO_vendor_support ioatdma i7core_edac edac_core ixgbe mdio igb
dca ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif ata_generic
pata_acpi ata_piix megaraid_sas dm_mod [last unloaded:
scsi_wait_scan]
Pid: 79910, comm: ls Tainted: G - T 2.6.32-131.6.1.el6.x86_64
#1 PRIMERGY RX900 S1
RIP: 0010:[<ffffffff81192957>] [<ffffffff81192957>]
commit_tree+0x77/0x100
RSP: 0018:ffff885f1484dab8 EFLAGS: 00010246
RAX: ffff881f5f43d3e8 RBX: ffff885f1484dab8 RCX: ffff885f1484dab8
RDX: ffff881f5f43d3e8 RSI: ffff881f5f43d3e8 RDI: ffff885f1484dab8
RBP: ffff885f1484dae8 R08: ffff881f5f43d3e8 R09: 0000000000000000
R10: ffff882080440a40 R11: 0000000000000000 R12: 0000000000000000

R13: ffff881f5f43d380 R14: ffff881f5fcba2c0 R15: 0000000000000000

FS: 00007f9b188177a0(0000) GS:ffff88011c700000(0000)
knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000018 CR3: 0000007fecaf5000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ls (pid: 79910, threadinfo ffff885f1484c000, task
ffff881fc4164b00)
Stack:
ffff881f5f43d3e8 ffff881f5f43d3e8 ffff881f5f43d380 ffff885f1484db08
<0> ffff881f5fcba2c0 ffff885f1484ddd8 ffff885f1484db48
ffffffff81192c6f
<0> ffff881c94a4d200 000000001484dbf8 ffff885f1484db08
ffff885f1484db08
Call Trace:
[<ffffffff81192c6f>] attach_recursive_mnt+0x28f/0x2a0
[<ffffffff81192d80>] graft_tree+0x100/0x140
[<ffffffff814dc686>] ? down_write+0x16/0x40
[<ffffffff81192e5f>] do_add_mount+0x9f/0x160
[<ffffffffa045ce2f>] nfs_follow_mountpoint+0x1bf/0x570 [nfs]
[<ffffffff811810a0>] do_follow_link+0x120/0x440
[<ffffffffa03112e0>] ? put_rpccred+0x50/0x150 [sunrpc]
[<ffffffff81180eeb>] __link_path_walk+0x78b/0x820
[<ffffffff8118164a>] path_walk+0x6a/0xe0
[<ffffffff8118181b>] do_path_lookup+0x5b/0xa0
[<ffffffff811819a7>] user_path_at+0x57/0xa0
[<ffffffff81041594>] ? __do_page_fault+0x1e4/0x480
[<ffffffff810ce97d>] ? audit_filter_rules+0x2d/0xa10
[<ffffffff81177cac>] vfs_fstatat+0x3c/0x80
[<ffffffff81177d5e>] vfs_lstat+0x1e/0x20
[<ffffffff81177d84>] sys_newlstat+0x24/0x50
[<ffffffff810d1ad2>] ? audit_syscall_entry+0x272/0x2a0
[<ffffffff814e054e>] ? do_page_fault+0x3e/0xa0
[<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
Code: 83 e8 68 eb 12 0f 1f 80 00 00 00 00 4c 89 a0 c0 00 00 00 48
8d 42 98 48 8b 50 68 48 8d 48 68 48 39 cb 0f 18 0a 75 e5 48 8b 45
d0 <49> 8b 54 24 18 48 39 d8 74 15 48 8b 0a 48 8b 5d d8 48 89 50 08
RIP [<ffffffff81192957>] c
ommit_tree+0x77/0x100
RSP <ffff885f1484dab8>
CR2: 0000000000000018
Troubleshooting Procedure Posted here Redhat Enterprise Linux Troubleshooting Kernel
Panic issues Part 2

Software: Pseudo-hangs

This are the common situations, that we commonly encounter where the system appears to be hung,
but some progress is being made, there are several reasons for this kind of behaviour, and they are

Livelock
if running a realtime kernel, application load could be too high,
leading the system into a state where it becomes effectively unresponsive in a
live lock/ busy wait state. The system is not actually hung, but just moving so
slowly that it appears to be hung.
Thrashing continuous swapping with close to no useful processing done

Lower zone starvation on i386 the low memory has a special significance
and the system may hang even when theres plenty of free memory

Memory starvation in one node in a NUMA system

Normally, Hangs which are not detected by the hardware are trickier to debug:

Use [sysrq + t] to collect process stack traces when possible

Enable the NMI watchdog which should detect those situations

Run hardware diagnostics when its a hard hang: memtest86, HP diagnostics

Sample Scenario 7:
The system is frequently getting hung and following error messages are getting logged
in /var/log/messages file while performing IO operations on the /dev/cciss/xx devices:
INFO: task cmaperfd:5628 blocked for more than 120 seconds.
echo 0 > /proc/sys/kernel/hung_task_timeout_secs disables this
message.
cmaperfd D ffff810009025e20 0 5628 1 5655 5577 (NOTLB)
ffff81081bdc9d18 0000000000000082 0000000000000000 0000000000000000
0000000000000000 0000000000000007 ffff81082250f040 ffff81043e100040
0000d75ba65246a4 0000000001f4db40 ffff81082250f228 0000000828e5ac68
Call Trace:
[<ffffffff8803bccc>] :jbd2:start_this_handle+0x2ed/0x3b7
[<ffffffff800a3c28>] autoremove_wake_function+0x0/0x2e
[<ffffffff8002d0f4>] mntput_no_expire+0x19/0x89
[<ffffffff8803be39>] :jbd2:jbd2_journal_start+0xa3/0xda
[<ffffffff8805e7b0>] :ext4:ext4_dirty_inode+0x1a/0x46

[<ffffffff80013deb>]
[<ffffffff80041bf5>]
[<ffffffff8805e70c>]
[<ffffffff88055abc>]
[<ffffffff8002cf2b>]
[<ffffffff800e45fe>]

__mark_inode_dirty+0x29/0x16e
inode_setattr+0xfd/0x104
:ext4:ext4_setattr+0x2db/0x365
:ext4:ext4_file_open+0x0/0xf5
notify_change+0x145/0x2f5
sys_fchmod+0xb3/0xd7

Troubleshooting Procedure Posted here Redhat Enterprise Linux Troubleshooting Kernel

Panic issues Part 2

Software: Out-of-Memory killer

In certain memory starvation cases, the OOM killer is triggered to force the release of some
memory by killing a suitable process. In severe starvation cases, the OOM killer may have to
panic the system when no killable processes are found:
Kernel panic not syncing:
Out of memory and no killable processes
The kernel can also be configured to always panic during an OOM by setting the vm.panic_on_oom
= 1 sysctl.

Sample Scenario 8 :
When the system panics kdump starts, but kdump hangs and does not output a vmcore. I see
following error messages on the console:
Kernel panic - not syncing: Out of memory and no killable processes...