Scribd
Upload
62 views

Disk IO Errors: Troubleshooting On Linux Servers

Disk IO errors are a common cause of poor server performance in Linux servers. Symptoms of disk IO errors include increased website latency, high server load, chkservd notifications, and slow email and cPanel interfaces. Tools like top, vmstat, and iostat can help identify if disk IO is the source of performance issues. Potential fixes include enabling write caching, direct memory access, or upgrading server hardware like the hard drive, RAM, or RAID controller.

Uploaded by

iftikhar ahmed
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
62 views

Disk IO Errors: Troubleshooting On Linux Servers

Disk IO errors are a common cause of poor server performance in Linux servers. Symptoms of disk IO errors include increased website latency, high server load, chkservd notifications, and slow email and cPanel interfaces. Tools like top, vmstat, and iostat can help identify if disk IO is the source of performance issues. Potential fixes include enabling write caching, direct memory access, or upgrading server hardware like the hard drive, RAM, or RAID controller.

Uploaded by

iftikhar ahmed
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 7

Disk IO Errors: Troubleshooting

on Linux Servers
Disk IO errors  (input/output) issues are a common cause of poor performance
on web hosting servers. Hard drives have speed limits, and if software tries to
read or write too much data too quickly, applications and users are forced to
wait. To put it another way, storage devices can be a bottleneck that stops the
server from reaching its full performance potential. 
Disk IO is not the only cause of slow servers, so in this article, we’ll explain
how to use Linux IO stats to identify disk IO issues and how to diagnose and
fix servers with storage bottlenecks. 

What Are the Symptoms of Disk IO Errors?


Because disk IO is so important to server performance, it can manifest in
many different ways:

 Increased website latency: sites hosted on the server take longer to


load than expected. 
 High server load: excessive IO can cause other components, including
the CPU, to work harder. You can learn more about measuring server
load in Troubleshooting High Server Loads.
 You receive a chkservd notification: cPanel & WHM’s built-
in chkservd monitoring tool will alert you if a service is unavailable. 
 Slow email delivery and webmail interfaces: Email software needs to
read data from and write data to the hard drives. 
 Slow cPanel & WHM interfaces: Many cPanel & WHM features rely on
fast hard drive or database access. 

If you observe any of these, high IO loads might be the culprit, but how do you
know it’s a storage bottleneck rather than a problem with the network or
processor?

Is Disk IO the Cause of Poor Performance?


From the user’s perspective, an IO bottleneck might look just like network
latency, among other problems. It’s prudent to make sure that the server’s
storage is the real culprit so we don’t waste time and money fixing the wrong
problem.
We’ll need to use diagnostic tools on the server’s command line, so log in with
SSH. 
First, let’s see if the CPU is waiting for disk operations to complete. Type “top”
and press enter. This launches the top tool, which shows server statistics and
a list of running processes. The wa metric shows IO-wait, the amount of time
the CPU spends waiting for IO completion represented as a percentage. 
IO-wait is one of a series of processor activity figures in the %CPU row. It also
includes:

 us: time spent on user processes.


 sy: time spent on system processes. 
 id: idle time.

On the single-CPU server in the example images, these are straightforward to


understand. Our server’s CPU spends 59 percent of its time waiting for IO
input instead of processing data.  An IO-wait above 1 may indicate the
server’s hard drives are struggling to supply the processor with data. 
On multi-core and multi-processor servers, it’s a little more complex.
Because top adds the CPU utilization figures for all cores, they can exceed
100 percent. As a rule, if the IO-wait percentage is bigger than 1 when divided
by the number of CPU cores, then the processor must wait before it can
process data. For example, on a 4-core system with a wa of 10 percent, the
IO-wait is around 2.5, so the processors are forced to wait. 
IO-wait times don’t always mean there is an IO bottleneck, but it is a valuable
clue, especially when it correlates with observed performance issues. To
discover the cause, we need to investigate further with vmstat, which shows
statistics for IO, CPU, and memory activity, among others. 

vmstat 1 10

We’re asking vmstat to show us ten readings at one-second intervals. The


first line shows average IO stats since the last reboot, and the subsequent
lines show real-time statistics.
We are interested in the io column, which is divided into input and output. It
shows that large amounts of data were written to a storage device throughout
the test period. Compared to the average loads in the first row, the IO system
is being seriously stressed. 
Next, we want to know which hard drive is under load. To find out, we can
use iostat. 

iostat -md

The -m option tells iostat to display statistics in megabytes per second, and -d
says we’re interested in device utilization. 

The device called vda is writing 730 MB of data each second. Whether that’s
a problem depends on the capabilities of the server and the device, but with
the observed performance degradation and large IO wait times, it’s
reasonable to conclude that excessive disk IO on vda is the cause of our
issues. 
There is one other piece of information that could help us narrow things down:
the mount point of the vda device. The mount point is the directory on the
server’s filesystem the device is connected to. You can find it with
the lsblk command.

We can see that vda has one partition called vda1 and that it’s mounted on


the root of the filesystem (/).  On this server, that information is not particularly
helpful; it only has one mounted device. However, on a server with several
storage devices, lsblk can help you to figure out where the data is being
written and which application is writing it. 

How To Fix Disk IO Issues


Once you have identified the affected drive, there are several approaches you
can take to mitigate disk IO issues.
For example, you may want to try changing a few settings to see if
performance improves before you upgrade hard drives or memory. Three hard
drive configuration settings you should try changing first are:
1. Turn on write caching
2. Turn on direct memory access
3. Upgrade Server Hardware
Here’s how to do that:

Turn on Write Caching


Write caching collects data for multiple writes in a RAM cache before writing
them permanently to the drive. Because it reduces the number of hard drive
writes, it can improve performance in some scenarios. 
Write caching can cause data loss if the server’s power is cut before the
cache is written to the disk. Don’t activate write caching if you want to
minimize the risk of data loss. 
The hdparm utility can turn write caching on and off. It may not be installed by
default on your server, but you can install it from CentOS’s core repository
with:

yum install hdparm

The following command turns write caching on:

hdparm -W1 /dev/sda

To turn write caching off, use:

hdparm -W0 /dev/sda

Turn on Direct Memory Access


Direct Memory Access (DMA) allows the server’s components to access its
RAM directly, without going via the CPU.  It can significantly increase hard
drive performance in some scenarios. 
To enable DMA, run the following command, replacing /dev/hda with your
hard drive:

hdparm -d1 /dev/hda

You can turn DMA off with:

hdparm -d0 /dev/hda

DMA isn’t available on all servers and, with virtual servers in particular, you
may not be able to modify hard drive settings.
Upgrade Server Hardware
If configuration tweaks don’t solve your IO issues, it’s time to think about
upgrading, replacing, or reorganizing the server’s hardware. 

 Upgrade the hard drive: If the drive is slow, consider upgrading to a


faster spinning disk drive or a solid state drive (SSD) that can better
cope with the IO load. 
 Split the application load between hard disks: Add hard drives and
divide the IO load between them. You may also want to consider giving
the operating system’s root filesystem its own drive, so that the
operating system and cPanel & WHM don’t have to compete with user
applications.
 Increase the server’s memory: If applications can cache more data in
RAM, they will not need to read from and write to the filesystem as
often.  For some applications, an in-memory cache such as Memcached
or Varnish may improve performance while reducing disk IO. 
 Check the RAID array hardware: A malfunctioning RAID controller can
degrade IO performance. 

Disk IO bottlenecks can be tricky to diagnose, but the process we’ve outlined
here will help you to quickly determine whether you have an IO problem,
which drives are affected, and what you can do to improve your server’s
performance. 
As always, if you have any feedback or comments, please let us know. We
are here to help in the best ways we can. You’ll find us on Discord, the cPanel
forums, and Reddit.

You might also like