Rodmach Troubleshooting
Rodmach Troubleshooting
Rod Mach [email protected]
Application Owner: I Changed Nothing
Storage Team: All Green Here
Storage Team: All Green Here
Network Team: No Problem on My End
VMware Team: Yeah I’ll Get Back To You
CPU MEM
STORAGE Network
CPU Issues
Is the VM Guest CPU Utilization High at 100% ?
Add a vCPU if App is MultiThreaded
Do NOT add vCPUs Without DataDriven Cause
If CPU Utilization in the VM Guest is Unusually Low
Check if App is waiting on an external dependency
Is the VM CPU Ready > 2000 ms?
ESX Host Saturated. Change Priority of VM or vMotion
Debugging CPU with ESXTOP
If PCPU >= 100% System CPU Saturated
If PCP
http://communities.vmware.com/docs/DOC Check %RDY
5420 for VMs > 10%
If %USED is > 100% x # vCPUS, add more vCPUS to VM
http://communities.vmware.com/docs/DOC5420
http://communities.vmware.com/docs/DOC5240
Switch to CPU
Select CPU
Ready in
Chart
Options
Looks Acceptable <
The Pie Method of CPU Allocation
Put Each VM in a Low, Normal, High Bucket
Understand Memory Overcommit
Memory Issues
Is the VM Swapping Inside The Guest VM?
Add more memory to the VM
Is Memory Ballooning Occurring Even If Memory Not
Overcommitted?
Check for Memory Limits on VMs. – REMOVE!
Chart Options – Your Friend
STACKED
Select Counter
Look at VMs on Host for Which Ones Swapping
VM Memory Swapping Solutions
Memory Swapping Solutions
• Reduce the level of memory overcommit.
• Enable the balloon driver in all VMs.
• Reduce memory reservations.
• Use resource controls to dedicate memory to
critical VMs.
Storage Issues
Excessive demand being placed on the storage
device.
Misconfigured storage. Disks per LUN, RAID
levels, Caches, HBA queues, IOPS, Alignment
Issues. Vendor Specific.
Switch to Datastore
If over 40 ms, there is
Storage Contention
Debugging Storage Issues
The Ultimate Tool for Storage Debugging vscsiStats
http://communities.vmware.com/docs/DOC10095
Look at your Storage Vendors Tools
Storage Resolutions
Storage I/O Control in vSphere 4.1 Can Fairly
Allocate Datastore Access Across ESX Hosts.
Must be Enabled in vSphere Client
Request both storage CAPACITY (TB) and
storage PERFORMANCE (IOPS). Examine the
IOPS of your workload versus your storage
capability. This is often mismatched.
Network Issues
Look at Dropped Packets
Choose Dropped Packets from Chart Options
Network Resolutions (Transmit)
Move some VMs with high network
demand to a different vSwitch.
Add additional uplink capacity to the
vSwitch.
Network Resolutions (Receive)
Add vCPUS
Try VMXNet3 Adapter Type
Add additional vNICS
Tune Guest VM OS
Thanks!
Rod Mach [email protected]