Agend
aMANAGING
L NUTANIX ENVIRONMENTS
• Cluster Monitoring
• NCC overview
• Prism Analysis (and Prism Central)
I I . TROUBLESHOOTING N UTANIX
ENVIRONMENTS
• General Troubleshooting
• Troubleshooting Scenarios
• Engaging support best practices
• Additional Resources
I I I.
Q/A
CONFERENCE
Monitoring
Pulse ,.•,
Emai
l SNMP
Syslog
Prism Alerts
CONFERENCE
Prism Alecs Pulse
HD
I N S GH T S
Pulse
HDurly Cluster
RepDrts
Deep Analytics
And InventDP/
/\UtDFFIBtIC
Case generatiDFl
Cluster PhDn e
Prism Alerts
HDme
Health Alerts
COINF
ERENCE
Auto-case Generation
Example:
Description Block Serlal Number:
alert tima: Tue Mar 22 2016 18:54:51 GMT-0700 (PDT)
aIert_type: PowerSupplyDown
alert msg: A1046:Bottom power supply iB down on
block
cluster id:
aIert„body: No Alert Body Available
New Alerts Appended
Block Serial Number:
alert time: Tue Mar 22 2016 21:46:25 GMT-0700 (PDT)
aIart_typa: PowerSupplyDown
a 6:Top power supply is down on block
cluster id:
aIert_body: No Alert Body Available
Resolution Scheduled Maintenance. As advised by customer
CONFERENCE
Auto-case Generation
THESE ALERTS WILL AUTO GENERATE SUPPORT CASES:
• Stargate process is down for more than 3 hours
(StargateTemporarilyDown)
• Curator scan fails (CuratorScanFailure)
• Running out of space on the cluster
• Running out of space on CVMs
• Hardware Clock Failure (HardwareClockFailure)
• Faulty RAM module (RAMFault)
• Power Supply failure (PowerSupplyDown)
If you want up to date information check
http://portal.nutanix.com/kb/1959 on the portal — KB
1959
For our customers leveraging our partners hardware platforms, we will
generate software based alerts which triggers auto support cases. CONFERENCE
Working with Prism Alerts
›I
i
COINFERE
NCE
Working with Prism Central Alerts Dashboard
COINFERE
NCE
NCC Health
ChecksCLI - (NCC HEALTH PRISM (AOS 5.X)
CHECKS RUN ALL)
- Summary of Cluster Check Executed on 4/28/2047, ¥
Passed
Total
CONFERENCE
DC Chcck Na mc
Checks• Aftecte a C V M s
NCC s a framewo of a tomatically diagnose cluster
scfi$
• Default
hea checks are non-disru we
• KB article for each NCC check
• Helps get a baselines
• NCC can be upgrade
Troubleshooting no impa
withrelevant
Information fincludinp KB) act to cluster
• Poperation
: The tested aspect of the cluster is healthy and no
further
action is required
cannot be evaluated as PASS/FAIL
CONFERENCE
’° w
CONFERENCE
Entity 8‹ Metric Charts
COINFERE
NCE
CONFERENCE
Troubleshooting Nutanix Environments: A Framework
• Problem Isolation
• Fixes and Mitigations
• Root Cause Analysis
• Product
Improvement
CONFERENCE
Troubleshooting by
Layers
A PPLICAT1ON
• SOL, VDI, Oracle RAC. etc.
CVM
• Stargate. Curator. Cassandra. etc.
HYPERVISOR
• AHV, ESXi, Hyper-V, XenServer
HARDWARE
• NVMe. SSD, HDD, Memory, NIC. Processor, etc.
N ETWOR K
• OVS. vswitch, Physical Switch, etc.
CONFERENCE
Troubleshooting: Problem Isolation
• Rapidly reduce failure domain scope. achieve faster resolution.
• Any recent changes in the environment*
IMPACT
• Is storage available*
• Are there performance issues*
• Can you reach Prism*
Use Build -In RE PORTS NG
• Prisrr Alerts
• Cluster Health
• NCC
• Cluster logs
• User Reports
CONFERENCE
Troubleshooting: Problem Isolation — Cluster
States
He \pful additionaJ commands
• cluster status I ex p -v UP showing condensed version
• genesis sBtus — shows only local
services/processes
CONFERENCE
Troubleshooting: Problem Isolation — allssh, hostssh, NCCR
Logging
• allssh
• NCC
• /home/nutanix/data/logs and sysstats
• INFO. WARN, ERROR, FATAL
• allssh "Is -Itr data/logs/*.FATAL”
• If FATALs are actively occurring and you’re experiencing issues, they may be related.
• hostssh "vmware -vl” instead of allssh ‘ssh -I root 192.168.5.1 "vmware -vl”’
• If you’re seeing an error, check the Nutanix Knowledge Base!
CONFERENCE
Problem Isolation - Data Resilenc States
O&
Rebuild capaclty
available
• r›cli cluster get-domain-fault-tolerance-status CONFERENCE
Typical Troubleshooting Scenarios
U PG RADE IS NOT PROG RESS ING
• Logging: genesis.out. host_upgrade.out, firmware_upgrade.out
• upgrade_status
• host_upgrade_status
• firmware_upgrade_status
STORAG E U NAVAI LAB LE
• Do all CVMs have connectivity to each other and to the hypervisor?
• Recent stargate FATALs*
• Cassandra status*
REPLICATION, SN A PS HOTTING , A ND METRO RELATED ISSUES
• Logging: Cerebro logs
NCC // HEALTH C H E C KS FAI LING
• Running NCC should indicate the nature oT the issue and give a KB describing
how to resolve the issue. .
CONFERENCE
rio - fflin
H
AH ĞK
V
CONFERENCE
Root Cause Analysis - Log Collection
Logs will be collected for all the no0es and components. Once the
task completes the bundle will de aveilabJe for download.
5 umm C he ck 0 C oiieci Logs startlng now
ary s E UTII 'I C 'C E R 7 STATL!S
Cluster }0b Succceded
Pun C h ec k s
BY CHI-CK S TA I US
Log Collecfor
Passed 39
1
C an cel
CONFERENCE
Best Practices for Engaging Suppor
• Update your break/fix contact via My Nutanix Portal
• Upgrade to the latest NCC and start a health
check
• Clear problem description
• What steps have you already taken?
• Keep components on the recommended version levels
• Press the Escalate Button in portal for immediate
attention
• Provide feedback after case closure. Surveys
matter!
CONFERENCE
Additional Resources
The Nutanix Bible - Architecture details
portal.nutanix.com - Nutanix Support Portal, KBs, Documentation, Software, etc.
portal.nutanix.com/ h/4530 — Additional troubleshooting details for Acropolis File Services
IF LIKED THIS SESSION, YOU MAY ALSO LIKE:
YOU
• Nutanix Architecture Deep Dive and the Deep Dive Super Session
• Getting the Network Right (The First Time)
• Fail Fast and Never Again
• AHV — Virtualization You Always Wanted
CONFERENCE
