Module 3: Tools for

Troubleshooting vSphere

© 2020 VMware, Inc.

Importance
By knowing how and when to use different troubleshooting tools to address various types of
problems, you can save time in identifying and solving problems.
You can use the GUI, the command-line interface (CLI), and log files to analyze problems and
resolve them effectively.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3-2

Module Lessons
1. Using the Command Line for Troubleshooting
2. Using Command-Line Tools
3. Logging and Log Files
4. VMware Skyline Overview

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3-3

Lesson 1: Using the Command Line for
Troubleshooting

© 2019 VMware Inc. All rights reserved.

Learner Objectives
After completing this lesson, you should be able to meet the following objectives:
• Identify ways to run commands
• Identify ways to access vSphere ESXi Shell
• Run commands to view, configure, and manage vSphere components

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3-5

vSphere Troubleshooting Toolkit
A vSphere troubleshooting toolkit consists of
all the tools that you need for troubleshooting
problems in your environment.
From these various tools, you can choose the
appropriate tool for any troubleshooting
situation.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3-6

Troubleshooting References and Documentation
To find information about concepts, processes, known issues, troubleshooting tips, and so on,
you can check the following sources:
• VMware knowledge base articles: https://kb.vmware.com
• VMware documentation: https://docs.vmware.com
• VMware blogs: https://blogs.vmware.com
• VMware community forums: https://communities.vmware.com
• VMware Compatibility Guide: https://www.vmware.com/resources/compatibility
• Product Interoperability Matrices:
https://www.vmware.com/resources/compatibility/sim/interop_matrix.php
• vSphere 7 release notes:
https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-vcenter-server-70-release-
notes.html
What other references and documentation are in your toolkit?

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3-7

User Interfaces: GUI
You use the vSphere Client for most of your
day-to-day operations. Other GUIs are
available for management and troubleshooting
tasks:
• vSphere Client
• VMware Host Client
• vCenter Server Appliance Management
Interface

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3-8

User Interfaces: CLI
If the vSphere Client becomes unavailable, the
command line is a useful alternative:
• ESXi direct console user interface (DCUI)
• SSH clients to ESXi hosts or vCenter Server
Appliance, such as PuTTY and MTPuTTY
• vCenter Server Appliance console

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3-9

Running Commands
You can access the command-line interface on an ESXi host in different ways:
• SSH to the ESXi host
• DCUI to the ESXi host and press ALT+F1 to access vSphere ESXi Shell
• Use Standalone ESXCLI remotely

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 10

Accessing vSphere ESXi Shell
You can access vSphere ESXi Shell either locally or remotely:

• Local access through the DCUI: • Remote access through SSH:

– Enable the vSphere ESXi Shell service, – Enable the SSH service, either in the
either in the DCUI or the vSphere Client. DCUI or the vSphere Client.
– Access vSphere ESXi Shell from the DCUI – Use an SSH client, such as PuTTY, to
by pressing Alt+F1. access vSphere ESXi Shell.
– Disable the vSphere ESXi Shell service – Disable the SSH service when not using
when not using it. it.
– Return to the main DCUI view by pressing
Alt+F2. This function leaves you logged
into the shell, if you return with Alt+F1,
so log out first if needed.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 11

vSphere ESXi Shell and SSH Timeout (1)
The Availability timeout setting for vSphere ESXi Shell determines how long new connections
are allowed after the service is enabled. This setting does not affect existing connections.
The Idle timeout setting logs out local and remote (SSH) users if their sessions are idle for the
defined period.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 12

vSphere ESXi Shell and SSH Timeout (2)
Availability and idle timeouts can be configured in the DCUI only when both the vSphere ESXi
Shell and SSH services are disabled.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 13

ESXCLI Commands
The ESXCLI commands are a comprehensive set of commands for managing most aspects of
the vSphere environment.

• esxcli device • esxcli nvme

• esxcli esxcli • esxcli rdma
• esxcli elxnet • esxcli sched
• esxcli fcoe • esxcli software
• esxcli graphics • esxcli storage
• esxcli hardware • esxcli system
• esxcli iscsi • esxcli vm
• esxcli network • esxcli vsan

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 14

Viewing vSphere Storage Information
You run the esxcli storage command to retrieve storage information, including multipathing
configuration, LUN specifics, and datastore settings.
The esxcli storage command includes options such as esxcli storage nfs41, esxcli
storage vflash, esxcli storage vvol, and so on.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 15

Viewing vSphere Network Information
You can run the esxcli network command to display physical and virtual network information.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 16

Viewing Standard Switch Information
You can run the esxcli network vswitch standard command to create standard switches.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 17

Viewing Distributed Switch Information
Although you cannot create a distributed switch from the command line, you can retrieve
information for existing distributed switches.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 18

Viewing Hardware Information
You can run the esxcli hardware command to discover the physical composition of ESXi
hosts.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 19

Using the vim-cmd Tool
The VMware Infrastructure Management command (vim-cmd) is a CLI tool for managing ESXi
hosts and their VMs from the command line:
• Communicates with hostd to perform tasks on the ESXi host
• Has several command categories

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 20

Using esxtop Utility
The esxtop utility provides much of the same information as the vSphere Client but
supplements the output with more detailed information. It displays information in percentages
of sample time (5 seconds by default).
You can use different esxtop modes:
• Interactive mode: Displays statistics in different panels
• Batch mode: Collects and saves resource utilization statistics in a file
• Replay mode: Replays resource utilization statistics collected using the vm-support tool.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 21

Labs
Lab: Using the Command Line
Lab: Using vim-cmd Commands

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 22

Lab 1: Using the Command Line
Use the command line to review the ESXi host configuration:
1. Access Your Student Desktop System
2. Validate the vSphere Licenses
3. Directly Access the DCUI of the ESXi Host
4. Remotely Access the DCUI of the ESXi Host
5. Use ESXCLI Commands to View Host Hardware Configuration
6. Use ESXCLI Commands to View Storage Information
7. Use ESXCLI Commands to View Virtual Switch Information

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 23

Lab 2: Using vim-cmd Commands
Use vim-cmd commands to manage ESXi hosts and VMs:
1. Get VM Information
2. Manage the ESXi Hosts
3. Register a VM
4. Power On a VM
5. Unregister a VM

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 24

Review of Learner Objectives
After completing this lesson, you should be able to meet the following objectives:
• Identify ways to run commands
• Identify ways to access vSphere ESXi Shell
• Run commands to view, configure, and manage vSphere components

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 25

Lesson 2: Using Command-Line Tools

© 2019 VMware Inc. All rights reserved.

Learner Objectives
After completing this lesson, you should be able to meet the following objectives:
• Use Standalone ESXCLI for troubleshooting
• Run ESXCLI commands from the Standalone ESXCLI
• Use Data Center CLI commands for troubleshooting
• Run commands to view, configure, and manage your vSphere components
• Identify the best tool to use for command-line interface troubleshooting

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 27

About Standalone ESXCLI
With the Standalone ESXCLI tool, you use the ESXCLI command set for the remote
management of ESXi hosts.
ESXCLI was previously part of vSphere CLI and is now provided as a separate package.
Features of Standalone ESXCLI include:
• Available in Linux and Windows versions
• Includes ESXCLI commands

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 28

ESXCLI Authentication from Standalone ESXCLI
You must be authenticated before you can use ESXCLI commands from a troubleshooting
server. You can authenticate in the following ways:
• Manual authentication:
– Add the thumbprint, user ID, and password to the command.
– Provide only ESXi host information.
• Digital certificate authentication:
– Load the digital certificate from vCenter Server and use it to authenticate.
– Provide the user ID and password for vCenter Server.
• Credential store authentication:
– Add the user ID, password, and thumbprint of each system to the credential store.
– No credentials are required for each esxcli command.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 29

Manual ESXCLI Authentication (1)
For manual authentication, you must enter all authentication information for each esxcli
command.
The first use of an esxcli command reports that the ESXi host is not trusted and prints the
thumbprint of the ESXi host.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 30

Manual ESXCLI Authentication (2)
Repeat the esxcli command with the following credentials:
• ESXi host user name and password
• ESXi host thumbprint

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 31

Digital Certificate Authentication
When using digital certificate authentication, you must download the certificate authority (CA)
digital certificate that vCenter Server is using and reference it in the command:
• Most ESXi hosts and vCenter Server instances use the CA built into the vCenter Server
instance.
• Procedures for downloading the certificate and installing it vary, depending on the operating
system that Standalone ESXCLI is hosted on.
• The certificate file, user name, and password of the vCenter Server instance are required in
the command.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 32

Credential Store Authentication
You can add the user names, passwords, and thumbprints of all ESXi hosts to your credential
store.
1. Add the user name and password for the ESXi host.
2. Add the thumbprint for the ESXi host.
This thumbprint is returned in the error that results when you attempt to connect to the host
in the previous step.
In vSphere 7.0, the credstore_admin.pl command is bundled with the vSphere SDK for Perl.
The command is available for Linux and Windows and is located in the /vmware-vsphere-cli-
distrib/apps/general directory on Linux systems and in C:\Program Files (x86)\VMware\
VMware vSphere CLI\Perl\apps\general on Windows systems.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 33

About DCLI
The Data Center CLI (DCLI) is a client of the vSphere Automation SDK for managing data
center services.
The DCLI communicates with a vCenter Server system and reports information at the vCenter
Server level.
A DCLI command talks to a vSphere Automation SDK endpoint to locate relevant information
and then runs the command and displays results to the user.
You can run DCLI commands interactively or from scripts:
• Run DCLI commands locally from a vCenter Server Appliance shell.
• Run DCLI commands from the standalone package installed on a Linux or Windows system.
• While in interactive mode, DCLI can prompt you for command options.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 34

DCLI Authentication
Most DCLI commands require authentication. VMware Cloud on AWS, NSX-T Data Center, and
vCenter Server use different authentication mechanisms.
• VMware Cloud on AWS and NSX-T Data Center both use a refresh token to perform
authentication.
• After you provide the refresh token, DCLI asks whether you want to save it to the credential
store. If you save the refresh token, you do not need to provide it each time you connect to
VMware Cloud on AWS or NSX-T Data Center.
• DCLI supports storing multiple refresh tokens in the credential store. Each refresh token
stored in the credential store also keeps information about the user and organization ID
associated with the token.
• You can authenticate to vCenter Server by providing a user name and password.
• If you save the user name and password in the credential store, you do not need to provide
credentials each time you connect to vCenter Server.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 35

Using DCLI and Certificates
On first connection, you must specify whether to use DCLI with or without security certificates.
Log in with an SSL certificate:
• dcli +interactive +server <vCenter Server FQDN> +cacert-file <SSL
certificate>

Log in without an SSL certificate:

• dcli +interactive +server <vCenter Server FQDN> +skip-server-verification

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 36

DCLI Credential Management (1)
After you connect, with or without a certificate, you must still enter login credentials.
On first use, you are asked if you want to store the current credentials.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 37

DCLI Credential Management (2)
You can list, add, and remove credentials.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 38

DCLI Interactive Command Help (1)
In interactive mode, DCLI helps you with commands.
All commands available in a specific namespace are displayed.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 39

DCLI Interactive Command Help (2)
When you enter previously used commands,
you get suggestions for similar commands.
You can use the tab key to autocomplete the
word that you started entering.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 40

Choosing a Method for Running CLI-Based Commands
Each method of running CLI-based commands has advantages and disadvantages depending
on the situation:
• Use the direct console when the management network connection to an ESXi host is down.
• Use SSH as a simple type of connection when the management network of an ESXi host is
up.
• Use Standalone ESXCLI when you require a single command-line platform to communicate
remotely with all your ESXi hosts and vCenter Server systems.
• Use ESXCLI as a command-line tool to manage and report the configuration of any ESXi host,
including hardware information.
The ESXCLI scope is limited to a single ESXi host, but a server that has Standalone ESXCLI
installed can be configured to connect and manage all ESXi hosts on a network.
• Use DCLI as a command-line tool to manage the vCenter Server configuration.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 41

About PowerCLI
PowerCLI is an extension to Microsoft PowerShell.

• PowerCLI has the following features: • Power CLI can manage and automate the
following components:
— A Windows PowerShell interface to the
vSphere API — vSphere
— Over 700 cmdlets — vCloud
— A function library — vRealize Operations Manager
— Can be called by vRealize Orchestrator — vSAN
— NSX-T Data Center
— VMware Cloud on AWS
— VMware HCX
— VMware Site Recovery Manager
— VMware Horizon

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 42

vSphere PowerCLI Cmdlet Structure
vSphere PowerCLI cmdlets are consistent with Windows PowerShell command syntax.
A cmdlet uses a verb-noun structure, where the verb represents the action and the noun
represents the object to operate on.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 43

vSphere PowerCLI Cmdlet Types
All cmdlets follow the same verb-noun structure. Commonly used cmdlet types fall into several
categories:
• Get cmdlets retrieve information about objects:
– Example: Get-VM
• New cmdlets create objects:
– Example: New-VM
• Set cmdlets change objects:
– Example: Set-VM
• Move cmdlets relocate objects:
– Example: Move-VM
• Remove cmdlets delete objects:
– Example: Remove-VM

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 44

Connecting with vSphere PowerCLI
Run the Connect-VIServer command to connect to all the vCenter Server systems or ESXi
hosts.
After connecting, run vSphere PowerCLI cmdlets to perform administration tasks. Run the
Disconnect-VIServer cmdlet to end the session.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 45

vSphere PowerCLI Cmdlet Example
Use the –Name parameter to filter the objects that are returned.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 46

Using the Show-Command
Using the Windows PowerShell Show-Command cmdlet, you can run vSphere PowerCLI cmdlets,
such as Get-VM in a command window.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 47

Lab 3: Using Standalone ESXCLI and DCLI
Use Standalone ESXCLI and DCLI to review the ESXi host configuration and the data center
configuration:
1. Log In to Standalone ESXCLI
2. Load the Digital Security Certificate from the vCenter Server System
3. Test the Digital Security Certificate from the vCenter Server System
4. (Optional) Add Credentials and Thumbprint for ESXCLI Commands
5. Use the DCLI to Manage vCenter Server

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 48

Review of Learner Objectives
After completing this lesson, you should be able to meet the following objectives:
• Use Standalone ESXCLI for troubleshooting
• Run ESXCLI commands from the Standalone ESXCLI
• Use Data Center CLI commands for troubleshooting
• Run commands to view, configure, and manage your vSphere components
• Identify the best tool to use for command-line interface troubleshooting

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 49

Lesson 3: Logging and Log Files

© 2019 VMware Inc. All rights reserved.

Learner Objectives
After completing this lesson, you should be able to meet the following objectives:
• Locate important log files
• Identify ways to use log files in troubleshooting

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 51

Location of vCenter Server Logs
Most vCenter Server log files are on vCenter Server Appliance in the /var/log/vmware/
directory:

• vCenter Server components have

subdirectories. For example, a vpxd-svcs
subdirectory contains vCenter Server
service logs.
• Logs created during the firstboot process
are written to /var/log/firstboot.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 52

vCenter Server Logs for Troubleshooting
The vpxd.log file is the main log file for vCenter Server. Most vCenter Server actions are
captured in vpxd.log, which is located in /var/log/vmware/vpxd/.
Several other useful logs can be found in the /var/log/vmware/ directory.

/var/log/vmware/ Directory Description

applmgmt/applmgmt.log VMware Appliance Management Service
sca/sca.log VMware Service Control Agent
vmafdd/vmafdd.log VMware Authentication Framework Daemon
vmon/vmon.log VMware Service Lifecycle Manager
vsphere-ui/logs/ vSphere Client
vsphere_client_virgo.log
vmdrird/vmdird-syslog.log VMware Directory Service daemon
vpostgres/postgresql-#.log vFabric Postgres database service
content-library/cls.log Content Library Service

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 53

vCenter Server Log: vpxd.log (1)
The vpxd.log file is the main vCenter Server log, consisting of internal tasks and events and
communication with the vCenter Server agent (vpxa) on managed ESXi or ESX hosts.
vpxd.log is located in /var/log/vmware/vpxd on vCenter Server and provides the following
types of information:
• Time the service was last started, pid of the vpxd process, and build and version
information:
2020-05-18T11:53:43.775Z - time the service was last started, Section for
VMware VirtualCenter, pid=57968, version=7.0.0, build=15952599,
option=Release
• Health status:
Wrote vpxd health XML to file /etc/vmware-sca/health/vmware-vpxd-health-
status.xml. Status: START_PENDING. Expiration: 255756

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 54

vCenter Server Log: vpxd.log (2)
• ESXi host heartbeat information:
Missed 8 heartbeats for host [vim.HostSystem:host-1044,sa-esxi-
03.vclass.local]
• Task details communicated to vpxa on the managed hosts:
Powering on VM '[sa-esxi-01-local] linux-a-01/linux-a-01.vmx' on host sa-
esxi-01.vclass.local

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 55

Location of ESXi Host Logs
The ESXi host log files are in the /var/log directory of the ESXi host.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 56

ESXi Host Logs for Troubleshooting
ESXi hosts write to multiple log files, depending on which action is performed.
Several ESXi host log files are useful for troubleshooting.

Log File Purpose

hostd.log Host management service logs
syslog.log Management service initialization, watchdogs, scheduled tasks, and
DCUI use
vmkernel.log Core VMkernel logs, including device discovery, storage and
networking device and driver events, and virtual machine startups

vmkwarning.log A summary of warning and alert log messages excerpted from the
VMkernel logs

vmksummary.log A summary of ESXi host startup and shutdown

An hourly heartbeat with uptime, number of virtual machines running,
and service resource consumption

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 57

ESXi and VM Start: vmkernel.log (1)
Located in /var/log, the vmkernel.log file logs the following entries during ESXi boot and VM
start:
• Initialization and loading of drivers and kernel modules:
Loading module migrate ...
Mod: 4997: Initialization of migrate succeeded with module ID 90.
migrate loaded successfully.
• Mounting of file systems:
LVM: 17187: File system '[Shared2, 5a9f7703-5de3241d-d830-00505601cab6]' (LV
5a9f7703-f3599d15-5ea0-00505601cab6) mounted in 'rw' mode.
• Service startup:
Starting service ESXShell

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 58

ESXi and VM Start: vmkernel.log (2)
• rc script startup:

Starting up rc scripts
init Completed /etc/rc.local.d
• Host boot status:
Boot Successful
• After the ESXi host is up and running, the vmkernel.log file contains entries for storage and
networking device events, which are useful for troubleshooting storage and networking
issues:
Adding world 'vmm0:linux-a-03' ....
Net: 2167: connected linux-a-03 eth0 to VM Network

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 59

ESXi Host Uptime and Availability: vmksummary.log
Located in /var/log, the vmksummary.log file contains information about ESXi host uptime
and availability:
• Host boot state changes:
2020-03-24T15:10:36Z bootstop: Host is rebooting
2020-03-24T15:14:01Z bootstop: Host has booted
2020-03-25T08:58:55Z bootstop: Host is powering off
• Hourly heartbeat information:
2020-05-17T09:00:00Z heartbeat: up 1d20h6m47s, 6 VMs; [[139245 vmx 303104kB]
[135482 vmx 311488kB] [135671 vmx 311640kB]] []
2020-05-17T10:00:00Z heartbeat: up 1d21h6m47s, 6 VMs; [[139245 vmx 303104kB]
[135482 vmx 311488kB] [135671 vmx 311640kB]] []

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 60

Observations About the VMkernel: vobd.log (1)
Located in /var/log, the vobd.log file contains observations about the VMkernel, which
includes the following events:
• Events sent to hostd:
An event (esx.audit.shell.enabled) could not be sent immediately to hostd;
queueing for retry.
• Enabling services:

The ESX command line shell has been enabled

SSH access has been enabled
• Configuration changes:
Firewall configuration has changed. Operation 'add' for rule set fdm
succeeded.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 61

Observations About the VMkernel: vobd.log (2)
• Synchronizing of the system clock:
... system clock synchronized to upstream time servers
• Storage path connectivity status:
Lost connectivity to storage device naa.60003ff44dc75adc95528de5d47b2a8e.
Path vmhba65:C0:T0:L3 is down. Affected datastores: "Shared2".
• Network link connectivity status:
Uplink: vmnic0 is up.#
Physical NIC vmnic0 linkstate is up

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 62

Managing ESXi Host Operations: hostd.log
The hostd agent is responsible for managing ESXi host operations, such as VM creation and
management, vSphere vMotion migrations, and ESXi host configuration, and so on.
Located in /var/log, the hostd.log file contains entries about the activity and tasks
performed by the hostd agent:
• Creation and completion of tasks:
Task Created : <task_identifier> ...
Task Completed : <task_identifier> ...
Status success
• VM state transitions:
State Transition (VM_STATE_POWERING_ON -> VM_STATE_ON)
• Host boot status:
Event 320 : Host is rebooting.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 63

Example of hostd.log Entries
When you power on a VM, the hostd.log file shows that a task to power on the VM is created.
Eventually, the task completes, and you see a success status.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 64

Monitoring the hostd Process: hostd-probe.log
The hostd-probe process monitors the hostd process and documents each connection to the
hostd process in the hostd-probe.log file, located in /var/log.
Every five minutes, the hostd-probe process checks whether hostd is running and documents
its findings in the hostd-probe.log file.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 65

Information About the vCenter Server Agent: vpxa.log (1)
The vCenter Server agent (vpxa) runs on the ESXi host.
If the host is managed by vCenter Server, the vpxa.log file contains information about the
agent that communicates with the vpxd process on the vCenter Server, and the host
management hostd agent on the ESXi host:
• Time of last log rotation, time of last service start, pid of the vpxa process, and build and
version information:
2020-05-19T07:03:00.342Z - last log rotation time, 2020-05-18T10:26:00.507Z -
time the service was last started, Section for VMware ESXi, pid=179693,
version=7.0.0, build=15843807, option=Release
• Host status and configuration changes sent to vCenter Server:
Increment master gen. no to (1093): Dvs:VpxaDvsHostd::ProcessDvsEvent
Increment master gen. no to (1094):
VmRuntime:VpxaInvtVmListener::RuntimeChanged
Increment master gen. no to (1095): VmRuntime:GuestHeartbeatStatusChanged

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 66

Information About the vCenter Server Agent: vpxa.log (2)
• QuickStats data sent to vCenter Server:
BEGIN lro-154 -- vpxa -- vpxapi.VpxaService.fetchQuickStats -- 527f86eb-8f14-
e8f4-9f4f-272178fd3f8e
• Virtual machine task information:
BEGIN task-6403 -- vpxa -- vpxapi.VpxaService.powerOffVm -- 522864fe-70e7-
e381-2004-f79990664381

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 67

VM Activity and Configuration: vmware.log
Located in the VM's home directory, the
vmware.log file contains information about
VM's activity and configuration, which includes
the following information:
• Power events
• VMware Tools status
• VM snapshot events
• Virtual hardware changes
• VM migration events

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 68

Using vSphere Log Files
Although log files contain a vast amount of information, you use a small percentage of that
information for troubleshooting.
Consider the following guidelines when using log files:
• Determine which log files to consult for any given problem.
• Identify useful keywords, phrases, or messages to search for within a log file.
• Use the object's name, such as the VM name, the ESXi host name, or the network uplink
name to improve your searches.
• Find the date and time that the problem occurred to narrow your search down to the most
relevant information. Most of the entries in these log files begin with a timestamp.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 69

Viewing Log Files in the DCUI
If vCenter Server is not available, you can use the DCUI to view log files.
Only the log files for a single ESXi host can be viewed in the DCUI.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 70

Lab 4: ESXi Command History
Determine commands run by each user in the ESXi Shell command history:
1. View Command History

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 71

Review of Learner Objectives
After completing this lesson, you should be able to meet the following objectives:
• Locate important log files
• Identify ways to use log files in troubleshooting

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 72

Lesson 4: VMware Skyline Overview

© 2019 VMware Inc. All rights reserved.

Learner Objectives
After completing this lesson, you should be able to meet the following objectives:
• Describe the benefits and capabilities of VMware Skyline
• Explain how VMware Skyline works
• Identify the types of health information provided by Skyline Health
• Identify uses for Skyline Advisor
• Describe the support bundle REST API
• Use the support bundle REST API

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 74

About VMware Skyline
VMware Skyline is a proactive support technology that provides predictive analysis and
proactive recommendations to help you avoid problems. VMware Skyline provides the following
benefits:
• Issue avoidance: • Personalized recommendations:
— Proactively identifies potential issues — Resolution is specific to your
based on environment-specific environment.
configuration, details, and usage.
• No additional cost:
— Resolves issues before they occur,
— You receive additional value with your
improving environment reliability and
stability. current support subscription (Basic,
Production, or Premier support).
• Shortens time to resolution:
— Environment-specific, data-driven
analytics accelerate problem resolution.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 75

VMware Skyline Family
The VMware Skyline family includes Skyline Health and Skyline Advisor.

Skyline Health Skyline Advisor

All VMware Customers Production and Premier Support Customers
Key capabilities: Key capabilities:
• vSphere and vSAN findings • Supports vSphere, vSAN, NSX for vSphere,
• Available in the vSphere Client vRealize Operations Manager, and VMware
• Supports vSphere 6.7 and later Horizon
• Supports vSphere 5.5 and later
• Tags VMware Validated Design, VxRail,
and VMware Cloud Foundation
deployments
• Automates log transfers with Log Assist
• Uses cloud-based ID and access
Premier Support Customers
Key capabilities:
• Advanced findings and reporting
• Tailored remediation plans
© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 76
Skyline Health (1)
Skyline Health provides findings for your vSphere and vSAN environment.

For vSphere: For vSAN:

• Findings are based on telemetry data • Findings are based on data from thousands
collected from your vSphere environment. of vSAN deployments.
• Data is analyzed and then provided in the • Findings draw on VMware knowledge base
vSphere Client. articles and best practices.
• Discovered findings can be related to • Discovered findings can be related to
stability and incorrect configurations in misconfigurations and the general health
vSphere. and well-being of hosts in a vSAN cluster.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 77

Skyline Health (2)
You access Skyline Health with
the vSphere Client.
For Skyline Health to function
correctly:
• You must join the Customer
Experience Improvement
Program (CEIP).
• Your vCenter Server
instance must have Internet
connectivity.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 78

Skyline Components: Collector and Advisor
Skyline Advisor and Skyline Collector work
together to provide proactive support:
• Skyline Collector is a standalone virtual
appliance that collects diagnostic data
about your vSphere, NSX for vSphere, vSAN,
VMware Horizon, and vRealize Operations
Manager environments.
• Skyline Advisor is a web-based portal where
you can view proactive findings and
recommendations on demand.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 79

How VMware Skyline Works
1. Install and configure:
• You install the Skyline Collector appliance into your environment.
• You configure Skyline Collector to have one or more endpoints (for vCenter Server, NSX
Manager, View, vRealize Operations).
2. Collect:
• Skyline Collector starts collecting product usage data and capturing patterns, events,
trends, configuration, design-compliance, and cross-product information.
3. Analyze:
• VMware Skyline analyzes the environment details and product usage and compares this
data against VMware best practices and the VMware knowledge base.
4. Act:
• VMware Skyline provides proactive findings and recommendations.
• You access this information through the Skyline Advisor, a web-based portal.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 80

Skyline Advisor
You use Skyline Advisor to view
account details, inventory
details, findings, and proactive
recommendations, to
download Operational
Summary Reports (OSR)
(Premier Support customers
only), and to transfer support
log bundles using Log Assist.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 81

Skyline Advisor: Proactive Findings
Proactive findings are at the heart of VMware Skyline.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 82

Log Assist
When using VMware Skyline
Log Assist, you do not have to
upload your log files manually
anymore.
Support log bundles can be
automatically uploaded to
VMware Technical Support.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 83

Skyline Features Comparison by Support Level

Features Basic Production Premier

Support Support Support
Skyline Health Yes Yes Yes
Basic findings and recommendations Yes Yes Yes
Moderated Skyline community Yes Yes Yes
Skyline Advisor No Yes Yes
Log Assist No Yes Yes
Proactive support for all supported products No Yes Yes
Export of findings, recommendations, and No Yes Yes
affected objects to a comma-separated file
Advanced findings and recommendations No No Yes
Custom operational summary reports No No Yes
Guided remediation from VMware Global Support No No Yes
Services

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 84

Skyline Privacy and Security Policy
VMware is committed to keeping your data private and secure:
• The Skyline Collector appliance is separate from your environment:
– Your production infrastructure is not exposed to the Internet.
• Collected data is forwarded to VMware using an encrypted channel:
– The data is only temporarily stored in the Skyline Collector appliance.
• Data is warehoused in a secure back-end system in the United States and is managed by
VMware:
– Access to your data is restricted to permitted users only.
• Information is limited to data such as host names:
– Data that is known to identify an individual person is not collected.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 85

About the Support Bundle REST API
vSphere 7 Update 1 provides a new appliance management
REST API to generate and download a support bundle, even if
the vCenter Server service is offline.
A downloaded bundle is deleted after 30 minutes. If a
download is in progress, the deletion is deferred for 30
minutes.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 86

Permissions When Using the Support Bundle REST API
Users must be a member of the new
SystemConfiguration.SupportUsers vCenter Single Sign-On
group.
This group provides users with the permissions to call only the
support bundle REST API.

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 87

Using the Support Bundle REST API (1)
To list the individual components that are gathered by the support bundle REST API:
• GET https://vcenter:5480/rest/appliance/support-bundle/components
To define the description, components, and partition details:
• POST https://vcenter:5480/rest/appliance/support-bundle?action=create&vmw-
task=True
– Takes the following parameters:
{
"spec": {
"description" : "Test support bundle",
"components" : {
"VirtualAppliance" : ["ApplianceManagement"]
},
"partition": "/storage/core"
}
}
– Returns a task ID

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 88

Using the Support Bundle REST API (2)
To return the status of the provided task ID:
• GET https://vcenter:5480/rest/cis/tasks/<task-id>
To return the completed details for the support bundle generation task:
• GET https://vcenter:5480/rest/appliance/support-bundle
– Returns the download URL of the generated support bundle
{
"supportbundle_operations": [
{
"description": "Test support bundle",
"status": "SUCCEEDED",
"generation_time": "2020-04-20T12:56:30.170Z",
"url": "https://vcenter:5480/diagnostics/vc-support-
MHNG68Xs.tgz",
"expiration_time": "2020-04-20T13:26:30.170Z",
"downloadable": true,
"size": 38296
}
]
}
© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 89
Troubleshooting the Support Bundle REST API
The support bundle REST API reports on errors and failures.
• In the example, the specified partition does not have enough space to store the support
bundle.
{
"supportbundle_operations": [
{
"description": "Test support bundle",
"status": "FAILED",
"failure_reason": {
"id": "supportbundle.failure.reason",
"default_message": "Partition /storage/netdump does not have
enough space to store the support bundle",
"args": []
}
}
]
}

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 90

Review of Learner Objectives
After completing this lesson, you should be able to meet the following objectives:
• Describe the benefits and capabilities of VMware Skyline
• Explain how VMware Skyline works
• Identify the types of health information provided by Skyline Health
• Identify uses for Skyline Advisor
• Describe the support bundle REST API
• Use the support bundle REST API

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 91

Key Points
• You can use vSphere ESXi Shell to run commands.
• You can use Standalone ESXCLI and DCLI to view and troubleshoot the system configuration
of ESXi hosts and vCenter Server systems.
• Log files are useful when trying to resolve vSphere problems.
• Using various vSphere features, you can collect, search, and export log files.
• Using VMware Skyline, you can reduce the time it takes to resolve a problem.
Questions?

© 2020 VMware, Inc. VMware vSphere: Troubleshooting | 3 - 92