Describing Cisco HX Software Components

Session Agenda

Hypervisor support for HyperFlex

Key Take Away

HyperFlex Data Platform • Understand Software Components

of Cisco HyperFlex

Log-Structured File System

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Software associated with HyperFlex Solution
• HyperFlex Solution requires the following software

• Hypervisor (VMware ESXi or Microsoft Hyper-V)

• HXDP (HyperFlex Data Platform)
• UCS firmware (A, B and C bundles)

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
VMware ESXi Hypervisor
The ESXi hypervisor in Cisco HyperFlex:

§ Serves as the installed operating system, providing the link

between software and hardware
§ Runs HyperFlex CVMs, which perform HyperFlex-specific tasks
§ Enables a virtual environment for running, deploying and
managing of VMs
§ Integrates virtual network components into the physical network
§ Provides the platform for deployment of HyperFlex-utilized
services such as the IOVisor, DRS, high-availability and VAAI
§ Allows for integration of HyperFlex solution into vCenter

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
HX Installer automates the configuration
• HyperFlex Installer automatically configures and automates the following
components:

• Controller VM
• The Network
• ESXi Configuration
• vCenter Integration

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Log-Structured File System

VMs: X Y Z
StorFS writes information to a disk sequentially and just creates an index where individual parts of a file
are, giving these benefits on spinning drives:
§ All the information is written to the first free space available, the drive does not have to randomly
seek free space
§ Drive does not have to seek on write, improving spinning drive performance
§ Since data is not deleted on write, this reduces write latency

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Log-Structured File System (cont.)
Common benefits of StorFS for All-Flash and Hybrid HyperFlex are:

§ Native deduplication rooted in the design of the file system

§ Superb file system level cloning and snapshot capabilities
§ Pointers can refer to storage that is reachable over the network on another
HyperFlex node and does not exist locally
§ Pointers can take into account redundant copies on HyperFlex storage, providing
redundancy
§ Physical storage on several physical machines can be presented as one contiguous
storage platform

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Cisco HyperFlex vs Regular Virtualized Server
Regular HyperFlex
VM VM VM VM VM VM

Datastore / Volume Datastore / Volume

HXDP
RAID
IOVisor

Hypervisor Hypervisor CVM

VAAI

SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD

HyperFlex virtual servers differ from regular servers in these key areas:
§ No RAID is required to consolidate disks into a shared data platform
§ CVM: Virtual appliance, which performs reading/writing, caching, deduplication and
compression
§ IOVISOR: Hypervisor driver, which mounts HyperFlex storage and distributes data
§ VAAI: vSphere storage API allowing file system-level snapshots and cloning
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 Traditional RAID
VM.vmdk
RAID cards enable striping as well as mirroring
and parity, with these features:
1 2 3 4
§ No load on the system resources, drives seem as one
drive to the operating system
Write

§ Hot replacement of drives available, depending on

configuration
RAID Card (RAID4)
§ Disk replacements require RAID rebuilds, taking a
long time
§ On RAID card failure, the RAID card compatibility can
1 2 4 3 1 2 be an issue
3 4 § Limited drives in a RAID field, depending on solution,
which limits scalability
Drive 1 Drive 2 Drive 3 Parity Drive
§ Only local drives can be in a RAID together
Max. 8 Drives in RAID Field
Stripe
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Mirror
Controller Virtual Machine (cont.)
• The CVMs manage the HyperFlex
cluster and are responsible for the API
explorer (HTTP), HX Connect (HTTP)
and the CLI (SSH)
• CVMs run the following:
§ HX Connect: HTML5 web interface, for
the management of the HXDP
§ HyperFlex CLI: Bash-based command line
for managing and troubleshooting Cisco
HyperFlex
§ HyperFlex API: REST automation
interface for control of Cisco HyperFlex

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Controller Virtual Machine
The CVMs have these features:
§ An Ubuntu-based VM running in the hypervisor of each individual server, having direct
access to the server’s storage
§ Installed automatically by the HyperFlex installer, configured through the installer
§ Needs network access to ESXi, other CVMs and management network
§ Performs caching, deduplication and compression of data
§ Utilizes IOVisor to distribute data across the HyperFlex cluster
§ Provides HX Connect management platform including HyperFlex CLI and API
§ CVMs are responsible for logging

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
IOVisor
Datastore / Volume
Network

IOVisor
VM VM VM VM VM VM

IOVisor IOVisor

Hypervisor CVM Hypervisor CVM Hypervisor CVM

VAAI VAAI VAAI

SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD

• Intercepts local virtual machines’ reads/writes and distributes them across the network, eliminating hotspots

• Provides redundancy when local CVM fails, offloading data processing to another CVM in the cluster

• Enables synchronous replication of data across individual HyperFlex nodes according to replication factor

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
VMware VAAI Storage API
VM VM VM Client
VAAI provides these functionalities:
Datastore / Volume API CLI
VMware
§ When a native snapshot is requested, the
Initiate snapshot request is processed by the hypervisor
§ Instead of the snapshot being performed
in hypervisor it is offloaded to the Cisco
Hypervisor VAAI CVM HXDP
§ Cisco HXDP creates a file system native
User request and snapshot, which is registered in the
Delegate Snapshotting snapshot info. vSphere

SSD SSD SSD SSD SSD

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Cisco HyperFlex Snapshots
Storage Datastore
VM-delta1.VMDK
VM-delta0.VMDK VM-delta1.VMDK
VM.VMDK
VM-delta0.VMDK ESXi Hypervisor

Read
Consolidated Data VM.VMDK VM Write

HyperFlex uses file system native snapshots and provides these features:
§ Automatically deduplicating data of snapshots through StorFS
§ No impact on the VM performance even after many writes
§ On data store level, snapshots work the same as vSphere snapshots
§ Consolidation of snapshots is still manual but not necessary
§ No limitations of age or number of snapshots. First snapshot must by HyperFlex

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 HyperFlex Ready Clones
• The cloning process is more efficient at the storage level
utilizing VMware VAAI.

• Clones are instantaneous

• Clones are created by re-using the parent VM blocks

on storage but are completely independent.

• Integration with all management platforms of Cisco

HyperFlex, including vCenter and batch customization.

- You can also create clones using ReadyClones using HX

Connect, REST API or stcli.

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Data Distribution in Cisco HyperFlex
VM VM.vmdk: A B C D E
Hypervisor CVM Hypervisor CVM Hypervisor CVM Hypervisor CVM
HX Datastore
A1 C2 D3 E1 A2 B1 D2 A3 B2 C1 E2 B3 C3 D1 E3
Local Storage Local Storage Local Storage Local Storage

Cisco HyperFlex upgrades the traditional RAID by:

§ Not limiting the number of drives, which are a part of the shared datastore
§ Distributing data across the hosts in HyperFlex cluster, not just locally
§ Hardware replacement initiates self-healing with minimal impact
§ Eliminating the need for additional hardware cards, while maintaining high
performance
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Data Optimization Process and Actual Data Savings

The process of data optimization is performed in this sequence:

1. On write, the local IOVisor sends the write to the primary CVM for that block
2. The primary CVM compresses the data, writes it to its cache drive and mirrors it
3. ACK is sent to the virtual machine that the write has been successfully performed
4. Once the write log is full, a destage is initiated, where the primary CVM performs a best
effort deduplication and writes the information across nodes

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Cisco HyperFlex vs Other HCI Solutions
1. Data local model performs data operations locally or it performs poorly
2. Optimally, VMs in HCI with data locality do not read from copies, creating load only on the
local node
3. On HyperFlex data operations are always done through the network
4. Blocks of information on HyperFlex are distributed across servers. HyperFlex uses all the
block available for storage operations, distributing the load to all nodes
Reads From Data on Local Server Reads From Any Data Instance in Cluster
VM1 VM2 VM3 VM1 VM2 VM3
1 3

2 4

(Regular HCI) (Hybrid HyperFlex)

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Log Structured File System (StoreFS) in HX
1. StoreFS writes data sequentially to the first free block that is available
2. StoreFS maintains pointers pointing to blocks on the storage.
3. It keeps individual blocks of files on storage as close together as possible, to speed up
reading the entire file.
4. It writes data so that there is as little block fragmentation as possible on the drive.

Reads From Data on Local Server Reads From Any Data Instance in Cluster
VM1 VM2 VM3 VM1 VM2 VM3
1 3

2 4

(Regular HCI) (Hybrid HyperFlex)

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
VM Migration with Data Locality
DRS
VM1 VM2 VM3 VM1
4 VM2 VM3 VM1
2

3
1
CPU 99% CPU 91% CPU 79% CPU 9%

Moving a VM to another server on a regular HCI follows these steps:

1. The initial server is overloaded, and DRS moves the VM to a less used server
2. The reads are remote since the destination server does not have the VM data, which is
suboptimal
3. To reach optimal performance, VM data is copied over to the new server, creating a lot of
load on the source and destination servers, and the connecting network
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
VM Migration without Data Locality
DRS
VM1 VM2 2 VM3 VM1 VM2 VM3 VM1
1 3
4

CPU 99% CPU 91% CPU 79% CPU 9%

1. First server is overloaded, and DRS decides to move the green VM another server
2. The VMs read the stored information across the network from all the servers
3. The VM is transferred to the new server; the reads are still done across the network as
normally
4. Since there is no local data required for optimal performance, there is no data migration
and no additional load on the servers or the network (hotspots)
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Reading in Cisco HyperFlex Read

• First look for data is in the Active write log:

Write Log
• Then the Passive write log

• Second look is in L1 cache: (10 GB in CVM)

• MRU (most recently used) L1
• MFU (most frequently used)

• Top K Majority Algorithm L2

• Third location is L2 in the cache SSD:
• If the file has been evicted from L1

• Last is HDD retrieves file into L1 cache

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Reading in Cisco HyperFlex
• Write-back: Only write information and most commonly used information are deposited in the cache
• Write-through (default): Only most commonly used data is cached, and most recently used
• No caching (SSD): With all-flash nodes, because there is little difference in read speeds between SSDs

Regular Hybrid VDI Hybrid All-Flash

VM (write-through) VM (write-back) VM (no read cache)
MFU MFU
MRU
Write Log Read Log Write Log Read Log Write Log

Capacity Capacity Capacity

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
MFU = Most Frequently Used
MRU = Most Recently Used
Data Distribution in Cisco HyperFlex (cont.)
Maximum Tolerable Failures Dependent on Replication Factor
5 or More Nodes
3-4 Nodes
(No LAZ)
Replication Factor 2 1 node or 1 drive failure 1 node or 1 drive failure

Replication Factor 3 1 node or 2 drive failures 2 node or 2 drive failures

Replication factor implies:

§ RF3 has 100% better protection of data while using 33% more space, and is recommended
§ When performing rolling upgrades, the upgraded node is down for maintenance, stretching
RF2 to the absolute limit
§ LAZ can increase fault tolerance of 8+ node cluster

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Disk Failure
Disk failure initiates this process:
1. If the replication factor is sufficient for the failure, the system is marked as unhealthy but remains
operational
2. VM running on the node is not migrated and the input/output continues from copies
3. Performance is almost unaffected. Sets 1-minute timer until self-healing starts
4. After one minutes, the missing pieces are recreated from the remaining instances

RF3
VM VM VM VM VM VM

IOVisor IOVisor IOVisor

Hypervisor CVM Hypervisor CVM Hypervisor CVM

VAAI VAAI VAAI

SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Node Failure
1. The system is marked unhealthy but remains operational
2. The VMs on the failed node are moved to another node by vSphere high availability
3. VMs keep reading from the remaining copies with minimal impact to performance
4. A 2-hour countdown initiates before self-healing process (can be changed by Cisco TAC)

HA
VM VM VM VM VM VM

IOVisor IOVisor IOVisor

Hypervisor CVM Hypervisor CVM Hypervisor CVM

VAAI VAAI VAAI

SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Expansion and Hardware Replacement
1. vSphere DRS migrates the virtual machines to the new node to balance the load
2. On node replace, the self-healing has to finish for the cluster to be healthy
3. The new node is already used for writing, but the old data is not migrated until the rebalance
process
4. Rebalance is initiated daily at 5:15 am or can be executed manually with the stcli cluster
rebalance command

HA DRS New
VM VM VM VM VM VM

IOVisor IOVisor IOVisor

Hypervisor CVM Hypervisor CVM Hypervisor CVM

VAAI VAAI VAAI

SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD SSD

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Logical Availability Zones
Cluster State: Online

LAZ function segments the cluster into

Zone 01
VM VM VM Controller VM VM VM Controller

resiliency zones to increase tolerable failures: Hypervisor Hypervisor

§ The function does not introduce any additional
storage usage, just rearranges the data

Zone 02
VM VM VM Controller VM VM VM Controller

§ Replication factor defines how many complete zones Hypervisor Hypervisor

can be lost before the system goes down
VM Controller VM Controller
§ In RF2, you can lose one complete zone. In RF3 you VM VM VM VM

Zone 03
can lose two complete zones Hypervisor Hypervisor
§ Only available on large clusters of 8 or more nodes,
with a three zone minimum VM VM VM Controller VM VM VM Controller

Zone 4
§ Can be activated while the cluster is active and the Hypervisor Hypervisor
data will be rebalanced into zones
Zone 5 VM VM VM Controller VM VM VM Controller

Hypervisor Hypervisor

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 LAZ Failure Scenario Cluster State: Online
Offline
CONTROLLER CONTROLLER CONTROLLER CONTROLLER
VM VM VM VM VM VM VM VM VM VM VM VM

HYPERVISOR HYPERVISOR HYPERVISOR HYPERVISOR

CONTROLLER CONTROLLER CONTROLLER CONTROLLER

VM VM VM VM VM VM VM VM VM VM VM VM

HYPERVISOR HYPERVISOR HYPERVISOR HYPERVISOR

CONTROLLER CONTROLLER CONTROLLER CONTROLLER

LAZ: Off VM VM VM VM VM VM VM VM VM VM VM VM

HYPERVISOR HYPERVISOR HYPERVISOR HYPERVISOR

CONTROLLER CONTROLLER
CONTROLLER CONTROLLER
VM VM VM VM VM VM VM VM VM VM VM VM

HYPERVISOR HYPERVISOR HYPERVISOR HYPERVISOR

CONTROLLER CONTROLLER
CONTROLLER CONTROLLER
VM VM VM VM VM VM
VM VM VM VM VM VM

HYPERVISOR HYPERVISOR HYPERVISOR HYPERVISOR

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 LAZ Failure Scenario Cluster State: Online
Offline
CONTROLLER CONTROLLER CONTROLLER CONTROLLER
Zone 01

VM VM VM VM VM VM VM VM VM VM VM VM

HYPERVISOR HYPERVISOR HYPERVISOR HYPERVISOR

CONTROLLER CONTROLLER CONTROLLER CONTROLLER

Zone 02

VM VM VM VM VM VM VM VM VM VM VM VM

HYPERVISOR HYPERVISOR HYPERVISOR HYPERVISOR

LAZ: On CONTROLLER CONTROLLER CONTROLLER CONTROLLER

Zone 03

VM VM VM VM VM VM VM VM VM VM VM VM

HYPERVISOR HYPERVISOR HYPERVISOR HYPERVISOR

CONTROLLER CONTROLLER
CONTROLLER CONTROLLER
Zone 04

VM VM VM VM VM VM
VM VM VM VM VM VM

HYPERVISOR HYPERVISOR HYPERVISOR HYPERVISOR

CONTROLLER CONTROLLER
CONTROLLER CONTROLLER
VM VM VM VM VM VM
VM VM VM VM VM VM

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Deduplication
File1.txt File2.txt
This is why the data is If you do not relocate it,
stored on the datastore the data is stored on 6 Blocks
action is taken the datastore
Blocks indefinitely
HDD

5 Blocks

17% reduction in written data

The deduplication works on the file level using the StorFS log-structured file
system by pointing to blocks which are parts of the file on the storage
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Compression

The compression replaces duplicate parts of the information and encodes them in
a way that they use up as little space as possible.

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
De-duplication savings

BEFORE
No Special Hardware
No Performance Impact
Inline Deduplication Inline Compression No Config lock-in
No Additional License

20–30% space savings 30–50% space savings Lower Cost

© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential