Microsoft SQL Server

Nutanix Best Practices

Version 3.0 • August 2019 • BP-2015

 Microsoft SQL Server

Copyright
Copyright 2019 Nutanix, Inc.
Nutanix, Inc.
1740 Technology Drive, Suite 150
San Jose, CA 95110
All rights reserved. This product is protected by U.S. and international copyright and intellectual
property laws.
Nutanix is a trademark of Nutanix, Inc. in the United States and/or other jurisdictions. All other
marks and names mentioned herein may be trademarks of their respective companies.

Copyright | 2
 Microsoft SQL Server

Contents

1. Executive Summary.................................................................................6

2. Introduction.............................................................................................. 7
2.1. Audience.........................................................................................................................7
2.2. Purpose.......................................................................................................................... 7

3. Nutanix Enterprise Cloud Overview...................................................... 8

3.1. Nutanix Acropolis Architecture.......................................................................................9
3.2. Nutanix Era.................................................................................................................... 9

4. Understanding Microsoft SQL Server................................................. 10

4.1. SQL Server Overview.................................................................................................. 10
4.2. SQL Server Workload Types....................................................................................... 11
4.3. SQL Server Licensing.................................................................................................. 12

5. Physical Hardware Design....................................................................13

5.1. NUMA Architecture and Best Practice.........................................................................13
5.2. Understanding RAM Configuration.............................................................................. 14
5.3. BIOS or UEFI Firmware...............................................................................................17
5.4. Power Policy Settings.................................................................................................. 17
5.5. Nutanix CVM Considerations.......................................................................................17

6. Virtual Hardware Design....................................................................... 20

6.1. CPU Definitions............................................................................................................20
6.2. Hyperthreading Technology......................................................................................... 21
6.3. CPU Hot-Add............................................................................................................... 22
6.4. CPU Ready Time......................................................................................................... 22
6.5. Memory Configuration..................................................................................................23
6.6. Network Configuration..................................................................................................25
6.7. Storage Configuration.................................................................................................. 29

3
 Microsoft SQL Server

7. Microsoft SQL Server Design...............................................................40

7.1. Instant File Initialization (IFI)........................................................................................40
7.2. SQL Server Memory Configuration..............................................................................41
7.3. SQL Server Performance Tuning................................................................................ 43
7.4. SQL Server Soft-NUMA............................................................................................... 44
7.5. SQL Server Compression Configuration..................................................................... 44
7.6. Lock Pages in Memory and Large Pages................................................................... 44
7.7. Windows Guest Power Policy......................................................................................45
7.8. High Availability Design............................................................................................... 45
7.9. Backup and Disaster Recovery Design....................................................................... 47
7.10. SQL Server Maintenance Scripts.............................................................................. 47

8. Best Practices Checklist.......................................................................49

8.1. General.........................................................................................................................49
8.2. Drive Configuration...................................................................................................... 49
8.3. SQL Server Data Files.................................................................................................49
8.4. SQL Server Log Files.................................................................................................. 50
8.5. TempDB........................................................................................................................50
8.6. VMware........................................................................................................................ 50
8.7. RAM..............................................................................................................................51
8.8. vCPUs.......................................................................................................................... 51
8.9. Networking....................................................................................................................51
8.10. Power Policy.............................................................................................................. 51
8.11. SQL Server Instance Scaling.....................................................................................52
8.12. High Availability.......................................................................................................... 52
8.13. Monitoring...................................................................................................................52
8.14. Manageability............................................................................................................. 52
8.15. SQL Server Compression.......................................................................................... 52
8.16. Nutanix Recommendations........................................................................................ 53
8.17. Backup and Disaster Recovery................................................................................. 53
8.18. SQL Server Maintenance...........................................................................................53

9. Conclusion..............................................................................................54

Appendix..........................................................................................................................55
Resources............................................................................................................................55
About the Authors............................................................................................................... 55
About Nutanix...................................................................................................................... 56

4
 Microsoft SQL Server

List of Figures................................................................................................................ 57

List of Tables.................................................................................................................. 59

5
 Microsoft SQL Server

1. Executive Summary
This document makes recommendations for designing, optimizing, and scaling Microsoft SQL
Server deployments on the Nutanix Enterprise Cloud. Historically, it has been a challenge to
virtualize SQL Server because of the high cost of traditional virtualization stacks and the impact
that a SAN-based architecture can have on performance. Businesses and their IT departments
have constantly fought to balance cost, operational simplicity, and consistent predictable
performance.
The Nutanix Enterprise Cloud removes many of these challenges and makes virtualizing a
business-critical application such as SQL Server much easier. The Acropolis Distributed Storage
Fabric (DSF) is a software-defined solution that provides all the features one typically expects in
an enterprise SAN, without a SAN’s physical limitations and bottlenecks. SQL Server particularly
benefits from the following DSF features:
• Localized I/O and the use of flash for index and key database files to lower operation latency.
• A highly distributed approach that can handle both random and sequential workloads.
• The ability to add new nodes and scale the infrastructure without system downtime or
performance impact.
• Nutanix data protection and disaster recovery workflows that simplify backup operations and
business continuity processes.
Nutanix lets you run both Microsoft SQL Server and other VM workloads simultaneously on
the same platform. Density for SQL Server deployments is driven by the database’s CPU and
storage requirements. To take full advantage of the system’s performance and capabilities,
validated testing shows that it is better to scale out and increase the number of SQL Server VMs
on the Nutanix platform than to scale up individual SQL Server instances. The Nutanix platform
handles SQL Server’s demanding throughput and transaction requirements with localized I/O,
server-attached flash, and distributed data protection capabilities.

1. Executive Summary | 6
 Microsoft SQL Server

2. Introduction

2.1. Audience
This best practice document is part of the Nutanix Solutions Library. We wrote it for those
architecting, designing, managing, and supporting Nutanix infrastructures. Readers should
already be familiar with a hypervisor (VMware vSphere, Microsoft Hyper-V, or the native Nutanix
hypervisor, AHV), Microsoft SQL Server, and Nutanix.
The document addresses key items for each role, enabling a successful design, implementation,
and transition to operation. Most of the recommendations apply equally to all currently supported
versions of Microsoft SQL Server. We call out differences between versions as needed.

2.2. Purpose
This document covers the following subject areas:
• Overview of the Nutanix solution.
• The benefits of running Microsoft SQL Server on Nutanix.
• Overview of high-level SQL Server best practices for Nutanix.
• Design and configuration considerations when architecting a SQL Server solution on Nutanix.
• Virtualization optimizations for VMware ESXi, Microsoft Hyper-V, and Nutanix AHV.

Table 1: Document Version History

Version
Published Notes
Number
1.0 April 2015 Original publication.
Updated for SQL Server 2016 and added discussion
2.0 September 2016
of Nutanix Volumes.
2.1 May 2017 Updated for all-flash platform availability.
3.0 August 2019 Major updates throughout.

2. Introduction | 7
 Microsoft SQL Server

3. Nutanix Enterprise Cloud Overview

Nutanix delivers a web-scale, hyperconverged infrastructure solution purpose-built for
virtualization and cloud environments. This solution brings the scale, resilience, and economic
benefits of web-scale architecture to the enterprise through the Nutanix Enterprise Cloud
Platform, which combines three product families—Nutanix Acropolis, Nutanix Prism, and Nutanix
Calm.
Attributes of this Enterprise Cloud OS include:
• Optimized for storage and compute resources.
• Machine learning to plan for and adapt to changing conditions automatically.
• Self-healing to tolerate and adjust to component failures.
• API-based automation and rich analytics.
• Simplified one-click upgrade.
• Native file services for user and application data.
• Native backup and disaster recovery solutions.
• Powerful and feature-rich virtualization.
• Flexible software-defined networking for visualization, automation, and security.
• Cloud automation and life cycle management.
Nutanix Acropolis provides data services and can be broken down into three foundational
components: the Distributed Storage Fabric (DSF), the App Mobility Fabric (AMF), and AHV.
Prism furnishes one-click infrastructure management for virtual environments running on
Acropolis. Acropolis is hypervisor agnostic, supporting two third-party hypervisors—ESXi and
Hyper-V—in addition to the native Nutanix hypervisor, AHV.

Figure 1: Nutanix Enterprise Cloud

3. Nutanix Enterprise Cloud Overview | 8

 Microsoft SQL Server

3.1. Nutanix Acropolis Architecture

Acropolis does not rely on traditional SAN or NAS storage or expensive storage network
interconnects. It combines highly dense storage and server compute (CPU and RAM) into a
single platform building block. Each building block delivers a unified, scale-out, shared-nothing
architecture with no single points of failure.
The Nutanix solution requires no SAN constructs, such as LUNs, RAID groups, or expensive
storage switches. All storage management is VM-centric, and I/O is optimized at the VM virtual
disk level. The software solution runs on nodes from a variety of manufacturers that are either
all-flash for optimal performance, or a hybrid combination of SSD and HDD that provides a
combination of performance and additional capacity. The DSF automatically tiers data across the
cluster to different classes of storage devices using intelligent data placement algorithms. For
best performance, algorithms make sure the most frequently used data is available in memory or
in flash on the node local to the VM.
To learn more about the Nutanix Enterprise Cloud, please visit the Nutanix Bible and
Nutanix.com.

3.2. Nutanix Era

Nutanix Era makes Nutanix Enterprise Cloud the ideal platform for running databases. Nutanix
Era is a software suite that automates and simplifies database administration, bringing one-click
simplicity and invisible operations to database provisioning and life cycle management (LCM).
With one-click database provisioning and copy data management (CDM) as its first services,
Nutanix Era enables DBAs to provision, clone, and refresh their databases to any point in time.
The API-first Nutanix Era architecture can easily integrate with your preferred self-service tools,
and every operation has a unique ID and is fully visible for auditing.
For more information, read our Nutanix Era solution brief.

3. Nutanix Enterprise Cloud Overview | 9

 Microsoft SQL Server

4. Understanding Microsoft SQL Server

4.1. SQL Server Overview

Microsoft SQL Server is a relational database management system (RDBMS) that supports
a wide variety of business intelligence, analytics applications, and transaction processing in
corporate IT environments. SQL Server is one of the most widely used database platforms in the
world, with organizations typically deploying from dozens to hundreds of instances across the
enterprise.

Figure 2: Microsoft SQL Server Architecture

The core component of Microsoft SQL Server is the SQL Server Database Engine, which
controls data storage, processing, and security. It includes a relational engine that processes
commands and queries and a storage engine that manages database files, tables, pages,

4. Understanding Microsoft SQL Server | 10

 Microsoft SQL Server

indexes, data buffers, and transactions. The Database Engine also creates and manages stored
procedures, triggers, views, and other database objects.
The SQL Server Operating System (SQLOS) that underlies the Database Engine handles
lower-level functions such as memory and I/O management, job scheduling, and locking data to
avoid conflicting updates. A network interface layer sits above the Database Engine and uses
Microsoft’s Tabular Data Stream protocol to facilitate request and response interactions with
database servers. At the user level, SQL Server DBAs and developers write T-SQL statements to
build and modify database structures, manipulate data, implement security protections, and back
up databases, among other tasks.

4.2. SQL Server Workload Types

When designing a new SQL Server database solution, it’s important to understand the different
SQL Server workload types. Each workload type has its own distinct characteristics and
requirements that feed into how the overall solution is delivered.

Online Transaction Processing (OLTP)

OLTP is a data-modeling approach used to facilitate and manage typical business applications.
Most applications are OLTP-based. Pure OLTP applications tend to have large numbers of small
queries and usually a 70:30 split between reads and writes. Sustained CPU usage during work
hours and high sensitivity to latency are common characteristics for OLTP workloads. Processors
with faster base clock speeds and faster turbo speeds tend to perform better with OLTP queries.
These workloads have more I/O operations per second than an equivalent data warehouse. With
a single OLTP database, administrators see mostly sequential write activity to the transaction log
file and more random write activity to the data files.

Online Analytic Processing (OLAP)

OLAP is an approach to answering multidimensional queries that was developed for
Management Information Systems (MIS) and Decision Support Systems (DSS). OLAP workloads
have several different components, and the common workflow reads data from multiple sources,
aggregates this data, processes the data, then returns it to customers, quite often in the shape
of a report or dashboard. OLAP workloads are characterized by their use of a large sequential
I/O profile, which means these workloads are geared toward throughput rather than latency.
Processors with higher physical core counts are very useful for OLAP workloads, as is having
a large amount of memory to process larger datasets. OLAP workloads tend to have a lot of
random I/O, so flash-based storage for the cube files can be very beneficial.

Decision Support Systems (DSS)

Decision Support Systems are often used in conjunction with OLAP to access and manipulate
data and support information-driven decision making in an organization. DSS workloads tend to

4. Understanding Microsoft SQL Server | 11

 Microsoft SQL Server

have a few longer-running queries that are resource-intensive (CPU, memory, I/O) and generally
occur during month-, quarter-, or year-end. DSS queries favor read over write, so it is important
for the Nutanix platform hosting this workload type to have nodes sized to allow 100 percent
of the dataset to reside in local flash. With Nutanix data locality, this sizing ensures that the
system can provide that data as quickly as possible. This advantage places a premium on having
processors with higher physical core counts.

Batch or ETL
These workload types tend to be write-intensive, run during off-peak hours, tolerate contention
better than OLTP workloads, and sometimes consume additional bandwidth. Organizations often
run batch workloads at the end of the business day to generate reports about transactions that
occurred during that day. We can break batch or ETL workloads down into three distinct phases:
1. Extract: The system contacts multiple sources and obtains data from these sources.
2. Transform: The system performs an action (or actions) on the obtained data to prepare it for
loading into a target system.
3. Load: The data loads into the target system (often a data warehouse).

4.3. SQL Server Licensing

It is the customer’s responsibility to ensure compliance with Microsoft licensing requirements
when deploying SQL Server in a virtualized environment. There are some differences between
licensing models depending on the SQL Server edition in use. It is important to deploy and
maintain every solution in a supported fashion. Be sure that your specific combination of Nutanix
AOS version, hypervisor type and version, and guest OS is supported. Please consult the
appropriate licensing guide for your supported SQL Server edition.

4. Understanding Microsoft SQL Server | 12

 Microsoft SQL Server

5. Physical Hardware Design

For organizations to successfully deploy SQL Server solutions, it is important to start with a solid
foundation for the database and applications. We cover several considerations related to the
platform’s physical hardware in this section.
When you choose Nutanix, one of the key benefits is that the physical hardware (a “node” in
Nutanix terminology) either arrives preconfigured from the factory (NX and OEM appliances) or
can be automatically configured using the Nutanix Foundation process. When using third-party
hardware from the Nutanix compatibility list (see the earlier Supporting SQL Server on Nutanix
section), you may need to validate these physical hardware configurations and settings.

5.1. NUMA Architecture and Best Practice

Nonuniform memory access (NUMA) is a computer memory design where the memory access
time depends on the memory location relative to the processor. With NUMA, a processor can
access its own local memory faster than nonlocal memory (memory local to another processor or
memory shared between processors). Because SQL Server uses a lot of memory and is aware
of the NUMA topology of the underlying host, it is important to ensure that CPU and physical
memory are properly installed and configured.
The following diagram shows a logical view of NUMA topology on a standard Intel-based dual-
socket system.

Figure 3: NUMA Architecture

5. Physical Hardware Design | 13

 Microsoft SQL Server

A NUMA topology has multiple nodes that are localized structures making up the physical host.
Each NUMA node has its own memory controllers and bus between the CPU and actual memory.
If a NUMA node wants to access nonlocal or remote memory, access times can be many times
slower than local memory access. Because SQL Server is NUMA-aware, it does most of its work
on a local node, preventing this remote memory access issue.

NUMA in a Virtualized Environment

Virtualization allows organizations to create isolated units of compute (commonly referred to as
VMs) that can vary in size and dimensions and do not have to align with the underlying physical
server.
At the hypervisor layer that provides this compute environment for VMs, try to map the NUMA
topology of the physical host to the guest OS and compatible applications, including SQL Server.
Because the three hypervisors supported with Nutanix (VMware ESXi, Microsoft Hyper-V, and
Nutanix AHV) are all NUMA compatible, they can present a virtual NUMA (or vNUMA) topology to
the VM for NUMA-aware applications such as SQL Server to consume.

5.2. Understanding RAM Configuration

The physical memory of a server is critical in any virtualized environment, but particularly so for
SQL Server, which requires a careful balance between memory capacity and memory speed.
The number of memory channels available to a system depends on the generation of its CPUs.
Currently, the latest Intel CPU architecture (known as Xeon or Skylake) includes six memory
channels per CPU, each with up to two DIMMs per channel (DPC), supporting a maximum of 12
DIMMs per CPU or 24 DIMMs per server (for a two-socket server). The previous generation (in
Nutanix nodes, the Intel E5 Broadwell processor family) includes four memory channels per CPU,
each with three DPC, supporting a maximum of 12 DIMMs per CPU.

5. Physical Hardware Design | 14

 Microsoft SQL Server

Figure 4: Intel Memory Architecture

To reduce stress on the memory controllers and CPUs, ensure that memory is in a Balanced
configuration—where DIMMs across all memory channels and CPUs are populated identically.
Balanced memory configurations enable optimal interleaving across all memory channels,
maximizing memory bandwidth. To further optimize bandwidth, configure both memory
controllers on the same physical processor socket to match. For the best system-level memory
performance, ensure that each physical processor socket has the same physical memory
capacity.
If DIMMS with different memory capacities populate the memory channels attached to a memory
controller, or if different numbers of DIMMs with identical capacity populate the memory channels,
the memory controller must create multiple interleaved sets. Managing multiple interleaved sets
creates overhead for the memory controller, which in turn reduces memory bandwidth.
For example, the following image shows the relative bandwidth difference between Skylake and
Broadwell systems, each configured with six DIMMs per CPU.

5. Physical Hardware Design | 15

 Microsoft SQL Server

Figure 5: Example DIMM Population Between Skylake and Broadwell

Populating DIMMs across all channels delivers an optimized memory configuration. When
following this approach with Skylake CPUs, populate memory slots in sets of three per memory
controller. If you are populating a server that has two CPUs with 32 GB DIMMs, the six DIMMs
across all channels per CPU deliver a total host capacity of 384 GB, providing the highest level
of performance for that server. For a Broadwell system, a more optimized configuration has
either four or eight DIMMs (as opposed to the six in Skylake). If you are populating a two-socket
system with the same 32 GB DIMMs, eight DIMMs across all channels per CPU deliver a total
host capacity of 512 GB, which is a higher performance configuration.

Note: Each generation of CPUs may change the way it uses memory, so it’s
important to always check the documentation from the hardware vendor to validate
the configuration of the components.

5. Physical Hardware Design | 16

 Microsoft SQL Server

5.3. BIOS or UEFI Firmware

To ensure that all components operate optimally and in a supported manner, Nutanix
recommends always keeping system firmware up to date. As described earlier, Nutanix LCM
can assist administrators in keeping firmware up to date across supported Nutanix and OEM
platforms.
The Nutanix node installation process applies appropriate BIOS or UEFI settings. Third-party
hardware platforms may need some additional configuration checks. As a best practice for
Nutanix and SQL Server, ensure that NUMA and hyperthreading are enabled, along with CPU
features such as Intel VMX and VT-x. Validate that all CPU C-states are disabled, as these
power management policies can impact CPU efficiency and memory latency, resulting in poor
application and VM performance.

5.4. Power Policy Settings

In addition to disabling the C-states at the BIOS or UEFI level, Nutanix recommends setting the
host power management policy to High Performance for critical applications. This setting only
applies to VMware ESXi and Microsoft Hyper-V; Nutanix AHV enables the high-performance
mode automatically.
In ESXi, select the host and navigate to the Configure tab. Select Power Management and click
High Performance.
In Hyper-V, open the Windows Control Panel and navigate to Power Options, then select High
Performance. You can apply this setting globally using Group Policy.

5.5. Nutanix CVM Considerations

Every host in a Nutanix cluster has a CVM that consumes some of the host’s CPU and memory
to provide all the Nutanix services. The CVM can’t live-migrate to other hosts, as the physical
drives pass through to the CVM using the host hypervisor’s PCI passthrough capability.

5. Physical Hardware Design | 17

 Microsoft SQL Server

Figure 6: Nutanix CVMs in a Cluster

The size of the CVM varies depending on the deployment; however, the typical size is 8 vCPUs
and 32 GB of memory. During initial cluster deployment, Nutanix Foundation automatically
configures the vCPU and memory assigned to the CVM. You should consult your Nutanix
partner and Nutanix Support if you need to make any changes to these values to ensure that the
configuration is support.
vCPUs are assigned to the CVM and not necessarily consumed. If the CVM has a low-end
workload, it uses the CPU cycles it needs. If the workload becomes more demanding, the CVM
uses more CPU cycles. Think of the number of CPUs you assign as the maximum number of
CPUs the CVM can use, not the number of CPUs immediately consumed.
When you size any Nutanix cluster, ensure that there are enough resources in the cluster
(especially failover capacity) to handle situations where a node fails or goes down for
maintenance. Sufficient resources are even more important when you’re sizing a solution to host
a critical workload like Microsoft SQL Server.
From Nutanix AOS 5.11 onward, AHV clusters can include a new type of node that does not
run a local CVM. These compute-only (CO) nodes are for very specific use cases, so engage
with your Nutanix partner or Nutanix account manager to see if compute-only is advantageous
for your workload. Because standard nodes (running CVMs) provide storage from the DSF, CO
nodes cannot benefit from data locality.

5. Physical Hardware Design | 18

 Microsoft SQL Server

Figure 7: Nutanix AHV Cluster with HCI and Compute Nodes

Note: CO nodes are only available when running Nutanix AHV as the hypervisor.

5. Physical Hardware Design | 19

 Microsoft SQL Server

6. Virtual Hardware Design

6.1. CPU Definitions

In a virtualized environment, many different components make up a CPU for an application such
as SQL Server to consume. The following diagram shows the physical and logical constructs of a
CPU.

Figure 8: CPU Terminology

• CPU package
The CPU package is the physical device that contains all the components and cores and sits
on the motherboard. Many different types of CPU packages are available, ranging from low-
core, high frequency models to high-core, low-frequency models. The right balance between
core and frequency depends on the requirements of the SQL Server workload, considering
factors such as the type of workload, licensing, or query (single-threaded or multithreaded).
There are typically two or more CPU packages in a physical host.
• Core

6. Virtual Hardware Design | 20

 Microsoft SQL Server

A CPU package typically has two or more cores. A core is a physical subsection of a
processing chip and contains one interface for memory and peripherals.
• Logical CPU (LCPU)
The LCPU construct presented to the host operating system is a duplication of the core
that allows it to run multiple threads concurrently using hyperthreading (covered in the next
section). Logical CPUs do not have characteristics or performance identical to the physical
core, so do not consider an LCPU as a core in its own right. For example, if a CPU package
has 10 cores, enabling hyperthreading presents 20 logical CPUs to the guest OS.
• Arithmetic logic unit (ALU)
The ALU is the heart of the CPU, responsible for performing mathematical operations on
binary numbers. Typically, the multiple local CPUs on the core share one ALU.
• Hypervisor
In a virtualized environment, the hypervisor is a piece of software that sits between the
hardware and one or more guest operating systems. Each of these guest operating systems
can run its own programs, as the hypervisor presents to it the host hardware’s processor,
memory, and resources.
• vSocket
Each VM running on the hypervisor has a vSocket construct. vCPUs virtually “plug in” to
a vSocket, which helps present the vNUMA topology to the guest OS. There can be many
vSockets per guest VM, or just one.
• vCPU
Guest VMs on the hypervisor use vCPUs as the processing unit and map them to available
LCPUs through the hypervisor’s CPU scheduler. When determining how many vCPUs to
assign to a SQL Server, always size assuming 1 vCPU = 1 physical core. For example, if
the physical host contains a single 10-core CPU package, do not assigned more than 10
vCPUs to the SQL Server. When considering how much to oversubscribe vCPUs in the virtual
environment, use the ratio of vCPU to cores, not logical CPUs.

6.2. Hyperthreading Technology

Hyperthreading is Intel’s proprietary simultaneous multithreading (SMT) implementation used
to improve computing parallelization. Hyperthreading’s main function is to deliver two distinct
pipelines into a single processor execution core. From the perspective of the hypervisor running
on the physical host, enabling hyperthreading makes twice as many processors available.

6. Virtual Hardware Design | 21

 Microsoft SQL Server

In theory, because we ideally want to use all available processor cycles, we should use
hyperthreading all the time. However, in practice, using this technology requires some additional
consideration. There are two key issues when assuming that hyperthreading simply doubles the
amount of CPU on a system:
1. How the hypervisor schedules threads to the pCPU.
2. The length of time a unit of work stays on a processor without interruption.
Because a processor is still a single entity, appropriately scheduling processes from guest VMs
is critically important. For example, the system should not schedule two threads from the same
process on the same physical core. In such a case, each thread takes turns stopping the other
to change the core’s architectural state, with a negative impact on performance. SQL Server
complicates this constraint because it schedules its own threads; depending on the combination
of SQL Server and Windows OS versions in use, SQL Server may not be aware of the distinction
between physical cores or know how to handle this distinction properly. There is also a hypervisor
CPU scheduling function to keep in mind, which is abstracted from the guest OS altogether.
Because of this complexity, be sure to size for physical cores and not hyperthreaded cores when
sizing SQL Server in a virtual environment. For mission-critical deployments, start with no vCPU
oversubscription. SQL Server deployments—especially OLTP workloads—benefit from this
approach.

6.3. CPU Hot-Add

VMware ESXi supports the ability to dynamically add CPU resources to running VMs. Microsoft
Hyper-V and Nutanix AHV do not support this capability. Although the hot-add function may seem
attractive, we strongly recommend disabling it. When CPU hot-add is enabled, vNUMA for the
guest VM is disabled, which has serious performance implications for both CPU scheduling and
accessing physical memory.

6.4. CPU Ready Time

One of the key statistics that administrators use to determine overall health in a virtualized
environment is CPU ready time. This metric tracks the amount of time a VM is ready to process
a thread but is instead waiting for the hypervisor scheduler for access to a physical processor.
When the guest VM (and thus SQL Server) is in this waiting state, no operations occur and the
database’s overall processing throughput drops.
Host vCPU oversubscription is the main reason administrators might see a high CPU ready time
—if there is a high ratio of vCPUs to available physical cores and the guest VMs need to use
CPU, then the hypervisor scheduler is under pressure to allot time to the guest’s vCPU. The only
effective way to avoid this contention is to right-size SQL Server VMs to use what they need, not
some larger amount.

6. Virtual Hardware Design | 22

 Microsoft SQL Server

As a rule, if CPU ready time is over 5 percent for a given VM, there is cause for concern.
Always ensure that the size of the physical CPU and the guest VM is in line with SQL Server
requirements to avoid this problem.

6.5. Memory Configuration

Virtual Memory Overview
In addition to providing CPU time for guest VMs, the hypervisor also virtualizes VM memory to
isolate workloads from each other and to provide a contiguous, zero-based memory space for
each guest operating system. Each supported hypervisor uses different techniques to manage
memory, but the overall concept is the same.
The following diagram demonstrates a conceptual overview of memory management in a virtual
environment.

Figure 9: Virtual Memory Overview

One of the key differences between Nutanix AHV and VMware ESXi or Microsoft Hyper-V is that
AHV does not support swapping VM memory to disk. With ESXi, it is possible to assign more

6. Virtual Hardware Design | 23

 Microsoft SQL Server

memory to all VMs on a host than there is physical host memory. This overcommitment can be
dangerous, as physical RAM is much faster than even the fastest storage available on Nutanix.
Similarly, with Hyper-V, technologies such as Dynamic Memory and Smart Paging allow the
system to share memory allocation between all guest VMs with physical disks as an overflow.

Memory Reservations
Because memory is the most important resource for SQL Server, do not oversubscribe memory
for this application (or for any business-critical applications). As mentioned earlier, Nutanix
AHV does not support memory oversubscription, so all memory is effectively reserved for all
VMs. When using VMware ESXi or Microsoft Hyper-V, ensure that the host memory is always
greater than the sum of all VMs. If there is memory contention on the hypervisor, guest VMs
could start to swap memory space to disk, which has a significant negative impact on SQL Server
performance.
If you are deploying SQL Server in an environment where oversubscription may occur, Nutanix
recommends reserving 100 percent of all SQL Server VM memory to ensure a consistent level
of performance. This reservation prevents the SQL Server VMs from swapping virtual memory to
disk; however, other VMs that don’t have this reservation may swap.

Memory Hot-Plug and Dynamic Memory

VMware ESXi and Microsoft Hyper-V offer built-in memory management techniques that allow
you to add memory to a running VM without needing to reboot. VMware has Memory Hot-Plug,
and Microsoft has Dynamic Memory.

ESXi Memory Hot-Plug

Memory hot-plug allows an administrator to add memory to ESXi VMs with no downtime. With
vSphere 6.5 and later, enabling memory hot-plug (which is not a default setting) adds memory
evenly to all vNUMA nodes, preserving the benefits of vNUMA topology for SQL Server.
It can be beneficial to enable this feature for SQL Server VMs, as you can add more memory to
the SQL Server VMs without restarting them; however, you do need to restart the SQL Server
instance (service). However, pay close attention to the total VM memory allocation, as it is
best to fit a VM within the memory footprint of one NUMA node. This arrangement provides
optimal memory performance across the entire address space. Also, be aware of the total
memory allocated to all VMs so you don’t oversubscribe memory on the host and potentially start
swapping to disk.
Once you have increased the memory on VMs where hot-plug is enabled, remember to adjust
the SQL Server maximum memory setting outlined in the later section SQL Server Memory
Configuration. Adjusting this value does not require a restart of the SQL Server service.

6. Virtual Hardware Design | 24

 Microsoft SQL Server

Hyper-V Dynamic Memory

Instead of allocating a fixed amount of memory to each VM, with the potential for overflow to disk
once physical host memory is exhausted, Dynamic Memory allows the system to see RAM as
a shared resource that it can dynamically and automatically divide between the running VMs.
Hyper-V does not overallocate memory on hosts as VMware ESXi does. With Dynamic Memory
enabled, Hyper-V can dynamically adjust VM memory in response to changes in the workload
of the running VMs. This capability allows the system to distribute memory resources more
efficiently between active VMs.
As with VMware’s CPU hot-add, enabling Dynamic Memory in the settings of a VM disables
vNUMA. If memory is split between NUMA nodes, there can be up to a 30 percent drop in
performance. Although enterprise editions of SQL Server can dynamically add or remove
memory, failing to maintain the vNUMA topology can lead to inconsistent performance. For this
reason, Nutanix recommends against using Dynamic Memory for SQL Server workloads.

6.6. Network Configuration

Networking Overview
Along with compute resources such as CPU and memory, the network is also a critical piece of
any successful SQL Server deployment. Without a robust, fully redundant network, access to
databases and supporting applications can be slow, intermittent, or even unavailable.

6. Virtual Hardware Design | 25

 Microsoft SQL Server

Figure 10: Virtual Networking Overview

Although each hypervisor implements virtual networking constructs slightly differently, the basic
concepts remain the same.

Networking Best Practices

Virtual network design is not specific to SQL Server. The following are generic best practices for
designing virtual networks:
• Always use redundant physical network connections to mitigate any failure of either a physical
switch or physical NIC interface on the host.
• Always size network bandwidth to ensure that the network can maintain the same level of
bandwidth during a component failure. For example, if the physical servers have two 10 GB
interfaces, do not configure teaming policies to provide 20 GB of bandwidth to the VMs. If the

6. Virtual Hardware Design | 26

 Microsoft SQL Server

VMs expect 20 GBps of bandwidth and one of the links fails, performance then drops by 50
percent.
• Consider using hypervisor-based network I/O control to ensure that transient high-throughput
operations such as vMotion or live migration do not negatively impact critical traffic.
• For VMware-based environments, consider using the VMware Distributed Switch. This
construct provides many benefits in a VMware environment, including the ability to use
technologies such as LACP and maintain TCP state during vMotion.
For more detailed information about networking in a Nutanix environment for supported
hypervisors, please refer to the following resources:
• Nutanix AHV Networking best practice guide
• vSphere Networking with Nutanix best practice guide
• Hyper-V 2016 Networking with Nutanix best practice guide

Use of Jumbo Frames

In an Ethernet network, the standard size of a network packet (referred to as the maximum
transmission size or MTU) is 1,500 bytes. A jumbo frame is a network packet greater than
the standard 1,500 bytes that can also carry up to 9,000 bytes in one frame. Nutanix’s
recommendation is to keep the default setting (1,500-byte MTU) and only change to jumbo
frames after testing carefully to ensure that the performance is indeed improved. Jumbo frames
can be useful in situations where you’re using Nutanix Volumes.

6. Virtual Hardware Design | 27

 Microsoft SQL Server

Figure 11: Jumbo Frames

Although larger Ethernet frame sizes can be beneficial for transmitting data across the network,
one of the key drawbacks to using jumbo frames is that you must configure all network
components in the path—switch ports, NICs, virtual switches, guest operating systems, and even
SQL Server itself—with the appropriate MTU. You must configure any network components you
introduce (like additional hosts or replacement switches that don’t have a default of 9,000 bytes)
with the larger nonstandard MTU size for the network to work.
After configuring the network with a size of 9,000 bytes, Nutanix recommends configuring SQL
Server with a network packet size of 8,192 bytes. This payload, with Ethernet overhead, fits in the
TCP 8,972-byte packet. To modify this setting, navigate to the SQL Server Server Properties
menu, then to the Advanced page.

6. Virtual Hardware Design | 28

 Microsoft SQL Server

Figure 12: SQL Server Network Packet Size

6.7. Storage Configuration

Nutanix Volumes
Nutanix Volumes is a native scale-out block storage solution that provides direct block-level
access via the iSCSI protocol to the Acropolis Distributed Storage Fabric (DSF). It enables
enterprise applications running on external servers to benefit from the hyperconverged Nutanix
architecture. Nutanix Volumes also allows guest VMs on any hypervisor to support clustered file
systems.

6. Virtual Hardware Design | 29

 Microsoft SQL Server

Figure 13: Nutanix Volumes

Nutanix Volumes supports SCSI-3 persistent reservations for the shared storage used by SQL
Server failover cluster instances (FCIs). Nutanix manages storage allocation and assignment
for Volumes through a construct called a volume group (VG). A VG is a collection of virtual disks
(or vDisks) presented as available block devices using the iSCSI protocol. All Nutanix CVMs in
a cluster participate in presenting these VGs, creating a highly redundant and scalable block
storage solution. When you create a Windows failover cluster for SQL Server, you need Nutanix
Volumes because Windows failover clusters require shared storage, but when you deploy SQL
Always On availability groups (AAGs), you don’t need Volumes because AAGs don’t require
shared storage.
Nutanix Volumes seamlessly manages failure events and automatically load balances iSCSI
clients to take advantage of all cluster resources. iSCSI redirection controls target path
management for vDisk load balancing and path resiliency. Instead of configuring host iSCSI client
sessions to connect directly to all CVMs in the Nutanix cluster, Volumes uses the Data Services
IP. This design allows administrators to add Nutanix nodes to the cluster in a nondisruptive
manner without needing to update every iSCSI initiator.
For more details on Nutanix Volumes, please refer to the Nutanix Volumes best practice guide.

6. Virtual Hardware Design | 30

 Microsoft SQL Server

Nutanix Volume Group Load Balancer (AHV Only)

Nutanix AHV makes the consumption of VGs easier than other hypervisors or physical servers.
VGs can connect directly to a VM running on AHV without the need to configure in-guest iSCSI
initiators.

Figure 14: Adding a Volume Group in Nutanix AHV

A volume group directly connected to an AHV VM exhibits the same behavior as a volume group
presented via iSCSI, with one exception: with VGs, vDisks are not load balanced across all
CVMs in the cluster by default. This lack of load balancing is equivalent to normal VMs running
on an AHV host and is not a concern for most workloads. Nutanix recommends starting with
the Volume Groups with Load Balancing (VGLB) feature disabled. Nutanix data locality and a

6. Virtual Hardware Design | 31

 Microsoft SQL Server

correctly sized SSD tier provides optimal performance for most SQL Server workloads without
enabling VGLB.

Figure 15: Nutanix AHV Volume Groups (Not Load Balanced)

There may be instances where VGLB can improve performance, especially with large workloads
where the size of the dataset is greater than the size of the underlying host or where throughput
is higher than what a single node can provide, like OLAP and DSS workloads. Particularly when
physical systems with a small number of drives are hosting VMs and volume groups with a high
number of vDisks, the solution is to use VGLB. This feature is not enabled with VGs by default,
so you must enable it on a per-VM basis.

6. Virtual Hardware Design | 32

 Microsoft SQL Server

Figure 16: Nutanix AHV Volume Groups (Load Balanced)

As illustrated in the previous image, enabling VGLB removes the bottleneck of the local Nutanix
CVM, so all CVMs can participate in presenting primary read and write requests. While this
approach can provide great performance benefits overall, using VGLB adds a small amount
of I/O latency because some data locality is lost when the vDisks shard across the CVMs on
the cluster. In the right use case, this cost is generally worth it, as VGLB gives the system a far
greater amount of available bandwidth and CPU resources to serve I/O.

SQL Server Storage Best Practice

The following sections outline the storage best practices for running SQL Server on Nutanix.

Drive Layout and Configuration

Nutanix is a scale-out platform that distributes disks and I/O across a shared-nothing cluster that
scales in a linear fashion. It is a general best practice to create many smaller disks as opposed
to a few large disks. One key reason for this recommendation is that each vDisk on Nutanix
(regardless of the hypervisor) contains a fixed amount of access to the random write buffer on
the underlying host (called the oplog). The optimal number and size of the virtual disks and their
corresponding SQL Server data or log files depends on the size and activity of the database.
Adding more files and properly aligning them with virtual disks can increase the I/O performance.
In keeping with the scale-out mindset, it is also important to ensure that the different types
of disks (system, data files, TempDB, and logs) are all placed across multiple virtual storage
controllers. This consideration is not relevant for Nutanix AHV, as AHV does not have the

6. Virtual Hardware Design | 33

 Microsoft SQL Server

construct of a virtual storage controller. However, for VMware ESXi and Microsoft Hyper-V,
ensure there are multiple virtual SCSI adapters for SQL Server.
As a starting point for storage, Nutanix recommends the following:
• Separate drives for the OS and SQL Server binaries (and backup or restore drive, if used).
Place these drives together on the same virtual storage controller (for example, controller 0).
• Two drives for database data files, with each drive containing two data files. Place these
drives together on the same virtual storage controller but ensure that it’s a different virtual
storage controller than the one containing the OS and binaries (for example, controller 1).
Read more in the section titled SQL Server Data Files.
• Two TempDB data drives, with one TempDB data file per vCPU across the drives, up to eight
TempDB data files. Place these vDisks on their own virtual storage controller (for example,
controller 2). Read more in the section titled TempDB Sizing and Placement.
• One drive for database log files and another drive for the TempDB log file, with both drives on
their own virtual storage controller (for example, controller 3). Read more in the section titled
SQL Server Log Files.
• Ensure that user databases and system databases are on separate drives.
The following diagram shows the suggested initial drive layout for SQL Server VMs on Nutanix.

6. Virtual Hardware Design | 34

 Microsoft SQL Server

Figure 17: Initial Drive and Controller Layout

In addition to providing performance and scalability, maintaining this separation helps with
manageability, as potential problems are easier to isolate. For example, when you separate
TempDB onto its own disks, you can configure the files to grow and fill the disk without worrying
about space requirements for other files (within certain limits, and filling the disk to 100 percent
capacity still causes issues to TempDB). The more separation you can build into the solution, the
easier it is to correlate potential disk performance issues with specific database files.
In a VMware environment, Nutanix recommends using the VMware paravirtual (PVSCSI)
controller for all data and log drives. For the OS drive, it is simpler to keep the default LSI SAS

6. Virtual Hardware Design | 35

 Microsoft SQL Server

adapter; otherwise, the system must load the PVSCSI drivers to allow the OS to boot, which you
can do when you create the Windows template.
In a Hyper-V environment, if you are using Generation 1 VMs, use SCSI disks for all drives
except the OS. For Generation 2 VMs, use SCSI disks for all drives including the OS.

SQL Server Data Files

To make the most out of a platform build, you need to employ a scale-out approach to the disk
and data files layout. Our recommendation is aligned with what Microsoft suggests, as splitting
the database across multiple files across multiple disks provides better performance compared
with a configuration based on a single data file and single disk. For example, you could split a
SQL Server with four vCPUs hosting a 500 GB database into four 125 GB database files spread
across two vDisks. In this configuration, it is important to grow files equally. For SQL Server 2016
and newer versions, use the following T-SQL command:
ALTER DATABASE (database name)
MODIFY FILEGROUP [PRIMARY] AUTOGROW_ALL_FILES
GO

For versions prior to SQL Server 2016, apply trace flag 1117 at SQL Server startup.

6. Virtual Hardware Design | 36

 Microsoft SQL Server

Figure 18: Applying SQL Server Startup Parameters

To avoid unnecessary complexity, add files to databases only if there is database page I/O latch
contention. To look for contention, monitor the PAGEIOLATCH_XX values and spread the data
across more files as needed. Several other factors, such as memory pressure, can also cause
PAGEIOLATCH_XX latency, so investigate the situation thoroughly before adding files.
Unlike transaction log files, SQL Server accesses data files in parallel, and access patterns can
be highly random. Spreading large databases across several vDisks can improve performance.
It is best to create multiple data files before writing data to the database so that SQL Server’s
proportional fill algorithm spreads data evenly across all data files from the beginning.
You should only use volume managers (dynamic disks, storages spaces) with the OS as last
resort after careful testing. Using multiple data files across multiple disks is the recommended
and default approach. Also, do not use autoshrink on database files, as the resulting
fragmentation can reduce performance.
It is not uncommon to have SQL Server host several small databases, none of which are
particularly I/O intensive. In this case, the design could have only one or two data files per
database. These files could easily fit on one or two disks and not spread over multiple disks. The

6. Virtual Hardware Design | 37

 Microsoft SQL Server

goal with disk and file design is to keep the solution as simple as possible while meeting business
and performance requirements.

SQL Server Log Files

The total size reserved for the database transaction log file should be based on the high-water
mark before the next backup and log truncation time. If the system performs daily backups,
estimate how much log space the database is likely to require in 24 hours. The log file space
requirement depends heavily on the application, so measure it appropriately for your specific
environment.
If the databases have an unusually high number of activities that influence the size of the log
file (inserts, updates, deletes, changes in the recovery model), it can be beneficial to perform
regular backups more often than every 24 hours. Increasing backup frequency on Nutanix does
not require a change in the existing backup strategy or design.
Physical transaction logs are made up of several smaller units called virtual log files (VLFs). VLFs
play a critical part in performing database backups and recovery operations. The number of VLFs
is determined at log creation and whenever there is any file growth. If the VLFs are too small,
database recovery and other operations can be very slow. If the VLFs are too large, backing up
and clearing logs can be slow.
Optimal VLF size is between 256 MB and 512 MB. Even if the logs reach 2 TB in size, they
should not contain more than 10,000 VLFs. To preserve this limit, preset the log file to either
4 GB or 8 GB and grow it by increments of the same starting amount to reach the desired log
size. If you need a 128 GB log file, initially create an 8 GB log file and grow it 15 times in 8 GB
increments. Configure the autogrow value to the same base (4 GB or 8 GB) to maintain the same
VLF size. In SQL Server Management Studio, issue the DBCC LOGINFO (dbName) command to
return the number of VLFs in a database.
Remember that, unlike data files, SQL Server log files are written to in a sequential or “fill
and spill” manner. Therefore, using multiple log files spread across several vDisks is not
advantageous and just makes the solution more complex than necessary. Because SQL Server
only accesses one log file at any given time, there’s no reason to use more than one log file.

TempDB Sizing and Placement

One of the most important elements of maximizing SQL Server performance is optimizing
TempDB. Applications can use TempDB as a scratch space and in most cases should not rely
on a single TempDB data file. If there are fewer than eight vCPUs, the recommendation is to
configure the number of TempDB data files to be equal to the number of assigned vCPUs. If
there are more than eight vCPUs, start with eight TempDB data files and look for contention for
in-memory allocation (PAGELATCH_XX). Scale up the number of TempDB files in increments
of four until the contention has been eliminated. During the installation, SQL Server 2016
automatically allocates TempDB files equal to the number of assigned vCPUs, up to eight.

6. Virtual Hardware Design | 38

 Microsoft SQL Server

The size and autogrowth settings for TempDB data files are also important. SQL Server allocates
space in TempDB data files proportionally based on their size, so it’s important to create
all TempDBs to be the same size. Size them properly up front to accommodate application
requirements; proper sizing is generally between 1 percent and 10 percent of the database
size. During a proof-of-concept deployment, monitor TempDB size and use the high-water mark
as the starting point for the production sizing. If you need more TempDB space in the future,
grow all data files equally. You can rely on autogrowth but autgrowth can grow the data files to
a size beyond the free space available on the disk, so we don’t recommend relying solely on
autogrowth.
Nutanix recommends starting with two TempDB data drives and one TempDB log drive. This
arrangement should suffice for most SQL Server workloads. Use trace flag 1118 to avoid mixed
extent sizes. When using trace flag 1118, SQL Server uses only full extents. SQL Server 2016
automatically enables the behavior of trace flag 1118 for both TempDB and user databases.
There is no downside involved in using this setting for SQL Server versions prior to 2016.

Windows Volume Configuration

When formatting NTFS volumes for SQL Server data and log drives, set the allocation unit size to
64 KB.

Figure 19: NTFS Allocation Unit Size

6. Virtual Hardware Design | 39

 Microsoft SQL Server

7. Microsoft SQL Server Design

7.1. Instant File Initialization (IFI)

When SQL Server data and log files are initialized, the process overwrites any existing data left
on the disk from previously deleted files. Performing the following operations initializes data and
log files:
• Create a database.
• Add data or log files to an existing database.
• Increase the size of an existing file (including autogrow operations).
• Restore a database or file group.
File initialization causes these operations to take longer.
In SQL Server, data files can be initialized instantaneously to avoid zeroing operations. Instant
file initialization reclaims used disk space without filling that space with zeros, allowing for fast
execution of the file operations listed earlier. Instead, disk content is overwritten only as new data
writes to the files. Please be aware that log files cannot be initialized instantaneously, so correctly
sizing the log files is a critical step.
Instant file initialization is enabled when the SQL Server service startup account has been
granted the SE_MANAGE_VOLUME_NAME right. Add the SQL Server service account or the
account used for backup operations to the Perform Volume Maintenance Tasks Windows
security policy. Starting with SQL Server 2016, you can grant instant file initialization permission
to the SQL Server service account during installation.
Nutanix recommends granting this right and enabling IFI. To validate that IFI is enabled, run the
following T-SQL query:
SELECT servicename as Service_Display_Name,service_account as Service_Account,
instant_file_initialization_enabled as Instant_File_Initialization_Enabled FROM
sys.dm_server_services WHERE servicename LIKE 'SQL Server (%'

If IFI is enabled, the query returns the following result.

Figure 20: SQL Server IFI Query Result

7. Microsoft SQL Server Design | 40

 Microsoft SQL Server

7.2. SQL Server Memory Configuration

There are two important memory settings in SQL Server: maximum server memory and minimum
server memory. Adjust these two settings in the Server Properties panel under Memory.

Figure 21: SQL Server Memory Settings

The default setting for maximum server memory is just over 2 TB (specifically, 2,147,483,647
MB). If you don’t change this value, SQL Server tries to consume all the memory in the VM,
which can have a negative impact on both SQL Server and the Windows OS.
As a rule of thumb, leave between 6 and 8 GB of memory for the OS and assign the remaining
memory to SQL Server. If you need a more specific sizing guide, use the following formula to
determine the optimal value for maximum server memory based on the VM configuration.

7. Microsoft SQL Server Design | 41

 Microsoft SQL Server

Figure 22: Calculating SQL Server Maximum Memory

The following table is a quick reference for setting maximum server memory for a 64-bit OS.

Table 2: SQL Server Maximum Memory Settings

VM CPU VM RAM Max Memory

2 8 GB 4,976 MB
2 12 GB 8,976 MB
4 16 GB 12,476 MB
4 24 GB 20,476 MB
4 32 GB 28,376 MB
8 48 GB 42,448 MB

7. Microsoft SQL Server Design | 42

 Microsoft SQL Server

VM CPU VM RAM Max Memory

8 64 GB 57,648 MB
12 96 GB 87,720 MB
12 128 GB 119,720 MB

You must manually reconfigure SQL Server maximum memory any time the memory assigned
to the VM changes. You must increase this parameter to allow SQL Server to take advantage of
memory added to the VM, and you must reduce this parameter to leave enough memory for the
Windows OS if memory is removed from the VM. When using large pages with the Lock pages in
memory local security policy, restart the SQL Server service for the new value to take effect; this
process incurs downtime.

7.3. SQL Server Performance Tuning

Several data points in SQL Server can indicate the performance efficiency of the database
engine. One such data point is the CXPACKET wait type, which is a result of parallel query
execution and indicates that a session is waiting for the synchronization of threads involved in the
parallel process to complete.
If CXPACKET waits are more than 5 percent of the total waits for OLTP workloads, the
environment could have a parallel processing problem. Keep these important considerations in
mind when trying to resolve a high percentage of CXPACKET waits:
• If the query uses too many threads, the overhead of managing the threads can reduce the
performance instead of improving it. To control the number of threads, use the maximum
degree of parallelism or MAXDOP setting. You can define the MAXDOP setting on the
instance, database, and query level. Setting the MAXDOP value to one disables parallel plans
altogether. Setting the MAXDOP value to zero lets SQL Server use all available cores if a
query runs in parallel.
• If small queries run in parallel, they consume more resources than if they ran serially. To
control the greater expense of parallel query execution, you can tune the cost threshold for
parallelism or CTFP setting. This setting adjusts the minimal cost for a query to be executed in
parallel. The cost is a custom SQL Server measure—it’s not a measure of time.
When they see low performance on their database and the CXPACKET wait time is high, many
DBAs decide to set the MAXDOP value to 1 to disable parallelism altogether. Because the
database may have large jobs that could benefit from processing on multiple CPUs, Nutanix does
not recommend this approach. Instead, we recommend increasing the CTFP value from 5 to
approximately 50 to make sure that only large queries run in parallel. Set the MAXDOP according
to Microsoft’s recommendation for the number of cores in the VM’s vNUMA node (no more than
8).

7. Microsoft SQL Server Design | 43

 Microsoft SQL Server

7.4. SQL Server Soft-NUMA

Microsoft introduced the soft-NUMA feature in response to the growing number of cores per CPU
package. Soft-NUMA uses software to partition available CPU resources inside one physical
NUMA node (hence the term “soft-NUMA”).
With SQL Server 2016, whenever the SQL Server Database Engine detects more than eight
physical cores per NUMA node or socket at startup, it automatically creates soft-NUMA nodes by
default. Hyperthreaded processor cores are not differentiated when counting physical cores in a
node.
If using soft-NUMA does not provide a performance benefit for the workload, investigate whether
you can change the vNUMA topology to expose smaller vNUMA to a VM hosting the instance of
SQL Server.

7.5. SQL Server Compression Configuration

Data compression can not only save space but also greatly improve performance. Microsoft
introduced the ability to compress rows or pages in SQL Server 2008 and enhanced it in SQL
Server 2012. Enabling compression can reduce the number of IOPS while allowing more data to
fit in the Nutanix hot tier (SSDs), providing space savings as well as a performance boost. You
can use compression with both OLTP and OLAP workloads, including data warehouse and batch
processing.
Administrators can use compression technologies at either the SQL Server layer or at the
Acropolis DSF layer. The results that compression can deliver vary based on the data in the
database. It could be advantageous to enable compression at both layers if the CPU cost-to-
space savings are worth it. SQL Server and Nutanix use different algorithms to compress; you
can enable Nutanix compression online without any downtime.

7.6. Lock Pages in Memory and Large Pages

The Lock Pages in Memory (LPIM) Windows policy determines which accounts can use the LPIM
process to keep data in physical memory, preventing the system from paging the data to virtual
memory on disk. Locking pages in memory can keep the server responsive when paging memory
to disk occurs.
Trace flag 834 causes SQL Server to use Microsoft Windows large-page allocations for the
memory that is allocated for the buffer pool. With this flag, SQL Server allocates large pages
at startup and keeps them throughout the lifetime of the process. Using large pages typically
improves performance by increasing the efficiency of the translation look-aside buffer (TLB) in the
CPU. You can use large pages if your environment meets the following conditions:

7. Microsoft SQL Server Design | 44

 Microsoft SQL Server

• The server is running SQL Server Enterprise Edition.

• The server has at least 8 GB of physical memory.
• The LPIM privilege is set for the service account.
Consider these factors if you plan to enable large pages:
• Large page allocation can increase SQL Server startup time. This approach changes the
way SQL Server manages memory, allocating all buffer pool memory on startup instead of
dynamically allocating it. Depending on the size of the instance, startup can take significantly
more time.
• Nutanix does not recommend using large pages if you are using the COLUMNSTORE feature.
See the Microsoft support article for more information on this issue.

7.7. Windows Guest Power Policy

In some cases, users may experience degraded overall performance on a Windows Server with
SQL Server when running with the default (balanced) power plan. The issue may occur on any
platform and in both native and virtual environments. The average response time for some tasks
may increase, and CPU-intensive applications may experience performance issues.
Nutanix recommends using the high-performance power plan instead. This change allows the
Windows scheduler to work most efficiently, as there is no dynamic scaling of CPU resources.
Enabling this plan across all SQL Servers ensures a consistent level of execution performance.

7.8. High Availability Design

Microsoft provides a variety of high availability (HA) options to limit potential downtime. Nutanix
recommends Always On availability groups (AAGs) as the preferred HA solution. AAGs are a
shared-nothing model introduced with SQL Server 2012, where SQL Server can synchronously
replicate transactions between nodes and provide very fast failover times. This approach doubles
the amount of storage required when you use two replicas, triples the amount of storage when
you use three replicas, and so on, as each node has a full copy of the database. AAGs are a
different framework than the traditional single-copy failover cluster, which requires shared SCSI
disks. You can use hypervisor HA to help minimize server downtime as well.
If the application does not support the use of AAGs, you can use log shipping as an alternative.
Log shipping is an older technology, but it may have wider application compatibility. AAGs are the
modern and much preferred HA solution for SQL Server. If you can’t use AAGs or log shipping,
Windows SQL failover cluster is fully supported across all hypervisors if you use in-guest iSCSI to
access a shared volume (as required for this type of solution).

7. Microsoft SQL Server Design | 45

 Microsoft SQL Server

AAGs can also serve as a built-in disaster recovery solution, if you replicate an AAG to a second
or third datacenter. Ensure that you have sufficient bandwidth to keep up with the replication
requirements and that latencies are acceptable.
When deploying FCIs or AAGs, use hypervisor antiaffinity rules to enforce placement of the
nodes on different physical hosts. This placement provides resiliency in the case of an unplanned
host failure. When designing your hypervisor clusters, allow for at least n + 1 availability. With this
allowance, VMs can continue with adequate resources, even if the system loses one host.
If using VMware HA, review the restart priority of VMs. You may want to increase the SQL Server
priority to high for production databases. The best priority setting depends on what other services
are running in your cluster.

Always On Failover Cluster Instance (FCI)

AAGs are a great solution and fit nicely in the Nutanix scale-out architecture, but they may not
be a good fit or even possible for certain environments. Some of the limiting factors for adopting
AAGs include:
• Space utilization: A secondary database copy consumes additional storage space. Some
administrators may prefer a single database copy where server HA is the primary use case.
• Synchronous commit performance: Synchronously copying transactions (insert, update,
delete) for AAG replication (in the context of an HA solution) has a performance overhead.
Administrators of latency-sensitive applications may prefer not to incur the additional response
time of waiting for transactions to be committed to multiple SQL Server instances.
• Distributed transactions: Some applications perform distributed transactions across databases
and SQL Server instances. Microsoft doesn’t support distributed transactions with AAGs in
SQL Server versions prior to SQL Server 2016 SP2, so application vendors can’t support
elements of those versions that use distributed transactions where AAGs are present. Please
check Microsoft documentation for the most updated information on this matter.
• Licensing: AAGs require Enterprise Edition for SQL Server 2012 and 2014. Some cost-
conscious organizations may prefer to use Standard Edition for many of their SQL Server
instances but still desire high availability at the SQL Server level. SQL Server 2016 and newer
provide support for basic AAGs with Standard Edition with several limitations.
In these cases, SQL Server Failover Cluster Instances (FCIs) can offer a solution. FCIs have
long been used as a means for HA with SQL Server. FCIs work with all current versions of SQL
Server and rely on shared storage to support the SQL Server instances.
The shared storage used can be block (LUN) based or, starting with SQL Server 2012, SMB
(file) based. In the case of LUN-based shared storage, SCSI-3 persistent reservations arbitrate
ownership of the shared disk resources between nodes. Refer to Microsoft’s documentation for
additional details on Always On FCIs.

7. Microsoft SQL Server Design | 46

 Microsoft SQL Server

Nutanix supports the use of FCIs across ESXi, Hyper-V, and AHV if you use in-guest iSCSI.
Hyper-V environments can also take advantage of the shared virtual hard disk functionality in
Windows Server 2012 R2.

7.9. Backup and Disaster Recovery Design

It is extremely important to meet recovery point objectives (RPOs) and recovery time objectives
(RTOs) in a SQL Server environment. Depending on the business requirements, these targets
may be very stringent. Sizing SQL Server VMs must take RTOs into account. For example,
a monster SQL Server VM takes much longer to restore than a smaller VM. If your backup
subsystem can restore at the rate of 1 TB per hour, and your RTO for a tier-1 SQL Server VM is
one hour, the VM should be less than 1 TB.
The additional time necessary to restore monster VMs is one reason that Nutanix recommends
scaling out SQL Server VMs rather than scaling up. Work with your backup team to establish
backup policies that meet RTOs and RPOs for each database.
As previously discussed, SQL Server AAGs are an excellent built-in disaster recovery solution
for your databases. Ensure that there is adequate bandwidth between the sites to sustain the
replication traffic.

7.10. SQL Server Maintenance Scripts

Maintaining your SQL Server is very important. For a comprehensive set of backup, integrity
check, index, and statistics maintenance scripts check out Ola Hallengren’s scripts.
Another handy script for checking the health of SQL Server is sp_Blitz by Brent Ozar.
Both solutions are free; carefully evaluate them for use in your environment. They can automate
much of the drudgery of maintaining SQL Server databases.

7. Microsoft SQL Server Design | 47

 Microsoft SQL Server

In a scale-out infrastructure, to make use of the architecture, use multiple I/O data streams and
multiple backup streams. Here’s an example script that configures a backup of a SQL Server DB
to write to multiple files.
BACKUP DATABASE [ntnxdb] TO
DISK = N'R:\SQLBACK\ntnxdb-tpcc-SQL2014-1000wh-01.bak',
DISK = N'R:\SQLBACK\ntnxdb-tpcc-SQL2014-1000wh-02.bak',
DISK = N'R:\SQLBACK\ntnxdb-tpcc-SQL2014-1000wh-03.bak',
DISK = N'R:\SQLBACK\ntnxdb-tpcc-SQL2014-1000wh-04.bak'
WITH DESCRIPTION = N'ntnxdb',
NOFORMAT, NOINIT, NAME = N'ntnxdb,
SKIP, NOREWIND, NOUNLOAD, COMPRESSION,
STATS = 5
GO

7. Microsoft SQL Server Design | 48

 Microsoft SQL Server

8. Best Practices Checklist

The following is a consolidated list of all the best practices covered in this document.

8.1. General
• Perform a current state analysis to identify workloads and sizing.
• Spend time up front to architect a solution that meets both current and future needs.
• Design to deliver consistent performance, reliability, and scale.
• Don’t undersize, don’t oversize—right size.
• Start with a proof of concept, then test, optimize, iterate, and scale.

8.2. Drive Configuration

• Distribute databases and log files across multiple vDisks.
• Distribute vDisks across multiple SCSI controllers.
• Use the VMware PVSCSI controller for database and log disks, or for all drives if possible.
• Use 64 KB NTFS allocation for database and log drives.
• Size for at least 20 percent free disk space on all drives.
• Do not use Windows dynamic disks or other in-guest volume managers.

8.3. SQL Server Data Files

• Enable instant file initialization (IFI).
• For each database, use multiple data files: one file per vCPU.
• For very write-intensive databases, distribute database files over four or more vDisks.
• Size all data files in a filegroup equally.
• Enable autogrow in 256 MB or 512 MB increments to start.
• Use trace flag 1117 for equal file autogrowth. With SQL Server 2016 and newer, use the
ALTER DATABASE AUTOGROW_ALL_FILES command instead.
• Do not shrink databases.

8. Best Practices Checklist | 49

 Microsoft SQL Server

8.4. SQL Server Log Files

• Base total size on the high-water mark before the next log truncation period.
• Optimal VLF size is between 256 MB and 512 MB.
• No log file should exceed 10,000 VLFs.
• Configure initial log file size to 4 GB or 8 GB and iterate by the initial amount to reach the
desired size.
• DBCC LOGINFO (“dbName”): The number of rows returned is the number of VLFs.
• One log per database (including TempDB) is sufficient, since SQL server can only use a single
log file at any given time.

8.5. TempDB
• Use multiple TempDB data files, all the same size.
• Use autogrow on TempDB files with caution to avoid situations where files use 100 percent of
the disk that hosts the log files.
• If there are eight or fewer cores, the number of TempDB data files is equal to the number of
cores.
• If there are more than eight cores, start with eight TempDB data files and monitor for
performance.
• Initially size TempDB to be 1 to 10 percent of database size.
• Use trace flag 1118 to avoid mixed extent sizes (full extents only). SQL Server 2016
automatically enables the behavior of trace flag 1118.
• One TempDB log drive should be sufficient for most environments.

8.6. VMware
• Use the VMXNET3 NIC.
• Use the latest VMware VM hardware version.
• Use the PVSCSI controller when possible.
• Remove unneeded hardware (floppy drive, serial port, and so on).
• Do not enable CPU hot-add, as this disables vNUMA.

8. Best Practices Checklist | 50

 Microsoft SQL Server

8.7. RAM
• More RAM can increase SQL Server database read performance.
• Enable large page allocations when using 8 GB of memory or more.
• Do not overcommit RAM at the hypervisor host level.
• Tier-1 databases reserve 100 percent of RAM.
• Configure SQL Server maximum memory per previous section guidance.
• For tier-1 workloads, lock pages in memory.
• Size each VM to fit within a NUMA node’s memory footprint.

8.8. vCPUs
• Do not overallocate vCPUs to VMs.
• For tier-1 databases, minimize or eliminate CPU oversubscription.
• Account for Nutanix CVM core usage.
• If possible, size VMs to fit within one NUMA node.

8.9. Networking
• Use hypervisor network control mechanisms (for example, VMware NIOC).
• Use VMware load-based teaming with the vDS.
• Connect Nutanix nodes with redundant 10 Gbps connections.
• Use multi-NIC vMotion or Live Migration.

8.10. Power Policy

• If using Hyper-V hosts, use the high-performance power option.
• If using VMware hosts, use balanced power mode unless the database is highly latency
sensitive.
• Always set the Power policy to High Performance in the VM guest OS.

8. Best Practices Checklist | 51

 Microsoft SQL Server

8.11. SQL Server Instance Scaling

• To make the most of a scale-out platform, configure the SQL DB to use multiple files across
multiple vDisks.
• We do not recommend scaling up one VM with multiple SQL Server instances.

8.12. High Availability

• Use SQL Server Always On availability groups for SQL Server HA.
• Leave VMware DRS automation at the default level (3).
• Let the hypervisor and DRS manage noisy neighbors and workload spikes.
• Use at least n + 1 for HA configuration.
• Use hypervisor antiaffinity rules to separate FCIs and AAG cluster nodes.
• Use a percentage-based admission control policy for VMware environments.
• Review SQL Server VM restart priorities.

8.13. Monitoring
• Choose an enterprise monitoring solution for all SQL Servers.
• Closely monitor drive space.

8.14. Manageability
• Standardize on SQL Server build and cumulative updates.
• Use standard drive letters or mount points.
• Use VM templates.
• Join the SQL Server to the domain and use Windows authentication.
• Use Windows cluster-aware updating for AAG instances.
• Test patches and roll them out in a staggered manner during maintenance windows.

8.15. SQL Server Compression

• Carefully evaluate using SQL Server compression for all databases.

8. Best Practices Checklist | 52

 Microsoft SQL Server

• Look into using SQL Server compression either at the row or page level, depending on which
one provides the best savings based on the workload. Be aware of licensing.

8.16. Nutanix Recommendations

• Enable compression at the container level.
• Do not use container-level deduplication.
• Only use Nutanix erasure coding (ECX) after carefully examining the workload pattern, since
ECX is best suited to datasets that aren’t written or rewritten often. Please reach out to the
Nutanix account team for further guidance.
• Use the fewest possible containers (one is fine).
• Take database growth into account when calculating sizing estimates.

8.17. Backup and Disaster Recovery

• Establish RTOs and RPOs for each database.
• Validate that the backup system can meet RTOs and RPOs.
• Size SQL Server VMs so they can be restored within RTOs.
• Use AAGs as a built-in disaster recovery solution.

8.18. SQL Server Maintenance

• Use Ola Hallengren’s SQL Server maintenance scripts.
• Use Bren Ozar’s SQL Server sp_Blitz health check script.

8. Best Practices Checklist | 53

 Microsoft SQL Server

9. Conclusion
Microsoft SQL Server deployments are crucial to organizations, as they are used in everything
from departmental databases to business-critical workloads, including ERP, CRM, and BI. At the
same time, enterprises are virtualizing SQL Server to shrink their datacenter footprint, control
costs, and accelerate provisioning. The Nutanix platform provides the ability to:
• Consolidate all types of SQL Server databases and VMs onto a single converged platform
with excellent performance.
• Start small and scale databases as your needs grow.
• Eliminate planned downtime and protect against unplanned issues to deliver continuous
availability of critical databases.
• Reduce operational complexity by leveraging simple, consumer-grade management with
complete insight into application and infrastructure components and performance.
• Keep pace with rapidly growing business needs, without large up-front investments or
disruptive forklift upgrades.

9. Conclusion | 54
 Microsoft SQL Server

Appendix

Resources
SQL Server Licensing
1. Microsoft product licensing for SQL Server

Supporting SQL Server on Nutanix

1. Nutanix third-party hardware compatibility lists
2. Nutanix Compatibility Matrix (including guest OS options supported on Nutanix AHV)
3. VMware Compatibility Guide (including guest OS options supported on VMware ESXi)
4. Hyper-V supported guest OS
5. Nutanix Software Documents (including release notes)
6. Nutanix End of Life Information
7. Microsoft support policy for SQL Server running in a virtualized environment
8. Microsoft Lifecycle Policy (to validate that your versions of SQL Server and Windows Server
are in support)
9. Server Memory Configuration Options

Nutanix Networking
1. Nutanix AHV Networking best practice guide
2. vSphere Networking with Nutanix best practice guide
3. Hyper-V 2016 Networking with Nutanix best practice guide

About the Authors

Bruno Sousa is a Senior Solutions Architect and a Nutanix Platform Expert (NPX-015) in the
Business-Critical Applications Engineering organization of Nutanix. Follow Bruno on Twitter at
@bsousapt.
Chris Jones is a Technical Product Manager in the Platforms team at Nutanix. Follow Chris on
Twitter at @cpjones44.
Michael Webster is the Global Technical Director of Business-Critical Applications Engineering for
Nutanix and a Nutanix Platform Expert (NPX-007). Follow Michael on Twitter at @vcdxnz001.

Appendix | 55
 Microsoft SQL Server

Derek Seaman is a Customer Success Enterprise Architect at Nutanix and a Nutanix Platform
Expert (NPX-014). Follow Derek on Twitter @vDereks.

About Nutanix
Nutanix makes infrastructure invisible, elevating IT to focus on the applications and services that
power their business. The Nutanix Enterprise Cloud OS leverages web-scale engineering and
consumer-grade design to natively converge compute, virtualization, and storage into a resilient,
software-defined solution with rich machine intelligence. The result is predictable performance,
cloud-like infrastructure consumption, robust security, and seamless application mobility for a
broad range of enterprise applications. Learn more at www.nutanix.com or follow us on Twitter
@nutanix.

Appendix | 56
 Microsoft SQL Server

List of Figures
Figure 1: Nutanix Enterprise Cloud................................................................................... 8

Figure 2: Microsoft SQL Server Architecture...................................................................10

Figure 3: NUMA Architecture...........................................................................................13

Figure 4: Intel Memory Architecture................................................................................ 15

Figure 5: Example DIMM Population Between Skylake and Broadwell...........................16

Figure 6: Nutanix CVMs in a Cluster...............................................................................18

Figure 7: Nutanix AHV Cluster with HCI and Compute Nodes........................................19

Figure 8: CPU Terminology..............................................................................................20

Figure 9: Virtual Memory Overview................................................................................. 23

Figure 10: Virtual Networking Overview.......................................................................... 26

Figure 11: Jumbo Frames................................................................................................28

Figure 12: SQL Server Network Packet Size.................................................................. 29

Figure 13: Nutanix Volumes.............................................................................................30

Figure 14: Adding a Volume Group in Nutanix AHV....................................................... 31

Figure 15: Nutanix AHV Volume Groups (Not Load Balanced)....................................... 32

Figure 16: Nutanix AHV Volume Groups (Load Balanced)..............................................33

Figure 17: Initial Drive and Controller Layout.................................................................. 35

Figure 18: Applying SQL Server Startup Parameters......................................................37

Figure 19: NTFS Allocation Unit Size..............................................................................39

Figure 20: SQL Server IFI Query Result......................................................................... 40

Figure 21: SQL Server Memory Settings........................................................................ 41

57
 Microsoft SQL Server

Figure 22: Calculating SQL Server Maximum Memory................................................... 42

58
 Microsoft SQL Server

List of Tables
Table 1: Document Version History................................................................................... 7

Table 2: SQL Server Maximum Memory Settings........................................................... 42

59