Nondistruptive Operations for NetApp

ONTAP 9.0

December 2016 | SL10281 Version 1.3

TABLE OF CONTENTS

1 Introduction...................................................................................................................................... 3

1.1 Nondisruptive Operations.........................................................................................................3

1.2 Lab Objectives........................................................................................................................... 4

1.3 Prerequisites.............................................................................................................................. 4

2 Lab Environment............................................................................................................................. 5

3 Lab Activities................................................................................................................................... 7

3.1 LIF Migration.............................................................................................................................. 7

3.2 Volume Move............................................................................................................................15

3.3 Storage Failover...................................................................................................................... 22

4 Appendix 1..................................................................................................................................... 37

4.1 Accessing the Command Line............................................................................................... 37

4.2 Accessing System Manager................................................................................................... 38

5 References......................................................................................................................................41

6 Version History.............................................................................................................................. 42

Nondistruptive Operations for NetApp ONTAP 9.0

2 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
1 Introduction
®
This lab introduces Nondisruptive Operations (NDO) for ONTAP 9 .
This lab guide utilizes as its starting point a completed instance of the “Basic Concepts for NetApp ONTAP 9.0”
lab, so if you have taken that lab then the configuration of the storage controllers and clients in this lab will be
familiar to you. However, such familiarity is not a requirement, as this lab is easy to follow even if you have not
taken the Basic Concepts lab.

1.1 Nondisruptive Operations

Nondisruptive Operations (NDO) are a set of features in NetApp ONTAP that help maintain continuous data
availability to clients and hosted applications during planned and unplanned maintenance events. Those features
are LIF migration, volume move, storage failover, and aggregate relocation.
LIF migration is the ability to dynamically move logical interfaces from one physical port to another in a cluster,
allowing you to migrate them to higher performing network ports, or take nodes offline for maintenance while
preserving data access. SAN LIFs do not support migrate in normal operation, as iSCSI and Fibre Channel
instead use multipathing and ALUA to protect against network path failure. LIF migration is non-disruptive for
NFS, and for newer SMB protocol versions.
Volume move is the ability to migrate an online volume from one aggregate to another, including to aggregates
on other nodes in the cluster. You can even move volumes to different types and tiers of storage; for example,
moving a volume from SATA to SAS to SSD as it’s performance needs change. Volume moves are non-disruptive
for all protocols, and ONTAP preserves data protection relationships over a move operation. You cannot directly
move a volume from one SVM to another, but you can replicate it to the other SVM using SnapMirror.
Storage Failover involves two nodes configured as a High Availability (HA) pair, with each node having access
to it’s partner's disks and networks. In the event that one of the nodes in the pair fails, the remaining node can
take over the functions of the failed node, providing access to all of it’s SVMs, aggregates, volumes, LUNs,
shares, exports, network identities, and so on, without any loss of data on the cluster, since both nodes mirror
each other’s NVRAM cache. (NVRAM caches acknowledged disk writes while the controller waits for a good
opportunity to actually write them to disk.)
Aggregate Relocate is primarily used to perform controller hardware upgrades of HA controller pairs. It involves
relocating the aggregates of one node in the pair to the other node while each node is undergoing an upgrade to
a new controller model. Aggregate relocate leverages the same capabilities in ONTAP that make volume move
and storage failover possible. Since this lab utilizes simulated storage controllers, this lab guide does not discuss
aggregate relocate any further.
NDO does not completely eliminate disruption in all cases, because some stateful protocols like CIFS have
inherent limitations that make them susceptible to interruption during certain kinds of downtime events. However,
even in many of those cases NDO still helps to minimize the overall extent of the disruption, as most CIFS clients
will attempt to reconnect to storage resources after an NDO event.
The “NDO by Protocol” table provides a summary of these NDO operations, and whether or not they are
disruptive for a given protocol.

Table 1: NDO by Protocol

Nondisruptive NAS SAN

Operation
SMB 3.0
SMB 2.X (Hyper-
SMB 1.0 NFS
and SMB 3.0 V and SQL
Server Only)
Possibly
LIF Migrate Nondisruptive Nondisruptive Nondisruptive N/A
Disruptive

Nondistruptive Operations for NetApp ONTAP 9.0

3 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 Volume Move Nondisruptive Nondisruptive Nondisruptive Nondisruptive Nondisruptive
Aggregate
Disruptive Disruptive Nondisruptive Nondisruptive Nondisruptive
Relocate
Storage
Disruptive Disruptive Nondisruptive Nondisruptive Nondisruptive
Failover

1.2 Lab Objectives

Here is a summary of the exercises available in this lab, along with their Estimated Completion Times (ECT):
• Nondisruptive Operations (ECT = 35 minutes total)
• LUN Migrate (Optional, ECT = 10 minutes)
• Volume Move (Optional, ECT = 10 minutes)
• Storage Failover (Optional, ECT = 15 minutes)
All of these lab sections are independent of the others, meaning that you can choose to complete or not complete
each section without affecting any of the other sections. Additionally, each section offers several choices of the
protocols you can use to generate client traffic during that section’s NDO exercise, and you are welcome to
complete more than one protocol in a section if you wish, but the completion times listed here assume that you
will complete one protocol per section.

1.3 Prerequisites
This lab assumes that you are familiar with the concepts introduced in the Basic Concepts for NetApp ONTAP 9.0
lab.
This lab also assumes that you know how to use PuTTY, and how to launch and log in to System Manager to
manage a cluster. If you are unfamiliar with any of those procedures then please review Appendix 1 of this lab
guide.

Nondistruptive Operations for NetApp ONTAP 9.0

4 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
2 Lab Environment
The following diagram illustrates the environment for this lab.

Figure 2-1:

All of the servers and storage controllers presented in this lab are virtual devices, and the networks that
interconnect them are exclusive to your lab session. While we encourage you to follow the demonstration steps
outlined in this lab guide, you are free to deviate from this guide and experiment with other ONTAP features that
interest you. While the virtual storage controllers (vsims) used in this lab offer nearly all of the same functionality
as physical storage controllers, they are not capable of providing the same performance as a physical controller,
which is why these labs are not suitable for performance testing.
The Lab Host Credentials table provides a list of the servers and storage controller nodes in the lab, along with
their IP address.

Table 2: Lab Host Credentials

Hostname Description IP Address(es) Username Password

Windows 20012R2 Remote
JUMPHOST 192.168.0.5 Demo\Administrator Netapp1!
Access host
RHEL1 Red Hat 6.7 x64 Linux host 192.168.0.61 root Netapp1!
RHEL2 Red Hat 6.7 x64 Linux host 192.168.0.62 root Netapp1!
DC1 Active Directory Server 192.168.0.253 Demo\Administrator Netapp1!
cluster1 ONTAP cluster 192.168.0.101 admin Netapp1!
cluster1-01 ONTAP cluster node 192.168.0.111 admin Netapp1!
cluster1-02 ONTAP cluster node 192.168.0.112 admin Netapp1!

The Preinstalled NetApp Software table lists the NetApp software that is pre-installed on the various hosts in this
lab.

Nondistruptive Operations for NetApp ONTAP 9.0

5 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 Table 3: Preinstalled NetApp Software

Hostname Description
Data ONTAP DSM v4.1 for Windows MPIO, Windows Unified Host Utility Kit
JUMPHOST
v7.0.0, NetApp PowerShell Toolkit v4.2.0
RHEL1, RHEL2 Linux Host Utilities Kit v7.0

Nondistruptive Operations for NetApp ONTAP 9.0

6 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
3 Lab Activities
You can perform all the exercises of the lab using the GUI, with a few CLI command sections included to help you
generate client I/O, and to gather some information that is not readily available using System Manager.
This lab makes use of aggregates, SVMs, volumes, LUNs, LIFs, protocols, and client connections that were
created during a fully completed execution of the Basic Concepts for NetApp ONTAP 9.0 lab.
At the beginning of each of the following sections of this lab guide is a table that provides a list of protocol
scenarios that you can choose to test during that exercise.
Note: Please only choose one scenario option at a time.
When you complete a section, you are welcome to go back and repeat that section using a different scenario if
you wish.

3.1 LIF Migration

In this section you use a NetApp utility named “SIO”, or Simple I/O Load Generator, to start an I/O load on a client
to a volume on svm1 using CIFS or NFS. You will then migrate a LIF supporting that connection to the other node
in the cluster, observing along the way that the I/O load to the volume is not disrupted by the LIF migration.
You can choose either of the two options listed in the LIF Migrate Exercise Choices table for this exercise.

Table 4: LIF Migrate Exercise Choices

Platform Protocol Client SVM Volume File

Linux NFS rhel1 svm1 engineering /svm1/engineering/nfs.sio
Windows CIFS jumphost svm1 engineering S:\engineering\cifs.sio

Note: The table does not contain entries for iSCSI because that protocol utilizes multipathing and ALUA
to protect against path failure, so LIF failover is not applicable.
Please only select one entry at a time for the LIF migration test in order to maintain optimum simulator
performance. Once you complete the test for one entry you are welcome to come back and test another.
Note: The “File” column of the table contains information you will be using later in this lab section.
1. You will be generating I/O load to the volume for your selected entry using the sio command, and you
will run sio from the client listed for your chosen entry from the LIF Migrate Exercise Choices table.
• If the listed client is rhel1, then you will need to open a PuTTY session to rhel1. The PuTTY
launch icon is on the taskbar of jumphost. Log in to rhel1 using the username root and the
password Netapp1!.

Figure 3-1:

If you need further instruction on how to use PuTTY, see Appendix 1.

Nondistruptive Operations for NetApp ONTAP 9.0

7 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 • If the listed client is Jumphost, then you will need to open PowerShell. The PowerShell launch
icon is on the taskbar of Jumphost.

Figure 3-2:

2. In the appropriate client CLI window for your exercise choice, launch sio using the following syntax:
sio 0 0 4k 0 50m 600 2 <File> -create,

where <File> is the “File” value that comes from LIF Migrate Exercise Choices table. This command will
initiate a sequential write operation to <File> for a period of 10 minutes, which should be sufficient time
to complete this LIF Migration exercise.
If you opted to use CIFS for your test, then the sio invocation would look like this:

Windows PowerShell
Copyright (C) 2013 Microsoft Corporation. All rights reserved.
PS C:\Users\Administrator.DEMO> sio 0 0 4k 0 50m 600 2 S:\engineering\cifs.sio -create
Version: 6.39
Read: 0 Rand: 0 BlkSz: 4096 BegnBlk: 0 EndBlk: 12800 Secs: 600 Threads: 2 Devs: 1 S:
\engineering\cifs.sio

The sio command will continue running until the specified duration is reached, or it encounters an error,
in which case it will generate an error message. If the sio command is disrupted by the LIF migration,
then you will see an error message in the sio output.
The remaining screenshots and CLI examples in this section show the steps required to complete the
Windows option in the LIF Migration exercise Choices table. If you chose a different entry you will need
to substitute your entry’s values where appropriate.
3. On the desktop of Jumphost, launch Chrome and log into System Manager. (If you need further
assistance in opening System Manager then see Appendix 1.)

Figure 3-3:

In System Manager, observe the current port assignments for the NAS LIFs for svm1.
4. In the command bar at the top of System Manager, select the Network tab.
5. In the “Network” pane, select the Network Interfaces tab.
6. In the list of interfaces, locate the entries for “svm1_cifs_nfs_lif1” and “svm1_cifs_nfs_lif2”, and note their
current port assignments, which should be cluster1-01:e0c and cluster1-02:e0c, respectively.

Nondistruptive Operations for NetApp ONTAP 9.0

8 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 4

Figure 3-4:

Svm1 uses DNS load balancing for it’s NAS LIFs, so you cannot predict in advance which of those two
LIFs the host running sio will be using to send I/O to the CIFS share or NFS-mounted volume. Since
you’ve already started sio on your client, you will now need to access the clustered Data ONTAP CLI so
you can determine which LIF is handling that traffic.
7. If you do not currently have a PuTTY session open to cluster1, open one now by right-clicking on
the PuTTY icon on the taskbar, and selecting PuTTY from the context menu. Log in to cluster1 with
username admin, and the password Netapp1!.

Figure 3-5:

Nondistruptive Operations for NetApp ONTAP 9.0

9 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 8. In the PuTTY session for cluster1, execute the following command to see current network statistics for
the two NAS LIFs on svm1. The command will take 5 seconds, or so, to generate any output.

cluster1::> statistics lif show -lif svm1_cifs_nfs_lif*

cluster1 : 10/01/2015 19:39:12
*Recv Sent
Recv Data Recv Sent Data Sent Current
LIF Vserver Packet (Bps) Errors Packet (Bps) Errors Port
------------------ ------- ------ ------- ------ ------ ----- ------ -------
svm1_cifs_nfs_lif2 svm1 1170 2461104 0 586 49234 0 e0c
svm1_cifs_nfs_lif1 svm1 0 0 0 0 0 0 e0c
cluster1::>

In the preceding command output, notice that “svm1_cifs_nfs_lif2” is carrying all the traffic, so in this
example this is the LIF that you would want to migrate. This is the LIF that the rest of the instructions in
this section will use, but remember that you will need to substitute the appropriate LIF name from your
lab where needed.
Now go back to System Manager to begin the LIF migration.
9. In the “Network” pane of System Manager, locate in the interface list the LIF you identified in the
preceding step and select it. Make note of the current port assignment for your LIF, as you will need this
information later. The LIF in this example is located on node cluster1-02 port e0c.
10. Right-click on the LIF.
11. In the context menu, select Migrate.

10

11

Figure 3-6:

A Warning dialog opens.

12. You are prompted that this LIF is supporting CIFS, and that migrating the interface may be disruptive
to CIFS connections. Since Jumphost is running Windows 2012 R2, it is using a newer version of SMB
which will not be affected. Click Yes to continue.

Nondistruptive Operations for NetApp ONTAP 9.0

10 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 12

Figure 3-7:

The Warning dialog closes, and is replaced by the “Migrate Interface” window.
It is in this window that you will select the node and port that you want to migrate the LIF to, a decision
you will make based on the LIF’s current node and port location. In the example shown here the LIF
“svm1_cifs_nfs_lif2” was located on node cluster-02 port e0c, in which case you would want to migrate
the LIF to node cluster1-01. Since “svm1_cifs_nfs_lif1” is already on port e0c of that node, choose the
e0d port as the target port for “svm1_cifs_nfs_lif2”.
13. Expand the ports list for your target node, cluster1-01 in this example, and select port e0d.
14. Click Migrate.

Nondistruptive Operations for NetApp ONTAP 9.0

11 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 13

14

Figure 3-8:

Although you are not using it in this exercise, notice the Migrate Permanently checkbox in this window.
If you check this box it indicates that the LIF’s home port should also be set to this new port value.
The “Migrate Interface” window closes, and focus returns to the Networks pane in System Manager.
The LIF quickly migrates over to the new node and port, and the sio program generates no error
messages during or after the migration, indicating that it was unaffected by the operation.
15. The “Current Port” value shown for the LIF in the Network Interfaces list has changed to reflect the
nodes’ new port assignment. The small red X next to the current port entry indicates that the LIF does
not currently reside on it’s configured home port.

Nondistruptive Operations for NetApp ONTAP 9.0

12 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 15

Figure 3-9:

Most likely you do not want to leave this LIF on this alternate node and port indefinitely. In most cases
you would perform any maintenance that required you to move the LIF in the first place, and when
finished you would move the LIF back to it’s original location. So now send the LIF back to it’s home
port.
16. Right-click on the list entry for the LIF.
17. Select Send to Home from the context menu.

Nondistruptive Operations for NetApp ONTAP 9.0

13 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 16

17

Figure 3-10:

The LIF migrates back to it’s home port, once again without disrupting the sio utility.
18. The “Current Port” value for the LIF returns to it’s original value in the Network Interfaces list, and the
red X disappears to indicate that the LIF is back on it’s home port.

Nondistruptive Operations for NetApp ONTAP 9.0

14 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 18

Figure 3-11:

When sio starts, it generates several lines of output, and then goes silent until it reaches the end of it’s
scheduled execution interval, at which point it outputs quite a few lines of statistical information about
the execution before exiting. If the LIF migration disrupted any of sio’s write operations to the target
file, then around the same time you will see obvious error messages in the sio output, and sio will have
terminated abnormally. You should see no such error messages during this exercise, as LIF migration
is non-disruptive. If sio is still running, you can terminate it by issuing a Ctrl-c within the Powershell
window.

3.2 Volume Move

In this section of the lab you will nondisruptively move a volume to another aggregate on a different node while
sending I/O to the volume using the sio command.
This exercise leverages one of the storage configurations from the Basic Concepts lab. You can choose any of
the four test scenarios listed in the Volume Move Exercise Choices table.

Table 5: Volume Move Exercise Choices

Platform Protocol Client SVM Volume Move File

Linux NFS rhel1 svm1 engineering /svm1/engineering/nfs_move.sio
Windows CIFS jumphost svm1 engineering S:\engineering\cifs_move.sio
Linux iSCSI rhel1 svmluns linluns /linuxlun/move.sio
Windows iSCSI jumphost svmluns winluns E:\move.sio

Nondistruptive Operations for NetApp ONTAP 9.0

15 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 Please only select one entry from the table at a time for the volume move test in order to maintain optimum
simulator performance. Once you complete the test for one entry you are welcome to come back and test
another.
Note: You have not previously used the the information in the File column of the table, that is information
you will be using later in this lab section.
1. You will be generating I/O load to the volume for your selected entry using the sio command, and you
will run sio from the client listed for your chosen entry.
• If the listed client is rhel1, then you will need to open a PuTTY session to rhel1 (if you do not
already have one). The PuTTY launch icon is on the taskbar of Jumphost. Log in to rhel1 using
the username root and password Netapp1!.

Figure 3-12:

• If the listed client is jumphost, then you will need to open PowerShell (if you do not already
have one open). The PowerShell launch icon is on the taskbar of jumphost.

Figure 3-13:

2. In the appropriate client CLI window for your exercise choice, launch sio using the following syntax:
sio 0 0 4k 0 50m 600 2 <File> -create where <File> is a value that comes from the Volume
Move Exercise Choices table. This will initiate a sequential write operation to <File> for a period of 10
minutes,which should be sufficient time to complete the volume move operation.
Note: The following screenshots and CLI examples in this section show the steps required to
complete the volume move exercise for the Linux and NFS entry from the Volume Move Exercise
Choices table. If you chose to utilize a different table entry then you will need to substitute your
entry’s corresponding values where appropriate.
Launch the sio job for your client using the appropriate <File> value from the Volume Move Exercise
Choices table. The following example is for the NFS entry.

[root@rhel1 ~]# sio 0 0 4k 0 50m 600 2 /svm1/engineering/nfs_move.sio -create

Version: 6.39
Read: 0 Rand: 0 BlkSz: 4096 BegnBlk: 0 EndBlk: 12800 Secs: 600 Threads: 2 Devs: 1 /svm1/
engineering/nfs.sio

Now that sio is running, examine the volume you plan to move.

Nondistruptive Operations for NetApp ONTAP 9.0

16 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 3. If you do not currently have a System Manager instance running, then on the desktop of Jumphost,
launch Chrome and log into System Manager. (Username = admin, password = Netapp1!. If you need
further assistance in opening System Manager then see Appendix 1.)

Figure 3-14:

4. In the command bar at the top of System Manager, select SVMs.

5. In the SVMs pane, select the SVM that corresponds to your chosen entry in the Volume Move Exercise
Choices table. In this example that volume is svm1.

Figure 3-15:

6. In the newly displayed pane for your chosen SVM, select the Volumes tab.
7. In the list of the SVM's Volumes, select the volume that corresponds to your chosen entry in the Volume
Move Exercise Choices Table. This example uses the “engineering” volume.
Note: This exercise assumes that your volumes initially exist on the aggregate
aggr1_cluster1_01, and that you will migrate your chosen volume to aggr1_cluster_02.
8. On the menu bar, select the Move button. You may need to expand the window to see the button.

Nondistruptive Operations for NetApp ONTAP 9.0

17 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 6
8

Figure 3-16:

The “Move Volume” window opens.

9. In the “Destination Aggregate” list select the only listed aggregate. If your volume is currently located
on aggr1_cluster1_01, then that listed entry should be aggr1_cluster1_02, otherwise the values will be
reversed.
10. Click the Move button to start the Move operation.

Nondistruptive Operations for NetApp ONTAP 9.0

18 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 9

10

Figure 3-17:

A “Move Volume” confirmation window opens asking if you are sure that you want to move the volume.
11. Click the Move button.

11

Figure 3-18:

Nondistruptive Operations for NetApp ONTAP 9.0

19 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 The “Move Volume” window changes to show that the volume move is in progress.
12. Make a mental note of the Job ID number for the move operation, as you will need that value in the next
step. Click on the Job ID number.

12

Figure 3-19:

The “Move Volume” window closes and focus returns to System Manager, which now displays the “Job”
pane, where System Manager lists the current jobs that are running on the cluster.
13. In the list of current jobs, select the entry that matches the Job ID for your move operation, and observe
the “State” column for your job. You may need to scroll down the window in order to see the entry for
your job.
14. If the job’s state is “queued”, or “running” then use the Refresh button to refresh the display every few
seconds until the Status changes to “success”.

14

13

Figure 3-20:

Nondistruptive Operations for NetApp ONTAP 9.0

20 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 Once the Job completes successfully, go back to the Volumes pane in System Manager to view the
volume’s aggregate information.
15. In the command bar at the top of System Manager, select the SVMs tab.
16. In the SVM pane, select the SVM that hosts the volume you chose to move.
Figure 3-21:

15

16

17. In the newly displayed pane for your chosen SVM, select the Volumes tab.
18. In the Volume pane, examine the entry for the volume you moved, and observe which aggregate it now
resides on.

17

18

Figure 3-22:

Nondistruptive Operations for NetApp ONTAP 9.0

21 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 19. Take a look at the PowerShell or PuTTy window where you started the sio command. If you completed
your volume move operation within 10 minutes from when you started sio, then sio should still be
running. When sio initially starts it generates a few lines of output and then goes silent until it either
finishes (at which point it prints a bunch of summary information about the completed execution), or
until it encounters an error (which is what you would see if sio was unable to write to the volume during
the move operation).
If you encountered an error you will see error messages in the sio output similar to the following
example.

[root@rhel1 ~]# sio 0 0 4k 0 50m 600 2 /svm1/engineering/nfs.sio -create

Version: 6.39
Read: 0 Rand: 0 BlkSz: 4096 BegnBlk: 0 EndBlk: 12800 Secs: 600 Threads: 2 Devs: 1 /svm1/
engineering/nfs.sio
WriteFile fails with The process cannot access the file because another process has locked a
portion of the file.
Actual io size doesn't match requested io size: No error
Filename: /svm1/engineering/nfs_move.sio op: write offset: 26140672 byte_count: -1
io_nbytes_this_loop: 4096
SIO_ERROR: do_io: file access error

If you do not see any error messages, or just see a group of summary statistics, then sio was
unaffected by the volume move.

[root@rhel1 ~]# sio 0 0 4k 0 50m 600 2 /svm1/engineering/nfs.sio -create

Version: 6.39
Read: 0 Rand: 0 BlkSz: 4096 BegnBlk: 0 EndBlk: 12800 Secs: 600 Threads: 2 Devs: 1 /svm1/
engineering/nfs.sio

If sio is still running, enter Ctrl-c to terminate the utility. When it terminates, sio may output a number of
lines of summary statistics that you can ignore for this exercise.
Important: In preparation for the next exercise, repeat the volume move procedure to move
your volume back to aggr1_cluster1_01.

3.3 Storage Failover

In this section you will initiate an HA failover on one of the HA nodes in your cluster while a client is writing to a
volume hosted on that node, and you will see that the client’s I/O to the volume is not disrupted by the HA failover
event.
This exercise leverages one of the storage configurations from the Basic Concepts lab. You can choose any of
the three test scenarios listed in the Storage Failover Exercise Choices table.

Table 6: Storage Failover Exercise Choices

Platform Protocol Client SVM Volume File

Linux NFS rhel1 svm1 engineering /svm1/engineering/sfo.sio
Linux iSCSI rhel1 svmluns linluns /linuxlun/sfo.sio
Windows iSCSI jumphost svmluns winluns E:\sfo.sio

Note: The table does not contain a CIFS entry because HA failover is disruptive to the CIFS configuration
used in this lab.
In order to maintain optimum simulator performance, please only select one entry at a time for the storage failover
test. Once you complete the test for one entry you are welcome to come back and test another.
Note: You have not previously used the information in the File column of the table. That is information
you will use later in this lab section.
1. You will generate an I/O load to the volume for your selected entry using the sio command, and you will
run sio from the client listed for your chosen entry.

Nondistruptive Operations for NetApp ONTAP 9.0

22 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 • If the listed client is rhel1, then you will need to open a PuTTY session to rhel1 (assuming you
do not already have one open). The PuTTY launch icon is on the taskbar of Jumphost. Log in
to rhel1 using the username root and the password Netapp1!.

Figure 3-23:

• If the listed client is Jumphost, then you will need to open PowerShell (assuming you do not
already have one open). The PowerShell launch icon is on the taskbar of Jumphost.

Figure 3-24:

2. The sio command syntax you use is: sio 0 0 4k 0 50m 1200 2 <File> -create where <File> is
the “File” value that comes from the Storage Failover Exercise Choices table. This command initiates
a sequential write operation to <File> for a period of 20 minutes, which should be sufficient time to
complete the storage failover exercise.
The remaining screenshots and CLI examples in this section show the steps required to complete
the storage failover exercise for the Windows and iSCSI entry from the Storage Failover Exercise
Choices table. If you chose a different entry then you will need to substitute your entry’s values where
appropriate.
Launch the sio job for your client using the appropriate <File> value from the Storage Failover Exercise
Choices table. The following example is for the Windows and iSCSI entry.

PS C:\> sio 0 0 4k 0 50m 1200 2 E:\sfo.sio -create

Version: 6.39
Read: 0 Rand: 0 BlkSz: 4096 BegnBlk: 0 EndBlk: 12800 Secs: 1200 Threads: 2 Devs: 1 E:\sfo.sio

Now that sio is running on your selected client you can look at the cluster’s current failover state.
3. If you do not currently have a System Manager instance running then, on the desktop of Jumphost,
launch Chrome and log into System Manager. (If you need further assistance in opening System
Manager then see Appendix 1.)

Nondistruptive Operations for NetApp ONTAP 9.0

23 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 3

Figure 3-25:

4. In the command bar at the top of System Manager, select the Configuration tab. You may need to
expand your browser window in order to see this tab.
5. In the left pane, under the Cluster Settings section, select High Availability.

Figure 3-26:

The “High Availability” pane shows the current failover state of the HA pair. The Cluster HA Status line at
the top of the window should show a green check mark, and the text “All nodes are paired and ready for
failover”, indicating that your cluster is correctly prepared for HA failover.
All the volumes you created in this lab were originally hosted on an aggregate on the node cluster1-01. If
you performed the volume move exercise in this lab, then during that exercise you would have moved a
volume on that node over to an aggregate on the node cluster1-02. At the end of the exercise you should
also have moved the volume back to it’s original location. With that in mind, in this exercise you will have
cluster1-02 take over for cluster1-01 so you can see what effect a takeover operation has on the SVMs,
LIFs, volumes, and aggregates that are currently hosted on cluster1-01.
Cluster1-01 also hosts the cluster management LIF you are using to manage the cluster. Having
cluster1-02 takeover cluster1-01 will also allow you to see how a cluster management LIF behaves
during a takeover event.

Nondistruptive Operations for NetApp ONTAP 9.0

24 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 Now it is time to initiate the takeover operation.
6. In the “High Availability” pane, click on the Action button under the image of cluster1-02, and select
Takeover node “cluster1-01” from the list of available actions.

Figure 3-27:

The “Takeover Confirmation” dialog opens, and includes warnings that the node cluster1-01 contains a
cluster management LIF that may be unreachable during the failover, and warns that CIFS sessions to
cluster1-01 may be disrupted as well.
7. Click the Takeover button to proceed.

7
Figure 3-28:

The “Takeover Confirmation” dialog closes, and focus returns to the “High Availability” pane in System
Manager.
8. The Cluster HA Status line at the top of the pane now shows a yellow caution icon, and states that a
storage failover is in progress.

Nondistruptive Operations for NetApp ONTAP 9.0

25 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 9. There is now also a yellow warning box over the cluster diagram that says that node cluster-01 is going
offline, and that cluster1-02 is taking it over.
10. Note the Refresh button. System Manager will automatically update the status information on this page
every minute or so, but if you want to see status changes more quickly, click the Refresh button.

10
8

Figure 3-29:

At some point, as you are monitoring the takeover activity, you will see an HTTP error window pop
open. This is expected behavior, and is the result of the loss of connectivity to the cluster management
LIF during takeover.
11. Click the Show Details button, and read through the details to gain a better understanding of why this
error has happened.

11

Figure 3-30:

12. Click OK to acknowledge the error.

Nondistruptive Operations for NetApp ONTAP 9.0

26 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 12

Figure 3-31:

Your web browser page may display a connection timeout, or a page not found error, at this point.
If this happens, periodically use the Refresh button inside the High Availability pane. It may take
several minutes before the window is able to correctly display again. If several minutes elapse and
System Manager has still not re-establish connectivity, then try using your browser’s refresh button.
If your browser cannot re-connect within a minute or two using the browser refresh button, close
your browser, re-open it, log in again to System Manager, and then navigate back to the Cluster >
cluster1 > High Availability page.
Once System Manager is connected again, and depending on how frequently you use the Refresh
button inside the High Availability page during the takeover operation, you may see the node shown
as “Offline”. During a takeover operation, the node being taken over is actually undergoing a reboot. If
this were a production node on which you wanted to perform hardware maintenance, you could at this
point switch over to the controller's serial console in order to interrupt the node's boot process, and then
power off the controller. If you did this, the High Availability page in System Manager would continue
to show the node as offline for the duration of the downtime. Once you power the node back on again,
ONTAP will automatically start booting up the node, but will pause before bringing the node fully online,
not proceeding further until the node that has taken it over initiates a giveback.

Nondistruptive Operations for NetApp ONTAP 9.0

27 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 Figure 3-32:

In this lab you will just let the node shut down and start it's reboot without interruption, so after a few
minutes, and without any action required on your part, the High Availability pane will change to indicate
that the node cluster1-01 is now ready for giveback.
13. The “Cluster HA Status” line indicates that some of the nodes are in giveback state.
14. The summary message above the diagram indicates that cluster1-01 is online and ready for giveback,
and that some LIFs are not on their home node.
15. The status line under the image for cluster1-01 states that the node is waiting for giveback.

Nondistruptive Operations for NetApp ONTAP 9.0

28 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 13

14

15

Figure 3-33:

16. Click on the Action button under cluster1-02, and select Give back node “cluster1-01”.

Nondistruptive Operations for NetApp ONTAP 9.0

29 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 16

Figure 3-34:

A “Giveback Confirmation” window opens asking you to confirm that you want to initiate a giveback
operation from cluster1-02 to cluster1-01.
17. Click Giveback.

17

Figure 3-35:

The “Giveback Confirmation” window closes, and focus returns to the “High Availability” pane in System
Manager.
18. The summary message above the diagram changes to say that cluster1-01 is in a partial giveback
state, meaning that the giveback operation is in process.
19. The status line under the image for cluster1-01 states that the node is in partial giveback.
20. This page will update every minute or so while the giveback operation is in process, but you can use
the Refresh button to update the status information more frequently. It takes several minutes for the
giveback operation to complete.

Nondistruptive Operations for NetApp ONTAP 9.0

30 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 20

18

19

Figure 3-36:

When the giveback operation completes, most, but not all, of the warning indicators displayed on the
High Availability pane will go back to green.
21. The icon on the Cluster HA Status line at the top of the pane reverts back to a green check mark, and
the line says that all nodes are paired and ready for takeover.
22. The summary messages above the cluster diagram indicate that both the nodes can perform takeover
on each other, but the warning that some LIFs are not on their home node remains unchanged.

Nondistruptive Operations for NetApp ONTAP 9.0

31 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 21
22

Figure 3-37:

During a giveback, ONTAP will by default leave non-SAN LIFs on the node that performed the
takeover, since you will probably want to verify that everything is working correctly on the node that was
given back before you start sending all of the LIFs back to their home ports.
Examine the current port assignments for your cluster’s LIFs.
23. In the command bar at the top of System Manager, select the Network tab.
24. In the Network pane select the Network Interfaces tab.
25. Click on the Interface Name heading to sort the entries in interface name order, then scroll to the
bottom of the list to find the “cluster_mgmt” and the “svm1_cifs_nfs_lif*” LIFs.
26. Notice that all of the NAS and management LIFS for svm1 have a red icon under the “Current Port”
column, which indicates that these LIFs are not currently running on their home node and port.

Nondistruptive Operations for NetApp ONTAP 9.0

32 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 23

24

25

26

Figure 3-38:

Now start sending those LIFs home.

27. Right-click on the cluster_mgmt LIF.
28. From the context menu select Send to Home.

Nondistruptive Operations for NetApp ONTAP 9.0

33 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 27

28

Figure 3-39:

A Warning dialog will open that indicates you have chosen to migrate a cluster management LIF which
could result is some disruption to management access, and asks you to confirm that you want to
proceed.
29. Click Yes.

29

Figure 3-40:

The Warning dialog closes, and shortly after an Error dialog may open stating the server is not
responding to HTTP. If this does happen (and it often does not here), this is again the result of the
cluster management LIF being temporarily unavailable as it migrates back to it’s home port.
30. Click OK.

Nondistruptive Operations for NetApp ONTAP 9.0

34 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 30

Figure 3-41:

If you encounter this Error dialog, then periodically use the Refresh button inside the Network
Interfaces tab until System Manager is properly responding again. If you have not re-established
connectivity within 1-2 minutes then you will probably need to use your browser’s refresh button to
refresh the page until it is able to communicate again with System Manager. If your browser cannot re-
connect within a minute or two using the browser refresh button, close your browser, re-open it, log in
again to System Manager, and then navigate back to the Network > Network Interfaces page.
31. The red icon next to the Network Interfaces list entry for the cluster_mgmt LIF is now gone, and the
“Current Port” column indicates that the LIF is back on port cluster1-01:e0c.

31

Figure 3-42:

Send the other LIFS with red icons home following this same procedure.

Nondistruptive Operations for NetApp ONTAP 9.0

35 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 32. Finally, take a look again at the sio job that you have been running throughout this whole storage
failover operation. If that sio job encountered any problems while writing to the target volume during the
takeover or giveback events, then sio will have generated error messages similar to the following:

PS C:\> sio 0 0 4k 0 50m 1200 2 E:\sfo.sio -create

Version: 6.39
Read: 0 Rand: 0 BlkSz: 4096 BegnBlk: 0 EndBlk: 12800 Secs: 1200 Threads: 2 Devs: 1 E:
\sfo.sio
WriteFile fails with The process cannot access the file because another process has locked a
portion of the file.
Actual io size doesn't match requested io size: No error
Filename: E:\sfo.sio op: write offset: 26140672 byte_count: -1 io_nbytes_this_loop: 4096
SIO_ERROR: do_io: file access error
PS C:\>

If sio encountered no problems while writing to the file during the takeover and giveback, then it will not
generate any additional output at all, unless if has finished it’s targeted 20 minute execution duration. In
that case it generates a number of lines of statistical output. That statistical output is not relevant for this
exercise.

PS C:\> sio 0 0 4k 0 50m 1200 2 E:\sfo.sio -create

Version: 6.39
Read: 0 Rand: 0 BlkSz: 4096 BegnBlk: 0 EndBlk: 12800 Secs: 1200 Threads: 2 Devs: 1 E:
\sfo.sio

This concludes the exercise. If sio is still running, issue a Ctrl-c in the PowerShell or PuTTY window to terminate
it’s execution.

Nondistruptive Operations for NetApp ONTAP 9.0

36 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
4 Appendix 1

4.1 Accessing the Command Line

PuTTY is the terminal emulation program used in the lab to log into Linux hosts and storage controllers in order to
run command line commands.
1. The launch icon for the PuTTY application is pinned to the taskbar on the Windows host Jumphost as
shown in the following screenshot; just double-click on the icon to launch it.

Figure 4-1:

Once PuTTY launches, you can connect to one of the hosts in the lab by following these steps. This
example shows a user connecting to the ONTAP cluster named “cluster1”.
2. By default PuTTY should launch into the “Basic options for your PuTTY session” display, as shown in the
screenshot. If you accidentally navigate away from this view, just click on the Session category item to
return to this view.
3. Use the scrollbar in the “Saved Sessions” box to navigate down to the desired host and double-click it
to open the connection. A terminal window will open, and you will be prompted to log into the host. You
can find the correct username and password for the host in the Lab Host Credentials table in the Lab
Environment section at the beginning of this guide.

Nondistruptive Operations for NetApp ONTAP 9.0

37 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 2

Figure 4-2:

If you are new to the clustered ONTAP CLI, the length of the commands can seem a little initimidating.
However, the commands are actually quite easy to use if you remember these three tips:
• Make liberal use of the Tab key while entering commands, as the ONTAP command shell
supports tab completion. If you hit the Tab key while entering a portion of a command word,
the command shell will examine the context and try to complete the rest of the word for you.
If there is insufficient context to make a single match, it will display a list of all the potential
matches. Tab completion also usually works with command argument values, but there are
some cases where there is simply not enough context for it to know what you want, in which
case you will just need to type in the argument value.
• You can recall your previously entered commands by repeatedly pressing the up-arrow key,
and you can then navigate up and down the list using the up and down arrow keys. When you
find a command you want to modify, you can use the left arrow, right arrow, and Delete keys
to navigate around in a selected command to edit it.
• Entering a question mark character “?? causes the CLI to print contextual help information.
You can use this character by itself, or while entering a command.
If you would like to learn more about the features of the ONTAP CLI, the Advanced Concepts for NetApp
ONTAP 9 lab includes an extensive tutorial on this subject.

4.2 Accessing System Manager

On the Jumphost, the Windows 2012R2 Server desktop you see when you first connect to the lab, open the web
browser of your choice. This lab guide uses Chrome, but you can use Firefox or Internet Explorer if you prefer one
of those. All three browsers already have System Manager set as the browser home page.

Nondistruptive Operations for NetApp ONTAP 9.0

38 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 1. Launch Chrome to open System Manager.

Figure 4-3:

The OnCommand System Manager Login window opens.

2. Enter the User Name admin, and the Password Netapp1!
3. Click the Sign In button.

Figure 4-4:

System Manager is now logged in to cluster1 and displays a summary page for the cluster. If you are
unfamiliar with System Manager, here is a quick introduction to its layout. Please take a few moments to
expand and browse these tabs to familiarize yourself with their contents.
4. The primary means of navigating within System Manager is through the tabs that appear under the blue
“NetApp OnCommand System Manager” bar at the top of the window. This bar of tabs is sometimes

Nondistruptive Operations for NetApp ONTAP 9.0

39 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 referred to as the “command bar”. Some command bar tabs may be obscured if the window is too small,
so you may need to expand the size of your browser window in order to see them all.
5. Additional tabs may appear with the main pane depending on which tab you selected in the command
bar.
6. The drop-down menu with the blue plus sign on the far right of the command bar provides a shortcut to
some of the more frequently used System Manager views.

5
6

Figure 4-5:

Tip: As you use System Manager in this lab, you may encounter situations where buttons at
the bottom of a System Manager pane are beyond the viewing size of the window, and no scroll
bar exists to allow you to scroll down to see them. If this happens, then you have two options;
either increase the size of the browser window (you might need to increase the resolution of
your Jumphost desktop to accommodate the larger browser window), or in the System Manager
window, use the tab key to cycle through all the various fields and buttons, which eventually
forces the window to scroll down to the non-visible items.

Nondistruptive Operations for NetApp ONTAP 9.0

40 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
5 References
We used following references when we created this lab guide.
• WP-7195: “Nondisruptive Operations and Clustered Data ONTAP”, March 2015
• TR-4100: “Nondisruptive Operations and SMB File Shares for Clustered Data ONTAP”, April 2013

Nondistruptive Operations for NetApp ONTAP 9.0

41 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
6 Version History
Version Date Document Version History
1.1 April 2015 Initial Release
1.2 October 2015 Insight 2015 Release
1.3 December 2016 Updated for ONTAP 9

Nondistruptive Operations for NetApp ONTAP 9.0

42 © 2016 NetApp, Inc. All rights reserved. NetApp Proprietary
 Refer to the Interoperability Matrix Tool (IMT) on the NetApp Support site to validate that the exact
product and feature versions described in this document are supported for your specific environment.
The NetApp IMT defines the product components and versions that can be used to construct
configurations that are supported by NetApp. Specific results depend on each customer's installation in
accordance with published specifications.

NetApp provides no representations or warranties regarding the accuracy, reliability, or serviceability of any
information or recommendations provided in this publication, or with respect to any results that may be obtained
by the use of the information or observance of any recommendations provided herein. The information in this
document is distributed AS IS, and the use of this information or the implementation of any recommendations or
techniques herein is a customer’s responsibility and depends on the customer’s ability to evaluate and integrate
them into the customer’s operational environment. This document and the information contained herein may be
used solely in connection with the NetApp products discussed in this document.

®
Go further, faster

© 2016NetApp, Inc. All rights reserved. No portions of this document may be reproduced without prior written consent
of NetApp, Inc. Specifications are subject to change without notice. NetApp, the NetApp logo, Data ONTAP®,
ONTAP®, OnCommand®, SANtricity®, FlexPod®, SnapCenter®, and SolidFire® are trademarks or registered
trademarks of NetApp, Inc. in the United States and/or other countries. All other brands or products are trademarks or
registered trademarks of their respective holders and should be treated as such.