Zeta File System

Cheat Sheet
Serge Y. Stroobandt

Copyright 2012–2017, licensed under Creative Commons BY-NC-SA

Why ZFS?
The data integrity problem can be best described as follows:1

System administrators may feel that because they store their data
on a redundant disk array and maintain a well-designed tape-backup
regimen, their data is adequately protected. However, undetected
data corruption can occur between backup periods. Subsequently
backing up corrupted data will yield corrupted data when restored.

The Zeta File System (ZFS) features the capability of being self validating and
self healing from silent data corruption or data rot through continuous data
block checksumming.2 Each block write operation yields a 256‑bit block check-
sum. The resulting block checksum is not stored with the block, but rather
with its parent block.3 Hence, the blocks of a ZFS storage pool form a Merkle
tree in which each block contains the checksum of all of its children.4 This al-

1
lows the entire pool to continuously self validate its data with every operation
on both accuracy and correctness, ensuring data integrity.

Figure 1: How ZFS self heals from silent data corruption or data rot through continu-
ous block checksumming. Source: root.cz

ZFS design principles include:3

• Pooled storage
◦ No volumes
◦ Virtualizes all disks
• End-to-end data integrity
◦ Everything Copy-On-Write
▪ No overwrite of live data
▪ On-disk state always valid
◦ Everything is checksummed
▪ No silent data corruption possible
▪ No panics due to corrupted metadata
• Everything transactional
◦ All changes occur together
◦ No need for journaling
◦ High-performance full stripe writes

2
ZFS server hardware
To run ZFS, a server needs to meet a set of stringent
hardware requirements. A separate article deals
with this matter in detail.

• ZFS only runs on 64-bit kernels because it contains 64-bit specific code.
• With deduplication switched off, ZFS runs on surprisingly modest
CPUs. Personally, I have ZFS running for several years on a 2010 vintage
Intel Atom D510 processor, clocked down to a mere 1 GHz clock speed to
save on the electricity bill.
• Install ZFS only on a system with lots of RAM. 1 GB is a bare minimum,
2 GB is better, 4 GB RAM would be preferred to start. Remember, ZFS
will use half of the available RAM for the ARC.
• Use ECC RAM when possible for scrubbing data in registers and
maintaining data consistency. The ARC is an actual read-only data cache
of valuable data in RAM.
• ZFS will be not happy running on a system without a swap partition.

Why Ubuntu LTS Server Edition?

I use ZFS on Ubuntu LTS Server Edition and Xubuntu LTS (the latter for mak-
ing offline backups). Since the advent of Ubuntu 16.04 LTS, ZFS packages are
provided in full support by the 64 bit distribution. As I am using Xubuntu LTS
on the desktop, it is a small step to use Ubuntu LTS Server Edition. Packages
are available under the same name and version number, which is really conve-
nient. Being Debian based, the sheer amount of available packages will suit all
needs and Ubuntu packages will be more recent than Debian.

I considered for a moment using CentOS for its ten years of support, compared
to a mere five years with Ubuntu Server LTS releases. However, in a home or
small office setting, one most probably expects a bit more from its server than
just file serving on the local network. The CentOS repository is extremely rudi-
mentary. Even for the most basic use cases, one grows dependent on third‑par-
ty repositories.

Neither are FreeBSD based distributions —like FreeNAS and NAS4Free—

an option, because of their lack of support for UTF-8 character encoding. This
might be fine in the Anglo-Saxon speaking part of the world, but certainly is
not for the vast majority in the rest of the world. This can be fixed, but I really
do not want to be dealing with something basic like this.

3
Installation
The Lawrence Livermore National Laboratory has
been working on porting the native Solaris ZFS
source to the Linux kernel as a kernel module. Fol-
low the installation instructions at zfsonlinux.org for
your specific GNU/Linux distribution.

Checking the installed ZFS version number can be

done as follows on Debian derived systems:

$ dpkg -l zfs*

Virtual devices
A virtual device (VDEV) is a meta-device that can represent one or more de-
vices. ZFS supports seven different types of VDEV.
• File - a pre-allocated file
• Physical Drive (HDD, SDD, PCIe NVME, etc)
• Mirror - a standard RAID1 mirror
• ZFS software raidz1, raidz2, raidz3 ‘distributed’ parity based RAID
• Hot Spare - hot spare for ZFS software raid.
• Cache - a device for level 2 adaptive read cache (ZFS L2ARC)
• Log - ZFS Intent Log (ZFS ZIL)

A device can be added to a VDEV, but cannot be removed from it. For most
home or small office users, each VDEV usually corresponds to a single phys-
ical drive. During pool creation, several of these VDEVs are combined into
forming a mirror or RAIDZ.
• Use whole disks rather than partitions. ZFS can make better use of
the on-disk cache as a result. If you must use partitions, backup
the partition table, and take care when reinstalling data into the other
partitions, so you don’t corrupt the data in your pool.
• Do not mix disk sizes or speeds in VDEVs, nor storage pools. If VDEVs
vary in size, ZFS will favor the larger VDEV, which could lead to
performance bottlenecks.
• Do not create ZFS storage pools from files in other ZFS datasets. This will
cause all sorts of headaches and problems.

4
Zpool creation
A zpool is a pool of storage made from a collection of
VDEVs. One or more ZFS file systems can be created
within a ZFS pool, as will be shown later. In practical
examples, zpools are often given the names pool ,
tank or backup , preferably followed by a digit to
designate between multiple pools on a system.

Identifying physical devices

$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:80 0 1.8T 0 disk

$ lsblk -o NAME,SIZE,SERIAL,TYPE,MOUNTPOINT
sdb 931.5G WD-WCAV51264701 disk

$ ls -l /dev/disk/by-id/
lrwxrwxrwx 1 root root 9 Jul 8 16:25 ata-WDC_WD20EFRX-68EUZN0_WD-
WCC4M4DKAVF1 -> ../../sdb

$ sudo hdparm -I /dev/sdb |grep Serial\ Number

Serial Number: WD-WCC4M4DKAVF1

Single drive zpool

Single drive zpools are useful for receiving a backup from a redundant mirror
or RAIDZ production server. Note that the lower-case -o option sets pool
properties, whereas the upper-case -O is required to set file system proper-
ties. Certain options are recommended to be set at every instance of a zpool
creation, even though these options are not set by default. These recommended
options are explained below.

$ sudo zpool create -f -o ashift=12 -o autoexpand=on -O compression=on -O

dedup=off backup0 /dev/disk/by-id/ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M4DKAVF1

$ sudo zpool list

NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH
ALTROOT
backup0 1.81T 300K 1.81T - 0% 0% 1.00x ONLINE -

5
-o ashift=12
The zpool create property -o ashift can only be set at pool creation time.
Its value corresponds to the base 2 logarithm of the pool sector size in kibibyte.
I/O operations are aligned to boundaries specified by this size. The default
9
value is 9, as 2 = 512, which corresponds to the standard sector size of operat-
ing system utilities used for both reading and writing data. In order to achieve
maximum performance from Advanced Format drives with 4 KiB boundaries,
12
the value should be set to ashift=12 , as 2 = 4096.

Here is a potential way of finding out the physical block size of your drives us-
ing the operating system. However, this method is not fail proof! Western Dig-
ital drives in particular may falsely report as being non-Advanced Format (see
inset below). Anyhow, by installing hdparm , one can query the microprocessor
on the printed circuit board of a hard drive. (Yes, hard drives are actually little
computers on their own.)

$ sudo apt-get install hdparm

$ sudo hdparm -I /dev/sdb |grep Sector\ size:
Logical Sector size: 512 bytes
Physical Sector size: 4096 bytes

Western Digital drives lie!!!

Advanced Format drives may emulate 512 byte
physical sectors, but still use 4096 byte sectors
underneath!
In 2010, the first drives featuring Advanced Format 4096 byte physical
blocks appeared on the market. Initially, compatibility with older op-
erating systems like Windows™ XP was an issue. Drive manufactur-
ers maintained backwards compatibility by reporting physical block
sizes of only 512 byte in utility commands such as lsblk , blktool
and occasionally even hdparm -I . Nowadays, most new hard drives
feature Advanced Format. However, Western Digital drives keep on
lying about their physical block size in a world where this no longer
necessary. Therefore, the only way of knowing for sure whether a dri-
ve is Advanced Format or not is be looking up the specification sheet or
perhaps reading the physical label on the drive. To achieve maximum
performance with Advanced Format drives, specify -o ashift=12 .

6
 Figure 2: The label of a 2010 model EARS 1 TB Western Digital Green drive,
featuring Advanced Format. Source: tomshardware.com

-o autoexpand=on
The pool property -o autoexpand=on must be set on before replacing a first
drive in the pool with a larger sized one. The property controls automatic pool
expansion. The default is off . After all drives in the pool have been replaced
with larger drives, the pool will automatically grow to the new, larger drive
size.

-O compression=on
Always enable compression. There is almost certainly no reason to keep it dis-
abled. It hardly touches the CPU and hardly touches throughput to the drive,
yet the benefits are amazing. Compression is disabled by default. This doesn’t
make much sense with today’s hardware. ZFS compression is extremely cheap,
extremely fast, and barely adds any latency to the reads and writes. In fact, in
some scenarios, your disks will respond faster with compression enabled than
disabled. A further benefit is the massive space benefits.

-O dedup=off
Even if you have the RAM for it, ZFS deduplication is, unfortunately, almost
certainly a lose.5 So, by all means avoid using deduplication; even on a ma-
chine built to handle it. Unlike compression, deduplication is very costly on
the system. The deduplication table consumes massive amounts of RAM.

-f
The force option -f forces the use of the stated VDEVs, even if these appear
to be in use. Not all devices can be overridden in this manner.

7
Zpool mirror
$ cd /dev/disk/by-id/
$ ls
$ sudo zpool create -f -o ashift=12 -O compression=on -O dedup=off pool0
mirror scsi-SATA_WDC_WD10EARS-00_WD-WCAV56475795 scsi-
SATA_WDC_WD10EARS-00_WD-WCAV56524564

In order to take advantage of ZFS self healing capabilities through constant

checksumming (Figure 1), a ZFS storage server should have inherent redun-
dancy offered by either mirroring or RAIDZ. Use redundancy when possible,
as ZFS can and will want to correct data errors that exist in the pool. You can-
not fix these errors if you do not have a redundant good copy elsewhere in
the pool. Mirrors and RAID-Z levels accomplish this.

A mirrored storage pool configuration requires at least two disks, when pos-
sible, connected to separate controllers. Personally, I prefer running a three-
way mirror using three disks, even though this consumes 50% more electric
power. Here is the reason why. When one physical drive in a two-way mirror
fails, the remaining drive needs to be replicated —resilvered in ZFS speak—
to a new physical drive. Replication puts additional stress on a drive and it
is not inconceivable that the remaining drive would fail during its replication
process. “When it rains, it pours.” By contrast, a three-way mirror with one
failed disk maintains 100% redundancy. A similar argument exists in favour of
RAIDZ‑2 and RAIDZ‑3 over RAIDZ‑1.

Zpool RAIDZ
• http://www.openoid.net/zfs-you-should-use-mirror-vdevs-not-raidz/
• When considering performance, know that for sequential writes,
mirrors will always outperform RAID-Z levels. For sequential reads,
RAID-Z levels will perform more slowly than mirrors on smaller data
blocks and faster on larger data blocks. For random reads and writes,
mirrors and RAID-Z seem to perform in similar manners. Striped
mirrors will outperform mirrors and RAID-Z in both sequential, and
random reads and writes.
• Consider using RAIDZ-2 or RAIDZ-3 over RAIDZ-1. You’ve heard
the phrase “when it rains, it pours”. This is true for disk failures. If a disk
fails in a RAIDZ-1, and the hot spare is getting resilvered, until the data
is fully copied, you cannot afford another disk failure during the resilver,
or you will suffer data loss. With RAIDZ-2, you can suffer two disk
failures, instead of one, increasing the probability you have fully
resilvered the necessary data before the second, and even third disk fails.

8
ZFS file system creation
Turning access time writing off with -o atime=off can result in significant
performance gains. However, doing so might confuse legacy mail clients and
similar utilities.

$ sudo zfs create -o atime=off pool0/home

• Avoid running a ZFS root file system on GNU/Linux for the time being.
It’s a bit too experimental for /boot and GRUB.
• However, do create datasets for /home/ , /var/log/ and /var/cache/ .
• For /home/ ZFS installations, setting up nested datasets for each user.
For example, pool0/home/atoponce and pool0/home/dobbs . Consider
using quotas on the datasets.

Scrubbing
Scrubbing examines all data to discover hardware
faults or disk failures, whereas resilvering examines
only that data known to be out of date. Scrubbing
ZFS storage pools does not happen automatically.
It is intiated manually. If possible, scrub consumer-
grade SATA and SCSI disks weekly and enterprise-
grade SAS and FC disks monthly.

$ sudo zpool scrub pool0

Snapshot creation
Snapshot frequently and regularly. Snapshots are
cheap, and can keep a plethora of file versions over
time. Consider using something like
the zfs-auto-snapshot script.

$ sudo zfs snapshot pool0/home@20120722

$ sudo zfs list -t snapshot

To see the snapshot creation time, add the following zfs list options:

9
 $ sudo zfs list -t snapshot -o name,used,refer,creation
NAME USED REFER CREATION
pool0/home@20120722 1.35M 373G Sun Jul 22 12:22 2012

Snapshots of file systems are accessible through the .zfs/snapshot/ directo-

ry within the root of the containing file system.

cd /home/.zfs/snapshot/20120722

Automated snapshots
• https://opensource.com/life/16/7/sanoid
• Sanoid
• http://askubuntu.com/a/391970/164341
• zfs-auto-snapshot

Local backup to another zpool

First time local backup
Snapshots are not a backup. Use zfs send and zfs receive to send snap-
shots to a local detachable external storage drive. screen

$ screen -S backup
$ sudo su - root
$ zfs list -t snapshot
$ zfs send -R pool0/home@20160707 |zfs receive -F backup0/home
$ zpool export backup0

Do not forget to export the backup zpool before detaching!

The option -R generates a replication stream package, which will replicate

the specified file system, and all descendent file systems, up to the named
snapshot. When received, all properties, snap‐shots, descendent file systems,
and clones are preserved.

If the -F flag is specified when this stream is received, snapshots and file sys-
tems that do not exist on the sender side are destroyed on the receiver side.

10
Subsequent incremental local backups
If the -i or -I flags are used in conjunction with an additional snapshot argu-
ment and the -R flag, an incremental or delta replication stream is generated.
This is a real time saver. By specifying the -R replication flag, property values,
the current snapshot and file system names are updated when the stream is re-
ceived.

TODO: http://docs.oracle.com/cd/E19253-01/819-5461/gbchx/

Remote backup
The use of zfs send is preferred over using rsync , as the zfs send command
preserves dataset properties. Perform regular (at least weekly) backups of
the full storage pool.

“Consider data as intrinsically lost, unless a tested, offline, off-site

backup is available.”

A backup consists of multiple copies. Having only redundant disks, does not
suffice to guarantee data preservation in the event of a power failure, hard-
ware failure, disconnected cables or a viral infection.
• http://www.openoid.net/why-sanoids-zfs-replication-matters/

Delegating receive permissions on the backup

client

Delegation of permissions through zfs allow and zfs unallow is

available on ZFSonLinux only since version 0.7.0. Even then, some
limitations apply.

sudo zfs allow desktop_user create,mount,receive backup0

Delegated permissions on the backup client can be checked with:

$ sudo zfs allow backup0

---- Permissions on backup0 ------------------------------------------
Local+Descendent permissions:
user serge create,mount,receive

11
 • Oracle Solaris ZFS Administration Guide
• simplesnap
• zrep

Sending a replication stream for the first time

In below example, a ssh session is opened from a desktop to the server called
d510 .

$ ssh -t d510 'sudo zfs list -t snapshot'

NAME USED AVAIL REFER MOUNTPOINT
pool0/home@20160707 124K - 450G -

$ ssh -t d510 'sudo zfs send -R pool0/home@20160707' |zfs receive -F

backup0/home

The option -R generates a replication stream package, which will replicate

the specified file system, and all descendent file systems, up to the named
snapshot. When received, all properties, snap‐shots, descendent file systems,
and clones are preserved.

If the -F flag is specified when this stream is received, snapshots and file sys-
tems that do not exist on the sender side are destroyed on the receiver side.

Subsequent incremental remote backups

If the -i or -I flags are used in conjunction with an additional snapshot argu-
ment and the -R flag, an incremental or delta replication stream is generated.
This is a real time saver. By specifying the -R replication flag, property values,
the current snapshot and file system names are updated when the stream is re-
ceived.

Monitoring
In order to preserve maximum performance, it is es-
sential to keep pool allocation under 80% of its full
capacity. The following set of monitoring commands
help to keep an eye on this and other aspects of zpool
health.

12
For example, the file system may get heavily fragmented due to the copy-on-
write nature of ZFS. It might be useful to e-mail capacity reports monthly.

$ sudo zpool list

NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH
ALTROOT
pool0 928G 475G 453G - 4% 51% 1.00x ONLINE -

$ sudo zfs list

NAME USED AVAIL REFER MOUNTPOINT
pool0 475G 424G 136K /pool0
pool0/home 475G 424G 450G legacy

$ sudo zfs list -t snapshot

NAME USED AVAIL REFER MOUNTPOINT
pool0/home@20120722 1.35M - 373G -
…
pool0/home@20160701 31.7M - 450G -
pool0/home@20160706 12.1M - 450G -
pool0/home@20160707 124K - 450G -

Use the zpool status command for status monitoring. Options are available
for verbose output or an automatic repeat after, for example, every five sec-
onds. Consult the Oracle Solaris documentation for details about
zpool status output. The second command line is a remote server example.

$ sudo zpool status -v 5

$ ssh -t d510 'sudo zpool status -v 5'

Input/output monitoring, verbose for individual VDEVs, repeated every five

seconds:

$ sudo zpool iostat -v 5

Mounting a backup
A backup drive will attempt to mount at the same point as its original. This re-
sults in a error similar to:

`cannot mount '/home': directory is not empty`.

Changing the mountpoint for the backup resolves this issue. The backup zpool
will need to be exported and imported again for this mountpoint change to
take effect.

13
 $ sudo zfs set mountpoint=/backup0/home backup0/home
$ sudo zpool export backup0
$ sudo zpool import backup0

Older snapshots of file systems on the backup drive are accessible through
the .zfs/snapshot/ directory within the root of the containing file system.

cd /backup0/.zfs/snapshot/home/20170803

Mounting as /home
Here is how to mount a ZFS file system as home:

$ sudo zfs set mountpoint=/home pool0/home

SFTP server
Workstations use the SSHFS (Secure Shell Filesystem) client to access server
files through SFTP (Secure Shell File Transfer Protocol).
• Complete file permission transparency is the main reason for preferring
SFTP over the Windows™ style Server Message Block (SMB). This,
despite the fact that ZFS has been integrated with the GNU/Linux
implementation, called Samba.
• Furthermore, SFTP handles well changing the case of filenames. This
cannot be said about the latest Samba versions!
• SFTP significantly simplifies things. If the server is accessible over SSH,
SSHFS should also work.
• Running only an OpenSSH SFTP server significantly reduces exposure
as SFTP is inherently more secure than SMB and NFS.
• Eavesdropping on the (W)LAN is not an issue, since all file transfers are
encrypted.
• The only drawback are the slower file transfer speeds due to
the encryption overhead.

14
SSHFS clients
Here is the bash script that I wrote to mount the server through SSHFS on any
client computer. It gets executed at login, in my case by specifying the script
in Xubuntu’s Session and Startup → Application Autostart . However,
the script can also be run manually, for example after connecting to a mobile
network.

The optimisation parameters are from the following article and tests. As a req-
uisite, the administrator needs to create a /$server/$USER mount point direc-
tory for every user on the client system.

#!/usr/bin/env bash

mountpoint="/$server/$USER"

options=' -o reconnect -o cache=yes -o kernel_cache -o compression=yes -o

large_read'
options+=" -o idmap=file -o uidfile=$HOME/.uidfile -o gidfile=$HOME/.gidfile
-o nomap=ignore"

{ sshfs $server:/home $mountpoint -p 22 $options > /dev/null 2>&1 && echo -e

'\nThe server is now mounted over LAN.\n'; } || \
{ sshfs $roam:/home $mountpoint -p 443 $options && echo -e '\nThe server is
now mounted over WAN.\n'; }

Export before a release upgrade

or moving drives!
Always export your storage pool before performing
a do-release-upgrade to a newer version of your distribution or be-
fore moving the drives from one physical system to another!

Failure to do so, may render the zpool unmountable.

~# zpool export pool0

umount: /home: target is busy

(In some cases useful info about processes that
use the\ device is found by lsof(8) or fuser(1).)
cannot unmount '/home': umount failed

15
Unmounting /home
Certain Zeta pool and file system operations require prior unmounting of
the file system. However, the \home Zeta file system will refuse to unmount
because of being in use. Here is how to overcome this conundrum:
1. Physical access to the server will be required.
2. For security reasons, set both PermitRootLogin to no and
PasswordAuthentication to no in the /etc/ssh/sshd_config
configuration file, and leave it like that. Here are Ansible tasks to take
care of this:

---
- name: 'PermitRootLogin no'
lineinfile:
path: /etc/ssh/sshd_config
regexp: '^PermitRootLogin '
line: 'PermitRootLogin no' # Was: PermitRootLogin prohibit-password
notify: restart ssh

- name: 'PasswordAuthentication no'

lineinfile:
path: /etc/ssh/sshd_config
regexp: '^PasswordAuthentication '
line: 'PasswordAuthentication no'
notify: restart ssh

- meta: flush_handlers

The corresponding Ansible handler is:

---
- name: 'restart ssh'
service: name=ssh state=restarted

3. Since Ubuntu is being used, and if a root password was not defined yet,
one should do so now using $ sudo passwd root . There really is no
other way.
4. In /etc/dhcp/dhclient.conf , set timeout to 15 . Otherwise, the next
step will cause the boot time to increase with 5 minutes (300 seconds)!

---
- name: 'Lower DHCP timeout to 15s.'
lineinfile:
path: /etc/dhcp/dhclient.conf
regexp: '^timeout'
line: 'timeout 15' # Was: timeout 300

16
 5. Reboot with the 8P8C Ethernet plug physically disconnected from
the RJ45 jack. This will ensure no other users are accessing the /home
file system.
6. Open a real text console with Ctrl + Alt + F1 .
7. Immediately log in as root .
8. Kill any X window manager services ~# service lightdm stop
9. List any open files ~# lsof |grep home on /home and kill any
pertaining processes when necessary.
10. Cautiously perform the exporting or mount point setting.
11. Switch to your normal user name as soon as possible for any other
tasks. This adds a level of fool proof safety to any shell commands.

Importing zpools
To import all known storage pools, simply type:

~# zpool import -a

Importing zpools by name is also possible.

Drive referencing by‑id

Here is how to convert a zpool with /dev/sdx designated devices to devices
designated by /dev/disk/by-id/ .

~# zpool export pool0

~# zpool import -d /dev/disk/by-id/ pool0

Renaming a zpool
Here is an example where a zpool called tank0 is renamed to pool0 .

~# zpool export tank0

~# zpool import tank0 pool0
~# zpool list

17
Attaching more drives
Before adding more drives to a zpool, it might be a good idea to
scrub the zpool first.

Doing so, will limit the resilvering process to a single drive. If the re-
silvering process does not run smoothly (e.g. slow data rate and/or
endless restarting), the drive to blame will be known. Should this
happen, simply detach the faulty drive and dispose of it in an envi-
ronmentally friendly way.

The command zpool attach is used to attach an extra drive to an existing dri-
ve in a zpool as follows:

$ sudo zpool attach [-f] [-o property=value] <pool> <existing-device> <new-

device>

It may seem a bit superfluous having to specify an existing device in addition

to the zpool name when adding a new device. However, this has all to do with
the fact that one is also offered the possibility to add a new device to a virtual
device.

$ cd /dev/disk/by-id/
$ ls
$ sudo zpool status

$ sudo zpool attach pool0 scsi-SATA_WDC_WD10EADS-00_WD-WCAV51264701 scsi-

SATA_WDC_WD10EARS-00_WD-WCAV56524564
$ sudo zpool status 5

The last line is to monitor the resilvering process. If zpool attach is com-
plaining about the new drive being in use, and you know what you are doing,
simply add -f to force zpool attach into what you want it to do.

$ sudo zpool attach -f pool0 scsi-SATA_WDC_WD10EADS-00_WD-WCAU4D512265 scsi-

SATA_WDC_WD10EARS-00_WD-WCAV56475795

Upgrading the zpool version

Occasionally, a zpool status report may yield a message similar to this:

18
 $ sudo zpool status
pool: pool0
state: ONLINE
status: The\ pool is formatted using an\ older on-disk format. The\ pool can
still be used, but some features are unavailable.
action: Upgrade the\ pool using 'zpool upgrade'. Once this is done, the
pool will no longer be accessible on older software versions.

All zpools can easily be upgraded while being online. Use the following com-
mand:

$ sudo zpool upgrade -a

Troubleshooting
When failing to create a zpool
I once ran into a case where zpool creation by-id did not work. Using the sdx
device name did work, however. Exporting and reimporting the pool by-id
kept everything nice and neat.

$ sudo zpool create -f -o ashift=12 -o autoexpand=on -O compression=on -O

dedup=off backup0 /dev/disk/by-id/ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M4DKAVF1
$ sudo zpool create -f -o ashift=12 -o autoexpand=on -O compression=on -O
dedup=off backup0 /dev/sdx
$ sudo zpool export backup0
$ sudo zpool import -d /dev/disk/by-id/ backup0

When a drive refuses to attach

Running a consistency check through the command zdb may cure a drive
from not attaching.

$ sudo zdb
$ sudo zpool attach pool0 15687870673060007972 /dev/disk/by-id/scsi-
SATA_WDC_WD10EADS-00_WD-WCAV51264701

Splitting off a backup

It is possible to split off a drive or virtual device as a stand-alone backup zpool:

$ sudo zpool split pool0 backup0 /dev/disk/by-id/scsi-

SATA_WDC_WD10EADS-00_WD-WCAV51264701

19
It looks tempting to later update such a backup using resilvering. However,
I strongly advise against this. Doing so leads to problems further down
the road.

Destroying a zpool
ZFS pools are virtually indestructible. If a zpool does not show up im-
mediately, do not presume too quickly the pool to be dead. In my
experience, digging around a bit will bring the pool back to life. Do
not unnecessarily destroy a zpool!

A zfs destroy can cause downtime for other datasets. A zfs destroy will
touch every file in the dataset that resides in the storage pool. The larger
the dataset, the longer this will take, and it will use all the possible IOPS out of
your drives to make it happen. Thus, if it take 2 hours to destroy the dataset,
that’s 2 hours of potential downtime for the other datasets in the pool.

$ sudo zpool offline tank1 /dev/disk/by-id/scsi-SATA_WDC_WD10EARS-00_WD-

WCAV56475795
$ sudo zpool destroy tank1

20
Other cheat sheets
• https://www.csparks.com/ZFS%20Without%20Tears.html
• https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/
• https://wiki.ubuntu.com/Kernel/Reference/ZFS
• http://www.datadisk.co.uk/html_docs/sun/sun_zfs_cs.htm
• http://lildude.co.uk/zfs-cheatsheet
• http://louwrentius.com/74tb-diy-nas-based-on-zfs-on-linux.html
• http://arstechnica.com/information-technology/2014/02/ars-
walkthrough-using-the-zfs-next-gen-filesystem-on-linux/1/

Real-world example
Here is a real-world ZFS example by Jim Salter of service provider Openoid,
involving reshuffling pool storage on the fly.

References
1. Michael H. Darden. Data integrity: The Dell|EMC distinction. 2002.
Available at: https://www.dell.com/content/topics/global.aspx/power/en/
ps2q02_darden.

2. Jim Salter. Bitrot and atomic COWs: Inside “next-gen” filesystems. 2014.
Available at: http://arstechnica.com/information-technology/2014/01/
bitrot-and-atomic-cows-inside-next-gen-filesystems/1/.

3. Robbin Harris. ZFS: Threat or menace? 2007. Available at:

http://storagemojo.com/zfs-threat-or-menace-pt-i/.

4. Jeff Bonwick. ZFS end-to-end data integrity. 2005. Available at:

https://blogs.oracle.com/bonwick/entry/zfs_end_to_end_data.

5. Jim Salter. ZFS dedup: Tested, found wanting. 2015. Available at:
http://www.openoid.net/zfs-dedup-tested-found-wanting/.

21
 This work is licensed under a Creative Commons
Attribution‑NonCommercial‑ShareAlike 4.0 International License.
Other licensing available on request.

Unattended CSS typesetting with .

This work is published at https://hamwaves.com/zfs/en/.

Last update: Wednesday, January 2, 2019.

22