Analyzing Metadata Caching in the

Windows SMB2 Client

David Kruse, Mathew George

Microsoft Corporation

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

Agenda

 Understand what kind of metadata is cached by

existing Windows SMB2 clients
 Discuss the coherency guarantees provided for these
metadata caches.
 Analyzing protocol performance under different
metadata intensive workloads.
 Outline possible protocol extensions to improve
metadata caching in SMB2 through directory leases.

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

Why is metadata caching important?

 Handle based I/O semantics & simplified SMB2 command set

=> more round trips to achieve a given operation.
 CREATE + QUERY/SET + CLOSE
 Compounding (predictive or lazy) can be to reduce round
trips.

CREATE CREATE
CREATE
QUERY
CLOSE
QUERY QUERY
CREATE
QUERY
CLOSE CLOSE CLOSE

Client Server Client Server

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

Why is metadata caching important?

 Metadata intensive “home folders” workload

characterized by FSCT.
 Significant fraction (60%) of the network round
trips involve metadata queries.
 Significant reduction in server load ( improved
server scalability.)
 Reduction in network traffic.
 Overall improvement in user perceived response
times (especially over high latency links.)

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

What is “best effort” caching?

 Maintaining metadata cache on the client without

associated state on the server.
 No explicit metadata cache coherency support in
the SMB2 protocol.
 Leverage existing protocol to achieve “near
consistent” view of metadata.
 Strike a balance between correctness and
performance by caching metadata for the shortest
possible time.

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

What is a “near coherent” cache?

 Present a consistent view of the

cache to single client apps.
 Update or invalidate cache
on I/O (writes, deletes,
renames, set-attributes)
Timeout
Metadata
Cache
 Short cache lifetimes.
 Leverage protocol exchanges as
much as possible.
 Avoid name aliasing issues as
much as possible.

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

6
Scaling cache timeouts

 Scaling based on network characteristics

 Windows 7 client extends cache lifetimes on high
latency links
 Scaling based on protocol exchanges.
 Windows clients extend cache lifetimes if there
are outstanding change notifications posted.
Network Latency Cache Timeout

< 50 msec 10 sec (default)

< 200 msec 120 sec

>= 200 msec 300 sec

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

The 3 metadata caches :
File Information Cache
 Caches file metadata (attributes, size, timestamps)
 FileNetworkOpenInformation structure.
 One entry per file identified by a128-bit ID.
 ID returned via the QFid create context.
 Default lifetime is 10 sec.
 Cache updated via CREATE, CLOSE, QUERY responses
 Cache is invalidated when
 File is locally modified by client app.
 Oplock/Lease break notification from the server.
 Directory change notification from the server.
 Application queries for FileNetworkOpenInformation,
FileBasicInformation and FileStandardInformation satisfied from
cache.
 Serves as a file existence cache.
2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.
The 3 metadata caches :
File Not Found Cache
 Caches names of files which are known not to exist
on the server.
 Allows the client to locally short circuit repeated
opens to the same name.
 Populated on a CREATE which fails with a “object
not found” error.
 Very short lifetime (5 sec.)
 Invalidated when the client creates a new file or
when the server notifies a directory change.
 Very useful in a “compile over network” workload.
 ~ 2000 / 6000 creates failed !
2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.
The 3 metadata caches:
Directory Cache
 Each directory cache entry caches an entire
directory enumeration.
 FileIdBothDirectoryInformation is cached.
 Populated when application enumerates directory.
 Can satisfy information queries for individual files.
 Individual files within a directory cache entry can be
updated / deleted when the client fetches updated
file information from the server.
 Default lifetime is 10 secs.
 Can serve as a file existence / non-existence cache.
 Invalidation is similar to the file information cache.

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

Pitfalls to watch out for!

 Distributed Applications
 Very typical in HPC scenarios which require data sharing
between concurrently running tasks on multiple nodes.
 Producer / consumer type of scenario
 Workaround is to force apps (or system) to use
mechanisms like change notifications.
 Legacy Applications
 Applications which mix long and short names.
 Applications coded for specific behavior patterns.
 Access based enumeration (ABE)
 Different views of a directory based on user.

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

Caching Parameters

 Cache size
 Maximum number of entries
 Elastic thresholds to handle bursts of activity
 Dynamic trimming of cache over time.
 Cache lifetime
 Fixed vs. dynamic
 Tradeoff between performance and correctness.
 Cache eviction policy

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

Scenario 1 : File browsing

The following chart shows the impact (in terms of the # of network frames
exchanged) of the file-info and directory caches for a simple file browsing
scenario of a directory with 60 files.

Frame Count
350
300
250
200
150
100 Frame Count
50
0
No FileInfo Dir+FileInfo
metadata cache Cache
caching

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

Scenario 2 : Engineering workload

 The data (frame counts) in the below table was provided by Marc Ullman
and Ken Harris from MathWorks to highlight the effect of the metadata
cache sizes for a “compile over the network” workload typical of large
engineering environments.
Windows 7 Windows 7 Windows 7 Windows 7
Default Settings (Large MTU enabled) (Large MTU enabled)
FileInfoCacheEntries = 64 FileInfoCacheEntries = 32K FileInfoCacheEntries = 32K FileInfoCacheEntries = 32K
DirectoryCacheEntries = 16 DirectoryCacheEntries = 32K DirectoryCacheEntries = 32K DirectoryCacheEntries = 32K
FileNotFoundCacheEntries=128 FileNotFoundCacheEntries=32K FileNotFoundCacheEntries=32K FileNotFoundCacheEntries=32K
Cache lifetime = 10s, 10s, 5s Cache lifetime = 60s, 60s, 60s Cache lifetime = 60s, 60s, 60s Cache lifetime = 600s, 600s, 600s

DormantFileLimit = 1024 DormantFileLimit = 4096 DormantFileLimit = 4096 DormantFileLimit = 16384

CacheFileTimeout = 10 sec CacheFileTimeout = 10 sec CacheFileTimeout = 600 sec CacheFileTimeout = 3600 sec

CREATE 180302 53860 52920 48396

CLOSE 173078 47030 46631 42359
QUERY_DIR 68580 21103 19536 13495
READ 19340 16474 16448 15976
QUERY_INFO 763 225 217 198
Total Ops 449575 146126 143208 127858
% Improvement
over default N/A 67% 68% 72%

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

 Looking Forward…

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

15
Caching Inefficiency

 A timeout based caching approach forces us to

release a potentially correct cache entry due to
uncertainty of its validity
 A timeout also results in us maintain (and return) an
incorrect cache entry for a period of time, resulting
in application or user confusion
 Both of these issues would exist with file data as
well, but coherency is reinforced with opportunistic
locking and leasing.

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

16
Directory Leasing

File Leasing Directory Leasing

Handle Caching The client is permitted to cache a Same
handle to the file. Revoked if
server receives a conflicting open,
etc.
Read Caching The client is permitted to cache The client is permitted to cache
data read from the file. Revoked if directory enumeration results from
a write or byte-range lock is taken the directory. Revoked if a meta-data
on the file by another client. modifying operation is received for
this directory from another client.
Write Caching The client is granted exclusive
access to the directory, and
permitted to cache writes and byte
range locks. Revoked if another
client attempts to access the file.

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

17
Directory Leasing

 Leasing model allows multiple clients to obtain RH

cache in collaboration or publication scenarios
 In scenarios with a single client (MyDocuments), the
directory cache lifetime will often be infinite
 When a modification is made by another client or a
local user, notification is sent immediately to
improve coherency of clients

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

18
File Lease Key

 Client associates a lease key File A

Lease Key { K(f1) }
for a file with a cache
 Client provides that key for Cached
Data
all opens to the same file
 Server ensures access on an Open #1 Open #2 Open #3
open do not break leases or
oplocks with a matching key File B
Lease Key { K(f2) }
 Coherency is preserved
even if the client is not Cached
Data
aware that File A is the
same as File B. Open #4 Open #5

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

19
Directory Leasing Keys

 For directories, operations on the children modify

the metadata of the parent (create, delete, modify)
 The children may themselves be a directory which
also has children
 Read caching is revoked by a change in my children
 Handle caching is revoked by a conflicting open to
myself
 Client selects lease keys to match their caching
structure (as seen previously)

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

20
Directory Leasing Keys

 Client provides lease key pair { Kself, Kparent }

{ Ka1, 0 }

{ Kb1, Ka1} { Kb2, Ka1 }

{ Kc1, Kb1} { Kc2, Kb1}

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

21
Directory Leasing Key

 Client maintains directory Directory A

Lease Key { K(d1), K(parent) }
cache on directory object,
associates key Cached
Directory
 Opens to children provide
key of parent directory as Open #1 Open #2
Kparent
 Modification of child will File B
Lease Key { K(f1), K(d1) }
not revoke read lease on
parent directory if Kparent of Cached
modified open is equal to Data

Kself of parent directory

Open #4

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

22
Lease Break Algorithm

 Handle Caching
 Handle caching is revoked if Kself of new open is not equal
to Kself of conflicting open at create time. (Same as
specified for opens to files.)
 Delete and rename of folders is the most common issue
that handle caching helps resolve.
 Read Caching
 Read caching is revoked on directory metadata update if
Kparent of the child who triggered the modification is not
equal to Kself of the open on which the least was obtained.
 Create new, modification of metadata, deletion of a file, or
updating timestamps on close would be most common.
2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.
23
Modeled Performance Results

 Modeled reduction of frames in FSCT scenarios

assuming directory oplock is held and cache is
synchronized before running scenario
Directory Oplocks Change Notify Close Create Query Directory

CmdLineFileDelete -2 -3 -2
CmdLineFileDownload -2 -2 -2
CmdLineFileUpload -1 -2
CmdLineNavigate -3 -4 -4
ExplorerDragDropFileDownload -4 -7 -2
ExplorerDragDropFileUpload -6 -8 -4
ExplorerFileDelete -2 -13 -14 -10
ExplorerNavigate -8 -11 -4
ExplorerSelect -1 -1 -2
WordEditAndSave -3 -6
WordFileClose -2 -17 -19 -8
WordFileOpen -21 -25 -14

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

24
Extrapolated Perf Results

Relative FSCT Latency

SMB 2.1 SMB vNext - DirOplock
100.0% 63.7%

 36% client latency improvement compared to

SMB 2.1 with directory caching

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

25
Key Takeaways

 If you are implementing an SMB2 client (or a SMB2 protocol

accelerator), you could employ some of the strategies we use
to ensure “best effort” consistency of the cached metadata.
 Depending on the application workload, the reduction in
network traffic / server load can be very significant.
 The metadata caching parameters can be tweaked on the
Windows SMB2 clients.
 Be aware that there will always be applications that require
very strict metadata consistency guarantees and “best effort”
caching may not work in those cases.
 Directory leasing could solve many issues relating to cache
coherency and helps clients to cache metadata for longer
periods of time, resulting in significant meta-data traffic
reductions.
2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.
 Questions?

Thanks!

2010 Storage Developer Conference. Microsoft Corporation. All Rights Reserved.

27