CIFS/SMB2 Deep-Dive Report

Created for Sample Report

Jun 25th 2013 Jul 8th 2013

Written by: Esben Laursen Hawk-Eye [email protected]

CIFS/SMB2 Deep-Dive Report

Jun 25th 2013 Jul 8th 2013 Sample Report

Page 2 of 13 June 12rd 2013

Contents
Introduction ................................................................................................................................................................. 3 Preface ...................................................................................................................................................................... 3 Executive Summary .................................................................................................................................................... 3 How sessions are counted ........................................................................................................................................... 4 Steelheads in Scope....................................................................................................................................................... 5 Optimization Errors vs. No Errors .............................................................................................................................. 6 Protocol Errors............................................................................................................................................................. 7 Error 261 - SMB2 blade disabled ............................................................................................................................... 7 Error 119 - Security signatures are required on the server .......................................................................................... 7 Error 125 - CIFS parser shutting down due to error ................................................................................................... 8 Error 109 - Negotiate response contains older CIFS dialect ........................................................................................ 8 Error 127 - UNKNOWN_SHUTDOWN_ERROR ......................................................................................................... 8 Top10 Steelhead Peers with Errors ........................................................................................................................... 10 Top10 Servers with Errors ......................................................................................................................................... 11 Top10 Client with Errors ........................................................................................................................................... 12 Apendix - Protocol Error Codes ................................................................................................................................ 13

CIFS/SMB2 Deep-Dive Report

Jun 25th 2013 Jul 8th 2013 Sample Report

Page 3 of 13 June 12rd 2013

Introduction
Preface
This report takes a deep-dive into the CIFS/SMB2 protocol optimization on Steelheads within your estate. This report includes an executive summary as well as details of the error profiles in the network. Additionally we provide recommendations on configuration changes, and best practice advice in order to facilitate fast identification of the Steelheads, servers or clients generating problems and the causes. This will enable the Steelhead administrator to conduct reconfiguration or further troubleshooting quickly and easily, saving time, money and resources.

Executive Summary
We have seen many problems with the CIFS/SMB2 protocol optimization, we have detected an error rate of >55% over the last 14 days. We have seen more than 410.000 sampled sessions with reduced optimization performance. This means that more than 410,000 sampled sessions have not been fully optimized, affecting both network performance and user experience. In general there are 2 issues; a) SMB2 is not enabled and b) SMB-signing is required a) SMB2 is easily remedied as all Steelheads supports SMB2 the configuration needs to be changed so this feature is enabled. b) Two common issues with SMB-signing are that it has not been enabled or it has been configured incorrectly. As the Steelheads are on a recent version of RiOS that supports SMB signing in transparent mode, it is highly recommended to join the all Steelheads as a Read-Only Domain Controller (RODC) and optimize it in transparent mode. If company security policy allows it, an alternative to joining the Steelheads as a RODC is lowering the SMB signing level to enabled from required. If this is implemented, there is no need to make additional configuration changes on the Steelheads as SMB signing will no longer be in use in the network.

CIFS/SMB2 Deep-Dive Report

Jun 25th 2013 Jul 8th 2013 Sample Report

Page 4 of 13 June 12rd 2013

How sessions are counted

Sampled vs. aggregated dataset Sampled dataset means we collect data on at fixed time intervals, only retaining the content of the register at the time of collection. Aggregated dataset means that the device is retaining all data, and when collection takes place we are passed the aggregated values accumulated since the previous collection took place. Which is the more precise method? Aggregated is considered a more valid sample, however if the time period is long enough and enough samples are collected, both methods will deliver similar results and conclusions. How we collect data and analyse We collect data using different methods; the primary method used for this report utilises the CLI (Command Line Interface) on the Steelhead. There is no detailed connection history of closed sessions on the Steelhead at the time data is collected; we only have a sampled dataset for this metric. This means that approximately every 5 minute we log on to the Steelhead and collect a sample of the Current Connection Report and analyse it. This also means that if a session is both opened and closed in between sample data being collected it will not be detected. Conversely this means that if the session is persistent, for example one lasting an hour, it will be counted 12 times in a 60 minute period. The disadvantage of a sampled dataset is that we only see traffic at the time we look (once every 5 minute) and we do not detect data outside this. This is not the most comprehensive method, but it is the only one available to us, and over time it gets more precise meaning the output and conclusions become more robust. We may also, depending on the result extract information from the logs collected (syslog) and/or from SNMP and include this in the report.

CIFS/SMB2 Deep-Dive Report

Jun 25th 2013 Jul 8th 2013 Sample Report

Page 5 of 13 June 12rd 2013

Steelheads in Scope
This is a list of Steelhead that has been analyzed

CIFS/SMB2 Deep-Dive Report

Jun 25th 2013 Jul 8th 2013 Sample Report

Page 6 of 13 June 12rd 2013

Optimization Errors vs. No Errors

In this section we have analyzed how many errors were logged, comparing it to the number of sessions optimized with no errors. Please note that an error is not a session that is broken from the user perspective its an error in the optimization. When we see optimization errors, it means the Steelhead cannot perform layer7 optimization, but is capable of bandwidth optimization only. Without Layer 7 optimization in place users will experience significant performance degradation, in particular this will impact those links and sites with high latency.

In this report, we see that more than 55% of the CIFS/SMB2 sessions were not being optimized as we would expect. It is normal that some sessions will not be optimized successfully, however this figure should be less than 1%, so 55% is extremely high and corrective action should be taken to address this.

CIFS/SMB2 Deep-Dive Report

Jun 25th 2013 Jul 8th 2013 Sample Report

Page 7 of 13 June 12rd 2013

Protocol Errors
When a protocol error occurs, the Steelhead lists a reason why the event occurred. . There will be many different causes, the most common are SMB-signing or SMB2 not enabled, but is can also be that the client or server operating system is not supported. The protocol error code list is the official Riverbed one and is available from the Riverbed knowledge base or from the appendix to this report. This view is global, but in later sections you will be able to see what Steelheads, servers or clients are causing the errors.

In this environment we can see 5 different protocol errors, however only error 261 and 119 is of priority.

Error 261 - SMB2 blade disabled

This error code is generated when the Steelheads SMB2 blade is not enabled. Older Steelheads do not have SMB2 optimization capabilities or enabled by default, and therefore traffic has no layer7 optimization. This error means users experience a significant performance penalty as they can only benefit from the Data Streamlining (data reduction) and not the Application Streamlining (Layer7). SMB2 is supported in RiOS 6.5 and later and Steelhead Mobile 4.0 and later. Recommendations As all Steelheads in this environment run versions of RiOS that support SMB2 optimization it is recommended that the configuration is changed to enable this feature.

Error 119 - Security signatures are required on the server

SMB-signing is a man-in-the-middle countermeasure, preventing an attacker from manipulating traffic in transit. As the Steelhead is a man-in-the-middle device (although the

CIFS/SMB2 Deep-Dive Report

Jun 25th 2013 Jul 8th 2013 Sample Report

Page 8 of 13 June 12rd 2013

good kind), SMB signing prevents the Steelheads from delivering layer 7 optimization. However, since RiOS v5.5 SMB-signing has been supported, but more importantly support for Windows 7 and Windows 2008 servers was not available until RiOS v6.5 was released. Riverbed is continuously upgrading and simplifying the SMB-signing support in the Steelhead. In RiOS v7.x and v8.x Steelheads can be joined as a Read-Only Domain Controller (RODC), this greatly simplifies the configuration and error rate associated with NTLM authentication against the client. If end to end Kerberos support is required RiOS v7 and above has to be used in combination with the Kerberos delegation user. Please refer to the Riverbed tech paper Optimization in a Secure Windows Environment for further details. Recommendations All the Steelheads in this environment are running RiOS 7.0.5d, it is recommended to join all Steelheads as RODC and enable NTLM delegation in transparent mode. This will allow the Steelheads to optimize most clients and servers (Windows XP, Vista, 7, 2003, and 2008) without further configuration. If there is a requirement in the Active Directory for the client to authenticate through the Kerberos protocol, a Kerberos delegation user (with domain replication privileges) must be configured in the active directory and added to the Steelheads. Please refer to the Optimization in a Secure Windows Environment tech paper on guides. You can also (if company security policy allows) downgrade the SMB-signing requirement on the server/client from required to enabled in Active Directory (GPO policy), this will also solve the problem without changing settings on the Steelhead appliances.

Error 125 - CIFS parser shutting down due to error

An unknown error occurred to the optimization. The optimization engine reports the error code but is not aware of any further details about the issue. There is a chance that INFO level logs (if available) can reveal further details. However since the number of sessions that within an acceptable level, it is out of scope of this report. Recommendations There have been so few of these errors (less than 1%) and they are within the anticipated volumes in an installation of this size, no further action is required.

Error 109 - Negotiate response contains older CIFS dialect

The server or the client is talking a CIFS dialect that is not supported (its considered obsolete). Upgrading the server or client is the only path to have enable this optimized with Application Streamlining Recommendations There have been so few of these errors (less than 1%) and they are within the anticipated volumes in an installation of this size, no further action is required.

Error 127 - UNKNOWN_SHUTDOWN_ERROR

An unknown error occurred to the optimization. The optimization engine reports the error code but is not aware of any further details about the issue. There is a chance that INFO level logs (if available) can reveal further details. However since the number of sessions that within an acceptable level, it is out of scope of this report.

CIFS/SMB2 Deep-Dive Report

Jun 25th 2013 Jul 8th 2013 Sample Report

Page 9 of 13 June 12rd 2013

Recommendations There have been so few of these errors (less than 1%) and they are within the anticipated volumes in an installation of this size, no further action is required.

CIFS/SMB2 Deep-Dive Report

Jun 25th 2013 Jul 8th 2013 Sample Report

Page 10 of 13 June 12rd 2013

Top10 Steelhead Peers with Errors

When we analyze a session, data is collected to identify the server-side Steelheads. We use this information to detect Steelheads that have problems, helping you to identify and prioritize the Steelheads that need to be focused on when troubleshooting issues. Note that the IP addresses are the In-path interfaces, this means that the same Steelhead can show up one or more times depending on how many interfaces it has.

This table shows that two Steelheads account for the majority of errors IP Addresses 10.129.0.51 & 10.64.0.5. We recommend these two appliances are prioritized for investigation and corrective action.

CIFS/SMB2 Deep-Dive Report

Jun 25th 2013 Jul 8th 2013 Sample Report

Page 11 of 13 June 12rd 2013

Top10 Servers with Errors

Within this section of the report we identify the servers causing the issues. It could be that a server is running a CIFS/SMB version or operating system that is not supported. Furthermore you can use this section to prioritize between servers/locations that have higher business priority.

The table above identifies one server that generating a high number of issues - 10.64.1.170. This server is running SMB2, however it is very likely that the Steelhead at this location is not enabled for SMB2 optimization (error code 261). The 2nd server on the list - 10.129.1.12 is likely to be a domain controller as it is also generates SMB signing (error code 119).

CIFS/SMB2 Deep-Dive Report

Jun 25th 2013 Jul 8th 2013 Sample Report

Page 12 of 13 June 12rd 2013

Top10 Client with Errors

Within the section of the report we identify the clients that are causing issues. Like the server section it could be that the client is running a CIFS/SMB version or operating system that is not supported. You can also use this section to prioritize between clients/locations that have higher business priority.

Its worth noticing that the Client 10.129.1.106 is by far the most active client (with errors), it is likely that is communicates with the server 10.64.1.170 seen in the previous section. This client is also experiencing common SMB2 issues.

CIFS/SMB2 Deep-Dive Report

Jun 25th 2013 Jul 8th 2013 Sample Report

Page 13 of 13 June 12rd 2013

Apendix - Protocol Error Codes

Protocol Error 102 103 104 109 111 112 113 116 118 119 122 123 124 125 126 127 128 258 259 261 262 263 Error Description Client sent unhandled NT Transact request type Client sent unhandled request type Client sent unhandled Trans2 request type Negotiate response contains older cifs dialect UNKNOWN_SHUTDOWN_ERROR Session setup request contains older cifs dialect SMB_SHUTDOWN_ERR_BAD_SSETUP_REQ See: https://supportkb.riverbed.com/support/index?page=content&id=S15714 Tree connect response contains older cifs dialect Security signatures are enabled on the server Security signatures are required on the server Server returned duplicate FID UNKNOWN_SHUTDOWN_ERROR Unable to optimize SMB2 traffic Cifs parser shutting down due to error - Disabling latency optimization - only bandwidth will be optimized Cifs parser shutting down due to error - Disabling latency optimization - only bandwidth will be optimized UNKNOWN_SHUTDOWN_ERROR UNKNOWN_SHUTDOWN_ERROR UNEXPECTED_SMB2_TRAFFIC SIGNING_IN_USE (SMB2 signing required) SMB2 blade disabled SMB2 Connection Blacklisted UNSUPPORTED_DIALECT