N

ot
fo
rr
es
al
e
or
di

Citrix NetScaler 11.0 Advanced

s
tri

Implementation
bu
tio

CNS-301-3I
n
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

2 © Copyright 2016 Citrix Systems, Inc.

Citrix NetScaler 11.0 Advanced
N

Implementation
ot
fo

CNS-301-3I
rr
es

March 2016
al

Version 3.0
e
or
di
s
tri
bu
tio
n
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

4 © Copyright 2016 Citrix Systems, Inc.

Table of Contents
Module 1: Advanced Troubleshooting .......................................................... 21
Advanced Troubleshooting Manual ....................................................................................... 23
Overview ........................................................................................................................... 23
Objectives ......................................................................................................................... 23
Troubleshooting Resources .............................................................................................. 23
Citrix Knowledge Center ................................................................................................... 23
Citrix Product Documentation ........................................................................................... 24
Auto Support .................................................................................................................... 25
Citrix Technical Support .................................................................................................... 27
Collected NetScaler Data .................................................................................................. 27
Troubleshooting Log ......................................................................................................... 27
N

NetScaler System Overview .............................................................................................. 28

ot

nCore Configuration Architecture ...................................................................................... 29

Built-In Tools ..................................................................................................................... 30
fo

Diagnostics Pane Overview ............................................................................................... 31

rr

NetScaler Logs ................................................................................................................. 32

Syslog ............................................................................................................................... 32
es

NetScaler Configuration Utility ........................................................................................... 32

Command-line Interface .................................................................................................... 32
al

Nslog ................................................................................................................................ 34
e

Real-Time Performance Statistics ..................................................................................... 39

Dashboard ........................................................................................................................ 40
or

Stat Commands ............................................................................................................... 41

Show Commands ............................................................................................................. 42
di

NetScaler Configuration Utility ........................................................................................... 44

s

NetScaler Visualizer ......................................................................................................... 45

tri

Historical Statistics ............................................................................................................ 46

bu

Reporting Tool .................................................................................................................. 47

Reporting Tool Components ............................................................................................. 47
t

Built-in Reports ................................................................................................................. 48

io

Custom Reports ............................................................................................................... 49

n

Charts ............................................................................................................................... 49
Data Collection ................................................................................................................. 49
Command Center and SNMP ........................................................................................... 50
Network Traffic Capture .................................................................................................... 50
Nstrace Options ................................................................................................................ 51
Default Settings ................................................................................................................ 51
Link Parameter Example ................................................................................................... 54
Trace Filter Syntax ............................................................................................................ 54
Creating a Trace File in the Configuration Utility ................................................................ 55
Shell Tools ........................................................................................................................ 56
Command-Line Interface Tools ......................................................................................... 58

© Copyright 2016 Citrix Systems, Inc. 5

 Third-Party Tools .............................................................................................................. 59
Network Protocol Analyzers ............................................................................................. 60
Web Browser Plug-ins ...................................................................................................... 60
SNMP Browsers ............................................................................................................... 60
FTP Clients ....................................................................................................................... 60

Module 2: Introducing Application Firewall .................................................... 61

Introducing Application Firewall Manual ................................................................................ 63
Overview ........................................................................................................................... 63
Objectives ......................................................................................................................... 63
Application Attacks ........................................................................................................... 64
Application Attack Description .......................................................................................... 64
Goals of Application Attacks ............................................................................................. 65
Most Common Types of Web Application Attacks ............................................................ 65
SQL Injection Attack ......................................................................................................... 66
N

Cross-Site Scripting .......................................................................................................... 67

ot

The Application Firewall Solution ....................................................................................... 68

Business Problems ........................................................................................................... 68
fo

The Benefits of Application Firewall ................................................................................... 69

rr

Application Layer Protection ............................................................................................. 70

Positive Security Model ..................................................................................................... 71
es

Negative Security Model ................................................................................................... 72

Deep Stream Inspection ................................................................................................... 72
al

Adaptive Learning Engine ................................................................................................. 73

e

Web Application Vulnerabilities ......................................................................................... 74

or

Security Audits and Application Firewalls .......................................................................... 74

Payment Card Industry Data Security Standard ................................................................ 75
di

Importance of PCI ............................................................................................................. 75

Common Coding Vulnerabilities ........................................................................................ 75
s

PCI-DSS Report ............................................................................................................... 78

tri

Packet Processing and Inspection .................................................................................... 78

bu

Request Process .............................................................................................................. 79

Response Process ............................................................................................................ 80
t io

Deployment Considerations .............................................................................................. 81

Profiles and Policies .......................................................................................................... 81
n

Profiles .............................................................................................................................. 81
Policies ............................................................................................................................. 82

Module 3: Profiles and Policies ..................................................................... 83

Profiles and Policies Manual ................................................................................................. 85
Overview ........................................................................................................................... 85
Objectives ......................................................................................................................... 85
Profiles .............................................................................................................................. 86
Profile Types ..................................................................................................................... 87
Default Profiles .................................................................................................................. 87

6 © Copyright 2016 Citrix Systems, Inc.

 Basic Profiles .................................................................................................................... 88
Advanced Profiles ............................................................................................................. 88
Creating a Profile in the Configuration Utility ...................................................................... 88
Creating a Profile in the Command-Line Interface ............................................................. 89
Action Settings ................................................................................................................. 90
Block ................................................................................................................................ 90
Logs ................................................................................................................................. 91
Statistics ........................................................................................................................... 91
Learn ................................................................................................................................ 91
Additional Action Settings ................................................................................................. 91
Modifying Action Settings in the Configuration Utility ......................................................... 92
Modifying Action Settings Using the Command-Line Interface .......................................... 92
Modifying Relaxations in the Configuration Utility .............................................................. 93
Sessionization and Security Checks ................................................................................. 94
How Sessionization Works ................................................................................................ 94
Profile Settings .................................................................................................................. 95
N

Error Page ........................................................................................................................ 96

ot

Setting the URL Redirect in the Configuration Utility ......................................................... 96

Setting the URL Redirect in the Command-Line Interface ................................................. 96
fo

HTML Comment Stripping ................................................................................................ 97

rr

Stripping HTML Comments in the Configuration Utility ...................................................... 97

Stripping HTML Comments in the Command-Line Interface ............................................. 97
es

XML Error Object .............................................................................................................. 98

Other Profile Settings ........................................................................................................ 98
al

Policies ............................................................................................................................. 98
e

Policy Creation .................................................................................................................. 99

or

Policy Binding ................................................................................................................... 99

Policy State .................................................................................................................... 100
di

Policy Priorities ............................................................................................................... 100

Creating a Policy in the Configuration Utility .................................................................... 100
s

Creating Policies in the Command-Line Interface ............................................................ 101

tri

Binding and Prioritizing a Policy in the Configuration Utility ............................................. 102

bu

Binding Policies in the Command-Line Interface ............................................................. 102

Engine Settings ............................................................................................................... 103
t io
n

Module 4: Regular Expressions .................................................................. 105

Regular Expressions Manual ............................................................................................... 107
Overview ......................................................................................................................... 107
Objectives ....................................................................................................................... 107
Regular Expressions ....................................................................................................... 108
Forms of Regular Expressions ........................................................................................ 109
Using Regular Expressions ............................................................................................. 109
Metacharacters and Literal Characters ........................................................................... 110
Metacharacters ............................................................................................................... 110
Escapes .......................................................................................................................... 113
Quantifiers ....................................................................................................................... 113

© Copyright 2016 Citrix Systems, Inc. 7

 Backreferencing .............................................................................................................. 114
Lookaheads .................................................................................................................... 115
Lookahead Example 1 .................................................................................................... 115
Lookahead Example 2 .................................................................................................... 115
Lookahead Example 3 .................................................................................................... 116
Lookahead Complexity ................................................................................................... 117
Regular Expression Scope .............................................................................................. 117
Practice: Interpreting Regular Expressions ...................................................................... 118
Practice: Creating Regular Expressions .......................................................................... 119

Module 5: Attacks and Protections ............................................................ 121

Attacks and Protections Manual ......................................................................................... 123
Overview ......................................................................................................................... 123
Objectives ....................................................................................................................... 123
Security Checks .............................................................................................................. 123
N

Profile Types ................................................................................................................... 124

ot

Common Security Checks .............................................................................................. 124

HTML Security Checks ................................................................................................... 126
fo

XML Security Checks ..................................................................................................... 127

rr

Request-Side and Response-Side Checks ..................................................................... 129

Response-Side Checks .................................................................................................. 129
es

HTTPS Web Applications ................................................................................................ 130

Buffer Overflow Exploits .................................................................................................. 131
al

Goals of a Buffer Overflow Attack ................................................................................... 131

e

Consequences of a Buffer Overflow Attack ..................................................................... 132

or

Buffer Overflow Protection .............................................................................................. 132

Default Maximum Values ................................................................................................. 133
di

Modifying Buffer Overflow Settings ................................................................................. 133

Modifying Buffer Overflow Settings in the Configuration Utility ......................................... 133
s

Modifying Buffer Overflow Settings in the Command-Line Interface ................................ 134

tri

Parameter Manipulation .................................................................................................. 134

bu

Parameter Manipulation Example .................................................................................... 135

Server Misconfiguration .................................................................................................. 135
t io

Deny URL Protection ...................................................................................................... 136

The Deny URL List .......................................................................................................... 136
n

Adding a Deny URL in the Command-Line Interface ....................................................... 137

Deleting a Deny URL in the Command-Line Interface ..................................................... 138
SQL Injection .................................................................................................................. 139
How SQL Injection Works ............................................................................................... 139
HTML SQL Injection Protection ....................................................................................... 140
SQL Keywords and Special Characters .......................................................................... 141
Modifying SQL Injection Action Settings .......................................................................... 141
Modifying SQL Injection Action Settings in the Configuration Utility ................................. 142
Modifying SQL Injection Action Settings in the Command-Line Interface ........................ 142
SQL Comments Handling ............................................................................................... 143
Relaxations ..................................................................................................................... 144

8 © Copyright 2016 Citrix Systems, Inc.

 Adding a SQL Injection Relaxation in the Command-Line Interface ................................. 145
Deleting a SQL Injection Relaxation Using the Command-Line Interface ......................... 145
XML SQL Injection Security Check ................................................................................. 146
Cross-Site Scripting ........................................................................................................ 147
Attacking the Trust Relationship ..................................................................................... 147
How Cross-Site Scripting Attacks Work ......................................................................... 148
Results of a Cross-Site Scripting Attack ......................................................................... 148
Preventing Cross-Site Scripting Attacks .......................................................................... 148
HTML Cross-Site Scripting Protection ............................................................................ 149
Cross-Site Scripting Action Settings ............................................................................... 149
Transform Cross-Site Scripts .......................................................................................... 149
Check Complete URLs for Cross-Site Scripting .............................................................. 150
Additional Action Settings ............................................................................................... 150
Relaxations ..................................................................................................................... 150
Modifying Cross-Site Scripting Action Settings ............................................................... 150
Modifying Additional Action Settings Procedure .............................................................. 151
N

Modifying Cross-Site Scripting Action Settings Using the Command-Line Interface ........ 151
ot

Adding a Cross-Site Scripting Relaxation Using the Command-Line Interface ................ 152
Deleting a Cross-Site Scripting Relaxation Using the Command-Line Interface ............... 152
fo

XML Cross-Site Scripting Security Check ....................................................................... 153

rr

Command Injection ......................................................................................................... 153

Command Injection Examples ........................................................................................ 154
es

Field Format Protection ................................................................................................... 154

Field Types and Field Formats ........................................................................................ 155
al

Predefined Field Types ................................................................................................... 155

e

Custom Field Types ........................................................................................................ 156

or

Field Format Configuration .............................................................................................. 156

Default Field Format ........................................................................................................ 157
di

Confidential Fields ........................................................................................................... 157

Adding a Custom Field Type ........................................................................................... 157
s

Adding a Custom Field Type Procedure ......................................................................... 157

tri

Adding a Custom Field Type Using the Command-Line Interface ................................... 158
bu

Setting a Default Field Type ............................................................................................ 158

Setting a Default Field Type Procedure ........................................................................... 158
t io

Setting a Default Field Type Using the Command-Line Interface ..................................... 159
Modifying Field Format Settings ...................................................................................... 160
n

Modifying Field Format Settings Procedure ..................................................................... 160

Adding a Field Format Using the Command-Line Interface ............................................. 161
Deleting a Field Format Using the Command-Line Interface ............................................ 162
Adding a Confidential Field ............................................................................................. 162
Adding a Confidential Field Procedure ............................................................................ 163
Adding a Confidential Field Using the Command-Line Interface ...................................... 163
Modifying a Confidential Field ......................................................................................... 164
Modifying a Confidential Field Procedure ........................................................................ 164
Modifying a Form Field Using the Command-Line Interface ............................................ 164
Cookie Tampering and Poisoning ................................................................................... 165
Types of Cookies ............................................................................................................ 165

© Copyright 2016 Citrix Systems, Inc. 9

 How Cookies Are Added ................................................................................................ 166
Web Server Sessions ...................................................................................................... 167
Cookie Consistency Protection ....................................................................................... 168
Sessionization and Cookies ............................................................................................ 169
Relaxations ..................................................................................................................... 170
Adding a Cookie Consistency Relaxation in the Command-Line Interface ....................... 171
Deleting a Cookie Relaxation in the Command-Line Interface ......................................... 171
Form/Hidden Field Manipulation ..................................................................................... 172
Example of Hidden Field Manipulation ............................................................................ 173
Form Field Consistency Protection ................................................................................. 174
Field Consistencies ......................................................................................................... 175
User Sessions ................................................................................................................. 175
Adding a Form Field Consistency Relaxation Using the Command-Line Interface ........... 175
Deleting a Form Field Consistency Relaxation Using the Command-Line Interface ......... 176
Forceful Browsing ........................................................................................................... 177
Forceful Browsing Protection .......................................................................................... 177
N

Start URLs ...................................................................................................................... 178

ot

The Start URL List .......................................................................................................... 179

Sessionization and Start URLs ........................................................................................ 179
fo

Modify Start URL Check ................................................................................................. 179

rr

Adding a Start URL in the Command-Line Interface ....................................................... 179

Deleting a Start URL in the Command-Line Interface ...................................................... 180
es

Backdoors and Misconfigurations ................................................................................... 180

Threats of Backdoors to an Environment ........................................................................ 181
al

Conventional and Unconventional Backdoors ................................................................. 181

e

Debug Code, Binaries, and Options ............................................................................... 181

or

Debug Options Attack Example ...................................................................................... 182

Application Firewall and Backdoors and Debug Options ................................................. 182
di

Broken Authentication ..................................................................................................... 182

Broken Access Control Lists and Weak Passwords ........................................................ 182
s

Protecting Against Misconfigurations .............................................................................. 183

tri

URL Closure ................................................................................................................... 183

bu

Enforcing URL Closure in the Configuration Utility ........................................................... 184

Enforcing URL Closure in the Command-Line Interface .................................................. 184
t io

Identity Theft Attacks ...................................................................................................... 185

Types of Identity Theft Attacks ........................................................................................ 185
n

Application Firewall Protection Against Identity Theft ....................................................... 186

Credit Card Protection .................................................................................................... 186
Predefined Credit Cards ................................................................................................. 187
Credit Card Settings ....................................................................................................... 187
Protecting Credit Cards .................................................................................................. 187
Protecting Credit Cards in the Configuration Utility ......................................................... 187
Protecting Credit Cards in the Command-Line Interface ................................................. 188
Errors Triggering Sensitive Information Leaks .................................................................. 189
Error Messages Resulting in Attacks ............................................................................... 189
Reducing the Risk of Information Leaks .......................................................................... 189
Safe Object Protection .................................................................................................... 190

10 © Copyright 2016 Citrix Systems, Inc.

 Defining a Safe Object .................................................................................................... 190
Adding a Safe Object ...................................................................................................... 191
Adding a Safe Object in the Configuration Utility ............................................................. 191
Adding a Safe Object in the Command-Line Interface .................................................... 191
Adaptive Learning for Security ........................................................................................ 193
Learning Over Time ........................................................................................................ 194
Learning Thresholds ....................................................................................................... 194
Generalized and Simple Rules ........................................................................................ 194
Learned Rules ................................................................................................................ 195
Enabling Learning ........................................................................................................... 195
Enabling Learning in the Configuration Utility .................................................................. 195
Enabling Learning in the Command-Line Interface .......................................................... 195
Setting Learning Thresholds ........................................................................................... 196
Setting Learning Thresholds in the Configuration Utility ................................................... 196
Managing Learned Rules ................................................................................................ 197
Managing Learned Rules in the Configuration Utility ....................................................... 197
N
ot

Module 6: Application Firewall Troubleshooting .......................................... 199

fo

Application Firewall Troubleshooting Manual ....................................................................... 201

rr

Overview ......................................................................................................................... 201

Objectives ....................................................................................................................... 201
es

Application Firewall and Applications .............................................................................. 201

HTTP Headers ................................................................................................................ 201
al

Dropped Request Headers ............................................................................................. 202

e

Dropped Response Headers .......................................................................................... 203

or

Modified Response Headers ........................................................................................... 203

Added Response Headers .............................................................................................. 203
di

HTML Comment Stripping .............................................................................................. 203

Configuration Issues ....................................................................................................... 204
s

Policy Issues ................................................................................................................... 204

tri

Profile Issues .................................................................................................................. 205

bu

Suggested Actions ......................................................................................................... 205

t io

Module 7: Queuing and Connection Tuning ............................................... 207

n

Queuing and Connection Tuning Manual ............................................................................ 209

Overview ......................................................................................................................... 209
Objectives ....................................................................................................................... 209
HTTP Connections ......................................................................................................... 209
Keep-alive HTTP Connections ........................................................................................ 209
HTTP 1.0 and 1.1 Behavior ............................................................................................ 210
Pipelined Requests ......................................................................................................... 210
HTTP Connection Management and NetScaler HTTP Behavior ...................................... 210
Client Keep-Alive ............................................................................................................. 211
Connection IP Address Control ....................................................................................... 211
Maximum Requests and Maximum Connections ............................................................ 211

© Copyright 2016 Citrix Systems, Inc. 11

 Connection Idle Settings ................................................................................................. 212
Trackable Connections ................................................................................................... 213
TCP Buffering ................................................................................................................. 213
Down-State Flush and Access Down Connection Settings ............................................. 214
TCP Optimization ............................................................................................................ 214
Advertised Window Size ................................................................................................. 214
Window Scaling .............................................................................................................. 214
Selective Acknowledgement ........................................................................................... 215
Surge Queue .................................................................................................................. 215
Surge Protection ............................................................................................................. 215
Request and Response Rates ........................................................................................ 215
Throttle Rate ................................................................................................................... 216
Disabling Surge Protection in the Configuration Utility ..................................................... 216
Disabling Surge Protection for a Service in the Configuration Utility ................................ 216
Setting Thresholds in the Configuration Utility ................................................................. 217
Priority Queuing .............................................................................................................. 217
N

Enabling Priority Queuing in the Configuration Utility ....................................................... 218

ot

Creating a Priority Queuing Policy in the Configuration Utility .......................................... 218

Binding Priority Queuing Policies in the Configuration Utility ............................................ 219
fo

Weighted Queuing .......................................................................................................... 219

rr

HTTP Denial-of-Service Protection .................................................................................. 220

Enabling HTTP DoS Protection in the Configuration Utility .............................................. 221
es

Adding a HTTP DoS Policy in the Configuration Utility .................................................... 221

Challenged JavaScript Responses .................................................................................. 221
al

Client Detection Tuning and JavaScript Challenge Response Rate ................................. 222
e

HTTP DoS Protection Deployment Guidelines ................................................................ 222

or

Attack Characteristics ..................................................................................................... 223

di

Module 8: Authentication, Authorization, and Auditing ................................ 225

s

Authentication, Authorization, and Auditing Manual ............................................................ 227

tri

Overview ......................................................................................................................... 227

bu

Objectives ....................................................................................................................... 227

Users, Groups, and Command Policies .......................................................................... 227
t io

Authentication, Authorization, and Auditing ..................................................................... 227

System and AAA User Groups ........................................................................................ 228
n

Local Accounts ............................................................................................................... 230

External Authentication ................................................................................................... 230
External Authentication for System Users ....................................................................... 231
Authentication Actions and Policies ................................................................................ 231
Configuring Local Authentication ..................................................................................... 232
Configuring External Authentication with Explicit Accounts ............................................. 232
Configuring External Authentication with Group Extraction .............................................. 233
Creating an External Authentication Policy ...................................................................... 233
Creating local groups in the Command-Line Interface .................................................... 234
Binding Groups in the Command-Line Interface ............................................................. 234
Command Policies .......................................................................................................... 234

12 © Copyright 2016 Citrix Systems, Inc.

 Example 1 ...................................................................................................................... 235
Example 2 ...................................................................................................................... 235
Creating an LDAP Authentication Action in the Command-Line Interface ........................ 236
Creating an Authentication Policy in the Command-Line Interface .................................. 237
Binding the Policy in the Command Line Interface .......................................................... 238
Authentication Troubleshooting ....................................................................................... 238
External Authentication Common Issues ......................................................................... 238
AAA for Application Traffic .............................................................................................. 239
Enabling AAA for Application Traffic ................................................................................ 239
AAA for Application Traffic .............................................................................................. 240
Basic AAA Setup for Application Traffic .......................................................................... 241
Workflow for AAA Application Traffic ............................................................................... 242
Configuration .................................................................................................................. 243
Creating an Authentication Virtual Server ....................................................................... 243
Creating an Authentication Virtual Server in the Command-Line Interface ....................... 244
Binding an SSL Certificate in the Command-Line Interface ............................................. 244
N

Binding a Virtual Server to an Authentication Policy in the Command-Line Interface ....... 245
ot

Configuring a Virtual Server to use an Authentication Virtual Server in the Command-Line

Interface .......................................................................................................................... 245
fo

Configuring the Authentication Virtual Server in the Configuration Utility .......................... 246
rr

Configuring Authorization Policies for Traffic Management .............................................. 247

Setting the Default Traffic Management Authentication Action to Deny in the Command-Line
es

Interface .......................................................................................................................... 247

Creating an Authorization Policy to Allow Access ............................................................ 247
al

SAML Authentication ...................................................................................................... 248

e

Audit Logging ................................................................................................................. 249

or

Audit Logging Troubleshooting ....................................................................................... 251

di

Module 9: AppExpert Rate Limiting, HTTP Service Callout, and Policy-based

s

Logging ...................................................................................................... 253

tri

AppExpert Rate Limiting, HTTP Service Callout, and Policy-based Logging Manual ........... 255
bu

Overview ......................................................................................................................... 255

HTTP Callout Overview ................................................................................................... 255
t io

HTTP Callouts ................................................................................................................ 256

n

Configuring HTTP Callouts .............................................................................................. 257

Callout Examples ............................................................................................................ 258
HTTP Callout Response Parsing ..................................................................................... 259
Avoiding HTTP Callout Recursion ................................................................................... 259
HTTP Callout Use Cases ................................................................................................ 260
Scenario 1: Filter Clients Based on an IP Address Blacklist ............................................ 260
Scenario 2: Fetch and Update content ........................................................................... 261
Creating HTTP-Callout-2 on the NetScaler System ......................................................... 261
Adding the Rewrite Action .............................................................................................. 261
Creating the Rewrite Policy ............................................................................................. 262
Binding the Rewrite Policy Globally in the Configuration Utility ........................................ 262
HTTP Callout Auditing ..................................................................................................... 262

© Copyright 2016 Citrix Systems, Inc. 13

 Configuring Rate Control ................................................................................................ 262
Limit Mode ...................................................................................................................... 263
Creating Rate-Limiting Policies ....................................................................................... 264
Practice A ....................................................................................................................... 264
Rate Control Policy Scenarios ......................................................................................... 265
Rate Control by Subnet .................................................................................................. 265
Using Cache as a Rate Control Action Example ............................................................. 266
Practice B ....................................................................................................................... 267
Policy Based Logging ..................................................................................................... 267
To create an audit message action ................................................................................. 268
Binding Audit Message Action to a Policy ....................................................................... 268

Module 10: Command Center .................................................................... 269

Command Center Manual ................................................................................................... 271
Overview ......................................................................................................................... 271
N

Objectives ....................................................................................................................... 271

ot

Command Center Introduction ....................................................................................... 271

Command Center Editions .............................................................................................. 271
fo

Command Center Features ............................................................................................ 272

rr

NetScaler and CloudBridge Support ............................................................................... 274

Command Center Installation types ............................................................................... 274
es

Command Center Modes .............................................................................................. 275

High Availability .............................................................................................................. 275
al

Server Requirements ...................................................................................................... 276

e

Disk Space Requirements ............................................................................................... 277

or

Command Center Management Communication ............................................................ 277

High Availability Communication ..................................................................................... 279
di

Command Center Installation .......................................................................................... 279

Capacity Planning ........................................................................................................... 279
s

Backup ........................................................................................................................... 280

tri

Installation Modes ........................................................................................................... 280

bu

Standalone Mode ........................................................................................................... 280

Service Mode ................................................................................................................. 281
t io

Installation Considerations .............................................................................................. 281

Command Center Functionality ....................................................................................... 282
n

Command Center Home Page ....................................................................................... 282

Discovery ........................................................................................................................ 283
Adding a New Device ..................................................................................................... 283
Discovery Process .......................................................................................................... 284
Discovery Settings .......................................................................................................... 287
Citrix Network and Device Management ......................................................................... 287
Troubleshooting Discovery Issues ................................................................................... 288
Fault Management .......................................................................................................... 289
SNMP ............................................................................................................................. 289
Events ............................................................................................................................. 290
Alarms ............................................................................................................................ 290

14 © Copyright 2016 Citrix Systems, Inc.

 Annotations ..................................................................................................................... 291
Event and Alarm Triggers ................................................................................................ 291
Monitoring ....................................................................................................................... 291
Syslogs ........................................................................................................................... 293
Configuration Management ............................................................................................. 293
Built In Tasks .................................................................................................................. 293
Custom Tasks ................................................................................................................ 294
Creating Custom Tasks from a Task File ........................................................................ 295
Creating Custom Tasks from a Command File ............................................................... 295
Creating New Tasks ....................................................................................................... 295
Task Operations ............................................................................................................. 296
Execute Sequentially ....................................................................................................... 296
Execute on Inaccessible System ..................................................................................... 297
Enable RBA .................................................................................................................... 297
Enable Auto Rollback ...................................................................................................... 297
Executing Custom Tasks ................................................................................................ 298
N

Change Management ..................................................................................................... 298

ot

Management Key Components ...................................................................................... 298

Audit Report Work-Flow .................................................................................................. 299
fo

Reports ........................................................................................................................... 301

rr

Centralized Certificate Management ............................................................................... 301

Threshold Configuration .................................................................................................. 302
es

Performance Monitoring and Reporting .......................................................................... 302

Configuring Performance Monitoring ............................................................................... 302
al

Configuring Counters for Polling ..................................................................................... 303

e

Counter Types ................................................................................................................ 305

or

Polling Counters ............................................................................................................. 305

Data Consolidation ......................................................................................................... 305
di

Trend Reports ................................................................................................................ 306

Thresholds ...................................................................................................................... 307
s

Performance Data Issues ................................................................................................ 307

tri

Command Center Administration .................................................................................... 307

bu

Security Administration ................................................................................................... 308

User Administration ......................................................................................................... 308
t io

Authentication ................................................................................................................. 308

Group Administration Overview ....................................................................................... 308
n

Administration Configuration Authentication .................................................................... 309

Group Administration ...................................................................................................... 309
Command Center Administration Node .......................................................................... 310
Additional Administration Tasks ..................................................................................... 311
Administration Configuration ........................................................................................... 312
Discovery Settings .......................................................................................................... 312
Trap Forward Settings .................................................................................................... 313

Module 11: Insight Center .......................................................................... 317

Insight Center Manual ......................................................................................................... 319

© Copyright 2016 Citrix Systems, Inc. 15

 Insight Center Overview .................................................................................................. 319
Objectives ....................................................................................................................... 319
AppFlow on the NetScaler System ................................................................................. 320
How AppFlow Works ...................................................................................................... 322
How Insight Center Collects AppFlow Data .................................................................... 322
HDX Insight ..................................................................................................................... 324
HTML Injection ................................................................................................................ 324

Module 12: NetScaler Web Logging ........................................................... 327

NetScaler Web Logging Manual ......................................................................................... 329
Overview ......................................................................................................................... 329
Objectives ....................................................................................................................... 329
NetScaler Web Logging Introduction .............................................................................. 329
Architecture Overview ..................................................................................................... 330
Communication Process ................................................................................................. 330
N

NetScaler System Configuration ..................................................................................... 331

ot

Enabling Web Logging in the Configuration Utility ........................................................... 331

Enabling Web Logging in the Command-Line Interface .................................................. 332
fo

Configuring the Buffer Size in the Configuration Utility ..................................................... 332

rr

Configuring the Buffer Size in the Command-Line Interface ............................................ 332

NSWL Client Installation .................................................................................................. 332
es

Logging System Components ........................................................................................ 333

Installing the NSWL Client on Windows .......................................................................... 333
al

NSWL Options ................................................................................................................ 334

e

Windows Service Registry Key ........................................................................................ 335

or

NSWL Client Configuration ............................................................................................. 335

NetScaler IP Addresses .................................................................................................. 336
di

Adding NetScaler IP Addresses ...................................................................................... 336

Log Filters ....................................................................................................................... 336
s

Creating Filters ................................................................................................................ 337

tri

Default Filters .................................................................................................................. 338

bu

Defining Log Properties .................................................................................................. 338

Default Settings for Log Properties ................................................................................. 340
t io

Running NSWL ............................................................................................................... 341

Verifying the Configuration .............................................................................................. 341
n

Troubleshooting Web Logging ........................................................................................ 341

NSWL Troubleshooting ................................................................................................... 342
NetScaler Troubleshooting .............................................................................................. 342
Buffer Overflow ............................................................................................................... 343

Module 13: Appendix A: Troubleshooting Common Issues ........................ 345

Troubleshooting Common Issues ....................................................................................... 347
Common Issues .............................................................................................................. 347
High Availability ............................................................................................................... 347
Configuration Synchronization Failure ............................................................................. 347

16 © Copyright 2016 Citrix Systems, Inc.

 File Synchronization Failure ............................................................................................. 348
Unexpected Failover ....................................................................................................... 349
Load Balancing ............................................................................................................... 349
Uneven Load Balancing .................................................................................................. 349
Service/Virtual Server Flapping ........................................................................................ 350
SSL Offloading ................................................................................................................ 351
Access to SSL VIP Address Failing ................................................................................. 351
Certificate-Related Warnings Occurring .......................................................................... 352
Intermediate Certificate Not Being Properly Linked ......................................................... 352
Browser Warning Showing an Insecure Web Page ......................................................... 353
Certificates Expiring ........................................................................................................ 353
Content Switching .......................................................................................................... 353
Global Server Load Balancing ......................................................................................... 353
Networking ..................................................................................................................... 354
Duplex Mismatch or Misconfigured Interface Settings ..................................................... 354
Inaccessible Content ...................................................................................................... 355
N

Caching .......................................................................................................................... 356

ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

© Copyright 2016 Citrix Systems, Inc. 17

Credits
Role Name
Instructional Designers: Howard Weise, Anton Mayers, Christopher
Rudolph

Graphic Artists: Benjamin Abraham, Veronica Fuentes, Ryan

Flowers

Manager: Ilan Belehssen

Editor: Anton Mayers

N

Subject Matter Experts: Jeff Apsley, Justin Aspley, Arvind Bangari, Paul
ot

Blitz, Mark Borrow, Erik Brandsberg, Colin

Christy, John Daniels, John Dell, Greg Dolan,
fo

Stefan Drege, Seema Vaibhav Dubey, Abhishek

rr

Gautam, Roland Geldner, Bino Gopal,

DeeLayna Hurst, Todd Hurst, Faisal Jahan,
es

Vamsi Korrapati, Prakash Mana, James Nagy,

Ronan O’Brien, Lokaraj Pedapalli, Glenn Porter,
al

Ram Prasad, Patrick Quinlan, Prabhu Rampur,

e

Kumaresan Rangasamy, Anoop Reddy, Guy

Rosefelt, Rhonda Rowland, Jacob Salassi,
or

Kawaljit Singh, Prakash Sinha, Erin Smith, Sam

Spence, Thilak Subburam, Raghu Varma
di

Tirumalaraju, Bjarne Traeholt, Chad Tripod,

s

Abhilash Verma, Gregor Visconty, Kit Wetzler,

tri

Don Williams, Lena Yarovaya, Tony Zhang

bu
tio
n
Notices
Citrix Systems, Inc. (Citrix) makes no representations or warranties with respect to the content or
use of this publication. Citrix specifically disclaims any expressed or implied warranties,
merchantability, or fitness for any particular purpose. Citrix reserves the right to make any changes
in specifications and other information contained in this publication without prior notice and
without obligation to notify any person or entity of such revisions or changes.
© Copyright 2015 Citrix Systems, Inc. All Rights Reserved.
No part of this publication may be reproduced or transmitted in any form or by any means,
electronic or mechanical, including photocopying, recording, or information storage and retrieval
systems, for any purpose other than the purchaser’s personal use, without express written
permission of:
Citrix Systems, Inc. 851 West Cypress Creek Road Fort Lauderdale, FL 33309 http://www.citrix.com
N

The following marks are service marks, trademarks or registered trademarks of their respective
ot

owners in the United States and other countries.

fo

Mark Owner
rr
es

Active Directory®, Microsoft®, Microsoft Internet Microsoft Corporation

Explorer®, Windows®, Win32™
al
e

ActivePerl® ActiveState Software, Inc.

or

American Express® American Express Company

di

Apache™ The Apache Software Foundation

s tri

Citrix®, Citrix NetScaler Gateway™, Citrix Citrix Systems, Inc.

Application Firewall™, Citrix Authorized
bu

Learning Center™, Citrix Certified

t

Administrator™, Citrix Certified Enterprise

io

Administrator™, Citrix Certified Integration

n

Architect™, EdgeSight®, ICA®, NetScaler®,

MyCitrix™

Diners Club® Diners Club International Ltd.

Discover® Discover Financial Services

Firefox® Mozilla Corporation

FreeBSD® FreeBSD Foundation

Google™ Google Inc.

Mark Owner
Intel®, Pentium® Intel Corporation

Java® Sun Microsystems, Inc.

JCB® JCB International Co., Ltd.

Linux® Linus Torvalds

LiveHTTPHeaders® Mozdev Community Organization, Inc.

MasterCard® MasterCard Worldwide

Microsoft®, .NET™, Active Directory®, Internet Microsoft Corporation

N

Explorer®, SQL Server®, Windows®

ot

Pearson VUE® Pearson Education, Inc.

fo

PuTTY® Simon Tatham

rr

Secure Shell®, SSH® SSH Communications Security Corp.

es

UNIX® The Open Group

al
e

Visa® Visa Inc.

or

WinSCP® Martin Prikryl, GNU General Public License,

Free Software Foundation, Inc.
di
s tri

Other product and company names mentioned herein might be the service marks, trademarks or
registered trademarks of their respective owners in the United States and other countries.
bu
t io
n
 1
Module 1

Advanced
Troubleshooting
N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

22 © Copyright 2016 Citrix Systems, Inc.

Advanced Troubleshooting Manual
Overview
When troubleshooting a NetScaler implementation, an administrator should:
• Reference troubleshooting resources, such as the Citrix Knowledge Center, product
documentation, Citrix Technical Support, collected NetScaler data and internal troubleshooting
logs.
• Be familiar with the NetScaler system architecture and processes.
• Use troubleshooting tools, including those available through the NetScaler system and through
third parties.
N

Objectives
ot
fo

After completing this module, you will be able to:

rr

• Use troubleshooting resources.

• Describe the NetScaler system architecture and processes.
es

• View, interpret and manage logged information.

al

• Access and use statistics.

e

• Capture and analyze network traffic.

or
di

Troubleshooting Resources
s tri

Useful troubleshooting resources include:

bu

• Citrix Knowledge Center

• Citrix product documentation
t io

• Auto Support
n

• Citrix technical support

• Collected NetScaler data
• Troubleshooting log

Citrix Knowledge Center

An important tool for research, the Citrix Knowledge Center is the official resource for technical
information on Citrix products, hotfixes, security bulletins, and troubleshooting guides. The
Knowledge Center contains forums through which external sources can be contacted. It also

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 23

contains a thorough library of documentation and white papers. The forum community is also very
knowledgeable and responsive.
The Citrix Knowledge Center can be accessed at the http://support.citrix.com.

Citrix Product Documentation

The most recent NetScaler product documentation is available at http://docs.citrix.com.
NetScaler administrator guides are now available in http://docs.citrix.com for online viewing or may
be downloaded as individual PDF files.
N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

Under the NetScaler Gateway 10.5 heading, topics related to administration, policy references,
traffic management and related features are covered. Major topic categories include:
• Release Notes
• Getting Started
• Hardware Installation
• Licensing, Upgrading, and Downgrading
• System - Basic operations, administration, Clustering, networking, high availability
• AppExpert - Policy and expression guides, Rewrite, Responder, Rate Limiting, HTTP Callout
topics.

24 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

• Traffic Management - Includes Load Balancing, Content Switching, GSLB, and other related
topics.
• Optimization - Includes compression, Integrated Caching, and other related optimization
topics.
• Security - Application Firewall, AAA Application Traffic, Content Filtering, and other related
topics.
• API
• Reference Material
The SSL VPN and NetScaler Gateway functionality is covered under the NetScaler Gateway
heading. Depending on the feature sets in use, administrators may need to reference both sets of
reference materials to have a comprehensive view of NetScaler administration.

Auto Support
N
ot

Auto Support (formerly Tools as a Service or TAAS) can help identify, troubleshoot, and resolve
issues through automatic analysis of configuration and error details. Auto Support is a free service
fo

and is available to any user with a MyCitrix ID; a support contract is not required.
rr

Auto Support can be used with XenApp, XenDesktop, XenServer, and NetScaler systems.
es

For NetScaler, the show techsupport command can be used to gather the necessary files for
upload to Auto Support for analysis. Auto Support only analyzes the actual crash files
al

(vmcore.xx.gz) and nCore packet crash files (NSPPE.*.gz). Additional files can be uploaded, but
e

they will not be analyzed by Auto Support; however, they can be submitted to tech support as part
or

of a support case.
Auto Support can be accessed at http://taas.citrix.com/autosupport or http://taas.citrix.com. Log on
di

with a valid MyCitrix account.

s

Data can be uploaded and the analysis viewed in the workspace. Past uploads can be viewed in the
tri

archive. If a support case exists, uploaded files can be associated with case number.
bu

When working with NetScalers in a high-availability pair or cluster, the tech support files may need
t

to be gathered from each node in the pair or cluster.

io

To generate files for upload using the Configuration Utility:

n

1. Connect to the NetScaler Configuration Utility (using a web browser).

2. Browse to the System > Diagnostics node.
3. Select Generate Support File under Technical Support Tools.
4. Download the support file through the interface and save it on the local workstation.
5. Upload the file to Auto Support at https://cis.citrix.com/.
To generate files for upload using the command-line interface:
1. Connect to the NetScaler command-line interface using SSH.

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 25

2. Run the following command:
show techsupport
3. Use SFTP to download the resultant file from the NetScaler appliance. The file is generated in
the /var/tmp/support/ directory with a <filename>.tar.gz extension.
4. Upload the file to Auto Support at http://taas.citrix.com/autosupport.
To upload files directly from the NetScaler:
1. Verify that the NetScaler can resolve the taas.citrix.com host to the IP address. If this DNS
resolution cannot be performed, then you must manually upload the files.
2. Download the taas_upload.pl script and the taas.citrix.com_root PEM from
CTX135876.
3. Upload both files to the /var/tmp/support/ directory on the target NetScaler appliance.
4. Update permissions on the files using the following commands:
N
ot

shell
fo

chmod 777 taas.citrix.com_root.pem

rr

chmod 777 taas_upload.pl

es

5. Generate the tech support file using the following command:

al

show techsupport
e
or

Note the name of the collector file to upload.

di
s tri

6. From the shell, type the command to run the upload script:
bu

./taas_upload.pl [-sr <service request number>] [-

description <description>] <collector file or trace file or
t io

other file>
n

A successful upload is confirmed by a return code 0.

For additional information, see the following references:

• CTX136396: How to Upload Data to Auto Support (TAAS)
• CTX127900: How to Use Show Techsupport Collection Script for NetScaler

26 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

 The collection script is only needed for releases prior to NetScaler 9.2.

• CTX135876: How to Upload a Collector File from a NetScaler Appliance to taas.citrix.com

Website Directly without Retrieving it from the Appliance
• Auto Support Forum: http://discussions.citrix.com/forum/298-citrix-auto-support

Citrix Technical Support

Citrix Technical Support can be an extremely useful resource when troubleshooting a NetScaler
deployment. When contacting Technical Support, compile all the previously gathered information
pertaining to the issue. Be prepared to engage the support engineer in an exchange of ideas.
N
ot

Collected NetScaler Data

fo

Gathering all available NetScaler data is important when beginning analysis. Data resources include:
rr

• Results of the show techsupport command, which provides configuration files,

performance log data, system messages and other relevant system information
es

• User feedback, which may include screen captures

al

• Documented steps for reproducing the issue

e

• Network packet traces

or

• Syslogs
di

• Web logs
s

• SNMP alarms
tri

• Network topology diagrams and other deployment documentation

bu
t

Troubleshooting Log
io
n

A troubleshooting log should not be confused with the various log files the NetScaler system
generates. Troubleshooting logs document any previously encountered issues and their resolutions.
Such logs are valuable when researching a current issue.
A troubleshooting log may be a simple spreadsheet that includes information such as:
• Issue type
• Description
• Date entered
• Entered by
• Priority

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 27

• Assigned to
• Issue resolution
• Date resolved

NetScaler System Overview

The NetScaler system has a unique architecture. It is helpful to understand some of the key
architectural concepts when troubleshooting.
N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

As the figure shows, the NetScaler design is based on a layered model between the NetScaler kernel
and the BSD operating system. BSD and the NetScaler system share memory management. The
NetScaler kernel operates below the BSD kernel and controls many of the features, including:
• CPU Time slicing for BSD
• Network packet processing
• SNMP and syslog processing
• SSL offload
BSD manages several integral features of the NetScaler system, including:

28 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

• Boot process
• File system access

File names and paths are case sensitive.

• Long-term logging
• Management processes

nCore Configuration Architecture

N
ot
fo
rr
es
al
e
or
di
s tri
bu

NetScaler nCore uses multiple CPU cores for packet handling. The NetScaler nCore architecture
includes the underlying NetScaler kernel and the cores, which are separate packet engines. The
t

packet engines are designed to work independently, however, the cores communicate with each
io

other using the core-to-core messaging.

n

Configuration changing commands, by default, are sent to all packet engines, while non-
configuration changing commands are sent to PE-0. Dist_send handles connection management,
sends and receives from the packet engines, and rolls back a single input-output control (IOCTL)
command if it fails on any one packet engine. If the packet engine moves to the user space then it
needs to be able to pick up packets from user space as well.
The queues of the network interface cards (NICs) are initiated in the kernel, and then sent to the
packet engine in the user space without involving the kernel. This function preserves the line rate
and does not introduce the kernel as an intermediary.
The following components are shown in the figure:
• PE represents the packet engine.

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 29

• GUI represents the Configuration Utility and other graphical user interfaces.
• CLI represents the command-line interface.
• nsaggregator collects and monitors data from the packet engines and communicates with
various aggregator clients, such as:
• nscollect
• nsconmsg
• stat command
• snmpd
• newnslog contains the performance data stored by nsconmsg
• The distributor sends configuration commands to all of the packet engines
• nsnetsvc includes the distributor module, which talks to the kernel and packet engines
• nsconf contains all of the configuration information
N
ot

Built-In Tools
fo

The NetScaler has a number of built-in tools to assist with issue diagnosis, troubleshooting, status,
rr

health checking, and monitoring of NetScaler resources. The following tools will be discussed:
es

• Diagnostics Pane
al

• NetScaler Logs
e

• Syslog
• Nslog
or

• Real-Time Performance Statistics

di

• Dashboard
s

• Stat Commands
tri

• Historical Statistics
bu

• Reporting Tool
t

• Command Center
io

• SNMP
n

• Viewing Objects
• Network Traffic Capture
• Command Line Tools
• Shell Tools

30 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

Diagnostics Pane Overview

N
ot
fo
rr
es

The figure shows the Diagnostics pane of the Configuration Utility, which displays the tools that
al

you can use to diagnose and troubleshoot appliance issues. Go to System > Diagnostics to view the
e

Diagnostics pane.
or

The tool groups within the Diagnostics Pane cover a number of troubleshooting and maintenance
tools for the NetScaler.
di

• View Configuration: Commands for viewing and comparing the saved, running, and GSLB
s

configuration states.
tri

• Utilities: Access to command-line shell utilities such as ping and traceroute, the
bu

NetScaler command-line interface, HA and Cluster file synchronization commands, and a

command to create batch scripts from a saved configuration file.
t io

• Technical Support Tools: Includes commands to generate support files ( show

n

techsupport), create network traces and download trace or core files, and Call Home
functionality (for support agreements with automatic support contact).
• Maintenance: Clear configuration and clean up commands for trace, core, and log files. Be
careful with the clear configuration options as they cannot be undone unless a backup
configuration has been created beforehand.
• Manage Logs: Includes shortcuts the most important and commonly used nsconmsg
commands.
• Troubleshooting Data: Additional shortcuts to debug commands for nsconmsg.
• Monitor Connections: Shortcuts to the show connectiontable and show
persistentsessions commands.

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 31

NetScaler Logs
The NetScaler maintains a number of log files to record many of the events that occur on the
appliance. Of these logs, the most useful for day-to-day operations and troubleshooting or the
Syslog file (/var/log/ns.log) and the Nslog file (/var/nslog/newnslog).

Syslog
The NetScaler audits all configuration changes, events, and certain action taken by specific features
within the syslog file. Features such as the NetScaler Gateway and Application Firewall log actions
that they take as part of the audit log. Actions by these features may overlap with security and audit
considerations. As a result, the log file is useful when troubleshooting the NetScaler and for meeting
requirements for audit logging.
• The current syslog file is located in /var/log/ns.log.
N

• The log file automatically rolls over every 100 KB or hourly.

ot

• The last 25 copies of the log file are retained and archived as ns.log.0.gz-ns.log.25.gz.
fo

• The file is in standard syslog format and can be archived with external syslog servers.
rr
es

NetScaler Configuration Utility

al

Syslog messages can be viewed from the System > Auditing node (node root).
e

• Recent audit messages: Displays the most recent events from the current syslog file. Default is
or

the last 20 messages received and all levels. This is equivalent to a tail command on the current
log file.
di

• Syslog messages: Displays the complete list of events in the current log file and can be used to
s

display any past ns.log files. This display also supports filtering of logs by module, event type,
tri

severity, and a message search. Log files may also be easily downloaded from the NetScaler
bu

from this interface.

t io

Command-line Interface
n

Syslog files are viewable from shell. From the command line, use the shell command.

Action Command
View current syslog file (all events)
more /var/log/ns.log

View latest events in current syslog file

tail /var/log/ns.log

32 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

Action Command
Search for specific messages in syslog file
more /var/log/ns.log | grep
The grep command is case-sensitive by default. lb_vsrv_rbg
The "-i" option can be used to make a search
pattern case-insensitive.
more /var/log/ns.log | grep
In the second example, the grep command will svc_red -i
find all instances of svc_red in the configuration
regardless of how it was entered.
more /var/log/ns.log | grep
In the third and fourth examples, searching on APPFW
APPFW and SSLVPN finds only the events
reported by the Application Firewall and
Gateway features since these are the literal more /var/log/ns.log | grep
N

MODULE names in the log file. By keeping the SSLVPN

search case-sensitive, you will only receive the
ot

list of events generated by these modules and

fo

will not receive any configuration commands

reported by the CMD_EXECUTED module
rr

when running set appfw profile

commands.
es
al

View syslog events as they occur. This is useful

for observing live reporting of audit commands tail -f /var/log/ns.log
e

as they occur.
or

The tail -f command keeps maintains an tail -

open file handle to the log file. f /var/log/ns.log | grep APPFW
di
s

Combining this with grep can allow an

tri

administrator to view only specific feature

events to simplify troubleshooting.
bu

View events in archived syslog file

t io

zcat /var/log/ns.log.##.gz |
more
n

zcat /var/log/ns.log.##.gz |
tail

zcat /var/log/ns.log.##.gz |
grep APPFW

The audit parameters (System > Auditing > Settings), control the default logging behavior of the
syslog and nslog logging to the local NetScaler system. Additional audit policies may be created to

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 33

log to external syslog servers. Audit policies may be bound to system global, VPN global, virtual
server, AAA groups, and AAA users.
By default, syslog does not include TCP connection logging, ACL logging, or AppFlow logging
events. This can be enabled if necessary, although it may cause more frequent log rollover.
While some features log key events and actions to ns.log as part of the syslog function, many
policy-based feature actions are not included in normal syslog tracking. Certain features support
custom log messages that can be triggered when policy actions go into effect. Features that support
custom logging include responder, rewrite, App Firewall, and content switching. To include the
custom log actions in the syslog, "user configurable log messages" must be enabled with the global
auditing parameters or the individual syslog server. When creating a custom log action, it is also
possible to log these events to the nslog file, but syslog is preferred.
N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

Nslog
The nslog file is the primary debug log for the NetScaler. All statistics, counters, and metrics
tracked by the NetScaler appliance are logged to the nslog including internal debug counters. The
log file also contains records of events and console messages that occur on the system. Statistics
information is written to nslog every 7 seconds. This information may be accessed easily using the
stat commands from the command-line interface or using the dashboard in the Configuration
Utility for real-time statistics. The Reporting Tool can display a historical view of these statistics

34 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

over time, though the historical metrics for reporting come from a separate metric database and are
not derived directly from the nslog file.
• The current log file is located in /var/nslog/newnslog.
• When the NetScaler is participating in a cluster (tri-scale cluster), each node member has a
separate nsconmsg file numbered after the node id. For example,
/var/nslog/newslog_nodeid.
• The log file automatically rolls over every 300 MB or every 48 hours.
• The last 100 log files are retained and archived as newnslog.0.gz-newnslog.99.gz.
• Statistics and events are logged every 7 seconds.
The nslog file is in binary format. Therefore it cannot be viewed directly from the command line.
Instead, the nsconmsg utility can be used to retrieve information from the log file. However, for
most standard operations available to administrators, the stat command, Dashboard, Reporting, and
Diagnostics utilities provide the easiest method for accessing the nslog information with requiring
N

raw nsconmsg commands.

ot

Performance Data Description

fo
rr

Stat (Command Line)

stat cmp
es

Stat commands display the same output

available in the Dashboard (tabular view).
al

stat lb vserver <lb vserver

For certain objects, the stat command name>
e

displays a summary set of counters.

or

stat system
The -details flag displays all statistics for an
di

object. stat <object> -detail

s tri

Dashboard Displays statistics information for common

bu

NetScaler objects in a tabular view and a

graphical view. Most dashboard statistics are
t

considered real-time and represent the current

io

and most recent time frame.

n

When troubleshooting the NetScaler, the most common commands are accessible from the System
> Diagnostics node using the Manage Logs tools.

Action Description
View log file duration Displays the time frame covered by a current or
archive newnslog file. Used to identify which
log contains the desired events.

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 35

Action Description
View events Displays events occurring on the NetScaler.
Events refer to discrete events that occur on the
NetScaler and can be used to identify state
changes. Events can include link state changes,
service events, or monitor events. This is a
useful troubleshooting log as it can indicate
possible issues caused by a misconfiguration or
an external entity failure such as a network
outage or an inaccessible server. Some events
may be reports as part of the syslog
(/var/log/ns.log) file and others may
only appear within the nslog
(/var/log/newnslog) event list.
N
ot

View console messages Displays console messages occurring on the

NetScaler. Console messages refer to error or
fo

warning messages that occur on the NetScaler

and may be a result of various events. Console
rr

messages may also cause the LCD panel to flash

es

when a warning is occurring. Console messages

can be used to identify issues such as IP address
al

conflicts, link failures due to switches muting

e

ports, NetScaler service failures, and other

errors.
or

View events from specific time Within a log file, an administrator can retrieve
di

events from a specific time to assist with event

s

correlation. The GUI prompts for log file and a

tri

time value within the start-end duration of the

bu

selected log file. The GUI can easily retrieve

details from the current nslog (newnslog) file or
t

an archived log (newnslog.##.gz). It is important

io

to select a date and time reference that occurs

n

within the selected log file. Use the "View log

file duration" command to identify the proper
log file before using the "View events from
specific time" command.

36 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

Action Description
Trim log files In order to isolate specific events or date from
specified time frame within a larger log file, the
trim log files command can be used. This
command is useful to strategically trim a much
larger log file down to the target time frame for
troubleshooting individually or with tech
support. An administrator must specify the log
file or archive to search, the start date/time for
the initial event, and the duration from this
event (in seconds) to capture. In addition, the
administrator must also supply the name of the
new file to create using the trimmed output. It
is important to ensure that the starting point
N

and the subsequent duration is contained within

ot

the select log file or archive; this should be

confirmed using the "view log file duration"
fo

command prior to executing the "trim log file"

rr

command.
es

Download log files Administrators can easily download current and

past nslog (newnslog) files direct from the
al

NetScaler configuration utility without needing

e

to launch an SFTP utility.

or

The command line equivalents of the above commands can be run from shell using nsconmsg.
di

Note all parameters are case sensitive. Other information can be retrieved from nsconmsg; however,
s

in most cases tech support will be involved when troubleshooting beyond the commands available
tri

in the Configuration Utility.

bu

Action Command
t io
n

View log file duration

nsconmsg -
K /var/nslog/newnslog -d setime

View events
nsconmsg -
K /var/nslog/newnslog -d event

View console messages

nsconmsg -
K /var/nslog/newnslog -
d consmsg

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 37

Action Command
View events from specific time
nsconmsg -
K /var/nslog/newnslog -s
time=DDmmmYYYYHH:MM:SS
-d event

Example:

nsconmsg -
K /var/nslog/newslog -
s time=20Dec201315:30:00 -d
event
N

Trim log files

ot

nsconmsg -
K /var/nslog/newnslog -s
fo

time=DDmmmYYYYHH:MM:SS
rr

-k <output file> -T
<secs> -d copy
es

Example:
al
e

To retrieve all events from 1:37 pm until 3:37

pm (2 hours), use the following command.
or

nsconmsg -
di

K /var/nslog/newnslog -
s

s time=20Dec201313:37:00 -k
tri

/var/nslog/trim1 -
bu

T 7200 -d copy
t

This will result in the output file

io

/var/nslog/trim1 being created and containing 2

n

hours worth of detail (7200 seconds) from 1:37

pm.

Retrieving events from past log files. With NetScaler 10.x, archived nslog files do not
need to be parsed by zcat or zmore first.
Instead, nsconmsg can be passed an archive and
it will automatically extract the files and display
the output.

38 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

Action Command
View log file duration
nsconmsg -
View events K /var/nslog/newnslog.##.gz -
View console messages d setime

nsconmsg -
K /var/nslog/newnslog.##.gz -d
event

nsconmsg -
K /var/nslog/newnslog.##.gz -d
consmsg
N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

Real-Time Performance Statistics

Real-time performance data and other critical performance statistics are gathered by the NetScaler
and available for review using the HTML dashboard or the stat commands from the NetScaler

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 39

command line. The statistics cover performance metrics, system throughput, and feature-specific
details that can be used for troubleshooting, system utilization, and analysis.
The information available from the stat commands is also available in the HTML dashboard in
both tabular (text) or graphical forms. Additional graphical reports are available in the HTML
dashboard enabling administrators to compare metrics for more detailed analysis than is possible
with the stat commands on their own. Invoking the statistics action within the Configuration
Utility will link an administrator to the entity or feature specific view within the dashboard.
All statistical information whether accessed from the dashboard or the stat commands within the
command-line interface are based on metrics stored within the nslog file
(/var/nslog/newnslog).

Dashboard
N

Statistics displayed within the HTML Dashboard are gathered and updated every 7 seconds. The
dashboard displays approximately the last 5 minutes of activity. Data is displayed graphically by
ot

default and may be switched to a tabular (text) display that is similar to the stat command in the
command-line interface as needed.
fo

The Dashboard provides an at-a-glance view of system health, including:

rr

• Management CPU and Packet CPU usage

es

• Memory usage
al

• Throughput numbers
e

• Syslog events
or

• System status including /flash and /var disk usage

di

Packet processing and feature specific statistics with both summary and detailed views:
s

• SSL
tri

• Integrated Caching
bu

• Compression
t

• Application Firewall
io

• Policy hit summaries for each feature

n

• Load balancing virtual server, service, and service group statistics

Network and web statistics including:
• Throughput statistics
• TCP connections
• Get vs. posts
• Client vs. server
In addition to the statistical display, built-in charts are included which allows comparison and
plotting of various statistics relative to each other or time indexes. The built-in chart displays may

40 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

be customized as either line, bar, or area charts as appropriate. Additional custom charts may be
created.

N
ot
fo
rr
es

Stat Commands
al
e

When working at the command-line interface, the stat command provides a quick and easy
or

method for querying the current system and feature performance statistics without having to view
raw details from the nslog file using nsconmsg. The stat command formats the output and makes
di

it easy to retrieve performance and debug related counters.

s

The stat command can be used with a number of the NetScaler object groups and features, such
tri

as:
bu

• system, system CPU, system memory

• ns
t io

• load balancing vserver, service, or service group

n

• cmp
• cache
• VPN and VPN vserver
• SSL
Common stat commands include:
• stat cmp
• stat cache
• stat system

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 41

• stat lb vserver <lb vserver name>
• stat service <service name>
The stat command also takes additional switch parameters which can modify the stat
command behavior. This includes toggling between summary view and detail view, displaying the
full names of counters and values. The stat command can also be used to iterate the output n-
number of times in order to simplify looking at statistics over time.
The following table lists additional switches for the stat command.

Switch Description
-detail Specifies detailed output instead of the default
summary view, for objects that support it.
The output can be quite voluminous. Without
N

this argument, the output will show only a

ot

summary.
fo

Not all objects contain additional details.

rr

This is equivalent between switching between

Summary and Detail view in the HTML
es

Dashboard.
al

-fullValues Specifies that numbers and strings should be

e

displayed in their full form .

or

Without this option, long strings are shortened

and large numbers are abbreviated.
di
s

-ntimes positive_integer Indicates the number of times, in intervals of

tri

seven seconds, the statistics should be displayed.

bu

This parameter results in the stat command

being repeated.
t io

The default value is 1.

n

-clearstats [basic | full] Clears the statistics or counters. Full can be

used to clear all stats for all global objects.

Show Commands
In order to characterize an issue, gather data about the network and the NetScaler system. Citrix
recommends to begin gathering information is to run the show command in the command-line
interface.
Entities that can be enumerated using the show command include:

42 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

Type Entity
System • ns.conf
• config
• runningconfig
• license
• mode

Object • vserver
• service
• server
• expression
N

• monitor
ot

• policy
fo

• profile
rr

• filter
• stats
es
al

Networking • interface
e

• vlan
or

• ip
• route
di

• arp
s tri

• acl
• node
bu

• channel
t io
n

For example, enter the following command to view the details of a particular Application Firewall
policy:

show appfw policy policy_name

Other commands can show entity and system configuration details:

show system global

show system group

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 43

 show system parameter

show lb vserver

show lb vserver <lb vserver name>

show route

While the show command is useful for showing settings and current state of an entity, sometimes
it is better to view the command used to create or configure the entity. This can also make it easy
to compare multiple entities in one or two lines and spot inconsistent settings. To view the actual
configuration commands, view the current running or saved configuration, and search for the
intended object or objects using grep.

show ns ns.conf
N
ot

show ns savedconfig
fo

show ns runningconfig
rr
es

show ns runningconfig | grep lb_vsrv_rbg

al

show ns runningconfig | grep "add lb vserver lb_vsrv_.*" -i

e
or

NetScaler Configuration Utility

di

The NetScaler Configuration Utility displays all configured objects and settings on the NetScaler.
s

Everything displayed within the GUI comes from a show command. Delegated administration can
tri

allow or prevent access to individual objects. Without show permissions, the object or node will not
bu

be accessible within the GUI.

t io
n

44 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

 N
ot
fo
rr
es
al
e
or
di

NetScaler Visualizer
s tri

The Visualizer tool can be used to display the policies and services associated with the different
bu

virtual servers on the NetScaler. This includes load balancing, content switching, and GSLB virtual
servers.
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 45

 N
ot
fo
rr
es
al
e
or
di

Historical Statistics
s tri

Nslog, nsconmsg utility, stat, and the dashboard are concerned with real-time metrics and data
bu

gathering. In order to assess NetScaler utilization and trends over time and to engage in past-event
correlation, a NetScaler administrator needs to be able to review and compare historical metrics
t

and metrics over time.

io

The nscollect process is responsible for collecting certain performance data (from the nslog metrics)
n

and consolidates this data for the historical performance database displayed by the reporting
console.
Historical data is consolidated over time, which means there is more granular data for the recent
history and less detailed data for older time periods. The default data collection process for
performance data is every 5 minutes (compared to the real-time data collection process of every 7
seconds).
The nscollect process used to gather data affects the CPU utilization of the Management CPU and
not the Packet Processing CPU(s). The larger the NetScaler configuration and the more traffic
being passed the more work that will be done by the nscollect process. But this should not be an
impact to overall NetScaler performance.

46 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

The Reporting tool provides easy access to built-in reports to compare and trend system resource
utilization. It can also be used to assess the effectiveness of compression and caching policies or
trends with Application Firewall security violations. With the built-in reports, charts are
automatically available to graph the data visually. Chart display options can be customized and the
time periods for the reports can be adjusted from a view of the last hour or last day to the last
month or year. Counters, display ranges, may be further adjusted.
The Reporting utility also supports creation of custom reports.

Reporting Tool
N
ot
fo
rr
es
al
e
or
di
s tri
bu
t

The figure shows the NetScaler reporting tool, which provides built-in reports that display statistics
io

collected by the nscollect utility. You can also create custom reports. The reports use charts to
n

display the statistics. You can modify the charts and add new charts. You can also modify the
operation of the nscollect utility and stop or start its operation.

Reporting Tool Components

The following table lists the reporting tool components.

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 47

Component Description
Navigation Pane The navigation pane extends down the left side
of the screen. It has two sections for each type
of report: Custom Reports and Built-in Reports.
Under Custom Reports, you can access custom
reports you create. Under Built-in Reports, a
collapsible menu contains links to different
categories of built-in reports.

Details Pane The details pane is the right portion of the

Reporting page, which displays the report you
clicked in the navigation pane. You can modify
a report, create custom reports, and view
reports for different time intervals using various
N

options available in the details pane.

ot

Report Toolbar You can use the options on the report toolbar
fo

in the details pane to do the following:

rr

• View reports in different time intervals

• Create new custom reports
es

• Save built-in reports as custom reports

al

• Delete reports
e

• Change the settings of the reports and

or

select a different data source

di

Chart Toolbar Each report is a collection of charts. Beneath

each chart is a chart toolbar with options for
s tri

changing the chart to a different type or

charting different counters. The chart toolbar
bu

also has icons for adding a new chart, deleting a

chart, or moving the chart up or down within
t io

the report.
n

Built-in Reports
The Reporting tool provides built-in reports for frequently viewed data. Built-in reports are
available for the following seven function groups:
• System
• Network
• SSL
• Compression

48 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

• Integrated Cache
• NetScaler Gateway
• Application Firewall
By default, the built-in reports are displayed for the last hour. However, you can view the reports
for the last day, last week, last month, or last year.

You need to save a built-in report as a custom report if you make any changes to it.

Custom Reports
NetScaler administrators can create up to 256 custom reports and save them with user-defined
N

names for reuse. Custom reports can plot different counters for different groups based on your
ot

requirements. You can save a built-in report as a custom report, or you can create a new report. By
default, a newly created custom report contains one chart named System Overview, which displays
fo

the CPU Usage counter plotted for the last hour. You can use the report toolbar to modify the
interval and to set the data source and time zone.
rr

All reports must have unique names (duplicate report names are not allowed even in different
es

folders). This simplifies the way of accessing reports through a URL (with the name as the primary
key).
al
e

Existing folders can be chosen or new folders can be added. The maximum number of folders
allowed is 128. All folders must have unique names.
or
di

Charts
s tri

Charts plot and monitor counters or groups of counters. Each report can contain up to four charts,
bu

and each chart can plot up to 16 counters. The counters can use different graphical formats (for
example, area and bar).
t io

In all report charts, the horizontal axis represents time and the vertical axis represents the value of
n

the counter.

Data Collection
When the NetScaler system starts, the nscollect utility automatically starts. The nscollect utility
retrieves data from the NetScaler system and updates the data source. The default data source is
/var/log/db/default.
If data is not updated accurately, or there is corrupted data displayed in the reports, you can stop
and then restart the utility. You may also want to stop the nscollect utility to back up the databases
or to create a new data source.

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 49

Enter the following command in the command-line interface to stop the nscollect utility:

/netscaler/nscollect stop

You can start nscollect on either the local system or a remote system. Enter the following command
in the command-line interface to start the nscollect utility:

/netscaler/nscollect start

Enter the following command to start the nscollect utility on the remote system:

/netscaler/nscollect start -U ip_address:UserName:Password -

ds data_source_name
N

Command Center and SNMP

ot

Command Center is an external system that can be used to configure, manage, and monitor
fo

multiple NetScaler, NetScaler VPX, and NetScaler SDX appliances. Command Center supports
rr

several key functions that can assist with troubleshooting, alerting, and performance monitoring.
es

Command Center acts as an SNMP manager and trap destination for the configured NetScaler
appliances. Command Center can then be used to generate alerts or notifications to administrators
al

when alerts and custom triggers occur.

e

As part of this solution, Command Center provides an at-a-glance dashboard functionality to verify
or

the state of multiple NetScaler appliances and all virtual server, service, and service group entities
on these managed systems.
di

Command Center also contains a performance monitoring database and can be used to control
s

which metrics and counters to archive, the polling interval, and metric retention periods. As a
tri

result, Command Center can provide a backup for the Reporting capabilities of the individual
bu

NetScaler appliances and can even be used by administrators who want more granular control over
the data gathered and reports to generate.
t io
n

Network Traffic Capture

A network trace is an essential tool for troubleshooting as it can confirm whether network traffic is
being passed between components and it can also be used to identify the protocol specific details of
requests and responses and the IP addresses and ports used to send and receive data. When using
the NetScaler to manage and optimize traffic, a trace can be used to identify the source and
destination IP addresses and ports of the client-side communication and the server-side
communication, along with the details of the communication. This can give administrators the
necessary information to identify, diagnose, and resolve configuration issues on the NetScaler and
communication issues with external components.

50 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

Since the NIC hardware of the NetScaler appliance is managed by the NetScaler kernel, the Linux
tcpdump command should not be used to generate network traces on the NetScaler appliance.
Instead, either the shell script nstcpdump.sh or the NetScaler command nstrace should be used
as both of these utilities generate Network traces based from the NetScaler kernel.

Nstrace Options
The nstrace utility can be invoked from the command line using the nstrace command and from
the configuration utility under System > Diagnostics node using the "Start new trace" command.
The NetScaler system can capture packets in the NetScaler native trace format or using the standard
tcpdump format (.pcap).
Native format includes additional information that is useful for debugging, including interface and
RX/TX information. If a NetScaler system defect is suspected and the system is live, then traffic
N

should always be captured in native format for submission to Citrix Technical Support.
ot

Default Settings
fo
rr

If nstrace is run from the command-line interface without options, then the following default
options are used:
es

• Stores trace files in NetScaler proprietary nstrace format as a .cap file.

al

• Generates a trace and logs to a single file for 1 hour; default number of files is 24, if trace is not
e

stopped. As a result, if a continuous trace is started it will run for 24 hours, with 24 separate
or

log files of 1 hour each.

• Captures first 164 Bytes for each packet.
di

These default behaviors of nstrace can be changed by specifying various options.

s tri

The nstrace capture behavior can be modified from the defaults in both the CLI and GUI
bu

commands. Useful options are included below:

t io

Switch Description
n

-tcpdump [enabled | disabled] Switches from option to create trace in native

nstrace format or in standard tcpdump format.
Nstrace is the default format therefore the flag -
tcpdump enabled must be used output standard
format.
Example:
nstrace -tcpdump Enabled

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 51

Switch Description
-size <integer> Size of packet to be logged. Set to 0 to capture
the full packet regardless of packet size; set to a
specific integer to capture only the specific bytes
of the packet.
Example:
nstrace -size 0 -
tcpdump Enabled

-pernic [enabled | disabled] When a trace is generated in tcpdump format, a

single network trace can be generated for traffic
across all NICs or a trace can be generated on a
N

per NIC basis with separate files for each NIC.

ot

Default option is to generated one trace for all

NICs: -pernic disabled.
fo
rr

-nf <integer> The -nf parameter specifies the total number of

output files to generate; default value is 24. The
-time <secs>
es

-time parameter specifies the total duration of

an individual trace (file); default value is 3600
al

seconds (1 hour).
e

In most situations, an administrator may start a

or

trace and then stop it within a few minutes so

only one file is generated. If the trace is allowed
di

to continue without being stopped, these two

s

parameters determine the total number of trace

tri

files to generate and the rollover interval for

each file.
bu

Example:
t io

nstrace -size 0 -
n

tcpdump enabled -nf 24 -time

3600

52 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

Switch Description
-mode <mode> Determines trace packet capture mode. Options
may be used to determine whether to capture
send or receive packets before or after NIC
pipelining. Modes are additive (based on a
bitmask).
Example:

nstrace -mode RX TX

nstrace -mode TXB RX

N

-filter <expr> The filter option allows an administrator to

ot

restrict the packet capture to only the traffic

meeting the filter expression criteria.
fo

The filter option was first introduced in

rr

NetScaler 9.0. The syntax for the filter

expression has expanded and the syntax support
es

in NetScaler 10.x is more advanced than the

filter syntax supported in earlier versions.
al
e

The filter option helps to generate a trace only

of the specific traffic of interest and can be used
or

to isolate the packets captured amidst all the

traffic being passed by the NetScaler appliance.
di

The option is frequently combined with the link

s

parameter.
tri

The syntax is discussed in more detail below.

bu

-link [Enabled | Disabled] The -link option determines whether the trace
t io

includes the associated peer traffic for a peer

connection. This is useful when a filter isolates
n

the traffic capture only to traffic originating

from a specific source IP address. The -link
enabled option will ensure that all parts of the
transaction (client-side and server-side) are
included in the trace.
This option is referred to as "Trace filtered
connection's peer traffic" within the Trace dialog
box in the GUI utilities.

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 53

Link Parameter Example
Example 1:

nstrace -size 0 -tcpdump enabled -filter

connection.SRCIP.EQ(192.168.200.10)

In this example, the -link parameter was omitted making the setting disabled (default value). As a
result the trace will only contain results for traffic originating from IP Address 192.168.200.10. This
may identify traffic such as client to NetScaler VIP, but may not include any other parts of the
transaction.
Example 2:

nstrace -size 0 -tcpdump enabled -filter

connection.SRCIP.EQ(192.168.200.10) -link enabled
N
ot

In the second example, the -link parameter was included. Now the trace will contain:
fo

• All traffic originating from 192.168.200.10 to any NetScaler IP address, such as a VIP.
rr

• All MIP/SNIP to backend server IP addresses associated with the client to NetScaler
communication.
es

• The return response server IP to MIP/SNIP.

al

• The return response from the VIP to the source IP address.

e

The benefit is that the trace is limited to traffic originating from a specific device, amongst all
or

traffic passing through the NetScaler and this trace contains all client to NetScaler and NetScaler to
server communications as part of the transaction, without requiring a more complicated filter
di

expression.
s tri

Trace Filter Syntax

bu

The filter parameter for nstrace is powered by the default policy engine and therefore can be based
t io

on a wide-range of criteria. Filter expressions may be based on simple or compound expressions

using Boolean operators (&&, ||) and parentheses.
n

Older syntax based on simple filters for source and destination IP address and ports are still
supported:
Filter Example 1:

sourceip == <IP address>

destip == <IP address>

sourceip == 192.168.200.10 || sourceip == 192.168.200.20

54 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

 sourceip == 192.168.200.10 && destport == 80

The default policy engine also supports a new top level object: connection. The connection object
can be used to identify traffic of interest for both client and server connections without being flow
specific. The connection object can be used to identify basic networking criteria of a packet such as:
• Source and Destination IP address for both IPv4 and IPv6
• Source and Destination Ports
• Interface, vlan IDs, and traffic domain
In addition, the connection object can also be used to define filter criteria based on NetScaler-
specific traffic criteria such as:
• Load Balancing or Content Switching virtual server entities or criteria
• Services and service type
N

• Additional filter criteria is available.

ot

Filter Example 2:
fo

CONNECTION.SRCIP.EQ(192.168.200.10)
rr
es

CONNECTION.SRCIP.EQ(192.168.200.10) ||
CONNECTION.SRCIP.EQ(192.168.200.20)
al

CONNECTION.SRCIP.EQ(192.168.200.10) || CONNECTION.DSTPORT.EQ(80)
e
or

CONNECTION.SRCIP.EQ(192.168.200.10) || !CONNECTION.DSTPORT.EQ(80)
di

The man pages for nstrace (man nstrace) contains a syntax reference for the filter parameter.
s

Within the trace utility in the Configuration Utility (GUI), a default policy engine expression build
tri

is available to simplify expression generation.

bu

In most situations, when using the filter expression parameter, also enable the link parameter (trace
filtered connections peer traffic).
t io
n

Creating a Trace File in the Configuration Utility

It is very common for a typical nstrace command to be based on the following commonly used
parameters and values:

nstrace -size 0 -tcpdump enabled -filter <expr> -link

enabled

The GUI equivalent of this command, using default values for the unspecified parameters will look
like the following. (This is not depicting default values for packet size, tcpdump, filter expression,
or link settings.)

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 55

 N
ot
fo
rr
es
al
e

Use following procedure to create a trace file, as shown in the figure:

or

1. Go to System>Diagnostics.
di

2. Click Start new trace to display the Trace dialog box.

s

3. Click Start to begin the trace.

tri

4. Click Stop to end the trace. The capture will end after 24 hours if you do not end it manually;
bu

1 hour per file for 24 files.

5. Trace files may be downloaded from /var/nstrace/ using SFTP or the Download option
t io

in the utility dialog box.

n

Shell Tools
The NetScaler uses a FreeBSD shell architecture that has a number of built-in tools. These shell
utilities can be used to test traffic flows. The following table list shell tools that are useful when
troubleshooting.

56 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

Shell Command Description
telnet Helps with troubleshooting connectivity issues
between the NetScaler system and other devices
on a specific port

ping Helps with troubleshooting connectivity

between two devices

This command is also available in the

command-line interface.

tail -f Displays the last lines of a log file and updates

the screen in real time as new lines are added to
the log; the file is kept as an open file handle for
N

continuous output to stdout.

ot

| (pipe) Sends the output of one command to another

fo

command
rr

grep Filters the output of other command-line

es

interface commands to the screen as they are

written
al

df Reports information about processes, memory,

e

paging, block I/O, traps, and CPU activity at

or

specified intervals (seconds between polls) and

count (number of times to poll)
di
s

nstcpdump.sh Allows for the capturing and analyzing of

tri

network traffic and prints out the headers of the

packets on a given network interface. Note:
bu

nstrace from the CLI or GUI is preferred over

nstcpdump.sh; nstcpdump.sh may add overhead
t io

to the CPU.
n

show techsupport Generates a tar file of the system configuration

data and statistics for submission to Citrix
Technical Support
The archive is named
/var/tmp/support.tgz for each
invocation of the command.

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 57

Command-Line Interface Tools
Enter the following command in the command-line interface to view recent audit messages:

show audit messages -logLevel log_level -

numOfMesgs positive_integer

The following table lists available arguments.

Argument Description
-logLevel log_level Specifies the log level filter

-numOfMesgs positive_integer Specifies the number of log messages to be

N

printed.
ot

The maximum value is 256, and the minimum

value is 1. The default value is 20.
fo
rr

Enter the following command in the command-line interface to invoke the ping command:
es

ping -c count -i interval -I interface -n -p pattern -q -s size -

al

S src_addr -t timeouthostname
e

The following table lists available arguments.

or
di

Argument Description
s tri

-c count Specifies the number of packets to send (default

bu

is infinite)
t

-i interval Specifies the waiting time in seconds (default is

io

1 second)
n

-I interface Specifies the network interface on which to

ping, if you have multiple interfaces

-S src_addr Specifies the source IP address to be used in the

outgoing query packets
If the IP address is not one of the NetScaler
system addresses, an error is returned and
nothing is sent.

58 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

Argument Description
-t timeout Specifies the timeout in seconds before the ping
exits

hostname Specifies the address of the host to ping

Enter the following command in the command-line interface to display the current TCP/IP
connection table:

show ns connectiontable

Enter the following command in the command-line interface to display all the entries in an ARP
table:
N
ot

show arp ip_address

fo

The following table lists available arguments.

rr
es

Argument Description
al

ip_address Specifies the IP address corresponding to an

e

Address Resolution Protocol entry

or

Third-Party Tools
di
s

The built-in NetScaler tools allow for monitoring the NetScaler system and any traffic that flows
tri

through it. But in the case of an issue originating outside the NetScaler system, third-party tools
bu

may be required.
t

Third-party tools include:

io

• Network packet analyzers, such as Wireshark

n

• HTTP header browsers, such as IE HTTP Headers for Internet Explorer or Live HTTP Headers
for Firefox
• SNMP MIB browsers
• Web browser plug-ins to view and modify headers
• FTP clients, such as WinSCP

© Copyright 2016 Citrix Systems, Inc. Module 1: Advanced Troubleshooting 59

Network Protocol Analyzers
Network protocol analyzers, such as Wireshark, are used to trace packets and observe what is
happening at the protocol level. Using such a tool requires thorough understanding of the protocols
at work but is often the only way to identify the causes of a complex issue. Network protocol
analyzers observe traffic packets without disrupting traffic, and so are suitable for gathering
information in a live environment. The NetScaler system, being a gateway device, provides a
location for inspecting traffic entering and exiting the network.
Network protocol analyzers can be useful while troubleshooting issues dealing with advanced
protocols. In a situation in which protocols such as TCP are operating but higher-level protocols
are not, simple tools such as the ping command will not yield much insight. Using a network
protocol analyzer to analyze the output of the nstrace.sh tool will allow inspection of the higher-
level protocols.
N

Web Browser Plug-ins

ot

HTTP headers encapsulate a large amounts of information passed during HTTP communications.
fo

Using an HTTP header browser, such as Live HTTP Headers, allows deeper inspection of HTTP
rr

traffic. The NetScaler has the ability to manipulate HTTP headers. It can add, remove, or edit the
value of header tags through the Responder/Rewrite feature. HTTP header browsers can be used to
es

troubleshoot features that work closely with HTTP traffic, such as compression. For example,
inspecting the HTTP headers can quickly identify which traffic is compressed.
al
e

SNMP Browsers
or
di

SNMP can be used to monitor many aspects of a network. Using an SNMP browser in conjunction
with thorough SNMP policies can help isolate the causes of an issue. The power of SNMP policies
s

is that they monitor a network condition passively and send out warning messages when something
tri

goes wrong. Linking SNMP to email allows emergency messages to reach network administrators in
bu

a timely manner.
t

SNMP is primarily a monitoring tool, but can also be very useful while troubleshooting intermittent
io

issues recurring over longer time scales. SNMP traps, or triggers, can be defined for the symptoms
n

of such an issue, and the resultant SNMP messages can be stored in a log. That log will provide
long-term information which may help isolate the source of such an issue.

FTP Clients
FTP clients, such as WinSCP, are commonly used to transfer files and to access log files.

60 Module 1: Advanced Troubleshooting © Copyright 2016 Citrix Systems, Inc.

 2
Module 2

Introducing
Application Firewall
N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

62 © Copyright 2016 Citrix Systems, Inc.

Introducing Application Firewall Manual
Overview
Organizations have a crucial need to protect their data and information from unauthorized users
and hackers. A network firewall does not provide enough protection against unauthorized access to
web applications. Rather, the best practice is to implement an application firewall in addition to a
network firewall to protect critical applications, especially those that contain customer and
employee data.
Hackers gain access to applications of an organization by exploiting vulnerabilities introduced by
human error and incomplete vendor updates, and by using new attack methods.
Application Firewall protects web applications from malicious attacks and unauthorized usage.
N

Application Firewall examines all incoming and outgoing traffic between protected web servers and
users for evidence of attacks or misuse of web server resources. It also blocks all known and
ot

unknown attacks.
fo

Application Firewall can be run as a stand-alone implementation on the NetScaler hardware and
functions as a dedicated Application Firewall appliance. Application Firewall is also available as a
rr

feature within the NetScaler Application Delivery System, which includes Application Firewall
es

functionality in addition to other NetScaler operating system features. Application Firewall

integrated with Citrix NetScaler is available with NetScaler Enterprise and Platinum editions.
al
e

Objectives
or

At the end of this module, you will be able to:

di
s

• Identify the goals of a web application attack.

tri

• Identify how Application Firewall protects web applications.

bu

• Identify how data flows through Application Firewall.

t io
n

© Copyright 2016 Citrix Systems, Inc. Module 2: Introducing Application Firewall 63

Application Attacks

N
ot
fo
rr
es
al
e
or
di

The figure shows how application attacks are mounted. Application Firewall protects critical web
s tri

applications and defends the infrastructure of any organization from identity theft, lost revenue,
brand erosion and other negative outcomes caused by application attacks.
bu
t io

Application Attack Description

n

An application attack starts with a hacker or virus sending a malicious but valid request to a web
server or servers. The malicious request exploits a programming error that results in unauthorized
individuals gaining access to important data, such as customer billing information or employee
records. An application attack has three possible outcomes:
• The attack causes the application to fail (denial of service) and prevents it from responding to
any request.
• The application fails but provides root or shell access to the attacker, leaving the application, its
data, and other areas of the network exposed.
• The application passes the malicious request, and the attacker obtains access to sensitive data in
the response.

64 Module 2: Introducing Application Firewall © Copyright 2016 Citrix Systems, Inc.

Goals of Application Attacks
Most types of application attacks on web servers and web sites are initiated to obtain sensitive and
private information or to obtain unauthorized access and control.
As opposed to network attacks, sensitive and private information contained in web applications can
include customer names, addresses, phone numbers, social security numbers, credit card numbers,
medical records, and other private information. Web applications may also contain information
related to system administration, test accounts, and passwords. The attacker can use this
information directly, sell it to others, or both. This type of breach can have serious consequences
for customers and organizations whose private information is compromised, such as having credit
ratings ruined or being blamed for criminal activities.
For example, in 2003, 400 customers of a cell phone company had their information stolen. The cell
phone company PDAs did not store information locally but instead stored information such as
phone books, pictures and email on central servers that served the information on demand.
N

The hacker discovered a flaw in the cell phone company web site that allowed PDA access
ot

passwords to be changed through forceful browsing. The hacker discovered that when clicking on
the forgot password link on the cell phone company web site, the hacker would be directed to:
fo
rr

http://www.cellphone.example.com/forgot_passwrd.html
es

This page had a logon screen for users to log into their service account before being directed to:
al

http://www.cellphone.example.com/reset_passwrd.html
e
or

This page was used to change PDA access passwords. The hacker discovered that the reset
password page could be accessed without having to log in. Because it was easy to figure out the
di

blocks of phone numbers that belonged to the cell phone company, the hacker was able to change
the access passwords of over 400 users and then download their data files.
s tri

Once the attack was discovered, the FBI was called to investigate the attack, but it took the FBI a
bu

year to catch the hacker. Although some of the affected customers were being kept informed of the
investigation, the hacker was able to evade the FBI because the cell phone company did not fix the
t

flaw in the web site for a year after it was discovered.

io
n

Most Common Types of Web Application Attacks

SQL injection and cross-site scripting (XSS) attacks are two specialized web application attacks that
occur most frequently.
• A SQL injection attack involves sending an active SQL command or commands to a web
application that, when passed to a back-end SQL database, runs the command and allows a
hacker to gain access to customer records and other sensitive information.
• A cross-site scripting (XSS) attack involves inserting or posting a malicious script to a
vulnerable web application that compromises the trust relationship between a user and a web

© Copyright 2016 Citrix Systems, Inc. Module 2: Introducing Application Firewall 65

 application, resulting in sending an attacker the user’s credentials. Confidential information can
then be accessed and used to steal that user’s identity.

SQL Injection Attack

N
ot
fo
rr
es
al
e
or
di
s tri
bu

As the figure shows, SQL injection attacks involve sending SQL commands to a web application,
t io

which the web application passes to the back-end SQL server and runs the command. This attack
allows the hacker to gain access to customer records and other sensitive information. SQL injection
n

attacks are exploiting vulnerabilities in some applications that do not properly clean or handle user
input:
• The database is the ultimate goal of the attacker, where the valuable corporate information is
stored.
• The web application is used as the conduit to the SQL database and has privileges to access the
database.
• A SQL injection attacks a back-end database through an application that has not validated the
input fields.
• A hacker looks for fields within the application simply by inserting SQL syntax, such as a '
mark.

66 Module 2: Introducing Application Firewall © Copyright 2016 Citrix Systems, Inc.

• A SQL error message that is returned to the hacker is an indication that there is access to the
back-end database through the application.
• The attack is very dangerous due to the potential for a hacker to access sensitive records, such
as account information, medical records, or customer information.

Cross-Site Scripting
N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

Cross-site scripting attacks exploit a similar vulnerability with user input fields, such as SQL
injection attacks. Attackers enter script or other executable code into a user input field. The web
application then runs the code and prompts other users for information that is sent back to the
attacker or redirects users to an alternate site to achieve the same goal.
A cross-site scripting attack involves inserting or posting a malicious script to a vulnerable web
application that compromises the trust relationship between a user and a web application, resulting
in sending an attacker the user’s credentials. Confidential information can then be accessed and
used to steal that user’s identity.

© Copyright 2016 Citrix Systems, Inc. Module 2: Introducing Application Firewall 67

The Application Firewall Solution

N
ot
fo
rr
es
al
e
or
di

When application attacks occur, one logical solution is to repair the application and then to
s tri

reinstall and redeploy it. However, this method is not always the easiest or most effective solution.
Several issues must be considered, such as whether the organization is dealing with legacy
bu

applications, Independent Software Vendor (ISV) applications or web server software

vulnerabilities.
t io

As the figure shows, Application Firewall protects mission-critical web applications and defends
n

customer web infrastructures from application attacks.

Application Firewall protects web sites and web service by filtering traffic to and from the web
servers. The traffic is inspected for security attacks and other harmful requests. Response traffic is
checked for sensitive data. Application Firewall blocks or renders harmless any attack or threat that
it detects.

Business Problems
One solution for increasing application security is to fix the application. However, factors may exist
that make fixing the application difficult. Even if a decision is made to apply fixes at the application

68 Module 2: Introducing Application Firewall © Copyright 2016 Citrix Systems, Inc.

level, there is still no guarantee that all vulnerabilities have been detected and addressed. For
example:
• Legacy applications are often mission-critical to organizations but do not meet governance or
corporate security requirements. The code must be rewritten, but capable developers may no
longer be employed by the company. The concern is that any attempts to fix an application
may cause more issues.
• ISV applications can be difficult to repair if they are written by another organization, and the
frequency of security updates from the vendor cannot be controlled. The organization may
need to depend on the ISV to release an update for the application and may expend time
testing and deploying the update. Productivity and profit loss are other possible results if the
application is disabled for a long period of time.
• Web server software vulnerabilities present the issue of keeping the site up while waiting on a
vendor update.
N

The Benefits of Application Firewall

ot

Application Firewall blocks application attacks while allowing legitimate traffic through to the
fo

application infrastructure and is a hardened security appliance that blocks all malicious attacks
rr

against web and web service applications. It was also named the highest-performing product in
attack protection, attack detection, traffic throttling and blocking by Forrester Research (2006).
es

Application Firewall provides protection against zero-day attacks and application attacks before an
al

ISV update is available.

e

Application Firewall can tune itself to both ISV and custom applications. It provides a solution for
protecting web applications that is significantly more effective than fixing the program or operating
or

with a network firewall alone. Application Firewall protects legacy applications without requiring
rewritten code, prevents new vulnerabilities from being exploited on web servers through ISV
di

applications and alleviates the need for immediate application updates and operating system (OS)
s

updates.
tri

Application Firewall secures and protects HTTP/HTTPS web applications and XML applications.
bu

Application Firewall:
t

• Uses a positive security model

io

• Performs Deep Stream Inspection on all traffic to and from web servers
n

• Defines safe application behavior using its Adaptive Learning Engine

© Copyright 2016 Citrix Systems, Inc. Module 2: Introducing Application Firewall 69

Application Layer Protection

N
ot
fo
rr
es
al
e
or
di

The figure explains the difference between network firewalls and application firewalls. A network
s tri

firewall protects at network layer (OSI layer 3). This type of protection is based on routing and
switching. A network firewall looks at the traffic source and destinations to determine which traffic
bu

should be allowed. The network firewall inspects the contents of packets in requests and responses
to determine if an attack is present or if data is being exposed. An attack that does not violate the
t io

network firewall rules is passed to the application.

n

Application Firewall protection occurs at the application layer (OSI layer 7). The NetScaler system
inspects all traffic going to and from the web servers. Sessions are terminated on the NetScaler
system, which decrypts, inspects, and re-encrypts SSL encrypted traffic before passing it to the web
servers.
Many organizations operate their IT environment with only a network firewall in place. A network
firewall protects against attacks at the network layer, but an application firewall is needed to protect
servers and applications against countless known and unknown attacks. Network firewalls also
cannot terminate SSL traffic, so attacks within SSL-encrypted sessions pass through without
inspection.

70 Module 2: Introducing Application Firewall © Copyright 2016 Citrix Systems, Inc.

Application Firewall is the best way to comprehensively address the challenges of delivering
centralized application-layer security for all web and web services applications and should be used
along with a network firewall.

Positive Security Model

N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io

The positive security model:

n

• Enforces correct application behavior

• Enforces Request For Comments (RFC) compliance
• Allows only safe client requests to the application
• Blocks client requests that violate best practices and protocol standards
• Requires no attack signature database
• Protects application and application logic
• Provides real-time protection for known and unknown attacks
Application Firewall uses a positive security model. Rather than relying on detecting attempts to
violate security, the Application Firewall recognizes valid or legitimate behavior and blocks

© Copyright 2016 Citrix Systems, Inc. Module 2: Introducing Application Firewall 71

everything else, allowing only traffic that is known to be valid through the firewall. Its filtering rules
are restrictive. If a request or response falls outside the boundaries of the security rules, then the
request or response is blocked. For the traffic to be allowed, the rules must be modified or relaxed
to allow the new traffic. As a result, the Application Firewall can protect web applications from
unknown attacks and zero-day attacks in addition to already known attacks.

Negative Security Model

A negative security model operates on a principle where everything is allowed except for what is
explicitly blocked. This model:
• Relies on using signatures to detect known attacks and specific types of suspicious behavior
likely to breach security on a protected server or network
• Resembles the functionality of an antivirus program and does not protect against unknown
attacks
N

• relies on identifying attack or threat and then defining a block or protection against such an
ot

attack
fo

Zero-day attack refers to the initial release of an attack or threat. Negative security models are
rr

vulnerable during the initial release of a new attack because a protection has not yet been defined.
Therefore, the application is subject to a window of vulnerability (a window of opportunity for an
es

attacker) until a block is defined.

al

Another limitation of this model is that if SSL connections are not terminated at the device, then
encrypted traffic is passed through the device without inspection.
e
or

Deep Stream Inspection

di
s

In a TCP/IP network, packets are the basic unit of data routed between an origin and a destination.
tri

A single message is divided into small units (packets) before being sent. In a typical scenario, each
packet is transmitted individually. Once all packets that form a message arrive at the destination,
bu

they are recompiled and then processed.

t io

Network firewalls perform basic packet inspection at layer 3. The network firewall does not look at
the payload of the packet or inspect each packet individually.
n

Application Firewall performs deep stream inspection of incoming and outgoing traffic. Rather than
just looking at individual packets in isolation of each other, Application Firewall reassembles all
application data for inspection, tracks individual sessions, and remembers what traffic came from
where, making full bi-directional analysis possible. As a result, Application Firewall can identify a
message containing an attack or threat.
Deep stream inspection uses full header and payload inspection to analyze all HTTP or HTTPS
traffic, searching for non-protocol compliance or predefined criteria to determine whether the
packet should be allowed to pass to or from the web server.

72 Module 2: Introducing Application Firewall © Copyright 2016 Citrix Systems, Inc.

Adaptive Learning Engine

N
ot
fo
rr
es
al
e
or

As the figure shows, Application Firewall incorporates an adaptive learning engine that discovers
and learns multiple aspects of application behavior, including:
di

• Modifications to URLs, cookies and form fields by client-side JavaScript

s

• Permissible data types for all application inputs

tri

Once application behavior is learned, Application Firewall generates policy recommendations using
bu

regular expressions, allowing an organization to set its security policies and deliver strict access
control to specific applications. The knowledge gained through policy recommendations and
t io

application learning also helps security managers better understand application behavior.
n

The adaptive learning engine automatically generates recommendations that:

• Are based on existing traffic patterns
• Reduce false positives
• Tailor themselves to changing application environments
The adaptive learning engine feature can be enabled in the user interface for the appliance and
customized in each profile for the web content it protects using that profile. Configurations can also
be disabled if necessary.
The adaptive learning engine identifies recommended settings. None of the recommendations are in
effect until an administrator deploys the rules.

© Copyright 2016 Citrix Systems, Inc. Module 2: Introducing Application Firewall 73

 The learning feature can be configured to assist in deploying the Application Firewall, but
it is not a requirement of deployment.

Web Application Vulnerabilities

Application Firewall provides superior web application security using a full positive security model
that protects against both known and unknown attacks. Application Firewall:
• Protects against web vulnerabilities automatically
• Protects against all web worms
• Protects against the OWASP (Open Web Application Security Project) Top 10 for 2013, which
are:
N

• Injection
ot

• Broken authentication and session management

• Cross-site scripting
fo

• Insecure Direct Object References

rr

• Security Misconfiguration
es

• Sensitive Data Exposure

al

• Missing Function Level Access Control

e

• Cross-Site Request Forgery (CSRF)

• Using Components with Known Vulnerabilities
or

• Unvalidated Redirects and Forwards

di

• Assists in achieving PCI-DSS (Payment Card Industry-Data Security Standard) compliance by

s

protecting against attacks detailed in PCI Section 6.5

tri
bu

Security Audits and Application Firewalls

t io

Secure coding practices and code review are core elements of a security-oriented application
n

development process. However, they are not sufficient to remove vulnerabilities. Code reviews are
an expensive undertaking and can take months to complete. Also, new application vulnerabilities
and attacks are regularly discovered, even in major commercial applications.
An organization relying solely on source code review and security audits may find themselves in an
expensive and time-consuming development lifecycle. The cost and time spent updating the
application, auditing, fixing the problem, auditing and then updating again is not the best business
approach compared to the protection offered by Application Firewall.
Application Firewall blocks vulnerabilities in all types of applications, complementing and enforcing
secure coding and security auditing best practices.

74 Module 2: Introducing Application Firewall © Copyright 2016 Citrix Systems, Inc.

Payment Card Industry Data Security Standard
PCI-DSS is a set of security standards defined by the Payment Card Industry to identify threats to
web applications (handling credit card payments).
PCI-DSS, first established in January 2005, provides a single global security standard that provides
specific technical guidance for protecting cardholder interests. The council that defines the standard
is made up of American Express, Discover Financial Services, JCB, MasterCard Worldwide, and
Visa International.

Importance of PCI
The current standard (PCI DSS v.1.1) specifically recognizes the unique security needs of web
applications. Organizations that directly interact with or support online credit card transactions
through a web application can be fined and possibly denied service by the credit card companies
N

for non-compliance with PCI-DSS requirements.

ot

In 2006, there were 268 reported data breaches. Credit cards and identities are being sold for
fo

approximately $1-20 United States dollars, depending on the depth of the personal information that
is provided.
rr

In December 2006, the largest credit card data theft that was disclosed involved a United States
es

clothing and housewares retailer, where it was believed that 45.7 million credit card numbers were
stolen. However, instead of reporting the breach to the federal government, as required by law, the
al

retailer believed that they could find the perpetrators and recover the numbers themselves.
e

By March 2007, the retailer realized that they could not locate the criminals or recover the stolen
or

credit card numbers. Therefore, they reported the theft to the federal government. The government
sanctioned and fined the retailer for being in breach of federal law by not reporting the theft and
di

started an investigation.
s

The retailer expected to incur $5 million in costs in connection with the computer intrusion.
tri

According to a study published by Forrester Research, information security breaches cost between
bu

$90 to $305 per lost record. Remediation costs were expected to approach $8.6 billion.
t

In December 2007, the federal investigation determined that the number of credit card numbers
io

lost was not 45.7 million; instead, it was over 90 million. The actual number of cards stolen is not
n

known due to the poor record keeping of the retailer. Experts now think remediation is over $12
billion.

Common Coding Vulnerabilities

The following table lists the common coding vulnerabilities in web application software, as defined
by PCI Section 6.5.

© Copyright 2016 Citrix Systems, Inc. Module 2: Introducing Application Firewall 75

Vulnerability Description
6.5.1 Unvalidated Input Information from web requests is not validated
before being used by a web application.
Attackers can use these flaws to attack back-end
components through a web application.

6.5.2 Broken Access Control (for example, Restrictions on what authenticated users are
malicious use of user IDs) allowed to do are not properly enforced.
Attackers can exploit these flaws to access other
users’ accounts, view sensitive files or use
unauthorized functions.

6.5.3 Broken Authentication and Session Account credentials and session tokens are not
Management (use of account credentials and properly protected. Attackers that can
N

session cookies) compromise passwords, keys, session cookies or

ot

other tokens can defeat authentication

restrictions and assume other users’ identities.
fo

6.5.4 Cross-site Scripting (XSS) Attacks The web application can be used as a
rr

mechanism to transport an attack to a user’s

es

browser. A successful attack can disclose the

user’s session token, attack the local system, or
al

forge content to mislead the user.

e

6.5.5 Buffer Overflows Web application components in some

or

programming languages that do not properly

validate input can be forced to fail and, in some
di

cases, used to take control of the process. These

s

components can include CGI, libraries, drivers,

tri

and web application server components.

bu

6.5.6 SQL Injection Flaws Web applications pass parameters when they
access external systems or the local operating
t io

system. If an attacker can embed malicious

n

commands in these parameters, the external

system may execute those commands on behalf
of the web application.

6.5.7 Improper Error Handling Error conditions that occur during normal
operation are not handled properly. If attackers
can cause errors to occur that the web
application does not handle, they can gain
detailed system information, deny service, cause
security mechanisms to disable the server.

76 Module 2: Introducing Application Firewall © Copyright 2016 Citrix Systems, Inc.

Vulnerability Description
6.5.8 Insecure Storage Web applications frequently use cryptographic
functions to protect information and
credentials. These functions and the code to
integrate them have proven difficult to code
properly, frequently resulting in weak
protection.

6.5.9 Denial of Service Attackers can consume web application

resources to a point where legitimate users can
no longer access or use the application.
Attackers can also lock users out of their
accounts or cause the entire application to fail.
N

6.5.10 Insecure Configuration Management Strong server configuration standards are

ot

critical to a secure web application. These

servers have many configuration options that
fo

affect security and are not secure.

rr

PCI-DSS v1.1 also lists 12 requirements for compliance with the standard when developing and
es

supporting web applications that support credit card transactions:

al

• Build and maintain a secure network.

e

• Protect cardholder data.

or

• Maintain a vulnerability management program.

• Implement strong access control measures.
di

• Regularly monitor and test networks.

s tri

• Maintain an information security policy.

bu

For more detailed information on web application vulnerabilities, types of application attacks and
requirements for standard compliance, refer to the following web sites:
t io

• PCI Security Standards Council: https://www.pcisecuritystandards.org/tech/

n

• Open Web Application Security Project (OWASP): http://www.owasp.org

© Copyright 2016 Citrix Systems, Inc. Module 2: Introducing Application Firewall 77

PCI-DSS Report

N
ot
fo
rr
es
al
e
or
di

The report identifies which requirements are relevant to the NetScaler system, which requirements
s tri

have been met, and what needs to be done to bring the system into compliance.
bu

Packet Processing and Inspection

t io
n

Application Firewall protects web servers by filtering traffic to and from the servers and by blocking
or rendering harmless any attacks or threats that it detects. A network appliance is installed
between web servers and users, usually behind a router or network firewall.
Depending on which Application Firewall model is being used and what other tasks it performs, it
may be installed in different locations and configured in a different manner. However, to function,
Application Firewall must be installed in a location where it can intercept traffic to the web servers
it is intended to protect as well as to the switch through which users access those servers.
NetScaler and Application Firewall features filter and process request traffic differently than
response traffic. With the NetScaler systems, policies regarding load balancing, content switching
and content filtering are primarily made against request traffic. Integrated caching involves policies
that apply to requests and responses. Cache policies are applied to requests to see if the request

78 Module 2: Introducing Application Firewall © Copyright 2016 Citrix Systems, Inc.

matches a cache hit and applied to responses to see if they match criteria to be cached for future
requests. Similarly, Application Firewall includes security checks that apply to requests, responses,
or both.

Request Process

N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io

The figure shows data flow of the request process at the device level when using a NetScaler
Application Firewall.
n

1. The user or client sends a request to a web application, which is intercepted by Application
Firewall.
2. The NetScaler system determines if the request is SSL encrypted.
• If the request is SSL encrypted, then the NetScaler system decrypts the data before passing
the data to the next module.
• If the request is not SSL encrypted, then the NetScaler system passes the data to the next
module.
3. Application Firewall evaluates traffic against Application Firewall profiles and policies
configured by administrators.

© Copyright 2016 Citrix Systems, Inc. Module 2: Introducing Application Firewall 79

 • If the request is deemed valid, then the traffic is passed to the caching module.
• If the request is deemed invalid, then the user is redirected to the error page.
4. The traffic management module examines the data and determines how to route the traffic to
the appropriate server depending on load balancing, content switching, and other
configurations.
5. The data is then passed to the rewrite module, which applies any necessary header rewriting
before the traffic is passed to the web server.
6. The NetScaler system determines whether the data should be passed in plain text or SSL.
• If the data is passed in clear text, then the traffic is forwarded to the web server.
• If the data is passed in SSL, then the traffic is encrypted and forwarded to the web server.

Response Process
N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

The figure shows the data flow of the response process at the device level when using Application
Firewall.

80 Module 2: Introducing Application Firewall © Copyright 2016 Citrix Systems, Inc.

 When traffic reaches an integrated or stand-alone Application Firewall, each system
handles traffic in the same manner.

1. The web server sends the response , which is intercepted by the Application Firewall, to the
user.
2. The NetScaler system determines if the response is SSL encrypted.
• If the request is SSL encrypted, then the NetScaler system decrypts the data before passing
the data to the next module.
• If the request is not SSL encrypted, then the NetScaler system passes the data to the next
module.
3. The NetScaler traffic management module determines which user originated the request and
forwards the data to the rewriting module.
4. The rewrite module rewrites the URL.
N

5. The Application Firewall module examines the traffic using both its internal rule set and any
ot

additions or modifications set by administrators.

fo

6. The traffic is compressed or encrypted as necessary.

rr

7. The response is sent to the user.

es

Deployment Considerations
al
e

The key item regarding this process is that when Application Firewall is enabled, the Application
or

Firewall security checks process the request before most other processing occurs. A request is only
sent to the web server when it does not violate any security checks. If a violation is detected, then
di

the request is blocked or modified to negate the risk. A response is only returned to the user after it
has been verified as not violating the security checks. Application Firewall inspects data to make
s

sure sensitive information is not being returned.

tri

Understanding the packet flow process and the order in which NetScaler features and Application
bu

Firewall processing occurs is important when deciding how to deploy a NetScaler system in the
t

environment and which features are enabled and disabled.

io
n

Profiles and Policies

Application Firewall evaluates traffic against profiles and policies configured by the administrator.
Profiles and policies are configured in the Configuration Utility or in the command-line interface.

Profiles
An Application Firewall profile is a collection of settings that tell Application Firewall which
security checks to use when filtering a particular request or response and how to handle a request

© Copyright 2016 Citrix Systems, Inc. Module 2: Introducing Application Firewall 81

or response that fails a security check. Application Firewall profiles allow immediate protection,
such as by preventing theft of sensitive data types and unauthorized access.
An administrator must create a profile during configuration to tell Application Firewall how to
protect content. A profile can be created in the Configuration Utility or in the command-line
interface.

Policies
Policies for both Application Firewall and NetScaler systems use the AppExpert Policy Engine.
Application Firewall policies allow administrators to assign filtering rules to different types of web
content. For example, an administrator can create policies that distinguish between a request made
for a simple text and graphics web page and one made for a JavaScript-enhanced form.
N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

82 Module 2: Introducing Application Firewall © Copyright 2016 Citrix Systems, Inc.

 3
Module 3

Profiles and Policies

N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

84 © Copyright 2016 Citrix Systems, Inc.

Profiles and Policies Manual
Overview
Profiles are a collection of settings that determine which security checks to apply to HTTP/S traffic.
Application Firewall distinguishes legitimate traffic from potential attacks and determines what
action to take when a request or response fails the check.
Policy expressions allow a NetScaler system to intelligently manage, switch, and validate application
traffic based on an aspect of a specific connection.

Objectives
N

At the end of this module, you will be able to:

ot

• Identify when to use a basic profile and when to use an advanced profile.
fo

• Identify the application profile types.

rr

• Create a profile.
• Configure profile settings.
es

• Identify the role profiles and security checks play in protecting web applications.
al

• Configure and bind Application Firewall policies.

e

• Understand Application Firewall Engine settings.

or
di
s tri
bu
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 3: Profiles and Policies 85

Profiles

N
ot
fo
rr
es
al
e
or
di

A profile is a collection of security checks that protect HTML, XML, or HTML/XML web
s tri

applications. The profile is applied against requests and responses to identify, block, or negate
security violations.
bu

The profile includes the following:

t io

• Profile settings
n

• Security checks with actions and settings

• Learning
Profiles are collections of settings. Profiles allow different security checks and settings to be used
with different web applications; the security checks, behavior, learning, blocking and relaxations are
configured specific to the web application.
As the figure shows, the security checks examine requests or responses and determines if the traffic
violates security or poses a risk. Security check actions, such as block or log, may be taken.

86 Module 3: Profiles and Policies © Copyright 2016 Citrix Systems, Inc.

Profile Types
There are three types of profiles available: Web Application, XML Application, and Web 2.0
Application (or Web Services). Each profile type determines which security checks or protections
are available. Security checks may apply to all profiles and application types or may be specific to
HTML applications or XML applications.
The key differences between these profile types are which security checks are available within the
profile:
• Web Application (HTML): Common and HTML security checks
• XML Application (XML, SOAP): Common and XML security checks
• Web 2.0 Application (HTML, XML, REST): Common, HTML, and XML security checks
The profile type is specified when the profile is created and cannot be changed after the fact, unless
the profile is recreated.
N
ot

Default Profiles
fo

The settings of a new profile are defined by one of two sets of defaults, which differ in the
rr

protections that are initially enabled or disabled. The following table lists the default profiles.
es
al

Profile Description
e

Basic Protects most types of web content with little

or

additional configuration or maintenance

di

Such profiles rarely cause false positives or

block legitimate responses or requests.
s tri

Advanced Protects more complex web content

bu

These profiles require initial configuration and

follow-up maintenance to make sure that they
t io

are working properly.

n

The advantage of beginning with basic or advanced defaults when creating a new profile is that an
administrator can optimize which security checks are applied to specific areas of a web application.
The key difference between basic and advanced defaults is that advanced defaults enable
sessionization, while basic defaults do not. Some protections enabled in advanced defaults require
sessionization, which affects performance and requires more CPU and memory than protections
that do not require sessionization.

© Copyright 2016 Citrix Systems, Inc. Module 3: Profiles and Policies 87

Basic Profiles
The Basic profile is intended to provide initial security protections against forceful browsing, cookie
tampering, web form tampering and buffer overflow attacks. Cross-site scripting and SQL injection
checks are also enabled. Sessionization is not enabled. Non-session based security checks are
enabled and session-based security checks are not enabled by default. Learning is not enabled.
The Basic profile default settings are intended to provide security for basic HTML and XML
applications without interfering in the function of the applications. If the HTML or XML content
does not access back-end SQL databases, does not contain active scripts, and does not access
sensitive data, then it is possible that the Basic default profile settings can be deployed with
minimal, if any, additional configuration. If the application does include any of these features, then
additional settings should be configured to provide security.

Advanced Profiles
N
ot

The Advanced profile is intended to provide a starting configuration for sites that contain legacy
CGI and complex JavaScript code, access SQL back-end databases, and otherwise involve sensitive
fo

information with the expectation that the profile needs to be further configured and customized to
rr

function with the application. The assumption is that by starting with the Advanced profile default
settings, the profile must be configured or modified with additional settings to work properly with
es

an application.
al

The Advanced profile default setting includes protections against forceful browsing, cookie
tampering, web form tampering, buffer overflow, SQL injection, and cross-site scripting. Cookie
e

tampering protection, which is session-based, is also included. The other significant difference is
or

that Learning is enabled for the security checks by default.

Sessionization is enabled in advanced profiles. The features that require sessionization are:
di
s

• Form Field Consistency

tri

• Cookie Consistency
bu

• URL Closure
t io

Creating a Profile in the Configuration Utility

n

Use the following procedure to add a new profile:

1. Go to Security > Application Firewall > Profiles.
2. Click Add.
3. Type a name for the profile in the Profile Name field. Example: Application_Basic
4. Determine whether to use basic or advanced defaults. >
• Select Basic if the profile applies to a variety of web content.
• Select Advanced if the profile applies to specific content.
5. Click OK.

88 Module 3: Profiles and Policies © Copyright 2016 Citrix Systems, Inc.

Creating a Profile in the Command-Line Interface
Enter the following command to create a profile:

add appfw profile name -defaults type

The following table lists the arguments.

Argument Description
name The name of the profile being configured

type The type of defaults with which a profile should

start, which can be either basic or advanced
N
ot

Security checks are separate objects that must be bound to the profile when using the command-
line interface.
fo

For example, enter the following command to create a new profile named checkout with advanced
rr

defaults:
es

add appfw profile checkout -defaults advanced

al
e
or
di
s tri
bu
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 3: Profiles and Policies 89

Action Settings

N
ot
fo
rr
es
al
e
or
di

Enabling an action setting causes Application Firewall to examine all traffic associated with the
s tri

profile for violations of that particular protection.

bu

Block
t io
n

The Block setting determines whether requests or responses that fail the security check should be
allowed through Application Firewall. Checking the box next to Block stops such traffic from
reaching its destination.
Blocking a response includes:
• Displaying part or none of the web page
• Masking the blocked content with X’s
• Removing the blocked content from the web page

90 Module 3: Profiles and Policies © Copyright 2016 Citrix Systems, Inc.

Logs
The Log setting determines whether requests or responses that fail the security check should be
logged for future review. Checking the box next to Log causes the Application Firewall to document
each violation as an event in the syslog log file, noting which profile and check initiated the event.

Statistics
The Statistics setting determines whether to calculate statistics on requests or responses that fail the
security check for future review. Checking the box next to Statistics will begin such calculations,
which can be viewed on the NetScaler Monitoring page.

Learn
N
ot

The Learn action enables learning for the security check. Learning is the process by which the
learning engine monitors traffic to and from the protected web sites and generates a list of
fo

recommended relaxations for the security check.

rr

When learning is enabled, violations to the security check are monitored. If violations occur and
exceed the learning thresholds, then the learning engine adds the field, URL or cookie to the list of
es

rules.
al

An administrator should review the rules and determine if the recommended relaxations represent
e

legitimate traffic that should be allowed, or if the traffic should be blocked. The administrator can
use the learning engine to generate very narrow rules to allow very specific relaxations or to
or

generalize rules to allow more broad relaxations. The rules suggested by the learning engine can
then be deployed without modification, modified and deployed, or skipped. The administrator uses
di

the learning engine to configure the Application Firewall security checks without having to
s

manually create all necessary relaxations. The learning engine also helps non-application
tri

administrators assess the applications and configure Application Firewall.

bu

During an initial deployment, learning can be enabled with blocking disabled. This enablement
allows the Application Firewall to monitor traffic, identify traffic that triggers a violation and
t io

generate rules based on the application learning without blocking traffic until Application Firewall
is fully configured. Once the device is configured, learning is often turned off.
n

Additional Action Settings

The following table lists additional actions that some security checks support.

© Copyright 2016 Citrix Systems, Inc. Module 3: Profiles and Policies 91

Action Description
X-Out Sensitive data, such as credit card, social
security and telephone numbers, is masked with
X's before being returned to the user. When
both X-Out and Block are selected, Block
overrides X-Out.

Remove The content that violates the security check is

removed from the response, resulting in part or
all of the response being omitted from the user.

Transform The request is allowed to proceed while

negating the risk associated with the attack. This
setting is available for SQL injection and cross-
N

site scripting.
ot
fo

Modifying Action Settings in the Configuration Utility

rr

Use the following procedure to modify the action settings for a security check in a profile.
es

1. Go to Security > Application Firewall > Profiles.

al

2. Double-click the profile to be modified.

e

3. Click the Security Checks node.

or

4. Double-click the security check to be modified.

5. Enable action settings:
di
s

• Select Block to stop all requests or responses that violate the security check.
tri

• Select Log to record all requests or responses that violate the security check as events in
bu

the log file.

• Select Statistics to generate statistics on all requests or responses that violate the security
t io

check.
n

6. Click OK, Save & Close then click Done to save the changes and to close the open dialog
boxes.

Modifying Action Settings Using the Command-Line

Interface
Enter the following command to modify the action settings for a security check in a profile:

set appfw profile name-checkActionactions

92 Module 3: Profiles and Policies © Copyright 2016 Citrix Systems, Inc.

The following table lists the arguments.

Argument Description
name The name of the profile

checkAction A security check with one or more of the

following settings:
• startURLAction
• cookieConsistencyAction
• fieldConsistencyAction
• bufferOverflowAction
• fieldFormatAction
N

• denyURLAction
ot

• CrossSiteScriptingAction
fo

• SQLInjectionAction
rr

• creditCardAction
• safeObjectAction
es
al

actions A list of action settings to be enabled, which can

include one or more of the following options:
e

• Block
or

• Log
di

• Statistics
s

• Learn
tri

Any setting that is listed is enabled. Any setting

bu

that is omitted is disabled, even if it was

previously enabled.
t io
n

For example, enter the following command to enable blocking and logging in the Start URL
security check of the pr_basic profile:

set appfw profile pr_basic -startURLAction block log

Modifying Relaxations in the Configuration Utility

Use the following procedure to modify the Start URL Check relaxation:
1. Go to Security > Application Firewall > Profiles.

© Copyright 2016 Citrix Systems, Inc. Module 3: Profiles and Policies 93

2. Double-click the profile to be modified.
3. Click the Relaxation Rules node.
4. Double-click Start URL.
5. Highlight the Start URL to be modified.
6. Enable or disable the Start URL check relaxation.
7. Note the literal string or PCRE-format regular expression that designates the URL in the Start
URL field.

The regular expression must consist of ASCII characters only.

8. Type a comment in the Comments field (optional).

9. Click Close and then Done.
N
ot

Sessionization and Security Checks

fo
rr

The HTTP protocol is stateless; therefore, either cookies or query data appended to the URL body
is required to track a user’s requests and responses.
es

To perform certain security checks, Application Firewall must sessionize the traffic (requests and
al

responses) between the user and the protected web site or application. Sessionization is the
Application Firewall process of creating a session, using a cookie, to track the progress and state of
e

a user.
or

Sessionization is required to enable more in-depth protection for the site by enabling the
di

Application Firewall to track and maintain state information across multiple requests and
responses.
s tri

For example, with sessionization, Application Firewall verifies a form submission in a client request
and verifies that the submitted form matches the actual form template in the original server
bu

response. This session-based inspection allows Application Firewall to verify that the form was not
t

altered or modified as a part of a form field manipulation attack. Without sessionization,

io

Application Firewall can only track the user’s current request (the altered web form) and cannot
n

compare the request with the original response to detect that a security violation occurred.

How Sessionization Works

When a new session begins, Application Firewall attaches a signed session cookie to the server
response before forwarding it to the client. Application Firewall cookies are still generated to track
state even when no application cookies are sent. The default session timeout is 15 minutes (900
seconds). Once the session times out, a user must access the application from a configured Start
URL to start a new session before being allowed to continue.

94 Module 3: Profiles and Policies © Copyright 2016 Citrix Systems, Inc.

Application Firewall also tracks outbound data in memory as part of the sessionization process in
addition to the cookie.
Session settings are configured within the Application Firewall Engine Settings and are global to the
system; they cannot be configured per profile. Administrators can configure the name of the session
cookie and the session timeout.
Additional considerations regarding sessionization are discussed in the deployment planning
module.

Profile Settings
N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

As the figure shows, profile settings control the profile behavior for the redirect URL to use when
blocking traffic and behavior for handling comments and other settings. Profile configurations are
viewed on the Settings tab of a profile configuration window.

© Copyright 2016 Citrix Systems, Inc. Module 3: Profiles and Policies 95

Error Page
When a request fails a security check that has blocking enabled, Application Firewall needs to know
where to redirect the user’s browser. Otherwise, the user receives no feedback that the request has
been blocked and may assume the problem resides with the connection to the site.
Therefore, each profile has an error page, which is a web page to which users are redirected in case
a request is blocked. By default, the error page is set to slash (/), which directs requests to the home
page of the protected web site.
The error page has a special implicit function when URL Closure is enabled. The error page acts as
an implicit start URL which, when set to /, allows the user access to the application even if no start
URL is defined.
Administrators may choose to designate another absolute or root-relative URL for the profile error
page. This page could include an external site or a special error page on the protected site that
informs the user that their request has been blocked. Citrix recommends to set the error page
N

outside the application.

ot

A redirect URL is specified for HTML web applications. An XML redirect object is specified for
XML-based web applications.
fo
rr

Administrators should carefully select the content of an error page. An error page should
not provide any information about the site or the firewall that hackers could use to their
es

advantage, nor should the error message taunt a would-be hacker for a failed attempt.
al

Taunting error messages could encourage hackers to try again.

e
or

Setting the URL Redirect in the Configuration Utility

di

Use the procedure in the following table to set a profile error page.
s

1. Go to the Security > Application Firewall > Profiles.

tri

2. Double-click the profile to be modified.

bu

3. Click the Profile Settings node.

t io

4. Under Redirect URL, type any valid relative or absolute URL to be used as the error page.
n

5. Click OK, then click Done to save the current settings and close the Profile Configuration
window.

Setting the URL Redirect in the Command-Line Interface

Enter the following command to set a profile error page:

set appfw profile name -errorURL "URL"

The following table lists the arguments.

96 Module 3: Profiles and Policies © Copyright 2016 Citrix Systems, Inc.

Argument Description
name The name of the profile

URL The absolute or relative URL of the error page

For example, enter the following command to set the error page of the pr_basic profile to a page
called error.html:

set appfw profile pr_basic -errorURL "/error.html"

HTML Comment Stripping

N

Comments are useful in development, but they can be hazardous to application security if they
ot

reach the outside world. Application Firewall can strip all HTML comments from a response of a
protected site before sending it to the user. This setting is disabled in HTML and Web 2.0 profiles
fo

by default.
rr

However, if JavaScript is inserted inside HTML comments, it is also removed when this feature is
es

enabled. Such scripts would be stripped along with the comments and would potentially break the
application. Therefore, an administrator should understand the dependencies within the application
al

and security risks when determining whether to use this setting.

e
or

Stripping HTML Comments in the Configuration Utility

di

Use the following procedure to strip HTML comments from a protected web application:
s tri

1. Go to Security > Application Firewall > Profiles.

bu

2. Double-click the profile to be modified.

3. Click the Profile Settings node.
t io

4. Click Strip HTML Comments drop down menu and select All.
n

5. Click OK, then Done to save the current settings.

Stripping HTML Comments in the Command-Line Interface

Enter the following command to strip HTML comments from a protected web application:

set appfw profile name -stripComments setting

The following table lists arguments.

© Copyright 2016 Citrix Systems, Inc. Module 3: Profiles and Policies 97

Argument Description
name The name of the profile

setting The new setting for HTML Comment Stripping,

which can be either ON or OFF

For example, enter the following command to enable HTML comment stripping in the pr_basic
profile:

set appfw profile pr_basic -stripComments ON

XML Error Object

N
ot

The XML error object is the redirect object for XML web applications and serves a similar function
as the HTML redirect URL. The XML Error object is the object to which Application Firewall
fo

directs blocked XML requests or responses. By default, it is not set. The XML error object for the
rr

XML application or web services application must be uploaded through the Import button. If
multiple error pages are imported, then different XML Error Objects to be used for each profile can
es

be selected.
al
e

Other Profile Settings

or

The Citrix Application Firewall Guide includes descriptions and considerations for the other Profile
di

Settings not discussed here. The default settings are generally acceptable for most applications.
However, settings can be modified to improve performance or compatibility for large
s tri

implementations.
bu

Policies
t io
n

Application Firewall uses classic policies, which evaluate basic characteristics of traffic and other
data. For example, classic policies can identify whether an HTTP request or response contains a
particular type of header or URL.
Application Firewall policies are similar to other policies in the rest of the NetScaler operating
system, but there are some differences and considerations to take into account. In other NetScaler
features, policies are typically composed of an expression (rule) and an action. The policy defines
what action to take on any entity matching the rule.
The Application Firewall policy defines an expression, which identifies the traffic that should be
filtered and the profile that should be used to process the traffic. When working with Application
Firewall profile and policy entities, the profile is essentially the policy action. The profile includes
all of the security checks and settings that need to be applied.

98 Module 3: Profiles and Policies © Copyright 2016 Citrix Systems, Inc.

One key difference regarding Application Firewall policies and other NetScaler policies is that the
application policies filter HTTP connections only (the NetScaler system treats HTTPS as HTTP).

Policy Creation
During the initial configuration of Application Firewall, an administrator should configure one
policy that protects all content traveling to and from a web application. If necessary, an
administrator can later create additional policies that define specific types of content or specific
parts of a web application that require customized protections.

Policy Binding
A policy is associated with, or bound to, an entity that enables the policy to be invoked. For
N

example, an administrator can bind a policy to request-time evaluation that applies to all virtual
servers. Policies can also be bound globally. A collection of policies that are bound to a particular
ot

bind point constitutes a policy bank.

fo

The following table lists the different types of policy bind points.
rr
es

Bind Point Description

al

Request Time Global A policy can be available to all components in a

e

feature at request time.

or

Response Time Global A policy can be available to all components in a

feature at response time.
di
s

Request Time, Virtual Server-Specific A policy can be bound to request-time

tri

processing for a particular virtual server. For

bu

example, an administrator can bind a request-

time policy to a cache redirection virtual server
t

to ensure that particular requests are forwarded

io

to a load balancing virtual server for the cache,

n

and other requests are sent to a load balancing

virtual server for the origin.

Response Time, Virtual Server-Specific A policy can also be bound to response-time

processing for a particular virtual server.

User-Defined Policy Label For advanced policies, an administrator can

configure custom groupings of policies (policy
banks) by defining a policy label and collecting
a set of related policies under the policy label.

© Copyright 2016 Citrix Systems, Inc. Module 3: Profiles and Policies 99

Bind Point Description
Other Bind Points The availability of additional bind points
depends on policy type (classic or advanced)
and specifics of the relevant NetScaler feature.
For example, classic policies for NetScaler
Gateway have user and group bind points.

Policy State
By default, a globally bound policy is immediately put into effect. However, in some cases, a policy
needs to be reviewed before it is active. By deselecting the State checkbox, an administrator can
bind a policy without putting it into effect. After the policy is reviewed, the State checkbox can be
selected to make the policy active.
N
ot

Policy Priorities
fo
rr

The lower the number, the sooner the policy is evaluated relative to other policies. For example, if
Application Firewall has three policies with priorities of 10, 100, and 1000, the policy assigned a
es

priority of 10 is performed first. All NetScaler features, except the rewrite feature, implement only
al

the first policy that a connection matches. The rewrite policy implements multiple policies in order
of priority, so policy priority is important to get the results that an administrator intends.
e

As a best practice, an administrator should leave room to add policies by setting priorities with
or

intervals of 50 (or 100) between each policy.

di
s

Creating a Policy in the Configuration Utility

tri
bu

Use the following procedure to create a policy for the Application Firewall:
t

1. Go to Security > Application Firewall > Policies > Firewall node.

io

2. Click Add.
n

3. Type a name in the Name field.

4. Select a profile in the Profile drop down list.
5. ClickExpression Editor.
6. Select the HTTP protocol.
7. Select a Request REQ or Response RES.
8. Select a qualifier from the Qualifier list.
9. Select an Operator from the Operator list.
10. Type a value in the Value field.
11. Type a length and offset, if appropriate.

100 Module 3: Profiles and Policies © Copyright 2016 Citrix Systems, Inc.
12. Click Done and then Create.

Creating Policies in the Command-Line Interface

Enter the following command to create a policy:

add appfw policy name rule profile_name

The following table lists the arguments.

Argument Description
name Designates the name of the policy object
N

rule Defines the expression used to evaluate traffic

ot

profile_name Designates the name of an existing profile

fo
rr

For example, enter the following command to create a policy that matches traffic sent to the
es

BadStore site with the pr_badstore profile:

al

add appfw policy BadStore "REQ.HTTP.URL CONTAINS BadStore"

e

pr_badstore
or

Enter the following command to remove a policy:

di

rm appfw policy name

s tri

A policy can also be created with a named expression. For example, enter the following command
bu

to add an expression:
t io

add exp Exp1 "REQ.HTTP.URL == /BadStore.net"

n

Enter the following command to add a profile:

add appfw profile BadStore_1

Enter the following command to add a policy:

add appfw policy BadStore Exp1 BadStore_1

Enter the following command to bind the policy:

© Copyright 2016 Citrix Systems, Inc. Module 3: Profiles and Policies 101
 bind appfw global BadStore 30

Binding and Prioritizing a Policy in the Configuration Utility

Use the following procedure to bind a policy:
1. Go to Security > Application Firewall.
2. Click Application Firewall Policy Manager.
3. Verify that Override Global is in the Bind Point and click Continue.
4. Click in the field below Select Policy.
5. Click the radio button next to the desired policy then click Select.
6. Click Bind then click Done.
N
ot

Binding Policies in the Command-Line Interface

fo

Enter the following command to bind a policy and set its priority:
rr
es

bind appfw global policy_name priority [-

state (ENABLED | DISABLED)]
al
e

The following table lists the arguments.

or

Argument Description
di
s

policy_name Designates the name of the policy object

tri
bu

priority Defines the policy priority as an integer

tio

For example, enter the following command to bind the BadStore policy:
n

bind appfw global BadStore 10

Enter the following command to unbind a policy:

unbind appfw global policyName

Enter the following command to display information about a policy:

show appfw policy name

Enter the following command to display information about all globally bound policies:

102 Module 3: Profiles and Policies © Copyright 2016 Citrix Systems, Inc.
 show appfw global

Engine Settings
Engine settings are global to Application Firewall and are not part of individual profiles.
Configuring the engine settings affects all profiles.
The following table lists engine settings.

Engine Setting Description

Cookie name The name of the cookie that stores the
NetScaler session ID
N
ot

Session timeout The maximum inactive period allowed. If a user

session shows no activity for this length of time,
fo

the session is terminated and the user is

rr

required to reestablish it by visiting a designated

start page.
es

Cookie post-encrypt prefix The string that precedes the encrypted portion
al

of any encrypted cookies

e

Maximum session lifetime The maximum amount of time, in seconds, that

or

a session is allowed to remain live. After this

period is reached, the session is terminated and
di

the user is required to reestablish it by visiting a

s

designated start page. This setting cannot be less

tri

than the session timeout. To disable this setting,

bu

so that there is no maximum session lifetime,

set the value to zero.
t io

Logging header name The name of the HTTP header that holds the
n

Client IP, for logging.

Undefined profile The profile applied when the corresponding

policy action evaluates as undefined.

Default profile The profile applied to connections that do not

match a policy.

© Copyright 2016 Citrix Systems, Inc. Module 3: Profiles and Policies 103
Engine Setting Description
Import size limit The maximum cumulative total byte count of all
files imported to the ADC, including signatures,
WSDLs, schemas, HTML and XML error pages.
During an import, if the size of the imported
objects would cause the cumulative total sizes of
all imported files to exceed the configured limit,
the import operation fails and the ADC displays
the following error message: ERROR: Import
failed - exceeding the configured total size limit
on the imported objects.

Learn message rate limit The maximum number of requests and

responses per second that the learning engine is
N

to process. Any additional requests or responses

ot

over this limit are not sent to the learning

engine.
fo

Entity decoding Decode HTML entities when running

rr

application firewall checks.

es

Log malformed request Enable logging of malformed HTTP requests

al

Use configurable secret key Use a configurable secret key for application
e

firewall operations.
or

Reset learned data Remove all learned data from the application
di

firewall. Restarts the learning process by

s

collecting fresh data.

tri
bu
t io
n

104 Module 3: Profiles and Policies © Copyright 2016 Citrix Systems, Inc.
 4
Module 4

Regular Expressions
N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

106 © Copyright 2016 Citrix Systems, Inc.

Regular Expressions Manual
Overview
Regular expressions are text strings formatted according to particular syntax rules and are used to
describe a search pattern. They save time and effort by requiring only a single pattern definition
that the regular expression engine then uses to search for strings that match the definition. For
example, rather than writing a separate literal string for each page on a web site, a regular
expression can be written that includes all pages whose URL starts with the address of the home
page. The address may or may not be followed by a forward slash and additional text.
Perl-Compatible Regular Expressions (PCRE) can be used to define settings for many of the
security checks in a profile.
N
ot

Objectives
fo

At the end of this module, you will be able to:

rr

• Understand PCRE syntax.

es

• Identify PCRE metacharacters.

• Write regular expressions using PCRE.
al
e
or
di
s tri
bu
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 4: Regular Expressions 107

Regular Expressions

N
ot
fo
rr
es
al
e
or
di

The power of regular expressions is in the ability to define patterns to match strings. The regular
s tri

expression language can be used to create incredibly complex and comprehensive patterns. A single
expression can be used to match multiple results, as the figure shows. Therefore it is important to
bu

be familiar with how the regular expressions are evaluated and verify that the patterns defined
match the strings desired and exclude everything else.
t io

Application Firewall can interpret regular expressions in a variety of rules related to profile
n

configuration. Almost all of the security checks can use regular expressions in place of literal strings
to ease the burden on administrators when defining URLs, fields and cookies.
Regular expressions can be used within a variety of rules within the Application Firewall settings,
including profile configuration and security checks. Almost all of the security checks support the
use of regular expressions in place of literal strings when defining URLs, fields, and cookies.

108 Module 4: Regular Expressions © Copyright 2016 Citrix Systems, Inc.

 • Multiple regular expression formats have been defined. Different regular expression
formats will use different meta-characters and special characters. When creating
regular expressions for NetScaler and Application Firewall, use Perl Compatible
Regular Expression (PCRE) format. Do not use POSIX-based regular expressions or
other formats with the NetScaler-based Application Firewall implementations.
• The Application Firewall does not consider case when interpreting regular
expressions.

Forms of Regular Expressions

Regular expressions come in multiple formats, each with its own set of rules for how to define
patterns.
Application Firewall uses PCRE to match patterns. Previous versions of Application Firewall not
N

based on the NetScaler OS used POSIX, which is similar to PCRE but differs in a few subtle ways.
ot

One important difference between POSIX and PCRE is how characters are interpreted without an
escape:
fo
rr

• In POSIX, nearly every character is assumed to be literal, unless preceded by a backslash, in

which case it is a metacharacter
es

• In PCRE, the only characters assumed to be literals are alpha-numeric. Punctuation marks used
as metacharacters are automatically interpreted as such, only requiring an escape if they should
al

be interpreted as a literal
e
or

Using Regular Expressions

di
s

Regular expressions can be used for a variety of rules in place of literal strings. The use of regular
tri

expressions is optional. However, regular expressions can be used to provide sophisticated pattern
matching and can reduce the total number of objects that must be configured.
bu

Regular expressions can be used to define:

t io

• Application Definitions
n

• Start URLs
• Relaxations for field consistency, cookie consistency, SQL injection, and other policies.
Pattern matching allows you to consolidate rules and application definitions. For example:
• Instead of defining Field Formats Names of address1, address2, and so on, instead define
address[0-9]
• To allow URLs www.company.com and www.othercompany.com, define it as:
www[.](company|othercompany)[.]com

© Copyright 2016 Citrix Systems, Inc. Module 4: Regular Expressions 109

Metacharacters and Literal Characters
Regular expressions contain metacharacters and literal characters. Metacharacters have special
meanings that are used to create pattern matches. For example, a metacharacter can define how
many times a certain character should be repeated or how the pattern must start or end.
Metacharacters can be used alone or in combination with each other and normal characters. Literal
characters are interpreted as they are, such as letters, numbers and punctuation, where exactly one
pattern will match a literal string.
In PCRE, the only characters assumed to be literals are alpha-numeric: a-z, A-Z, 0-9. If a backslash
precedes one of these alpha-numeric characters, then its special meaning as a metacharacter can be
invoked.
All other characters have a special meaning which will be used unless the character is escaped
(preceded by a "\"), in which case it will be treated as a literal character. By default, all punctuation
marks are interpreted as metacharacters.
N
ot

Metacharacters
fo

The following characters must be preceded by the "\" character to match the character and not the
rr

metacharacter:
es

• \
al

• ^
e

• .
• $
or

• |
di

• ()
s

• []
tri

• *
bu

• +
t

• ?
io

• {}
n

• ,
The following table lists important metacharacters for Application Firewall.

110 Module 4: Regular Expressions © Copyright 2016 Citrix Systems, Inc.

Metacharacter Function Example

\ Escapes the next metacharacter. 3\.14 matches the number 3.14

Uses the character following
the backslash as a regular
character instead of using
another metacharacter

() Groups related expressions (10\.0\.0\.\d{1,3})|(34\.0\.1\.\d{1

,3}) returns a true value for an
IP address between 10.0.0.0 and
10.0.0.255 or between 34.0.1.0
and 34.0.1.255

^ Matches the beginning of a line ^www matches anything that

N

begins with www

ot

. Matches any single character, 10\.0\.0\.. returns a true value

fo

except newline characters for IP addresses between

10.0.0.0 and 10.0.0.9
rr

The first three '.' have a '\' in

es

front of them to match a

al

decimal point and not any

character.
e
or

$ Matches the end of a line, 5$ returns true values for IP

except newline characters addresses ending in 5
di
s

| Denotes an OR expression (400)|(404) returns true values

tri

if one of the error codes is

found
bu

\w Matches an alphanumeric firewall\w returns a true value

t io

character or underscore (a-z, for firewalla, firewall1 or

A-Z, 0-9 and _) firewall_
n

\W Matches any non-alphanumeric computer\W1 returns a true

character (such as value for computer#1,
~!@#$%^&*+={}[]:;), white computer~1 or computer=1
space, carriage returns, and line
feed characters. This
metacharacter does not match
underscores "_".

© Copyright 2016 Citrix Systems, Inc. Module 4: Regular Expressions 111

Metacharacter Function Example

\d Matches any digit, 0-9 ^232\d$ returns a true value

for a port between 2320 and
2329

\D Matches any non-digit value firewall\D returns a true value

for firewalla and firewall_200
but not firewall2

\b Matches at a word break (a \b1000\b returns a true value

word character "\w" followed for 1000 at the beginning,
by a non-word character "\W") middle or end of a line, but not
10000, 21000 or 100
N

Inside a character class

([\b]), a \b represents a
ot

backspace.
fo

\B Matches at a non-word break port\B1000 returns a true value

rr

(middle of either "\w\w" or for port1000, but not

es

"\W\W") port_1000, port~1000,

port=1000 or port 1000
al

\s Matches any white space port\s1000 returns a true value

e

character for port 1000, but not for

or

port_1000
di

\S Matches any non-white space port\S1000 returns a true value

s

character for port=1000, port~1000,

tri

port:1000, port1000 or
port_1000 but not port 1000
bu

\t Matches a tab character ip\t192\.168\.1\.2 returns a true

t io

value for tab-delimited values,

such as ip 129.168.1.2
n

[] Matches one character within firewall_[abd] returns a true

the set, which can include a value for hits to firewall_a,
range or a list of characters; a firewall_b or firewall_d
character class
firewall[1-6] returns a true
value for firewall1 to firewall6.

'-' indicates a range when

used in a character class.

112 Module 4: Regular Expressions © Copyright 2016 Citrix Systems, Inc.

Metacharacter Function Example

[^] Matches one character not in [^abc] and [^a-c] matches d, e,

the set, which can include a f
range or a list of characters
firewall[^02468] returns a true
value for odd numbered
firewalls in the system.

Escapes
To separate normal interpretations from metacharacters, an escape is used, which tells the engine to
ignore how that character is usually interpreted. In many cases, this escape is a backslash (\).
Metacharacters are standalone characters that have reserved, special meanings. Examples: \ . + * ^ $
N
ot

To treat these metacharacters as literal characters, they must be escaped. For the metacharacters,
they must be preceded by a backslash: \\ \. \+ \* \^ \$
fo

All alphanumeric characters are treated as literals. In order to use alphanumerics for special
rr

meanings within a regex, they must be escaped.

es

Example: Escaping d and D as \d and \D are used to match digit and non-digit patterns. Numbers
can be escaped for use with backreferences.
al

Remember:
e

• Escape metacharacters to use them as literal values.

or

• Escape regular literals (alphanumerics) to use their special meanings.

di

When using the dot to define URLs or other content, it is also acceptable -- and
s

recommended for easier reading -- to escape this metacharacter by including it in a set.

tri

Remember the square brackets denotes a single character set. For example, the
bu

www[.]citrix[.]com URL will be interpreted as a literal string, just like the

www\.citrix\.com URL.
t io
n

Quantifiers
Quantifier characters search for the number of occurrences of a character or set of characters.
Quantifiers are listed in the following table.

Quantifier Function Example

* Matches 0 or more of the 13*2 returns a true value for
preceding character port 12, 132 and 1332

© Copyright 2016 Citrix Systems, Inc. Module 4: Regular Expressions 113

Quantifier Function Example
+ Matches one or more of the 13+2 returns a true value for
preceding character port 132 or 1332 but not 12

? Matches zero or one of the 1?8 returns a true value for

preceding character server 18 or 8

{n} Matches exactly n of the 10\.0\.0\.\d{2} returns a true

preceding character value for an IP address between
10.0.0.10 and 10.0.0.99

{n,} Matches at least n times 10\.0\.0\.\d{1,} returns a true

value for an IP address between
10.0.0.0 and 10.0.0.255
N
ot

{n,m} Matches at least n, but not 10\.0\.0\.\d{1,2} returns a true

more than m, times value for an IP address between
fo

10.0.0.0 and 10.0.0.99

rr

More information may be received from a search than is expected because quantifiers are
es

"greedy." For example, when searching for a 1 followed by a 3 in the string 123, where the
al

last character is 3, using /.+3/ will return the entire string.

e
or

Backreferencing
di

Backreferencing allows an expression to reuse a matching text string in a future segment of the
s

same expression. Although multiple text strings can potentially match an expression, using a
tri

backreference enforces a character-for-character match between the previous and future segments.
bu

A backreference is made up of the following two parts:

t

• The substring to be repeated, set apart by parentheses

io

• A numeric reference to the substring, based on the order of its group compared to all other
n

parenthetical groups in the pattern

If the substring is the first group, then the backreference will be \1
The following examples show how backreferencing can be used:
• (.)\1 matches any repeated character, such as aa, OO or 22
• (\w+) (\w+) \2 \1. matches any two strings that are then repeated in reverse order, such as "one
two two one"

114 Module 4: Regular Expressions © Copyright 2016 Citrix Systems, Inc.

 Administrators should avoid expressions that contain .+ or .* because they do not limit the
length or the content of matching strings.

Lookaheads
Lookaheads are special regular expression constructs that allow a regular expression to check if a
character string matches a pattern without capturing the substring within the match.
Positive lookaheads identify whether the lookahead regex exists in the string.
Syntax: (=regex)
Negative lookaheads identify whether the lookahead regex does not exist in the string.
Syntax: (?!regex)
N

Lookaheads essentially return a "match" or "not a match". In the following examples, bold/underline
ot

is used to identify the matches within the test strings.

fo
rr

Lookahead Example 1
es

A lookahead that matches a word followed by a semicolon but does not include the semicolon in
al

the match:
e

RegEx: \w+(?=;)
or

String Result
di
s

a test; string The string is a match, but only test is included

tri

in the match result.

bu

a; second test string; The match is for a and for string.

t io

a test string Does not return a match since there is no ";"

n

Lookahead Example 2
A negative lookahead that matches any occurrence of 'foo' not followed by 'bar'.
RegEx: foo(?!bar)

© Copyright 2016 Citrix Systems, Inc. Module 4: Regular Expressions 115

String Result
foo Matches foo

afoo Matches foo

foobar Does not return a match because foo is followed

by bar

afoobar Does not return a match

a string of foo, foo, and foobars Does not match the foo in foobar

Lookahead Example 3
N
ot

A regular expression to enforce password complexity rules:

• Must start and end with a letter.
fo

• Can be between 3 and 7 characters long.

rr

• Must contain at least one number.

es

RegEx: ^[a-zA-Z](?=.*[0-9]{1})[0-9a-zA-Z]{1,5}[a-zA-Z]$
al

• ^[a-zA-Z] enforces starting with a letter.

e

• (?=.*[0-9]{1}) requires that at least one number is within the string, at any position.
or

• [0-9a-zA-Z]{1,5} enforces a valid password containing letters or numbers between the starting
and ending chars. Total string length is 3-7 characters.
di

• [a-zA-Z]$ enforces ending with a letter.

s tri
bu

String Result
t

zz Does not match

io
n

zaz Does not match

z1z Matches

z1a1z Matches

aa1aaz Matches

z1aaa1z Matches

zaaaaaz Does not match

116 Module 4: Regular Expressions © Copyright 2016 Citrix Systems, Inc.

String Result
z11111z Matches

z00000z Matches

Lookahead Complexity
Lookaheads can be complex to implement properly if you are unfamiliar with regular expressions.
Therefore, do not implement without thorough testing.

PCRE regular expressions also support Lookbehinds. The function is similar to

Lookaheads, but Lookbehinds do require that a maximum match length is specified.
Lookaheads and Lookbehinds are collectively referred to as Lookarounds. A more in depth
N

look at these topics is beyond the scope of this course. See PCRE regular expression syntax
ot

references for more details.

fo
rr

Regular Expression Scope

es

Because a single expression can match multiple strings and a string can be described by multiple
al

expressions, no "right" or "wrong" way to use PCRE exists. Many ways to describe a pattern, each of
which is correct in a particular situation, are possible. However, when using PCRE to define
e

content in the Application Firewall, administrators should double-check their expressions to make
or

sure that they are not:

• Too narrow, which limits the content that can match the expression, reducing regular
di

expressions to no more than complicated literal strings

s tri

• Too broad, which allows potentially malicious content into the site by matching too many
strings
bu

Regular expressions are very powerful. Improperly configured regular expressions can leave a
t

protected web site vulnerable or prevent the application from working as expected. Test and review
io

regular expressions thoroughly to verify proper scope.

n

© Copyright 2016 Citrix Systems, Inc. Module 4: Regular Expressions 117

Practice: Interpreting Regular Expressions

N
ot
fo
rr
es
al
e
or
di
s

Line numbers are not part of the text.

tri
bu

1. ^[A-Z]..$
t io

2. ^[A-Z][a-z ]*3[0-5]
n

There is a space after z.

3. [a-z]+[.]
4. ^.*[A-Z][a-z][a-z]$
5. [A-Za-z]+[^,][A-Za-z]+$
6. [0-9]{2}
7. (.)\1

118 Module 4: Regular Expressions © Copyright 2016 Citrix Systems, Inc.

Practice: Creating Regular Expressions
1. Write a single regular expression that matches either HTTP or HTTPS.
2. Write three strings that match [1234]{2}.
3. Write a regular expression that matches either .html or .gif.
4. What is the difference between bug+ and (bug)+?
5. Write three strings that match [A-Z]([a-z])\1[a-z].
6. Identify any security risks that may be associated with using the following regular expression to
define legitimate pages in a web application:

^http://www[.]badstore[.]net/(.*([.]html|[.]cgi|[.]css))\?[A-Za-
z0-9]*$
N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 4: Regular Expressions 119

 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

120 © Copyright 2016 Citrix Systems, Inc.

 5
Module 5

Attacks and
Protections
N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

122 © Copyright 2016 Citrix Systems, Inc.

Attacks and Protections Manual
Overview
The positive security model used by Application Firewall allows traffic known to be safe to pass and
blocks all other requests. Application Firewall also contains specific features that protect against
most common and dangerous types of web attacks.
Application Firewall validates all incoming and outgoing data from the user to the protected web
servers. All unvalidated input (PCI Section 6.5.1) is blocked from reaching the application
infrastructure.
By modifying the security checks within a profile, administrators can maximize the efficiency with
which Application Firewall protects web applications. This module identifies the types of attacks
N

and vulnerabilities that can be used against web applications and the protections against these
attacks included with Application Firewall. This module reviews the security checks which protect
ot

against the attacks and reviews the settings and configurations that can be configured within each
security check to modify the behavior or to enable relaxations for the protection.
fo
rr

Objectives
es
al

At the end of this module, you will be able to:

e

• Identify how regular expressions are used in a profile.

or

• Identify various attacks and protections.

• Configure security checks.
di

• Enable learning per security check.

s tri

• Identify administration guidelines related to configuring learning.

• Configure the generation of simple and generalized rules.
bu

• Review learned rules per security check.

t io

• Deploy, skip, and edit learned rules.

n

Security Checks
Security checks provide a protection for a type of attack, vulnerability or exploit. Requests or
responses that violate a security check are handled as specified. Security checks may be configured
with relaxations to allow a request/response that would otherwise violate the security check.
This section presents the attacks and protections available to web applications. Specifically, we focus
on the Common and HTML security checks specific to HTML applications. Once you understand
the attack/protection model and how to use and configure the security checks, applying appropriate

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 123
XML security checks should not be an issue. Refer to the Application Firewall Users Guide for
specific information involving XML profiles and security checks not covered here.

Profile Types
The profile type is specified when creating an Application Firewall profile. The profile type defines
which security checks are available in the profile. The profile type cannot be modified once it has
been created. The following table lists the available profile types.

Profile Type Description

HTML Web Application This type is used for applications that use basic
web technologies, including HTML, XHTML,
N

CSS, and HTTP.

ot

XML Web Application This type is used for applications that exchange
information using XML and support standard
fo

APIs such as SOAP, e-Business XML or Plain-

rr

Old-XML.
es

Web 2.0 Application This type is used for applications that exchange
information using XML and support Web 2.0
al

technologies, such as Asynchronous JavaScript,

e

AJAX, REST, RSS or ATOM. The Web 2.0

or

Application profile includes security checks for

the HTML and XML application profiles. If you
di

have applications that need both HTML and

XML security checks, define the profile type as a
s

Web 2.0 Application.

tri
bu

Common Security Checks

t io
n

Common security checks are relevant to all types of profiles. The following table lists the six
security checks included in all three profile types.

Check Description
Start URL The start URL check examines the URLs of
incoming requests and blocks connections to
URLs that are not listed in the Start URLs list.
This check applies to requests only.

124 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Check Description
Deny URL The deny URL check examines the URLs of
incoming requests and blocks connections to
URLs that are commonly accessed by hackers,
malicious code, or any other URLs you specify.
The deny URL check prevents attacks against
various known security weaknesses in different
web server software or on many web sites. The
deny URL check takes priority over the start
URL check, and thus denies malicious
connection attempts even when a start URL
relaxation would normally allow a request to
proceed.
N

Cookie Consistency The cookie consistency check examines cookies

ot

returned by users to see that they match cookies

your web application set for that user. If a
fo

modified cookie is found, the cookie is stripped

from the request before the request is forwarded
rr

to the web server. This check applies to requests

es

only.
al

Buffer Overflow The buffer overflow check detects attempts to

e

cause a buffer overflow on the web server. If

Application Firewall detects a URL, cookie, or
or

header longer than the specified maximum

length in a request, it blocks that request
di

because it might be an attempt to cause a buffer

s

overflow.
tri
bu

Credit Card The credit card check provides special handling

for credit card numbers. A web application does
t

not usually send a credit card number in a

io

response to a user request, even when the user

n

supplies a credit card number in the request.

The Application Firewall examines web server
responses, including headers, for credit card
numbers. If it finds a credit card number in the
response, and the administrator has not
configured it to allow credit card numbers to be
sent, it responds by either blocking the request
or replacing all but the final group of digits in
the credit card with x’s.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 125
Check Description
Safe Object The safe object check provides user-configurable
protection for sensitive business information,
such as customer numbers, order numbers, and
country- or region-specific telephone numbers
or postal codes. A user-defined regular
expression or custom plug-in tells Application
Firewall the format of this information, and
defines the rules to be used to protect it.

HTML Security Checks

HTML security checks are relevant to the Web Application (HTML) profiles and are included in
N

the Web 2.0 application profiles. The following table lists the available HTML Security Checks.
ot
fo

Check Description
rr

Form Field Consistency The form field consistency check examines the
es

web forms returned by users of the web

al

application, and verifies that the web form was

not modified inappropriately by the client. This
e

check applies only to HTML requests that

or

contain a web form, with or without data. It

does not apply to XML requests.
di

Field Formats The field formats check requires that

s tri

Application Firewall be told about the type and

length of data expected in each form field on
bu

each web form to be protected. It then examines

the data users return using web forms on the
t io

web application and verifies that the data meets

the formatting restrictions you set for that form
n

field. If any web form data does not meet the

formatting restrictions, Application Firewall
blocks the user’s request.

HTML Cross-Site Scripting The HTML cross-site scripting check provides

special defenses against cross-site scripting
attacks. Application Firewall examines user
requests for possible cross-site scripting attacks.
If it finds a possible cross-site scripting attack, it
either transforms the request to render the
attack harmless or blocks the request.

126 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Check Description
HTML SQL Injection Check The HTML SQL injection check provides
special defenses against injection of
unauthorized SQL code that might break
security. This check applies to HTML profiles
only; it is not used with XML profiles.

Form field consistency and field format security checks are only available with the HTML profiles.
The HTML checks for cross-site scripting and SQL injection are specific for HTML applications
since form fields need to be defined to configure relaxations for exempted fields.

XML Security Checks

N
ot

XML security checks apply to both of the XML-based profile types: XML web apps and Web 2.0
apps. The following table lists the available XML security checks.
fo
rr

Check Description
es

XML Format The XML format check examines the XML

al

format of incoming requests, and blocks those

e

requests that are not well formed. The purpose

of the XML Format check is to prevent a
or

malicious user from using a poorly-formed

XML request to breach security.
di
s

XML Denial of Service The XML denial of service (XML DoS) check
tri

examines incoming XML requests to determine

whether they match the characteristics of a DoS
bu

attack, and blocks those requests that do. The

t

purpose of the XML DoS check is to prevent an

io

attacker from using XML requests to launch a

n

denial-of-service attack on your server or web

service.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 127
Check Description
XML Cross-Site Scripting The XML cross-site scripting check examines
incoming requests for possible cross-site scripts
errors, and blocks those requests that contain
such errors. The purpose of the XML cross-site
scripting check is to prevent misuse of the
scripts in protected XML web services to breach
security. It does this by blocking scripts that
violate the same origin rule, which states that
scripts should not access or modify content on
any server but the server where they are located.

XML SQL Injection The XML SQL injection check examines

incoming requests for inappropriate SQL special
N

characters and keywords that might indicate an

ot

attempt to inject SQL code, and blocks those

requests. The purpose of the XML SQL injection
fo

check is to prevent an attacker from using an

XML web service to send SQL commands to a
rr

back-end SQL database server and either

es

obtaining information that they were not

entitled to obtain, or gaining control of the
al

server.
e

XML Attachment The XML attachment check examines incoming

or

requests for malicious attachments, and blocks

those requests that contain attachments that
di

might breach web services security. The purpose

s

of the XML attachment check is to prevent an

tri

attacker from using an XML attachment to

bu

breach security on your server.

t

Web Services Interoperability (WS-I) The WS-I check examines both requests and
io

responses for adherence to the WS-I standard,

n

and blocks those requests and responses that do

not adhere to this standard. The purpose of the
WS-I check is to block requests that might not
interact with other XML appropriately. An
attacker can use inconsistencies in
interoperability to launch an attack on a web
service.

128 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Check Description
XML Message Validation The XML message validation check examines
requests that contain XML messages to ensure
that they are valid. If a request contains an
invalid XML message, Application Firewall
blocks the request. The purpose of the XML
Validation check is to prevent an attacker from
using specially-constructed invalid XML
messages to breach security on a web service.

The XML checks for SQL injection and cross-site scripting are similar to the HTML version of
these security checks; however, there is no configuration for exempted fields or to inspect URL
parameters with XML. If you understand the attack and the protection provided by the HTML
N

security checks, you can manage to implement a corresponding XML security check.
ot

Request-Side and Response-Side Checks

fo
rr

Request-side security checks evaluate data passing from the client to the server. The following
es

checks involve request-side processing:

• Start URL
al

• Cookie consistency
e

• Form Field consistency

or

• Buffer overflow
di

• Field formats
s

• Deny URL
tri

• Cross-site scripting
bu

• SQL injection
t io

Response-Side Checks
n

Two security checks evaluate data passing from the server to the client, checking for sensitive data
that should not reach the outside world or at least being limited in the degree to which it is
released. The following checks involve response-side processing:
• Credit card
• Safe object

To minimize the impact on response times, enable response-side security checks only in
profiles that protect the parts of the web application that serve the sensitive content.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 129
HTTPS Web Applications
HTTPS can affect start URLs and deny URLs. Rules need to account for whether HTTP or HTTPS
are both allowed or whether rules need to be restrictive for one or the other option. When web sites
or parts of web sites need to support both HTTPS and HTTP connections, an appropriate start and
deny URLs take both into account. SSL may be supported on front-end (virtual servers) or back-
end (services) as appropriate to the application.
For example, the following regular expressions could identify parts of a web site for use with start
URLs:
• ^http://www\.example\.com/$
• ^http://www\.example\.com/shoppingcart$
• ^http://www\.example\.com/aboutus$
• ^http://www\.example\.com/images/
N

If requests to the site or parts of the site, such as the shopping cart, and requests to images within
ot

the shopping cart are made through HTTPS, then the start and deny URLs need to incorporate the
appropriate references to HTTP, HTTPS, or both within the regular expressions:
fo

• ^https?://www\.example\.com/$
rr

• ^(http|https)://www\.example\.com/shoppingcart$
es

• ^http://www\.example\.com/aboutus$
al

• ^https?://www\.example\.com/images/
e

Account for the use of HTTP or HTTPS within the start URL and deny URL regular expressions
based on what is appropriate for the application. Otherwise, the SSL offload configuration with
or

Application Firewall is the same as an SSL offload configuration with any other load balancing or
content switching virtual server. Depending on the configuration, SSL can be specified on the client
di

side (virtual server) only, server side (service) only or both. SSL can be supported on the client side
s

of the configuration, to support secure access and SSL offload by configuring a standard load
tri

balancing virtual server of protocol type SSL. SSL can be supported on the server-side of the
bu

configuration by configuring services of protocol type SSL.

For applications that require access through HTTP and HTTPS, then both HTTP and SSL virtual
t io

servers (and services) can be created.

n

130 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Buffer Overflow Exploits

N
ot
fo
rr
es
al
e
or
di

As the figure shows, buffer overflow is a common type of web application attack that allows an
s

attacker to gain unauthorized access. This type of attack overflows a memory buffer with excessive
tri

data. The buffer is a memory location allocated to contain data, such as strings or integers. The web
bu

server can behave in a manner that is useful to the attacker when a buffer overflows. Buffer
overflow attacks are the culprit in many known security issues with web server software.
t io

Buffer overflow exploits fall under PCI Section 6.5.5: Buffer Overflows.
n

Goals of a Buffer Overflow Attack

Hackers run buffer overflow attacks by sending a request to a web server that contains an extremely
long URL, cookie or other data, such as a header. The excessive data is intended to overflow the
buffer, causing the web server or operating system to fail and provide access to the hacker executing
the attack.
The goal of a hacker who is executing a buffer overflow attack includes gaining unauthorized
information, compromising a web server, or both. Another use of a buffer overflow attack is to run

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 131
arbitrary code not covered in the base security policy of the application to subvert or gain access to
another security device.

Consequences of a Buffer Overflow Attack

Buffer overflows are an extremely serious security vulnerability for software programs. Some of the
consequences and results of a successful buffer overflow attack are:
• The attacker is able to run a remote shell on the system and gain the same system privileges as
the application being attacked.
• The applications or an entire web server exhibit erratic behavior and run very slowly.
• The applications allow a hacker access to the command line.
• The server is no longer available to users or administrators.
N
ot

Buffer Overflow Protection

fo
rr
es
al
e
or
di
s tri
bu
t io
n

As the figure shows, the buffer overflow check detects attempts to cause a buffer overflow on the
web server running a protected site. If Application Firewall detects a request with a URL, cookie or

132 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
header that is longer than the specified maximum length, it blocks that request to protect
vulnerable operating systems or web server software. This security check is enabled by default.

Default Maximum Values

Application Firewall sets maximum values for URL, cookie, and header lengths as shown in the
following table.

Item Maximum Value

URL 1024 bytes

Cookie 4096 bytes

N

Header 4096 bytes

ot
fo

The default settings should meet the needs of most web applications, but each site and its users are
unique. To customize protections of specific sites, administrators may need to:
rr

• Increase the maximum lengths if legitimate content violates the buffer overflow rule. The new
es

maximum should be set just high enough to accommodate the URL, cookie, or header in
question, but not so high that it allows potential attacks through Application Firewall.
al

• Decrease the maximum lengths if an application does not have enough memory allocated to its
e

buffer to accommodate URLs, cookies or headers of the default lengths.

or
di

Modifying Buffer Overflow Settings

s tri

Administrators can customize the buffer overflow protection of a profile by modifying the
bu

maximum values for URL, cookie, and header lengths.

t io

Modifying Buffer Overflow Settings in the Configuration

n

Utility
Use the following procedure to change the maximum lengths for URLs, cookies, and headers.
1. Go to Security > Application Firewall > Profiles.
2. Double-click the profile to be modified.
3. Click the Security Checks node.
4. Double-click the Buffer Overflow security check.
5. Type an integer value between 0 and 65535 for the following three fields:
• Maximum URL Length

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 133
 • Maximum Cookie Length
• Maximum Header Length
6. Click OK, OK, Save & Close then Done to save the settings and to close any open dialog
boxes.

Modifying Buffer Overflow Settings in the Command-Line

Interface
Enter the following command to modify buffer overflow settings:

set appfw profile name -bufferOverflowMaxURLLength URL_int

-bufferOverflowMaxHeaderLength header_int
-bufferOverflowMaxCookieLength cookie_int
N
ot

The following table lists the available arguments.

fo

Argument Description
rr
es

name Identifies the profile

al

URL_int Sets the new maximum URL length

e

header_int Sets the new maximum header length

or

cookie_int Sets the new maximum cookie length

di
s

For example, enter the following command to set the maximum URL length in the pr_badstore
tri

profile to 1050:
bu

set appfw profile pr_badstore -bufferOverflowMaxURLLength 1050

t io
n

Parameter Manipulation
Parameter manipulation is the process of tampering with query parameters in web URLs that are
used by dynamic Common Gateway Interface (CGI) scripts on the back-end web servers. This type
of attack can be used by hackers to obtain unauthorized information.
Parameter manipulation falls under PCI Section 6.5.6: Injection Flaws.

134 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Parameter Manipulation Example
The following shows how a hacker uses parameter manipulation to view the new product
information of a company. The form field consistency feature of Application Firewall helps to block
this type of attack.
1. The hacker sees that the following URL provides product descriptions:
http://www.example.com/proc.cgi?file=prod.xml
2. The hacker manipulates the script parameter in an attempt to retrieve another file:
http://www.example.com/proc.cgi?file=newprod.xml
3. The hacker manipulates the script parameter in an attempt to retrieve the password file:
http://www.example.com/proc.cgi?file=../../etc/passwd
N
ot

Server Misconfiguration
fo
rr

Server misconfiguration consists of oversights in server configuration, such as allowing unrestricted

access to administrative pages, failing to disable default accounts, and exposing untested
es

functionality. An example of an oversight includes an administrator neglecting to disable a

showcode.asp file for server demonstration and the product being shipped with code that is either
al

untested or intended to be kept within the company. The code can be used by hackers to attack the
e

application and the users accessing it.

or

Only one oversight in a server configuration can produce a vulnerability that could undermine the
security of an entire infrastructure. Proper configuration of the web infrastructure is vital to
di

maintaining application security.

s

Server misconfiguration falls under PCI Section 6.5.10: Insecure Configuration Management.
tri

Rather than trusting that software developers will never make human errors, an administrator can
bu

deploy Application Firewall, which uses deny URLs, start URLs and URL closure to protect against
server misconfigurations.
t io

Organizations should also conduct regular testing for proper infrastructure configuration.
n

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 135
Deny URL Protection

N
ot
fo
rr
es
al
e
or
di

Administrators may want to block all external traffic to certain pages on a protected site. As the
s

figure shows, the deny URL check ensures that any request going through Application Firewall
tri

pointing to a URL on the deny URL list will be blocked. The only way to access these URLs is to
bu

connect directly to that server without the traffic passing through Application Firewall.
t

This security check is enabled by default.

io
n

Administrators can use Deny URLs to narrow the scope of a broadly defined start URL
regular expression. The deny URL check takes precedence over any start URL, as well as
pages accessed through URL closure.

The Deny URL List

Application Firewall comes with multiple suggested deny URLs, which are written as regular
expressions. These URLs protect against known attacks and vulnerabilities and can be enabled as an
administrator chooses, based on the particular vulnerabilities of their web applications and servers.

136 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
For example, to protect a printer on the internal network from external printer buffer overflow
attacks, an administrator can enable the IIS executable file parsing vulnerability-3 deny URL.

The figure shows the Modify Deny URL Check the dialog box in the Configuration Utility. By
N

default, all of the suggested deny URLs are disabled. For improved performance, an administrator
ot

should only enable deny URLs on an individual, as-needed basis, such as for a specific application.
fo
rr

Adding a Deny URL in the Command-Line Interface

es

Enter the following command to add a deny URL:

al
e

bind appfw profile name -denyURL "expression" -comment "string"

-state ( ENABLED | DISABLED )
or

The following table lists the available arguments.

di
s tri

Argument Description
bu

name Identifies the profile

t io

expression Identifies the URL as either a literal string or a

n

regular expression

string Defines an optional comment about the Deny

URL

For example, enter the following command to add a Deny URL to the pr_badstore profile that
protects the administrator’s page at www.example.com/admin.html:

bind appfw profile pr_badstore -

denyURL "www.example.com/admin.html"
-state ENABLED

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 137
Deleting a Deny URL in the Command-Line Interface
Enter the following command to delete a deny URL:

unbind appfw profile name -denyURL "expression"

Arguments include:

Argument Description
name Identifies the name of the profile

expression Identifies the URL as either a literal string or a

regular expression
N
ot

For example, to delete the Deny URL protecting the administrator’s page at
www.example.com/admin.html from the pr_badstore profile, type the following command in the
fo

command-line interface:
rr

unbind appfw profile pr_badstore -

es

denyURL "www.example.com/admin.html"
al
e
or
di
s tri
bu
t io
n

138 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
SQL Injection

N
ot
fo
rr
es
al
e
or
di

As the figure shows, SQL injection attacks are made against web applications in an attempt to
s

create, read, update or delete any data available to the application. SQL injection is most commonly
tri

performed on forms. With the ability to alter data, the hacker can access unauthorized information,
bu

such as bank or credit card accounts or medical records. A successful SQL injection attack is
capable of completely compromising a web application.
t io

Command and SQL injection falls under PCI Section 6.5.6: Injection Flaws.
n

How SQL Injection Works

Databases are frequently used in conjunction with web applications. Many applications run queries
against relational databases using SQL commands to collect data from a user to add to a database.
The database structure and the data it contains can be exposed through an unprotected web
application.
A SQL injection attack is an attack that tries to run unauthorized SQL code and queries against a
database. The attacker inserts (or injects) code through the web application which runs against the
database. For example, an attacker submits SQL queries through form fields. If the application does

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 139
not properly validate or clean field values for special characters or SQL reserve words, then the
query commands embedded in the field values are run against the database. The attacker uses SQL
commands against the application and gets additional commands to run other than the intended
queries.
In other cases, the web application may not properly handle errors from the database server from
malformed queries, and the web application may display the results of the error message in place of
the expected query results. This scenario provides an attacker with additional information about the
database platform, known vulnerabilities, or details about the database schema and architecture.
Any of these can be used by the attacker to refine the attacks against the database.

HTML SQL Injection Protection

N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

As the figure shows, the HTML SQL injection security check protects against SQL injection attacks.
It does this by checking web forms and URLs for a combination of SQL characters and keywords.
Violations can be blocked or transformed (negated).
The security check is enabled (with blocking) by default.
Default Basic/Advanced Profile settings are:

140 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
• BLOCK is enabled.
• Learning is disabled by default with the Basic profile settings but is available. Learning is
enabled by default with the Advanced profile settings.
• Additional actions and parameters can be adjusted.
The HTML SQL injection protection is configured by specifying the actions and fields that are
checked. The security check may be further modified by creating relaxations, defining fields that
should be exempt from the SQL injection security check if the protection is blocking legitimate and
allowed data.
Only disable this security check if your web application has no access to back-end SQL databases.

SQL Keywords and Special Characters

Application Firewall recognizes certain SQL keywords, such as SELECT and OR, which can signal a
N

SQL injection attack. Keywords are recognized as a threat only if they are not adjacent to an
ot

alphanumeric character. For example:

fo

• SELECT is recognized as a SQL keyword.

• SELECTED and SELECT1 are not recognized as SQL keywords.
rr

Administrators have the option to ignore keywords unless they appear in a field that also includes a
es

special character. If the Restrict checks to fields containing SQL special characters option is
al

checked, the following actions will occur:

e

• SELECT ‘ is recognized as a SQL keyword.

or

• SELECT is not recognized as a SQL keyword.

di

Modifying SQL Injection Action Settings

s tri

The SQL injection security check contains the following unique action settings that administrators
bu

may choose to enable within a profile:

t

• Transform SQL special characters

io

• Restrict checks to fields containing SQL special characters

n

SQL relaxations that are defined in SQL relaxations will not be subject to the
transformations listed above.

SQL comments handling allow an administrator to observe the response of the web server to profile
the behavior of an SQL server and to identify which SQL software it uses. With relaxations,
administrators can instruct Application Firewall to ignore a particular field.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 141
Modifying SQL Injection Action Settings in the
Configuration Utility
Use the following procedure to enable or disable additional action settings for the SQL injection
check of a profile:
1. Go to Security > Application Firewall > Profiles.
2. Select the profile to be modified and click Edit.
3. Click the Security Checks node.
4. Select the HTML SQL Injection security check and click Action Settings.
5. Determine whether to transform SQL special characters.
• Select Transform SQL special characters if the SQL special characters should be
transformed so that they are harmless.
N

• Deselect Transform SQL special characters if the SQL special characters should not be
transformed.
ot

6. Determine whether to check keywords independently.

fo

• Select SQL Special Character from the Check Request Containing drop down menu if
rr

the SQL keywords should trigger a violation only if they coincide with a SQL special
character.
es

•
al

Deselecting Restrict checks to fields containing SQL special characters is not

recommended for profiles that protect a variety of content because it can lead to
e

multiple false, positive violations.

or

7. Click OK to save the current settings.

di

8. Click OK to acknowledge that the Security Checks was successfully modified.

s

9. Click Save & Close, then click Done to close the Profile Configuration window.
tri
bu

Modifying SQL Injection Action Settings in the Command-

t io

Line Interface
n

Enter the following command to modify SQL injection action settings:

set appfw profile name -

SQLInjectionTransformSpecialChars ( ON | OFF )

-SQLInjectionOnlyCheckFieldsWithSQLChars ( ON | OFF )

The name argument identifies the profile.

142 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
For example, enter the following command to restrict the SQL injection check to fields that contain
SQL characters in the pr_badstore profile:

set appfw profile pr_badstore -

SQLInjectionOnlyCheckFieldsWithSQLChars ON

SQL Comments Handling

The SQL Comments Handling field tells Application Firewall how to handle SQL comments in the
web pages that are checked. By default, Application Firewall treats all SQL comments like any type
of content and checks entire requests for SQL special characters and keywords.
Because SQL servers ignore text that is recognized as part of a comment, an administrator can
prevent false positives by configuring Application Firewall to recognize SQL comments and to skip
N

them when checking a request for SQL injection check violations. Administrators can also observe
the response of the web server in order to profile the behavior of the SQL server.
ot

The following table lists the available SQL Comments Handling field settings.
fo
rr

Setting Description
es

ANSI The Application Firewall recognizes SQL ANSI

al

comments, which are normally used by Unix-

e

based SQL databases, and skips those comments

when filtering requests for violations of the SQL
or

injection check.
di

Nested The Application Firewall recognizes nested SQL

s

comments, which are normally used by

tri

Microsoft SQL Server, and skips those

bu

comments when filtering requests for violations

of the SQL injection check.
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 143
Setting Description
ANSI/Nested The Application Firewall recognizes comments
that adhere to both the ANSI and nested SQL
comment standards and skips those comments
when filtering requests for violations of the SQL
injection check. Comments that match only the
ANSI standard or only the nested standard will
not be skipped.

An administrator should choose either the

Nested or the ANSI/Nested option only if
the back-end database runs on Microsoft
SQL Server because most other types of
SQL databases do not recognize nested
N

comments.
ot

Check all Comments Application Firewall checks the entire request

fo

for violations of the SQL injection check and

rr

skips nothing.
es

Relaxations
al
e

After the SQL injection security check has been enabled, all fields and URLs within a protected site
or

will be examined based on the action settings defined by the administrator. In some situations,
however, a field that does not interact with a SQL database may require a SQL keyword or special
di

character to perform a legitimate function. For example, an online banking logon page may require
its customers to choose their account type from a drop-down menu, which includes "President’s
s tri

Select Checking" as an option. The apostrophe and use of the keyword "Select" would normally
block this option.
bu

By setting a relaxation, administrators can instruct the Application Firewall to ignore a particular
t

field, rather than allow it to continually violate the SQL injection rule.
io

A relaxation is created by specifying the field name and the URL where the field occurs. Field
n

names and URLs can be specified using regular expressions to identify multiple fields or URLs in a
single relaxation. However, when using regular expressions, an administrator should be careful and
ensure that only the desired fields are being identified and that the relaxation is not being applied
too broadly.

For additional protection, administrators may set field format checks on fields that are
exempted from the SQL injection check.

144 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Adding a SQL Injection Relaxation in the Command-Line
Interface
Enter the following command to add SQL injection relaxation:

bind appfw profile name -SQLInjection "field_name" "action_URL"

-isRegex ( REGEX | NOTREGEX ) -comment string -
state ( ENABLED | DISABLED )

The following table lists the available arguments include:

Argument Description
name Identifies the profile being modified
N
ot

field_name Identifies the field in the relaxation being added

fo

action_URL Identifies the action URL associated with the

rr

field being relaxed

es

string Defines an optional comment about the

relaxation
al
e

For example, enter the following command to add a relaxation in the pr_badstore profile for the
or

field_ex field, which has an action URL of www.example.com/example.cgi:

di

bind appfw profile pr_badstore -SQLInjection "field_ex"

s

"www.example.com/example.cgi" -isRegex NOTREGEX -state ENABLED

tri
bu

Deleting a SQL Injection Relaxation Using the Command-

t io

Line Interface
n

Enter the following command to delete a SQL injection relaxation:

unbind appfw profile name -SQLInjection "field_name" "action_URL"

The following table lists the available arguments.

Argument Description
name Identifies the profile being modified

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 145
Argument Description
field_name Identifies the field in the relaxation being
deleted

action_URL Identifies the action URL associated with the

field being relaxed

For example, enter the following command to delete the SQL injection relaxation in the
pr_badstore profile for the field_ex field, which has an action URL of
www.example.com/example.cgi:

unbind appfw profile pr_badstore -SQLInjection "field_ex"

"www.example.com/example.cgi"
N
ot

XML SQL Injection Security Check

fo
rr

XML-based SQL injection attacks are similar to HTML SQL injection attacks. With XML-based
SQL injection attacks, unauthorized SQL code is submitted through XML requests instead of
es

through HTML form fields.

al

The XML SQL injection security check supports the block, log, and statistics options. Learning and
e

transform options are not available. By default, the XML SQL injection security check is disabled in
both basic and advanced profile settings. The XML version of the security check supports the same
or

settings to restrict checks to fields containing SQL special characters and to configure SQL
comment handling. No relaxations on the security check can be configured.
di
s tri
bu
t io
n

146 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Cross-Site Scripting

N
ot
fo
rr
es
al
e
or
di

As the figure shows, hackers who run cross-site scripting attacks have the goal of obtaining
s

unauthorized information, compromising a web server, or both. Cross-site scripting attacks can
tri

result in identity theft of unsuspecting application users.

bu

XSS falls under PCI Section 6.5.4: Cross-site Scripting (XSS) Attacks.
t io
n

Attacking the Trust Relationship

Cross-site scripting takes advantage of the trust relationship between users and the web application
they are using. Cross-site scripting is commonly performed on forms, search engine boxes, online
forums, and public blogs, but can also occur in URLs. An innocent user working with a trusted web
application can unknowingly download malicious code to an application because a hacker has
successfully run a cross-site scripting attack. After the malicious code has been transferred, the user
may, also unknowingly, grant that code the same privileges normally given to the web application.
Without having any malicious intentions, the user actually takes part in the attack, violating the
trust of both the user and the provider of the application.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 147
How Cross-Site Scripting Attacks Work
Cross-site scripting attacks are carried out using a script that bypasses the same origin policy of a
web application, a security measure that prohibits one web site from obtaining or setting properties
for any content on another web site. Scripts on a web site can access data and modify content on
the site, which is what hackers use cross-site scripting to accomplish. JavaScript is used most often,
but any scripting language supported by the victim’s browser can be used.
1. A typical cross-site scripting attack begins with a user viewing a web application that allows
posting of user information.
2. Rather than posting a description of an item or the information that the user requests, the
hacker will post an executable script on the browser.
3. The script may appear in the form of a fake logon screen, a dialog box or a command to
transfer the user’s information to the hacker’s location. The hacker’s goal is to steal the user’s
credentials by forcing them to log in.
N

4. If the user logs in, the hacker gains complete access to their account.
ot
fo

Results of a Cross-Site Scripting Attack

rr

Cross-site scripting attacks can be extremely damaging to web applications and the users that access
es

them. The result of a successful cross-site scripting attack provides a hacker with the ability to:
al

• Change user settings.

e

• Hijack accounts and steal user identities.

or

• Access restricted sites.

• Launch false advertisements.
di

• Deface web sites.

s

• Insert hostile content.

tri
bu

Preventing Cross-Site Scripting Attacks

t io

Cross-site scripting flaws occur when a web application accepts data from a user and sends it to the
n

web server or browser without first encoding or checking to see if the content is valid. OWASP
recommends a strategy of rejecting all invalid input rather than attempting to filter potentially
hostile data.
The positive security model used by Application Firewall blocks all content not known to be valid.
Application Firewall also contains a cross-site scripting security check, which examines requests for
scripts and either renders them harmless before forwarding the valid data to the correct location or
blocks the connection.

148 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
HTML Cross-Site Scripting Protection

N
ot
fo
rr
es
al
e
or
di

This security check is enabled in both the basic and advanced defaults. Application Firewall protects
s

against cross-site scripting attacks by examining requests for scripts that could be used to
tri

compromise the trust relationship between a protected site and other users. When the Firewall
bu

finds such a script, it either transforms the script before forwarding it to its destination or blocks
the connection.
t io
n

Cross-Site Scripting Action Settings

The HTML cross-site scripting security checks form fields and URLs for the presence of scripts that
violate the security check. If a violation is found, the request is blocked or transformed based on
the action settings.

Transform Cross-Site Scripts

Enabling the Transform action causes Application Firewall to modify any cross-site scripts it detects
in a request to render it harmless before forwarding the request to the web server.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 149
The Transform action works by performing the following transformations:
• Left angle brackets (<) are transformed to the HTML character entity equivalent (&lt;).
• Right angle brackets (>) are transformed to the HTML character entity equivalent (&gt;).
This transformation prevents a browser from interpreting the HTML tag and from trying to run
the tag. An embedded <script> tag or other HTML tag is rendered useless as &lt;script&gt;.

Check Complete URLs for Cross-Site Scripting

By default, the Check complete URLs for cross-site scripting setting is disabled in both basic and
advanced profiles. The HTML cross-site scripting security check inspects the query portion of the
URL for violations. When this setting is enabled, the security check checks the entire URL string
and not just the query portion.
N

Additional Action Settings

ot
fo

In addition to the standard action settings found in every security check, the cross-site scripting
check offers two optional features for administrators to customize a profile:
rr

• Transform cross-site scripts modifies potentially harmful scripts before allowing them to
es

continue to their destinations (for example, < is changed to &lt;). Transforming the scripts
al

does not affect other action settings; block, log and stat still apply if enabled.
e

• Check complete URLs for cross-site scripting causes Application Firewall to check URLs in
addition to form fields for cross-site scripts. When this feature is disabled, only fields are
or

examined.
di
s

Relaxations
tri
bu

If a protected site includes form fields where legitimate scripts or data could be flagged as cross-site
scripting, relaxations can be set by designating field names and form URLs that should not be
t

blocked or transformed.
io

Cross-site scripting relaxations can be disabled without removing them from the list. Disabling a
n

relaxation allows an administrator to later re-enable a relaxation without having to redefine it.

For additional protection, administrators may set Field Format checks on fields that are
exempted from the cross-site scripting security check.

Modifying Cross-Site Scripting Action Settings

The cross-site scripting security check contains the following unique action settings that
administrators may choose to enable within a profile:

150 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
• Transform cross-site scripts.
• Check complete URLs for cross-site scripting.

Fields that are defined in Cross-Site Scripting relaxations will not be subject to the action
settings listed above.

Modifying Additional Action Settings Procedure

Use the following procedure to enable or disable additional action settings for the cross-site
scripting security check in a profile:
1. Go to Security > Application Firewall > Profiles.
2. Select the profile to be modified and click Edit.
N

3. Click the Security Checks tab.

ot

4. Select the HTML Cross-Site Scripting security check and click Action Settings.
fo

5. Determine whether to transform cross-site scripts.

rr

• Select Transform cross-site scripts if violating scripts should be transformed before being
passed along to the protected site.
es

• Deselect Transform cross-site scripts if violating scripts should not be transformed.

al

6. Determine whether to check complete URLs.

e

• Select Check complete URLs for cross-site scripting if URLs should be checked for cross-
site scripts along with form fields.
or

• Deselect Check complete URLs for cross-site scripting if only form fields should be
di

checked.
s

7. Click OK to save the current settings.

tri

8. Click OK to acknowledge that the security check was successfully modified.

bu

9. Click Save & Close, then click Done to close the Profile Configuration window.
t io

Modifying Cross-Site Scripting Action Settings Using the

n

Command-Line Interface
Enter the following command to modify cross-site scripting action settings:

set appfw profile name -

crossSiteScriptingTransformUnsafeHTML ( ON | OFF ) -
crossSiteScriptingCheckCompleteURLs ( ON | OFF )

The name argument identifies the profile.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 151
For example, enter the following command to enable both cross-site scripting action settings in the
pr_badstore profile:

set appfw profile pr_badstore -

crossSiteScriptingTransformUnsafeHTML ON -
crossSiteScriptingCheckCompleteURLs ON

Adding a Cross-Site Scripting Relaxation Using the

Command-Line Interface
Enter the following command to add a cross-site scripting relaxation:

bind appfw profile name -

N

crossSiteScripting "field_name" "action_URL" -

ot

isRegex ( REGEX | NOTREGEX ) -comment string -

state ( ENABLED | DISABLED )
fo

The following table lists the available arguments.

rr
es

Argument Description
al
e

name Identifies the profile

or

field_name Identifies the field in the relaxation being added

di

action_URL Identifies the action URL associated with the

s

field
tri

string Defines an optional comment about the

bu

relaxation
t io

For example, enter the following command to add a relaxation in the pr_badstore profile for the
n

field_ex field with an action URL of www.example.com/example.cgi:

bind appfw profile pr_badstore -crossSiteScripting field_ex

"www.example.com/example.cgi" -isRegex NOTREGEX -state ENABLED

Deleting a Cross-Site Scripting Relaxation Using the

Command-Line Interface
Enter the following command to delete a cross-site scripting relaxation:

152 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
 unbind appfw profile name -
crossSiteScripting "field_name" "action_URL"

The following table lists the available arguments.

Argument Description
name Identifies the profile being modified

field_name Identifies the field in the relaxation being

deleted

action_URL Identifies the action URL associated with the

field being relaxed
N
ot

For example, enter the following command to delete the relaxation that was set in the pr_badstore
profile for the field_ex field with an action URL of www.example.com/example.cgi:
fo
rr

unbind appfw profile pr_badstore -

es

crossSiteScripting "field_ex" "www.example.com/example.cgi"

al
e

XML Cross-Site Scripting Security Check

or

XML-based cross-site scripting attacks are similar to HTML cross-site scripting attacks. With XML
cross-site scripting attacks, unauthorized code is submitted through XML documents instead of
di

through HTML form fields.

s tri

The XML cross-site scripting security check supports the block, log, and statistics option. Learning
and transform are not available. By default, this security check is disabled in both the basic and
bu

advanced profile settings. No relaxations on the security check can be configured.

t io
n

Command Injection
Command injection involves inserting system commands that the server inadvertently runs.
Injection of commands into a URL are blocked by Application Firewall by using deny URLs or
URL closure. Injection of commands into form fields are blocked by enforcing input validation
using field format validation.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 153
Command Injection Examples

N
ot
fo
rr
es
al
e
or
di

The embedded commands can trick the web application to shell out to the operating system and
s

run the commands. Typical command injection attacks include attempting to trick the web
tri

application to display file directories and information in files, such as a password file, delete data,
bu

or injecting Trojan programs and scripts.

t

Many of the URL vulnerabilities to command injection attacks can be blocked or prevented using
io

deny URLs.
n

Vulnerable fields can be protected using field formats to define expressions that restrict the types of
allowed and disallowed content which can be entered in the field.

Field Format Protection

The field formats security check protects against many potential attacks, including cross-site
scripting, SQL injection, and buffer overflow. If a request returns data through a web form field
that does not match the pre-specified format or length for that field, the request will be blocked.
This security check is enabled by default.

154 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
 Because violations that are blocked by the field formats security check will redirect the
user to the error page with no additional feedback, this check is not a substitute for input
validation in the protected application.

Confidential fields offer additional protection by allowing an administrator to designate web form
fields as confidential. The information that a user types into a confidential field is displayed by a
masking character when logged by Application Firewall.

Field Types and Field Formats

N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

For example, if a registration page contains a field for new users to type their name, which includes
only alphabetic characters, an administrator may set a field format for that field using the
predefined field type alpha.

Predefined Field Types

Application Firewall comes with five predefined field types, which are listed in the following table,
along with their regular expressions. Predefined field types cannot be deleted or modified.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 155
Field Type Regular Expression
integer ^[+-]?[0-9]+$

alpha ^[a-zA-Z]+$

alphanum ^[a-zA-Z0-9]+$

nohtml ^[^&<>]*$

any ^.*$

Setting a field format to "any" may leave fields vulnerable to a variety of attacks, which is
the same as turning off the field formats check for the field in question. Therefore, it
N

should be used with extreme caution.

ot
fo

Custom Field Types

rr

Administrators may define custom field types. Each new type requires a regular expression and a
es

priority setting that the learning engine can use when evaluating strings that match more than one
field type definition.
al

For example, the predefined field types integer and alphanum will both match strings that contain
e

only numbers. However, integer has a higher priority setting, so the learning engine checks for
or

matches on this field type first. Otherwise, strings would match alphanum and never have the
opportunity to match integer.
di

When administrators create their own field types, they must keep potential overlaps like these in
s

mind. Priorities should be set so that the most specific field types have the highest priority or the
tri

lowest number.
bu
t

Field Format Configuration

io
n

The field formats check requires configuration after it is enabled; otherwise, the check remains
dormant. Before Application Firewall can check for any formats, individual fields must be identified
and their respective formats defined. Identifying a field requires the administrator to know the
name of the field in question and the action URL that specifies where the form data is sent for
processing.
Administrators may choose to set a minimum and maximum number of characters in a string that
matches the specified field type. The default settings are 0 and 65535, respectively.

156 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Default Field Format
An administrator can also set a global, default field format, which applies to all fields protected by
the profile in question. Doing so requires all fields to return data in the specified format unless
otherwise specified as an individual definition. For example, if the default field format is set to
alphanum, but FieldA has been individually set for integer, then FieldA must return data in the
integer format and all other fields must return data in the alphanum format.
An administrator should be careful when defining a default field format because it could require
specifying a field type for each explicit field and form protected by the profile that does not meet
the default field format. An administrator should not set "any" as the workaround default field
format because the security protection for any fields that are not explicitly defined are disabled.

Confidential Fields
N

A confidential field is a web form field that accepts private information, that could be used in
ot

identity theft or other types of fraud. The Application Firewall confidential fields feature allows an
administrator to designate web form fields as confidential. Confidential fields mask the values in
fo

logs, and data is logged on nslog.

rr

Common types of information that an administrator should consider protecting with a confidential
es

field designation include:

• Passwords
al

• Credit card numbers, validation codes, and expiration dates

e

• Social security numbers

or

• Home addresses and telephone numbers

di
s

Adding a Custom Field Type

tri
bu

Administrators can define custom field types that will be available to all profiles within Application
Firewall. The new field type will be listed as an option in all field format definitions.
t io
n

Adding a Custom Field Type Procedure

Use the following procedure to add a custom field type to Application Firewall.
1. Go to Security > Application Firewall > Manage Field Types.
2. Click Add.
3. Type a name for the new field type in the Name field.
4. Type a regular expression that describes the new format in the Regular Expression field.
5. Enter a positive integer value in the Priority field.
6. (Optional) Enter a comment about the field type in the Comments field for future reference.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 157
7. Click Create.
8. Click Close to close the Configure Application Firewall Field Type window.

Adding a Custom Field Type Using the Command-Line

Interface
Enter the following command to add a field type to the Application Firewall:

add appfw fieldType name "expression" priority -comment string

Arguments include:

Argument Description
N
ot

name Identifies the new field type for future use

fo

expression Defines the new field type in the form of a

rr

regular expression
es

priority Sets the priority of the field type as a positive

integer, with lower integers meaning higher
al

priority
e
or

string Defines an optional comment about the Deny

URL
di
s

For example, enter the following command to add a field type named TwoDigitNumbers, described
tri

by the expression ^[0-9]{2}$ with a priority of 35:

bu

add appfw fieldType TwoDigitNumbers "^[0-9]{2}$" 35

t io
n

Setting a Default Field Type

Administrators can choose to set a field type as the default for all fields protected by a profile. After
a default field type is defined, data returned through all fields must be of that type, unless a specific
field format has been set.

Setting a Default Field Type Procedure

Use the following procedure to set a default field type for all fields protected by a profile.

158 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
1. Go to Security > Application Firewall > Profiles.
2. Select the profile to be modified and click Edit.
3. Click the Security Checks node.
4. Highlight Field Formats and click Action Settings.
5. Determine the data format in the Field Type drop-down menu under Default Field Format.
• Select integer if the data returned in this field should be any string of numbers.
• Select alpha if the data returned in this field should be any string of letters.
• Select alphanum if the data returned in this field should be any string of numbers or
letters.
• Select nohtml if the data returned in the this field should not contain HTML.
• Select the name of the custom field type If the data returned in this field should be a
custom field type defined by an administrator.
N

• Select any if the data returned in this field should be anything at all.
ot

6. Type a new integer value between 0 and 65535 in the Minimum Length field (optional).
7. Type a new integer value between 0 and 65535 in the Maximum Length field (optional).
fo

8. Click OK, OK, Save & Close then Done to close the Profile Configuration window.
rr
es

Setting a Default Field Type Using the Command-Line

al

Interface
e
or

Enter the following command to set a default field type:

di

set appfw profile name -defaultFieldFormatType type [-

defaultFieldFormatMinLength int_min_length] [-
s tri

defaultFieldFormatMaxLength int_max_length]
bu

The following table lists the available arguments.

t io

Argument Description
n

name Identifies the profile

type Identifies the field type being set as the default

int_min_length Defines the minimum length of this field type

int_max_length Defines the maximum length of this field type

For example, enter the following command to set the default field type of all field formats in the
pr_badstore profile to alpha with a maximum length of 20:

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 159
 set appfw profile pr_badstore -defaultFieldFormatType alpha
-defaultFieldFormatMaxLength 20

Modifying Field Format Settings

To specify a field type for a single field (or group of fields, if defined by a regular expression),
administrators need to add a field format to the profile protecting that field. A field format can be
modified after creation.

Modifying Field Format Settings Procedure

Use the following procedure to add a field format:
N

1. Go to Security > Application Firewall > Profiles.

ot

2. Select the profile to be modified and click Edit.

fo

3. Click the Relaxation Rules node.

rr

4. Select the Field Formats security check and click Edit.

5. Determine whether to add a new field format or modify an existing field format.
es

• To create a field format, click Add.

al

• To modify a field format, select the setting from the list and click Modify.
e

6. Determine whether to enable the field format.

or

• If the field format should be an active part of the security check settings, then select
Enabled.
di

• If the field format should be saved for later, deselect Enabled.

s tri

7. Determine whether to declare the field name as a regular expression.

• Select Is Form Field Name Regex if the field name will be written as a regular expression.
bu

• Deselect Is Form Field Name Regex if the field name will be written as a literal string.
t io

8. Type a literal string or PCRE-format regular expression in the Field Name field.
n

9. Type the action URL of the web form that contains the field being defined.
10. Determine the data format to specify in the Field Type/CharMap Regex drop-down menu.
• Select integer if the data returned in this field should be any set of numbers.
• Select alpha if the data returned in this field should be any string of letters.
• Select alphanum if the data returned in this field should be any string of numbers or
letters.
• Select nohtml if the data returned in the this field should not contain HTML.
• Select the name of the custom field type If the data returned in this field should be a
custom field type defined by an administrator.
• Select any if the data returned in this field should be anything at all.

160 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
11. Type a new integer value between 0 and 65535 in the Minimum Length field (optional).
12. Type a new integer value between 0 and 65535 in the Maximum Length field (optional).
13. Type a comment about the field type in the Comments field for future reference (optional).
14. Determine whether this is a new field format.
• If the field format is new, then click Create.
• If the field format is being modified, then click OK.
15. Click Close to acknowledge that the field format was added or modified successfully.
16. Click OK to acknowledge that the field formats security check was successfully modified.
17. Click Done to close the Profile Configuration window.

Adding a Field Format Using the Command-Line Interface

N

Enter the following command to add a field format:

ot

bind appfw profile name -

fo

fieldFormat "field_name" "action_URL" field_type [-

rr

fieldFormatMinLength int_min_length] [-
fieldFormatMaxLength int_max_length] -
es

isRegex ( REGEX | NOTREGEX ) -comment "string" -

state ( ENABLED | DISABLED )
al
e

The following table lists the available arguments.

or

Argument Description
di
s

name Identifies the profile

tri
bu

field_name Identifies the field in the new field format

t

action_URL Identifies the action URL associated with the

io

field
n

field_type Sets the predefined field type being assigned to

the field

int_min_length Sets a positive integer as the minimum length of

the field format

int_max_length Sets a positive integer as the maximum length

of the field format

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 161
Argument Description
string Defines an optional comment about the field
format

For example, enter the following command to set a field format in the pr_badstore profile for the
field_ex field, which has an action URL of www.example.com/example.cgi requiring all data to be of
the type integer with a minimum length of 3:

bind appfw profile "field_ex" "www.example.com/example.cgi" -

fieldFormatMinLength 3 -isRegEX NOTREGEX -state ENABLED

Deleting a Field Format Using the Command-Line Interface

N
ot

Enter the following command to delete a field format:

fo

unbind appfw profile name -fieldFormat "field_name" "action_URL"

rr

The following table lists the available arguments.

es
al

Argument Description
e
or

name Identifies the profile

di

field_name Identifies the field in the field format being

deleted
s tri

action_URL Identifies the action URL associated with the

bu

field
t io

For example, enter the following command to delete the field format set for the field_ex field,
n

which has an action URL of www.example.com/example.cgi in the pr_badstore profile:

unbind appfw profile pr_badstore -

fieldFormat "field_ex" "www.example.com/example.cgi"

Adding a Confidential Field

Confidential fields can be added in the Configuration Utility and in the command-line interface

162 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Adding a Confidential Field Procedure
Use the following procedure to add a confidential field:
1. Go to Security > Application Firewall > Profiles.
2. Select the profile to be modified and click Edit.
3. Click the Relaxation Rules node.
4. Highlight Form Field Consistency then click Edit.
5. Click Add.
6. Determine whether to enable the confidential immediately.
• Select Enabled to enable the confidential field immediately.
• Deselect Enabled to enable the confidential field later.
7. Check the Is Form Field Name Regex box to use a regular expression as the name field.
N

8. Type the name of the form field in the Form Field Name box.
ot

9. Type the action URL in the Action URL box.

10. Type an explanation of why this form field has been designated as confidential.
fo

11. Click Create.

rr

12. Click Close.

es

13. Click Done.

al
e

Adding a Confidential Field Using the Command-Line

or

Interface
di

Enter the following command to set a confidential field:

s tri

add appfw confidField name "action_URL" -

bu

isRegex ( REGEX | NOTREGEX ) -comment "string" -

state ( ENABLED | DISABLED )
t io

The following table lists the available arguments.

n

Argument Description
name Identifies the profile

action_URL Identifies the action URL associated with the

confidential field

string Defines an optional comment about the

confidential field

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 163
Modifying a Confidential Field
Confidential fields can be modified in the Configuration Utility and in the command-line interface.

Modifying a Confidential Field Procedure

Use the following procedure to modify a confidential field:
1. Go to Security > Application Firewall > Profiles.
2. Select the profile to be modified and click Edit.
3. Click the Relaxation Rules node.
4. Highlight Form Field Consistency then click Edit.
5. Highlight the Form Field Consistency Relaxation Rule to be modified and click Edit.
N

6. Determine whether to enable the confidential field.

ot

• Select Enabled if the confidential field should be enabled.

• Deselect Enabled if the confidential field should be disabled temporarily.
fo

7. Check the Is Form Field Name Regex box to use a regular expression as the name field.
rr

8. Type the name of the form field in the Form Field Name field
es

9. Type the action URL in the Action URL field.

al

10. Type an explanation of why this form field has been designated as confidential (optional).
e

11. Click OK.

or

12. Click Close.

13. Click Done.
di
s tri

Modifying a Form Field Using the Command-Line Interface

bu

Enter the following command to modify a confidential field:

t io

set appfw confidField name "action_URL" -

n

isRegex ( REGEX | NOTREGEX ) -comment "string" -

state ( ENABLED | DISABLED )

The following table lists the available arguments:

Argument Description
name Identifies the profile

164 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Argument Description
action_URL Identifies the action URL associated with the
confidential field

string Defines an optional comment about the

confidential field

Cookie Tampering and Poisoning

N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

Cookie tampering and poisoning falls under PCI Section 6.5.3: Broken Authentication and Session
Management.

Types of Cookies
There are two types of cookies: session (or transient) cookies and persistent (or stored or
permanent) cookies.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 165
Session cookies are stored in temporary memory within the web browser and are destroyed after
the browser is closed; they are not retained. Session cookies are typically used to track session
information for navigation.
Persistent cookies are stored as a text file by the browser on the client system. Persistent cookies
stay on the client system after the session ends and the browser is closed.
Persistent cookies may be used to store information regarding user preferences or selections for
application purposes. For example, a persistent cookie could allow a site to remember the name and
profile of a user and build recommendations for that user.
Both types of cookies are captured and encoded by Application Firewall. Persistent cookies are
indicated by the letters "wlf" or will live forever. Transient cookies are indicated by the letters "wat"
or will act transiently.

Existing persistent cookies stored in client browsers will be stripped the first time they are
seen by Application Firewall because they will not have been signed. Cookie consistency
N

blocking, as well as learning, should be deselected after Application Firewall has been
ot

deployed to avoid this situation. After sufficient traffic has passed through Application
Firewall, a relaxation for any persistent cookies should be deployed, provided that doing so
fo

will not introduce a security risk.

rr
es

How Cookies Are Added

al

The following table describes how cookies are added.

e
or

Before Application Firewall After Application Firewall

di

0 cookies 1 citrix_ns_id session cookie

s tri

1 persistent cookie
1 application persistent cookie
bu

1 citrix_ns_id cookie
t io

1 citrix_ns_id_ wlf cookie

n

2 persistent cookies
2 application persistent cookies
1 citrix_ns_id cookie
1 citrix_ns_id_ wlf cookie

166 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Before Application Firewall After Application Firewall

1 persistent cookie 2 original application cookies

1 session cookie 1 citrix_ns_id cookie
1 citrix_ns_id_ wat cookie
1 citrix_ns_id_ wlf cookie

Web Server Sessions

The client and web server must constantly pass session state information between themselves to
verify the user context because the HTTP used by web applications is stateless. Along with the URL
N

and Body, the Header portion of HTTP requests contain the cookie with the session state
information. The web browser also stores the information in a temporary memory.
ot

In an Application Firewall environment, Application Firewall adds its own session cookie, keeping
fo

track of users and their corresponding sessions.

rr
es
al
e
or
di
s tri
bu
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 167
Cookie Consistency Protection

N
ot
fo
rr
es
al
e
or
di

Application Firewall takes several different actions to ensure cookie consistency and to prevent
s tri

cookie tampering and poisoning:

bu

• Application Firewall verifies that cookies have not been added or modified when a web server
sends the client a cookie, and the client returns the cookie to the server. If the Application
t

Firewall detects an added or modified cookie, the cookie is stripped before the request is sent
io

to the web server.

n

• Application Firewall does not modify the cookie after it receives it from the client. Rather, it
applies a signature to each cookie it reads. Then it adds its own cookie to validate that the
session was not tampered and that the cookies were not altered by the client. The following
cookies are added as needed:
• Application Firewall session cookies
• Tracking cookies for permanent or transient application cookies

168 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Sessionization and Cookies
Web applications are stateless, and many applications do not maintain state across request and
response pairs. The client and the server must constantly pass session state information to
remember the nature of the connection. Session state information passes in one of three ways:
• URL: Through session IDs in the URL
• Header: Through cookies containing session state information
• Body: Through When hidden fields and the Viewstate parameters in Microsoft applications
Sessionization as provided in Application Firewall tracks session state and prevents session-based
attacks. An example of sessionization is remembering the maximum length associated with an input
field, such as a password.
Session cookies allow Application Firewall to match the requests from a particular user to the
appropriate server responses. Although application Firewall does not modify application cookies, it
N

tracks sessions by appending data to application cookies if cookie consistency is enabled.

Application Firewall uses these hashed cookies to confirm that the cookies were not modified on
ot

the client side of the transaction.

fo

By default, the session cookie name is citrix_ns_id, but this name can be modified on the Engine
rr

Settings dialog box.

es
al
e
or
di
s tri
bu
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 169
 N
ot
fo
rr
es
al
e
or

The figure shows how Application Firewall maintains session state by adding a session cookie to the
di

server response. The session cookie is returned on the next client request. If any part of the session
state has been modified, it will no longer match the information stored in the Application Firewall
s

session cookie on the return. Therefore, Application Firewall will block the request.
tri
bu

Relaxations
t io
n

170 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
The figure shows the Add Cookie Consistency Check Relaxation dialog box in the Configuration
Utility. After the cookie consistency check has been enabled, all cookies must return unaltered. If a
protected web site uses cookies that need to be added or modified by users, administrators can
specify relaxations for certain cookies.
Cookie relaxations can be disabled without removing the cookies from the list. Disabling cookies
allows an administrator to later re-enable a cookie relaxation without having to redefine it.

Adding a Cookie Consistency Relaxation in the Command-

Line Interface
Enter the following command to add a cookie consistency relaxation:

bind appfw profile name -cookieConsistency "cookie_name" -

N

isRegex ( REGEX | NOTREGEX ) -comment "string" -

ot

state ( ENABLED | DISABLED )

fo

The following table lists the available arguments.

rr

Argument Description
es
al

name Identifies the profile being modified

e

cookie_name Identifies the name of the cookie in the

or

relaxation being added

di

string Defines an optional comment about the

s

relaxation
tri

Initial Sets the initial state of the relaxation, which can

bu

be either ENABLED or DISABLED

t io

For example, enter the following command to add a relaxation that allows users to alter the cookie
n

named cookie_ex in the pr_badstore profile:

bind appfw profile pr_badstore -cookieConsistency "cookie_ex"

-isRegex NOTREGEX -state ENABLED

Deleting a Cookie Relaxation in the Command-Line

Interface
Enter the following command to delete a cookie consistency relaxation:

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 171
 unbind appfw profile name -cookieConsistency "cookie_name"

The following table lists the available arguments.

Argument Description
name Identifies the profile

cookie_name Identifies the name of the cookie in the

relaxation being deleted

For example, enter the following command to delete the relaxation previously set for cookie_ex in
the pr_badstore profile:
N
ot

unbind appfw profile pr_badstore -cookieConsistency "cookie_ex"

fo

Form/Hidden Field Manipulation

rr
es

Forms are a common method of collecting user information in web applications. However, form
fields can be manipulated and are vulnerable to attacks. Malicious code and data can be injected
al

into the application using read-only form fields or hidden fields, causing the web server or
e

application to be compromised.
or

By using hidden or form field manipulation, hackers can manipulate the data that is inserted into
form fields and perform unauthorized actions on a web site.
di

Form/Hidden Field Manipulation falls under PCI Section 6.5.3: Broken authentication and session
s

management.
tri
bu
t io
n

172 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Example of Hidden Field Manipulation

N
ot
fo
rr
es
al
e
or
di

The pricing information is in a hidden field on the web site not viewable in the user's browser
s tri

space. Using a proxy server, a hacker could access and manipulate the HTML code containing the
price information before the form is posted to the server.
bu

Other form manipulation attacks could be used to modify hidden parameters or values associated
t

with the user. The functionality of the application and the purpose of the form determines the
io

severity of this type of manipulation. For example, field manipulation can result in the modification
n

of account privileges so that instead of user-level permissions the account has full application
administrator permissions.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 173
Form Field Consistency Protection

N
ot
fo
rr
es
al
e
or
di

Application Firewall contains several checks and security measures to block form field and hidden
s tri

field manipulation attacks. For each user session, Application Firewall ensures that:
bu

• Each field is returned.

• No fields were added by the client.
t io

• No read-only or hidden fields were altered.

n

• Lists and buttons conform to requirements.

• Maximum field length was not exceeded.
Application Firewall remembers all values on a form that is served to the client in a given session
and prevents them from being manipulated.
Application Firewall modifies forms by inserting a hidden variable named as_fid. The hidden field
is used for field consistency, and it stores a form tag to identify with which form these name-value
pairs are associated.
For example:

174 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
 <input type=hidden name=as_fid value=211478770231/>

Field Consistencies
The form field consistency check verifies the following:
• The read-only or hidden fields have not been altered by the user.
• The user did not add new fields to a response.
• The fields sent to the user were returned by the user.
• The HTML field lengths and types have not been altered by the user.
• The response data corresponds to one or more of the predefined values in the fields, such as
list box or option fields.
N
ot

User Sessions
fo

When form field consistency is enabled, Application Firewall signs each form sent by the protected
rr

site before passing it along to the user. When the user returns that form, Application Firewall
verifies that the structures of the fields match what was originally sent.
es

To validate form fields, Application Firewall needs to keep track of user sessions. Administrators
al

should therefore assign only profiles with this check enabled to sites that use forms. Doing so
e

increases overall performance by not storing unnecessary session data.

or

Adding a Form Field Consistency Relaxation Using the

di

Command-Line Interface
s tri

Enter the following command to add a form field consistency relaxation:

bu
t

bind appfw profile name -

io

fieldConsistency "field_name" "action_URL" -

n

isRegex ( REGEX | NOTREGEX ) -comment "string" -

state ( ENABLED | DISABLED )

The following table lists the available arguments.

Argument Description
name Identifies the profile

field_name Identifies the field in the relaxation to be added

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 175
Argument Description
action_URL Identifies the action URL associated with the
field

string Defines an optional comment about the

relaxation

For example, enter the following command to add a relaxation for a form field named field_ex that
has an action URL of www.example.com/example.cgi to the pr_badstore profile:

bind appfw profile pr_badstore -

fieldConsistency "field_ex" "www.example.com/example.cgi" -
isRegex NOTREGEX -state ENABLED
N
ot

Deleting a Form Field Consistency Relaxation Using the

fo

Command-Line Interface
rr
es

Enter the following command to delete a form field relaxation:

al

unbind appfw profile name -

e

fieldConsistency "field_name" "action_URL"

or

The following table lists the available arguments.

di
s

Argument Description
tri
bu

name Identifies the profile

t

field_name Identifies the field in the relaxation being

io

deleted
n

action_URL Identifies the action URL associated with the

field

For example, enter the following command to delete the relaxation for the form field named
field_ex with an action URL of www.example.com/example.cgi from the pr_badstore profile:

unbind appfw profile pr_badstore -

fieldConsistency "field_ex" "www.example.com/example.cgi"

176 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Forceful Browsing

N
ot
fo
rr
es
al
e
or
di

As the figure shows, forceful browsing is a type of application attack where hackers manipulate
s

request URLs to gain access to content that they are not entitled to view. This is accomplished by
tri

manually entering URLs to get access to parts of the site that are not directly exposed across the
bu

interface or links provided to users. Forceful browsing is called forceful because the attacker forces
the path to resolve instead of relying on configured paths and links provided across the application
t io

interface and in-application navigation directives. It is considered a brute-force attack on the

application infrastructure.
n

Forceful browsing can be used to trigger buffer overflows, find and access content that users were
not intended or not authorized to access directly, or expose a backdoor into secure areas of the web
server and the web server infrastructure.
Forceful browsing falls under PCI Section 6.5.1: Unvalidated Input.

Forceful Browsing Protection

Application Firewall offers forceful browsing protection with deny URLs and the start URL features.
URL closure adds to this protection by eliminating the need to configure all the URLs of the site.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 177
Only requests that are explicitly allowed by an administrator through the start URL setting are
allowed. If URL closure is enabled, then Application Firewall parses the server responses and
maintains all the links that were sent to the user. These URLs are automatically allowed and do not
need to be configured as start URLs. Typing a different URL in the address line of the browser in
an attempt to subvert links will be unsuccessful.
Administrators configure the start URL feature at the same time as they configure URL closure. If
the application learning feature is used, additional URLs may be added to the start URL list.
Application Firewall can also be configured to block the request for access to web pages, such as the
supplier and accounts directory that may contain sensitive information.

Start URLs
N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

As the figure shows, the start URL check protects against forceful browsing by examining the URLs
to which incoming requests are directed and blocking connections to URLs that are not on the start
URL list. This check is enabled by default.

178 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
The Start URL List
The start URL list defines the URLs that are allowed to be accessed. These URLs can be regular
expressions or fixed paths.
The predefined Start URLs in a basic profile include:
• ^[^?]+[.](html?|shtml|js|gif|jpg|jpeg|png|swf|pif|pdf|css|csv)$
• ^[^?]+[.](cgi|aspx?|jsp|php|pl)([?].*)?$

Sessionization and Start URLs

Sessionization allows Application Firewall to track users’ states as they access a protected site.
Without this feature enabled, Application Firewall must treat each incoming request as a new entry
to the site. To prevent administrators from having to list every page on a protected site individually,
N

the basic defaults include two generic start URLs written as regular expressions. The PCRE
ot

expressions will match most requests for content, including queries. As an administrator adds
specific content to the start URL list, it may be necessary to modify or disable the generic start
fo

URLs.
rr
es

Modify Start URL Check

al
e
or
di
s tri
bu
t io
n

The figure shows the Start URL Settings and Start URL Relaxation Rules tabs, which an
administrator can use to modify the start URL check.

Adding a Start URL in the Command-Line Interface

Enter the following command to add a start URL:

bind appfw profile name -startURL "expression" -comment "string" -

state ( ENABLED | DISABLED )

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 179
The following table lists the available arguments.

Argument Description
name Identifies the profile being modified

expression Identifies the URL being added, written as

either a literal string or as a regular expression

string Defines an optional comment about the Start

URL

For example, to add www.example.com to the start URL list of the pr_badstore profile, enter the
following command in the command-line interface:
N
ot

bind appfw profile pr_badstore -startURL "www.example.com" -

state ENABLED
fo
rr

Deleting a Start URL in the Command-Line Interface

es
al

Enter the following command to delete a start URL:

e

unbind appfw profile name -startURL "expression"

or

The following table lists the available arguments.

di
s tri

Argument Description
bu

name Identifies the profile being modified

t io

expression Identifies the URL being deleted from the list

n

For example, enter the following command to remove www.example.com as a start URL in the
pr_badstore profile:

unbind appfw profile pr_badstore -startURL "www.example.com"

Backdoors and Misconfigurations

A backdoor is a hidden, undocumented entrance to a system or network that can be used to bypass
authentication requirements and other types of security measures. After a hacker has gained

180 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
entrance through the backdoor into the system, the hacker can inflict serious harm to the system or
gain concealed access to other systems.
Attacks based on backdoors and debug options fall under the same category as server
misconfiguration. Backdoors and debug options fall under PCI Section 6.5.1: Unvalidated Input.

Threats of Backdoors to an Environment

Backdoors are common system vulnerabilities and should be considered a major security breach.
Some enterprise environments contain a backdoor to a middle-tier system that does not require
authentication. A user in the enterprise, or a hacker, can access the system and exploit it to gain
sensitive information or to perform unauthorized actions.
Often, the impact of a backdoor existing on the system is underestimated and becomes a serious
problem. Creation of a backdoor can result in additional backdoors and can affect multiple systems
N

across an enterprise. Valuable resources are consumed during the effort to close the backdoor and
minimize the impact of the security breach, a process that is also very difficult. The existence of
ot

backdoors is best prevented, or at least mitigated, during the application design and development
phases.
fo
rr

Conventional and Unconventional Backdoors

es
al

The two main types of backdoors are conventional and unconventional. Unconventional backdoors
are more dangerous because organizations are often unaware of their origin. However, backdoors
e

are often created by company developers as a quick way to check an application during the
or

development phase. Often, these backdoors are never disabled because developers find it useful,
forget about it, do not know about it, or leave the company.
di

Knowledgeable hackers know how to search for backdoors and how to gain access to an entire
s

system. Preventing this attack is vital to the security of the application and to the entire system and
tri

its components. Just as the OWASP has defined their Top 10 Web Application Vulnerabilities, they
bu

also maintain a Top 10 list of Backdoors, including additional information on backdoors.

t io

Debug Code, Binaries, and Options

n

During development of an application, it is common for engineers to write code specifically

designed for debugging and testing. This code is not intended to be included in the final product;
however, it is not unusual for this code to remain because of human error. Leftover debugging code
exposes the application to hackers who can then use the code to take unauthorized actions with the
application. Hackers can also use the information they gain from the debugging code to target the
framework, database, or other application resources.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 181
Debug Options Attack Example
If a hacker knows that a debug code exists or wants to check, the hacker can take action upon
finding a URL that reads:

http://www.example.com/Cart.asp?user=joe

In an attempt to access debug code, the hacker can alter the URL to read:

http://www.example.com/Cart.asp?user=test

http://www.example.com/Cart.asp?debug=7

If the attack is successful because of a leftover debug code, then the application may produce details
the hacker is not authorized to see.
N
ot

Application Firewall and Backdoors and Debug Options

fo

Application Firewall reduces the risk of backdoors and debug code by limiting access parameters
rr

and the number of accessible pages. Some of the protections provided by Application Firewall that
es

reduce the risk of backdoors and debug codes include:

al

• Start URLs
e

• Deny URLs
• Form field consistency check
or

• Safe objects
di
s

Broken Authentication
tri
bu

Broken authentication vulnerabilities are often caused by administrators not protecting credentials
throughout their entire lifecycle. Results of broken authentication flaws can be hackers taking
t io

control of user or administration accounts and privacy violations. When a web server authenticates
n

to a back-end server, the credentials should be encrypted. If an administrator fails to protect these
credentials or if the password to gain access is weak, then a hacker can gain access to sensitive data.

Broken Access Control Lists and Weak Passwords

Administrators that do not configure a system correctly can leave URLs open for access by hackers
because they are not protected by access control lists. Using hacked access control lists can provide
an attacker with unauthorized access to sensitive documents and information. Hackers also test
logons for weak passwords, sometimes using forceful browsing. These credentials should be
protected and regarded as important to security.

182 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Protecting Against Misconfigurations
Application Firewall diverts many miscellaneous attacks, including insecure storage and
configuration management, because of its positive security model that blocks all traffic not known
to be valid. However, some attacks exist that do not conform to any of the security checks.
Avoiding insecure configuration management of the system is another way to protect against
miscellaneous attacks.
Application Firewall protects against miscellaneous attacks by using a combination of start URLs,
field consistency, deny URLs and safe objects. Several NetScaler features that are not explicitly
included as part of the Application Firewall engine can also provide security and protection features
for web applications that complement Application Firewall protections. These features include
responder, rewrite and URL transformation functionality.

URL Closure
N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

When URL closure is enabled, the start URL list should only contain pages through which users are
likely to enter the site initially or bookmark for future access, as shown in the figure. One point of
entry into an application must exist, such as the front page, and there can be many points of entry,

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 183
such as through bookmarks. Without URL closure, Application Firewall will block all requests for
pages that are not on the start URL list. The URL closure feature, which requires sessionization, is
not enabled in the basic defaults.

URL Closure works only on HTML links. JavaScript links will be blocked unless the pages
they refer to are added to the start URL list.

With URL closure, Application Firewall identifies all the hyperlinks included within the web page.
Users are allowed to access any web page that is in the list of start URLs or that a user goes to from
a link within the current session. With URL closure, a user is allowed to go to links included within
the allowed pages.

Enforcing URL Closure in the Configuration Utility

N

Use the following procedure to enable URL closure:

ot

1. Go to Security > Application Firewall > Profiles.

fo

2. Double-click the profile to be modified.

rr

3. Click the Security Checks node.

es

4. Double-click the Start URL security check.

5. Select Enforce URL Closure.
al

6. Click OK, OK, Save & Close then Done to save the settings and to close any open dialog
e

boxes.
or
di

Enforcing URL Closure in the Command-Line Interface

s tri

Enter the following command to enable or disable URL closure:

bu

set appfw profile name -startURLClosure ( ON | OFF )

t io

The name arguments identifies the profile.

n

For example, enter the following command to enable URL closure in the pr_badstore profile:

set appfw profile pr_badstore -startURLClosure ON

184 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Identity Theft Attacks

N
ot
fo
rr
es
al
e
or
di

As the figure shows, criminals engage in identity theft in order to make purchases in someone else’s
s tri

name, or to steal sensitive data such as social security numbers or health care records.
bu

Identity Theft Attacks falls under PCI Section 6.5.1: Unvalidated Input.
t io

Types of Identity Theft Attacks

n

Cross-site scripting is generally used to gain access to user accounts on web applications and to
steal the user’s identity for financial gain.
SQL injection is often performed to gain access to a web site database and to steal customer
information.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 185
Application Firewall Protection Against Identity Theft
Application Firewall can perform credit card checks that prevent the disclosure or leaking of credit
card numbers. During configuration, an administrator can limit how many card numbers can been
seen on the page or can replace the numbers with XXX.
Administrators can also define other types of information that should remain confidential as safe
objects and define them using regular expressions. safe objects can be used for employee ID
numbers, account numbers, social security numbers, and other types of sensitive information.

Credit Card Protection

N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

As the figure shows, the credit card security check prevents hackers from accessing the credit card
numbers belonging to users of a protected site. Major credit cards can be protected by requiring
Application Firewall to check all responses for number strings that match the patterns of those
cards.
This security check is disabled by default. Administrators may enable it for profiles that protect
commerce-related content with access to customer credit card numbers.

186 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
 If block is enabled, Application Firewall will block the credit card number, as well as the
remainder of the response from the server.

PCI-DSS reports in Application Firewall indicate whether Application Firewall configuration is in

compliance with the PCI-DSS criteria. If Application Firewall is not in compliance, the report
provides information on how to configure Application Firewall to become PCI-DSS compliant.

Predefined Credit Cards

Administrators can choose to protect the following major credit cards:
• American Express
• Diners Club
N

• Discover
ot

• JCB
• Master Card
fo

• Visa
rr

By default, all card types are unprotected. Administrators can select which patterns to require the
es

firewall to check server responses for, based on the cards that a protected site accepts as payment.
al
e

Credit Card Settings

or

Administrators can do more than block, log, or keep statistics on responses that match a protected
di

card type. The following additional action settings are available:

s

• Maximum credit cards allowed per page sets the number of protected card numbers that may
tri

be seen before any action is taken. By default, this number is set to 0, which means no cards
bu

will be displayed.
• X-Out replaces all but the last four digits in a credit card number with X before forwarding the
t io

response to the user.

n

Protecting Credit Cards

Administrators can find a list of major credit cards in the settings tab. From here, they can choose
whether to protect each card type.

Protecting Credit Cards in the Configuration Utility

Use the procedure in the following table to enable protection for one or more of the major credit
cards:

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 187
1. Go to Security > Application Firewall > Profiles.
2. Double-click the profile to be modified.
3. Click the Security Checks node.
4. Double-click the Credit Card security check.
5. Check a credit card (or multiple cards) to protect.
6. De-select the credit cards that should no longer be protected.
7. Click OK, OK, Save & Close then Done to save the settings and to close the open dialog
boxes.

Protecting Credit Cards in the Command-Line Interface

Enter the following command to modify Credit Card Protection:
N

set appfw profile name -creditCard card_names

ot

The following table lists the available arguments.

fo
rr

Argument Description
es
al

name Identifies the profile being modified

e

card_names Lists the cards being protected, which can

or

include one or more of the following:

• visa
di

• mastercard
s tri

• discover
bu

• amex
• jcb
t io

• dinersclub
n

Any card not listed will be set as

unprotected.

For example, enter the following command to protect Visa and Mastercard numbers in the
pr_badstore profile:

set appfw profile pr_badstore -creditCard visa mastercard

188 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Errors Triggering Sensitive Information Leaks
Errors triggering sensitive information leaks are common because web applications display many
error messages to users. Insecure configuration management and other vulnerabilities can cause
web applications to handle errors improperly and leak sensitive information.
Error triggering sensitive information leaks falls under PCI Section 6.5.1: Unvalidated Input.

Error Messages Resulting in Attacks

Hackers can purposely insert invalid data into a web application to view an error message. The
error message can reveal information that helps the hacker devise more targeted attacks. Some of
the information that hackers collect from error messages are server headers, error messages, SQL
and Java exceptions, failed logon messages, stack traces, debugging information, and non-existent
file errors. Skilled hackers can often figure out implementation details and other data to help them
N

identify vulnerabilities in the system with this information.

ot

Another method employed by hackers is to continually supply the same username with a different
fo

password for the logon screen. A response from the application is often the same text, such as no
such user or invalid password, but occasionally an application produces an alternate error code that
rr

reveals valuable information.

es
al

Reducing the Risk of Information Leaks

e

Application Firewall employs safe objects as a protection against errors triggering sensitive
or

information leaks. Other Application Firewall features that help mitigate the risk of information
leaks are deny URLs and field formats.
di
s tri
bu
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 189
Safe Object Protection

N
ot
fo
rr
es
al
e
or
di

As the figure shows, the safe object security check works in a similar way to the credit card check.
s tri

The only difference is that administrators use regular expressions to define custom objects that they
do not want displayed, such as employee ID numbers or certain types of error messages. After the
bu

objects are defined, all responses that match a safe object pattern will be altered according to the
action settings for that object.
t io

This security check is not enabled by default. After an administrator defines one or more safe
n

objects, each object must be enabled or disabled, and each has its own action settings.

Defining a Safe Object

Safe objects require a regular expression and a maximum match length. The regular expression
defines a pattern that Application Firewall can use to examine responses from a protected site.

The maximum match length prevents longer strings from being trapped in the safe object
definition.

190 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Adding a Safe Object
The first step in protecting custom content types from leaking out through a web site is to define it
in the safe object security check.

Adding a Safe Object in the Configuration Utility

Use the following procedure to create a custom safe object:
1. Go to Security > Application Firewall > Profiles.
2. Double-click the profile to be modified.
3. Click the Relaxation Rules node.
4. Double-click the Safe Object security check.
5. Click Add.
N
ot

6. Determine whether to enable the safe object immediately.

• Select Enable if the safe object is being defined for immediate use.
fo

• Deselect Enable if the safe object is being defined for future use.
rr

7. Type a name for the new safe object in the Safe Object Name field.
es

8. Determine which action settings to enable.

• Select Block to stop all matching responses.
al

• Select Log store all matching responses as events in the log file.
e

• Select Stats to generate statistics on all matching responses.

or

• Select X-Out to X out matching responses.

di

• Select Remove to remove the matching string from the response.

s

9. Type a regular expression that defines the content to be protected in the Regular Expression
tri

field.
bu

10. Type a positive integer that represents the maximum length of any content that would match
the new safe object in the Maximum Match Length field.
t io

11. (Optional) Type a comment about the safe object in the Comments field for future reference.
n

12. Click Create.

13. Click Close then click Done to acknowledge that the safe object was successfully created and
close the remaining dialog boxes.

Adding a Safe Object in the Command-Line Interface

Enter the following command to add a safe Object:

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 191
 bind appfw profile name -
safeObject object_name "expression" max_match -action actions -
comment string -state ( ENABLED | DISABLED )

The following table lists the available arguments.

Argument Description
name Identifies the profile

object_name Sets the name of the new safe object

expression Defines the new safe object as a regular

expression
N
ot

max_match Sets the maximum match length

fo

actions Lists the enabled action settings of the new safe

object, which can be one or more of the
rr

following:
es

• Block
• Log
al

• Stat
e

• Xout
or

• Remove
di

string Defines an optional comment about the safe

s

object
tri
bu

For example, enter the following command to define a safe object named ALLCAPS, which is
t

defined by the expression [A-Z]+ with a maximum length of 10 characters, and to enable logging of
io

this object in the pr_badstore profile:

n

bind appfw profile pr_badstore -safeObject ALLCAPS "[A-Z]+" 10 -

action log -state ENABLED

192 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Adaptive Learning for Security

N
ot
fo
rr
es
al
e
or
di

For many types of web applications a profile protects, listing each field name, URL, and cookie
s

name in the security check settings can be a daunting task. To assist administrators in customizing
tri

profiles, Application Firewall includes an adaptive learning engine, which can be applied to certain
bu

security checks, as shown in the figure. Applying learning is especially helpful with profiles that
protect advanced content, which is why the advanced defaults have learning enabled for all relevant
t io

security checks.
n

Adaptive learning functions as a repetitive activity filter that identifies typical user behavior when
accessing protected web sites. Based on these behaviors, Application Firewall recommends
appropriate settings for the request inspection and input validation filters.

The learning engine works under the assumption that the majority of traffic to a protected
site includes normal, non-malicious requests. Therefore, administrators should not enable
learning while a vulnerability scanner is applied to a protected site because it will suggest
inappropriate rules.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 193
Learning Over Time
Enabling learning on a particular security check is just the first step. Learning occurs as traffic flows
through Application Firewall, so administrators must assign a profile to a policy bound to active
content.
The amount of time it will take to generate valid learning suggestions depends on how often users
access content protected by a particular security check. If an application is used frequently, then the
learning engine can recommend rules in a short amount of time. Applications or pages that are
used less frequently will take longer to generate learned rules.

Learning Thresholds
How often the learning engine suggests rules is determined by the learning thresholds set for each
security check. The first threshold determines the minimum number of sessions before a repeated
N

behavior is identified as a pattern.

ot

Minimums are only part of the equation. Administrators should also specify a percentage of times
fo

that a particular behavior occurs. For example, a BadStore administrator could set a start URL
learning threshold that requires at least 20 percent of users to enter BadStore.net through the
rr

"What’s New?" page before the learning engine can suggest this page as a Start URL.
es

When a minimum threshold is set, URL requests that are made by mistake are less likely to be
included in the suggested rules list. The percentage of times that a request is made also helps
al

eliminate infrequent requests.

e
or

Generalized and Simple Rules

di

Administrators have two options for how to view the list of suggested rules for a security check.
s

The Simple tab provides a list of all suggestions written as literal strings. Depending on the variety
tri

of traffic flowing through Application Firewall and the learning thresholds that were set, this list
bu

could be quite long. In addition, deploying rules from the simple list could be too narrow, causing
similar fields, URLs or cookies to be neglected.
t io

The Generalized tab lists the same recommendations as the simple tab, but they are grouped
n

together through the use of regular expressions. Administrators can limit the number of expressions
shown on this list, which will cause the firewall to restrict or expand the expressions to fit the set of
recommendations. If this limit is set too low, the expressions will be too general and may allow
dangerous content through Application Firewall.
Deploying the learning engine recommendations is a balance between potentially restricting too
much content or too little content.

194 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Learned Rules
Not all suggested rules will be appropriate, which is why administrators must actively decide
whether to add a rule to the profile that generated the rule. Each rule comes with three options:
• Deploy: add the rule to the current settings of the security check in question.
• Edit: manually alter the rule before deploying it.
• Skip: remove the rule from the list without deploying it.

Enabling Learning
Learning can be enabled for the following security checks:
• Start URL
N

• Cookie consistency
ot

• Form field consistency

• Field formats
fo

• Cross-site scripting
rr

• SQL injection
es
al

Enabling Learning in the Configuration Utility

e

Use the following procedure to enable learning for a security check:

or

1. Go to Security > Application Firewall > Profiles.

di

2. Double-click the profile to be modified.

s

3. Click the Security Checks node.

tri

4. Select the Learn check box next to the desired security check.
bu

5. Click OK.
t io

6. Click Done.
n

Enabling Learning in the Command-Line Interface

Enter the following command to enable the learning function:

set appfw profile name-checkAction learn

The following table lists the available arguments.

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 195
Argument Description
name Identifies the profile

checkAction Identifies the security check for which learning

is being enabled, which can be one of the
following:
• startURLAction
• cookieConsistencyAction
• fieldConsistencyAction
• fieldFormatAction
• crossSiteScriptingAction
• SQLInjectionAction
N
ot

This command is the same as the one used to enable or disable standard action settings,
fo

such as Block or Log. If the syntax listed above is used, all action settings not listed will be
disabled.
rr
es

Setting Learning Thresholds

al
e

Learning thresholds determine how often the Application Firewall needs to witness a particular user
or

behavior before creating a recommendation. Higher thresholds lead to a more restrictive list.

Learning Thresholds are best handled in the Configuration Utility. Command-line

di

interface commands are not included for this procedure.

s tri
bu

Setting Learning Thresholds in the Configuration Utility

t io
n

Use the following procedure to set learning thresholds for certain security checks:
1. Go to Security > Application Firewall > Profiles.
2. Double-click the profile to be modified.
3. Click the Learned Rules node.
4. Select a security check in the pane on the left and click Settings.
5. Enter the thresholds for learning.
6. Click OK, then click Done.

196 Module 5: Attacks and Protections © Copyright 2016 Citrix Systems, Inc.
Managing Learned Rules
After a security check has learning enabled and relevant traffic has flowed through the Application
Firewall, suggested rules for each check will be listed in one location. From here, the administrator
can examine each rule and decide whether to deploy, edit or skip.

Learned rules are best handled in the Configuration Utility. Command-line interface
commands are not given for this procedure.

Managing Learned Rules in the Configuration Utility

Use the following procedure to manage learned rules:
1. Go to Security > Application Firewall > Profiles.
N

2. Double-click the profile to be modified.

ot

3. Click the Learned Rules node.

fo

4. Select a security check in the pane on the left.

rr

5. Click Edit.
es

6. Determine whether to view simple or generalized rules.

• Click the Simple tab to view each learned suggestion as a literal string.
al

• Click the Generalized tab to view all suggestions grouped together as PCRE regular
e

expressions.
or

7. Select a learned rule.

di

8. Determine what action to take with this suggestion.

s

• Click Edit and Deploy to alter the rule before adding it to the active settings.
tri

• Click Deploy to add the rule to the active settings of the security check.
bu

• Click Skip to remove the rule from the list of suggestions until the repeated behavior
crosses the thresholds again.
t io

9. Click Close and then Done.

n

© Copyright 2016 Citrix Systems, Inc. Module 5: Attacks and Protections 197
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

198 © Copyright 2016 Citrix Systems, Inc.

 6
Module 6

Application Firewall
Troubleshooting
N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

200 © Copyright 2016 Citrix Systems, Inc.

Application Firewall Troubleshooting Manual
Overview
To effectively troubleshoot Application Firewall, an administrator must understand the following
points:
• How Application Firewall affects applications
• How to gather network, device, and application data
• How to characterize an issue
After data has been gathered and analyzed, an administrator can form hypotheses as to the nature
of the problem and can take appropriate actions to resolve the issue.
N
ot

Objectives
fo

After completing this module, you will be able to:

rr

• Identify how Application Firewall affects applications.

es

• Identify troubleshooting techniques.

• Gather troubleshooting data.
al

• Identify common HTTP errors.

e

• Resolve common configuration issues.

or
di

Application Firewall and Applications

s tri

While the Application Firewall does not require any changes to back-end web applications, it may
bu

appear to affect those applications because it intercepts traffic and can maintain sessions. In reality,
the Application Firewall does the following actions:
t io

• Appends session cookies

n

• Drops, adds or modifies HTTP headers

• Makes minor changes to forms

HTTP Headers
HTTP requests and responses use headers to send information about the HTTP message. Some
header fields are used in both request and response headers, while others are used in one or the
other. Many request header fields will allow the client to specify several acceptable options in the
value and, in some cases, even rank the preference of each option. For more information about
HTTP headers, see the W3.org web site.

© Copyright 2016 Citrix Systems, Inc. Module 6: Application Firewall Troubleshooting 201
Dropped Request Headers
The Application Firewall drops compression and caching headers so that every request can be
viewed within the context of a session. The following table describes several additional HTTP
request headers dropped by the Application Firewall.

HTTP Header Description

Range
Used to recover from failed or partial file
transfers
This header is dropped to prevent, for example,
half a credit card from appearing in one
segment and the other half from appearing in
N

another.
ot

If-Range
Used to retrieve a partial object when it
fo

contains a part of that object in its cache already

rr

(conditional GET)
This header is dropped to prevent, for example,
es

half a credit card from appearing in one

al

segment and the other from half appearing in

another.
e
or

If-Modified-Since
Used to send a 304 (not modified) error if the
requested object has not been modified since
di

the time specified in this field

s tri

This header is dropped because all requests and

associated responses must be inspected by the
bu

Application Firewall.
t io

Accept-Encoding
Used to determine what compression methods
n

are allowed for a particular client

This header is dropped to prevent compression
from the server to the Application Firewall.

If-None-Match
Used to allow efficient updates of cached
information with a minimum amount of
overhead
This header is dropped because the Application
Firewall prohibits the caching of pages.

202 Module 6: Application Firewall Troubleshooting © Copyright 2016 Citrix Systems, Inc.
Dropped Response Headers
The Application Firewall drops the Content-Length and Last_Modified headers from responses
because the Application Firewall may block or modify content (such as by masking a credit card
number) that would result in a size mismatch.

Modified Response Headers

The Application Firewall modifies the headers listed in the following table.

HTTP Header Modification

Accept-Range: none Prohibits access to partial objects
N

Cache-Control: no-cache, no-store Prohibits caching of pages

ot

Pragma: no-cache Prohibits caching of pages

fo
rr

Expires: date Sets a page expiration date to the past so that it

is always refreshed
es
al

Added Response Headers

e
or

If the Application Firewall has modified a response, it sets, in addition to the citrix_ns_id cookie,
the Transfer-Encoding header to be chunked. This header results in content being streamed back to
di

a client without having to know the total length of the response first. Therefore, the server starts
s

writing as soon as the internal response buffer has been exceeded, rather than waiting for the
tri

complete response.
bu

The purpose of chunking pages is so that the Application Firewall does not have to process an
entire page before presenting it to the user; instead, the user can see the chunks of the page that
t io

have passed inspection immediately. If the Application Firewall encounters, for example, a credit
card violation, it will not display the remainder of the page.
n

HTML Comment Stripping

HTML Comment Stripping, which is disabled by default, causes the Application Firewall to strip
any data that appears between the

<!--

and

© Copyright 2016 Citrix Systems, Inc. Module 6: Application Firewall Troubleshooting 203
 -->

comment tags. The feature is designed to remove any comments a developer includes in the HTML
code of an application before the response is sent to the user. If a web site includes extensive
HTML comments, stripping the comments can improve response times by reducing the amount of
data that must be sent to the user, as well as deny hackers information they might find useful. For
more information about enabling the feature, see "HTML Comment Stripping" on page 127.

Configuration Issues
When troubleshooting and checking configuration issues, an administrator should address the
following questions:
• Are the protected servers defined correctly?
N

• Are the ports defined correctly?

ot

• Are the SSL certificates and keys correct?

• Are the virtual servers and services bound properly?
fo

• Are there issues with the security inspections?

rr

• Are the Start URLs defined correctly?

es

• Have the regular expressions been verified?

al

• Are there issues with the cookies?

e

• What issues have been recorded in the log file?

or

• Have the profiles and policies been created correctly?

• Have the appropriate features and checks been implemented?
di
s tri

Policy Issues
bu

If configured protections do not seem to be applied to a web application, an administrator should

t

verify that the policy is defined correctly. Questions to consider include:

io

• How do users access the application? Is it by FQDN or IP address?

n

• Do the expressions in the policy define the correct type and content of requests to the
application?
• Does the policy include the correct profile in the action field?
• Does the policy counter increment when the application is accessed? If not, then no traffic is
hitting that policy.

204 Module 6: Application Firewall Troubleshooting © Copyright 2016 Citrix Systems, Inc.
Profile Issues
A variety of issues may occur if a profile is not configured correctly. The following questions can
help narrow the possible issues:
• Is the relevant profile defined correctly?
• Do any errors appear in the log? Application Firewall events should be checked. Is something
being blocked that should not be?
• Are start URLs configured correctly? If URL closure is not selected, all pages that the users are
allowed to see must be defined by one or more start URLs.
• Are regular expressions defined properly? A typo or poorly designed regular expression could
cause the wrong traffic to be blocked or too much traffic to be let through to the web server.
• Is the error page defined correctly? By default, the error page is set to "/".
• Are the associated policy counters incrementing?
N
ot

Suggested Actions
fo

Information in the following table may help focus the profile troubleshooting process.
rr
es

Issue Suggested Action

al

Users cannot access the home page. A home Verify that:

e

page redirect fails. • A start URL is defined for the application

or

URL and default home page.

di

• Application Firewall can connect to the web

server.
s tri

• The users can connect to the Application

Firewall VIP address.
bu

Users cannot access any links. Verify that:

t io

• One or more start URLs define every page

n

in the site.
OR
• A start URL is defined for the home page,
and URL closure is enabled.

Images do not display. A CSS style is not being Verify that CSS and image rollovers are
applied to the site. included in the start URL list.

Users cannot log on or register new accounts. Verify field types and start URLs are defined
Users cannot submit forms. correctly.

© Copyright 2016 Citrix Systems, Inc. Module 6: Application Firewall Troubleshooting 205
Issue Suggested Action
Some pages are not being served or do not • Check the firewall log for cookie validation,
display properly. field consistency, and Start URL errors.
• Verify that HTML comment stripping is
not affecting page display or removing
active scripts.
N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

206 Module 6: Application Firewall Troubleshooting © Copyright 2016 Citrix Systems, Inc.
 7
Module 7

Queuing and
Connection Tuning
N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

208 © Copyright 2016 Citrix Systems, Inc.

Queuing and Connection Tuning Manual
Overview
The NetScaler system optimizes HTTP requests and responses through queuing and active
connection management. Through surge protection and surge queuing, connections to the server
occur at a rate that the server can handle and acts as a temporary HTTP request placeholder before
being transmitted to the server. In addition, priority queuing filters incoming HTTP traffic based
on certain criteria. Various factors impact which optimizations are implemented, including which
behaviors the client and server expect. The major factors that impact optimization include if the
client and server support or use keep-alive connections, if the server depends on the true IP address
of the client, how the application behaves in regards to timing.
N

Objectives
ot

At the end of this module, you will be able to:

fo
rr

• Configure surge protection.

• Configure priority queuing.
es

• Explain how the NetScaler system uses HTTP connections.

al

• Identify which options impact which behaviors.

e

• Tune HTTP and TCP behaviors.

or

• Identify connection tuning best practices.

di
s

HTTP Connections
tri
bu

HTTP connection characteristics that affect traffic behavior include keep-alive HTTP connections,
HTTP 1.0 and 1.1 behavior, and pipelined requests.
t io
n

Keep-alive HTTP Connections

A TCP connection can be reused for many requests with a keep-alive connection. A non-keep-alive
connection can only be used for a single request and response. Regardless of the HTTP version
being used, a browser and server can control keep-alive behaviors using the Connection: Keep-alive
or the Connection: Close headers. With the close behavior, the server is responsible for closing the
TCP connection after it finishes transmitting the requested content, allowing for backwards
compatibility with older HTTP clients that cannot track a completed response.

© Copyright 2016 Citrix Systems, Inc. Module 7: Queuing and Connection Tuning 209
HTTP 1.0 and 1.1 Behavior
Key differences between HTTP 1.0 and 1.1 impact NetScaler connection behaviors.
• HTTP 1.0 does not define keep-alive behaviors. However, when HTTP 1.0 use increased, the
Connection: Keep-alive header started allowing clients to indicate to servers that they support
connection reuse.
• HTTP 1.0 does not allow response chunking, which is generally used for dynamic content.
When a server fails at using chunking, then if often inserts a Connection: Close header.
• HTTP 1.1 defaults to using keep-alive unless the connection header specifies otherwise,
although all browsers using HTTP 1.1 continue to insert the Connection: Keep-alive header for
compatibility with HTTP 1.0 servers.
• HTTP 1.1 requires a host header, while HTTP 1.0 does not.
N

Pipelined Requests
ot

Pipelined requests are keep-alive requests that are allowed but not required for HTTP 1.1 support.
fo

With pipelined requests, many distinct requests are sent before responses are received, which
rr

should improve performance. However, pipelined requests are rarely used due to various factors,
including the following:
es

• Not knowing how long a response takes makes is difficult to determine the optimal order to
al

send requests and on what connection.

e

• Servers can close connections after the first response, forcing requests to be retransmitted.
or

• Many load balancing devices do not cleanly support pipelining due to potential DoS issues,
forcing large amounts of request data to be buffered.
di

• No mechanism exists to cancel a pending request on one connection and retransmit the
s

request on another connection due to slow responses from a prior request.

tri
bu

HTTP Connection Management and NetScaler HTTP

t

Behavior
io
n

The NetScaler system uses connection multiplexing for handling HTTP requests. Many requests
from different clients can be dispatched to the server one at a time across the same TCP
connection. Various conditions dictate which connections are used for a particular request,
including:
• If use source IP address (USIP) is specified, then only a connection matching the IP address of
the client is used. However, requests are not necessarily mapped 1:1 between the client
connection and server connection.
• If TCP buffering is not enabled, then the client and server TCP connection maximum segment
size (MSS) is matched up, ensuring that the packet size transmitted by the server is not
changed when it is sent to the client. This action minimizes the amount of data repackaging

210 Module 7: Queuing and Connection Tuning © Copyright 2016 Citrix Systems, Inc.
 that needs to be done by the NetScaler. If TCP buffering is enabled, then all connections to the
server use the maximum MSS for the server, reducing the number of TCP connections but
increasing the load on the NetScaler system.
As the system multiplexes, it attempts to touch the server to client packets as little as possible,
which is why the MSS is matched. This action allows the NetScaler system to depend on the
target server to handle retransmission without using memory resources on the NetScaler
system, although it can be tuned.

Client Keep-Alive
To ensure that client and server connections are not prematurely closed, the system provides
Connection: Close interception. This function determines when a request contains a Connection:
Close header and modifies it in such a way that it will not be recognized. Since this header is
intercepted, the client connection will remain open. This header act as an intermediary between the
N

client and server so that expected behavior in each direction is met. In the client to server direction,
ot

it is implied to prevent denial of service attacks, and cannot be overridden. In the server to client
direction, it is enabled using the client keep-alive behavior, which can be configured on a service-
fo

by-service basis (not globally). A best practice is to disable client keep-alive unless it is certain that
the server will unnecessarily close connections frequently.
rr
es

Connection IP Address Control

al
e

By default, the NetScaler system uses a system-owned IP address to establish connections to the
server. This IP address can be a mapped IP address (MIP) or a subnet IP address (SNIP). By
or

activating USIP mode, the NetScaler system spoofs ownership of the IP address that the client itself
is using and establishes connections to the server on its behalf. A best practice is to avoid USIP
di

mode for HTTP traffic because it negates many benefits of connection multiplexing except in
s

extreme cases where large numbers of requests come from the same IP address.
tri

In the event that a back-end server requires the IP address of the client and USIP mode is not used,
bu

client IP address header insertion can be enabled. With this function, a custom header contains the
client IP address and is added to the HTTP request. Because this function is frequently used in
t io

proxy servers, many software packages provide a way to extract this value and internally use it for
n

logging and access control. For more information about how to handle various software packages,
refer to the Citrix Knowledge Center and available NetScaler system documentation.

Maximum Requests and Maximum Connections

The max-request and max-connection settings can be used to control server connections. The
maximum request setting controls how many requests can traverse a given HTTP connection
before that TCP connection is closed. Although this setting in not often used, it is needed in case
the servers have issues with memory leakage because memory can only be freed by closing the TCP
connection.

© Copyright 2016 Citrix Systems, Inc. Module 7: Queuing and Connection Tuning 211
The max-connection option controls how many connections are made to the server at one time in
the established state. With Microsoft IIS, this option is not always needed, but it may improve
performance in surge situations. Apache, however, has a corresponding setting. This value should
be lower than the value configured on the server by some small amount, ensuring that the
NetScaler system acts as a bottleneck instead of leaving it to the server. For Apache, the default
number of connections is often 256. Therefore, a setting of 250 on the service is appropriate. IIS
does not have a default connection limit, but 250 is often acceptable. With servers that generate
large amounts of dynamic content, this value should be lowered, sometimes by a factor of 10 or
more to 25 or fewer connections to prevent databases from being over-saturated by too many
concurrent queries.
When using the max-connection setting, an administrator should ensure that other settings do not
cause conflict. If USIP mode is enabled and the max-connection is set, then new users from new IP
addresses may not be able to make connections to the server if the connection limit has been
reached. If this configuration is necessary, then the server idle timeout value should be set
sufficiently low so that the number of server connections is maintained below the limit under
N

normal conditions. Care should also be taken to monitor the state and send alerts if the limit is
ot

almost reached. TCP-based monitors are disabled when the max-connection setting is reached
because it is assumed that the server no longer has connection resources. If the setting has been set
fo

lower than what the server itself considers the limit, then the monitors do not exceed the max-
client limit.
rr
es

Connection Idle Settings

al
e

Connection idle timeouts can be adjusted in two places: the virtual server and the service.
or

The virtual server has only a client idle timeout value and not a server idle timeout value. The
service has both a client and a server idle timeout values. With load balancing, the service client
di

idle timeout value is not used, which can cause confusion. This value is only used in transparent
configurations, which are rarely implemented. Therefore, the two values to consider are the virtual
s tri

server client timeout value and the service server timeout value.
bu

A timeout occurs for an active transaction when either the client or the server timeout is exceeded
without any traffic passing on the link. If a TCP window update occurs without any data passing,
t

this update is not considered activity and does not reset the timeout. If a server or client connection
io

is idle, such as no active transaction on the connection, then only the server or the client timeout
n

applies.
Another factor affecting how idle connections are closed is the zombie cleanup process. This
process is a garbage collection routine that runs periodically and initiates the RST packet when
connections are flagged as idle. Until the zombie cleanup process runs, any connection that exceeds
its idle timeout may still pass data and be unflagged from the idle state. By default, the zombie
cleanup process is set to occur every 120 seconds. However, it can be globally reduced to a lower
value, though setting it less than every one second is not recommended.
A best practice is to tune the server and client idle timeout values according to the maximum
length of time objects take to be returned or the idle timeout of the client browser or server. Most
browsers close connections that do not have pending requests within 60 seconds, and most servers

212 Module 7: Queuing and Connection Tuning © Copyright 2016 Citrix Systems, Inc.
do the same in 300 seconds or less. Therefore, idle timeouts in between, such as 180 seconds, are
frequently used.

Trackable Connections
For connection multiplexing to work properly, the NetScaler system must be able to detect when a
request and a response has been received completely, a process called tracking. However, although
rare, legal or potentially illegally formatted requests or responses can be received by the system that
breaks the tracking process, leading to non-trackable connections. The most common situations
that can result in non-trackable connections are:
• Requests that are longer than 24KB or 16 packets
• Invalid content-length values for either request or response body data
• Server responses that are formatted using MIME headers
N

By default, the system stops reusing connections that are in this state, resulting in pinned
ot

connections between the client and server that cannot be reused for other client requests. These
connections result in connection buildup over time, especially if it happens frequently. A client DoS
fo

attack using such requests can be prevented when a global flag of drop invalid requests is provided,
rr

which is a best practice. For compatibility with certain applications, however, this option is disabled
by default. If non-trackable server responses are encountered, lowering the server idle timeout is the
es

most effective solution to resolve the problem until the application can be corrected.
al

Client IP address insertion does not work on non-trackable traffic.

e
or
di

TCP Buffering
s tri

TCP buffering is an optimization technique where the NetScaler system takes responsibility for
bu

repackaging data from the server that is being transmitted to the client, ensuring that:
t

• The server does not have to process retransmits.

io

• The NetScaler system sends an ACK for data transmitted by the server, increasing server
n

performance.
• The number of connections needed is reduced since each connection can be reused faster, even
if the client has not yet sent an ACK for the data.
• The client has a low MSS, allowing the server to transmit fewer frames and reducing overhead
on the network interfaces.
• Denial of service attacks can be prevented because clients force the response to move slowly by
using a small MSS and by delaying ACK packets.
TCP buffering results in approximately 30% higher resource utilization on the system , but the
improved performance and increased robustness of the application justifies it. If the NetScaler

© Copyright 2016 Citrix Systems, Inc. Module 7: Queuing and Connection Tuning 213
system runs out of memory, then the server is forced to buffer data through the use of TCP
window size control.

Down-State Flush and Access Down Connection Settings

Other settings that control how connections to a server are managed are the down-state flush and
access down settings. Down-state flush addresses whether or not all established connections will be
forcibly terminated when a service goes down. In some situations, especially when monitors are
aggressive, this setting may not be desired because a service may have to go down because of slow
response times, which is not always an indication that the service is actually offline.
The access down option impacts connections that are not actually load balanced but are instead
being handled transparently. This option is often the result of an external monitoring device
performing health checks against the server port directly, as well as traversing the NetScaler system
to do it. While not apparent, the NetScaler system terminates such requests as it does when load
N

balancing. Furthermore, it does not open a new back-end connection to the server if it is flagged as
ot

down for any reason. The access down flag allows the transparent connection requests to bypass the
state control and access the server.
fo
rr

TCP Optimization
es

The NetScaler system improves TCP performance through manipulation windows, which are the
al

amount of data that can be transmitted without the receiver acknowledging the data. In older
e

implementations of TCP, this window is a fixed size, implying that after a certain amount of data is
transmitted, then an ACK packet needs to be received before more data is transmitted. To optimize
or

TCP behaviors, the NetScaler system provides two extensions to TCP to optimize the TCP window
behaviors. It supports the configuration of the initial advertised window size, allowing for the use of
di

window scaling and selective acknowledgments (SACK).

s tri
bu

Advertised Window Size

t io

The advertised window size is the initial size that the NetScaler system provides in SYN or SYN-
ACK packets to clients or servers. The larger the window size, the more likely the burst can
n

overwhelm buffers on intermediate devices, although reasonably large window sizes, less than 128K,
can be used. Therefore, the advertised window size must be larger than the normal request size for
most web sites. When TCP buffering is enabled, a best practice is to have the window size set to the
same size as the per-connection TCP buffer.

Window Scaling
Window scaling relates to how the TCP window is handled after initial connection and how it
grows as traffic is handled. When enabled, the window grows larger and larger until packet loss
occurs as the NetScaler system passes data. At this point, it stops increasing in size. It is important

214 Module 7: Queuing and Connection Tuning © Copyright 2016 Citrix Systems, Inc.
to enable window scaling when large responses are a typical website behavior, or when remote
clients connect to the site through a high latency connection, such as when Internet users
worldwide attempt to download large files.

Selective Acknowledgement
Selective acknowledgement (SACK) optimizes retransmission data handling. When TCP detects a
dropped packet, retransmission starts from the last packet and continues, which also includes data
that was not dropped, forcing unnecessary data transmission twice. SACK only allows the data that
was not received to be retransmitted. Most modern TCP stacks provide SACK support and can
improve performance on networks with large packet loss, especially when combined with large
window sizes.

Surge Queue
N
ot

A surge queue acts as a temporary placeholder for HTTP requests before they are transmitted to a
server. Under certain conditions, such as those involving memory availability, requests cannot be
fo

processed immediately.
rr
es

Surge Protection
al

When a surge in client requests overloads a server, the server response becomes slow because the
e

server cannot respond to new requests. The surge protection feature ensures that connections to the
or

server occur at a rate that the server can handle. The response rate depends on how surge
protection is configured. The NetScaler system tracks the number of connections to the server and
di

adjusts the rate at which it opens new server connections.

s

Surge protection is enabled by default on the NetScaler system and can be disabled as appropriate.
tri

To configure surge protection, an administrator must set the throttle value to determine the
bu

aggressiveness of managing connection attempts. To configure a different maximum number of

concurrent connections allowed before surge protection is triggered by the default throttle value, an
t io

administrator should set the base threshold value independently.

n

If the NetScaler system is installed at the edge of the network, where it interacts with
network devices on the client side of the Internet, then the surge protection feature must
be disabled. Surge Protection must also be disabled if USIP mode is enabled on the system.

Request and Response Rates

When surge protection is disabled and a surge in requests occurs, the server accepts as many
requests as it can process concurrently. Then, it begins to drop requests. As the server becomes
more overloaded, the response rate is reduced to zero. When the server recovers, often after several
minutes, it sends resets for all pending requests. Because this behavior is abnormal, the server

© Copyright 2016 Citrix Systems, Inc. Module 7: Queuing and Connection Tuning 215
responds to new requests abnormally. The process repeats for each surge in requests. If the server is
under DDoS attack and receives multiple surges of requests, then it may become unavailable to
legitimate users.
When surge protection is enabled and a surge in requests occurs, surge protection manages the rate
of requests to the server. Requests are sent to the server as fast as the server can handle those
requests, enabling the server to respond to each request correctly in the order that it was received.
When the surge is over, the backlogged requests are cleared as quickly as the server can handle
them until the request rate matches the response rate.

Throttle Rate
The throttle is the rate per seconds at which the NetScaler system allows new connections to the
server to be opened once surge protection is triggered. The throttle can be set to the following
values:
N

• Aggressive. This option should be chosen when the connection-handling and surge-handling
ot

capacity of the server is low and the connections need to be managed carefully. When the
throttle is set to aggressive, the rate of new allowed connections is 16 per seconds.
fo

• Normal. This option should be chosen when there is no external load balancer behind the
rr

NetScaler system or downstream. When the throttle is set to normal, the rate of new allowed
es

connections is 200 per seconds. Normal is the default throttle option.

• Relaxed. This option should be chosen when the NetScaler system is load balancing between a
al

large number of web servers and can handle a high number of concurrent connections. When
e

the throttle is set to relaxed, the rate of new allowed connections is 500 per seconds.
or

Disabling Surge Protection in the Configuration Utility

di
s

The surge protection feature is enabled by default and can be disabled as needed. Use the following
tri

procedure to disable Surge Protection:

bu

1. Go to System > Settings.

t

2. Click Change advanced features under the Modes and Features section. The Configure
io

Advanced Features dialog box opens.

n

3. Deselect Surge Protection.

4. Click OK. The dialog box closes.

Disabling Surge Protection for a Service in the

Configuration Utility
Use the following procedure to disable Surge Protection for a specific service:
1. Go to Traffic Management > Load Balancing > Services.

216 Module 7: Queuing and Connection Tuning © Copyright 2016 Citrix Systems, Inc.
2. Select the service to disable surge protection and click Edit. The Load Balancing Service dialog
box opens.
3. Click the Edit Icon next to Settings.
4. Deselect Surge Protection in the Settings section.
5. Click OK, then click Done. The Load Balancing Service dialog box closes.

Surge protection only works when both the feature and the service settings are
enabled.

Setting Thresholds in the Configuration Utility

Use the following procedure to set thresholds for surge protection:
N

1. Go to System > Settings.

ot

2. Click Change global system settings in the Settings section. A dialog box opens.
fo

3. Manually specify the maximum number or concurrent server connections allowed before surge
protection is triggered in the Base Threshold field (200 default).
rr
es

The base threshold is the maximum number of server connections that can be open
before surge protection is activated. The maximum value for this setting is 32,767
al

server connections.
e

4. Select a Throttle rate in the Throttle drop-down list.

or

5. Click OK. The dialog box closes.

di
s

Priority Queuing
tri
bu

The priority queuing feature filters incoming HTTP traffic based on categories that are created and
defined. Priority queuing directs high-priority requests to the server ahead of low-priority requests,
t

ensuring that users who need resources for defined uses receive expedited access to protected Web
io

servers.
n

Priority queuing policies specify a priority, weight, threshold and implicit action. When an
incoming request matches a priority queuing policy, the request is processed as the associated
action indicates.
An administrator can bind up to three priority queuing policies to a single load balancing virtual
server. The priority levels are:

© Copyright 2016 Citrix Systems, Inc. Module 7: Queuing and Connection Tuning 217
Level Definition
Level 1 A Level 1 policy processes priority requests. A
weight can range from 0 to 101, where 101
represents infinite weight.

Level 2 A Level 2 policy processes requests that receive

responses as soon as priority requests have been
cleared from the queue.

Level 3 A Level 3 policy processes non-priority requests

that receive responses only after requests in the
first two queues have been cleared.
N

A unique name must be assigned to each priority queuing policy. Multiple policies bound to the
same load balancing virtual server cannot have the same priority level. No two virtual servers that
ot

have one or more common underlying physical services can have priority queuing configured on
both virtual servers simultaneously.
fo
rr

Enabling Priority Queuing in the Configuration Utility

es
al

Use the following procedure to enable priority queuing:

e

1. Go to System > Settings.

or

2. Click Change advanced features under the Modes and Features section. The Configure
Advanced Features dialog box opens.
di

3. Select Priority Queuing.

s

4. Click OK. The dialog box closes.

tri
bu

Creating a Priority Queuing Policy in the Configuration

t io

Utility
n

Use the following procedure to create a priority queuing policy:

1. Go to Security > Protection Features > Priority Queuing.
2. Click Add. The Create PQ Policy dialog box opens.
3. Enter a policy name in the Policy Name field.
4. Enter a policy expression in the Rule field or click Expression Editor to create a policy
expression. Perform the following steps to create a new policy expression:
a. Click Expression Editor. The Add Expression dialog box opens.
b. Select an expression type in the Select Expression Type list.

218 Module 7: Queuing and Connection Tuning © Copyright 2016 Citrix Systems, Inc.
 c. Select a flow type, protocol, qualifier and operator from the appropriate lists.
d. Type a value in the Value field and click Done. The Add Expression dialog closes.
5. Enter the priority value in the Priority field.
6. Enter the weight in the Weight field.
7. Enter a queue depth value in the Queue Depth field or a policy queue depth value in the Policy
Queue Depth field.

The queue depth defines the total number of waiting clients or requests on the virtual
server to which the policy is bound. The policy queue depth defines the number of
waiting clients or requests belonging to the policy.

8. Click Create. The Create PQ Policy dialog closes.

Binding Priority Queuing Policies in the Configuration Utility

N
ot

Use the following procedure to bind priority queuing policies:

fo

1. Go to Traffic Management > Load Balancing > Virtual Servers.

rr

2. Select the virtual server that will be bound to the priority queuing policy and click Edit. The
Load Balancing Virtual Server dialog opens.
es

3. Click the Policies node.

al

4. Click the + sign next to Policies.

e

5. Select Priority Queue from the Choose Policy drop-down list.

or

6. The Choose Type should be Request then click Continue.

7. Click in the field below Select Policy.
di

8. Select the PQ Policy. then click Select.

s tri

9. Click Bind.
bu

10. Click Done. The Load Balancing Virtual Server dialog closes.
t io

Weighted Queuing
n

With priority queuing set, lower-priority requests are often kept on hold while higher-priority
requests are served. Therefore, the lower-priority requests may be delayed if there is a constant flow
of higher-priority requests.
To prevent delays for low-priority requests across multiple priority levels, an administrator can
configure weighted queuing for serving requests. The default weights for the priorities are:

Level Definition
Gold Priority 1, Weight 3

© Copyright 2016 Citrix Systems, Inc. Module 7: Queuing and Connection Tuning 219
Level Definition
Silver Priority 2, Weight 2

Bronze Priority 3, Weight 1

An administrator should assign zero as the minimum weight to requests that are sent to the server
if no requests are stored in the other queues. An administrator should assign 101 as the maximum
weight to requests that are sent to the server immediately, ahead of any requests stored in the other
queues. Weights between these weights set the relative priority of a particular queue in relation to
the other queues. Queues with a higher weight are processed first.

The weight assigned to a higher-priority queue must be larger than the weight assigned to
a lower-priority queue. For example, the weight assigned to the Gold (Priority 1) queue
must be greater than the weight assigned to the Silver (Priority 2) queue.
N
ot
fo

HTTP Denial-of-Service Protection

rr

HTTP denial of service protection feature provides an effective way to prevent HTTP-level attacks
es

from being relayed to a server. This feature also ensures that a NetScaler system located between
the Internet cloud and the servers is not brought down by an attack while protecting the servers.
al

When this feature is enabled, the NetScaler monitors the number of incoming connections. Once
e

the queue hits a specified depth, the NetScaler determines it is under attack. When the NetScaler
or

system detects an attack, it responds to incoming requests based on the value of the client detect
rate parameter with a JavaScript containing a simple refresh and cookie. Real clients parse the
di

request and return it with the cookie, and false client responses are dropped.
s

When a POST request is received, it is checked for a valid cookie. If the request has a valid cookie,
tri

then the request goes forward. If the request does not contain a valid cookie, then the NetScaler
bu

system sends a JavaScript to the client asking it to resend the information with a new cookie. If the
client sends a new cookie, then the cookie becomes invalid after four minutes, and every response
t

to the client is sent with the new cookie.

io

The cookie in a legitimate POST request may become invalid under the following conditions:
n

• The GET request is made when the NetScaler is not under attack and the POST request is
made when the NetScaler system is under attack.
• The client processing time exceeds four minutes.
Although both of the scenarios are rare, they are not impossible. In addition, the layer 7 DoS
protection feature has the following limitations:
• Under an attack, all POST requests are dropped, and an error page with a cookie is sent.
• Under an attack, all embedded objects without a cookie are dropped, and an error page with a
cookie is sent.

220 Module 7: Queuing and Connection Tuning © Copyright 2016 Citrix Systems, Inc.
The HTTP DoS protection feature may affect other NetScaler features. Using DoS protection for a
particular content switching policy creates additional overhead because the policy engine must find
the matching policy. SSL requests incur overhead from data decryption. However, because most
DoS attacks are not on a secure network, the attacks are less aggressive. Under an attack, requests
without proper cookies are placed in a low priority queue. Although queuing incurs overhead, the
Web servers are protected from false clients.
With HTTP DoS protection, a minimal effect on throughput exists because the test JavaScript is
sent to the client at a slow rate. The default value is one percent of the server HTTP response rate
and can be changed by tuning the client detect rate setting. The latency of requests is also increased
because the client reissues the request, which is queued, after it receives the JavaScript.

Enabling HTTP DoS Protection in the Configuration Utility

Use the following procedure to enable HTTP DoS Protection:
N

1. Go to System > Settings.

ot

2. Click Change advanced features in the Modes and Features section. The Configure Advanced
fo

Features dialog opens.

rr

3. Select HTTP Dos Protection.

4. Click OK. The Configure Advanced Features dialog closes.
es
al

Adding a HTTP DoS Policy in the Configuration Utility

e
or

Use the following procedure to add a HTTP DoS policy:

1. Go to Security > Protection Features > HTTP DoS.
di

2. Click Add. The Create HTTP DoS Policy dialog opens.

s tri

3. Type a name in the Name field.

bu

4. Type the queue size in the Queue Depth field.

5. Type the client detection rate in the Client Detect Rate field.
t io

This value is the percentage of traffic to which the HTTP DoS policy is applied.
n

6. Click Create.

Challenged JavaScript Responses

If more than 200 clients are waiting in the NetScaler surge queue for a service with a maximum
client value of 200, then the HTTP DoS protection function is triggered. The default rate of
challenged JavaScript responses sent to the client is one percent of the server response rate.

© Copyright 2016 Citrix Systems, Inc. Module 7: Queuing and Connection Tuning 221
For example, if the server generates 100 responses per second and there are 200 clients in the surge
queue, then the HTTP DoS protection feature chooses one pending client request per second from
the surge queue (1% of 100) and sends a challenge JavaScript response to the suspect client at the
rate of one JavaScript per second.
If the client runs the received challenge JavaScript, generates the cookie and resends the original
HTTP request with the JavaScript-generated cookie. Sending the cookie demonstrates that it is a
legitimate browser-based client.
The HTTP DoS protection feature queues the HTTP requests of the client in its higher-priority
legitimate client queue to be served faster.

Client Detection Tuning and JavaScript Challenge

Response Rate
N

The default client detection and JavaScript challenge response rate of one percent is inadequate in
ot

many real attack scenarios and therefore needs to be tuned.

For example, there are 500 responses/sec and sent from the server, and the server is under attack
fo

with 10,000 GETs/second. If one percent of the server responses are sent as JavaScript challenges,
rr

then the responses are reduced to almost none. With five client (500 x 0.01) JavaScript responses
for 10,000 waiting client requests, approximately 0.05% of the real clients receive JavaScript
es

challenge responses.
al

If the client detection and JavaScript challenge response rate is very high (for example, 10%,
e

generating 1,000 challenge JavaScript responses per second), it may saturate the upstream links or
harm the upstream network devices. Citrix recommends to carefully choose client detect rate
or

values.
di

If the configured triggering surge queue depth is, for example, 200, and the surge queue size is
s

fluctuating between 199 and 200, then the NetScaler system switches between the attack and no-
tri

attack scenario, which is undesirable.

bu

To prevent the attack and no attack scenario from occurring, a window mechanism is provided.
When the surge queue size reaches 200 and the attack scenario is detected, then the surge queue
t

size must fall for the NetScaler system to enter no-attack mode. If the value of WINDOW_SIZE is
io

set to 20, then the surge queue size must be under 180 before the NetScaler system enters no-attack
n

mode. During configuration, an administrator must specify a value that is greater than the
WINDOW_SIZE for the QDepth parameter when adding or setting a DoS policy. The triggering
surge queue depth should be configured based on prior knowledge of the traffic characteristics.

HTTP DoS Protection Deployment Guidelines

A best practice is to deploy the HTTP DoS protection feature in a tested and planned manner, as
well as to closely monitor what occurs after initial deployment. The following considerations should
be used to fine-tune the deployment of HTTP DoS Protection:
• The maximum number of concurrent connections supported by the servers

222 Module 7: Queuing and Connection Tuning © Copyright 2016 Citrix Systems, Inc.
• The average and normal values of the concurrent connections supported by the servers
• The maximum output rate (responses/second) that the server generates
• The average traffic that the server handles
• The network bandwidth
• The maximum bandwidth available upstream
• The limits faced by the bandwidth, for example, external links, routers and the critical devices
on the path that may suffer from a traffic surge
• The decision whether allowing a greater number of clients to connect is more important than
protecting upstream network devices

Attack Characteristics
An administrator should consider the following questions when fighting DoS attacks:
N
ot

• What is the rate of incoming fake requests that have occurred in the past?
• What types of requests have been received, such as complete posts and incomplete gets?
fo

• Did previous attacks saturate downstream links? If not, what was the bandwidth?
rr

• What types of source IP addresses and source ports did the HTTP requests arrive from, such as
es

IP addresses from one subnet, constant IP addresses and ports increasing by one?
• What types of attacks have occurred in the past and what types of attacks are anticipated for
al

the future?
e

• What other information can be gathered to tune DoS attack protection?

or
di
s tri
bu
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 7: Queuing and Connection Tuning 223
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

224 © Copyright 2016 Citrix Systems, Inc.

 8
Module 8

Authentication,
Authorization, and
Auditing
N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

226 © Copyright 2016 Citrix Systems, Inc.

Authentication, Authorization, and Auditing
Manual
Overview
Most networks concentrate their user credentials in one centralized location. This aids in
management and security. The NetScaler system can use common authentication, authorization,
and auditing (AAA) systems for its system users. AAA can also be applied to traffic passing
through it.
AAA for Application Traffic uses authentication virtual servers to provide AAA functionality for
load balancing and content switching traffic. This allows the NetScaler to perform authentication,
authorization, auditing functionality in front of traffic management virtual servers. This gives
N

administrators the ability to provide single sign-on, access control, session, and traffic policy
ot

capabilities for non-VPN traffic. AAA for Application Traffic uses the NetScaler to manage access
requirements for multiple web sites without needing full VPN style connectivity.
fo

AAA for Application Traffic uses many of the policy types and design concepts as the SSLVPN
rr

functionality, but streamlined for access control only.

es

Objectives
al
e

At the end of this module you will be able to:

or

• Identify users, groups and command policies.

di

• Configure external authentication for system users.

s

• Configure AAA for application traffic.

tri

• Configure logging and auditing.

bu
t

Users, Groups, and Command Policies

io
n

Users, groups, and policies involve:

• Authentication: Who is allowed access
• Authorization: What resources the user is allowed or denied access
• Auditing: The log and record of who is allowed or denied access to what resources

Authentication, Authorization, and Auditing

Authentication determines who can log on to the system, and authorization determines to which
resources an administrator has access. The two processes are used together to manage and control

© Copyright 2016 Citrix Systems, Inc. Module 8: Authentication, Authorization, and Auditing 227
access to environments and resources. Auditing is the built-in mechanism for logging the
authentication, authorization and access records.
NetScaler provides separate authentication/authorization processes for NetScaler system access and
AAA user access across a user entry point like an SSL VPN virtual server or an authentication
virtual server. System access controls determine what administrators can access the management IP
addresses of the NetScaler and which commands are authorized to run against the system
configuration. AAA access determines which users are allowed to log on through a SSL VPN or
authentication virtual server and which resources a user can access across these entry points. While
the mechanisms are similar, system and AAA access is managed separately on the NetScaler.
Authentication. Authentication identifies which accounts are allowed to log on. Authentication
alone does not grant users access to resources and may only determine whether the user has logged
on successfully. On the NetScaler system, authentication can be based on locally defined users and
groups or on external authentication services such as LDAP (Active Directory), TACACS+,
RADIUS.
N

The authentication behavior is managed through authentication policies. Authentication policies are
ot

used to determine whether local accounts are used or whether the authentication process is
integrated with specified external authentication services. This usage provides administrators the
fo

flexibility on managing accounts and whether or not to use their existing directory services within
their environment.
rr

Authorization. Authorization identifies to which resources authenticated users are allowed or

es

denied access. Authorization policies allow administrators to control access to resources based on
account or group. Authorization behavior is managed through authorization policies. The only
al

authorization actions are allow and deny. Administrators define the global authorization action and
e

then use policies applied to specific users or groups to override the global authorization behavior.
or

As a result, Citrix recommends to have the global authorization action set to deny and to have
additional policies created for allowed resource access.
di

Auditing. Auditing tracks and logs authentication and authorization activity results. With logs,
s

administrators determine which accounts are denied when attempting to access an environment or
tri

a resource and which accounts are accessing resources successfully. Also logged are the accessed
bu

resources. All authentication and authorization results, as well as resource access, are logged in the
/var/log/ns.log NetScaler syslog file.
t io
n

System and AAA User Groups

System users and groups access the NetScaler system for management and configuration. AAA
users and groups access the NetScaler system to own and control resources, including:
• SSL VPN authentication and VPN resources (NetScaler Gateway)
• Virtual servers
The NetScaler system manages two types of users and groups:
• System users and groups. These users and groups are used for logging into the system directly
for NetScaler management functions through the web-based NetScaler Configuration Utility or
the NetScaler command-line interface (using SSH). Command policies can be defined which

228 Module 8: Authentication, Authorization, and Auditing © Copyright 2016 Citrix Systems, Inc.
 determine the level of permissions each account or group contains when performing
management functions. These policies are command specs or command policies, for example:
• Read-only
• Operators
• Network
• Superuser
• AAA users and groups. The AAA users and groups are a set of accounts that are defined on
the NetScaler system and are used to access the managed and controlled resources through
both VPN virtual servers and authentication virtual servers. Policies are defined to determine
which resources the accounts can access. AAA accounts do not have management access to the
NetScaler system.
Traditionally, AAA accounts were used in conjunction with the SSL VPN/NetScaler Gateway
configuration and were used solely for logging into and accessing SSL VPN resources allowed by
the policies. Additional policies - including session, network, and traffic policies - are defined to
N

further manage the SSL VPN account behavior.

ot

AAA for application traffic can also use AAA users and groups to manage resource access that is
fo

not part of an SSL VPN configuration. The functionality to use AAA for application traffic allows
an administrator to configure the NetScaler system to use AAA accounts to manage access to web
rr

sites and web resources. AAA for application traffic allows an administrator to use the same AAA
es

capabilities that are available with a VPN implementation to provide authentication to resources
and to manage allowed and denied authorizations to those resources for non-SSL VPN scenarios.
al

For example, an administrator has three websites that are accessible to users:
e

• corp.example.com
or

• salesportal.example.com
di

• info.example.com
s

The administrator wants to provide added security and access control to the websites by requiring
tri

users to authenticate before accessing the sites. The sites themselves do not provide this
bu

functionality, so the administrator is going to use the AAA for application traffic feature to
frontend the site. When a user accesses any of the three sites, they will be redirected to a logon
t io

page hosted at
n

• secureaccess.ecample.com
Once the user authenticates, the administrator can then control which sites the user is authorized
or prevented from accessing. The administrator can use authorization policies to allow or deny
access to the entire sites: corp, salesportal, or info. Or the policies can be used to restrict access to
certain paths within the site. Such as:
• corp.example.com/techsupport
• corp.example.com/hr
• corp.example.com/store
In this example, the NetScaler AAA for application traffic feature provides this functionality
without configuring a VPN.

© Copyright 2016 Citrix Systems, Inc. Module 8: Authentication, Authorization, and Auditing 229
Local Accounts
For both system and AAA accounts, users and groups are created and maintained as local accounts
on the NetScaler system. User accounts, passwords and group membership are set and managed on
the NetScaler system. If local accounts are used for authentication, then the local users and groups
must be created and maintained individually on each NetScaler system.
Each NetScaler system has two local system accounts that are always maintained as local accounts:
• nsroot. This account is the default administrative account for the NetScaler system and cannot
be disabled or removed from the system. The account has superuser permissions. A best
practice is to change the default account password after initial NetScaler appliance setup.
• #nsinternal#. This account is used for GSLB and high availability communications through the
rpc nodes. The command set rpcnode implicitly uses the #nsinternal# account. Its password
can be changed by adjusting the password of the rpc nodes.
N

External Authentication
ot
fo

The NetScaler system integrates with the following external authentication and directory services
systems:
rr

• LDAP
es

• RADIUS
al

• TACACS+
e

• SAML
or

With external authentication, the NetScaler system uses the existing, accounts and passwords and
groups managed within the directory services infrastructure to authenticate users to the
di

environment.
s tri

An administrator defines authentication policies and actions that identify which directory service
resource to connect when performing external authentication. The NetScaler system uses the
bu

external directory service to verify that the account and password supplied is valid in the directory
service and to identify the group membership. The results are then returned to the NetScaler
t io

system, and the appropriate authentication and authorization decisions are made.
n

When using external authentication, administrators can simplify account management by using
group extraction. With this functionality, the administrator only defines the groups on the
NetScaler system. The group names added to the NetScaler system, either as system groups or AAA
groups, exactly match the group names and case within the external directory service. The
individual user accounts do not need to be created or maintained on the NetScaler system. Instead,
when a user account is supplied for authentication, the external authentication configuration relies
on group extraction to identify to which group the user account belongs. The group memberships
correspond to the directory service configuration. Authentication and authorization policies and
permissions are managed and assigned at the group level.

230 Module 8: Authentication, Authorization, and Auditing © Copyright 2016 Citrix Systems, Inc.
External Authentication for System Users
While both system and AAA can use the same authentication policy types, authentication in use
can be configured independently for local and system accounts. Local and external authentication
can be configured and managed separately for system and AAA accounts. The authentication type
used for one does not have to be the same methods in use for the other. For example:
• System authentication could be configured to use local credentials while AAA authentication
could be configured to use LDAP (Active Directory)
• System authentication could be configured to use TACACS+ while AAA authentication uses
two-factor authentication with both LDAP and RADIUS.
System authentication manages administrative access to NetScaler systems. Authentication policies
are bound to the global system object. Authorization is handled across command policies
(command specs) bound to system groups and systems user accounts.
For SSL VPN vservers, authentication policies can be bound to the Global VPN object or individual
N

VPN virtual servers. Authorization policies are then used to determine resource access.
ot

Similarly, AAA for application traffic authentication is controlled by binding authentication policies
fo

to authentication virtual servers.

rr

It is important to note that system authentication is single-factor only. AAA authentication can be
configured using single-factor, two-factor authentication, or three-factor (with SAML
es

authentication).
al
e

Authentication Actions and Policies

or

Like NetScaler classic policies, authentication policies are composed of an expression and an action.
di

Each directory services and authentication options has different authentication policy and action
s

types. The authentication actions include the information required to perform the authentication
tri

behavior and are also referred to as authentication servers. The authentication policy determines
when the policy matches based on the defined expression. The policy is then bound to global
bu

objects or virtual servers on the NetScaler system.

t

The NetScaler system defines the policies and actions:

io
n

Policy Action
localPolicy no associated action

ldapPolicy ldapAction

radiusPolicy radiusAction

tacacsPolicy tacacsAction

© Copyright 2016 Citrix Systems, Inc. Module 8: Authentication, Authorization, and Auditing 231
Policy Action
samlPolicy samlAction

Configuring Local Authentication

When configuring the default local authentication, an administrator must create a system user
account and group on the NetScaler system. One or more user accounts must be bound to each
group. Permissions can then be managed at the group level.
For system access, the appropriate command policies must be bound to either the system user
accounts or the system groups. Command policies determine system management permissions
levels, which are superuser, network, operator, and read-only. Custom permissions may also be
configured.
N
ot

Configuring External Authentication with Explicit Accounts

fo
rr

This example demonstrates the configuration of external authentication using LDAP and explicit
accounts. Group extraction is not used. For this scenario, LDAP is an Active Directory
es

implementation.
al

1. Accounts and groups are configured within the LDAP directory service.
e

• AD domain accounts and groups are created.

or

• Accounts are made members of the appropriate groups within the domain.
2. System groups are created on the NetScaler system. The names of the groups are case-sensitive
di

and must match the group names within LDAP.

s

3. System users are created that correspond to the explicit LDAP user accounts that are allowed to
tri

access the NetScaler system. Passwords are not set on the system users because the credentials
bu

in the external authentication system should be used.

4. The system users are bound to the appropriate system groups. The relationships must match
t io

the LDAP configuration.

n

5. The appropriate command policy is bound to either the system user accounts or the system
groups. Command policies determine system management permission levels, which are
superuser, network, operator and read-only.
6. An authentication ldapAction is created. This entity defines all the necessary components
needed to connect to the LDAP infrastructure to authenticate the user account. Information
required for the ldapAction includes:
• LDAP server IP address and LDAP port
• LDAP base DN
For the domain example.com, the base DN is "DC=example,DC=com"
• LDAP bind DN and password

232 Module 8: Authentication, Authorization, and Auditing © Copyright 2016 Citrix Systems, Inc.
 A user account and password with permissions to browse the LDAP directory
• Other required LDAP settings
7. An authentication ldapPolicy is created. The policy includes an expression and the ldapAction.
For this example, because the authentication policy is bound globally, the expression can be
based on "true value" (ns_true), meaning that the policy always evaluates to true. Depending on
the environment, an actual expression may be required to determine when to authenticate
using this policy as opposed to an alternate policy.
8. The authentication policy is bound to the global object or to the appropriate system user,
group or entity.
When configuring external authentication without group extraction, an administrator must create
and maintain both the group and user objects on the NetScaler system. External authentication
without group extraction requires maintenance on the NetScaler system as accounts are added,
removed and modified within LDAP. With group extraction, only the group accounts must be
created and maintained on the NetScaler system.
N
ot

Configuring External Authentication with Group Extraction

fo
rr

The procedure for configuring external authentication with group extraction is similar to
configuring external authentication with explicit accounts, except that user accounts are not created
es

or bound to the groups. Additionally, other policies are bound only at the system group level.
al

For successful group extraction, an administrator must identify the groups needed and create only
those groups on the NetScaler system, taking into account that the group name in the directory
e

service is case-sensitive.
or

With group extraction, the NetScaler system benefits from all group and account management
occurring within the directory service. For example, dedicated groups can be created for NetScaler
di

administrators within LDAP. The NetScaler system is configured with the corresponding system
s

group NetScaler Admins. Command policies are set at the group level, and LDAP authentication is
tri

enabled for system authentication. As users are added or removed from the NetScaler Admins
bu

group, the NetScaler configuration does not have to be modified.

External authentication with group extraction is supported with LDAP and Radius authentication
t io

sources for both system and AAA authentication.

n

Creating an External Authentication Policy

Use the following procedure to create an authentication policy:
1. Create system groups on the NetScaler.
Do not include user accounts in the group. Group names must exactly match group names in
authentication source (case-sensitive).
2. Bind the appropriate command policies to the system group.
3. Create the authentication action for the authentication source.

© Copyright 2016 Citrix Systems, Inc. Module 8: Authentication, Authorization, and Auditing 233
4. Create the authentication policy.
5. Bind the policy to the System Global bind point.

Creating local groups in the Command-Line Interface

Enter the following command to create a local group:

add system group group_name

Replace group_name with the name of the group

Group names must correspond to the group in the directory service. Groups are case sensitive.

Binding Groups in the Command-Line Interface

N
ot

Enter the following command to bind a group:

fo

bind system group group_name -policyName cmd_policypriority

rr
es

Arguments include:
al

Argument Definition
e
or

group_name The group name

di

cmd_policy The command policy to which the group is

s

bound
tri

priority The priority of the policy after binding

bu
t io

Command Policies
n

For system accounts, NetScaler configuration and delegated administration is managed by

command policies. Command policies determine which NetScaler commands an account is allowed
to run. Command policies govern all allowed CLI and therefore GUI commands permitted to a
system account.
A global default "deny -all" policy exists for system administration. Unless an explicit command
policy is bound to a system group or system user account granting permissions to run a command,
the command is denied based on the default settings.
Command policies can be bound to individual system user accounts or to a system group.
Command policy precedence, follows the following guidelines:

234 Module 8: Authentication, Authorization, and Auditing © Copyright 2016 Citrix Systems, Inc.
• Priority determine which policy is more important if multiple policies are applied. If policies
have the same priority, than the user policy is more important than the group policy.
• Priority 1 (lower number) is more important (higher priority) than a policy with a higher
number (lower priority). Policy with priority 1 is more important than a policy with priority
10. Priority 10 is more important than priority 100.
If a user account belongs to multiple groups, the user may receive the allowed permissions from
both groups. Permissions with command policies are additive. If a policy says to allow a command
and another policy says to deny a command, then whichever policy is higher precedence, will
determine whether the command is allowed or denied.

Example 1
Using the built-in policies for superuser and operator, this example demonstrates the effect of
command policies when a user is a member of multiple groups.
N
ot

• NSAdmins: Superuser (priority 20)

• NSOperators: Operator (priority 10)
fo
rr

add system group NSAdmins

es

add system group NSOperators

al

add system user johndoe

e
or

bind system group NSAdmins -username johndoe -

policyName superuser 20
di
s

bind system group NSOperators -username johndoe -

tri

policyName operator 10
bu

User account johndoe is a member of both NSAdmins and NSOperators groups. Even with the
priorities listed, johndoe will retain superuser permissions. Because there is no deny command
t io

encountered, when johndoe attempts to run a command like save ns config, the command is
first compared against the Operator policy where no command match is found. Then the command
n

is compared against the lower priority superuser policy where the command is allowed. Therefore,
the command is successful.

Example 2
In this example, two new command policies have been created for a custom LB Administrator. It is
based on the built-in Operator policy, so it grants read-only access to almost all entities, but then
allows the administrator to modify server, services, and lb vserver entities.

© Copyright 2016 Citrix Systems, Inc. Module 8: Authentication, Authorization, and Auditing 235
 add system cmdPolicy custom_lbadmin ALLOW
"(^man.*)|(^show\\s+(?!system)(?!configstatus)(?!ns
ns\\.conf)(?!ns savedconfig)(?!ns runningConfig)(?!gslb
runningConfig)(?!audit
messages)(?!techsupport).*)|(^stat.*)|(^(add|rm|enable|disable|set
|unset|bind|ubind) (server|service|(lb vserver)).*)"

A second policy was created to explicitly prevent access to certain lb vservers by creating a policy
with a DENY action (as opposed to allow).

add system cmdPolicy custom_lbadmin_noRBG DENY

"(^(add|rm|enable|disable|set|unset|bind|ubind)
(server|service|(lb vserver)) lb_vsrv_rbg.*)"

The user janedoe is a member of the LBAdmins group.

N

• LBAdmins: custom_lbadmin (priority 10) - Uses an ALLOW action.

ot

• LBAdmins: custom_lbadmin_norbg (priority 20) - Uses a DENY action.

fo

bind system group lbadmins -userName janedoe

rr
es

bind system group lbadmins -policyName custom_lbadmin 10

al

bind system group lbadmins -policyName custom_lbadmin_noRBG 20

e

In this configuration, the allow commands take precedence and an attempt for janedoe to modify
or

lb_vsrv_rbg will succeed.

di

If the priorities are reversed, then the deny action will prevent janedoe from modifying lb_vsrv_rbg,
s

but allow creation of other lb vserver entities.

tri
bu

Creating an LDAP Authentication Action in the Command-

t

Line Interface
io
n

Enter the following command to create an LDAP authentication action:

add authentication ldapAction action_name

Replace action_name with the name of the action.

The following argument list explains the LDAP parameters to be configured in the add ldapAction
command below:

236 Module 8: Authentication, Authorization, and Auditing © Copyright 2016 Citrix Systems, Inc.
Argument Definition
Server IP address The IP address of the LDAP server.

Server port The port for LDAP. The default value is 389.

LDAP base The LDAP string for base domain

LDAP bind DN The account with permissions to browse the

LDAP DS

LDAP bind DN password The password for the LDAP bind DN account

LDAP logon name The sAMAccountName value with the case as

indicated
N
ot

For example:
fo

add authentication ldapAction authe_ldap_action -

rr

serverIP 10.29.0.20 -serverPort 389 -

ldapBase "DC=Education,DC=ctx" -
es

ldapBindDN "[email protected]" -
al

ldapBindDNPassword Password1 -ldapLoginName sAMAccountName -

groupAttrName memberOf -subAttributeName CN
e
or

Creating an Authentication Policy in the Command-Line

di

Interface
s tri

Enter the following command to create an authentication policy:

bu
t

add authentication ldapPolicy policy_name ns_true action_name

io
n

Arguments include:

Argument Definition
policy_name The name of the policy

action_name The name of the action

© Copyright 2016 Citrix Systems, Inc. Module 8: Authentication, Authorization, and Auditing 237
Binding the Policy in the Command Line Interface
Enter the following command to bind the policy:

bind system global policy_name -priority priority

Arguments include:

Argument Definition
policy_name The name of the policy

priority The priority of the policy after binding

N
ot

Authentication Troubleshooting
fo

To troubleshoot issues with authentication, view the following pipe and log:
rr

• aaad.debug. The aaad.debug is a named pipe which displays the authentication process and
any success or error messages that occur. The named pipe is viewed like a file handle and can
es

be useful when troubleshooting authentication issues. The pipe is located in

al

/tmp/aaad.debug.
e

The contents of the named pipe for a set amount of time are displayed. The cat command
times out, so an administrator may need to stop and repeat the command if performing
or

extensive testing.
di

• auth.log. The auth.log file includes a log of system authentication events that have occurred
s

on the NetScaler system. The information in this log file is useful when identifying issues with
tri

rpcnode password conflicts. The file is located in /var/log/auth.log.

bu

External Authentication Common Issues

t io
n

Common mistakes related to external authentication include:

• Group names on the NetScaler system that do not match the case and spelling of the groups in
the directory service
• User accounts that are not members of the appropriate groups in the directory service
• Command policies that are not bound to the local groups and should be verified with the
following command:
show system group groupname

When external authentication is not working due to issues with authentication parameter settings,
the following methods can be used to troubleshoot:

238 Module 8: Authentication, Authorization, and Auditing © Copyright 2016 Citrix Systems, Inc.
• Create a trace of the authentication traffic between the NetScaler system and the external
authentication system and view in Wireshark.
• View the authentication process by monitoring aaad.debug. The aaad.debug is a file handle,
and the output can be viewed in the BSD shell with the following command:
cat /tmp/aaad.debug

The output of the aaad.debug file should be monitored while attempting authentication
through a second NetScaler command-line interface session.

AAA for Application Traffic

AAA is available for application traffic management. The feature is also referred to as AAA for
Traffic Management.
N
ot

Enabling AAA for Application Traffic

fo

To enable AAA for Application Traffic, enable the feature Authentication, Authorization and
rr

Auditing in the features node or enable AAA - Application Traffic under the Security node.
es

Remember that the SSL Offload feature must be enabled along with Load Balancing or Content
Switching. Caution should be used when configuring both NetScaler Gateway and AAA for
al

Application Traffic on the same system as some settings may overlap between the two features
e

(such as the global authorization action).

or
di
s tri
bu
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 8: Authentication, Authorization, and Auditing 239
 N
ot
fo
rr
es
al

AAA for application traffic is also referred to as AAA for traffic management. The figure shows
e

where the feature is enabled in the Configuration Utility. The authentication, authorization and
or

auditing (AAA) feature of the NetScaler system is a security model that allows users accessing
traffic management resources (load balancing and content switching virtual servers) to establish a
di

session and authenticate over a secure channel (SSL). The NetScaler system provides authentication
s

for traffic management virtual servers that are associated with the authentication virtual server. An
tri

authentication virtual server enables the NetScaler system to authenticate the client that accesses the
traffic management virtual server. The appliance also uses the auditing framework to log
bu

application traffic.
t io
n

AAA for Application Traffic

The authentication virtual server provides authentication capabilities to traffic management virtual
servers using HTTPS. It supports SSL on port 443 and authenticates users that access the load
balancing virtual server using a standard authentication server that is external to the NetScaler
system. The authentication virtual server can be configured from the Security > AAA - Application
Traffic > Policies > Authentication node. An authentication LDAP action can be added in the
NetScaler command-line interface.
To control access to the web site, the NetScaler system divides the authentication traffic from the
application traffic to the web site, which allows the sharing of authentication information for the
web sites in that domain.

240 Module 8: Authentication, Authorization, and Auditing © Copyright 2016 Citrix Systems, Inc.
AAA for application traffic provides security for a distributed Internet environment.
For example, an organization has the web site https://employee.example.com for its employees. The
content on this web site is confidential and requires secure access. Any user who accesses the web
site must have a valid user name and password. When a user first visits the site, the NetScaler
system sends the logon page on which the user must enter credentials. The NetScaler system
validates the user credentials locally or using external authentication servers, such as LDAP and
RADIUS. After the user credentials are validated, the NetScaler system allows the user access to the
web site. In this configuration, the NetScaler system performs the following actions:
• Collects the user credentials over a secure HTTPS connection at
https://secure.employee.example.com
• Maintains a session timeout that can be configured for keeping sessions active for a configured
time
• Creates audit logs with user access information, including invalid logon attempts
N

The authentication and origin web sites must be in the same domain because the NetScaler system
maintains a user session in a cookie that the authentication virtual server issues. If the two web sites
ot

are in different domains, then the browser cannot return the cookie on subsequent requests.
fo
rr

Basic AAA Setup for Application Traffic

es

AAA for application traffic:

al

• Redirects the user to the authentication virtual server and prompts the user for a username and
e

password
or

• Includes the creation of authorization policies that specify allow (default) or deny actions using
the AppExpert classic policy engine
di

• Includes the example "REQ.HTTP.URL == /admin/*"

s

• Has authorization policies that can be bound to AAA groups

tri

• Includes the creation of local users that are added to groups on the NetScaler appliance
bu

• Has customers use an external authentication provider and extracts the user groups upon log
t

on
io

Configuring a basic but functional AAA setup for application traffic involves enabling AAA,
n

creating an authentication virtual server, and binding certificates. Then, the domain with the
authentication virtual server must be configured and a load balancing or content switching virtual
server must be set up and configured, serving as a base for various configuration tasks.

© Copyright 2016 Citrix Systems, Inc. Module 8: Authentication, Authorization, and Auditing 241
Workflow for AAA Application Traffic

N
ot
fo
rr
es
al
e
or
di

The figure shows the workflow for AAA Application Traffic management.
s tri

1. The user makes an HTTP request to a load-balancing virtual server which is configured for
bu

AAA.
2. The load-balancing virtual server responds with a form. The body content is similar to:
t io

<html><head><META HTTP-EQUIV="Content-
n

Type" CONTENT="text/html; charset=UTF-

8"></head><body onLoad='document.forms[0].submit();'><form
action="https://AuthVServer.example.com/cgi/tm"
method="post"><input type=hidden name=loc
value="http://lbvserver.example.com/tmindex">If you are not
automatically redirected click <input type="submit"
value="Continue"></form></body></html>
3. The result is a POST by the browser of the user to the
https://AuthVServer.example.com/cgi/tm URL.

242 Module 8: Authentication, Authorization, and Auditing © Copyright 2016 Citrix Systems, Inc.
4. The NetScaler system replies to this post with 302 to the logon page, which is the
http://lbvserver.example.com/tmindex.html web site.
5. The client browser GETS tmindex.html, which is the logon page.
6. The 200 response occurs and a logon page appears that is used to submit credentials.
7. The user submits credentials.
8. The NetScaler system contacts the authentication server if it is external or checks the
credentials itself if it is local.
9. The NetScaler system performs group extraction if it succeeds, and based upon the
authorization polices, it allows or denies access.
10. A 302 redirect is sent to the browser to the original URL that was requested with SET-
COOKIE headers if the request is allowed. These cookies are sent to the browser and allow the
user through the protected load-balancing virtual server. If the request is not allowed, then a
403 error with a Connection: Close header is sent and if configured, the NetScaler system
custom SERVER header is sent as configured by:
N
ot

set ns httpParam -insNSSrvrHdr ON "NS9.0-67.1"

This header is useful when receiving 403 errors because is it from the NetScaler system or from
fo

the back-end server itself.

rr

11. The user makes a request to the original URL that it requested, as sent in the previous 302
es

redirect, and includes the NSC_TMAA cookie.

al

Configuration
e
or

The AAA virtual server configuration requires an SSL cert to be configured. The SSL certkey is
bound to the authentication virtual server. The authentication policies are bound to the
di

authentication virtual server with two-factor and dual password authentication supported. The
s

policies can also be bound directly to the virtual server for:

tri

• Traffic management, which specifies the timeout and default action of ALLOW or DENY
bu

• Auditing and logging policies to log events pertaining to authentication events

t

Authorization policies can also be bound to these groups.

io
n

When the AAA virtual server is configured for authentication:

• Configure the authentication settings on the HTTP load-balancing virtual servers to protect.
• Enter the FQDN of the authentication virtual server, which is passed to the user in the 200
response and post action to the load-balancing virtual server.

Creating an Authentication Virtual Server

Use the following procedure to create an authentication policy:
1. Create an authentication virtual server.

© Copyright 2016 Citrix Systems, Inc. Module 8: Authentication, Authorization, and Auditing 243
2. Bind an SSL certificate to the authentication virtual server.
3. Bind the authentication policy to the authentication virtual server.
4. Configure one or more traffic management virtual servers (Load Balancing or Content
Switching).
5. Configure the traffic management virtual server for integration with the authentication virtual
server
• Enable Authentication (Advanced tab)
• Specify the Authentication vServer
• Set the Authentication FQDN to be the FQDN of the authentication vServer

Creating an Authentication Virtual Server in the Command-

Line Interface
N
ot

Enter the following command to create an authentication virtual server:

fo

add authentication vserver vserver_name SSL ip_address -

rr

AuthenticationDomain domain
es

Arguments include:
al
e

Argument Definition
or

vserver_name The name of the virtual server

di

ip_address The IP address of the authentication virtual

s

server (VIP).
tri
bu

domain The domain name of the virtual server

This is the domain portion of the authentication
t io

virtual server FQDN only.

n

Example: If the FQDN is secure.example.com;

the domain is example.com only.

Binding an SSL Certificate in the Command-Line Interface

Enter the following command to bind an SSL certificate to the authentication virtual server:

bind ssl vserver vserver_name -certKeyName cert_name

Arguments include:

244 Module 8: Authentication, Authorization, and Auditing © Copyright 2016 Citrix Systems, Inc.
Argument Definition
vserver_name The name of the virtual server

cert_name The name of the SSL certificate-key pair on the

NetScaler.

Binding a Virtual Server to an Authentication Policy in the

Command-Line Interface
Authentication virtual servers can be configured with two-factor authentication. To use single-
factor authentication bind a policy to the primary bank only:
N

bind authentication vserver vserver_name -policy policy_name

ot

To use two-factor authentication, bind additional authentication policies to the secondary policy
fo

bank:
rr

bind authentication vserver vserver_name -policy policy_name -

es

secondary
al

Priorities can be used with authentication policies.

e

Arguments include:
or
di

Argument Definition
s tri

vserver_name The name of the virtual server

bu

policy_name The name of the policy

t io
n

Configuring a Virtual Server to use an Authentication Virtual

Server in the Command-Line Interface
Enter the following command to configure a virtual server to use an authentication virtual server:

set lb vserver lb_vsrv_afweb -AuthenticationHost hostname -

Authentication ON -authnVsName authe_vsrv

The following table lists available arguments.

© Copyright 2016 Citrix Systems, Inc. Module 8: Authentication, Authorization, and Auditing 245
Argument Definition
hostname The hostname of the authentication virtual
server.

authnVsName The authentication virtual server on this

NetScaler.

Configuring the Authentication Virtual Server in the

Configuration Utility
To configure the traffic management virtual server to integrate with the authentication virtual
server, configure the Authentication settings on the advanced tab.
N

Multiple load balancing or content switching virtual servers can be integrated with the same
ot

authentication virtual server.

fo
rr
es
al
e
or
di
s tri
bu
t io
n

246 Module 8: Authentication, Authorization, and Auditing © Copyright 2016 Citrix Systems, Inc.
Configuring Authorization Policies for Traffic Management
Authorization policies can determine allow or deny behavior for authenticated users. Authorization
policies can be used to allow or deny access based on group membership or other criteria.
Authorization policies can provide allow or deny decisions for the sites as a whole or can be used to
restrict access to certain destination paths.
Authorization settings can be managed across Authorization policies or the authorization setting
using session policies.
Authorization policies can only be bound to AAA groups and AAA users. Use the following
procedure to configure an authorization policy for traffic management:
1. Set the default traffic management authorization action to DENY: Security > AAA -
Application Traffic > Change Global Settings > Default Authorization Action.
2. Create an authorization policy to allow access to the targeted page or pages.
N

3. Create AAA groups on the NetScaler system. These groups should match the group names in
LDAP or Radius if using group extraction.
ot

4. Bind the authorization policies to the AAA groups (or AAA user accounts).
fo

Session policies can also control the authorization behavior and can be used to override the global
rr

authorization settings. Session policies can be bound to authentication virtual servers, AAA groups,
and AAA users.
es
al

Setting the Default Traffic Management Authentication

e

Action to Deny in the Command-Line Interface

or
di

Enter the following command to set the default traffic management authentication action to deny:
s tri

set tm sessionParameter -defaultAuthorizationAction DENY

bu

Creating an Authorization Policy to Allow Access

t io
n

Because the default traffic management action is set to DENY, policies must be put in place to
allow access.
Enter the following command to create a policy to allow access:

add authorization policy policy_name "expression" ALLOW

Arguments include:

© Copyright 2016 Citrix Systems, Inc. Module 8: Authentication, Authorization, and Auditing 247
Argument Definition
policy_name The name of the policy

expression A classic expression that identifies the pages to

be allowed, which can be done with the
"REQ.HTTP.URL.CONTAINS [...]" expression.

SAML Authentication
Starting with NetScaler 10.0, SAML authentication policies are supported for use with SSLVPN
virtual servers and authentication virtual servers (AAA for Application Traffic).
SAML authentication can be integrated as a primary or secondary authentication policy method.
N

When combined with other authentication policies, SAML authentication can be used to direct
ot

authentication validation to an external authentication point. SAML authentication can be used as

part of a single-factor, two-factor, or even three-factor authentication process. SAML double-source
fo

authentication is also supported.

rr

SAML authentication policies can be created just like other authentication policies. The policies
consists of an expression and an action, where the SAML policy action (samlaction) specifies the
es

SAML authentication requirements (or server).

al

With SAML authentication, user authentication is redirected to an external authentication page

e

hosted on a SAML server (not on the NetScaler). After the authentication is complete, the user is
redirected back to the NetScaler entity. Traffic is either sent to the original load balancing or virtual
or

server if single-factor authentication was used (SAML only) or the traffic is redirected to the
authentication virtual server to continue with LDAP/RADIUS or the other configured
di

authentication methods.
s tri

For example, if authentication is setup to include:

bu

• SAML
• LDAP
t io

• Radius
n

The user's request is first directed to the SAML server logon page and then returned to the
NetScaler's Traffic Management page for authentication using LDAP and Radius. This results in
two log on pages, with three different credentials.
Typically, SAML policies would be bound to the primary bank. To combine SAML, LDAP, and
Radius. SAML would be bound to the primary bank first, with LDAP in the primary bank as
cascade authentication and Radius in the secondary bank.
SAML policies can be created within the GUI. To bind or adjust policy priorities, the command
line may need to be used depending on NetScaler version.

248 Module 8: Authentication, Authorization, and Auditing © Copyright 2016 Citrix Systems, Inc.
 N
ot
fo
rr
es
al
e
or

Audit Logging
di

A global auditing policy can be bound to the authentication virtual server, or the policy can be
s

bound globally. These actions can be performed in the Auditing tab under the AAA virtual server
tri

settings and will log auditing events in the syslog and nslog formats.
bu

The information captured in the logs include:

t io

• Group extraction details:

n

AAA EXTRACTED_GROUPS 6499 : Extracted_groups

"students1,XenAppUsers"
• Client information details:
"Client_ip 10.90.147.83 -Nat_ip "Mapped Ip" -
Vserver 10.90.196.13:443 -Browser_type "Mozilla/5.0"
• Requests denied by authorization policies:
Remote_host lbvserver.aaa.com -
Denied_url GET /citrix_logo.jpg -Denied_by_policy "No-Jpgs"

© Copyright 2016 Citrix Systems, Inc. Module 8: Authentication, Authorization, and Auditing 249
The following log message shows AAA extracted groups:

Mar 12 15:22:08 <local0.info> 10.90.196.10 03/12/2009:15:22:08

GMT Ronan : AAA EXTRACTED_GROUPS 6439 : Extracted_groups
"students1,XenAppUsers"

Log messages are generated with every successful logon. For example:

Mar 12 15:22:08 <local0.info> 10.90.196.10 03/12/2009:15:22:08

GMT Ronan : AAATM LOGIN 6440 : Context [email protected] -
User student01 - Client_ip 10.90.147.83 - Nat_ip "Mapped Ip" -
Vserver 10.90.196.13:443 -
Browser_type "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-
US; rv:1.8.1.20) Gecko/20081217 Firefox/2.0.0.20" -
Group(s) "students1"
N

The following log message shows a successful AAATM HTTP request.

ot
fo

Mar 12 15:22:08 <local0.debug> 10.90.196.10 03/12/2009:15:22:08

GMT Ronan : AAATM HTTPREQUEST 6441 : Context
rr

[email protected] -
es

lbvserver.aaa.com User student01 : Group(s) students1 : Vserver

10.90.196.12:80 - 03/12/2009:15:22:08 GMT GET / - -
al

The following log message shows a failed attempt at resource access:

e
or

Mar 12 15:22:09 <local0.alert> 10.90.196.10 03/12/2009:15:22:09

GMT Ronan : AAATM HTTP_RESOURCEACCESS_DENIED 6442 : Context
di

[email protected] - User student01 -

s

Vserver 10.90.196.12:80 - Total_bytes_send 433 -

tri

Remote_host lbvserver.aaa.com - Denied_url GET /citrix_logo.jpg -

bu

Denied_by_policy "No-Jpgs" - Group(s) "students1"

t

The information captured in the log oncludes session termination. For example:
io

• logon time
n

• Logout time
• Duration
• Polices that are allowed and denied
• Logout method
The following log message shows an example of logout information:

250 Module 8: Authentication, Authorization, and Auditing © Copyright 2016 Citrix Systems, Inc.
 Mar 12 16:14:51 <local0.info> 10.90.196.10 03/12/2009:16:14:51
GMT Ronan : AAATM LOGOUT 6629 : Context [email protected] -
User student01 - Client_ip 10.90.147.83 - Nat_ip "Mapped Ip" -
Vserver 10.90.196.13:443 - Start_time "03/12/2009:15:30:26 GMT" -
End_time "03/12/2009:16:14:51 GMT" - Duration 00:44:25 -
Http_resources_accessed 1 - Total_TCP_connections 0 -
Total_policies_allowed 1 - Total_policies_denied 1 -
Total_bytes_send 0 - Total_bytes_recv 0 -
Total_compressedbytes_send 0 - Total_compressedbytes_recv 0 -
Compression_ratio_send 0.00% - Compression_ratio_recv 0.00% -
LogoutMethod "KickedOut" - Group(s) "students1"

Audit Logging Troubleshooting

N

An administrator can use Firefox Live HTTP Headers or Internet Explorer HTTP Headers to view
ot

cookies, POST content, 302 redirect details and HTTP response error headers, as shown in the
figure.
fo
rr
es
al
e
or
di
s tri
bu
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 8: Authentication, Authorization, and Auditing 251
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

252 © Copyright 2016 Citrix Systems, Inc.

 9
Module 9

AppExpert Rate
Limiting, HTTP
Service Callout, and
N

Policy-based
ot

Logging
fo
rr
es
al
e
or
di
s
tri
bu
t io
n
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

254 © Copyright 2016 Citrix Systems, Inc.

AppExpert Rate Limiting, HTTP Service
Callout, and Policy-based Logging Manual
Overview
At the end of this module you will be able to:
• Describe HTTP callout.
• Configure HTTP callout.
• Describe the AppExpert rate control feature and its components.
• Identify HTTP callout use cases with Rate Control.
• Configure AppExpert rate control policies on the NetScaler system.
N

• Explain Rate Control policy scenarios.

ot

• Describe policy-based Audit Logging.

fo
rr

HTTP Callout Overview

es

For certain types of incoming requests it may be necessary to stall policy evaluation while external
al

information is retrieved from a server, and based on the response from the external server, certain
e

actions may be required. Additionally, you may also want to update information to a database or
content on a Web Server based on specific information that is retrieved from the external server.
or

HTTP callouts enable an administrator to perform these tasks by enabling incoming packets to be
di

processed by an external service that can be a virtual server on the NetScaler system, a back-end
s

server, or a third-party service.

tri

Traditionally, the NetScaler system verifies incoming packets internally using built-in policies.
bu

However, with external specialized services that are available for validation, they can be integrated
with the NetScaler system using the HTTP callout feature.
t io

HTTP callouts can modify any part of an incoming packet seamlessly and proceed with the
n

transaction. Use case scenarios for HTTP callout include:

• IP address blacklisting
• Spam verification
• Access control
• Management integration
• Custom content rewrite
• Universal Description Discovery and Integration (UDDI access)

© Copyright 2016 Citrix Systems, Inc. Module 9: AppExpert Rate Limiting, HTTP Service Callout, and
Policy-based Logging 255
HTTP Callouts
When the NetScaler appliance receives a client request, the appliance evaluates the request against
the policies bound to various bind points. During this evaluation, if the appliance encounters the
HTTP callout expression, SYS.HTTP_CALLOUT(<name>), it stalls policy evaluation briefly and
sends a request to the HTTP callout agent by using the parameters configured for the specified
HTTP callout. Upon receiving the response, the appliance inspects the specified portion of the
response, and then either performs an action or evaluates the next policy, depending on whether
the evaluation of the response from the HTTP callout agent evaluates to TRUE or FALSE,
respectively. For example, if the HTTP callout is included in a responder policy, if the evaluation of
the response evaluates to TRUE, the appliance performs the action associated with the responder
policy.
An HTTP callout consists of a NetScaler policy expression that sends a simple HTTP/HTTP(s)
request to an external service, waits for the response, and parses the response to determine the
result. HTTP callouts are available for any feature that supports asynchronous and advanced
N

policies, including:
ot

• HTTP and TCP content switching

fo

• Rewrite
rr

• Responder (content filtering)

• Token load balancing
es

The type of expression limits how the HTTP callout expression is used. For example, if an
al

HTTP.RES expression is used in an HTTP callout, then an administrator cannot use that callout in
e

an HTTP request time policy bank or in a TCP content switching policy bank.
or
di
s tri
bu
t io
n

256 Module 9: AppExpert Rate Limiting, HTTP Service Callout, and Policy-based Logging ©
Copyright 2016 Citrix Systems, Inc.
 N
ot
fo
rr
es
al
e
or

The figure describes the service callout process. HTTP service callouts invoke external functionality
from within NetScaler policies and are available for multiple features. During the HTTP service
di

callout process:
s tri

1. The user sends a request

bu

2. The policy sends the HTTP request to an external service.

3. The policy uses the result like other policy expression evaluation results.
t io
n

Configuring HTTP Callouts

HTTP callouts can be configured in the AppExpert node of the Configuration Utility.
To configure an HTTP callout, the administrator must:
1. Create the HTTP Callout
2. Specify the server
3. Define the request to send to the server.
4. Define the server response.
5. Configure the external server

© Copyright 2016 Citrix Systems, Inc. Module 9: AppExpert Rate Limiting, HTTP Service Callout, and
Policy-based Logging 257
 Configuring the external server is not covered in this course because it is unique to the
customer environment.

Use the following procedure to create an HTTP callout to compare the IP address of an incoming
connection against a list of blacklisted IP addresses:
1. Go to AppExpert > HTTP Callouts.
2. Click Add in the HTTP Callouts pane. The Create HTTP Callout dialog box opens.
3. Type a name for the callout in the Name field.
4. Select IP Address under Server to receive callout request.
5. Type the IP address of the server to receive the callout request in the IP Address field.

The IP address contains the server blacklist.

N
ot

6. Type the port number of the service in the Port field.

fo

7. Select Attribute-based or Expression-based and configure as appropriate under Request to send

rr

to the server.
es

8. Select TEXT in the Return Type field under Server Response.

9. Type "HTTP.RES.BODY(100)" in the Expression to extract data from the response field.
al

10. Click Create. The new callout appears in the HTTP Callouts pane.
e

Use the following to configure HTTP callout in the Command-Line Interface:

or
di

Set policy httpcallout name

s tri

Callout Examples
bu
t

Consider the HTTP callout expression SYS.HTTP_CALLOUT.

io

Enter the following command to add an HTTP callout policy:

n

add policy httpCallout callout_name

This command adds the simplest HTTP callout with default behavior.
For example, create an HTTP callout policy:

add policy httpCallout auth_call

For all HTTP callouts:

258 Module 9: AppExpert Rate Limiting, HTTP Service Callout, and Policy-based Logging ©
Copyright 2016 Citrix Systems, Inc.
• An undefined event (UNDEF) is raised by SYS.HTTP_CALLOUT if an HTTP callout cannot
be made.
• An HTTP callout request contains standard headers by default. An administrator can specify
additional headers.
Examples of other asynchronous default policy expressions are HTTP.RES.BODY(1000) and
CLIENT.IP.DST.MATCHES ("example.com").
The HTTP callout name is any valid entity name and is used in the SYS.HTTP_CALLOUT
expression. For example: SYS.HTTP_CALLOUT(auth_call).
A default HTTP Callout configured this way always send a GET / request when invoked.

HTTP Callout Response Parsing

Enter the following command to set an HTTP callout policy:
N
ot

set policy httpCallout callout_name [-resultExpr string]

fo

An administrator can build the result using default policy expressions. The default policy expression
rr

given in the -resultExpr argument must match the -returnType specified in the add policy
httpCallout command.
es

The return type controls how SYS.HTTP_CALLOUT can be extended. For example,
al

SYS.HTTP_CALLOUT(auth_call).CONTAINS("example")matches the returned data as text.

e

SYS.HTTP_CALLOUT(auth_call).GT(500) matches the returned data as a number.

or

Only HTTP.RES based PI expressions can be used to build the result, and it is possible to use
asynchronous default policy expressions.
di
s

Avoiding HTTP Callout Recursion

tri
bu

Even though the NetScaler appliance does not check for the validity of the HTTP callout request, it
parses the request once before it sends the request to the HTTP callout agent. This allows the
t io

appliance to treat the callout request as any other incoming request, which allows the administrator
n

to configure several useful NetScaler features (such as integrated caching, SureConnect, and Priority
Queuing) to work on the callout request.
However, during this parsing, the HTTP callout request can hit the same policy and therefore
invoke itself recursively. The appliance detects the recursive invocation and raises an undefined
(UNDEF) condition. However, the recursive invocation results in the policy and HTTP callout hit
counters being incremented by two counts each instead of one count each.
To prevent a callout from invoking itself, you must identify at least one unique characteristic of the
HTTP callout request, and then exclude all requests with this characteristic from being processed by
the policy rule that invokes the callout. You can do so by including another default syntax
expression in the policy rule. The expression must precede the SYS.HTTP_CALLOUT(<name>)
expression so that it is evaluated before the callout expression is evaluated.

© Copyright 2016 Citrix Systems, Inc. Module 9: AppExpert Rate Limiting, HTTP Service Callout, and
Policy-based Logging 259
 When you configure a policy rule in this way, the compound rule evaluates to FALSE once
the appliance generates the request and parses it. The callout is not generated a second
time, and the hit counters are incremented correctly.

One way by which you can assign a unique characteristic to an HTTP callout request is to include a
unique custom HTTP header when you configure the callout.

HTTP Callout Use Cases

The following examples illustrate how to use HTTP callouts to perform various tasks. The NetScaler
system performs a callout to an external server where a callout agent is configured to respond to
the request that is based on external server data.
HTTP callouts can be used in the following scenarios:
N

• Filter clients based on an IP blacklist.

ot

• Fetch and update content using Edge Side Includes (ESI) markup language.
fo

Scenario 1: Filter Clients Based on an IP Address Blacklist

rr
es

HTTP callouts are used to verify the IP address of the client that started an HTTP session. After the
source IP address of the incoming packet is checked against the blacklist, the transaction is either
al

blocked or allowed to continue on the NetScaler system.

e

HTTP callout facilitates this filtering by allowing the appliance to communicate with another server
or

that contains a database of blacklisted IP addresses. Because a list can be very long and it is time
consuming to search for every incoming packet, Citrix recommends to not have it on the appliance
di

consuming the appliance processing power.

s

When an HTTP callout is made to an external server, packet processing continues on the appliance
tri

until a response is received from the server. Depending on the response received, the appliance
bu

either continues with the transaction, drops it or allows other features, such as Responder, to take
appropriate action.
t io

In the following example, an HTTP callout is created to take the source IP address from an HTTP
n

header and to send it to an external server with IP address 192.168.1.2 on port 80. This callout
verifies if the source IP address is part of any pre-defined IP address blacklist. The external server
verifies the source IP address against the IP address blacklist that it maintains and returns either
VALID or INVALID based on the source IP address.
HTTP callouts can be used to block requests from clients that are blacklisted by an administrator.
This list of clients can be a publicly known blacklist, one that is maintained specifically by the
administrator or a combination of both. The source IP address of the incoming client request is
checked against the external pre-configured blacklist. Based on whether the IP address has been
blacklisted or not, the transaction is either blocked by the NetScaler system or continues to be
processed normally.
To implement this configuration:

260 Module 9: AppExpert Rate Limiting, HTTP Service Callout, and Policy-based Logging ©
Copyright 2016 Citrix Systems, Inc.
1. Enable the Responder feature.
2. Create an HTTP callout and configure it with details about the external server and other
required parameters.
3. Create a responder policy to analyze the response.
4. Bind the responder policy globally.
5. Create a callout agent on the remote server.

Scenario 2: Fetch and Update content

Edge Side Includes (ESI) is a markup language for the edge-level dynamic web content assembly. It
helps in accelerating dynamic web-based applications by defining a simple markup language to
describe cacheable and non-cacheable Web page components that can be aggregated, assembled,
and delivered at the network edge.
N
ot

Creating HTTP-Callout-2 on the NetScaler System

fo

Create HTTP-Callout-2 with the following parameter settings:

rr

• Name: HTTP-Callout-2
es

• IP address: 10.102.56.51
al

• Port: 80
e

• Method: GET
or

• Host Expression: 10.102.56.51:80

• URL Stem Expression: "HTTP.RES.BODY(500).AFTER_STR(\"src=\").BEFORE_STR(\"/>\")"
di

• Headers Name: Name

s tri

• Value Expression: Callout

bu

• Return Type: TEXT

• Expression to extract response from: HTTP.RES.BODY(100)
t io
n

Adding the Rewrite Action

Create Rewrite action Action-Rewrite-1 to replace the ESI content with the callout response body.
Create the action with the parameters settings:
• Name: Action-Rewrite-1
• Type: Replace
• Expression to choose target text reference:
"HTTP.RES.BODY(500).AFTER_STR(\"<esi:example>\").BEFORE_STR(\"</esi:example>\")"
• String expression for replacement text: "SYS.HTTP_CALLOUT(HTTP-Callout-2)"

© Copyright 2016 Citrix Systems, Inc. Module 9: AppExpert Rate Limiting, HTTP Service Callout, and
Policy-based Logging 261
Creating the Rewrite Policy
Create Rewrite policy Policy-Rewrite-1 with the following parameter settings:
• Name: Policy-Rewrite-1
• Action: Action-Rewrite-1
• Undefined-Result-Action: Global undefined-result-action
• Expression: "!HTTP.REQ.HEADER(\"Name\").CONTAINS(\"Callout\")"

Binding the Rewrite Policy Globally in the Configuration

Utility
The rewrite policy must be bound on the NetScaler system to ensure that the HTTP callout is
N

applied to incoming traffic.

ot

Use the following procedure to bind rewrite policy Policy-Rewrite-1 globally:

fo

1. Select AppExpert > Rewrite.

rr

2. Click Rewrite policy manager under Policy Manager.

3. Click the Response under Connection Type.
es

4. Click Continue.
al

5. Under Policy Binding, click in the field below Select Policy.

e

6. Click the radio button next to Policy-Rewrite-1, then click Select.

or

7. Click Bind and then Done.

di

HTTP Callout Auditing

s tri

HTTP callouts have statistics available in the NetScaler Configuration Utility, command-line
bu

interface, and SNMP.

t io

Enter the following command to view HTTP callout statistics in the command-line interface:
n

nsconsmsg -d stats | grep hc_

When a callout is highlighted in the left-hand window pane of the Monitoring screen, then the
statistics are listed.

Configuring Rate Control

To configure rate control:
1. Define a stream selector

262 Module 9: AppExpert Rate Limiting, HTTP Service Callout, and Policy-based Logging ©
Copyright 2016 Citrix Systems, Inc.
2. Define a limit identifier
3. Define a rate limit policy
When defining the limit selector, a NetScaler administrator can select a specific limit mode and
limit type. The limit mode specifies the mode of tracking for rate control such as request rate or
connections. The limit types specify the type of limit tracking and available types include bursty
and smooth.
Stream selectors, which are simple expressions, are created to parse traffic for information. The
information is used to differentiate traffic. An administrator can use the procedure in the following
table to create a limit selector in the configuration utility.
1. Expand the AppExpert node.
2. Expand Rate Limiting in the AppExpert pane.
3. Click Add in the Selectors pane. This opens the Create Selector dialog box.
4. Type the name of the limit selector in the Name field.
N

5. Click Insert and enter an expression in the Expressions field and click Insert.
ot

6. Click Create. The Create Limit Selector dialog box closes.

fo
rr

Limit Mode
es

The limit modes are:

al

• REQUEST_RATE. This default mode tracks the number of active requests per client. The
e

timeout is specified in 10 ms increments and can be either Bursty or Smooth.

or

• CONNECTIONS. This mode tracks the number of active transactions for the client and is
useful for HTTP traffic expected to open many concurrent connections, such as web crawlers
di

and spiders. This mode is always Bursty.

s

• NONE. This mode tracks the bandwidth usage of the client, which is collected over time for
tri

each client. The decision to activate the policy or to let the traffic through is made at the time
bu

of the request. Once a request has been allowed through, there is no rate control of the
bandwidth consumed by the request.
t io

The limit types are:

n

• Bursty. This limit type is where the client exhausts its quota at any time throughout the
timeslice. Bursty is useful when users open a web page, and three to four connections are
opened simultaneously followed by a period of inactivity.
• Smooth. This limit type is where the quota is averaged across the timeslice. Smooth is useful
for traffic patterns where users open a new connections at a fixed rate.
Limit identifiers require a maximum threshold and a defined time period when they are created.
An administrator can use the following procedure to create a limit identifier in the Configuration
Utility
1. Expand the AppExpert node.
2. Expand Rate Limiting in the AppExpert pane.

© Copyright 2016 Citrix Systems, Inc. Module 9: AppExpert Rate Limiting, HTTP Service Callout, and
Policy-based Logging 263
3. Select Limit Identifiers in the Rate Limiting pane.
4. Click Add in the Limit Identifiers pane. The Create Limit Identifier dialog box.
5. Type the name of the limit identifier in the Name field.
6. Select a limit selector from the Selector list.
7. Select a limit identifier mode from the Mode list.
8. Select a limit type from the Limit Type list.
9. Type the threshold in the Threshold field.
10. Type the time slice the in the Time Slice (msec) field.
11. Type the maximum bandwidth in the Max Bandwidth field.
12. Type the number of traps in the Traps field.
13. Click Create. The Create Limit Identifier dialog box closes.
N

Creating Rate-Limiting Policies

ot
fo

Rate-limiting policies, like all policies, require an expression and an action. An administrator must
place the SYS.CHECK_LIMIT call as the last component of the expression.
rr
es

Practice A
al
e

• Limit identifier
• Default Policy
or

• Limit selector
di
s

Term Description
tri
bu

Counter variable that tracks the amount of

t

traffic that passes through a NetScaler system

io

and the number of times that it has been

n

invoked during a specific time interval

Defines the action that the NetScaler system

takes when encountering excess traffic

Optional filter expression that restricts which

traffic to monitor

264 Module 9: AppExpert Rate Limiting, HTTP Service Callout, and Policy-based Logging ©
Copyright 2016 Citrix Systems, Inc.
Rate Control Policy Scenarios
Rate control defends a network from attacks while still allowing acceptable traffic to pass through
the system. This feature flexibility limits traffic based on a variety of criteria.
Common attributes for rate control are:
• DNS
• Source IP address
• Bandwidth consumption (primarily for streaming protocols)
Common responses to excess traffic are:
• Drop requests
• Redirect requests
• Cache responses
N

Configuration examples are:

ot

• Rate control by subnet

fo

• No selector with responder

rr

• Integrated caching
es

Rate Control by Subnet

al
e

A rate limiting policy that limits traffic based on the subnet of the request can be configured in the
or

command-line interface. An administrator can use the following command to create the limit
selector test1, which parses the traffic for the subnet of the client in the command-line interface:
di
s

add ns limitSelector test1 Q.URL "CLIENT.IP.SRC.SUBNET(24)"

tri
bu

The following command creates the limit identifier identifier1, which allows five requests within 20
seconds to pass through and uses selector test1:
t io

add ns limitIdentifier rslm1 -threshold 5 -timeSlice 20000 -

n

mode REQUEST_RATE -limittype bursty -selectorName test1

The excess traffic is redirected to http://www.example.com. The following command creates an

action that the policy undertakes when traffic exceeds the threshold:

add responder action identifier1 redirect

"\"http://www.example.com/\""

The following command creates responder policy policy1, which checks for requests containing
testsite2:

© Copyright 2016 Citrix Systems, Inc. Module 9: AppExpert Rate Limiting, HTTP Service Callout, and
Policy-based Logging 265
 add responder policy policy1 "REQ.URL.CONTAINS(\"testsite2\") &&
SYS.CHECK_LIMIT(\"test1\")" identifier1

This request triggers the limit identifier, which creates or increments an instance ID based on the
subnet of the request. If the instance ID exceeds the threshold defined in the limit identifier, then
the responder action is run.
The following command binds the policy:

bind responder global policy1 8 END -type DEFAULT

A selector does not need to be defined if the limit identifier is bound exclusively to a policy. For
example, the following example adds a limit identifier:

add ns limitIdentifier example1 -threshold 20000 -timeSlice 1000 -

N

trapsInTimeSlice 400
ot

If this limit identifier is used by another policy, then the limit identifier increments when either
fo

policy is exercised.
rr

Using Cache as a Rate Control Action Example

es
al

The following command configures the responder action to redirect excess traffic to the
e

http://example.com web site:

or

add responder action resp_vip_act redirect

"\"http://example.com\""
di
s

The following command checks for requests containing example_sale.html:

tri
bu

add responder policy resp_vip_pol

"Q.URL.CONTAINS(\"example_sale.html\") &&
t io

SYS.CHECK_LIMIT(\"example1\")" resp_vip_act
n

The following command binds the policy:

bind lb vserver lb_vip -policyName resp_vip_pol -priority 5 -

gotoPriorityExpression END

The Integrated Caching feature is used to cache a URL only if the number of requests per second
exceeds a threshold. The following command to configure a selector identifies traffic based on a
requested URL:

add limitselector cache_test Q.URL

266 Module 9: AppExpert Rate Limiting, HTTP Service Callout, and Policy-based Logging ©
Copyright 2016 Citrix Systems, Inc.
The following command rate-limits five hits every two seconds:

add limitidentifier cache_identifier -threshold 5 -

timeslice 2000 -selectorName cache_test

The following command applies the policy to all GET requests and uses the CACHE predefined
action:

add cache policy cac_policy -

rule "HTTP.REQ.METHOD.EQ(GET) &&
SYS.CHECK_LIMIT(\"cache_identifier\")" -action CACHE

The following command binds the policy:

bind cache global cac_policy -priority 7

N
ot

Practice B
fo
rr

Indicate whether each of the following statements is true or false.

es

If a limit identifier is configured with no selector, then only a single instance ID is created for all
traffic.
al

A. True
e

B. False
or

Bursty is the limit type is where the quota is averaged across the timeslice.
di

A. True
s tri

B. False
bu

The Integrated Caching feature caches a URL only if the number of requests per second exceeds a
threshold.
t io

A. True
n

B. False

Policy Based Logging

You can configure policy based logging for a variety of features, including Rate Limiting, on the
NetScaler appliance. Policy based logging enables you to specify a format for log messages. The
contents of the log message are defined by using a default syntax expression in the policy.
To configure policy based logging for a policy, you must first configure an audit message action.

© Copyright 2016 Citrix Systems, Inc. Module 9: AppExpert Rate Limiting, HTTP Service Callout, and
Policy-based Logging 267
To create an audit message action
By using the command line interface:

add audit messageaction <name> <logLevel> <stringBuilderExpr> [-

logtoNewnslog (YES|NO)] [-bypassSafetyCheck (YES|NO)]

By using the configuration utility

1. Go to System > Auditing > Message Actions.
2. In the Auditing Message Actions pane, click Add.
3. In the Create Audit Message Action dialog box, configure the message action using default
expressions to build the custom log entry.
4. Enter a Name and Log Level.
5. Enter an Expression in the Expression field.
N
ot

6. Click Create.
fo

Binding Audit Message Action to a Policy

rr
es

After you have created an audit message action, you must bind it to a rewrite or responder policy.
For more information about binding log message actions to a rewrite or responder policy, see Citrix
al

product documentation at http://docs.citrix.com.

e
or
di
s tri
bu
t io
n

268 Module 9: AppExpert Rate Limiting, HTTP Service Callout, and Policy-based Logging ©
Copyright 2016 Citrix Systems, Inc.
10
n
t io
bu
stri
di
or
Command Center

e
al
es
rr
fo
ot
N
Module 10
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

270 © Copyright 2016 Citrix Systems, Inc.

Command Center Manual
Overview
Utilizing multiple NetScaler systems is a requirement as a deployment grows. The larger a
deployment grows, the more management is impacted. Citrix Command Center is a centralized
monitoring, configuration and management solution for Citrix NetScaler, Citrix NetScaler Gateway
Enterprise Edition, Citrix CloudBridge and Citrix CloudBridge Platform. Citrix Command Center
offers performance monitoring, management, and troubleshooting functionality for an enterprise-
wide global application delivery infrastructure.

Objectives
N

At the end of this module, you will be able to:

ot

• Install Command Center.

fo

• Explain Command Center functionality.

rr

• Manage Command Center.

es

• Troubleshoot Command Center.

al
e

Command Center Introduction

or

Command Center supports management of NetScaler MPX, SDX, and VPX appliances along with
Citrix CloudBridge products. Command Center allows administrators to centrally manage multiple
di

appliances and supports configuration, reporting, alerting, and performance monitoring

s

functionality.
tri
bu

Command Center Editions

t io

Citrix Command Center is available in a physical hardware appliance and as a web-based software
n

installation that can run on Windows or Linux operating systems.

The Command Center appliance is a self-contained solution with a pre-loaded database and no
external dependencies. The hardware appliance provides the same functionality as the software-
based solutions. It is not discussed in this class.
Citrix Command Center can be deployed on Windows or Linux servers as a software-based
solution. The software solution requires a separate database server running Microsoft SQL Server,
MySQL, or Oracle. It also supports a high-availability configuration. Command Center provides a
web-based configuration and management console allowing access to Command Center
functionality from any web-accessible system.

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 271
Command Center Features
Command Center supports centralized management of both CloudBridge and NetScaler systems,
although some features are limited to NetScaler systems only.
Command Center supports management and monitoring of NetScaler and NetScaler Gateway MPX
and VPX appliances. It also provides provisioning functions and specific management task for
NetScaler SDX appliances.
Command Center also supports centralized management, configuration, and reporting for the
CloudBridge product line. While Command Center provides discovery, fault management, and
monitoring for CloudBridge; some features are limited to NetScaler only.
The following table lists Command Center features.

Feature Description
N
ot

Discovery Allows Command Center to identify and add

NetScaler devices to a Command Center
fo

configuration for management. This includes

rr

management of standalone systems, high-

availability pairs, and clusters.
es

Provisioning Command Center can be used to provision

al

VPX instances on XenServer pools and on

e

NetScaler SDX platforms. Provisioning includes

create the VPX instances.
or

This feature is NetScaler only.

di

Fault Management Displays SNMP events and alarms triggered by

s tri

NetScaler devices in the network. Displays

syslog events from the NetScaler nslog file.
bu

Configuration Management Provides centralized configuration for multiple

t io

NetScaler devices.
n

Administrators can create, customize, and

execute tasks against one or more machines
from a single centralized management console
without having to configure NetScaler devices
individually. Tasks may be run on demand or
scheduled.

272 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
Feature Description
Monitoring Displays state and configuration details of
services, service groups and virtual servers
configured on NetScaler systems.
Displays dashboard summary of CloudBridge
traffic activity.

Change Management Configuration management also supports

configuration auditing and reporting.
This allows administrators to define templates
and policies for expected configuration settings.
It also runs comparisons against devices in the
environment to determine if systems match the
N

expected configurations or if changes are

ot

present and supports comparing running

configurations with saved configurations and
fo

reviewing change history.

rr

This feature is for NetScaler devices only.

es

Certificate Management Allows centralized management of SSL

al

certificates on NetScaler devices.

e

An administrator can view all SSL certificates

installed across all devices, as well install,
or

update, and manage certificates across all

devices from a central console. Alerts are also
di

displayed regarding certificate expirations.

stri

This feature is for NetScaler devices only.

bu

Performance Monitoring Gathers and maintains performance data across

all systems in the environment. Performance
t io

data includes appliance performance data and

feature performance data for NetScaler and
n

CloudBridge systems.
Additional functionality includes dashboard,
reports, and log views tailored to the
Application Firewall and NetScaler Gateway
features.
Performance monitoring consolidates data
within the central database and supports the
generation of reports, custom reports and
graphs.

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 273
Feature Description
Data Consolidation Backs up configuration, SSL certificates, and
license files for managed devices. Command
Center stores these files as part of the device
inventory within the database in order to
support device recovery.

NetScaler and CloudBridge Support

The following versions of Citrix NetScaler are supported:
• NetScaler 7.0 - Monitoring of NetScaler entities is not supported.
• NetScaler 8.0 and later (including NetScaler 11.x)
N

• NetScaler MPX, VPX, and SDX

ot

The following versions of the Citrix CloudBridge (formerly known as WANScaler and Branch
fo

Repeater) product line are supported:

rr

• Repeater and Branch Repeater 4.5.0 and later

es

• Branch Repeater with Windows Server (2003 and 2008) 2.0.0 or later
• Branch Repeater VPX 5.6.0 or later
al

• This includes support for CloudBridge 7.0 products.

e

Repeater (formerly known as WANScaler) refers to physical appliances intended for use in
or

datacenters and regional headquarters and support between 10 Mbps - 2 Gbps throughput.
di

CloudBridge appliances, formerly known as Branch Repeater appliances, are physical appliances
s

intended for use Branch Offices support 2-10 Mbps throughput.

tri

CloudBridge VPX virtual appliances are virtual machines for smaller deployments and support
bu

between 0.5-45 Mbps and are useful for locations where physical appliances may not be ideal.
t

This course focuses on the use of Command Center with the NetScaler product line;
io

however, occasional notes regarding CloudBridge support may be noted where

n

appropriate.

Command Center Installation types

Command Center supports two types of installations: evaluation and typical.
The evaluation installation type is useful for proof-of-concept and test deployments of Command
Center. No external database is required and instead the installation uses a PostGRE SQL database
installed locally to the Command Center server. The evaluation installation does not support

274 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
exporting of report data from the database. While it supports all other functionality of Command
Center, the evaluation installation should not be used for production deployments.
The typical installation provides full access to all installation options and requires the user to have
an existing database server running MySQL with InnoDB, Oracle, or Microsoft SQL Server available
for use with reporting. This is the preferred installation type for production deployments.

Command Center Modes

Command Center can be deployed in a single-tier mode where a single Command Center server
performs Command Center tasks and also collects data to manage and monitor Citrix devices.
In a multiple-tier configurations, a central Command Center server is deployed along with
additional Command Center agents. The agent systems gather performance and metric data from
the assigned Citrix devices and are used to push configuration changes, while all command center
N

tasks are managed centrally through the Command Center server. This reduces load on the central
Command Center server by distributing some functions to the agent systems.
ot

The Command Center server is responsible for discovery, trap processing, monitoring entities,
fo

syslog messages, and configuration using tasks.

rr

The Command Center agents perform the following tasks:

es

• monitoring entities and syslog messages

• polling and collecting data used for performance monitoring (CPU, resource utilization, and IP
al

bytes transmitted)
e

• managing certificates
or

Devices are assigned to a specific agent for management within the Command Center
di

configuration. It is important to note that only certain tasks are performed by the agents;
s

the Command Center server handles the other points of monitoring and alerting.
tri
bu

High Availability
t io

Command Center server can be deployed as a single-server, standalone system. However, if the
n

Command Center server is offline, then there will be no data gathering or alerting available from
Command Center.
To avoid gaps in data being gathered or monitoring the environment, Command Center may also
be deployed in a High Availability (active/passive) configuration. In the High Availability
configuration, two Command Center servers are setup and configured to work in an HA pair. One
system is primary (active) and is responsible for monitoring systems, gathering data, and updating
the database. The other system takes on the role of the secondary system, which monitors the
primary server. In the event of an issue, the secondary can take over as the new primary. The old
primary takes on the role of new secondary. The database should be housed on a server separate
from the Command Center servers.

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 275
Server Requirements
Hardware requirements:

Component Minimum requirement

Processor Type Pentium 4

Processor Speed 1.2 GHz

Memory 1 GB RAM

Hard Disk Space 20 GB (See Disk Space Requirements)

N

Supported operating system requirements include:

ot

• Windows 2003 Service Pack 2

fo

• Windows 2003 Server R2 Standard x64 edition

rr

• Windows 2008
• Windows 2008 R2
es

• Windows 2012
al

• Windows 2012 R2
e

• Red Hat Enterprise Linux AS 4.0

or

• Red Hat Enterprise Linux ES 4.0 and 5.1

• CentOS 5.5 version
di
s

Database requirements:
tri
bu

Database Type Version

t io

MySQL 5.1.x with InnoDB storage engine

n

Oracle 10g / 11g

Microsoft SQL Server 2005, 2008, and 2008 R2

Requires: SQL Server authentication mode;
Windows authentication mode is not supported
in Command center.

276 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
Disk Space Requirements
A minimum of 20 GB is the recommended disk space for a basic production implementation.
Actual hard disk space requirements are dependent on the configuration of features, such as
Performance Management, the number of counters polled and counter retention policies, and the
number of devices in an environment.
Command Center stores performance data across three tables:
• The first table contains granular data gathered every 5 minutes for the last 14 days. Data is
overwritten beginning on the 15th day.
• The second table contains one polling result per hour for the last 30 days. Data is overwritten
on the 31st day.
• The third table contains one polling result per 24-hour period for the last 365 days. Data is
overwritten on the 366th day.
N

Using the default polling interval, Command Center will require almost 500 MB to maintain data
ot

for 100 counters for 10 devices.

fo

The following table can used for reference when sizing the database.
rr

No. of Counters Polling Interval No. of Devices Disk Space

es

Required
al
e

100 5 min 10 488 MB

or

300 5 min 10 1.4 GB

di

500 5 min 10 2.4 GB

s tri

1000 5 min 25 11.9 GB

bu

5000 5 min 50 119.1 GB

t io

10000 2.5 min 50 426 GB

n

Command Center Management Communication

Command Center provides a web-based administrative console that allows administrators to make
configuration changes and to interact with the reporting and dashboard functionality of Command
Center server. The Command Center client is any system connecting to the Command Center
server through a web browser. The following ports are used for Command center client to server
communication:

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 277
Purpose TCP Ports
HTTPS communication between Command 8443
Center client and server
Example: https://commandcenterfqdn:8443

HTTP communication between Command 9090

Center client and server
Example: http://commandcenterfqdn:9090

The Command Center server is responsible for issuing commands to and receiving alerts from
managed NetScaler systems. Configuration, SNMP polling and alerting, and performance reporting
communication occurs between Command Center server and the managed NetScaler NSIP
N

addresses. Communication does not occur from the Command Center client to the managed
NetScaler systems except when using the invoke CLI, GUI, or dashboard commands.
ot
fo

Purpose Ports
rr
es

SNMP polling between Command Center server UDP 161

and NetScaler
al

SNMP alerting between Command Center UDP 162

e

server and NetScaler

or

SSH and SFTP between Command Center TCP 22

di

server and NetScaler

s tri

TCP 22
Invoke CLI command exchanges
bu

communication between client and NetScaler

(SSH)
t io

Does not use Command Center server.

n

Invoke GUI and Invoke Dashboard makes 80 or 443

NetScaler configuration utility using web
browser between client and NetScaler
(HTTP or HTTPS). Does not use Command
Center server.

278 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
High Availability Communication
Command Center servers can be deployed in a high availability configuration to provide
redundancy and ensure continuous data gathering for Command Center. The two Command
Center servers will point to the same database. One Command Center server acts as the primary
(active) member and handles all data gathering and configuration tasks. The other server acts as the
secondary (passive) member which monitors the status of the primary and takes over data
gathering in the event of failover.
High Availability Communication includes a heartbeat interval which requires the Command
Center to update its status in the database (default: 60 seconds). Failover Interval is the interval at
which the secondary checks on the status of the primary system to determine if failover is required
(default: 75 seconds). The secondary server can be configured to check the status multiple times
before triggering failover (default retry count is 1). And a backup interval determines how often the
secondary system creates a backup of the configuration files from the primary.
N
ot

Purpose TCP Ports

fo

Communication between Command Center 2014, 1099

rr

servers in a High Availability pair

es

Communication between Command Center 6011

servers in a High Availability pair with a firewall
al

separating servers
e
or

Command Center Installation

di

Use the following procedure to prepare for installing Command Center:

s tri

1. Download Command Center from Citrix.com. Log on with a MyCitrix account and go to
bu

Downloads > Command Center.

2. Verify the operating system and database requirements.
t io

3. Check the client browser requirements.

n

4. Verify the ports.

5. Review the installation procedures for Command Center on Windows or Linux based on the
target operating system
Command Center installations are available as full installations or as updates to existing
installations.

Capacity Planning
Be aware of the following for capacity planning:

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 279
• Polled data is gathered at 5 minute intervals and is maintained for the last 14 days.
• Hourly consolidated data is maintained for the last 30 days.
• Daily consolidated data is maintained for last 365 days.
Command Center storage requirements varies based on total number of devices being monitored,
total number of counters being polled, and changes to the default polling intervals and retention
periods. Sample space requirements for database sizing is provided under Disk Space
Requirements. Disk space requirements for performance counters affects the database sizing if run
separately from the Command Center server. Some performance counters generate more data than
others; so monitor database growth after initial deployment when monitoring vector metrics.
Command Center also gathers the configuration files and the license files of managed NetScaler
systems and stores the files in the database as well. For each managed device, changes to the
configuration triggers a download of the updated configuration. Command Center stores the last 10
copies of configuration files for each managed system. Retention periods can be adjusted.
N

Faults and alert events also contribute to the database sizing.

ot

Backup
fo
rr

Database backups of the Command Center database and the configuration files gathered should be
es

handled by the database platform being used. Database administrators should coordinate regular
backups of the Command Center database for the MySQL, MSSQL, or Oracle.
al

When using the Command Center appliance, the database backup can be scheduled from the
e

appliance itself.
or

Installation Modes
di
s

After installation, Command Center can be started in either standalone or service mode and is run
tri

on Windows and Linux platforms.

bu
t

Standalone Mode
io
n

In standalone mode, the server is started as a standalone application that runs in the foreground.
Command Center must be manually started and stopped for a client to be able to connect to the
Command Center server. The application is not auto-started following a restart unless explicitly
included in a startup script or other task.
When Command Center is installed, the installation path will vary from Windows to Linux
systems. To simplify reference, <cc_home> will be used to refer to the default path. For Windows,
the default path will be: C:\Program Files\Citrix Command Center\.
Command Center initially installs in Standalone Mode. After installation, Command Center
initializes and can be used. During the first connection to the web console, administrators will
prompted to accept the EULA.

280 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
To manually start Command Center after installation, use the following script:
• <cc_home>\bin\startncc.bat

Service Mode
Command Center installs by default in Standalone mode and can be changed to a Windows Service
after installation. Running Command Center as a service (daemon) allows the application to run in
the background and automatically starts after a system restart and can be used to ensure continuous
operation.
Before Command Center is run as a service, it must first be run as a standalone application to
accept the EULA. If the EULA has not been accepted, then Command Center fails to start as a
service.
To install or uninstall Command Center as a Windows service, run the following scripts:
N

• <cc_home>\bin\InstallCCAsService.bat
ot

• <cc_home>\bin\UninstallCCAsService.bat
fo

The service is then managed through the services MMC console (Services.msc).
rr

To install and run the Command Center server as a Linux service:

es

1. Install the Linux startup service.

al

2. Set up the service using chkconfig -add NSCCService.

e

3. Start the service using /etc/init.d/NSCCService.

or

4. Run /usr/sbin/ntsysv and select NSCCService to configure the service to start on system
restart.
di

5. Stop the service using /etc/init.d/NSCCService as appropriate.

s

To uninstall the Linux service, use chkconfig -del NSCCService and rm -rf
tri

/etc/init.d/NSCCService.
bu

An administrator must uninstall the Linux service before uninstalling the Citrix Command Center
t

server.
io
n

Installation Considerations
The very first time Command Center is started should be in Standalone mode to accept the EULA.
Without the acceptance, Command Center as service will fail to start on both Windows and Linux.

If Command Center as a Service fails to start, try starting Command Center in standalone
mode. The console will display useful debug messages.

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 281
Command Center Functionality
Command Center functionality includes:
• Command Center Home Page
• Discovery
• Fault Management
• SNMP
• Monitoring
• Syslogs
• Configuration Management
• Change Management
• Centralized Certificate Management
N

• Performance Monitoring
ot
fo

Command Center Home Page

rr

The Citrix Command Center home page provides an administrator with a high-level view of the
es

performance of the Citrix network. The home page contains graphical and tabular representation of
the following statistics about the devices on the network:
al
e

Inventory A category-wise listing of all devices on the network with their

or

current alert status.

di
s

Active Alarms A graphical representation of the number of currently active alarms

tri

by their severity. Clicking any classifications on the pie chart

bu

displays alarm details with that status.

t io

Alarm Summary A list of the total alarms listed by alarm category instead of by
n

device.

Recent Alarms A list of most recent alarms with details such as date or time,
severity, category, source of alarm (system IP address) and
description

282 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
Discovery
Discovery is the process by which Command Center identifies and configures NetScaler and
CloudBridge devices deployed across the network. Automatic device backup configurations are
included, and files can be found in the database. The device icons depict standalone, high
availability pairs, and clustered devices and are color-coded to depict device state. Command Center
can also identify NetScaler SDX and VPX systems.
Devices can be added to Command Center for management by specifying the management IP
addresses (NSIP, CLIP, and SDX management IP) of the appropriate devices. These IP addresses
can be entered individually, as a list, as a range within a single Class-C subnet mask, or imported
from file. When specifying a range of IP addresses, Command Center goes through a discovery
process to determine if the IP address represents a supported device or whether the IP address
should be excluded from management. This makes it easy for an administrator to quickly add
multiple managed devices using whatever mechanism is easiest.
N

Before a device can be managed by Command Center, a device profile needs to be created which
identifies common credentials and SNMP settings to use for a group of devices. This allows
ot

Command Center to make the necessary configuration changes to manage and monitor the system
and to download the necessary configuration files. Devices requiring different credentials or settings
fo

can be managed using separate profiles.

rr

Finally, devices can be organized within Command Center into containers called Maps. Maps are
es

optional and may be used or not. Maps can be used to group related devices together and allows
administrators to perform operations at the map level. Maps may also contain submaps.
al
e

Adding a New Device

or

Devices are individual NetScaler, NetScaler Gateway, and CloudBridge appliances being monitored
di

and managed through Command Center.

s tri

When devices are added:

bu

• Multiple devices can be added by importing IP addresses or host names from a file.
• Devices can be separated by commas or entered on separate lines.
t io

• Devices can be manually entered individually or by supplying multiple values separated by a

n

comma directly through the interface.

• Devices can be entered by specifying the hostname, IP address, a range of IP addresses, or a
combination.
Before adding a device one or more device profiles must be created. A profile defines the
credentials, SNMP and other related settings needed to update the devices for management by
Command Center. Separate profiles can be created for managing the following system types.
• NetScaler (both MPX and VPX systems, also includes NetScaler Gateway devices)
• NetScaler SDX Platform
• CloudBridge

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 283
• CloudBridge Advanced Platform
• XenServer (XenServer management is for purposes of using command center to provision
NetScaler VPX instances on standard XenServer pools; use NetScaler SDX Platform settings to
manage SDX appliances and to provision VPX instances on the SDX platform.)
The typical settings required for a profile to manage a NetScaler system include:
• Account credentials for a superuser account (usually nsroot).
• SNMP version, community, and port details.
• SNMP credentials if configuring SNMP v3
The settings required for a profile to manage a NetScaler SDX system include:
• Account credentials for the SDX appliance administration.
• SNMP community details
• A specific NetScaler profile to use to manage the VPX instances on the SDX system. One
N

Profile should be common to all VPX instances.

ot

The settings required for a XenServer profile include:

fo

• Account credentials needing rights to create and provisioning virtual machines from a
rr

template.
• XenServer communication port (80 or 443); 443 is default.
es

• A specific NetScaler profile to use to manage the VPX instances provisioned on the XenServer
al

environment.
e

XenServer management is limited to using Command Center to provision NetScaler VPX

or

systems only. A template for the NetScaler VPX system must already exist in XenServer
and contain the word "NetScaler" in the template name.
di
s

CloudBridge is not being examined in this course.

tri

By default, the credentials used for device management are also used for both SSH and sFTP, but
bu

separate credentials can be specified if required. To specify different credentials, SNMP community
or discovery settings to different devices, an administrator should create separate profiles and apply
t

to the appropriate devices.

io
n

Command Center requires NetScaler credentials with superuser rights on the system because
Command Center updates the NetScaler configuration during the discovery process to properly
backup license and configuration files, as well as invoke various NetScaler interfaces for
configuration.

Discovery Process
Discovery is the process by which Command Center identifies and connects to devices to be
managed.

284 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
The figure shows the steps required for discovery:

Step Description
N

SNMP ping Command Center attempts an SNMP GET for a

Citrix-specific SNMP OID from the device.
ot

SNMP ping fails if the device is not a Citrix

device, if SNMP is disabled, or if the device is
fo

unreachable. Even if the SNMP ping fails,

rr

Command Center attempts the next discovery

step.
es
al

Find Citrix Device Command Center attempts a connection to the

device over SSH. If successful, Command
e

Center issues a command-line interface

or

command to verify if it is a NetScaler device. If

either the SSH connection or the CLI command
di

fails, the device is discarded as a non-Citrix

device and discovery fails without proceeding to
s tri

additional steps.
bu

Enable SNMP Command Center configures the SNMP

community. If the step fails, an issue affecting
t io

SNMP exists and is noted in the device

discovery status. The issue could be caused by
n

having SNMP disabled or too many SNMP

managers already configured. If SNMP
configuration fails, discovery still proceeds, the
failure is noted, and data gathering will be
limited until the SNMP issue is resolved.

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 285
Step Description
Add Trap Destination Command Center adds its IP address as a trap
destination on the device. Each NetScaler
system has a maximum of 20 trap destinations
(for NetScaler 9.2 and later; 10 or less for older
systems depending on version). If the limit is
reached, discovery fails. The issue must be
resolved before discovery can succeed.

Inventory Collection Command Center retrieves information about

the device using SNMP. This step will fail if
Command Center is not configured as an
SNMP manager in the "Enable SNMP" step.
During inventory collection, Command Center
N

gathers data on the device and entities on the

ot

device for monitoring. This includes identify the

services, service groups, and virtual servers
fo

configured on the NetScaler.

rr

Download Files Command Center uses sFTP to download the

es

configuration and license files of the NetScaler

device for backup and recovery. License files are
al

downloaded from the /nsconfig/license/

e

directory. The current saved configuration is

downloaded from /nsconfig/ns.conf. For
or

CloudBridge systems only the configuration files

are downloaded. These files are stored in the
di

Command Center database. The number of

s

configuration files to backup and the retention

tri

period is configurable after the initial discovery

bu

is complete.
t

Other Reasons This is an additional status that could identify

io

other potential issues.

n

With the exception of SNMP ping, Command Center ceases the discovery process if any of the
steps fail.
Once a device has been added to the configuration, the discovery process can be monitored and
checked for errors. If issues occur with discovery, an administrator should check the device status.
If issues occur with adding a trap destination, an administrator should verify the list of trap
destinations that are configured on the target device. A limit of 20 SNMP managers on the
NetScaler system exists. If the limit has already been reached, then Command Center cannot
configure itself as a trap destination on the NetScaler system.

286 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
If a trap destination cannot be added, then an administrator should view the NetScaler SNMP
configuration to check if other SNMP managers have been configured. If they have been
configured, then Command Center must be added to the list because the NetScaler system only
accepts listed devices as trap destinations.

Discovery Settings
Discovered devices are rediscovered periodically. Events depict the latest state of a device within a
map, following discovery. Discovery settings are global to the Command Center implementation.
The parameters that affect discovery behavior are:
• SNMP timeout: Default of 5 seconds
• SNMP retries: Default of 3
• Re-Discovery interval: Default of 60 minutes
N

• Status poll interval: Default of 1800 seconds

ot

To modify the discovery parameters and settings go to Administration > Settings > Discovery
Settings.
fo

A device that is identified as a Citrix device (passes SNMP ping or Find Citrix device phases) but
rr

fails discovery due to a network issue, configuration issue, or SNMP issue will appear in the
es

Inaccessible Systems list. A device may also appear in this list if it was offline during a rediscovery
event. The Discovery status for each device can be reviewed to determine the cause of the failure.
al

These failed devices can be re-discovered to return to normal management by Command Center.
e

These failed devices can be identified under Citrix Network > Inaccessible Systems.
or
di

Citrix Network and Device Management

s tri

Within a map, Command Center provides a graphical display of all included devices. From this
view, administrators can monitor, manage, and configure the device by selecting the Device menu.
bu

Each discovered device is available for monitoring and management under Citrix Network >
t

Device Inventory > Devices.

io
n

The Device Management commands include:

• Status: Discovery status shows whether all discovery steps succeeded or if status fails.
• Device Properties: shows device summary information. Information includes system summary
details equivalent to the nsinfo command. Configuration change summary such as most
recent configuration change points and shortcuts to the configuration comparison commands
and the show ns runningconfig and show saved config commands. It also displays
the current license state and configured modes.
• Invoke the NetScaler CLI and Configuration Utility: Invokes an SSH connection or a web-
connection to the NetScaler Configuration utility. Communication using these commands
occurs from the client direct to the NetScaler device. Whereas other commands are issued from
the Command Center Server to the managed NetScaler device. The Invoke CLI and

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 287
 Configuration Utility commands will fail if the client system is unable to connect to the
NetScaler management IP (NSIP or CLIP) over the required ports.
• Alarms and Events: displays alarms and events generated by the selected device
• Show Tech Support, Ping, TraceRoute: This is the same as running the commands from the
diagnostics pane under the NetScaler console.
• Unmanage and Rediscover: Commands can be used to remove the device from the Command
Center configuration or to trigger rediscovery to update device settings.
• Replicate Config and Replication Status: This command can be used to replicate the config
from one NetScaler to one or more additional systems. This command should be used with
caution as this will result in a forced clear config of the target systems and could result in
duplicate configurations on independent systems, resulting in IP Address conflicts.

Troubleshooting Discovery Issues

N
ot

Verify that the Command Center server can successfully connect to the NetScaler system using SSH
with the NSIP address specified and the correct credentials as listed in the profile. If this issue
fo

persists, then check network connectivity between the two devices.

rr

If the Command Center server is multi-homed, check which NIC and IP address it should be using
to connect to the NetScaler system, as well as the ports between two devices: Administration >
es

Settings > Server Settings.

al

If SNMP managers are configured in the devices and the Command Center IP address is not listed
as one of them, add the Command Center IP address to the list of manager IP addresses.
e
or

If the trap destination limit of 20 devices is reached on the Managed NetScaler and Command
Center cannot add itself as another trap destination, then discovery will fail. Delete any unused trap
di

destinations and add the Command Center IP address.

s

NetScaler trap limits were increased from 10 to 20 for each trap type for NetScaler 9.2 and
tri

later.
bu
t

If the logon credentials for SSH and sFTP entered in the device profile are incorrect, then enter the
io

correct credentials. Command Center must be configured with superuser permissions so either use
n

the nsroot account or setup a Command Center specific use account with superuser permissions.
If the NetScaler device is not reachable from the Command Center installation network, then ping
the device from the Command Center server to check if it is a network issue.
If there is a considerable delay in the SSH connection from the Command Center network to the
NetScaler device, then consider modifying the timeout values.

288 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
Fault Management
Command Center provides centralized management for multiple NetScaler and CloudBridge
devices. Part of this centralized management includes the ability to consolidate event and error
reporting into a single dashboard view, allowing administrators to easily identify issues in the
environment and current system states.
In addition to the dashboard functionality, Command Center can act as a central SNMP Manager
and Trap destination to view the state of all events and alarms in the environment. If necessary,
administrators can also use Command Center for issue management and tracking since alarms can
be assigned to administrators for action and resolutions may be documented.
Command Center centralizes events and alarms reported from managed devices into a single
console. In this console, an administrator can view real-time device status, as well as view, assign,
manage and configure events and alarms. Events represent incident occurrences, are generated
when a failure or fault is detected and are correlated to form alarms based on their significance.
N

Fault management features:

ot

• Display errors and events occurring in environment.

fo

• Generate alarms and configure notifications.

rr

• Manage custom views on target data based on custom criteria.

es

When managed devices are added to Command Center, Command Center adds itself to the SNMP
manager and trap destination configuration for the device. As a result, Command Center is a
al

centralized SNMP manager for all managed Citrix devices within the environment. Through
Command Center, any event or trap occurring on a managed device is received and displayed
e

within Command Center, providing administrators with a consolidated and comprehensive view of
or

the environment.
di

Command Center also centralizes the management of all Syslog reporting from NetScaler and
CloudBridge systems. Syslog provides a record of the primary audit events of all configuration
s

changes and is a troubleshooting log for general feature operation especially when using NetScaler
tri

Gateway and AppFirewall functionality. By configuring audit policies on the NetScaler, Command
bu

Center can be used as a remote syslog server.

t io

SNMP
n

Command Center acts a central SNMP Manager and Trap destination for the entire environment.
As part of the initial discovery process, Command Center is included in the managed devices list of
SNMP Trap destinations. Command Center may be used in place of or alongside other SNMP
managers in the environment.
Fault management with SNMP involves the following settings:
• Events: Conditions that have occurred on the NetScaler. Not all events are an alert or error
condition. Triggers, Severity, and views may be configured.
• Alarms: Alerts that have occurred on the NetScaler. Triggers and view may be configured.
Alerts may be assigned to administrators for handling.

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 289
• My Assignments: Specific list of alerts assigned to the logged on user in Command Center.
This acts as an issue tracking and management functionality.

Events
An SNMP trap is a notification event issued by a managed device to the SNMP manager when a
significant event occurs. This significant event can include numerous types of events and include an
outage, fault, issue or security violation.
Traps include:
• Notifications that the NetScaler configuration has been changed or saved
• Changes in state for services and virtual servers from UP to DOWN and from DOWN to UP
• Certificate expiration warnings
• High availability heartbeats failing
N
ot

• CPU and memory thresholds exceeding defined limits

• Power supply failures
fo

Command Center displays all events occurring within the environment and identifies the source,
rr

time and date, category, description and severity of the event, including Clear, Warning, Minor,
Major, Critical, Info and Unknown. The severity of events can be modified and set based on the
es

needs of the environment.

al

The events themselves do not necessarily indicate alarms. However, Command Center can correlate
e

events to alarms and allow administrators to define custom alarm conditions for events.
or

From the Fault tab, administrators have access to a complete list of the current network events and
alarms. Events and alarms can be assigned to specific Command Center users, and these
di

assignments are displayed in the My Assignments list.

s tri

Alarms
bu
t

Alarms are generated from events based on the impact of the event to the environment, its
io

criticality and the need for a response. Alarms identify events and issues occurring in the
n

environment that require a response to either resolve an issue that is occurring or to prevent an
issue before it can occur.
Command Center can display traps based on the NetScaler device SNMP configuration and
additional performance monitoring thresholds and alert triggers configured on the Command
Center itself.
• Any configured SNMP trap on the NetScaler system
• Triggers configured on the Command Center server for certain performance conditions and
other identified conditions.
Alarms can be assigned or picked by administrators. Alarms may be specifically assigned to a
Command Center user. Once assigned, the alarm is only available to this user until resolved or

290 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
reassigned. Picking an alarm allows an administrator to review and annotate the alarm, but the
alarm is still visible by other administrators and within Command Center. Alarms include details
regarding the severity of the alarm, source and object from where the alarm originates, as well as an
alarm description and details.
Custom views give the ability to create custom views for events of interest. By customizing views,
an administrator is able to focus on specific aspects of an alarm.

Annotations
Alarms can be annotated with notes that are tracked and included in the alarm history, which helps
track resolutions for current issues and provides a reference for future issues. Annotations can be
exported to a file. The alarms generated can be customized by creating alarm triggers.
N

Event and Alarm Triggers

ot

The ability to filter events and alarms is based on certain parameters and actions. Triggers may be
fo

configured by creating a filter for a specific condition and then specifying an action to take. These
rr

actions can include: Send email Action, Suppress Action, Run command Action, and Send Trap
Action.
es

A message can be included with the trigger which be displayed as the description or cause of the
al

alert. File attachments may also be incorporated.

e
or

Monitoring
di

Command Center provides real-time monitoring of virtual server state and health, allowing
s

administrators to track the state of virtual server objects across an environment. Administrators can
tri

view virtual server details, monitor services and service groups that are bound to a virtual server
and can view audit trails for the enable and disable operations on virtual servers, services and
bu

service groups on managed NetScaler systems, as shown in the figure.

t io
n

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 291
 N
ot

Bound services and service groups can be viewed, enabled, and disabled from Command Center.
fo
rr
es
al
e
or
di
s tri
bu
t io
n

292 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
Syslogs
Syslog on NetScaler devices is the audit log for the system. It contains a record of all configuration
changes made and identifies the administrator, client IP, change, and timestamp of change. In
addition, as an audit log, various features also record actions that are taken against traffic. For the
NetScaler Gateway, user authentication events, authorization actions, and connection details are
contained within the log which allows Syslog to audit usage of SSL VPN and NetScaler Gateway
resources but it can also be used to troubleshoot unexpected connectivity issues. The AppFirewall
similarly audits any and all application violations that occurs and the action taken. Other features of
the NetScaler that used the default (formerly advanced policy engine) can also be configured to log
to syslog when policy actions are applied, if custom log messages are configured. This can allow the
NetScaler syslog to both record these events and to be used for troubleshooting a wide range of
features from NetScaler Gateway, AppFirewall, Responder, Rewrite, and other features on the
NetScaler.
To configure Syslog, administrators must configure audit policies on the NetScaler with the
N

command center IP address as the syslog server (action) destination IP address. The audit policy
ot

must then be bound to the appropriate entity for information to be sent to the NetScaler: global
system, global VPN, virtual server (lb, cs, vpn, authentication, or other), AAA groups, and AAA
fo

users.
rr

To capture everything that goes to the regular syslog file (/var/log/ns.log), bind to the global system
object.
es

Audit policies can be configured on the respective NetScalers or use the built-in configuration task
al

included within Command Center:

e

Configuration > Configuration > Built-in Tasks: NSConfigureSyslogServer

or

An administrator can view the Syslog events per NetScaler or create custom views that can filter the
syslog events for certain devices and specific message types.
di
s tri

Configuration Management
bu

With centralized configuration management, an administrator can:

t io

• Setup one or more commands to be run.

n

• Configure one or more managed devices through a single action using either built-in or custom
tasks.
• Perform upgrade and downgrade operations on multiple systems with built-in tasks.
• Run tasks immediately or schedule to run them later.

Built In Tasks
Command Center includes several built-in configuration tasks for the NetScaler system that cannot
be added, modified, or deleted, as shown in the figure. Built-in tasks include:

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 293
• Configuring filter policies
• Configuring compression policies
• Configuring syslog server (audit policies)
• Installing SSL certificates
• Importing application templates
• Upgrading and downgrading software
Upgrade and downgrade actions are only performed in the built-in tasks. Custom tasks should not
be used for NetScaler upgrades.
N
ot
fo
rr
es
al
e
or
di
s tri

With built-in tasks, an administrator can:

bu

• Schedule configuration tasks for later execution

t

• View a list of scheduled events for a given task

io

• Stop and resume scheduled tasks

n

• View the latest task state and individual commands in a task under execution with the
execution log
• Export built-in tasks to use as a template and to create new custom tasks

Custom Tasks
Custom tasks define tasks to make configuration changes across device groups. They can be
created, modified, and deleted. Custom tasks can be:
• Run on demand or scheduled

294 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
• Involve any combination of NetScaler command-line interface, shell, or sFTP commands
• Created by importing a task or command file or defined and built within the Command Center
interface

Creating Custom Tasks from a Task File

Task files are XML files that define a task. Task files can be generated by exporting an existing task
from Command Center and using these tasks as templates to create or modify new tasks. When
exporting a task in Command Center, the file is exported to the
<CC_Home>\provisioningtemplates\exportedtemplates\<filename>.xml
location. Files may be selected through the server location
/provisioningtemplates/exportedtemplates if the Command Center client is run
from a remote system.
N

Creating Custom Tasks from a Command File

ot
fo

A command file is a basic text file consisting of a list of NetScaler commands, which can be part or
all of an ns.conf command or a set of manually compiled commands. Each command must be a
rr

NetScaler command-line interface, shell or sFTP command call.

es

For example, the following commands can be added to the text file CmdScript:
al

add service svc_red 192.168.20.15 HTTP 80

e

add lb vserver lb_vsrv_color HTTP 192.168.30.10 80

or

bind lb vserver lb_vsrv_color svc_red

save ns config
di
s

This file is then imported into Command Center and used as a template for a custom task. The task
tri

may be further modified after it is imported, including changing the actual service IP address or
replacing the virtual IP address with a variable within the task. The file must be located on the local
bu

system or on the Command Center server to upload into Command Center.

t io

Creating New Tasks

n

New tasks can be created within the Command Center interface and are built by specifying the
commands.
Running Shell commands in a task script is the same as configuring a NetScaler system from the
command-line interface. For example:

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 295
 CLI: show ns license
Shell: shell
cp /var/testlicense.lic /nsconfig/license/
exit
CLI: save ns config

Running shell commands using the inline shell commands to remain at the command-line interface
level is not effective and can result in commands timing out within the Command Center script.
For this reason, the shell [(command)] format should not be used.

show ns ip
shell "echo \"this is a test\" > /nsconfig/test.txt"
show ns feature
N

Task Operations
ot
fo

Task operations options include:

rr

• Run Sequentially
• Run on Inaccessible System(s) also
es

• Enable RBA (role-based authorization)

al

• Enable Auto-Rollback
e

• Save configuration on success

or

Role-based authorization is affected by task-level settings, whether RBA is disabled by default

within Command Center, and by global settings affecting task execution with user credentials. An
di

administrator must determine whether tasks are run using the Command Center's device profile
s

credentials or whether users must explicitly supply credentials to run tasks on devices. RBA
tri

requires that a user execute tasks against managed devices using their individual Command Center
bu

credentials instead of the profile credentials used by Command Center.

t io

Execute Sequentially
n

Setting determines whether tasks are executed across all devices in parallel or on one device at a
time.
• When disabled, tasks are executed on a set of devices. If task fails on one device, task continues
to execute on other devices in set.
• When enabled, task is executed on one device at a time. If task fails on any device, execution
does not continue to the next device.
• By default, execute sequentially is disabled.

296 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
Execute on Inaccessible System
The execute on an inaccessible system setting determines whether a custom task is run against
inaccessible systems. If a task is applied to a device list that includes both available and inaccessible
devices, then:
• The task is executed on managed devices only when disabled. Inaccessible devices included in
the list are excluded from execution.
• The task is executed on managed and inaccessible devices when enabled.
• By default, setting is disabled.

Enable RBA
Role-based authorization (RBA) is tied to the global setting for task execution user credentials. By
N

default, RBA is disabled at the task level, and task execution user credentials is disabled at the
ot

global setting. Therefore, once a user logs into Command Center, all tasks are executed using the
same credentials specified during the device discovery process within the discovery profile.
fo

Both the RBA and task execution user credentials settings are implemented to ensure that only
rr

authorized users execute tasks against devices. If either RBA or task execution user credentials is
enabled, then users must supply credentials when configuring a task to execute, supporting auditing
es

and delegating administration. Users can only apply tasks to devices to which they have access.
al

If the global RBA setting is enabled, then users are required to supply credentials when executing or
e

scheduling a task, regardless of whether the task-level RBA setting is enabled or disabled. If the
global RBA setting is disabled:
or

• Tasks execute using the discovery account if task-level RBA is disabled. Users are not prompted
di

to supply credentials.
s

• Role-based authorization is enforced if task-level RBA is enabled. Administrators must supply

tri

user credentials to execute or schedule the task.

bu

Enable Auto Rollback

t io
n

Auto rollback is only supported on NetScaler versions 8.1 and later. Command Center detects the
versions and determines appropriate action if the option is enabled. The rollback process ensures
that if a command within a task fails, then the entire task is rolled back and all configuration
changes are undone. If auto-rollback is enabled, then Command Center automatically determines
the required rollback commands needed to reverse an action if an error occurs during task
execution.
If the option is disabled, then rollback commands should be entered for each command if rollback
support is required. If the option is enabled, then Command Center generates the necessary
rollback commands. This task:
• Must include at least one command-line interface command, or else enabling Auto Rollback
generates errors.

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 297
• Generates errors if it is run against a device for which Auto-Rollback is not supported and is
only used with NetScaler 8.1 or later.
A preview function displays the auto-generated rollback command prior to task execution for
review.

Executing Custom Tasks

To run a task, an administrator:
• Chooses whether to execute or schedule a task.
• Selects device or device list tasks of which to be applied.
• Provides user input values for any variables if required.
• Specifies the date and time for scheduled tasks, or runs the task.
N

The procedures for executing or scheduling custom tasks is the same as executing or scheduling a
ot

built-in task.
fo

Change Management
rr
es

With large deployments coupled with complex configuration, making efficient configuration
changes and monitoring the effects are a challenge. Change management:
al

• Detects conflicts and mismatches.

e

• Uses audit templates to define and expected configuration and correlates events with
or

configuration changes.
di

• Reports what, by whom, and when the changes were effected.

s

• Suggests corrective commands to resolve configuration conflicts.

tri

Within Command Center, change management is handled by defining audit templates that are used
bu

to compare expected configuration settings with the configuration of managed NetScaler devices.
Therefore, changes to the configuration can be identified and tracked throughout the environment.
t io
n

Management Key Components

Change Management compares the current running configuration of a device with either the
expected configuration defined within an audit template or the saved configuration. Therefore,
Change Management identifies changes to the NetScaler configuration. The following table lists
management components.

298 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
Component Description
Audit Templates Audit templates define the expected settings that
should be present in the NetScaler ns.conf file.
The template is a subset of an ns.conf file.

Audit Policies Audit policies define the type of configuration

comparison that should be made and the
reports to run. Policies determine what to
compare to the running configuration, such as
one or more audit templates or the saved
configuration. Options include:
• Running configuration versus the chosen
audit templates, where multiple instances
may be specified.
N

• Running configuration versus the saved

ot

configuration, where differences in the

fo

running and saved configuration are

returned.
rr

A single policy may include both types of

es

reports running configurations versus templates

and saved configurations.
al
e

Audit Reports Audit reports include a summary of the results

when policies are executed or scheduled to run
or

against different devices. Results are listed per

device.
di
s tri

Audit Report Work-Flow

bu

The work-flow to generate an audit report consists of:

t io

1. Defining one or more audit templates.

n

2. Creating audit policies and choosing the desired reports that need to be generated.
3. Executing or scheduling the policy.
4. Viewing and analyzing the audit report.

Audit Element Description

Audit Templates A template that monitors the sysname value of a
NetScaler system. It is assigned to devices when
the audit policy is configured.

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 299
Audit Element Description
Creating Audit Policies A policy is created that generate reports based
on comparing the running configuration to the
audit template named
change_sysname_template.
Comparisons across multiple templates may be
configured. The running versus saved
configuration can also be included as part of
this policy or in a separate policy.

Running Audit Policies Command Center includes two pre-defined

audit policies, which are
RunningVsSavedConfiguration and
N

ConfigurationChangeHistory.
ot

Any additional policies are displayed in the

Audit Policies list. Policies can be run as needed
fo

through the execute policy option, and policies

or policies can be scheduled to execute on
rr

specific days and times. Scheduling audit

es

policies is useful to monitor changes in the

environment on a regular basis.
al
e

Applying Policies to Devices The devices to be audited are specified when

choosing to execute or schedule a policy.
or

Devices are specified individually or by selecting

device lists. Device lists can be created at the
di

time they are needed.

s tri

Audit Policy Results Reports Once an audit policy has completed execution,
bu

the results are available in the Reports section.

Results are listed by policy and then by device.
t

Within the Reports section, the policies that are

io

run are listed by the start and end times. A

n

result indicates if a change exists. When an

administrator selects a policy, a list of the
results per device are displayed. The results of
each device, including the exact lines that are
different in the running configuration versus the
template or saved configuration are displayed
with suggested commands to correct the issue.

300 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
Reports
The figure shows a sample report of device details.

N
ot
fo
rr
es
al
e
or
di
s

Centralized Certificate Management

tri
bu

The Certificate Manager:

t

• Displays certificates (certkeys) across all devices

io

• Displays the days to expiration and indicates severity level

n

• Updates certificates and generates a CSR output file that can be sent for signing
• Provides a centralized view of SSL certificates installed across all managed devices
• Generates events when a certificate nears expiration based on a threshold
Selecting Poll Now refreshes certificate information on all devices. Command Center supports the
certificate manager feature on NetScaler 7.0.52 or later only.

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 301
Threshold Configuration
Severity levels notify administrators when certificates approach expiry. Values may be modified.
Certificates are polled from managed devices once per day; polling interval is also configurable.
Default certificate severity levels include:
• Critical: Expires within 0 days
• Major: Expires within 7 days
• Minor: Expires within 30 days
• Warning: Expires within 90 days

Performance Monitoring and Reporting

N

Performance monitoring monitors the health of a device by polling the performance counters
supported by a device.
ot

Performance monitoring:
fo

• Generates a report and plots a chart

rr

• Creates a highly customizable reports

es

• Applies aggregate functions on counters

al

• Defines and sets thresholds

e

• Generates events
or

• Sets severity levels and event messages

• Sets an event to a clear value
di

With NetScaler systems, over 300 counters are available to monitor performance, including
s

statistics related to entities, requests, responses and traffic performance. The polling intervals for
tri

counters can be set to determine how much data is gathered within the environment. The degree of
bu

granularity of the performance data can be increased or decreased by setting the polling interval.
Command Center gathers and consolidates the data across all managed devices in the environment
t

and then provides administrators with reporting tools to review and assess the environment.
io
n

In addition, Command Center provides dashboard functionality specific to AppFirewall and

NetScaler Gateway and additional reporting for AppFirewall violations. Entity monitoring also
provides a status view of all NetScaler service, service group, and virtual server entities.

Configuring Performance Monitoring

Use the following procedure to configure performance monitoring:
1. Within Command Center, go to Reporting.
2. Configure Performance settings and configure polled counters (Performance node top-level).

302 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
 Scalar counters are enabled by default; additional counters may be enabled. Only configured
parameters are gathered. Vector counters are entity-specific; the more entities on the NetScaler
the more data that is gathered. Default polling interval is 5 minutes (300 seconds); minimal
polling interval is 30 seconds.
3. Configure the reports to be generated.
Quick reports are used when specific counters are selected, and a report is generated. Custom
reports are used when specific counters are selected and the configuration is saved. A report is
run to view metrics and graph performance.
4. Schedule reports to run at regular intervals.
The frequency of the report must be configured, a CSV or XML format specified and the
report emailed.
5. Configure thresholds if required.
Thresholds allow Command Center to assign a severity and message to the condition; events
N

will appear within the fault list in Command Center. An email notification can be triggered in
ot

the event a specific performance condition has been encountered.

fo
rr

Configuring Counters for Polling

es

The figure shows where an administrator can enable and disable counters for polling. The five
al

minute default polling interval is configurable and includes scalar and vector counters.
e
or
di
s tri
bu
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 303
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

304 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
Counter Types
Scalar counters:
• Are enabled for polling by default
• Are device-level
• Include TCP and UDP counters
• Are listed by name, for example: Compression, DNS and GSLB
Vector counters:
• Are disabled by default and must be explicitly enabled
• Are entity-level counters that display statistics for entities, including interfaces, virtual servers,
services, and service groups
• Impact Command Center performance for a large number of entities
N

• Are denoted within a list by a + sign following the name, such as CPU Usage + and Cache
ot

Redirection Policies +
fo

Polling Counters
rr
es

The minimum polling interval is 30 seconds, and the default polling interval is 300 seconds (5
minutes). When choosing the polling interval, an administrator should take into consideration the
al

number of devices, the number of counters selected for polling and the amount of network load
e

that might result. If there are a large number of devices and entities in the environment, a higher
(less frequent) polling interval should be configured.
or

Polling counters support frequent polling intervals for a limited number of counters for which a
di

high degree of granularity is required. Counters are configured for polling within the Custom
s

Reports node.
tri
bu

Data Consolidation
t io

Command Center collects performance data from each device at a 5 minute default interval. The
n

consolidation intervals are configurable in the 3.2 release. The figure shows a sample data chart.

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 305
 N
ot
fo
rr
es
al
e
or
di
stri

Command Center maintains one month of hourly consolidated data, one year of daily consolidated
bu

data and 14 days of granular polled data. Consolidated data can be viewed with trend reports.
Command Center also maintains three months of events at 250 bytes each. The polling interval can
t io

be increased or decreased as necessary. However, this level of granular performance data can
generate large volumes of data over long periods of time.
n

Trend Reports
When generating trend reports, administrators can:
• Review data by hour for the previous month.
• Review data by month for the last year.
• Analyze minimum, maximum and average trends.
• View results in graphs.

306 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
• Generate reports in PDF.
Trend reports are consolidated historical data reports that allow an administrator to analyze
performance trends over time and offer extensive trend analysis capabilities for monitoring
NetScaler devices that are customizable.
Using this feature, an administrator can:
• Select multiple counters from one or more discovered devices.
• Aggregate counters across multiple devices.
• Generate daily, weekly, or monthly reports based on aggregated historical data.

Thresholds
Thresholds can be configured on any counters being polled. They help with monitoring counters
N

and generating alerts when the counter exceeds a set value. Thresholds generate events and alerts
based on performance monitored metrics.
ot

A threshold can be associated to one or more counters. Events are generated when a threshold is
fo

exceeded, and these events are then available as part of fault management.
rr
es

Performance Data Issues

al

If the No polled data found error message occurs, possible reasons include:
e

• The selected counter is not enabled for polling.

or

• The selected counter is deprecated for this version of the device.

di
s

Command Center Administration

tri
bu

Command Center administration includes:

t

• Security administration
io

• Administration operations
n

• Administration configurations
• Server details
After installation, Command Center settings and administration are configured through the Admin
tab within the Command Center interface.

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 307
Security Administration
An administrator can secure access to managed devices by securing access to Command Center.
Authorization settings can be configured to ensure that users only have access to tasks and
resources that they are allowed to manage and configure.
To secure administration, an administrator should configure:
• User administration
• Group administration
• Audit logs
Authorization is:
• Simple because it specifies the operations that a group is allowed to perform
• Fine-grained because it limits the data on which a specified operation can be performed
N

Authentication includes:
ot

• Local authentication
fo

• External authentication with Active Directory, RADIUS and TACACS+

rr
es

User Administration
al

With user administration, an administrator can add and manage user account that log on to
e

Command Center. An administrator can set password expiration rules and add users to one or
more groups to determine permitted operations.
or
di

Authentication
s tri

Authentication is configured in Administration > Security. Administrators can configure the

bu

external or local authentication settings, create user accounts, and add accounts to groups.
Permissions within command center are assigned at the group level.
t io
n

Group Administration Overview

An administrator can add and manage groups and group memberships, as well as their operations.
By default, two groups are created within Command Center, Users and Admins. Admins have full
control over all operations, and Users are not authorized to perform operations. Additional groups
can be created to support granular control over operations and to support limited or delegated
administration.

308 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
 N
ot
fo
rr
es

Administration Configuration Authentication

al
e

Authentication configuration is used in conjunction with authorization settings, such as the security
or

admin task group, to manage access to Command Center and its managed devices.
In the authentication node, administrators configure whether to use local authentication through
di

accounts managed in Command Center or to use external authentication with:

s tri

• Active Directory, including server name, server port and domain

bu

• RADIUS, including server name, server port, secret key, client IP address, client identifier and
password encoding
t

• TACACS+, including server name, server port, secret key and password encoding
io
n

Details on configuring the various authentication services are provided in the Command Center
Administration Guide.

Group Administration
Operations can be configured through the Assigned Operations link. Tasks are applied to groups,
and administrators have granular control over task assignments.
Benefits of group administration include:
• Simple, fine grained and custom view scope that are based on groups

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 309
• All users in a group having a common set of operations available
• Time saved in administration

N
ot

The figure shows that when a user tries to configure a device, the user does not have the necessary
authorization.
fo
rr

Command Center Administration Node

es

The Command Center Administration node contains settings to allow administrators to configure
al

how Command Center operates, including:

e

• Security: Configuration of command center delegated administration, external authentication

or

settings, and group permissions.

• Information: Displays summary of Command Center Server information and currently logged
di

on users.
s

• Tools: Allows administrators to install certificates on Command Center to support secure

tri

communication to the Command Center webclient, to update and manage the database
bu

password, and to generate command center support logs for troubleshooting.

• Agent Setup: Allows administrators to configure Command Center Agents associated with
t io

the Server deployment and to assign managed systems to agent.

n

• Device Profiles: Configure new and edit existing device profiles which can then be used
when discovering devices for management.
• Device Lists: Create management groups of associated devices so settings can be more
easily applied to related devices collectively.
• Logging: View Logs includes a display of all Command Center logs which can be used for
troubleshooting and the Logs Settings which can be used to control the Command Center log
generation such as lines/file, rollover file count, and the log level detail.

310 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
Additional Administration Tasks
Command Center database management should be conducted using the appropriate MySQL,
MSSQL, or Oracle database management tools as these settings cannot be managed from within
Command Center. If using the evaluation edition with the included PostGRESQL database, limited
settings can be managed from the PostGRESQL tools. This functionality is not directly exposed in
Command Center console.
Commands to Start and Stop Command Center after the initial installation is performed using
scripts within the <CC_Home>\bin directory. This functionality is not exposed in the Command
Center server console. Start and Stop procedures vary if running Command Center as a service as
opposed to a standalone process.

Option Description
N

Server Shutdown / Startup (Standalone) Stopping the Command Center process prevents
ot

gathering of data. Scripts are located in the

<CC_Home>\bin\ directory. From a
fo

command prompt, run the following commands

to start or stop the process. Scripts must be run
rr

from the <CC_Home> directory, so use the

es

change directory command (cd) first and run

from a command prompt.
al

Shutdown Command Center

e
or

cd <CC_Home>\bin\
di

shutdown.bat
stri

Start Command Center

bu

cd <CC_Home>\bin\
t io

startncc.bat
n

The startup script will display a status summary

of all services, so if issues are encountered
during startup additional troubleshooting can be
performed..

Server Shutdown / Startup (Service) The Command Center service can be started
and stopped through the Windows Services
console. From a command line, the same
startup and shutdown scripts can be used as
with the standalone process.

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 311
Option Description

Start/Stop PostGRESQL If running the evaluation edition of Command

Center, the PostGRESQL service can also be
started and stopped from the command line.
This service needs to be running when starting
Command Center. If database errors are present
during startup, ensure the database is started
using the following commands:
Start/Stop PostGRESQL:

cd <CC_Home>\bin\

startPostgresql.bat
N
ot

cd <CC_Home>\bin\
fo

stopPostgresql.bat
rr
es

Administration Configuration
al

This administration configuration task group includes options for configuring Command Center
e

server operations. Settings here affect the Command Center server installation and global settings.
or

The Administration tab also contains a list of Settings to control various aspects of Command
di

Center operation:
s

• Discovery Settings
tri

• Inventory Settings
bu

• Server Settings
• High Availability Settings
t io

• Mail Server Settings

n

• Access Settings
• Trap Forward Settings

Discovery Settings
• Inventory Settings
• Server Settings
• High Availability Settings
• Mail Server Settings

312 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
• Access Settings

Trap Forward Settings

The following table lists the configurable settings for Command Center under Administration >
Settings.

Setting Description
Discovery Settings
Discovery parameter defaults include timeouts
for SNMP responses during initial discovery
process:
N

• SNMP timeout: 5 seconds

ot

• SNMP retries: 3
• Re-discovery interval: 60 seconds
fo

• Status poll interval: 1800 seconds

rr

Configure Inventory Settings Command Center archives the configuration

es

ns.conf, SSL certificates, and license files for

every discovered NetScaler devices. settings
al

allows administrators to control when archiving

e

occurs.
or

Settings that can be configured and defaults

include:
di

• Archive on receiving the save config trap

s tri

default: Disabled
bu

• Archive on every schedule interval in hours

default: Default: 12 hours
t

• Number of previous archive files to retain

io

default: 50
n

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 313
Setting Description
Configure Server Settings These settings represent the SNMP details and
retention periods for performance monitoring.
SNMP Settings:
• SNMP Trap Destination and Port: Defaults
to command center IP.
• Device Label: System IP or hostname.
• SSL Certificate management: enable
gathering of certificates or exclude.
• Task Execution User Credentials: This is
the global RBA setting; if this is enabled
than all tasks will require user permissions
N

to execute via command center.

ot

• Monitoring: Enabled/Disabled.
fo

• Syslog Clean interval (in days): 90 days

rr

Performance Durations: Adjust retention

periods for default performance data gathering.
es

• 5 minute interval duration: 14 days

al

• Hourly interval duration: 30 days

e

• Daily consolidated duration: 365 days

or

Be careful if increasing the performance data

retention periods as this may cause an increase
di

in storage requirements for all performance

s

monitoring data.
tri
bu

High Availability Settings Configure and adjust High Availability settings

when in an HA Pair.
t io

Mail Server Settings An administrator can configure SMTP settings

n

to support email alerts and notifications that are

tied to alarms within the environment.

Access Settings Change or modify the Command Center

listening ports for the web client. Default is
HTTPS:8443 and HTTP:9090. The Protocol and
port can be changed here. Changes require a
restart of the Command Center Server.

314 Module 10: Command Center © Copyright 2016 Citrix Systems, Inc.
Setting Description
Trap Forward Settings Configure SNMP Trap forwarding to another
SNMP destination. This allows alerts received
by Command Center to be sent to an additional
SNMP destination.

N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 10: Command Center 315
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

316 © Copyright 2016 Citrix Systems, Inc.

11
n
t io
bu
stri
di
or
e
al
es
rr
Insight Center
fo
ot
N
Module 11
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

318 © Copyright 2016 Citrix Systems, Inc.

Insight Center Manual
Insight Center Overview
NetScaler Insight Center is a virtual appliance that is installed on a hypervisor. Insight Center
collects detailed information about web and virtual desktop traffic that passes through a NetScaler.
Insight Center collects data from a single or multiple NetScalers, is available as a XenServer XVA,
has no external dependencies and has built-in persistent storage.
Insight eliminates the need to collect metrics with 3rd party tools that use other means of collecting
this data. By leveraging the built-in support for the AppFlow protocol and EdgeSight Monitoring
(HTML injection) capabilities, there is no need to install agents, take packet captures, or other
intrusive techniques as the NetScaler can send statistics in real-time for the traffic that passes
N

through the NetScaler providing complete visibility. All of this happens transparently and in real-
time with no network interruption or additional load on your backend infrastructure.
ot

Using NetScaler Insight Center in conjunction with NetScaler AppFlow and EdgeSight Monitoring
fo

(HTML injection), we can generate a wealth of application visibility information, not only for web
application deployments in a cloud or enterprise environment, but also, for published resources
rr

delivered by our Desktop Delivery infrastructure.

es

Insight Center has two components:

al

• Web Insight – monitors HTTP, SSL, and TCP traffic passing through a load balancing and
e

content switching virtual server defined on the NetScaler appliance. Web Insight supports
HTTP, SSL, and TCP virtual servers.
or

• HDX Insight – monitors ICA traffic passing through NetScaler Gateway virtual servers defined
di

in the NetScaler appliance.

s

HDX Insight does not support standalone Access Gateway appliances.

tri
bu
t io

Insight Center can be installed on the following platforms:

n

• Citrix XenServer 5.6 or later

• VMware ESX 4.1 or later
For detailed information about installing Insight Center, see the product documentation for
Installing NetScaler Insight Center at http://docs.citrix.com/en-us/netscaler-insight/11-0/installing-
insight-center.html.

Objectives
At the end of this module, you will be able to:

© Copyright 2016 Citrix Systems, Inc. Module 11: Insight Center 319
• Monitor the NetScaler system using Insight Center.
• Describe how Insight Center collects AppFlow data.
• Explain how to rewrite content in HTTP responses and requests with HTML injection.

AppFlow on the NetScaler System

Why use AppFlow?
N
ot
fo
rr
es
al
e
or

AppFlow is a new standard for application monitoring and reporting that does not require network
taps or span ports. It is an extension of an existing protocol, IPFIX, already used by multiple
di

vendors in the industry to report vital metrics and other statistics about the network. AppFlow
takes it a step further and includes application specific information and packages it in the same
s

IPFIX format for it to be sent to a centralized collector for further processing and reporting.
tri
bu
t io
n

320 Module 11: Insight Center © Copyright 2016 Citrix Systems, Inc.
 N
ot
fo

The NetScaler system is a central point of control for all application traffic in the datacenter. It
rr

collects information about flow and about the end user's session that is valuable for application-
es

performance monitoring, analytics, and business intelligence applications. It also collects web page-
performance data and database information. AppFlow transmits the information by using the
al

Internet Protocol Flow Information eXport (IPFIX) format. IPFIX is the standardized version of
Cisco's NetFlow and is widely used to monitor network flow information. Using UDP as the
e

transport protocol, AppFlow transmits the collected data, called flow records, to one or more IPv4
or

collectors. The collectors aggregate the flow records and generate real-time or historical reports.
AppFlow provides visibility at the transaction level for HTTP, SSL, TCP, and SSL_TCP flows.
di

AppFlow uses actions and policies to send records for a selected flow to a specific set of collectors.
s

An AppFlow action specifies which set of collectors will receive the AppFlow records. Policies,
tri

which are based on default expressions, can be configured to select flows for which flow records
bu

will be sent to the collectors specified by the associated AppFlow action. To limit the types of flows,
you can enable AppFlow for a virtual server. AppFlow can also provide statistics for the virtual
t io

server. You can enable AppFlow for a specific service, representing an application server, and
monitor the traffic to that application server.
n

© Copyright 2016 Citrix Systems, Inc. Module 11: Insight Center 321
How AppFlow Works

In a common deployment scenario, inbound traffic flows to a virtual IP address (VIP) on the
N

NetScaler system and is load balanced to a server. Outbound traffic flows from the server to a
ot

mapped or subnet IP address on the NetScaler system and from the VIP to the client. A flow is a
unidirectional collection of IP packets identified by the following five tuples: sourceIP, sourcePort,
fo

destIP, destPort, and protocol.

rr

NetScaler Insight inventories load balancing, content switching, and VPN vservers and uses SSH
es

and Nitro over HTTP or HTTPS to communicate with NetScaler systems. In order to control data
collection, the NetScaler uses AppFlow using default expressions.
al

When configuring NetScaler Insight, the system can automatically enable the required features on
e

the NetScaler such as rewrite, EdgeSight monitoring (HTML injection), and AppFlow.
or

Once AppFlow has been configured you need to create AppFlow policies so the information is
forwarded to the Insight Center for analysis.
di
s tri
bu
t io
n

How Insight Center Collects AppFlow Data

Insight Center retrieves AppFlow logging information for each monitored application, analyzes the
information, and presents it as visual reports. To enable data collection, you must enable AppFlow

322 Module 11: Insight Center © Copyright 2016 Citrix Systems, Inc.
on each virtual server from which you want Insight Center to retrieve the data. Services bound to
those virtual servers must also have AppFlow enabled.
The NetScaler appliance is usually the central point where traffic flows through the network making
it an ideal place to collect network flow information. The AppFlow feature in a NetScaler appliance
uses the Internet Protocol Flow Information eXport (IPFIX) format to transmit network flow
information. AppFlow provides visibility at the transaction level for HTTP, SSL, TCP, and
SSL_TCP flows. AppFlow uses actions and policies to send records for a selected flow to specific set
of collectors. An AppFlow action specifies which set of collectors will receive the Appflow records.
Policies, which are based on Advanced Expressions, can be configured to select flows for which
flow records will be sent to the collectors specified by the associated AppFlow action.
AppFlow NetScaler Integration supports up to four collectors and uses UDP port 4739. Each
AppFlow packet contains sequence number of flow records which helps the collector detect packet
drops/out of sequence and templates built for end user specific information.
AppFlow defines a set of templates, one for each type of flow. Each template contains a set of
N

standard Information Elements (IEs) and Enterprise-specific Information Elements (EIEs). IPFIX
ot

templates define the order and sizes of the Information Elements (IE) in the flow record. The
templates are sent to the collectors at regular intervals, as described in RFC 5101.
fo

Some of the metrics collected by Web Insight:

rr

• Number of hits received by the NetScaler

es

• Bandwidth of the NetScaler

al

• Number of hits received by an application

e

• Bandwidth used by an application

or

• Response time for an application

• Number of hits received for a URL
di

• Number of requests sent by a client

s

• Latency of the client-side network

tri

• Number of hits received by the virtual servers

bu

• Processing time on a virtual server

t

• Number of hits by HTTP request method

io

• Number of hits by HTTP response status

n

Some of the metrics collected by HDX Insight:

• Latency of the client-side network
• ICA RTT
• Bandwidth used in an ICA session
• Total number of active sessions in a given time interval
• Total number of applications active during a given time period
• Average rate at which data is transferred over the ICA session
• Total number of unique user sessions in a given time period

© Copyright 2016 Citrix Systems, Inc. Module 11: Insight Center 323
• Licenses in use for SSL VPN and ICA traffic

HDX Insight
HDX Insight allows administrators of Citrix XenApp and Citrix XenDesktop environments a way
to monitor the users and performance of the applications hosted on those products. HDX Insight
captures data about the ICA traffic that flows between the clients and the servers, generates
AppFlow records by doing deep packet inspection of the data, and presents the records as visual
reports on the Insight Center dashboard.
HDX Insight currently supports collecting AppFlow data from NetScalers in single-hop mode. In
single-hop mode, users access the NetScaler through a virtual private network, such as the
NetScaler Gateway.
N

HTML Injection
ot

NetScaler Edgesight Monitoring (HTML Injection) is a form of content rewriting that allows you to
fo

inject data into chosen HTTP responses served by the NetScaler to a client. These responses can be
rr

chosen on the basis of parameters in the HTTP request or response or both. The injected data can
then be used by Insight, to measure web site performance.
es

Data can be injected either before the response body or after the response body. Your choice
al

depends on when you want the data to be analyzed. Pre-body content is the first content that the
client reads in the HTTP response, and post-body content is the last content that the client reads.
e

To inject data into the HTTP response body, you create pre-body or post-body files, depending on
or

where in the HTTP response body you want to inject data. These are JavaScript files that make use
of the NetScaler predefined variables. The files are run on the client, in the client browser (provided
di

that JavaScript is enabled within the browser).

s tri

Before configuring HTML Injection, make sure that it is enabled, and configure files containing the
data to be injected. Then, create actions to inject the data. Also create policies to select the
bu

responses that will be affected. As you create a policy, you associate it with an action. To complete
the configuration, bind the policies to the virtual servers that are to inject the data.
t io

The figures show examples of the Edgesight Monitoring and Responder Policies:
n

324 Module 11: Insight Center © Copyright 2016 Citrix Systems, Inc.
 N
ot
fo
rr

Once configured the results are sent by the client to the Web Insight enabled VIP and responder
policy catches the request and initiates callout to Insight. Using this feature, the Insight Center can
es

report on website page load and render times.

al

The figures show JavaScript being injected both pre-body and post-body using the NetScaler
e

Edgesight Monitoring (HTML Injection) feature:

or
di
s tri
bu
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 11: Insight Center 325
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

326 © Copyright 2016 Citrix Systems, Inc.

 12
Module 12

NetScaler Web
Logging
N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

328 © Copyright 2016 Citrix Systems, Inc.

NetScaler Web Logging Manual
Overview
The NetScaler system generates log files on the system (newnslogs) for performance and events.
HTTP/HTTPS transaction data is written on the NetScaler system in a buffer only and is not logged
on a form that can be viewed or accessed. Web logging provides a mechanism for making this
transaction data available for analysis and review.

Objectives
At the end of the module, you will be able to:
N

• Identify the components within the NetScaler web logging architecture.

ot

• Discuss the web logging communication process.

• Install the NSWL client.
fo

• Configure the NetScaler system for web logging.

rr

• Verify the web logging configuration.

es

• Create filters.
al

• Configure buffer size.

e

• Troubleshoot the common web logging issues.

or

NetScaler Web Logging Introduction

di
s

Client requests are terminated at the NetScaler system, which then opens a connection to the back-
tri

end server. Client requests are typically made to a virtual IP (VIP) address on the NetScaler system,
bu

and requests to the server from the NetScaler system are made from the mapped IP (MIP) or
subnet IP (SNIP) addresses. As a result, client systems may see only the VIP address owned by the
t io

NetScaler system. Server systems may only see the MIP or SNIP address of the NetScaler as the
client IP address originating the request, instead of the actual client IP address.
n

NetScaler web logging provides a mechanism to obtain the source IP address of the client
originating the request to specific VIP addresses and the corresponding back-end server IP
addresses through the transaction logs. Transaction data acquired through web logging may be used
for traffic analysis, troubleshooting, statistical data, or for commercial or billing purposes. NetScaler
web logging also logs integrated cache transactions, same as back-end server transactions. The web
logging data identifies which transactions are served from cache.

© Copyright 2016 Citrix Systems, Inc. Module 12: NetScaler Web Logging 329
Architecture Overview

N
ot
fo
rr
es
al
e
or
di

The NetScaler system keeps track of complete web transaction log activity and is aware of the client
s tri

connection from the client IP address to the virtual IP address and the server-side connection from
the MIP/SNIP address to the actual server IP address. The NetScaler system can also track which
bu

transactions are delivered from cache. The NetScaler system tracks the web transaction log
information within a buffer only; the information is not tracked in a log file on the NetScaler.
t io

To track this data, the NSWL client can be installed on a separate server or workstation (Windows,
n

Linux, FreeBSD). The NSWL client then receives the web logging and transaction data from the
associated NetScaler and outputs the details to a log file. These log files can then be archived and
reviewed for information.

Communication Process
The NSWL client is configured with the NSIP address and credentials of one or more NetScaler
systems from which to retrieve web logging information. When started, the NSWL client initiates a
long-lived TCP connection to the NetScaler system on TCP port 3011, starting with a 3-way TCP

330 Module 12: NetScaler Web Logging © Copyright 2016 Citrix Systems, Inc.
handshake. The NSWL client connects to the NetScaler system using the NSIP address and not a
MIP/SNIP address.
After the NSWL client establishes a connection with the NetScaler system, the NSWL client
authenticates to the NetScaler system using the login credentials supplied through the NetScaler
LOGIN routine. The login is performed through the NetScaler remote procedure call protocol (NS-
RPC). The NetScaler system responds with whether the login was a success or failure.
Once the login is successful, the NSWL client notifies the NetScaler system that it is ready to
receive data by sending the LOG DATA request. While the NSWL client sends the initial
notification to the NetScaler system to send data, the NSWL client does not pull the data from the
NetScaler system. The NetScaler system pushes the data to the NSWL client as data is available
within the buffer.
The size of the buffer on the NetScaler system and the hardware running the NSWL client are
important to ensure that the NSWL client can keep up with the transaction data. If the NWSL
client is unable to keep up with the transaction log, then a buffer overflow could occur, resulting in
N

the loss of some transaction data. In this situation, to avoid loss of data, either increase the available
ot

resources of the system running the NSWL client or increase the web logging buffer size on the
NetScaler system. Refer to the Citrix NetScaler Administration Guide for minimum recommended
fo

settings.
rr

For web logging to be successful, the NetScaler system must have the feature enabled and the
NSWL client must be able to connect to the NSIP address over TCP port 3011.
es

Firewalls, proxy servers, or other devices between the NSWL client and the NetScaler
al

system can cause the connection to fail.

e
or
di

NetScaler System Configuration

s tri

Enabling the web logging feature allows the NetScaler system to push web logging data to an
authenticated NSWL client running on the logging system. The default Web Log buffer size is 16
bu

MB. The buffer size may be increased or decreased if appropriate for the environment. If the NSWL
t

client is unable to keep up with the flow of log data, increase the buffer size.
io
n

Enabling Web Logging in the Configuration Utility

Use the following procedure to enable web logging:
1. Go to System>Settings.
2. Click Change Advanced Features.The Configure Advanced Features dialog box opens.
3. Select Web Logging and click OK.
4. Click Yes in the Enable/Disable Features(s)? dialog box.

© Copyright 2016 Citrix Systems, Inc. Module 12: NetScaler Web Logging 331
Enabling Web Logging in the Command-Line Interface
Enter the following command to enable web logging:
enable ns feature WL

Configuring the Buffer Size in the Configuration Utility

Use the following procedure to configure the buffer:
1. Go to System>Settings.
2. Click Change Global System Settings.The Change Global Settings dialog opens.
3. Enter a value in the Buffer_Size (in MBytes) text box.
4. Click OK.
N
ot

Configuring the Buffer Size in the Command-Line Interface

fo
rr

Enter the following command to configure the buffer size:

es

set weblogparam -b size

al
e

NSWL Client Installation

or

Both the NetScaler system and a separate logging system running the NSWL client must be
di

configured. The NSWL client can be installed on the following platforms:

s

• Windows (XP, 2003 Server)

tri

• Linux
bu

• Macintosh
t

• Solaris
io

• FreeBSD
n

Refer to the Citrix NetScaler Administration Guide for specific operating system versions supported
and minimum hardware requirements as well as platform specific installation and configuration
instructions.

This module provides instructions specific to the Windows operating system.

The logging parameters (filters, log formats, and NetScaler system to monitor) are set in the
configuration file on the logging system on which the NSWL client is installed. The log files will be

332 Module 12: NetScaler Web Logging © Copyright 2016 Citrix Systems, Inc.
generated in a specific location depending on the host platform. Once parameters have been
configured, start the service and verify transaction data is being logged.
The NSWL client executable can be downloaded from the ftp.netscaler.com web site or MyCitrix
with appropriate credentials.

The NSWL client version/build number and the NetScaler version/build number must
match. For example, the NSWL client 9.0.66.12 would be used with NetScaler systems
running build 9.0.66.12.

Logging System Components

The following table lists the logging system components.
N

Component Description
ot

NSWL.exe NSWL client appropriate to the NetScaler build

fo

in use. The NSWL component can run as a

rr

process or as a service, depending on the OS

platform being configured.
es

Log.conf file NSWL configuration file. File includes details of

al

what to log, format to log transaction data in,

e

and the NetScaler system or systems to receive

log data from. The log.conf file includes the
or

NSIP address and authentication credentials

(superuser) to establish a connection with the
di

designated NetScaler systems.

s tri

Log files Web Logging output files which contain the

bu

details of the transaction data being sent from

the NetScaler systems. The log files capture the
t

transaction data originating in the NetScaler

io

web transaction buffer.

n

Debug files Log files that report the status of the NSWL
component on the web logging server, including
success and error messages. Debug log level can
be varied from 1 (minimal detail) to 3
(maximum detail).

Installing the NSWL Client on Windows

Use the following procedure to install NSWL on a Windows operating system:

© Copyright 2016 Citrix Systems, Inc. Module 12: NetScaler Web Logging 333
1. Copy the NSweblog.exe file (zipped executable) into a temporary directory.
The extracted files are installed in the following directories.
• \NS\BIN
• \NS\ETC
• \NS\SAMPLES
2. Run the following command from the \NS\BIN path to install web server logging as a service:
nswl -install -f directory_path\log.conf

NSWL Options
The following table describes the options for use with the NSWL client executable.
N
ot

Command Description
fo

nswl -help This command displays all available NSWL

rr

options.
es

nswl -addns -f filepath This command specifies the system that gathers
al

the log transaction data. You are prompted to

enter the IP address of the system. Enter a valid
e

username and password.

or

nswl -verify -f filepath This command checks for syntax or semantic

di

errors in the configuration file (for example,

log.conf).
s tri

nswl -start -f filepath This command starts web server logging based
bu

on the settings in the configuration file (for

example, log.conf).
t io

nswl -install -f filepath (Windows only) This command installs the web server logging
n

client as a service in Windows.

nswl -startservice (Windows only) This command starts web logging on the
logging system, using the settings in the
configuration file (for example, log.conf)
specified in the nswl install option. Web server
logging can also be started from Start>Control
Panel>Services.

nswl -stopservice (Windows only) This command stops web logging on the
logging system.

334 Module 12: NetScaler Web Logging © Copyright 2016 Citrix Systems, Inc.
 Command Description
nswl -remove This command removes the web logging service
from the registry.

nswl cmd -f filepath -d debug This command specifies a debug level of 1-3.
The debug parameter is optional and may be
omitted. The default level is 1 if none is
specified.

Windows Service Registry Key

Installing web logging as a Windows service updates the following registry key:
N

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\nswlsvc\
ot

The ImagePath value contains the path for the NSWL.exe service and the location of the log.conf
fo

file supplied during the service installation.

rr

Example path:
es

ImagePath =C:\NSWL\bin\nswl.exe -log -f C:\NSWL\etc\log.conf

al
e

While the registry could be modified manually, the preferred method is to use the following
commands to install the service and set the log.conf file path with a corrected value
or

nswl.exe -remove
di

nswl.exe -install
s

.
tri
bu

NSWL Client Configuration

t io

Web logging must be configured after it is installed. The settings configured on the NetScaler
n

system include whether the web logging feature is enabled or disabled and the size of the web
logging buffer.
The NSWL client, which is running on a logging system external to the NetScaler system, is
configured with the NetScaler IP (NSIP) address of one or more NetScaler systems and the system
credentials to authenticate to the NetScaler with. The NetScaler NSIP address and credential
information is part of the NSWL log.conf configuration file. The NSWL client authenticates to each
of the NetScaler systems configured in its nswl log.conf file.
Once the connection is established, each NetScaler involved in web logging, pushes data from its
transaction log buffer to the NSWL client.

© Copyright 2016 Citrix Systems, Inc. Module 12: NetScaler Web Logging 335
NetScaler IP Addresses
Before web logging can begin gathering log data, it must be configured with the NetScaler IP
address of one or more NetScaler systems. The log.conf file includes the NSIP address and
connection credentials for each NetScaler system.
To enable web logging on NetScaler systems in a high-availability pair, add the NSIP addresses of
both nodes to the NSWL log.conf file to ensure proper transaction logging when either system
becomes the active (primary) node.
Add NetScaler systems to the log.conf only using the nswl -addns command; do not add NetScaler
systems to the file manually. Remove NetScaler systems by manually deleting the appropriate line.

Adding NetScaler IP Addresses

N

Use the following procedure to add NetScaler IP addresses:

ot

1. Enter the following command at the command prompt:

fo

nswl -addns -f directory_path\log.conf

rr

2. Enter the following information at the prompt:

es

• NSIP: Specify the IP address of the system

• Username: Specify the username to log on
al

• Password: Specify the password to log on

e
or

Log Filters
di
s

Log filters determine which transactions are logged. Only transactions matching a filter are
tri

included in the log output files. The default filter, if enabled, will catch all transactions for which no
filter is defined. Filters provide administrators with the ability to determine which data is logged
bu

and how that data is logged. Each filter can be configured with different logging properties.
t io

Filters consist of two parts:

n

• A filter definition, which identifies the transactions to log

• Filter properties, which determine how the transactions are logged
Filters can be defined to identify transactions based on host IP address, domain name, or hostname
of the web servers. Filters can identify traffic on server IP addresses or virtual server IP addresses.
Transactions can be based on one or more individually specified IP addresses or a subnet.
Filters are created within the log.conf file. The file comes with a set of example filters and a pre-
defined default filter. All filters are commented out with the exception of the default filter. The
default filter is enabled. Administrators may modify the included filters or construct new filter
definitions.

336 Module 12: NetScaler Web Logging © Copyright 2016 Citrix Systems, Inc.
Creating Filters
Enter the following command in the log.conf file to create a filter:
Filter filter_name HOST name IP ip_address ON | OFF
The following table lists available arguments.

Argument Description
filter_name Name of the filter (maximum 64 alphanumeric
characters)

HOST name Host name of the server for which the

transactions are being logged
N

IP ip_address IP address of the server for which transactions

ot

are to be logged (for example, if the server has

multiple domains that have one IP address)
fo
rr

ON | OFF Enable or disable the filter to log transactions

es

If no argument is selected, the filter is enabled

(ON).
al
e

Filters can be created that match transactions based on hostname, IP Address, multiple IP
or

addresses, or subnets. Parameters can be combined so that HOST names and IP addresses can be
specified within a single filter.
di

Filters can be enabled or disabled by setting the ON / OFF value appropriately. If not specified, the
s

filter is assumed to be ON. This allows filters to be turned on and off without having to remove the
tri

settings from the log.conf file.

bu

Example filters:
t io

Filter F1 HOST www.netscaler.com ON

n

Filter F2 HOST www.netscaler.com IP 192.168.100.151 OFF

Filter F3 IP 192.168.100.152 192.168.100.153 ON
Filter F4 IP 192.168.100.0 NETMASK 255.255.255.0
Filter default

The examples demonstrate creating four separate filters:

• Filter F1 logs transactions involving hostname www.netscaler.com
• Filter F2 logs transactions matching both the hostname and the IP address. Filter is disabled
(OFF).
• Filter F3 demonstrates use of multiple IP addresses.

© Copyright 2016 Citrix Systems, Inc. Module 12: NetScaler Web Logging 337
• Filter F4 demonstrates use of configuring a filter for a subnet.

Default Filters
The default filter can be specified as any of the following:
• Filter default
• Filter default ON
• Filter default OFF
The default filter does not identify explicit transaction criteria (host names or IP addresses) and is
used to catch all transactions that do not match any other filter.

Defining Log Properties

N
ot

Log properties associated with the filter are applied to all log entries associated with the filter. Log
property definition begins with the keyword BEGIN and ends with END.
fo

For example:
rr
es

BEGIN filtername
logFormat ...
al

logFilenameFormat ...
e

logInterval ...
logFileSize ....
or

logExclude ....
logTransactions ...
di

END
s tri

The following table lists available log properties.

bu
t

Property Description
io
n

logFormat Specifies the web server logging feature: W3C

(default), NCSA, or Custom

338 Module 12: NetScaler Web Logging © Copyright 2016 Citrix Systems, Inc.
Property Description
logInterval Specifies the intervals at which log files are
created:
• Hourly: A file is created every hour
• Daily: A file is created every day at
midnight (default)
• Weekly: A file is created every Sunday at
midnight
• Monthly: A file is created on the first day of
the month at midnight
• None: A file is created only once, when web
server logging starts
N

logFileSize Specifies the maximum size of the log file in MB

ot

It can be used with any log interval. A file is

fo

created when the maximum file size limit is

rr

reached or when the defined log interval time

elapses. To override this behavior, specify the
es

size as the logInterval property so that a file is

created only when the log file size limit is
al

reached. The default logFileSize is 10 MB.

e

logFilenameFormat Specifies the file name format of the log file.

or

The name of the file can be of the following

types:
di

• Static: Specifies a constant string that

s

contains the absolute path and file name.

tri

• Dynamic: Specifies an expression

bu

containing the following format:

t

• Server IP address (%A)

io

• Date (%{format}t)
n

• URL suffix (%x)

• Host name (%v)

The date format %t specified in the

logFilenameFormat command overrides the
log interval property for that filter. To
prevent a new file being created every day
instead of when the specified log file size is
reached, do not use %t in the
LogFilenameFormat.

© Copyright 2016 Citrix Systems, Inc. Module 12: NetScaler Web Logging 339
Property Description
logExclude Prevents logging of transactions with the
specified file extensions

logTransactions Specifies the type of transaction between client

and the server
Transactions between the client and server are
either complete or incomplete. A complete
transaction can be either the server completing
the request successfully or sending back an
error code to the client. Incomplete or canceled
transactions occur for many reasons, such as
client termination, network timeout, or
N

messages dropped on NetScaler if policies or

rules fail. The LogTransactions setting has the
ot

following possible values:

fo

• All: Logs both completed and canceled

transactions
rr

• Completed: Logs only completed

es

transactions (default)
al

• Canceled: Logs only canceled or incomplete

transactions
e

When the logTransactions setting is set to

or

Aborted, the status code field displays a zero

value for each log entry and the number of
di

received bytes field displays the number of bytes

s

received from the server until the time the

tri

transaction was failed.

bu
t

Default Settings for Log Properties

io
n

The following default settings apply to the filter:

• logFormat: W3C Extended
• logInterval: Daily
• logFileSize: 10
• logFileNameFormat: Ex%{%m%d%y}t
• logTime: GMT
• logTransactions: Completed.

340 Module 12: NetScaler Web Logging © Copyright 2016 Citrix Systems, Inc.
Running NSWL
Once NSWL has been configured with the NSIP address of one or more NetScaler systems and the
log.conf has been fully configured with the filters, run NSWL to begin capturing log data.
When running NSWL as a service, the service will not start if the log.conf file is not properly
configured. When running NSWL as a service on Windows, the configuration file path is specified
during the service installation process and therefore the path does not need to be specified when
starting the service.
When running NSWL as a service on any other operating system, the path to the configuration file
must be specified when starting the process. Finally, the path to the configuration file is always
specified when running NSWL as a standalone process.

Verifying the Configuration

N

Enter the following command to check the configuration file (log.conf) for syntax errors to ensure
ot

that logging works correctly:

fo

nswl -verify -f directory_path\log.conf

rr
es

Where directory_path is the path to the configuration file (log.conf).

If the file is misconfigured, the command will return a brief description of the error and identify
al

the debug log file and current logging level. If there are no errors, the command will confirm the
e

file is correct and return a "Done!!" message.

or

Errors with the log.conf file can range from missing NetScaler IP, invalid credentials, or a
malformed log filter definition. The verify command in conjunction with the debug log output files
di

can be used to troubleshoot configuration and operational issues with NSWL.

s tri

Troubleshooting Web Logging

bu
t

Most common issues are log files not being generated as expected or the service failing to start.
io

First verify all the basic configuration settings:

n

• The NSWL feature is enabled on all NetScaler systems involved in logging.

• The NSIP address and credentials are present in the log.conf file.
• The NetScaler and NSWL version/build numbers are the same. Not only should the major
version number, such as 8.0/8.1/9.0, match, but the components should also be running the
same minor build numbers as well. For example, NetScaler systems running 9.0.67.7 should be
logged to the corresponding NSWL components version 9.0.67.7. A separate NSWL instance
should be used for NetScaler systems running 9.0.66.12. If multiple NetScaler systems are being
monitored, make sure all NetScaler systems in a given log.conf file are on the version as the
NSWL component.
• NSWL is running, either as a service or standalone process.

© Copyright 2016 Citrix Systems, Inc. Module 12: NetScaler Web Logging 341
• Both NetScaler systems in a high-availability pair are listed in the log.conf file, and both are
listed by NSIP address (not MIP or SNIP address).
Be sure to verify TCP port 3011 is not blocked between the target NetScaler systems and the NSWL
clients. A quick test is to verify you can connect to the Configuration Utility using a web browser
from the system where you are running NSWL. A successful login and connection to the GUI
should confirm connectivity to the NSIP across TCP port 3011. If NSWL is being run from a client
which is multi-homed, make sure the NSWL client is using the right NIC to connect to the
NetScaler systems.
If NAT is involved, verify proper configuration. Refer to the Citrix NetScaler 9.0 Administration
Guide for more information.

NSWL Troubleshooting
N

The following items should be examined from the system running the NSWL client:
ot

• View the transaction log files. Determine if transactions are or are not being logged. See if the
transactions are indicating an issue that is external to NSWL, such as an issue with a specific
fo

web server.
rr

• Review the NSWL debug files. Run NSWL with debug levels 2 or 3 to get more details about
the exact issues that are occurring.
es

• If you are working with custom filters or custom log file formats and are not receiving the
output you are expecting, first run the -verify command to verify that the log.conf file is valid.
al

In some cases, you may want to revert either to the default log.conf file or at least the default
e

filter in order to isolate issues. If the issue appears to only affect some traffic on a specific
or

NetScaler, then test the configured filters one at a time to see if there is an issue with a specific
filter definition or log parameters.
di
s

NetScaler Troubleshooting
tri
bu

The following items can be examined on the NetScaler system itself, when troubleshooting web
logging issues:
t io

• Make sure the web logging feature is enabled.

n

• Make sure the NSIP address and account credentials are valid on the target NetScaler system.
• Verify the web log buffer size.
The NetScaler newnslog components can be used to gather additional NSWL event information.
Check the Web Logging feature status and the weblogparam (show ns weblogparam) to verify
settings are consistent with expected behavior.

342 Module 12: NetScaler Web Logging © Copyright 2016 Citrix Systems, Inc.
Buffer Overflow
Buffer overflow occurs when data from the NetScaler transaction log is being lost because it is not
being sent to the client before the buffer on the NetScaler system must be flushed or overwritten
with new data. Buffer overflow statistics can be reviewed using the nsconmsg command. Nsconmsg
is retrieving performance data written to the nslog in the /var/nslog/newnslog directory.
Enter the following command in shell:

nsconmsg -d stats | grep tcp_tot_log_bufOverruncount

values in dec

Look for tcp_tot_log_bufOverruncount.

Possible causes for buffer overflow values incrementing:
• Network congestion between NSWL client and the NetScaler are resulting in loss of data
N
ot

Network issues prevent all data from being sent from the NetScaler to the NSWL client before
the NetScaler must overwrite the buffer.
fo

• Slow web logging client, meaning the NSWL client is unable to perform at speeds necessary to
rr

receive the data at the rate it is being sent from the NetScaler
es

Solution:
• Enter the following command in the command-line interface to increase the buffer size on the
al

NetScaler system:
e

set ns weblogparam - bufferSizeMB integer

or

• Increase the CPU/memory of the NSWL workstation to allow the NSWL client system to
di

handle more transactions and write data to the log files faster to keep pace with the NetScaler
transaction log. Typically, this is only an issue if the NSWL client system is underpowered,
s tri

performing other high-CPU tasks, or writing to numerous log files simultaneously, or if there
are an excessive number of NetScaler systems with high traffic load are logging to a single
bu

NSWL client.
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 12: NetScaler Web Logging 343
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

344 © Copyright 2016 Citrix Systems, Inc.

 13
Module 13

Appendix A:
Troubleshooting
Common Issues
N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

346 © Copyright 2016 Citrix Systems, Inc.

Troubleshooting Common Issues
Common Issues
Common issues encountered during NetScaler operations include:
• High availability
• Load balancing
• SSL offloading
• Networking
• Global server load balancing
• Content Switching
N
ot

High Availability
fo

High-availability issues include:

rr

• Configuration synchronization failure

es

• File synchronization failure

al

• Unexpected failover
e
or

Configuration Synchronization Failure

di

Synchronization failure can be a result of connectivity issues, duplex mismatches, packet drops, or
s

the /netscaler/nsnetsvc process not running. Perform the following tasks if synchronization between
tri

the primary and secondary node fails:

bu

• Verify that the primary and secondary nodes can communicate with each other. Management
and heartbeat messages are sent through layer 2 protocols. Layer 2 connectivity between the
t

two high-availability nodes must allow the heartbeat to be received within 3 seconds.
io

• Ensure that any configured ACLs permit communication between the pair.
n

• Enter the following command to check inetd.conf file to ensure the /netscaler/nsnetsvc process
is not disabled:
ns# more /etc/inetd.conf
Ensure the nsnetsvc stream tcp nowait root /netscaler/nsnetsvc nsnetsvc line is not
commented out.
• Enter the following command in the shell to check the ns_com_cfg.conf file on the secondary
node:
ls -l

© Copyright 2016 Citrix Systems, Inc. Module 13: Appendix A: Troubleshooting Common Issues 347
 Ensure that the /tmp directory has write permissions. For example:
drwxrwxrwt 4 root wheel 512 Aug 17 21:28 /tmp
• Verify that the two nodes are not running different versions of the NetScaler operating system.

File Synchronization Failure

Both nodes in an high-availability pair may need a set of common configuration or certificate files,
depending on the deployment needs. If so, files may need to be manually synchronized. For
example, if SSL offload is enabled, then SSL certificates must be copied to the same location
(directory) on both nodes. Similar examples include vsr.html (for SureConnect), any manually
customized files, or any other batch files containing configuration commands.
Enter the following command in the command-line interface to manually synchronize files between
nodes in a high-availability pair:
N
ot

sync ha files mode

fo

The following table lists available arguments.

rr
es

Argument Description
al

mode Specifies the sync mode

e

Possible values include:

or

• all
di

• bookmarks
s

• ssl
tri

• htmlinjection
bu

• imports
t io

The following table lists paths corresponding to synchronization mode.

n

Mode Path
all /nsconfig/ssl/
/var/vpn/bookmarks/
/nsconfig/htmlinjection/

ssl /nsconfig/ssl/

348 Module 13: Appendix A: Troubleshooting Common Issues © Copyright 2016 Citrix Systems, Inc.
Mode Path
bookmarks /var/vpn/bookmarks/

htmlinjection /nsconfig/htmlinjection/

Unexpected Failover
If the NetScaler systems are failing over unexpectedly, then enter the following command in
command-line interface to view current events that may be causing the failover.

root@ns# nsconmsg -d event

N

Possible causes include:

ot

• An interface is down.
• An SSL acceleration card is down.
fo

• The primary node has failed.

rr
es

Load Balancing
al
e

Load-balancing issues include:

or

• Uneven load balancing

• Service/virtual IP address flapping
di
s tri

Uneven Load Balancing

bu

The following table lists items to check when looking to explain and diagnose uneven load
t

balancing.
io
n

© Copyright 2016 Citrix Systems, Inc. Module 13: Appendix A: Troubleshooting Common Issues 349
Item Description
Slow Start The NetScaler system performs a slow start to
avoid overloading physical servers. During the
slow start phase, the NetScaler system
distributes requests by round robin, regardless
of the actual load balancing method configured
on the virtual server. However, it does honor
the configured weight on bindings even during
round robin. A slow start occurs in any of the
following conditions:
• The load balancing method changes
• A new server is bound to a virtual server
• An existing server binding is removed from
N

a virtual server
ot

• A server changes its status from DOWN to

UP
fo

A contact slow start indicates service flapping.

rr

Persistence When enabled, persistence may create uneven

es

loading because the NetScaler system must

al

direct requests from a specific client to a specific

server. Only unique or expired clients get
e

balanced according to the load balancing

or

method.
di

Inconsistent Server Performance Performance across a set of servers is seldom

consistent. Therefore, some servers serve
s tri

requests faster and based on the Load Balancing

method configured, will likely cause an uneven
bu

balance. The degree of imbalance increases as

the degree of inconstancy between the servers
t io

increases.
n

Service Weights Service weights can result in uneven load

balancing. The services with less weight serve
fewer requests.

Service/Virtual Server Flapping

A service flaps mostly likely because its monitors are failing.

350 Module 13: Appendix A: Troubleshooting Common Issues © Copyright 2016 Citrix Systems, Inc.
Enter the following command in the command-line interface to obtain detail monitoring and status
information:
show service service
In most cases, assume that the monitor is failing legitimately and troubleshooting the issue on the
servers themselves. It is also possible that the monitor configuration is causing service flapping by
monitoring too frequently or for the wrong response. The configuration should then be inspected
for any issues.
A virtual server goes down when all of the bound services go down, so a virtual server that flaps is
likely the result of services that are flapping. Follow the advice for service flapping to determine
why the virtual server is flapping.

SSL Offloading
N

SSL offloading issues include:

ot

• Access to the SSL VIP address failing

fo

• Certificate-related warnings occurring

• Intermediate cert not being properly linked
rr

• Browser warning showing an insecure web page

es
al

Access to SSL VIP Address Failing

e
or

This issue typically occurs when the certkey (certificate-key pair entity on the NetScaler system) is
not bound. Enter the following command in the command-line interface to show this state:
di

show ssl vserver vserver_name

s tri

The following table lists the available arguments.

bu

Argument Description
t io
n

vserver_name Specifies the name of the SSL virtual server

Enter the following command in the command-line interface to resolve the issue:

bind ssl certKey vserver_nameservice_name certkey_name

The following table lists available arguments.

© Copyright 2016 Citrix Systems, Inc. Module 13: Appendix A: Troubleshooting Common Issues 351
Argument Description
vserver_name Specifies the name of the SSL virtual server
name to which the certificate-key pair needs to
be bound

service_name Specifies the name of the SSL service to which

the certificate-key pair needs to be bound
Use the add service command to create this
service.

certkey_name Specifies the object name for the certificate-key

pair
N

Certificate-Related Warnings Occurring

ot
fo

Potential causes of this issue include:

rr

• The certkey domain does not match the domain in the browser address bar.
es

• The site is accessed by IP address.

• The certificate is expired.
al

• The intermediate certificate is not installed or is not bound to the certkey.

e

The first two causes can be resolved by using the correct domain in the browser address bar. The
or

third cause requires installation of an updated valid cert. The fourth cause requires a linked
intermediate certificate.
di
s

The fourth cause can also be resolved by the end user accepting the cert when accessing an
tri

internal site. This option is not a good practice with test certificates, as those certificates
can be used on public sites. Once the certificate is accepted, the end user will never be
bu

prompted and may not be aware they are trusting a site with an invalid certificate.
t io
n

Intermediate Certificate Not Being Properly Linked

Certain server certificates might be issued by a CA that is not in the browser's trusted store by
default. In that case, an intermediate certificate which established the chain of trust would need to
added to the NetScaler system and linked to the certkey. When the certificate is presented to the
end user, the intermediate certificate is also provided. This trust is established. The intermediate
certificate needs to be added and then linked to the certkey in question

352 Module 13: Appendix A: Troubleshooting Common Issues © Copyright 2016 Citrix Systems, Inc.
Browser Warning Showing an Insecure Web Page
This commonly occurs with static content like images that are served up on an SSL encrypted page;
this is not a problem and either the images can be provided securely or the user can ignore the
warnings.

Certificates Expiring
The NetScaler system can alert on certificate expiry and new certificates can be uploaded and
bound to the SSL virtual server by unbinding the old cert and binding the new one.

Content Switching
N

The following table lists content-switching issues.

ot
fo

Issue Resolution
rr

Traffic not hitting the intended load balancing Make sure that content switching is enabled, the
es

virtual IP address policies are configured, and the load-balancing

virtual server is bound with a policy to the
al

content switching virtual server.

e

Policy not being matched properly Use the policy evaluator tool to ensure the
or

policy matches the expected content.

di

No content being served Make sure the back-end resources are available
s

for the load-balancing virtual server and that

tri

the services passing health checks.

bu
t

Global Server Load Balancing

io
n

The following table lists global-server load-balancing issues.

Issue Description
Metric Exchange Protocol (MEP) not being First, make sure that MEP is enabled on both IP
formed addresses and that they are pointing to the
correct ones on each site. Then check that
communication between the sites for the MEP
IP addresses is working and that there are no
firewalls or ACLs blocking the traffic.

© Copyright 2016 Citrix Systems, Inc. Module 13: Appendix A: Troubleshooting Common Issues 353
Issue Description
Remote site not coming up When this issue occurs, make sure that the load
balancing virtual server on the remote site is
passing all its health checks and that the other
site is either reachable by MEP or by direct
health checks from the working site.

DNS not working If the NetScaler system is resolving DNS queries

(ADNS mode), then make sure that the entry
for that domain is present. If so, then make sure
it is accepting DNS queries on the ADNS
service and that it is up. If the NetScaler system
is operating as a DNS proxy, then make sure
that the back-end DNS servers are responding
N

and that the virtual server and services are up.

ot
fo

Networking
rr

Networking issues include:

es

• Duplex mismatch or misconfigured interface settings

al

• Unresponsive system
e

• Inaccessible content
or

Several common issues can be checked and discarded early in the troubleshooting process,
including a slow NetScaler system due to a duplex mismatch, an unresponsive system and
di

inaccessible content.
s tri

The NetScaler system always draws power when it is plugged in. Therefore, an occasional
simple reboot does not clear severe console hang conditions. Completely remove the
bu

power from the units (unplug them from the outlets) for 30 seconds and then power them
t

back on.
io
n

Duplex Mismatch or Misconfigured Interface Settings

If the duplex setting on the NetScaler system does not match the setting on the switch or router on
which it is connected, the NetScaler system may be slow to respond to requests. To investigate this
issue, click View console messages in the Diagnostics pane of the System node.
IP address conflicts or a duplex mismatch alert should be checked. Interfaces can be configured in
the Interfaces screen of the Network node.
Enter the following command in the command-line interface to correctly set the interface:

354 Module 13: Appendix A: Troubleshooting Common Issues © Copyright 2016 Citrix Systems, Inc.
 set interface id -speed speed -duplex duplex_mode

The following table lists available arguments.

Argument Description
id Specifies the interface ID, for example 1/1 or
1/5

-speed speed Specifies the network speed, which can be set to

Auto, 10, 100 or 1000

-duplex duplex_mode Specifies the duplex mode, which can be Auto,

Full or Half
N
ot

Enter the following command to obtain detailed interface statistics and switch port settings:
fo

sh interface n/n
rr

the following table lists available arguments.

es
al

Argument Description
e
or

n/n Specifies the appropriate interface

di

Incorrect interface settings usually result in an interface that will not come up. It is
s tri

important to ensure that the configuration on the NetScaler interface closely resembles the
configuration on the corresponding switch/router interface. Ensure that speed, duplex, and
bu

flow control settings match.

t io

Inaccessible Content
n

If content located behind the NetScaler system is inaccessible, the following items should be
verified:
• Have configuration changes been made to servers or network devices?
• Have configuration changes been made to server, service, or virtual server objects?
• Can the site be accessed directly (in other words, bypassing the NetScaler system)?
• Can the server and port be accessed using Telnet?

© Copyright 2016 Citrix Systems, Inc. Module 13: Appendix A: Troubleshooting Common Issues 355
Caching
The following table lists cache issues.

Issue Description
Incorrect content being cached Improper configuration of the Integrated
Caching feature may cause users to see incorrect
content served from the cache.
Enter the following command in the command-
line interface to determine whether a particular
object is in the cache:

show cache object -url url -

N

host hostname
ot

Enter this command after making an initial

fo

request for the object (for example, using a web

rr

browser) to give the NetScaler system a chance

to cache the object. If the object is not in the
es

cache, the NetScaler system reports "ERROR:

No such resource."
al

To determine if an object served from the cache

e

does not match the object served from the back-

or

end Web server(s), compare the HTTP

responses from:
di

• A client with a connection through the

s

NetScaler system
tri

• A client with a connection not through the

bu

NetScaler system
t

If the responses differ, the responses received

io

from clients passing through the NetScaler

n

system are erroneous and the issue requires

troubleshooting.
Verify if there are any cache selectors
configured that could serve the same object for
different VIP addresses.

Expired content being served Proper expiry headers are not being provided;
check the content on the servers and the
invalidation parameters.

356 Module 13: Appendix A: Troubleshooting Common Issues © Copyright 2016 Citrix Systems, Inc.
Issue Description
Cache expiry causing traffic surge to the back- Decrease the timeouts for the cached content or
end create different content groups that expire at
different times so all the content is not expiring
at the same time. Enable the prefetch option.

N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

© Copyright 2016 Citrix Systems, Inc. Module 13: Appendix A: Troubleshooting Common Issues 357
 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

358 © Copyright 2016 Citrix Systems, Inc.

 N
ot
fo
rr
es
al
e
or
di
s
tri
bu
t io
n

© Copyright 2016 Citrix Systems, Inc. 359

 N
ot
fo
rr
es
al
e
or
di
s tri
bu
t io
n

851 West Cypress Creek Road Fort Lauderdale, FL 33309 USA (954) 267 3000 www.citrix.com
Rheinweg 9 8200 Schaffhausen Switzerland +41 (0) 52 63577 00 www.citrix.com
© Copyright 2016 Citrix Systems, Inc. All rights reserved.