Top 10 ASA Firewall and VPN

Troubleshooting Techniques
Omar Santos
[email protected]
Cisco PSIRT
Security Research and Operations

Agenda

Introduction of ASA Packet Flow

Top 5 Techniques when Troubleshooting

Firewall Problems

Top 5 Techniques when Troubleshooting

VPN Problems

Q&A

2014 Cisco and/or its affiliates. All rights reserved.

Understanding the

2013-2014 Cisco and/or its affiliates. All rights reserved.

Understanding Packet Flow

To effectively troubleshoot a connectivity problem, one must first understand the packet path

through the network.

Attempt to isolate the problem down to a single device.
Then perform a systematic walk of the packet path through the device to determine where the

problem could be
For problems relating to the Cisco ASA, always:

Determine the flow: Protocol, Source IP, Destination IP, Source Port, Destination Port

Determine the logical (named) interfaces through which the flow passes
TCP outside

172.16.164.216:5620 inside

192.168.1.150:50141, idle 0:00:00, bytes 0, flags saA

All firewall connectivity issues can be simplified to two

interfaces (ingress and egress) and the policies tied to both
4

2014 Cisco and/or its affiliates. All rights reserved.

Example Flow

With the Flow defined,

examination of configuration
issues boils down to just the two
Interfaces: Inside and Outside

TCP Flow
Source IP

: 10.1.1.9

Destination IP : 198.133.219.25

Source Port

: 11030

Destination Port

80

Interfaces
Source: Inside

Destination: Outside

Packet Flow

Servers

10.1.1.9

Eng

Accounting
Outside

2014 Cisco and/or its affiliates. All rights reserved.

198.133.219.25

Packet Processing: Ingress Interface

IPS or CX
Module

Yes
RX
Pkt

Ingress
Interface

Existing
Conn

No

NAT
Untranslate

Yes

ACL
Permit
No

Packet arrives on ingress interface

DROP

Yes
Stateful
Inspection

NAT IP
Header

No
DROP

Egress
Interface

L3
Route

No

No

DROP

DROP

Yes

L2
Addr

Yes

TX
Pkt

No
DROP

Input counters incremented by NIC and periodically retrieved by CPU

Software input queue (RX ring) is an indicator of packet load

Overrun counter indicates packet drops (usually packet bursts)

2014 Cisco and/or its affiliates. All rights reserved.

asa# show interface outside

Interface GigabitEthernet0/3 "outside", is up, line protocol is up
Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
MAC address 0026.0b31.36d5, MTU 1500
IP address 148.167.254.24, subnet mask 255.255.255.128
54365986 packets input, 19026041545 bytes, 0 no buffer
Received 158602 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
[]
input queue (blocks free curr/low): hardware (255/230)
output queue (blocks free curr/low): hardware (254/65)

Packet Processing: Locate Connection

IPS or CX
Module
Yes
RX
Pkt

Ingress
Interface

Existing
Conn

No

NAT
Untranslate

ACL
Permit

No
DROP

Yes

Yes
Stateful
Inspection
No
DROP

NAT IP
Header

Egress
Interface

L3
Route

No

No

DROP

DROP

Yes

L2
Addr

Yes

TX
Pkt

No
DROP

Check first for existing connection in conn table

If conn entry exists, bypass ACL check and process in Fastpath
asa# show conn
TCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO

If no existing connection
TCP SYN or UDP packet, pass to ACL and other policy checks in Session Manager

TCP non-SYN packet, drop and log

ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH ACK
interface inside

2014 Cisco and/or its affiliates. All rights reserved.

on

Packet Processing: NAT Un-Translate

IPS or CX
Module
Yes
RX
Pkt

Ingress
Interface

Existing
Conn

No

NAT
Untranslate

ACL
Permit

No
DROP

Yes

Yes
Stateful
Inspection

NAT IP
Header

No
DROP

Egress
Interface

L3
Route

No

No

DROP

DROP

Yes

L2
Addr

Yes

TX
Pkt

No
DROP

Incoming packet is checked against NAT rules

Packet is un-translated first, before ACL check

In ASA 8.2 and below, incoming packet was subjected to ACL check prior to un-translation
NAT rules can determine the egress interface at this stage

2014 Cisco and/or its affiliates. All rights reserved.

Packet Processing: ACL Check

IPS or CX
Module
Yes
RX
Pkt

Ingress
Interface

Existing
Conn

No

NAT
Untranslate

ACL
Permit

Yes

No
DROP

Yes
Stateful
Inspection

NAT IP
Header

No
DROP

Egress
Interface

L3
Route

No

No

DROP

DROP

Yes

L2
Addr

Yes

TX
Pkt

No
DROP

First packet in flow is processed through ACL checks

ACLs are first configured match
First packet in flow matches ACE, incrementing hit count by one
asa# show access-list inside
access-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1)

Denied packets are dropped and logged

ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside:198.133.219.25/80 by access-group "inside"

2014 Cisco and/or its affiliates. All rights reserved.

Packet Processing: Stateful Inspection

IPS or CX
Module
Yes
RX
Pkt

Ingress
Interface

Existing
Conn

No

NAT
Untranslate

ACL
Permit

No
DROP

Yes

Yes
Stateful
Inspection
No
DROP

NAT IP
Header

Egress
Interface

L3
Route

No

No

DROP

DROP

Yes

L2
Addr

Yes

TX
Pkt

No
DROP

Stateful inspection ensures protocol compliance at TCP/UDP/ICMP level

(Optional) Customisable application inspection up to Layer 7 (FTP, SIP, and so on)

Rewrite embedded IP addresses, open up ACL pinholes for secondary connections

Additional security checks are applied to the application payload

ASA-4-406002: FTP port command different address: 10.2.252.21(192.168.1.21) to 209.165.202.130 on
interface inside
ASA-4-405104: H225 message received from outside_address/outside_port to inside_address/inside_port
before SETUP

2014 Cisco and/or its affiliates. All rights reserved.

10

Packet Processing: NAT IP Header

IPS or CX
Module
Yes
RX
Pkt

Ingress
Interface

Existing
Conn

No

NAT
Untranslate

ACL
Permit

No
DROP

Yes

Yes
Stateful
Inspection

NAT IP
Header

Egress
Interface
No

No

DROP

DROP

No
DROP

L3
Route

Yes

L2
Addr

Yes

TX
Pkt

No
DROP

Translate the source and destination IP addresses in the IP header

Translate the port if performing PAT
Update header checksums

(Optional) Following the above, pass packet to IPS or CX module

Real (pre-NAT) IP address information is supplied as meta data

2014 Cisco and/or its affiliates. All rights reserved.

11

Packet Processing: Egress Interface

IPS or CX
Module
Yes
RX
Pkt

Existing
Conn

Ingress
Interface

No

NAT
Untranslate

ACL
Permit

No
DROP

Yes

Yes
Stateful
Inspection
No
DROP

NAT IP
Header

Egress
Interface

L3
Route

No

No

DROP

DROP

Yes

L2
Addr

Yes

TX
Pkt

No
DROP

Packet is virtually forwarded to egress interface (not forwarded to the Ethernet NIC yet)
Egress interface is determined first by translation rules or existing conn entry, only THEN

the routing table

If NAT does not divert to the egress interface, the global routing table is consulted to

determine egress interface

Inside
DMZ

172.16.0.0/16

Outside

172.16.12.0/24
172.16.12.4

2014 Cisco and/or its affiliates. All rights reserved.

Packets received on outside and destined to

192.168.12.4 get routed to 172.16.12.4 on
inside based on NAT configuration.
nat (inside,outside) source static 172.16.0.0-net 192.168.0.0-net
nat (dmz,outside) source static 172.16.12.0-net 192.168.12.0-net
12

Packet Processing: L3 Route Lookup

IPS or CX
Module
Yes
RX
Pkt

Ingress
Interface

Existing
Conn

No

NAT
Untranslate

ACL
Permit

No
DROP

Yes

Yes
Stateful
Inspection

NAT IP
Header

No
DROP

Egress
Interface

L3
Route

No

No

DROP

DROP

Yes

L2
Addr

Yes

TX
Pkt

No
DROP

Once at egress interface, an interface route lookup is performed

Only routes pointing out the egress interface are eligible
Remember: NAT rule can forward the packet to the egress interface, even though the

routing table may point to a different interface

If the destination is not routable out of the identified egress interface, the packet is dropped
%ASA-6-110003: Routing failed to locate next hop for TCP from inside:192.168.103.220/59138
to dmz:172.15.124.76/23

2014 Cisco and/or its affiliates. All rights reserved.

13

Packet Processing: L2 Address Lookup

IPS or CX
Module
Yes
RX
Pkt

Ingress
Interface

Existing
Conn

No

NAT
Untranslate

ACL
Permit

No
DROP

Yes

Yes
Stateful
Inspection

NAT IP
Header

Egress
Interface

No
DROP

L3
Route

No

No

DROP

DROP

Yes

L2
Addr

Yes

TX
Pkt

No
DROP

Once a Layer 3 route has been found, and next hop IP address identified, Layer 2

resolution is performed
Layer 2 rewrite of MAC header
If Layer 2 resolution fails no syslog

show arp will not display an entry for the L3 next hop
debug arp will indicate if we are not receiving an ARP reply
arp-req: generating request for 10.1.2.33 at interface outside
arp-req: request for 10.1.2.33 still pending

2014 Cisco and/or its affiliates. All rights reserved.

14

Packet Processing: Transmit Packet

IPS or CX
Module
Yes
RX
Pkt

Ingress
Interface

Existing
Conn

No

NAT
Untranslate

ACL
Permit

No
DROP

Yes

Yes
Stateful
Inspection
No
DROP

NAT IP
Header

Egress
Interface

L3
Route

No

No

DROP

DROP

Yes

L2
Addr

Yes

TX
Pkt

No
DROP

Packet is transmitted on wire

Interface counters will increment on interface

Underrun counter indicates drops due to egress interface oversubscription

TX ring is full

2014 Cisco and/or its affiliates. All rights reserved.

asa# show interface outside

Interface GigabitEthernet0/1 "outside", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
MAC address 503d.e59d.90ab, MTU 1500
IP address 172.18.124.149, subnet mask 255.255.255.0

273399 packets output, 115316725 bytes, 80 underruns

input queue (blocks free curr/low): hardware (485/441)

output queue (blocks free curr/low): hardware (463/0)
15

*NOT RANKED BY IMPORTANCE

2014 Cisco and/or its affiliates. All rights reserved.

17

Uses of Syslogs
Primary mechanism for recording connections to and through the firewall
The best troubleshooting tool available

Archival Purposes

Live Debugging Purposes

Console
Syslog/FTP Server

Trap
SNMP Server

2014 Cisco and/or its affiliates. All rights reserved.

Syslog.

Flash

Local
Buffer

ASDM

18

ASA Syslog Level vs. Number of Messages

Log
Level

Number of Messages (SUM)

Description
Ver. 7.0

Ver. 7.2

Ver. 8.0

Ver. 8.1

Ver. 8.2

Ver. 8.3

Ver. 8.4

Ver. 9.1

Emergencies

Alerts

62 (62)

77 (77)

78 (78)

87 (87)

87 (87)

95 (95)

109 (109)

117 (117)

Critical

29 (91)

35 (112)

49 (127)

50 (137)

56 (143)

57 (152)

63 (172)

72 (189)

Errors

274 (365)

334 (446)

361 (488)

363 (500)

384 (527)

408 (560)

448 (620)

521 (710)

Warnings

179 (544)

267 (713)

280 (768)

281 (781)

315 (842)

324 (884)

357 (997)

420 (1130)

Notifications

161 (705)

206 (919)

216 (984)

218 (999)

237 (1079)

246 (1130)

265 (1242)

285 (1415)

Informational

234 (939)

302 (1221)

335 (1319)

337 (1336)

368 (1447)

377 (1507)

395 (1637)

430 (1845)

Debugging

217 (1156)

258 (1479)

266 (1585)

267 (1603)

269 (1716)

269 (1776)

276 (1913)

295 (2140)

2014 Cisco and/or its affiliates. All rights reserved.

19

Custom Syslog Levels

Assign any syslog message to any available level

Problem:

Levels

You want to record what exec commands are

being executed on the firewall; syslog ID 111009
records this information, but by default it is at
level 7 (debug)
ASA-7-111009: User johndoe executed cmd: show run
The problem is we dont want to log all 1775
other syslogs that are generated at debug level

0Emergency
1Alert
2Critical
3Errors

4Warnings
5Notifications
6Informational

asa(config)# logging message 111009 level 3

7Debugging

ASA-3-111009: User johndoe executed cmd: show run

2014 Cisco and/or its affiliates. All rights reserved.

20

TCP Connection Termination Reasons

If a TCP flow was built through the ASA, it will always log a teardown reason
TCP teardown message is logged at level 6 (informational) by default

If you are having problems abnormal connection termination, temporally increase your logging

level (or change the syslog level, and check the teardown reason
%ASA-6-302014: Teardown TCP connection 90 for outside:10.1.1.1/80 to inside:192.168.1.101/1107 duration 0:00:30 bytes 0

SYN Timeout
%ASA-6-302014: Teardown TCP connection 3681 for DMZ:172.16.171.125/21 to inside:192.168.1.110/24245 duration 0:01:03
bytes 12504 TCP Reset-O

21

2014 Cisco and/or its affiliates. All rights reserved.

21

TCP Connection Termination Reasons

Reason

Description

Conn-Timeout

Connection Ended Because It Was Idle Longer Than the Configured Idle Timeout

Deny Terminate

Flow Was Terminated by Application Inspection

Failover Primary Closed

The Standby Unit in a Failover Pair Deleted a Connection Because of a Message

Received from the Active Unit

FIN Timeout

Force Termination After Ten Minutes Awaiting the Last ACK or After Half-Closed Timeout

Flow Closed by Inspection

Flow Was Terminated by Inspection Feature

Flow Terminated by IPS

Flow Was Terminated by IPS

Flow Reset by IPS

Flow Was Reset by IPS

Flow Terminated by
TCP Intercept

Flow Was Terminated by TCP Intercept

Invalid SYN

SYN Packet Not Valid

Idle Timeout

Connection Timed Out Because It Was Idle Longer than the Timeout Value

IPS Fail-Close

Flow Was Terminated Due to IPS Card Down

SYN Control

Back Channel Initiation from Wrong Side

2014 Cisco and/or its affiliates. All rights reserved.

22

TCP Connection Termination Reasons

Reason

Description

SYN Timeout

Force Termination After Twenty Seconds Awaiting

Three-Way Handshake Completion

TCP Bad Retransmission

Connection Terminated Because of Bad TCP Retransmission

TCP Fins

Normal Close Down Sequence

TCP Invalid SYN

Invalid TCP SYN Packet

TCP Reset-I

TCP Reset Was Sent From the Inside Host

TCP Reset-O

TCP Reset Was Sent From the Outside Host

TCP Segment Partial Overlap

Detected a Partially Overlapping Segment

TCP Unexpected Window Size Variation

Connection Terminated Due to a Variation in the

TCP Window Size

Tunnel Has Been Torn Down

Flow Terminated Because Tunnel Is Down

Unauth Deny

Connection Denied by URL Filtering Server

Unknown

Catch-All Error

Xlate Clear

User Executed the Clear Xlate Command

2014 Cisco and/or its affiliates. All rights reserved.

23

NetFlow Secure Event Logging (NSEL)

NetFlow v9 support added in ASA 8.1+

Provides a method to deliver binary logs at high speeds

Reduce processing overhead in printing logs
Combine multiple events into one NetFlow record
FlowSets Supported:

Flow Creation
Flow Teardown
Flow Denied

Flow Update in ASA 8.4(5)+ and 9.1(2)+

Remove redundant syslog messages

asa(config)# logging flow-export-syslogs disable

2014 Cisco and/or its affiliates. All rights reserved.

24

Case Study: Excessive Logging

logging
logging
logging
logging
logging
logging
logging
logging

enable
buffered debugging
console debugging
trap debugging
history debugging
host inside 192.168.1.10
host inside 192.168.1.11
host DMZ 192.168.2.121

snmp-server host inside 192.168.1.10

snmp-server host inside 192.168.1.11
snmp-server host DMZ 192.168.2.121

flow-export destination inside 192.168.1.10

flow-export destination inside 192.168.1.11
flow-export destination DMZ 192.168.2.121

4 logging destinations (buffer,

console, SNMP, and syslog)
3 syslog servers
3 SNMP servers
3 Netflow collectors
4 messages per PAT
connection (over 550 bytes)

%ASA-6-305011: Built dynamic TCP translation from inside:192.168.1.101/4675 to

1 connection:
outside:172.16.171.125/34605
%ASA-6-302013: Built outbound TCP connection 3367663 for outside:198.133.219.25/80
32 syslog messages
(198.133.219.25/80) to inside:192.168.1.101/4675 (172.16.171.125/34605)
26+ packets sent
%ASA-6-302014: Teardown TCP connection 3367663 for outside:198.133.219.25/80 to
100K connections/sec:
inside:192.168.1.101/4675 duration 0:00:00 bytes 1027 TCP FINs
2.8Gbps
%ASA-6-305012: Teardown dynamic TCP translation from inside:192.168.1.101/4675
to
outside:172.16.171.125/34605
duration
0:00:30
2014 Cisco and/or its affiliates. All rights reserved.

25

Case Study: Logging Optimization

Not logging to buffer
unless troubleshooting

Console logging is a
bottleneck (low rate)

Using minimum number of syslog

servers and Netflow collectors

logging enable
logging flow-export-syslogs disable

Do not duplicate syslogs

and Netflow data

logging list FAILOVER message 104003

Reduce severity
level for syslogs

logging trap errors

logging history FAILOVER
logging host inside 192.168.1.10

Send only certain

syslogs as SNMP traps

logging host DMZ 192.168.2.121

snmp-server host inside 192.168.1.10
snmp-server host DMZ 192.168.2.121 poll

Not all SNMP servers

need to receive traps

flow-export destination inside 192.168.1.10

flow-export destination DMZ 192.168.2.121
2014 Cisco and/or its affiliates. All rights reserved.

26

Logging Common Issues

logging flash-bufferwrap should only be used when logging to buffer at Level 1
logging history should only be used when you really have an SNMP server that you want to

receive all syslogs

logging console should only be enabled while actively troubleshooting on the console
logging standby should only be used if you want to receive double the syslogs
logging permit-hostdown should always be used with TCP syslogging

2014 Cisco and/or its affiliates. All rights reserved.

27

2
*NOT RANKED BY IMPORTANCE.
2014 Cisco and/or its affiliates. All rights reserved.

28

Xlate Table
show xlate displays information about NAT translations through the ASA

Second biggest memory consumer after conn table, no hardcoded size limit
You can limit the output to just the local or global IP
asa# show xlate local 10.2.1.2
5014 in use, 5772 most used
TCP PAT from inside:192.168.103.220/57762 to outside:10.2.1.2/43756 flags ri
idle 0:00:00 timeout 0:00:30
TCP PAT from inside:192.168.103.220/57761 to outside:10.2.1.2/54464 flags ri
idle 0:00:00
timeout
0:00:30
Depleted
NAT/PAT
pools
may cause connectivity issues

asa# show nat pool

TCP PAT pool outside, address 10.2.1.2, range 1-511, allocated 1
TCP PAT pool outside, address 10.2.1.2, range 512-1023, allocated 0
TCP PAT pool outside, address 10.2.1.2, range 1024-65535, allocated 64102
2014 Cisco and/or its affiliates. All rights reserved.

29

Detailed NAT Information

show nat displays information about the NAT table of the ASA

detail keyword will display object definitions

Watch the hit counts for policies that are not matching traffic
asa# show nat detail
Manual NAT Policies (Section 1)
1 (inside) to (outside) source static science-obj science-obj destination static vpn-obj vpn-obj
translate_hits = 0, untranslate_hits = 0
Source - Origin: 192.168.0.0/16, Translated: 192.168.0.0/16
Destination - Origin: 172.16.1.0/24, Translated: 172.16.1.0/24
Auto NAT Policies (Section 2)
1 (dmz) to (outside) source static webserver-obj 14.36.103.83
translate_hits = 0, untranslate_hits = 3232
Source - Origin: 192.168.22.32/32, Translated: 14.36.103.83/32
2 (inside) to (outside) source dynamic science-obj interface
translate_hits = 37723, untranslate_hits = 0
Source - Origin: 192.168.0.0/16, Translated: 14.36.103.96/16

Translate hits indicate

connections from real to
mapped
2014 Cisco and/orinterfaces
its affiliates. All rights reserved.

Untranslate hits indicate

connections from mapped to
real interfaces

Check specific
translation policies in
the applied order.

30

CONNECTION TABLE

3
*NOT RANKED BY IMPORTANCE.
2014 Cisco and/or its affiliates. All rights reserved.

31

Connection Table
asa# show conn detail
2 in use, 64511 most used
Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN,
B - initial SYN from outside, b - TCP state-bypass or nailed,
C - CTIQBE media, c - cluster centralized,
D - DNS, d - dump, E - outside back connection, F - outside FIN, f - inside FIN,
G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data,
i - incomplete, J - GTP, j - GTP data, K - GTP t3-response
k - Skinny media, M - SMTP data, m - SIP media, n - GUP
O - outbound data, P - inside back connection, p - Phone-proxy TFTP connection,
q - SQL*Net data, R - outside acknowledged FIN,
R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN,
s - awaiting outside SYN, T - SIP, t - SIP transient, U - up,
V - VPN orphan, W - WAAS,
X - inspected by service module,
Narrow down the output with x - per session, Y - director stub flow, y - backup stub flow, Bidirectional byte count; use
NSEL to report each
show conn address <ip> Z - Scansafe redirection, z - forwarding stub flow
TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101,
flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127
UDP outside:172.18.124.1/123 dmz:10.1.1.9/123,
flags -, idle 15s, uptime 16s, timeout 2m, bytes 1431

Conn flags indicate current

state
2014 Cisco and/or its affiliates. All rights reserved.

direction separately.

detail option adds uptime

and timeout information

32

Local Host Table

A local-host entry is created for every IP tracked by the ASA
It groups xlates, connections, and AAA information

Useful for monitoring connections terminating on servers or offending clients

asa# show local-host detail connection tcp 50
Interface dmz: 0 active, 0 maximum active, 0 denied
Interface inside: 1 active, 1 maximum active, 0 denied
local host: <192.168.103.220>,
TCP flow count/limit = 798/unlimited
TCP embryonic count to host = 0
TCP intercept watermark = unlimited
UDP flow count/limit = 0/unlimited
Conn:
TCP outside:172.18.124.76/80 inside:192.168.103.220/34078,
flags UO, idle 0s, uptime 0s, timeout 30s, bytes 0
TCP outside:172.18.124.76/80 inside:192.168.103.220/34077,
flags UO, idle 0s, uptime 0s, timeout 30s, bytes 0
(output
truncated)
2014 Cisco and/or its affiliates. All rights reserved.

Only display hosts that have

more than 50 active TCP
connections.

33

Connection Flags
Outbound Connection

inside

client

Inbound Connection

outside

ASA

2014 Cisco and/or its affiliates. All rights reserved.

inside

server

client

outside

ASA

server
34

PACKET CAPTURE

4
*NOT RANKED BY IMPORTANCE.
2014 Cisco and/or its affiliates. All rights reserved.

35

Packet Capture
Inside Capture

Outside Capture

In-line capability to record packets passing through ASA

Inside

Outside
Capture OUT

Two key steps in troubleshooting with captures

Capture IN
Apply capture under unique name to ingress and egress interfaces
Define the traffic that you want to capture, use pre-NAT on the wire information
Tcpdump-like format for displaying captured packets on the box
asa# capture OUT interface outside match ip any host 172.18.124.1
asa# capture IN interface inside match ip any host 172.18.124.1
asa# show capture IN

Unlike ACL, match covers

both directions of the flow

4 packets captured

1: 10:51:26.139046
2: 10:51:26.139503
3: 10:51:27.140739
4: 10:51:27.141182
4 packets shown
asa# no capture IN
2014 Cisco and/or its affiliates. All rights reserved.

802.1Q
802.1Q
802.1Q
802.1Q

vlan#10
vlan#10
vlan#10
vlan#10

P0
P0
P0
P0

172.18.254.46 > 172.18.124.1:

172.18.124.1 > 172.18.254.46:
172.18.254.46 > 172.18.124.1:
172.18.124.1 > 172.18.254.46:

icmp:
icmp:
icmp:
icmp:

echo
echo
echo
echo

request
reply
request
reply

Remember to remove the captures

when done with troubleshooting
36

Packet Capture
Capture buffer maintained in RAM (512KB by default, 30 MB max)

Stops capturing when full by default, circular option available

Default recorded packet length is 1518 bytes
May elevate CPU utilization on multiple-core ASA when applied
Copy captures off via TFTP or retrieve through HTTPS with your web browser

Do this before removing the capture with no capture

https://x.x.x.x/admin/capture/OUT/pcap/outsidecapture.pcap
Configured capture name

Save capture file under this name

Download binary PCAP to

open in your favorite packet
analyser (such as Wireshark)
2014 Cisco and/or its affiliates. All rights reserved.

37

Where Packets Are Captured in Packet Flow

IPS or CX
Module
Yes
RX
Pkt

Ingress
Interface

Existing
Conn

No

NAT
Untranslate

ACL
Permit

No
DROP

Ingress Packets
Captured

Yes

Yes
Stateful
Inspection

NAT IP
Header

Egress
Interface

No
DROP

L3
Route

No

No

DROP

DROP

Yes

L2
Addr

Yes

TX
Pkt

No
DROP

Egress Packets
Captured

Packets are captured at the first and last points they can be in the flow
Ingress packets are captured before most packet processing
Egress packets are captured after all processing
Transit packets show the destination MAC address rewritten
Self-sourced packets may show an empty MAC address (0000.0000.0000)
2014 Cisco and/or its affiliates. All rights reserved.

38

Accelerated Security Path (ASP)

Packets and flows dropped in the ASP will increment a counter

Frame drop counters are per packet

Flow drops are per flow

See command reference under show asp drop for full list of counters
asa# show asp drop

Frame drop:
Invalid encapsulation (invalid-encap)
Invalid tcp length (invalid-tcp-hdr-length)
Invalid udp length (invalid-udp-length)
No valid adjacency (no-adjacency)
No route to host (no-route)
Reverse-path verify failed (rpf-violated)
Flow is denied by access rule (acl-drop)
First TCP packet not SYN (tcp-not-syn)
10942
Bad TCP Checksum (bad-tcp-cksum)

10897
9382
10
5594
1009
15
25247101
36888
893

2014 Cisco and/or its affiliates. All rights reserved.

39

Capturing ASP Drops

Capture all frames dropped in the ASP
asa# capture drops type asp-drop all

Capture all frames with a specific drop reason

asa# capture drop type asp-drop ?
acl-drop
rule
all
bad-crypto
bad-ipsec-natt
bad-ipsec-prot
bad-ipsec-udp
bad-tcp-cksum
bad-tcp-flags

Flow is denied by configured

All packet drop reasons
Bad crypto return in packet
Bad IPSEC NATT packet
IPSEC not AH or ESP
Bad IPSEC UDP packet
Bad TCP checksum
Bad TCP flags

ASP flow drops are non-atomic and cannot be captured

asa# capture drops type asp-drop tcp-not-syn
2014 Cisco and/or its affiliates. All rights reserved.

40

PACKET TRACER

5
*NOT RANKED BY IMPORTANCE.
2014 Cisco and/or its affiliates. All rights reserved.

41

Packet Tracer
Unique capability to record the path of a specially tagged packet through ASA

Best way to understand the packet path in the specific software version
Inject a simulated packet to analyse the behaviour and validate configuration
Feature order
and name

asa# packet-tracer input inside tcp 192.168.1.101 23121 172.16.171.125 23 detailed

Phase: 1
Type: CAPTURE
Ingress interface
Subtype:
Result: ALLOW
Config:
Additional Information:
[]

2014 Cisco and/or its affiliates. All rights reserved.

Packet information as it
enters the ingress interface
Include detailed internal flow and
policy structure information

42

Sample Packet Tracer Output

asa# packet-tracer input outside tcp 172.18.124.66 1234 172.18.254.139 3389
Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
MAC Access list
Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

2014

Phase: 3
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (outside,dmz) source dynamic any interface destination static interface Win7-vm service rdp-outside rdp-outside
Additional Information:
NAT divert to egress interface dmz
Untranslate 172.18.254.139/3389 to 192.168.103.221/3389
Cisco and/or its affiliates. All.
rights reserved.

43

Sample Packet Tracer Output

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_in in interface outside
access-list outside_in extended permit tcp any any eq 3389
Additional Information:

Phase: 8
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (outside,dmz) source dynamic any interface destination static interface Win7-vm service rdp-outside rdp-outside
Additional Information:
Dynamic translate 172.18.124.66/1234 to 192.168.103.221/1234

Phase: 12
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 16538274, packet dispatched to next module
2014 Cisco and/or its affiliates. All rights reserved.

44

Packet Tracer in ASDM

Launch from Tools >
Packet Tracer

Define simulated packet

Feature type and

resulting action

Direct link to edit policy

Associated
configuration

Final outcome (allowed or

dropped) and egress
interface information
2014 Cisco and/or its affiliates. All rights reserved.

45

Packet Tracer: Tracing Captured Packet

Enable packet tracer within an internal packet capture
asa# capture IN interface inside trace trace-count 20 match tcp any any eq
Trace inbound
packets only

Traced packet count per

capture (50 by default)

Find the packet that you want to trace in the capture

asa#
68
1:
2:
3:
4:
5:

show capture inside

packets captured
15:22:47.581116 10.1.1.2.31746 > 198.133.219.25.80:
15:22:47.583465 198.133.219.25.80 > 10.1.1.2.31746:
15:22:47.585052 10.1.1.2.31746 > 198.133.219.25.80:
15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80:
15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746:
...

S
S
.
P
.

ack
ack
ack
Ack

Select that packet to show the tracer results

asa# show capture inside trace packet-number 4
2014 Cisco and/or its affiliates. All rights reserved.

46

Packet Tracer Video

Embedded Event Manager

Troubleshooting tool added in 9.2(1), similar to IOS EEM
Powerful way to run CLI commands based on ASA events (syslogs) and save
the output
Trigger Event

Action

Syslog ID

Execute show commands

Console

Time based

Execute config commands

File on disk

Crash

Output Destination

None

Manual

48

2014 Cisco and/or its affiliates. All rights reserved.

48

Embedded Event Manager

Time-based events
Every midnight back up the ASA configuration to your tftp server
Every 3 hours gather the output of show memory detail and save it to the flash

Syslog based events

If the available 1550 byte blocks become depleted, gather show blocks pool 1550
dump and save to the disk
If the AAA server is marked down: ping tcp to the server on port 49, show aaa-server
to gather statistics, save to a file on disk, use SCH to email the file contents

Manual events
Gather the output of 10 different commands and save to a file

49

2014 Cisco and/or its affiliates. All rights reserved.

49

Embedded Event Manager

Goal: Backup the configuration when a user logs in, and again when they log off of a SSH
session
Determine the syslogs that should trigger the event
%ASA-6-605005: Login permitted from 14.36.103.220/54785 to 36net:14.36.103.88/ssh for user "cisco"
%ASA-5-611103: User logged out: Uname: cisco

Configure the event applet

event manager applet loginConfigBackup
event syslog id 605005
event syslog id 611103
action 1 cli command "show running-config"
output file rotate 50
!

Applet name
Trigger syslogs
Action Command
Output Destination

Files written to disk when a user logs in and then out

261
260
259

-rwx
-rwx
-rwx

161286
161331
161277

16:46:27 May 05 2014

16:46:14 May 05 2014
16:46:07 May 05 2014

eem-loginConfigBackup-0.log
eem-loginConfigBackup-1.log
eem-loginConfigBackup-2.log
50

2014 Cisco and/or its affiliates. All rights reserved.

50

VPN AUTHENTICATION DEBUGS

1
*NOT
RANKED BY IMPORTANCE.
2014 Cisco and/or its affiliates. All rights reserved.

52

Authentication Problems
debug webvpn <1-255>
Good Authentication

WebVPN: calling AAA with ewsContext (-925550560) and nh (927982512)!

WebVPN: started user authentication...
WebVPN: AAA status = (ACCEPT)
WebVPN: user: (user1) authenticated.
Bad Authentication

WebVPN: started user authentication...

webvpn_free_auth_struct: net_handle = 0xc839fc30
webvpn_allocate_auth_struct: net_handle = 0xc839fc30
webvpn_free_auth_struct: net_handle = 0xc839fc30
webvpn_auth.c:webvpn_aaa_callback[5107]
WebVPN: AAA status = (ERROR)
WebVPN: callback data is not valid!!
webvpn_remove_auth_handle: auth_handle = 5
53

2014 Cisco and/or its affiliates. All rights reserved.

53

RADIUS Authentication Problems

debug radius
RADIUS packet decode (authentication request)
-------------------------------------Raw packet data (length = 150).....
01 11 00 96 53 90 89 8e af bc 45 9a cb a8 c1 66
a7 54 fd f2 01 07 75 73 65 72 31 02 12 07 6f 5c
c4 03 ae cf cc bf df ec 1d 58 0f 31 38 05 06 00
00 70 00 1e 11 32 30 39 2e 31 36 35 2e 32 30 30
2e 32 32 35 1f 11 32 30 39 2e 31 36 35 2e 32 30
30 2e 32 32 36 3d 06 00 00 00 05 42 11 32 30 39
2e 31 36 35 2e 32 30 30 2e 32 32 36 04 06 0a 0a
0a fe 1a 24 00 00 00 09 01 1e 69 70 3a 73 6f 75
72 63 65 2d 69 70 3d 32 30 39 2e 31 36 35 2e 32
30 30 2e 32 32 36
Parsed packet data.....
Radius: Code = 1 (0x01)
Radius: Identifier = 17 (0x11)
Radius: Length = 150 (0x0096)
Radius: Vector: 5390898EAFBC459ACBA8C166A754FDF2
Radius: Type = 1 (0x01) User-Name
Radius: Length = 7 (0x07)
Radius: Value (String) =
75 73 65 72 31
Radius: Type = 2 (0x02) User-Password
Radius: Length = 18 (0x12)
send pkt 172.18.104.83/1645
RADIUS_SENT:server response timeout
RADIUS
RADIUS_DELETE
2014 Cisco and/or its affiliates. All rights reserved.
remove_req 0xcbeb5d00 session 0x14 id 17

|
|
|
|
|
|
|
|
|
|

....S.....E....f
.T....user1...o\
.........X.18...
.p...209.165.200
.225..209.165.20
0.226=.....B.209
.165.200.226....
...$......ip:sou
rce-ip=209.165.2
00.226

user1

Server not Responding

54

54

Domain Authentication Problem

debug ntdomain
Domain Controller Communication Problem

smb: negotiate phase failed: syserr = Network is down

Cifs_Connect_Server() returned FALSE, error_code = 18
ntdomain_process_ntinfo - state is NTDOMAIN_DELETE
INFO: Attempting Authentication test to IP address <172.18.85.123>
(timeout: 12 seconds)
ERROR: Authentication Server not responding: No error

Note: In this Example the Administrator Attempts to Authenticate to the Active Directory Server Using the TEST Utility Within ASDM
55

2014 Cisco and/or its affiliates. All rights reserved.

55

Additional Authentication Debugs

For Your
Reference

You can combine the debugs listed above with the debug webvpn
and debug aaa common when troubleshooting clientless
authentication problems.
56

2014 Cisco and/or its affiliates. All rights reserved.

56

Authentication Test Utility

Using the CLI:

test
aaa-server authentication NYGroup host 172.18.85.123 user domainuser password 123qweasd
2014 Cisco and/or its affiliates. All rights reserved.
57

57

 2014 Cisco and/or its affiliates. All rights reserved.

58

Useful Show Commands

show vpn-sessiondb
asa# show vpn-sessiondb
--------------------------------------------------------------------------VPN Session Summary
--------------------------------------------------------------------------Active : Cumulative : Peak Concur : Inactive
---------------------------------------------AnyConnect Client
:
12 :
22 :
12 :
0
SSL/TLS/DTLS
:
12 :
22 :
12 :
0
--------------------------------------------------------------------------Total Active and Inactive
:
12
Total Cumulative :
22
Device Total VPN Capacity
:
25
Device Load
:
0%
----------------------------------------------------------------------------------------------------------------------------------------------------Tunnels Summary
--------------------------------------------------------------------------Active : Cumulative : Peak Concurrent
---------------------------------------------AnyConnect-Parent
:
12 :
22 :
12
SSL-Tunnel
:
12 :
22 :
12
DTLS-Tunnel
:
12 :
22 :
12
--------------------------------------------------------------------------Totals
:
12 :
6
2014 Cisco and/or its affiliates. All rights reserved.

59

show vpn-sessiondb additional options

asa# show vpn-sessiondb ?
exec mode commands/options:

anyconnect
detail
email-proxy
full
index
l2l
license-summary
ra-ikev1-ipsec
ratio
summary
vpn-lb
webvpn
|
<cr>

AnyConnect sessions
Show detailed output
Email-Proxy sessions
Output formatted for data management programs
Index of session
IPsec LAN-to-LAN sessions
Show VPN License summary
IKEv1 IPsec/L2TP-IPsec Remote Access sessions
Show VPN Session protocol or encryption ratios
Show VPN Session summary
VPN Load Balancing Mgmt sessions
WebVPN sessions
Output modifiers

2014 Cisco and/or its affiliates. All rights reserved.

60

debug webvpn
omar-asa# debug webvpn ?
<1-255>
anyconnect

webvpn anyconnect debugging

chunk
cifs

webvpn chunk debugging

webvpn cifs debugging

citrix
compression

webvpn citrix debugging

webvpn (anyconnect) compression debugging

cstp-auth
customization

webvpn cstp-auth debugging

webvpn customization debugging

failover
html

webvpn failover debugging

webvpn html debugging

javascript
kcd

webvpn javascript debugging

webvpn kcd debugging

listener
mus

webvpn listener debugging

webvpn MUS debugging

nfs
request

webvpn nfs debugging

webvpn request debugging

response
session

webvpn response debugging

webvpn session debugging

transformation
url

webvpn transformation debugging

webvpn url debugging

util
xml

webvpn util debugging

webvpn xml debugging

2014 Cisco and/or its affiliates. All rights reserved.

<cr>

61

SSL VPN TROUBLESHOOTING VIDEOS

 2014 Cisco and/or its affiliates. All rights reserved.

63

DEBUG DAP TRACE

ASA(config)# debug dap trace
The DAP policy contains the following attributes:
------------------------------------------------1: action = continue
DAP_open: C9EEE930
DAP_add_CSD: csd_token = [4287F77A4F7347A553F4619C]
[ 0]: aaa.cisco.username = user2
[ 1]: aaa.cisco.tunnelgroup = DefaultWEBVPNGroup
dap_add_to_lua_tree:aaa["cisco"]["username"] = "user2";
dap_add_to_lua_tree:aaa["cisco"]["tunnelgroup"] = "DefaultWEBVPNGroup";
dap_clienttype_to_string(3) returns CLIENTLESS
dap_add_to_lua_tree:endpoint["application"]["clienttype"] = "CLIENTLESS";
dap_add_csd_data_to_lua:
endpoint.os.version = "Windows XP";
endpoint.os.servicepack = "2";
endpoint.location = "Default";
endpoint.protection = "secure desktop";
endpoint.fw["MSWindowsFW"] = {};
endpoint.fw["MSWindowsFW"].exists = "true;
64

2014 Cisco and/or its affiliates. All rights reserved.

64

Continuation of the debug dap

trace output

DEBUG DAP TRACE

endpoint.fw["MSWindowsFW"].description = "Microsoft Windows Firewall";

endpoint.fw["MSWindowsFW"].enabled = "true";
endpoint.av["McAfeeAV"] = {};
endpoint.av["McAfeeAV"].exists = "true";
endpoint.av["McAfeeAV"].description = "McAfee VirusScan Enterprise";
endpoint.av["McAfeeAV"].version = "7.0.0";
endpoint.av["McAfeeAV"].activescan = "true";
endpoint.av["McAfeeAV"].lastupdate = "132895";
endpoint.as["SpyBot"] = {};
endpoint.as["SpyBot"].exists = "true";
endpoint.as["SpyBot"].description = "Spybot - Search & Destroy 1.4";
endpoint.as["SpyBot"].version = "1.4";
endpoint.as["SpyBot"].activescan = "false";
endpoint.as["SpyBot"].lastupdate = "996895";
endpoint.enforce = "success";
Selected DAPs: McAfee-7,SpyBot

dap_request: memory usage = 19%

dap_process_selected_daps: selected 3 records
dap_aggregate_attr: rec_count = 3
DAP_close: C9EEE930
65

2014 Cisco and/or its affiliates. All rights reserved.

65

 2014 Cisco and/or its affiliates. All rights reserved.

66

IPSec Debugs and Show Commands

omar-asa# debug crypto ?
ca

Set PKI debug levels

condition

Set IPSec/ISAKMP debug filters

engine

Set crypto engine debug levels

ike-common

Set IKE common debug levels

ikev1

Set IKEV1 debug levels

ikev2

Set IKEV2 debug levels

ipsec

Set IPSec debug levels

ss-api

Set Crypto Secure Socket API debug levels

vpnclient

Set EasyVPN client debug levels

2014 Cisco and/or its affiliates. All rights reserved.

67

IKEv2 Debug Commands

debugs specific for IKEv2

debug crypto ikev2 platform

Debugs ASA processing of IKEv2, not protocol specific exchanges.
This debug is useful for AAA and session management issues. Also to troubleshoot
the ASA cryptographic module performing encryption and decryption.
debug crypto ikev2 protocol
Debugs IKEv2 protocol specific exchanges.
debug crypto ikev2 timer

Debugs IKEv2 timer expiration. Useful when clients are complaining that their
connection is being timed-out too often.
Note: debug crypto ike-common can be used for both IKEv1 and IKEv2

68

2014 Cisco and/or its affiliates. All rights reserved.

68

show crypto ipsec sa

ciscoasa(config)# show crypto ipsec sa
interface: outside
Crypto map tag: def, local addr: 10.132.0.17

local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)

remote ident (addr/mask/prot/port): (172.20.0.21/255.255.255.255/0/0)
current_peer: 172.20.0.21
dynamic allocated peer ip: 10.135.1.5

#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0

#pkts decaps: 1145, #pkts decrypt: 1145, #pkts verify: 1145
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 2, #pre-frag failures: 1, #fragments created: 10
#PMTUs sent: 5, #PMTUs rcvd: 2, #decapstulated frags needing reassembly: 1
#send errors: 0, #recv errors: 0

2014 Cisco and/or its affiliates. All rights reserved.

69

show crypto ipsec sa (cont.)

local crypto endpt.: 10.132.0.17, remote crypto endpt.: 172.20.0.21
path mtu 1500, ipsec overhead 60, media mtu 1500
current outbound spi: DC15BF68
inbound esp sas:
spi: 0x1E8246FC (511854332)
transform: esp-3des esp-md5-hmac
in use settings ={L2L, Transport, Manual key, (OSPFv3), }
slot: 0, conn_id: 3, crypto-map: def
sa timing: remaining key lifetime (sec): 548

IV size: 8 bytes
replay detection support: Y
outbound esp sas:
spi: 0xDC15BF68 (3692412776)

transform: esp-3des esp-md5-hmac

2014 Cisco and/or its affiliates. All rights reserved.

70

show crypto ipsec sa (cont.)

transform: esp-3des esp-md5-hmac
in use settings ={L2L, Transport, Manual key, (OSPFv3), }
slot: 0, conn_id: 3, crypto-map: def
sa timing: remaining key lifetime (sec): 548
IV size: 8 bytes
replay detection support: Y
Crypto map tag: def, local addr: 10.132.0.17
local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)

2014 Cisco and/or its affiliates. All rights reserved.

71

 2014 Cisco and/or its affiliates. All rights reserved.

72

AnyConnect Diagnostics and Reporting Tool

useful for troubleshooting AnyConnect installation and connection problems

To Launch DART go to
the Status Overview
Tab and click on
Diagnostics
1
73

2014 Cisco and/or its affiliates. All rights reserved.

73

DART Wizard
Under Bundle Creation Option, select Default or Custom. The Default option includes the typical log files and
diagnostic information. DARTBundle.zip is saved to the local desktop. If you choose Custom, the DART wizard
allows you to specify where and what files want to include in the bundle.

74

2014 Cisco and/or its affiliates. All rights reserved.

74

DART Wizard
continued

75

2014 Cisco and/or its affiliates. All rights reserved.

75

DART Bundled Files

Advanced Detailed Logs for each Installed Module in AnyConnect
DART BUNDLE SUMMARY
Username:
Time:

unknown (user is offline, or username was not specified in Request)

Tue Apr 05 17:12:17 2011

OS:
OS username:
Upload URL:
DART Mode:
Bundle on client computer:

Win7 : WinNT 6.1.7600

omar
None (offline mode)
User-Initiated/Offline Mode
C:\Users\omar\Desktop\DARTBundle_0405_1353.zip

=============================================================================================================================================
Cisco AnyConnect Secure Mobility Client:
Files Included in Bundle:
ID
Filename
Description
Truncate? Final Size Orig. Size
---------------------------------------------------------------------------------------------------------------------------ac-install
update_pre3.0.txt
AnyConnect install logs. Includes web
No
10 bytes
10 bytes
and standalone install logs
ac-install
anyconnect-win-2.3.0254-web
AnyConnect install logs. Includes web
No
322.35K
322.35K
-deploy-k9-install-22203701
and standalone install logs
062010.log
ac-install
update.txt
AnyConnect install logs. Includes web
No
10 bytes
10 bytes
and standalone install logs
ac-install
VPNManifest.dat
AnyConnect install logs. Includes web
No
181 bytes
181 bytes
and standalone install logs
ac-install
AnyConnectLocalPolicy.xml
AnyConnect install logs. Includes web
No
589 bytes
589 bytes
and standalone install logs
ac-install
UpdateHistory_20110405_1244
AnyConnect install logs. Includes web
No
705 bytes
705 bytes
00_log.txt
and standalone install logs
ac-logs
AnyConnect_pre3.0.txt
AnyConnect application logs
No
3.62M
3.62M
ac-logs
AnyConnect.txt
AnyConnect application logs
No
227.40K
227.40K
ac-logs
AnyConnect.evtx
AnyConnect application logs
No
1.06M
1.06M
ac-profile
CALO.xml
AnyConnect Profile
No
1.46K
1.46K
ac-profile
AnyConnectProfile.xsd
AnyConnect Profile
No
93.22K
93.22K
global-preferenc
preferences_global.xml
AnyConnect Global Preferences
No
546 bytes
546 bytes
es
user-preferences
preferences.xml
AnyConnect User Preferences
No
590 bytes
590 bytes
va-runtime
setupapi.app.log
Virtual Adapter runtime logs
No
320.88K
320.88K
va-runtime
setupapi.dev.log
Virtual Adapter runtime logs
No
9.70M
9.70M
2014 Cisco and/or its affiliates. All rights reserved.
----------------------------------------------------------------------------------------------------------------------------

MANY, MANY, MANY, MANY more

76

76

ANYCONNECT STATISTICS VIDEO

Thank you.