Common Troubleshooting Hints and Tricks

Config Files
Review basic navigation of the CLI, looking at software versions and license specifics.
Two methods of viewing the configuration are available from the CLI: show config
running from the main prompt and show from the configure prompt.
To view the configuration in set mode rather than XML format, set cli configoutput-format set should be entered from the main prompt. This will then change
the output shown in configure mode.
Moving a configuration file over from a TFTP server is done with the tftp import
configuration command.
The difference between the running and candidate configs can be seen with show
config diff.
The commit command from the configure prompt will apply changes.
Key points:
Comfort with XML and set formats for configuration
Process for loading and committing from the CLI.

PAN-EDU-311

Useful commands include:

show interface all
show running security-policy/pbf-security/nat-policy
show routing summary
Key points:
Reviewing policy on CLI

Tech Support Files

Creation of a tech support file from the CLI can also be done with the scp export
tech-support command. This can also be done via the GUI.
Note: the command request tech-support dump will generate a tech-support file,
but you cannot download this file from the CLI. You can download this from the GUI,
however.
Extraction of the contents of the tech support file will require a tool which can open a
TAR archive. 7-Zip is one such program and is provided for the lab.
System processes can be viewed with show running resource-monitor and
show system resources. Disk space can be examined with show system diskspace.
Discussion/Caveats:
Tech support file generation is more elegant through the GUI due to the ability to
generate it and export it later.
There is no good/simple way to view what debug commands are enabled on the
system. Looking through show system state for references to debug can
provide some insight, but there is no show debug command to present all the data
in a pretty manner.
Key points:
Generation, export, and viewing of tech support file

Lab Manual Rev A

PAN-OS 5.0

Page 2

PAN-EDU-311

Counters and Debug Commands

Counters can be viewed with show counter. Look at the interface and global
counters. Filters on the global counters can help restrict data to be viewed. show
counter global filter delta yes provides a view which only shows value
changes since the last viewing.
Multiple steps may be required to properly perform the packet capture:
(optional but recommended) debug dataplane packet-diag set filter
match
(optional but recommended) debug dataplane packet-diag set filter on
debug dataplane packet-diag set capture stage [stage] file
[filename]
debug dataplane packet-diag set capture on
Capture can be viewed on device with view-pcap filter-pcap [filename]
Discussion/Caveats:
Interface counters do not have a delta filter
There is a good example of the packet capture process in the 201/205 training guides
Applying filters to packet capture are important to targeted collection of data and limiting
its potential to consume significant system resources
Viewing PCAP data on the box is a good method of ensuring it is actually working prior
to attempting to move the file to an external system.
PCAP is an excellent tool but requires practice to perform with ease and understand the
impact of various operations.
The GUI in 4.X now provides the same functionality as CLI packet capture. The
concepts, commands, and ordering are the same so reinforcement of concepts is
important regardless of the interface.
Best practices include:
Capture what you need then turn it off
Use filters and use them properly
Do not clear filters while capture is still running. Turn off the capture before changing
filters
Packet capture only exists for six debug areas data plane, dhcpd, ike, l3svc, pppoe,
and routing
Key points:
Performing packet capture from the CLI
Setting capture stage is important, often overlooked

Lab Manual Rev A

PAN-OS 5.0

Page 3

PAN-EDU-311

Management/Admin issues
Management interface settings.
> show system services
To turn on services use commands like the ones below (for https GUI)
# set deviceconfig system service disable-https no
Discussion/Caveats:
Interface access is configured by service.

Admin Roles
There can be multiple roles configured on the firewall (Students and students). Roles
can be customized to have limited functionality.
Discussion/Caveats:
System logs can provide general details.
General searching of log files groups can be performed (grep mp-log *
pattern [word]) to find which log files contain which concepts
Key points:
Pay attention to which role is assigned to a user account.

User accounts can be authenticated by different mechanisms.

Some authentication problems are not corrected on the firewall, but may be RADIUS
or AD problems.

Key points:
Pay attention to how a user is authenticated.

Dynamic Update Problems

The command request system software check will demonstrate connectivity
breakage.
A review of the system log looking for information on update will show the improper
server name of update.paloaltonetworks.com: show log system | match update.

Lab Manual Rev A

PAN-OS 5.0

Page 4

PAN-EDU-311

This can be corrected in configuration mode with set deviceconfig system

update-server
Existing software versions can be seen in the show system info command.
Discussion topics and caveats:
Visibility of the update server name is easily visible in the GUI. The error can be clearly
seen through the CLI.
Log file review is key to identifying the specifics of the problem.
Key points:
Log availability on CLI vs. GUI
Familiarization with CLI access and viewing of log files

DHCP issues
In this scenario, the DHCP server may be misconfigured to give out a gateway address
(ie:192.168.1.100 instead of 192.168.1.1) The desktops will not be able to ping (or
access) the default gateway and only by looking at the correct topology will one be able
to identify and correct the problem.
The default gateway for the firewall could also configured incorrectly. (The next-hop is
configured for 172.16.X.253 when according to the topology, the correct next-hop
should be 172.16.X.254).
Discussion topics and caveats:
Knowing the topology will help troubleshoot this problem
After the DHCP problem is solved, debug flow will help identify the incorrect routing
issue.
Key points:
Look at the known topology
Dont take anything for granted
Ask if you are not sure!

Page Display issues (get part of a web page, but not all of it)
In this scenario, there is a Policy Based Forwarding policy configured to route the webbrowsing application to a next-hop which is unreachable.
Discussion topics and caveats:
DNS is not using PBF, so the title pages are resolved. Web-browsing is being blackholed, so the content on the pages will not display.
The traffic logs will show the application as incomplete.

Lab Manual Rev A

PAN-OS 5.0

Page 5

PAN-EDU-311

Key points:
Troubleshooting PBF may be difficult, but debug flow should help.
Solution is to delete or disable the PBF rule or use a Monitor profile that if the next hop
is unreachable, use the routing table.

Routing Issues
The adjacent OSPF router does not have an OSPF authentication profile or password,
while this firewall does (palaalto).
Viewing the routed log file (less mp-log routed.log) an error message will show
an authentication mismatch. The neighbor router will show authentication type 0, while
this one has type 1. You will need to infer that this means the neighbor router does not
have an authentication profile/password configured, and they will have to remove it from
the OSPF interface configuration on the local firewall.
Once the auth-profile is removed, they will be able to see, again in the log file, a
mismatched hello timer. This can also be corrected through the CLI. This will complete
the lab. At this point, a show routing protocol ospf neighbor should show the
neighbor. show routing route should show plenty of OSPF routes.
Students may also attempt a packet capture on the interface to view the packet in its
entirety. This would allow correction of the problem in one stage rather than two.
Discussion/Caveats:
System logs can provide important details.
General searching of log files groups can be performed (grep mp-log *
pattern [word]) to find which log files contain which concepts
Key points:
Logs are useful for this exercise; this level of log detail would not be visible in the GUI (a
PCAP would be required)

NAT issues
Testing of the NAT policies using the test command will first show if NAT policies are in
the incorrect order and must be moved must be moved Point out the move
command in the config CLI, which will make this process much easier.
Discussion topics and caveats:
Diagnosing the impact of NAT rules is straightforward via the CLI
Re-ordering of NAT rules is easier via the GUI. The move command allows rule
reordering in the CLI.

Lab Manual Rev A

PAN-OS 5.0

Page 6

PAN-EDU-311

Once the first NAT rule is matched, no other translation will be applied. For any
destination NAT, the requirement for source translation needs to be considered.
Key points:
Order of processing matters for all rule bases (NAT, security policy, QoS)
Only one rule hits, the first match, in a rule base
Source and destination can be included in the same rule

VPN Tunnel Issues

A static route needs to be in place to direct traffic through the tunnel.
Problems:
Peer Gateway IP address is incorrect.
The preshared key does not match.
No-pfs is set on the even numbered FW for P2.
Examine your routing table and interfaces to identify how this traffic should flow and
then identify what is broken.
Useful commands include: (be sure to try to send traffic to neighbor first)
show routing route
show VPN flow
show VPN gateway [name]
show vpn ike-sa
ikemgr log file
debug ike pcap on (when you view this file, you should use verbose++ option, i.e.
view-pcap verbose++ yes debug-pcap ikemgr.pcap
Key points:
Documenting the lab, understanding the architecture
Utilizing the logs
Multiple paths exist to isolating the problem

Lab Manual Rev A

PAN-OS 5.0

Page 7

PAN-EDU-311

Security Policy Rules

Example

In this example there is no permit for web-browsing, so any websites listed will be
inaccessible. This is the key concept of application dependency.
Explicit allows for applications (ping, ssl) and applications (LinkedIn, Facebook,
Pandora, Gmail)
Explicit denies for Gmail-chat (prior to Gmail)
Explicit deny all at the end
Also note that web browsing is not equivalent to port 80.
Look in show log traffic action equal deny to see web-browsing denies.
The application dependency error is visible in the mp-log ms.log. A grep of all mp-logs
for the keyword dependency would reveal this.
Discussion topics and caveats:
It is important to understand the impact of application dependencies. Problems in this
space can be seen in the mp-log files.
Lab Manual Rev A

PAN-OS 5.0

Page 8

PAN-EDU-311

Ordering of applications may not make a difference in control but can help with
readability.
The GUI commit is more clear in understanding dependencies, cannot simply rely on its
output for complex configurations
Copy and paste through CLI set mode works, but command ordering adds challenge.
Key points:
Web browsing is an application, not equivalent to port 80
Apps change within a single session
Application dependency concept

Decryption
The encrypted (HTTPS) nature of the Gmail site prevents blocking of the talk
application.
A decryption filter needs to be put in place to allow this content to be processed as the
proper application.
A certificate must be generated, set as the forward ssl-decrypt cert, and a decryption
policy enabled.
> request certificate generate ca yes certificate-name ssl-9 name
10.30.11.9
# set shared ssl-decrypt forward-trust-certificate ssl-9
# set rulebase decryption rules ssl-9-decrypt from trust to
untrust source any destination any category any action decrypt
type ssl-forward-proxy
show system setting ssl-decrypt setting will show if forward proxy cert is
ready.
Discussion/Caveats:
Lack of visibility of data due to encryption is common.
This decryption puts the firewall into man-in-the-middle decryption. This functionality
can be visible to end users.
Decryption is intended to allow policy application rather than interception of traffic
Key point:
Decryption required to apply policy to an encrypted session

Lab Manual Rev A

PAN-OS 5.0

Page 9

PAN-EDU-311

Deny All Security Rules

If there is a Deny ALL security policy with logging which is common in corporate
environments wishing to log all dropped traffic. In this scenario, however, the source
and destination zone are both set to ANY which causes traffic within the same zone.
Discussion topics and caveats:
The use of ANY in a zone should be used carefully. Also be aware that a Deny ALL rule
(explicit) will break the implicit intra-zone allow traffic. In this case, you would have to
explicitly allow intra-zone traffic and place those policies above the Deny ALL rule.
Key points:
When you use ANY- ANY in the source and destination zones with the action drop, it will
also drop packets to the firewall. In the VPN case, the IKE packets to the firewall will be
dropped.

User ID issues
Key points:
View mapping of User-ID attributes in the CLI
show user ip-user-mapping all
show user user-id-agent statistics

Lab Manual Rev A

PAN-OS 5.0

Page 10