Royal Protocol Forcepoint Project

 Problem 1: After the migration at HQ, all services were working fine except for the SSH
service to the network devices that was failing to connect. The rule to allow the SSH traffic
was in place still it was not allowing the connection.
Solution: The SSH request was traversing through another firewall in between that was
doing NAT to the originating traffic. By default, the ForcePoint firewall does a “Connection
Tracking”.
Connection tracking means that the engine keeps a record of all currently open connections
(stateful inspection).
With connection tracking, the engine can verify that the connection proceeds according to
the protocol standards. Connection tracking is also required for changing addresses using
NAT and enforcing some other connection-related settings. By default, connection tracking
is on.
However, it is not necessary to track the state of certain kinds of connections (for example,
SNMP traps). Some applications can also communicate in a non-standard way that
prevents them from being allowed through the engine when connection tracking is used.
For those connections, you can disable connection tracking in the Access rules, which
allows the engine to function as a simple packet filter for those connections. However,
disabling connection tracking also prevents the use of some features that require connection
tracking.

When connection tracking is off, each packet that you want to allow must match an Access
rule that allows the packet. This means that even if two-way communications are always
opened one way, both directions of communication must be explicitly allowed in Access
rules. Reply packets are not recognized, so they are not automatically allowed through. If
done carelessly, turning off connection tracking reduces the benefits you gain from your
engine and might even weaken security. You might have other options: sometimes using the
correct Protocol Agent helps.
 Problem 2: Remote access VPN was not working after migration.

Solution: The DHCP configuration for VPN users was configured to be relayed from AD
server. However, the defined DHCP range was from a different subnet as the actual NIC
subnet of the AD server. After creating a new DHCP range from the same subnet as the AD
NIC address, the DHCP relay worked and VPN started working.

 Problem 3: Diriya branch firewall was able to do initial contact with the SMC server (with
appropriate policies to publish the SMC server) in RoyalProtocol HQ data center but we were
unable to push policies to Diriya firewall from the SMC server.

Solution: We needed to enable the “Node initiated contact to Management Server” option
to make this work since the firewall was behind the router, the SMC could not contact it
with its private IP.

Apart from this, we also saw in logs that “SG Control” traffic from Diriya firewall towards the
SMC Server (HQ FW wan IP) was getting blocked. When we enabled this traffic inside the
policy, the policy push worked. As per theory, this policy was not required but for some
reason it was not working without this policy. I didn’t have much time to test it again by
removing this policy.
  Problem 4: No internet access to users after Diriya branch firewall migration.

Solution: Diriya branch firewall was implemented behind a NAT router that was connected
to the internet. The firewall WAN interfaces were configured with private IP ranges
(192.168.101.0/24) as that branch had just one public IP that was configured on the NAT
router. That NAT router was doing the NAT of the private IP to the public IP. The firewall was
configured with just the IPv4 rule to allow any-any. The NAT policy was not configured
because the NAT was being done on the router. This didn’t work. The problem here was that
the user subnet was 10.0.0.0/16 network and the NAT configured on the router was natting
192.168.101.2 (firewall virtual ip) to the public IP. As the router didn’t know about the 10
network, it was not natting it and hence the users were not getting allowed to the internet
(they were getting allowed through the firewall though as any-any IPv4 policy was in place).
To resolve this, we configured a NAT policy on the firewall to NAT any source that wanted
to go to the internet, with the firewall VIP (192.168.101.2). hence the 10 network would
get translated to 192.168.101.2 and then the WAN router would translate this to the public
IP.

PSS Data Center Project at Tabuk and Medina

 Problem 1: The port-channel between the management switch and the Fortigate firewall
was not coming up.

Solution: Two ports (10Gig) from the management switch were connected to two ports
(25Gig) on the firewall. LACP was configured at both ends and ideally the ports should do
auto negotiation of the speed and the port channel should come up. In our case it did not
happen. When we issued the command “show interface status” on switch end, we saw that
the port is in down-down state and showing as “not connected”. Initially I though that this is
a switch side issue may be the SFP or the patch chord at switch end as it was a layer 1 issue
and port was showing as not connected as switch end. The actual problem however was that
at the firewall end, we were using 25Gig ports and at the switch end we were using the
10Gig ports. This is fine, but the SFP installed in the firewall 25Gig slots were 10Gig SFP’s.
This would not work. If the port is 10 Gig and we install a 25 Gig SFP in it, it will negotiate
10Gig speed automatically. However, the opposite is not true. We cannot install a 10Gig SFP
in a 25Gig port. The port will not come up. To resolve this, we statically configured the
firewall 25Gig slot to operate at 10Gig speed.

Config system interface

Edit port 28
Set speed 10000full
As soon as we did this, the port-channel came up.
NGHA BlueCoat Proxy Project

 One of the issues that you were facing was slow browsing through the Proxy. This we were
able to resolve today by changing the web authentication method from “Proxy” to
“Proxy_IP”. This effectively reduced the number of authentication requests sent from proxy
to the AD and thus improved the overall proxy response time to user request. “Proxy” mode
sends user authentication challenge “407 requests” with each client session whereas, the
“Proxy_IP” mode sends challenge only for the first session. For the subsequesnt sessions,
the proxy keeps a user to IP mapping in its “Surrogate Cache” and no authentication
challenges are sent by the proxy for subsequent sessions from the same IP. The default
reauthentication time for “Proxy_IP” mode is 15 minutes but it can be changed.

Slowness could also be due to high CPU utilization. High CPU was reported at Riyadh site.
We bypassed several HTTPS sites from SSL interception to reduce the load on the proxy
(streaming sites) as a troubleshooting step. This reduced the CPU utilization a bit but it was
still high. High CPU could also be due to ICAP scanning. We bypassed several sites from ICAP
scanning that reduced the CPU utilization a bit again but not considerably.

The actual issue was the OS version. We had to upgrade the OS to the recommended
version by Symantec to resolve the issue.

 Another issue was in Jeddah deployment where they wanted policies based on hostnames.
We enabled this on the CLI as per the guide but still it was not working. Th policy trace was
showing that it is getting blocked because of the configured policy (although there was no
policy blocking it). The problem was that you need to enable RDNS at two places: one on
the CLI and the other on the VPM.
Through the CLI, you give the command, “policy restrict-rdns none”. Then on the VPM >
Configuration > Set reverse DNS lookup restrictions > select none.

 One of the prominent issues was “error connecting to SG” error message while connecting
to the proxy HTTPS web console. You need to bypass the proxy ip and port in browser proxy
settings. Or you can simply remove all proxy settings from the PC you wish to connect to
the proxy web console. ProxySG v6.7 and later do not support proxied sessions to the web
console.

 We faced issue in reporter configuration. The admin account was not getting accepted
while trying to integrate Proxy with the Reporter. To integrate the proxySG appliances to the
Reporter, you need to create a non-admin account on the reporter with complex password
and use the same credential on the proxySG to integrate it to the reporter. For each region
proxy, you create a separate DB and a user role to access his own DB on the reporter. This
same user must be selected in reporter FTP settings and FTPS setting drop down menu on
reporter configuration.

 Few users were getting “Appliance Error” while accessing the internet. If a user is restricted
to login from few authorized workstations, he will get “Appliance Error” while browsing
through the proxy as in the AD proxy is not one of the authorized workstations for this user.
To resolve this, you need to add the proxy hostname to the list of authorized workstations
for this user.
 ITB Network Migration Project

 Problem 1: I tested the VPN access (both SSL and IPSEC) on 2P link before migration and it was
working fine. But after migration, the VPN was not getting connected.

Solution: During the testing, I connected just the 2P internet link to the firewall so the incoming and
outgoing traffic had just one route. However, after migration, I connected the other WAN link, i.e the
STC internet along with the 2P internet link. The VPN traffic was hitting the 2P interface from
outside. But the return traffic was taking the STC route as STC route had lower “Distance” of 5 as
compared to 2P route which had a distance of 10 which is default for static routes. This was failing
the RPF check and hence the VPN connection was failing. To resolve this, we reduced the distance
value of 2P interface to be same as STC interface which was 5. This created another issue that both
the links would be considered as ECMP and user normal internet traffic would be load balanced
across both links. However, we wanted to use only STC link for outgoing internet traffic and use 2P
link only for VPN traffic. To resolve this, we set the “Metric” for 2P route to be 10 which was higher
than STC route which was 0 (dynamic learned routes have a metric of 0 by default.)

2P route settings:

From Fortinet website, we have below explanation on this scenario:

 Problem 2: After migration, Avaya phones in VLAN 20 were not getting registered with Avaya server
in VLAN 1.

Solution: Avaya phones send broadcast messages in that VLAN where they are connected to search
for the server to get configuration from it. Since the phone and the server were in different VLAN’s.
the broadcast messages from IP phones were not reaching the Avaya server. To resolve this, we
needed to add option 242/176 (new phones/old phones) with the Avaya server IP to make this
request unicast from phones to the server IP directly.
ip dhcp pool VOICE_POOL
network 192.168.20.0 255.255.255.0
default-router 192.168.20.1
option 242 ascii
"MCIPADD=192.168.1.130,MCPORT=1719,HTTPSRVR=192.168.1.130,L2QVLAN=20" ---> For
new Avaya phones
option 176 ascii
"MCIPADD=192.168.1.130,MCPORT=1719,HTTPSRVR=192.168.1.130,L2QVLAN=20" ---> For
old Avaya phones
Notes:
1. 192.168.1.130 is Avaya server, which is in vlan 1. Phones are in vlan 20.
2. L2QVLAN is voice vlan tag in dhcp option. MCIPADD=192.168.1.130 command is instructing
phones to contact server at 192.168.1.130.
2. When the phones comes up initially, it is assigned data vlan and then the switch recognizes it
as IP phone and then bounce it to the voice vlan.
3. Remember to enable LLDP on switch globally.
4. Voice vlan should be different from vlan 1 in this case as voice traffic needs to be tagged and
vlan 1 is untagged (native vlan)

KACARE Proxy Project

 Problem 1: Getting “Error: Keyring does not have a certificate authority's certificate” while
installing the AD signed SSL certificate on the proxy.

Solution: For using an SSL certificate on proxy for SSL interception, the certificate needs to be signed by
using “Subordinate CA Template”. The signed certificate must have the CA subordinate attribute that
will be there only if it is signed using the subordinate CA template. Since the proxy will use this certificate
to unencrypt the HTTPS certificate, analyze it and then re-encrypt it again, the certificate should have the
signing capability. This capability is attained by creating this certificate using the subordinate CA
template.
The certificate that the KACARE team provided, was signed by their “Web Server template” which was
missing the subordinate CA attribute and hence we were getting this error.
 Problem 2: Authenticating users in transparent deployment.

Solution: In transparent authentication, we need to create two IWA realms, one for authenticating HTTP
request, and the other for authenticating HTTPS request. Both these realms will have different “Virtual
URL’s”. Create a new service of type “HTTPS Reverse Proxy”, on a new port like port 4433, source any
and destination explicit. This new port will be used in HTTPS virtual URL for transparent authentication.
 Virtual URL for nonSSL_Auth IWA realm: http://proxy-dns-name
 Virtual URL for SSL_Auth IWA realm: https://proxy-dns-name:4433

Note that the proxy-dns-name mentioned here should be resolvable by DNS (should not be FQDN
name).

 Problem 3: There was a limitation in explicit proxy deployment from KACARE side. The users when
configured with explicit browser settings, would lose access to internet when they go home and try
to connect to internet from there since the proxy would not be reachable.

Solution: We have several solutions for this:

 Remove explicit proxy settings from browser when users go home. But this was not acceptable
by KACARE team as they did not want manual work.
 Use PAC file script to use direct connection when the proxy IP is unreachable. This would work
fine but again there are several applications that do not understand PAC configuration or they
just pick the first configured option (proxy IP) and do not fall back to other options. This would
again create problem and was non acceptable by KACARE team.
 You can configure the PAC file to be "smart", and allow the user to go DIRECT if the proxy
cannot be found. 
But instead of using GPO to push PAC settings to the client web browsers, Its recommend
pushing the PAC file out using DHCP WPAD or DNS WPAD. When you use this method, your
clients will only receive the PAC file instructions when they are on your corporate network (and
have access to your DHCP or DNS server).  You will not change their browser settings and the
browser will retain the "“Automatically detect settings” option.  This way, when the user goes to
their home network, they will not have any interruptions. 
Until you figure out how to get DHCP or DNS WPAD working, you can push the PAC file out with
your original method.  But once you master WPAD, switch the user browser setting back to
"“Automatically detect settings”.

On Cisco IOS: option 252 ascii "http://X.X.X.X/wpad.dat "

Note: Make sure after the .dat at the end there is a space before configuring the closing (“).

On Windows DHCP Server:

To create an option 252 entry in DHCP

1. Click Start, point to All Programs, point to Administrative Tools, and

then click DHCP.
2. In the console tree, right-click the applicable DHCP server, click Set
Predefined Options, and then click Add.
3. In Name, type WPAD.
 4. In Code, type 252.
5. In Data type, select String, and then click OK.
6. In String, type http://Computer_Name:Port/wpad.dat where:
o Computer_Name is the fully qualified domain name of the
ISA Server computer.
o Port is the port number on which automatic discovery
information is published. You can specify any port number.
By default, ISA Server publishes automatic discovery
information on port 8080.
7. Right-click Server options, and then click Configure options.
8. Confirm that Option 252 is selected.

Problem 4: SSL certificate errors on Google Chrome and Mozilla Firefox after installing KACARE AD
signed certificate. Microsoft Edge and Internet Explorer were working fine.

Solution: The certificate signed by KACARE AD was using SHA1 as hashing algorithm. This was not
trusted/supported by Chrome and Firefox as they are more locked down on security. They were
presenting certificate errors while visiting HTTPS sites and the proxy was intercepting it using the AD
signed certificate (using SHA1 algorithm). Internet explorer and Edge are not that particular on these
settings and hence they were not presenting any certificate errors on interception.
To resolve this issue, KACARE migrated their AD to new version that could use SHA256 algorithm for
signing certificates. After migration and generating new certificate, the issue was resolved.

Problem 5: Users were getting login prompt whenever they visited new website.

Solution: Since this was a transparent deployment, the authentication was being done transparently for
each new website that the user was visiting. This is default behavior in transparent proxy deployment.
To mitigate this issue, BlueCoat proxy uses virtual URL for transparent authentication. The first user
request is authenticated against this virtual URL and then the login information is stored in form of
cookie. For all subsequent authentication requests, the proxy would use the information stored in cookie
and will not prompt the users for login for 15 minutes by default (idle timeout). This timer can be
tweaked.
The solution to this problem has two steps:
I. Adding the virtual URL’s to “Trusted Sites” under “Internet Option” settings of windows.
This is valid for Internet explorer, Edge and Chrome.
 For Firefox however, you need to do it separately under “about:config” settings.

II. Enabling IWA support under “Internet Option > Security > Trusted Sites > Custom
Level” settings of windows. This is valid for Internet explorer, Edge and Chrome.
 For Firefox however, you need to do it separately under “about:config” settings.

EDIT-1: There is no need to create two realms with newer Browser and new proxy OS versions. We
need to create just one realm with the virtual URL as http://proxy-name-without-fqdn. The detect
protocol option under “External HTTP” will automatically redirect HTTPS traffic to HTTP virtual URL
for authentication. We use the same realm in user-based policies under web access layer.

Mistake 1: In the “Request URL Category”, we cannot use IP addresses here. Only domain
names will be read from here. Entering IP addresses here will not give any error, but it will
not take effect. IP addresses will not be read from here.
Also. the domain names should not contain * or any other characters. It can be either the
complete domain, or the exact subdomains.

Mistake 2: While adding IP addresses, if it is a single

host, you need to specify the network mask as
255.255.255.255. You cannot simply use the IP address for single host.

Mistake 3: Added the customer Primary and Secondary DNS servers in Primary DNS and Alternate DNS
settings in proxy which is wrong. The proxy will always query the servers in the Primary DNS list for
resolution. If the DNS Servers in the Primary DNS list are down, the proxy will NOT contact the servers
in the Alternate DNS list. Only if all the servers in the primary list are unable to resolve the name, the
Alternate DNS list will be used. For this reason, always add all customer DNS servers in the primary DNS
list in order of priority (top to bottom).

ITB Fortigate FSSO Implementation

Problem: Usernames were not getting logged in reports.

Solution: The first issue was that the FSSO fabric (AD Connector) was down. It was down because of the
login credentials that was used for AD integration. We were using service account for this. This
integration needs domain privilege account. After entering the domain privilege credentials, it came up.

The second issue was that the we checked firewall debug and found that FortiGate is sending a read
request for event ID 4776 (that contains User information). However, when we checked the windows AD
security events, we found the events are generating with ID 4624. This was the reason there was no
response from AD for the query because of event ID mismatch. As per customer support, there is no
workaround for this and we need to install FSSO agent on the AD in order to implement UID.

Debug commands used on FortiGate and the output as per below screenshot:

We see that the FortiGate is sending query request for event ID 4776