Web Enumeration & Information

Gathering
Introduction
Alexis Ahmed
Senior Penetration Tester & Red Teamer @HackerSploit
Red Team Instructor @INE

[email protected]
@HackerSploit
@alexisahmed
Course + Introduction To Web Enumeration & Information
Topic Gathering
Overview + Finding Website Ownership & IP Addresses
+ Reviewing Webserver Metafiles For Information
Leakage
+ Search Engine Discovery
+ Web App Fingerprinting
+ Source Code Analysis
+ Website Crawling & Spidering
+ Web Server Fingerprinting
+ DNS Enumeration
+ Subdomain Enumeration
+ Web App Vulnerability Scanning
+ Automated Recon Frameworks
+ Basic familiarity with the
web (TCP/IP, UDP &
HTTP) Prerequisites
+ Familiarity with
Windows & Linux
 ● You will learn how to use the OWASP Web Security Testing
Guide as a methodology for web app pentesting engagements.
● You will be able to perform passive web app information
gathering.
● You will learn how to to perform passive and active DNS
enumeration.
● You will learn how to detect web application firewalls (WAF).
Learning ● You will be able to utilize Google Dorks to find additional
information on target websites/web applications.
Objectives: ● You will learn how to perform spidering and crawling to identify
the content structure of websites.
● You will learn how to perform subdomain enumeration through
publicly available sources and through subdomain brute-force
attacks..
● You will learn how to perform file & directory enumeration.
● You will learn how to utilize Automated recon frameworks like
OWASP Amass.
 Disclaimer

In this course you will see some examples of tools and techniques being
used on real-world (public) IT assets.

Never run any of these tools and techniques on these addresses or on any
server and network without proper authorization!

In the context of this course, i will explicitly specify what websites you can
run tests on.

For most of the techniques, you will be provided with labs where you can
practice what you have learnt.
Let’s Get Started!
Introduction To Web Enumeration &
Information Gathering
 What is Information Gathering?
● Information gathering is the first step of any penetration test and
involves gathering or collecting information about an individual,
company, website or system that you are targeting.
● The more information you have on your target, the more successful you
will be during the latter stages of a penetration test.
● Information gathering is typically broken down into two types:
○ Passive information gathering - Involves gathering as much information as
possible without actively engaging with the target.
○ Active information gathering/Enumeration - Involves gathering as much
information as possible by actively engaging with the target system. (You
will require authorization in order to perform active information gathering)
 What is Information Gathering?
● Gathering information about the target server/web app is the initial
phase of any penetration test, and is arguable the most important phase
of the entire engagement.
● One of the nuances of this phase is that there is no unnecessary
information, everything you collect should be recorded/saved for future
use.
● In the context of web application penetration testing, the information
collected in this phase will become extremely useful in understanding
the website/web application logic and structure during the initial
access/exploitation phase.
What Information Are We Looking For?
● Website & domain ownership.
● IP addresses, domains and subdomains.
● Hidden files & directories.
● Hosting infrastructure (web server, CMS, Database etc).
● Presence of defensive solutions like a web application firewall
(WAF).
 Passive Information Gathering
Passive Information Gathering
+ Identifying domain names and domain ownership
information.
+ Discovering hidden/disallowed files and directories.
+ Identifying web server IP addresses & DNS records.
+ Identifying web technologies being used on target sites.
+ WAF detection.
+ Identifying subdomains.
+ Identify website content structure.
Active Information Gathering/Enumeration
Active Information Gathering
+ Downloading & analyzing website/web app source code.
+ Port scanning & service discovery.
+ Web server fingerprinting.
+ Web application scanning.
+ DNS Zone Transfers.
+ Subdomain enumeration via Brute-Force.
OWASP Web Security Testing Guide
Demo: Using The OWASP Web Security
Testing Guide
WHOIS
 WHOIS
● WHOIS is a query and response protocol that is used to query databases
that store the registered users or organizations of an internet resource
like a domain name or an IP address block.
● WHOIS lookups can be performed through the command line interface
via the whois client or through some third party web-based tools to
lookup the domain ownership details from different databases.
Demo: Identifying Ownership Information
With WHOIS
Website Fingerprinting With Netcraft
Passive DNS Enumeration
 DNS Enumeration
● Now that we have gathered some valuable information about our target,
we can start digging deeper into the data we found to build a
map/topology of the target site and it’s underlying infrastructure.
● A valuable resource for this information is the Domain Name System
(DNS).
● We can query DNS to identify the DNS records associated with a
particular domain or IP address.
 DNS
+ Domain Name System (DNS) is a protocol that is used to resolve domain
names/hostnames to IP addresses.
+ During the early days of the internet, users would have to remember the
IP addresses of the sites that they wanted to visit, DNS resolves this
issue by mapping domain names (easier to recall) to their respective IP
addresses.
+ A DNS server (nameserver) is like a telephone directory that contains
domain names and their corresponding IP addresses.
+ A plethora of public DNS servers have been set up by companies like
Cloudflare (1.1.1.1) and Google (8.8.8.8). These DNS servers contain the
records of almost all domains on the internet.
 DNS Records
+ A - Resolves a hostname or domain to an IPv4 address.
+ AAAA - Resolves a hostname or domain to an IPv6 address.
+ NS - Reference to the domains nameserver.
+ MX - Resolves a domain to a mail server.
+ CNAME - Used for domain aliases.
+ TXT - Text record.
+ HINFO - Host information.
+ SOA - Domain authority.
+ SRV - Service records.
+ PTR - Resolves an IP address to a hostname
Demo: Passive DNS Enumeration
Reviewing Webserver Metafiles
Practical Demo
Google Dorks
Practical Demo
Web App Technology Fingerprinting
Practical Demo
WAF Detection
Practical Demo
Copying A Website With HTTRack
Practical Demo
Website Screenshots With EyeWitness
Practical Demo
Passive Crawling & Spidering With Burp
Suite & OWASP ZAP
 Crawling
● Crawling is the process of navigating around the web application,
following links, submitting forms and logging in (where possible) with
the objective of mapping out and cataloging the web application and the
navigational paths within it.
● Crawling is typically passive as engagement with the target is done via
what is publicly accessible, we can utilize Burp Suite’s passive crawler to
help us map out the web application to better understand how it is setup
and how it works.
 Spidering
● Spidering is the process of automatically discovering new resources
(URLs) on a web application/site.
● It typically begins with a list of target URLs called seeds, after which the
Spider will visit the URLs and identified hyperlinks in the page and adds
them to the list of URLs to visit and repeats the process recursively.
● Spidering can be quite loud and as a result, it is typically considered to
be an active information gathering technique.
● We can utilize OWASP ZAP’s Spider to automate the process of
spidering a web application to map out the web application and learn
more about how the site is laid out and how it works.
Practical Demo
Web Server Fingerprinting
Practical Demo
DNS Zone Transfers
 DNS
+ Domain Name System (DNS) is a protocol that is used to resolve domain
names/hostnames to IP addresses.
+ During the early days of the internet, users would have to remember the
IP addresses of the sites that they wanted to visit, DNS resolves this
issue by mapping domain names (easier to recall) to their respective IP
addresses.
+ A DNS server (nameserver) is like a telephone directory that contains
domain names and their corresponding IP addresses.
+ A plethora of public DNS servers have been set up by companies like
Cloudflare (1.1.1.1) and Google (8.8.8.8). These DNS servers contain the
records of almost all domains on the internet.
 DNS Records
+ A - Resolves a hostname or domain to an IPv4 address.
+ AAAA - Resolves a hostname or domain to an IPv6 address.
+ NS - Reference to the domains nameserver.
+ MX - Resolves a domain to a mail server.
+ CNAME - Used for domain aliases.
+ TXT - Text record.
+ HINFO - Host information.
+ SOA - Domain authority.
+ SRV - Service records.
+ PTR - Resolves an IP address to a hostname
 DNS Interrogation

+ DNS interrogation is the process of enumerating DNS records for

a specific domain.
+ The objective of DNS interrogation is to probe a DNS server to
provide us with DNS records for a specific domain.
+ This process can provide us with important information like the IP
address of a domain, subdomains, mail server addresses etc.
 DNS Zone Transfer
+ In certain cases DNS server admins may want to copy or transfer
zone files from one DNS server to another. This process is known
as a zone transfer.
+ If misconfigured and left unsecured, this functionality can be
abused by attackers to copy the zone file from the primary DNS
server to another DNS server.
+ A DNS Zone transfer can provide penetration testers with a
holistic view of an organization's network layout.
+ Furthermore, in certain cases, internal network addresses may be
found on an organization's DNS servers.
Practical Demo
Subdomain Enumeration
Practical Demo
Web Server Scanning With Nikto
Practical Demo
File & Directory Brute-Force
Practical Demo
Automated Web Recon With
OWASP Amass
Practical Demo
Web Enumeration & Information
Gathering
Conclusion
 ● You will learn how to use the OWASP Web Security Testing
Guide as a methodology for web app pentesting engagements.
● You will be able to perform passive web app information
gathering.
● You will learn how to perform passive and active DNS
enumeration.
● You will learn how to detect web application firewalls (WAF).
● You will be able to utilize Google Dorks to find additional
Learning information on target websites/web applications.
● You will learn how to perform spidering and crawling to identify
Objectives: the content structure of websites.
● You will learn how to perform subdomain enumeration through
publicly available sources and through subdomain brute-force
attacks.
● You will learn how to perform file & directory brute-forcing to
discover hidden files and directories..
● You will learn how to utilize Automated recon frameworks like
OWASP Amass.
Thank You!