Intro to Web Technologies

INFR 4662U – Winter 2020

Garrett Hayes

Excerpts and concepts taken from the Web Application Hacker’s Handbook 2nd Edition License: Creative Commons
Stuttard & Pinto, Wiley Press
 2

About Me
MSc Computer Science Offensive Security Certified
@UOIT Professional (OSCP)

Blockchain Security and

Garrett Hayes
Bug Bounty Researcher
!@UOIT
 3

The HTTP Protocol

 4

Protocol: HTTP
§ Stateless and plaintext protocol

§ Based on client requests and server

responses
§ Headers are followed by a request
or response body

§ HTTP requests must use specific

request methods

§ Data is passed via key=value pairs

§ Responses use a status code

 5

URLs vs. Restful URLs

§ URLs indicate a resource requested by the client

from the target server, in the form:

protocol://hostname[:port]/[path/]file[?param=value]

https://example.com:8080/user.php?id=1&name=me

§ URLs can also be relative, implying the current

domain is to be used:

/user.php?id=2&name=you

§ Modern URLs may follow the Representational

State Transfer (REST) style format, such as:

https://example.com:8080/user/id/2/name/you
 6

HTTP Request Components

§ All basic HTTP requests include the same core

components:

1. HTTP Verb
The method used to communicate with
the server. For example: GET, POST,
OPTIONS, TRACE, etc..

2. The Requested URL

Excludes the domain and protocol used.
May also contain GET parameters/data.

3. HTTP Version Number

1.0, 1.1 and 2.0 are all valid.
 7

HTTP Request Components

§ All basic HTTP requests include the same core

components:

4. The hostname of the server

The full hostname of the server being
contacted, including any non-standard
port numbers (80 is implied).

5. User Agent String

A (semi) unique string identifying the
browser used to contact the server.

6. Referrer (Optional)
The URL of the originating server (i.e. the
page you were on when you clicked a link).
 8

HTTP Request Components

§ All basic HTTP requests include the same core

components:

7. Cookie(s)
One or more (hopefully) random values
used to uniquely identify a client or
store session data.

8. Other Headers
A variety of other components can be
added, including custom headers used
for authentication, authorization, or to
enforce client-side security mechanisms.

See: https://mzl.la/2QWIuND
 9

Example: A Simple HTTP Request

GET / HTTP/1.1
Host: uoit.ca
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:71.0)
Gecko/20100101 Firefox/71.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-CA,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
DNT: 1
Connection: close
Upgrade-Insecure-Requests: 1
 10

Example: A Complex HTTP Request

POST /x/RemoteUls.ashx?waccluster=US7 HTTP/1.1
Host: excel.officeapps.live.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:71.0) Gecko/20100101 Firefox/71.0
Accept: */*
Accept-Language: en-CA,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: https://excel.officeapps.live.com/
X-Requested-With: XMLHttpRequest
[snip]
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Content-Length: 225
Origin: https://excel.officeapps.live.com
DNT: 1
Connection: close
Cookie: [snip]

{"T":1578493198615,"L":[{"G":595947916,"T":3071,"M":"Aria Transport Stats:

{\"processed\":12,\"sent\":16,\"dropped\":0,\"rejected\":0,\"retried\":0}","C":383,"D":50}],"S":"977dc2f5-4cd5-48ba-8c0f-
1088c2eaf57d","I":1826,"V":1}
 11

HTTP Response Components

§ All basic HTTP responses include the same core

components:

1. HTTP Version Number

1.0, 1.1 and 2.0 are all valid.

2. HTTP Status Code

A numerical representation of the result of
the request. For example: 200, 301, 404,
etc..
3. Textual HTTP Status Code
A textual representation of the result of the
request. For example: 200 OK.
 12

HTTP Response Components

§ All basic HTTP responses include the same core

components:

4. Server Header
An optional, (semi) unique string identifying
the server software used to render the
response.

5. Other Headers
A variety of other headers may be added to
a response, including Set-Cookie, Pragma,
security settings, and other functionality we
will see later.
 13

HTTP Response Components

§ All basic HTTP responses include the same core

components:

6. Content Type Header

Indicates the type of content to follow in the
body of the response (e.g. HTML, JSON,
XML, etc.).

7. Content Body
The actual content of the response. For
example, the web page’s HTML, an API JSON
response, etc..

8. Content Length Header

The length of the content body in bytes.
 14

Example: A Simple HTTP Response

HTTP/1.1 200 OK
Date: Wed, 01 Jan 2020 14:21:30 GMT
Server: Apache/2.4.18 (Ubuntu)
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Length: 1638
Connection: close
Content-Type: text/html

<!doctype html>
<html>
<head>
<title>ontariotechu.ca</title>
<META HTTP-EQUIV="Refresh" CONTENT="1;URL=https://ontariotechu.ca/">
</head>
[snip]
 15

HTTP Method: GET

§ The purpose of this method is to
retrieve resources from a remote
server

§ This method can also pass request

data via a URL query string

§ Examples:

GET / HTTP/1.0
GET /index.php HTTP/2.0
GET /blog.php?user=bob HTTP/1.1
 16

HTTP Method: POST

§ The purpose of this method is to
perform actions on a remote server

§ This method can pass request data

via a URL query string or in the body
of the request (often the latter)

§ Example:

POST /cp/home/login HTTP/1.1

Host: portal.mycampus.ca
pass=password&user=100123456
 17

HTTP Methods: [Other]

§ HTTP does support additional request
methods that indicate other actions:

§ HEAD receives the response of a GET

request without the response body

§ PUT replaces a target’s resource with

a request’s resource

§ DELETE removes the target resource

§ PATCH modifies a resources on the

server without first deleting it
 18

HTTP Methods: [Other]

§ HTTP does support additional request
methods to indicate other actions:

§ TRACE performs a message loopback

and test(s) on the way to the target
resource

§ OPTIONS requests which methods

are supported by the server

§ CONNECT opens a tunnel to the

remote server (often involving a
proxy)
 19

HTTP Status Codes

§ Each HTTP response includes a status
code to indicate the state of the executed
request:

§ 1xx Informational Response

§ 2xx Success
§ 3xx Redirection
§ 4xx Client Error
§ 5xx Server Error
 20

HTTP Status Codes

§ Although dozens of codes exist, only a few
are used regularly on the web:

§ 200 OK
§ 301 Moved Permanently
§ 302 Found
§ 304 Not Modified
§ 400 Bad Request
§ 401 Unauthorized
§ 403 Forbidden
§ 404 Not Found
§ 500 Internal Service Error
 21

HTTP Cookies
§ Cookies are used to exchange persistent data
between a client and server

§ A server may provide data to a client using a

cookie, of which is retransmitted to the server
during the following request

Cookies are set using a response header:

Set-Cookie: username=admin; authed=true

§ This mechanism is often used to

authenticate clients, preventing them
from having to log in during every page
request
 22

HTTP Cookies
§ Once a cookie is set by the server, all
subsequent client requests automatically
include the cookie value in the request
header:

Cookie: username=admin&authed=true

§ Additional cookie attributes can also be set,

including:
§ An expiry date
§ The domain and path for which the
cookie is valid
§ The secure flag, indicating the cookie
should only be sent over HTTPS
§ The HttpOnly flag, preventing the cookie
from being read directly via JavaScript
 23

HTTP Authentication
§ Three mechanisms are built into the HTTP protocol
to authenticate clients, all of which can also occur
over HTTPS!

1. Basic Authentication
User credentials are base64 encoded and sent to
the server for processing

2. NTLM Authentication
The Windows NTLM challenge-response mechanism
is used to verify credentials without sending them

3. Digest Authentication
A MD5 challenge-response mechanism is used to
verify credentials without sending them
 24

S e r v e r- s i d e Te c h n o l o g i e s
 25

Static vs. Dynamic Content

§ Many web applications take input from users and
generate tailored content when rendering a page

§ In the past, all pages were static (i.e. coded in

HTML with changes pushed to all users)

§ Input can be provided to backend systems via

GET, POST, cookies, or RESTful URL requests

§ Backend web technologies like ASP.Net, PHP, Java,

etc. are leveraged to render content into HTML on
the fly, allowing custom content to be pushed to
visitors

§ JavaScript is often used to render client-side

content after receiving data from a backend
 26

Backend Technology: PHP

§ PHP is a heavily extended scripting framework
used to deploy dynamic applications on the web

§ Originally a functional language, but

recently evolved to support OOP as well

§ Web applications can be coded manually using

PHP; however, many developers choose to
leverage existing PHP frameworks that wrap and
automate common functionality

§ Some common frameworks include:

CakePHP, PHPBB, CodeIgniter, and Zend

§ Most web applications run on PHP, unfortunately

(more on this later)
 27

Backend Technology: Java

§ Java is a cross-platform language that
compiles Java source code into byte code
§ This bytecode is then interpreted on the
fly upon execution using the Java
Runtime Engine (JRE)
§ The JRE runs across a variety of
platforms, including Windows, Linux,
MacOS, Android, etc.

§ Java code that receives and responds with

HTTP requests is called a servlet

§ The underlying platform allowing one or more

servlets to run is called a web container
§ Examples: Tomcat, Jboss, and WebLogic
 28

Backend Technology: ASP.Net

§ ASP.Net is a Java-like event-driven web

application framework that uses Microsoft’s
existing .Net Framework

§ This means that any .Net language can

run on the web (e.g. C#, VB, etc.)

§ Code runs in an execution VM called the

Common Language Runtime (CLR)
§ Powerful APIs are provided to ease the
implementation of business logic

§ Common web security defenses are built into

the framework, automatically preventing
attacks like XSS, CSFR, etc.
 29

Backend Technology: Ruby

§ Ruby on Rails is a Model-View-Controller style
framework that leans heavily on structured
and logic-siloed code

§ Rails is capable of significantly simplifying the

structuring and modification of backend
database data using auto-generated models
and views

§ This functionality significantly eases

development of data-heavy applications
like APIs

§ Rails is particularly powerful when building

complex web applications that fulfill specific
business processes
 30

Data Structures
 31

Data Transmission: XML

§ The Extensible Markup Language (XML) was
designed to support the encoding and parsing of
complex data

§ A content’s data is separated from its markup,

just like any other markup language

§ Markup is represented by tags like <teacher>

or <student>

§ Tags can also include attributes like:

<teacher faculty=“fbit”>

§ Tags and attributes are entirely user-defined

and can be infinitely nested, allowing for
extreme flexibility
 32

Data Transmission: JSON

§ The JavaScript Object Notation (JSON) standard
facilitates the transmission of lightweight, text-
based data between applications

§ Data is commonly requested from a

backend using AJAX or WebSockets and is
used to update a web frontend

§ JSON data is simply a collection of objects &

arrays that are structured to aid machine-to-
machine data transmission

§ Example:
{
“firstName”: “john”,
“lastName”: “doe”,
}
 33

K e y C l i e n t- s i d e Te c h n o l o g i e s
 34

Basic Frontend Technologies

§ Cascading Style Sheets (CSS) are used to modify the

presentation of a document written in HTML
§ Examples: color text, add a shadow, etc.

§ JavaScript allows a browser to render, modify, and

fetch content in an asynchronous manner

§ This improves an application’s performance

since content can be rendered as it’s received

§ Content can be dynamically updated based on

a user’s actions

§ VBScript is an alternate version of JavaScript that

runs on Microsoft browsers
§ It’s hardly ever used but can be used to
bypass some security controls
 35

Frontend Technology: Forms

§ HTML forms provide input from users by POSTing

collected data to an application’s backend system
(a PHP page, for example)

§ Form input fields collect data from a user:

<input type=“text” name=“firstName”>

§ A form action dictates where the form’s data

will be POSTed:

<form action=“/submit.php” method=“post”>

§ Submitted forms generate a POST request whose

body includes the form data
 36

Example: A Sample HTTP Form Submission

The HTML form used by the client’s browser:

<form action=”/login.php” method=”post”>

Username: <input type=”text” name=”username”><br/>
Password: <input type=”password” name=”password”><br/>
<input type=”submit” name=”submit” value=”Login”>
</form>

The POST request send by the client’s browser upon submission:

POST /login.php HTTP/1.1 Host: example.com

Content-Type: application/x-www-form-urlencoded
Content-Length: 26
username=joe&password=blow
 37

Frontend Technology: DOM

§ The Document Object Model conceptually

represents a HTML document using a series of APIs,
allowing the original document to be
programmatically parsed, and updated using
technologies like JavaScript

§ Clients (i.e. browsers) can access and update

elements in a HTML document using the
element’s id or class

§ The DOM also provides access to cookie data,

as well as control over the current window,
including its location (i.e. URL)

§ Responsive applications build using AJAX lean

heavily on the DOM to provide dynamic
content to users without reloading pages
 38

Frontend Technology: AJAX

§ Asynchronous JavaScript and XML (AJAX) is a

collection of programming techniques used by
browsers to synchronously or asynchronously
manipulate the DOM and/or query a web
application’s backend system

§ Instead of processing user actions (like logins, for

example) through POST requests requiring a page
reload, asynchronous requests are sent in the
background and the DOM is updated to reflect any
changes

§ Backend requests are sent using a XMLHttpRequest

§ Requests need not use XML – they often use

JSON in modern web applications
 39

Example: A Sample HTTP AJAX Request

An XHR (AJAX) request sent from our cloud panel when deploying an instance:

GET /horizon/api/keystone/user-session/ HTTP/1.1

Host: cloud.hrl.uoit.ca
[snip]
X-Requested-With: XMLHttpRequest
Content-Type: application/json;charset=utf-8

The JSON response sent by the backend web application:

{"username": ”admin", "project_name": "INFR-4662-Instructors", "user_domain_id":

"default", "roles": [{"name": "user"}], "services_region": "RegionOne",
"user_domain_name": "Default", "enabled": true, "domain_name": null, [snip]}
 40

Frontend Security: SOP

§ The same-origin (SOP) policy is enforced by

browsers to ensure content between different web
applications remains isolated

§ Web pages can send & receive data from the

same application/domain only

§ A page can load a remote script from any domain to

be rendered on the local DOM only

§ A page on one domain cannot read/modify the

DOM of any other domain’s page loaded in a
different tab or window

§ A page on one domain can send a request to a page

on another domain, but cannot process the data
returned from the request
 41

Encoding
 42

URL Encoding Scheme

§ Encoding is used to safely represent bad, dangerous, or
unsupported characters during the transmission of data
§ Input data is encoded, sent, then decoded by the
receiver
§ Note that Encoding != encryption

§ URLs can only contain standard US-ASCII characters in the

range of 0x20 – 0x7e
§ Unsupported characters (like other language characters
such as î or ø) must be encoded

§ URL encoded characters are prefaced by % and followed by

the character’s ASCI hex code

§ For example, a space is %20 and a new line is %0a

§ The special character + also represents a space but doesn’t

require encoding
 43

Unicode Encoding Scheme

§ Unicode is character encoding standard that includes all
standard characters from all languages, including emojis
and other small images !

§ Unicode characters are typically 16-bit and are prefaced by

%u and followed by the 4+ digit codepoint

§ For example, is represented as %u2705

§ Unicode characters not used in a URL can also be

represented as:

§ ! à U+1F600 or even à :grinning:

§ URLs including Unicode often use Punycode to support

internationalized domain names

§ These domains begin with xn--

 44

HTML Encoding Scheme

§ HTML encoding allows HTML-specific characters to be
represented and then rendered into an HTML
document

§ HTML encoded characters will render and look

like HTML, but they cannot actually affect the
content of the page itself

§ Some examples of HTML encoded include:

§ &amp; renders to: &

§ &quot; renders to: “

§ HTML encoding using ASCII is also possible:

§ &#34 renders to: “

 45

Base64 Encoding Scheme

§ When binary data needs to be transmitted over a text-
based channel (think in a URL, form, etc.), it needs to
be encoded using a scheme like base64

§ Base64 simply allows the representation of binary data

in text form using the following characters:

ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopq
rstuvwxyz0123456789+/

§ Data is processed into blocks of 3 bytes, which are

then split into four 6-bit pieces, which are mapped to
the character set above – padding is provided using =

§ It’s also possible to use standard hex encoding!

 46

Let’s break!
S e e Yo u N e x t T i m e