Database Models

Vu Tuyet Trinh
[email protected] Department of Information Systems, Faculty of Information Technology Hanoi University of Technology

Data Models
A data model is a plan for building a database* A database
a set of concepts used to describe the database structure
data types Constraints

Some existing database models

Hierarchical model Network model Relational model Object-Oriented model

*http://www.computerworld.com/databasetopics/data/story/0,10801,80205,00.html

A Brief History
Hierarchical model
IMS, System 2k, ...

Relationa l model

DB2, ORACLEExtended Relational 10i, SQL Server ... Model

System R(81), DB2, ORACLE, SQL Server, Sybase, ...

XML

???
1990 1995 2000 2005 2010

1965

1970

1975

1980

1985

Entity-Relationship Model
Network model
DMS(65), CODASYL (71), IDMS, IDS

O2, ORION, IRIS, ...

IRDS(87) ,CDD+, ...

Object-Oriented model

Semi-structure Model

Lore (97), ...

Some Data Models

Hierarchical model Network model Relational model Entity Relationship Model Object-Oriented model Semi-structure Model S i t t M d l

Network data model

Represented as oriented graph Concepts
Record
Fields Record type

Link
name type: 1-1, 1-n, n-1, recursive

Operation
Navigating: FIND, FIND member, FIND owner, FIND NEXT Function: GET

Example
teacher
teach

class
include

study

subject
notes

student

have

note

Discussion
Discussion about the advantages and g inconvenient of this model

Hierarchical model
Represented as tree
Parent/child relationshop Each node has only one parent node 1 DB = set of trees

Concepts
Record Link Operation: GET, GET UNIQUE, GET NEXT, GET NEXT WITHIN PARENT, ...

Example
class
include study

teacher
teach

studen
have

subject

subject
noted

subject

note

note

Discussion
Discussion about the advantages and g inconvenient of this model

Relational Model
Represented as table Based on set theory Concept
Attribute/Field/Column
Name Data type, Domain value

Relation: defined on a set of attributes Tuple Key: Operation: union, intersection, cartesian product, selection, projection, join, ...

Example: University Database

Takes Student St dent
Id 1108 3936 8507 8452 Name Robert Glen Norman Mary Suburb Kew Bundoora Bundoora Balwyn SID 1108 1108 1108 8507 8507 SNO 21 23 29 23 29

Enrol
SID 3936 1108 8507 Course 101 113 101

Course C
No 113 101 Name BCS MCS Dept CSCE CSCE

Subject j
No 21 23 29 18 Name Systems Database VB Algebra Dept CSCE CSCE CSCE Maths

Discussion
Discussion about the advantages and g inconvenient of this model

Entity-Relation
ER diagram Concepts
Entity: represents an object in real-world Set of entities Attribute: property of an entity Key: Relationship: Association between two (or more) entities. entities
o o

o o

May have attribute 1-1, 1-n, n-m, recursion

Set of relationship

Example
ID Name Credit

examine

Subject

program

ID Name Birthday Sex Address ID Name Year

student

include

Class
head Fac.

Discussion
Discussion about the advantages and g inconvenient of this model

Object-Oriented Model
Represented by class diagram p y g Concepts
Object: having an unique identity Attribute: a property of the object, Method : an operation on the object Class: represents for a set objects having the same set of attributes and methods.

V d
class student { string ID; string Name; date Birthday; boolean Sex; string Address; string Class; string getName(); string getBirthday(); string getAddress(); string getClass(); void setAddress(string DC_moi); void setClass(string class);

Discussion
Discussion about the advantages and g inconvenient of this model

Database D i D t b Design Top-down approach

Database Design
Extended Entity Relationship y p
Top Down Conceptual/Abstract View

Functional Dependencies
Bottom Up Implementation View Synthesise relations

Process of Database Design

1: Analyse 1 A l

Requirements Collection

Modelling data (ex: using ER diagram)

2: Design 3: Implementation

Using a DBMS

Logical Data Schema (ex: using relational schema)

ER Model
well-suited to data modelling for use with databases
easy to represent and explain readily translated to relations.

Basic concepts
Attribute
represent a property/characteristic of an object in real world

Entity
defined as a set of attributes

Entity Set
Set of all entity instances of the same entity type

Relationship Set
Set of all relationship instances of the same relationship type

Entity & Attribute

EMPLOYEE

Name

Salary

Address

Age

Types of Attribute
Atomic
Ex: Database; 3083; Bundoora

Composite
Ex: Address ~ (Number, Street, and Suburb) 5+ Plenty Road+ Bundoora
EMPLOYEE

Name Age Address Number Street

Salary

Suburb

Types of Attribute
Multi-valued
Ex: Degrees of a person BCS, PhD
TEACHER

Name Degree ID Birthday

Derived
Ex: Derived a persons age from attribute Date of birth Age

Relationships
Degree of relationship g p
Unary Binary Ternary
M
SUBJECT
prerequisite i it

STUDENT

Cardinality
1-1 1-n n-m
RENTAL SHOP

Enrols in

SUBJECT

owns

VIDEO

1
BORROWER

Relationships can have attributes.

Result Name

Enrols in Title
N M

ID Address

STUDENT

SUBJECT

Code

Week entity
Definition
Existence depends on the existence of one or more other entities. Primary key partially or totally derived from the owner entity.

Example
EMPLOYEE is the owner entity, and DEPENDENT is the weak entity.
Occupation Employee No Dependent Name

EMPLOYEE

has

DEPENDENT

From ER schema to relational schema

STEP1: For each entity in the ER model, create a relation, i.e a table that includes all the simple attributes
EMPLOYEE

Name

EmployeeNo

Address

Age

Employee(EmployeeNo, Name, Address, Age)

STEP2: For each weak entity in the ER model, create a relation which includes all the simple attributes. The primary key of the relation is the combination of the primary key/s of the owner and the the key of the weak entity itself.
Employee No Occupation Dependent Name

EMPLOYEE

has

DEPENDENT

DEPENDENT(EmployeeNo, Dependent Name, Occupation)

STEP3: For each binary 1 TO 1 Relationship identify the two relations that correspond to the entities participating in the relationship. Choose one of the Relation and include as foreign key the primary key of the the other relation.
Dept# Deptname Manstaff# Name Address DEPARTMENT
MANAGED BY

MANAGER

DEPARTMENT (Dept#, Dept Name, ManStaff#)

STEP4: For each binary 1 TO N Relationships identify the relations that represents the participating entity at the N (i.e many) side of the relationship. Include as foreign key in the relation that holds the N side, the primary key of the other entity (that hold the 1 side)
E# Ename Dept# Deptname

EMPLOYEE

WORKS FOR

DEPARTMENT

EMPLOYEE (E#, EName, Dept#)

STEP5: For each binary M:N Relationships create a new relation to represent the relationship. The primary key of the new relation is the combination of the primary keys of the two connected entities.
E# Ename P# Ptitle

EMPLOYEE

WORKS ON

PROJECT

WORKS_ON (E# , P#)

STEP6: For each multivalued attribute, create a new relation that includes the multivalued attribute and the primary key of the entity where the multivalued attribute is attached.
E# Ename

Degree

EMPLOYEE

EMPLOYEE (E#, Ename) EDEGREE(E#, Degree)

STEP7: For each n-ary ( > 2 ) relationships create a new relation to represent the relationship. The primary key of the new relation is the combination of the primary keys of the participating entities that hold the N side.
Sup# Supname SQuantity SDate P# Ptitle

SUPPLIER M

SUPPLY

PROJECT N

N PART

Part# Part Name

SUPPLY (Sup# , P#, Part# , SQuanity, SDate)

Exercises

Database Design Bottom-up approach

Vu Tuyet Trinh
[email protected] Department of Information Systems, Faculty of Information Technology Hanoi University of Technology

Database Design
Extended Entity Relationship
Top Down Conceptual/Abstract View

Functional Dependencies
Bottom Up Synthesise relations List all attributes Consider the relationships between them Decompose attributes into tables in order to eliminate the redundancy.

1. 2. 3.

Using functional dependencies toSynthesise relations

studno studno studno familyname studno givenname studno hons studno tutor studno slot studno year studno, courseno studno, courseno courseno courseno courseno lecturer lecturer lecturer roomno roomno roomno year year yeartutor yeartutor STUDENT (studno,givenname,familyname,hons,tutor,slot,year)

labmark exammark courseno subject equip

ENROL(studno,courseno,labmark,exammark)

COURSE(courseno,subject,equip)

lecturer roomno appraiser lecturer roomno appraiser year yeartutor year yeartutor

STAFF(lecturer,roomno,appraiser) hons hons YEAR(year,yeartutor) faculty hons

SCHOOL(hons,faculty)

stud no s1 s1 s1 s2 s2 s3 s4 s5 s6 null null s7

name jones jones jones brown brown smith blogg jones peters null null patel

tutor bush bush wibby kahn kahn goble goble zobel kahn capon null null

roomn o 2.26 2.26 2.26 IT206 IT206 2.82 2.82 2.34 A17 A14 null null

course no cs250 cs260 cs270 cs250 cs270 cs270 cs280 cs250 cs250 null cs290 null

labmark 65 80 47 67 65 49 50 0 2 null null null

subject prog graphics elecs prog elecs comms design prog prog null specs null

F studno name, tutor tutor roomno roomno tutor courseno subject studno, courseno labmark

F+ studno, courseno name partial studno roomno transitive

Process of Normalization
Represent all user views (e.g forms, reports etc) as a collection of relations Normalize these relations Combine relations that have exactly the same primary key/s.

First Normal Form (1NF)

Definition there are no repeating groups. a unique key has been identified for each relation all attributes are functionally dependent on all or part of the key. Example
STUDENT_DETAILS (studno, name, tutor, roomno, {courseno, labmark, subject}) studno name, tutor courseno subject tutor roomno, roomno tutor studno, courseno labmark STUDENT (studno, name, tutor, roomno) studno name, tutor tutor roomno, roomno tutor

ENROL (studno, courseno, subject, labmark) courseno subject studno, courseno labmark

Second Normal Form (2NF)

Definition
the relation is in 1 NF all non-key attributes are fully functionally dependent on the entire key partial dependency has been removed

Example
ENROL (studno, courseno, subject, labmark) courseno subject studno, courseno labmark ENROL (studno, courseno, labmark) studno, courseno labmark COURSE (courseno, subject) courseno subject

Third Normal Form (3NF)

Definition
the relation is in 2NF all transitive dependencies have been removed. Transitive dependency: non-key attribute dependent on another non-key attribute. STUDENT (studno, name, tutor, roomno) studno name, tutor tutor roomno roomno tutor STUDENT (studno, name, tutor) studno name, tutor TUTOR (tutor, roomno) tutor roomno roomno tutor

Example
STUDENT (studno, name, tutor) studno name, tutor TUTOR (tutor, roomno) tutor roomno roomno tutor ENROL (studno, courseno, labmark) studno, courseno labmark COURSE (courseno, subject) courseno subject

Boye-Codd Normal Form (BCNF)

Definition
the relation is in 3NF any remaining anomalies that result form functional dependencies have been removed.

More Normal Forms

Fourth Normal Form (4NF)
the relation is in BCNF any multivalued dependencies have been removed.

Fifth Normal Form (5NF)

the relation is in 4NF any remaining anomalies that result form join dependencies have been removed.

Remarks
only in rare situations that a relation in 3NF is not in 4NF or 5NF. most relations that are in 3NF are also in BCNF.

Lossless or Non-additive Join

Given R ~ a relational scheme, F ~ a set of functional dependencies on R. Decomposition R = (R1, R2) The decomposition of R is non-additive if at least one of the following functional dependencies are in F+ R1 R2 R1 R1 R2 R2 The decomposition of R is non-additive if for every state r of R that satisfies F <R1> (r)* ...* <Rm> (r) = r

Lossless or Non-additive Join

ENROL (studno, courseno, subject, labmark) courseno subject studno, courseno labmark ENROL (studno, courseno, labmark) studno, courseno labmark COURSE (courseno, subject) courseno subject

ENROL COURSE = courseno courseno subject (courseno, subject) = COURSE

Lossless or Non-additive Join

STUDENT studno name s1 jones s2 brown s3 smith s4 bloggs s5 jones s6 peters
STUDENT1 studno name s1 jones s2 brown s3 smith s4 bloggs s5 jones s6 peters tutor bush kahn goble goble zobel kahn

tutor bush kahn goble goble zobel kahn

roomno 2.26 IT206 2.82 2.82 2.34 IT206

studno name studno tutor tutor roomno roomno tutor

studno name studno tutor

TUTOR tutor kahn bush goble zobel

roomno IT206 2.26 2.82 2.34

tutor roomno roomno tutor

STUDENT1 * TUTORS = STUDENT

Dependency Preservation
The union of dependencies that hold on the individual relations in decomposition D must be equivalent to F. Given F on R, F(Ri) where Ri R is the set of dependencies X Y in F+ such that the attributes in X Y are all contained in Ri Decomposition D = {R1, R2, ..., Rm} of R is dependency preserving w.r.t. F if (F(R1)) .... F(Rm)))+ = F+ Given the restriction of functional dependencies to a relation is the fds that involve attributes of that relation Fi for Ri n n U Fi F possible, but... (U Fi)+ = F+ i=1 i =1

Dependency Preservation
STUDENT studno name s1 jones s2 brown s3 smith s4 bloggs s5 jones s6 peters tutor bush kahn goble goble zobel kahn roomno 2.26 IT206 2.82 2.82 2.34 IT206 appraiser capon watson capon capon watson watson

studno name studno tutor tutor roomno tutor appraiser roomno tutor roomno appraiser studno appraiser studno roomno studno appraiser studno roomno

STUDENT' studno name s1 jones s2 brown s3 smith s4 bloggs s5 jones s6 peters

tutor bush kahn goble goble zobel kahn

TUTOR studno s1 s2 s3 s4 s5 s6

roomno 2.26 IT206 2.82 2.82 2.34 IT206

appraiser capon watson capon capon watson watson

studno name studno tutor

STUDENT

* TUTOR = STUDENT

Designing a relational schema

Build a relational database without redundancy normalisation without loss of information or gain of data lossless join decomposition without losing dependency integrity dependency preservation

Exercises

Database Architecture
Vu Tuyet Trinh
[email protected] Department of Information Systems, Faculty of Information Technology Hanoi University of Technology

Architecture
Application Query Processing

DBMS

Transaction Management

Storage Management

Data

Data

QP Trans. Mgt

Storage Management
Responsible for storing and accessing data data. Buffer manager
responsible for partitioning the available main memory into buffers
Storage Management Buffer Mgt File Mgt

Storage Mgtt

Trans. Mgt

File Management
responsible for interacting with file system

Metadata & Data dictionary

Data & index

Stored Data
Data: the contents of database itself
Metadata:
the database schema that describes the structure of and constraints on the database

Statistics
information gathered and stored by the DBMS about data properties such as the size of, and values in, various relations or other components of the database

Indexes
data structures that support efficient access to the data.

QP Trans. Mgt

Query Processing
Composed of
Parser Optimizer Execution Engine
Query Processing

Stoarge Mgt

Parser Optimizer Exec. Engine

Parser
responsible for verifying query syntax and semantic and translating query into query plan Query plan is a sequence of operations implementing relational algebra operation, to be performed on the data.

Storage Mgt

Metadata & Data dictionary

Data & index

QP Trans. Mgt

Query Processing (2)

Optimizer
responsible for performing query plan transformation for the best evaluation Query Processing

Stoarge Mgt

Parser Optimizer Exec. Engine

Execution engine
Responsible for executing each of steps in the chosen query plan. interacts with most of the other components of the DBMS

Storage Mgt

Metadata & Data dictionary

Data & index

Transaction Management
Accept transaction commands from an application which tell the transaction manager when transactions begin and end, as well as information about the expectations of the application (some may not wish to require atomicity, for example). The transaction processor performs the following tasks:
Logging Recovery Concurrency control

Transaction Management
Logging
every change in the database is logged separately on disk disk. The log manager follows one of several policies designed to assure that no matter when a system failure or crash occurs.

Recovery manager
able to examine the log changes and restore the database to some consistent state.

Concurency control
transactions must appear to execute in isolation. pp the schedeler (concurrency control manager) must assure that the individual actions of multiple transactions are executed in such an order that the net effect in the same as if the transactions had in fact executed in their entirely, once-at-atime. A typical scheduler does its work by maintaining locks on certain pieces of the database.

3-tier Schema Model (ANSI-SPARC Architecture)

EXTERNAL VIEW1

END USERS

EXTERNAL LEVEL

.. ..

EXTERNAL VIEWn

External/Conceptual Mapping

CONCEPTUAL LEVEL
Conceptual Internal Mapping

CONCEPTUAL SCHEMA

INTERNAL LEVEL

INTERNAL SCHEMA

STORED DATABASE

Internal schema
describes the physical storage structure of the database.

Conceptual schema
describes the structure of the whole database for a community of users.

External schemas or user views.

describes the part of the database that a particular user group is interested in and hides the rest of the database from that user group.

DBMS Utilities
Loading
To load existing data files into the database

Backup & recovery

To create a backup copy of the database To recovery an old state of DB from a backup file

File reorganization
To reorganize a database file in order to improve performance

Report generation
To T generate reports based on the information from the t t b d th i f ti f th database

Performance monitoring
To provide the DBA with statistical data about the DB usage.

Concurrency
Vu Tuyet Trinh
[email protected] Department of Information Systems, Faculty of Information Technology Hanoi University of Technology

Example
500USD Account A read(A) If A > 500 then B: B 500 B:=B+500 A:=A-500 Crash Account B

What happen pp ???

Transaction
A sequence of read and write operations on data items that logically functions as one unit of work
Cho php m bo tnh nht qun v tnh ng n ca d liu

ACID Properties
Atomicity Consistency y Isolation Durability
Concurrency Control Recovery

Automicity
guarantee that either all of the tasks of a transaction are performed or none of them are Example T: Read(A,t1); If t1 > 500 { Read(B,t2); t2:=t2+500; Write(B,t2); t1:=t1-500; Write(A,t1); }

crash

Consistency
ensures that the DB remains in a consistent state before the start of the transaction and after the transaction is over Example A+B = C
T: Read(A,t1); If t1 > 500 { Read(B,t2); t2:=t2+500; Write(B,t2); t1:=t1-500; Write(A,t1); }

A+B = C
5

Isolation
ability of the application to make operations in a transaction appear isolated f t ti i l t d from all other operations. ll th ti Example A= 5000, B= 3000 T: Read(A,t1); If t1 > 500 { Read(B,t2); t2:=t2+500; Write(B,t2); t1:=t1-500; Write(A,t1); }

T: A+B (= 5000+3500) (A+B = 4500+3500)

6

Durability
guarantee that once the user has been notified of success, th t the transaction will persist, and not be undone ti ill i t d tb d V d: A= 5000, B= 3000 T: Read(A,t1); If t1 > 500 { Read(B,t2); t2:=t2+500; Write(B,t2); t1:=t1-500; Write(A,t1); }

crash A= 4500, B=3500

Transaction States

Transaction Management Interfaces

Begin Trans Commit () Abort() Savepoint Save() Rollback (savepoint) (savepoint = 0 ==> Abort)

Concurrency Control
Objective:
ensures that database transactions are performed concurrently without the concurrency violating the data integrity guarantees that no effect of committed transactions is lost, and no effect of aborted (rolled back) transactions remains in the related database.

Example
T0: read(A); A := A -50; write(A); read(B); B := B + 50; write(B); T1: read(A); temp := A *0.1; A := A -temp; write(A); read(B); B := B + temp; write(B);

10

Scheduling

(1)

(2)

(3)

11

Serializability
A schedule of a set of transactions is a linear ordering of their actions
e.g. for the simultaneous deposits example:

R1(X) R2(X) W1(X) W2(X) A serial schedule is one in which all the steps of each transaction occur consecutively A serializable schedule is one which is equivalent to some serial schedule

Lock
Definition
a synchronization mechanism for enforcing limits on access to DB in concurrent way. one way of enforcing concurrency control policies

Lock types
Shared lock (LS) readable but can not write Exclusive lock (LX): c v ghi DL UN(D): unlock

LS LS LX true

LX false
13

Compatibility

false false

Example
T0: LX(A); read(A); A := A -50; write(A); LX(B); read(B); B := B + 50; ( ); write(B); UN(A); UN(B); T1: LX(A); read(A); temp := A *0.1; A := A -temp; write(A) LX(B); read(B); B:=B+temp; p; write(B); UN(A); UN(B);

14

Well-Formed, two-phased transaction

A transaction is well-formed if it acquires at least a shared lock on Q before reading Q or an exclusive lock on Q before writing Q and doesnt release the lock until the action is performed
Locks are also released by the end of the transaction

A transaction is two-phased if it never acquires a lock after unlocking one

i.e., there are two phases: a growing phase in which the transaction acquires locks, and a shrinking phase in which locks are released

2Phase Locking (2PL)

Phase 1
locks are acquired and no locks are released

Phase 2
locks are released and no locks are acquired

BOT

EOT

t Phase lock Phase unlock

Example
T1 Lock(A) Read(A) Lock(B) Read(B) B:=B+A Write(B) Unlock(A) Unlock(B) U l k(B) T2 Lock(B) Read(B) Lock(A) Read(A) Unlock(B) A:=A+B Write(A) W it (A) Unlock(A) T3 Lock(B) Read(B) B=B-50 Write(B) Unlock(B) Lock(A) Read(A) A A+50 A=A+50 Write(A) Unlock(A) T4 Lock(A) Read(A) Unlock(A) Lock(B) Read(B) Unlock(B) Pritn(A+B)

2PL Not 2PL

Deadlock
T0: LX(B); read(B); B := B +50; write(B); LX(A); read(A); A := A - 50; ( ); write(A); UN(A); UN(B); T1: LX(A); read(A); temp := A *0.1; A := A -temp; write(A) LX(B); read(B); B:=B+temp; p; write(B); UN(A); UN(B);

18

Resolving Deadlock
Detecting g
Recovery when deadlock happen
rollback

Used waiting-graph

Avoiding
Resource ordering g Timeout Wait-die Wound-wait

Waiting Graph
Graph
Node handling lock or waiting for lock Edge T U
U handle L(A) T wait to lock A T must wait until U unlock A

If there exists a cycle in the waiting graph

deadlok

Timeout
Set a limit time for each transaction If time-out do rollback

Exercises

Crash Recovery
Vu Tuyet Trinh
[email protected] Department of Information Systems, Faculty of Information Technology Hanoi University of Technology

Transaction
collection of action that preserve consistency p y

Consistent DB

Consistent DB

with assumption
IF THEN T starts with consistent state + T executes in isolation T leaves consistent state

How can constraints be violated?

Transaction bug DBMS bug Hardware failure
e.g., disk crash

Data sharing
e.g., T1 and T2 in parallel

Failures
Events Desired Undesired Expected Unexpected processor
CPU

memory
M

disk
D

Recovery
Maintaining the consistency of DB by ROLLBACK to the last consistency state. Ensuring 2 properties
Atomic Durability

Using LOG

Transaction Log
A sequence of log record keeping trace of actions executed by DBMS <start T>
Log the beginning of the transaction execution

<commit T>
transaction is already finished

<abort T>
Transaction is calcel

<T, X, v, w>
Transaction makes an update actio, before update X=v, after update x = w

Transaction Log
Handled in main memory and put to external y p memory (disk) when possible

A = 8 16 B = 8 16 Actions Log

Data Log g Disk

Memory

Checkpoint
Definition:
moment where intermediate results and a log record are saved to disk. being initiated at specified intervals

Objective
minimize the amount of time and effort wasted when restart the process can be restarted from the latest checkpoint rather than from the beginning. beginning

Log record <checkpoint> or <ckpt>

Undo-logging
Step 1 2 3 4 5 6 7 8 9 10 11 12 Action Read(A,t) t:=t*2 Write(A,t) Read(B,t) t:=t*2 Write(B,t) Flush log Output(A) Output(B) Flush log t 8 16 16 8 16 16 16 16 Mem A 8 8 16 16 16 16 16 16 Mem B Disk A 8 8 8 8 8 8 16 16 Disk B 8 8 8 8 8 8 8 16 <commit T> <T, B, 8> Mem Log <start T>

<T, A, 8>

8 8 16 16 16

Undo-Logging Rules
(1) For every action generate undo log record (containing old value) (2) Before X is modified on disk, log records pertaining to X must be on disk (write ahead logging: WAL) (3) Before commit is flushed to log, all writes of transaction must be reflected on disk

Undo Logging Recovery Rules

Let S is set of unfinished transactions
<start Ti> in log <commit Ti> or <abort Ti> is not in log

For each <Ti, X, v> in log

If Ti S then - Write(X, v) - Output(X)

For each Ti S
Write <abort Ti> to log

Undo-Logging & Checkpoint

<start T1> <T1, A, 5> <start T2> <T2, B, 10> <T2, C, 15> <T2, D, 20> <commit T1> <commit T2> <checkpoint> <start T3> <T3, E, 25> <T3, F, 30> scan <start T1> <T1, A, 5> <start T2> <T2, B, 10> <start ckpt (T1,T2)> <T2, C, 15> <start T3> <T1, D, 20> <commit T1> <T3, E, 25> <commit T2> <end ckpt> <T3, F, 30> scan

Redo-logging
Step Action 1 2 3 4 5 6 7 8 9 10 11 Read(A,t) t:=t*2 Write(A,t) Read(B,t) t:=t*2 Write(B,t) Write(B t) Flush log Output(A) Output(B) t 8 16 16 8 16 16 Mem A Mem B 8 8 16 16 16 16 Disk A 8 8 8 8 8 8 Disk B 8 8 8 8 8 8 Mem Log <start T>

<T, A, 16>

8 8 16

<T, B, <T B 16> <commit T>

16 16

16 16

16 16

16 16

8 16 <T, end>

Redo-logging Rules
(1) For every action, generate redo log record (containing new value) (2) Before X is modified on disk (DB),all log records for transaction that modified X (including commit) must be on disk (3) Flush log at commit (4) Write END record after DB updates flushed to disk

Redo-logging Recovery Rules

Let S = set of transactions with
<Ti, commit> in log no <Ti, end> in log

For each <Ti, X, v> in log, in forward order (earliest latest)

If Ti S then write(X, v) output(X) For each Ti S write <Ti, end>

Redo Logging & Checkpoint

<start T1> <T1, A, 5> <start T2> <commit T1> <T2, B, 10> <start ckpt (T2)> <T2, C, 15> <start T3> <T3, D 20> D, scan <start T1> <T1, A, 5> <start T2> <commit T1> <T2, B, 10> <start ckpt (T2)> <T2, C, 15> <start T3> <T3, D, 20> <end ckpt> <commit T2> <commit T3> scan

Discussion
Undo Logging
need to write to disk as soon transaction finishes Access disk

Redo Logging
need to keep all modified blocks in memory until commit Use memory

Undo/Redo Loggin
Step 1 2 3 4 5 6 7 8 9 10 11 Action Read(A,t) t:=t*2 Write(A,t) Read(B,t) t:=t*2 Write(B,t) Write(B t) Flush log Output(A) Output(B) t 8 16 16 8 16 16 16 16 Mem A 8 8 16 16 16 16 16 16 Mem B Disk A 8 8 8 8 8 8 16 16 Disk B 8 8 8 8 8 8 8 <commit T> 16 16 Mem Log <start T>

<T, A, 8, 16>

8 8 16 16

<T, B, 8, <T B 8 16>

Undo/Redo Logging Rules

Page X can be flushed before or after T commit g Log record flushed before corresponding updated page (WAL) Flush at commit (log only)

Undo/Redo Logging & Checkpoint

<start T1> <T1, A, 4, 5> <start T2> <commit T1> <T2, B, 9, 10> <start ckpt (T2)> <T2, C, 14, 15> <start < t t T3> <T3, D, 19, 20> <end ckpt> <commit T2> scan <start T1> <T1, A, 4, 5> <start T2> <commit T1> <start T3> <T2, B, 9, 10> <T3, E, 6, 7> <start ckpt (T2, T3)> <T2, C 14 15> C, 14, <T3, D, 19, 20> <end ckpt> <commit T2> scan

Undo/Redo Logging Recovery Rules

Backwards pass (end of log latest valid checkpoint start) Constructing set S of committed transactions undo actions of transactions not in S undo pending transactions follow undo chains for transactions in (checkpoint active list) S Forward pass (latest checkpoint start end of log) redo actions of S transactions

Exercises

Index Management
Vu Tuyet Trinh
[email protected] Department of Information Systems, Faculty of Information Technology Hanoi University of Technology

File Organization
Data storage in file
records, records blocks and access structures

Organisation Heap files: for full file scans or frequent updates

Data unordered Write new data at end

Index: if retrieved in key attributes

Need to store index

Sorted Files: if retrieved in sort order or want range

Need external sort or an index to keep sorted

Hashed Files: if selection on equality Collection of buckets with primary & overflow pages Hashing function over search key attributes

Heap File
Organization
Data unordered Write new data at end Data Page Header Page Data Page Data Page Data Page Pages with Free Space Data Page Data Page Full Pages

Need a full scan file for Search, Insert, Update, Delete operations

Indexing technique
Search key y
Any subset of the fields of a relation can be the search key Search key may not be the key in relation

Index
a collection of k data entries supports efficient retrieval with a given key value k.

Classes of Indexes
Primary vs. secondary: primary has primary key Clustered vs. unclustered: order of records and index approximately same
Alternative 1 implies clustered, but not vice-versa A file can be clustered on at most one search key

Dense vs. Sparse: dense has index entry per data value; sparse may skip some
Alternative 1 always leads to dense index Every sparse index is clustered! Sparse indexes are smaller; however, some useful optimizations are based on dense indexes

Clustered vs. Unclustered Index

Suppose Index Alternative (2) used, records are stored used in Heap file
Perhaps initially sort data file, leave some gaps Inserts may require overflow pages
Index entries direct search for data entries

CLUSTERED

UNCLUSTERED

Data entries

Data entries (Index File) (Data file)

Data Records

Data Records

B Tree Technique
Organization
Root is either a leaf node or a node having at least 2 childrent nodes Except root and leaf node, every node have [m/2] n m con. Length of every path from the root to a leaf is equal

Index Entries (Direct search)

Data Entries ("Sequence set")

Example
Search begins at root, and key comparisons direct it to a leaf.
Root
13 17 24 30

2*

3*

5*

7*

14* 16*

19* 20* 22*

24* 27* 29*

33* 34* 38* 39*

Search for 5*, 15*,

Operations
Inserting g Deleting Updating Searching
Discussion about -How to do -Complexity

B+ Tree Summary
B+ tree and other indices ideal for range searches, good for equality searches.
Inserts/deletes leave tree height-balanced; logF N cost. High fanout (F) means depth rarely more than 3 or 4. Almost always better than maintaining a sorted file. Typically, 67% occupancy on average. Note: Order (d) concept replaced by physical space criterion in practice ( at least half full) (at half-full ).
Records may be variable sized Index pages typically hold more entries than leaves

Hashing Technique
A familiar idea:
Requires good hash function (may depend on data) Distribute data across buckets Often multiple items in same bucket (buckets might overflow)

Types of hash tables:

Static Extendible (requires directory to buckets; can split) Linear (two levels, rotate through + split; bad with skew) Can be the basis of disk-based indices!
We wont get into detail because of time, but see text

Example
h(x) = x mod 4 1 2 4 3 Store hash

1 1

2 2

3 3

4 4

h(x) = x mod 4 10 12 6 Store hash

1 1

2 2 10 6

3 3

4 4 12

Operations
Inserting g Deleting Updating Searching
Discussion about -How to do -Complexity

Query Processing
Vu Tuyet Trinh
[email protected] Department of Information Systems, Faculty of Information Technology Hanoi University of Technology

Query Processing

Query plans & exec strategies Standard relational operators Query Optimization

Query Plans
Data-flow graph of g p relational algebra operators Typically: determined by optimizer
SELECT * FROM PressRel p, Clients C WHERE p.Symbol = c.Symbol AND c.Client = Atkins AND c.Symbol IN (SELECT CoSymbol FROM EastCoast)

Join J i
PressRel.Symbol = EastCoast.CoSymbol

Join
PressRel.Symbol = Clients.Symbol

Project
CoSymbol

Select
Client = Atkins

Scan
PressRel

Scan
Clients

Scan
EastCoast

Execution Strategy Issues

Granularity & y parallelism:
Pipelining vs. blocking Materialization
Join J i
PressRel.Symbol = EastCoast.CoSymbol

Join
PressRel.Symbol = Clients.Symbol

Project
CoSymbol

Select
Client = Atkins

Scan
PressRel

Scan
Clients

Scan
EastCoast

Iterator-Based Query Execution

Execution begins at root
open, next, close Propagate calls to children
May call multiple child nexts

Join J i
PressRel.Symbol = EastCoast.CoSymbol

Efficient scheduling & resource usage

Join
PressRel.Symbol = Clients.Symbol

Project
CoSymbol

Select
Client = Atkins

Scan
PressRel

Scan
Clients

Scan
EastCoast

Basic Principles
Many DB operations require reading tuples, tuple vs. previous tuples, or tuples vs. tuples in another table Techniques generally used:
Iteration: for/while loop comparing with all tuples on disk Index: if comparison of attribute thats indexed, look up matches in index & return those Sort: iteration against presorted data (interesting orders) Hash: build hash table of the tuple list, probe the hash table

Must be able to support larger-than-memory data

Basic Operators
One-pass operators:
Scan Select Project

Multi-pass operators:
Join
Various implementations Handling of larger-than-memory sources

Semi-join Aggregation, union, etc.

1-Pass Operators: Scanning a Table

Sequential scan: read through blocks of table Index scan: retrieve tuples in index order

Cost in page reads -- b(T) blocks, r(T) tuples

b(T) pages for sequential scan Up to r(T) for index scan if unclustered index Requires memory f one bl k R i for block

Relation S

Multi-pass Operators: Nested-Loops Join

Principle p
reading the external relation R & iterating over the internal relation S one-and-haft pass, nonblocking Matching
Tuple R

Variations

Tuple R p

Tuple S p

Tuple-based NLJ, blockbased NLJ, index-based NLJ

SOURCE R

SOURCE S

Two-Pass Algorithms
Sort-based
Need to do a multiway sort first (or have an index) Approximately linear in practice, 2 b(T) for table T

Hash-based
Store one relation in a hash table

(Sort-)Merge Join
Requires data sorted by join attributes
Merge and join sorted files reading sequentially a files, block at a time

Maintain two file pointers

While tuple at R < tuple at S, advance R (and vice versa) While tuples match, output all possible pairings

Preserves sorted order of outer relation Very efficient for presorted data Can be hybridized with NL Join for range j y g joins May require a sort before (adds cost + delay) Cost: b(R) + b(S) plus sort costs, if necessary In practice, approximately linear, 3 (b(R) + b(S))

Hash-Based Joins
Allows partial p p p pipelining of operations with g p equality comparisons Sort-based operations block, but allow range and inequality comparisons Hash joins usually done with static number of hash buckets
Generally have fairly long chains at each bucket What happens when memory is too small?

Hash Join (HJ)

Principle Process
using a hash table (HT) of R two-pass, blocking algorithm build the HT of R Matching M t hi
Hash Table S

build b ld

probe every tuple of S with corresponding tuples in the HT of R

probe

Tuple R hash(Tuple R) SOURCE R

Tuple S hash(Tuple S) SOURCE S

Overview of Query Optimization

Plan: Tree of R.A. ops, with choice of alg for each op.
Each operator typically implemented using a `pull interface: when an operator is `pulled for the next output tuples, it `pulls on its inputs and computes them.

Two main issues:

For a given query, what plans are considered?
Algorithm to search plan space for cheapest (estimated) plan.

How is the cost of a plan estimated?

Ideally: Want to find best plan. Practically: Avoid worst plans!

Query optimization [Ioa00]

Rewriter
Rewriting Stage (Declarative) Planning Stage (Procedural)

Algebraic Space Planner Method-Structure Space

Cost Model

Size-Distribution Estimator

Rewriting
Principle [Lev01]
reformulating a client query to use available views replacing views by their definition View def.:

client query

V V T U

Query optimization [Ioa00]

Rewriter
Rewriting Stage (Declarative) Planning Stage (Procedural)

Algebraic Space

Search strategy
Planner

Cost Model

Method-Structure Space

Size-Distribution Estimator

Search space

Cost estimation

Search Space
Algebraic space
operators execution orders that are to be considered by the Planner for each query sent to it

Method-structure space
implementation choices that exist for the execution of each specified ordered series of operators

Relational Algebra Equivalences

Allow us to choose different join orders and to `push selections and projections ahead of joins. Selections: (Cascade)

c1 ... cn ( R) c1 ( . . . cn ( R) )

c1 ( c 2 ( R) ) c 2 ( c1 ( R) )

Projections: a1 ( R) a1 . . . ( an ( R) ) Joins: R (S (R
Show that:

(Commute) (Cascade) (Associative) (Commute)

T) (R R) T) (T (S

S) R)

T S

S) (S R

More Equivalences
A projection commutes with a selection that only uses attributes retained by the projection projection. Selection between attributes of the two arguments of a crossproduct converts cross-product to a join. A selection on just attributes of R commutes with R S. (i.e., (R S) (R) S ) a Similarly, if a projection follows join R S, we can `push it by retaining only attributes of R (and S) that are needed for the join or are kept by the projection.

Enumeration of Alternative Plans

There are two main cases:
Single-relation plans Multiple-relation plans

For queries over a single relation, queries consist of a combination of selects, projects, and aggregate ops:
Each available access path (file scan / index) is considered, and the one with the least estimated cost is chosen. The different operations are essentially carried out together (e g if an index is (e.g., used for a selection, projection is done for each retrieved tuple, and the resulting tuples are pipelined into the aggregate computation).

Cost estimation
Cost factors
CPU cost + I/O cost + Communication cost Size of (intermediate) relations

For each plan considered, must estimate cost:

Must estimate cost of each operation in plan tree.
Depends on input cardinalities.

Must also estimate size of result for each operation in tree!

Use information about the input relations. For selections and joins, assume independence of predicates.

Search strategy
Objective
exploring the set of alternative execution plans and finding the cheapest one

Taxonomy
Polynomial vs. Combinatorial Heuristics vs. Systematic Deterministic vs. Randomized Transformative vs Constructive vs.

orthogonal but related

Queries Over Multiple Relations

Fundamental decision in System R: only left-deep join trees are considered.
As the number of joins increases, the number of alternative plans grows rapidly; we need to restrict the search space. Left-deep trees allow us to generate all fully pipelined plans. Intermediate results not written to temporary files. Not all left-deep trees are fully pipelined (e.g., SM join).

D C
A B C D

D C A B

Enumeration of Left-Deep Plans

Left-deep Left deep plans differ only in the order of relations the access method for relations, each relation, and the join method Enumerated using N passes (if N relations joined):
Pass 1: Find best 1-relation plan for each relation. Pass 2: Find best way to join result of each 1-relation plan (as outer) to another relation. (All 2-relation plans.) Pass N: Find best way to join result of a (N-1)-relation plan (as outer) to the N th relation. Nth relation (All N-relation plans ) N relation plans.)

For each subset of relations, retain only:

Cheapest plan overall, plus Cheapest plan for each interesting order of the tuples.

Enumeration of Plans (Contd.)

ORDER BY, GROUP BY, aggregates etc. handled as a final

step, using either an `interestingly ordered plan or an interestingly ordered addional sorting operator. An N-1 way plan is not combined with an additional relation unless there is a join condition between them, unless all predicates in WHERE have been used up.
i.e., avoid Cartesian products if possible.

In spite of pruning plan space this approach is still space, exponential in the # of tables.

Recent T d & Future R t Trends F t Directions

Vu Tuyet Trinh
[email protected] Department of Information Systems, Faculty of Information Technology Hanoi University of Technology

Trends Perspectives
Extending data model g
More extensible More flexible

Distribution
More distributive

Heterogeneity g y

Example of XML
Processing Instr. <?xml version="1.0" encoding="ISO-8859-1" ?> <dblp> Open-tag <mastersthesis mdate 2002 01 03 key="ms/Brown92"> mdate="2002-01-03" key ms/Brown92 > <author>Kurt P. Brown</author> <title>PRPL: A Database Workload Specification Language</title> <year>1992</year> Element <school>Univ. of Wisconsin-Madison</school> </mastersthesis> <article mdate="2002-01-03" key="tr/dec/SRC1997-018"> <editor>Paul R. McJones</editor> Attribute <title>The 1995 SQL Reunion</title> <journal>Digital System Research Center Report</journal> <volume>SRC1997-018</volume> <year>1997</year> Close-tag <ee>db/labs/dec/SRC1997-018.html</ee> <ee>http://www.mcjones.org/System_R/SQL_Reunion_95/</ee> </article> </dblp>

XML as a Data Model

XML information set includes 7 types of nodes:
Document (root) Element Attribute Processing instruction Text (content) Namespace: Comment

XML Data Model Visualized (and simplified!)

Root ?xml dblp article mdate key

root p-i i

attribute element l t
text

mastersthesis mdate
2002

key author title year school

1992 PRPL Kurt P. Univ.

editor title journal volume year ee ee

tr/dec/ The Digital Paul R. 1997 db/labs/dec SRC http://www.

2002

ms/Brown92

Document Type Definitions (DTDs)

a grammar defining XML structure t t define special significance for attributes:
IDs special attributes that are analogous to keys for elements IDREFs references to IDs IDREFS a nasty hack that represents a list of IDREFs Example DTD: <!ELEMENT dblp((mastersthesis | article)*)> <!ELEMENT mastersthesis(author,title,year,school,co mmitteemember*)> <!ATTLIST mastersthesis(mdate CDATA #REQUIRED key ID #REQUIRED advisor CDATA #IMPLIED> <!ELEMENT author(#PCDATA)> Example use of DTD in XML file: <?xml version="1.0" encoding="ISO-88591" ?> <!DOCTYPE dblp SYSTEM my.dtd"> <dblp>

XML Schema

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" > <xsd:element name=mastersthesis" type=ThesisType"/> XML syntax <xsd:complexType name=ThesisType"> < sd comple T pe name ThesisT pe"> Better way of defining <xsd:attribute name=mdate" type="xsd:date"/> keys using XPaths <xsd:attribute name=key" type="xsd:string"/> Type sub-classing <xsd:attribute name=advisor" type="xsd:string"/> more complex than in <xsd:sequence> a programming <xsd:element name=author" language type=xsd:string"/> <xsd:element name=title" type=xsd:string"/> Domains and built-in <xsd:element name= year name=year" data-types d t t type=xsd:integer"/> <xsd:element name=school" type=xsd:string/> <xsd:element name=committeemember" type=CommitteeType minOccurs=0"/> </xsd:sequence> </xsd:complexType>

Schema Example

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:element name=mastersthesis" type=ThesisType"/> <xsd:complexType name=ThesisType"> name= ThesisType > <xsd:attribute name=mdate" type="xsd:date"/> <xsd:attribute name=key" type="xsd:string"/> <xsd:attribute name=advisor" type="xsd:string"/> <xsd:sequence> <xsd:element name=author" type=xsd:string"/> <xsd:element name=title" type=xsd:string"/> <xsd:element name=year" type=xsd:integer"/> <xsd:element name=school" type=xsd:string/> yp g <xsd:element name=committeemember" type=CommitteeType minOccurs=0"/> </xsd:sequence> </xsd:complexType>

Discussion
Comparison of DTD and XML Schema p

Web & Database

Internet System
Not just a web server or web application An application built over the Internet
Having many participants in client-server or server-to-server fashion Exchanging data / code in distributed fashion for operating the application Partitioning, replicating, translating, data Having code written in different environments, languages, etc. Handling failures, firewalls,

Web: surface ... inside

Rethinking the Web

Tim Berners-Lee (creator of the web) [The Semantic Web, 2001]
the Web ~ interlinking meaning rather than hyperlinks Annotating information with its semantics Make easier mapping between different interpretations

Semantic Web [http://www.w3.org/2001/sw/Activity]

"The Semantic Web is a vision: the idea of having data on the Web defined and linked in such a way that it can be used by machines y y not just for display purposes, but for automation, integration and reuse of data across various applications."

Data integration
Aims
An integrated view on multi data sources A transparent access to multi data sources

Architecture:
Three tier (mediation) .... Multi-tier (hierarchical mediation

Problems
How to define the integrated view How to locate data sources How to forward user queries

Mediator-based architecture
MEDATION N
Mediation Schema

R(#K, A, B, C)
Q: R1 R2 R3

SOURCE ES

R1(#K, A) R2(#A, B) Source Schema 1

R3(#B, C) Source Schema 2

Integrating: Global As View approach

Global as view

The global schema is defined as a view on exported schemas

Global schema GAV

A stems from source 1

Source 1

Integrating: Local As View approach

Local as view
A

Schma global

Exported schemas are defined as views on the global schema

LAV

Source 1

Source 1contains A

Peer-to-Peer Systems
A distributed system with a large number of peers. Every participing peer
as a client and a server providing access to (some of) its ressources
Network

* Resources * Services * Information

Dynamic discovery by contamination

* Client * Server * Client/Server

Search: Resource discovery, Index Storage: Replication, Managing updates, Robustness & fault tolerance Security: Authenticity, Privacy & confidentiality

18

Classification
Unstructured systems : no restriction on data placement
Existing catalog/index Search : (mostly) flooding techniques, keyword search

Structured systems : existing hash table

(Distributed) Hash table
Applying hash function on key attributes Searching based on key values

Search : using index on key attributes

19

Classification (2)
Centralized model
Global index (catalog) A central authority node (as a server)

Decentralized model :
No global index No central coordination

Hierarchical model (partially decentralized)

Several Super-peers (as servers) No overall coordination but sub-network coordination
20

Data Management in P2P Main Issues

Topologies, structured and unstructured p p p g p2p systems Caching and replication Gossip-based dissemination Distributed query processing

21

GRID
Database b Machine

Submission Machine UI

22

Layered Grid Architecture

Application A li ti
Coordinating multiple resources: ubiquitous infrastructure services, app-specific distributed services Sharing single resources: negotiating access, controlling use Talking to things: communication (Internet protocols) & security Controlling things locally: Access to, & control of, resources Internet Protocol A Architecture

Collective Resource Connectivity Fabric

Application

Transport Internet Link

From I. Foster

Mains problems
Resource sharing
Computers, storage, sensors, networks, Heterogeneity of device, mechanism, policy Sharing conditional: negotiation, payment,

Coordinated problem solving

Integration of distributed resources Compound quality of service requirements

Dynamic, multi-institutional virtual orgs

Dynamic overlays on classic org structures Map to underlying control mechanisms

Ideas
Need for interoperability when different groups want to share resources
Diverse components, policies, mechanisms E.g., standard notions of identity, means of communication, resource descriptions

Need for shared infrastructure services to avoid repeated development, installation

E.g., one port/service/protocol for remote access to computing, / / f not one per tool/application E.g., Certificate Authorities: expensive to run

A common need for protocols & services

Basic services
Authentication Authorization Activity control Resource information Resource brokering Scheduling Job submission, data access/migration and execution Accounting

Grid Data Management

Application Metadata Service Planner: Data location, Replica selection, Selection of compute and storage nodes Replica Location Service Information Services Security and Policy Executor: Initiates data transfers and computations Data Movement Data Access Location based on data attributes Location of one or more physical replicas State of grid resources, performance measurements and predictions

Compute Resources

Storage Resources

Mobile Computing Architecture

MU: Mobile Unit MSS: Mobile Support Station Fixed Host: No Wireless interface
28

A Scenario from Mobile Commerce

Consumers are moving on different routes and equipped with locationaware devices A group of the consumers is looking for services (motels, restaurants, parking lots, entertainment, etc. ) Another group is targeted by marketers (special offers, coupons,etc.) The area is managed by a mobile Source: MapQuest computing application The application includes a location management database

29

Sensor Network

Kirk Martinez, Jane K. Hart, and Royan Ong, Environmental Sensor Networks, Computer, Volume 37, Issue 8, Aug. 2004 Page(s): 50 - 56

30

TinyDB DB for Sensor Network

http://telegraph.cs.berkeley.edu/tinydb/

and so many others

To keep up to date http://www-db.cs.wisc.edu/cidr/ http://www.vldb.org http://www.edbt.org http://www.sigmod.org http://www.informatik.uni-trier.de/~ley/db/conf/icde/