Hash Tables and

Query Execution
March 1st, 2004
 Hash Tables
• Secondary storage hash tables are much like
main memory ones
• Recall basics:
– There are n buckets
– A hash function f(k) maps a key k to {0, 1, …, n-1}
– Store in bucket f(k) a pointer to record with key k
• Secondary storage: bucket = block, use
overflow blocks when needed
 Hash Table Example
• Assume 1 bucket (block) stores 2 keys +
pointers
e
• h(e)=0 0

• h(b)=h(f)=1 1
b
f
• h(g)=2 g
2
• h(a)=h(c)=3
a
3
c
 Searching in a Hash Table
• Search for a:
• Compute h(a)=3
e
• Read bucket 3 0

• 1 disk access 1
b
f
g
2
a
3
c
 Insertion in Hash Table
• Place in right bucket, if space
• E.g. h(d)=2
e
0
b
1
f
g
2
d
a
3
c
 Insertion in Hash Table
• Create overflow block, if no space
• E.g. h(k)=1
e
0
b k
1
f
g
2
d
• More over- 3 a
flow blocks c

may be needed
 Hash Table Performance
• Excellent, if no overflow blocks
• Degrades considerably when number of
keys exceeds the number of buckets (I.e.
many overflow blocks).
• Typically, we assume that a hash-lookup
takes 1.2 I/Os.
 Where are we?
• File organizations: sorted, hashed, heaps.
• Indexes: hash index, B+-tree
• Indexes can be clustered or not.
• Data can be stored in the index or not.

• Hence, when we access a relation, we can

either scan or go through an index:
– Called an access path.
 Current Issues in Indexing
• Multi-dimensional indexing:
– how do we index regions in space?
– Document collections?
– Multi-dimensional sales data
– How do we support nearest neighbor queries?

• Indexing is still a hot and unsolved

problem!
User/ Query
Query Execution
Application update
Query compiler
Query execution
plan
Execution engine
Record, index
requests
Index/record mgr.
Page
commands
Buffer manager
Read/write
pages
Storage manager

storage
 Query Execution Plans
SELECT S.sname buyer
FROM Purchase P, Person Q
WHERE P.buyer=Q.name AND

Q.city=‘seattle’ AND City=‘seattle’ phone>’5430000’

Q.phone > ‘5430000’

Query Plan: Buyer=name (Simple Nested Loops)

• logical tree
• implementation Purchase Person
(Table scan) (Index scan)
choice at every
node Some operators are from relational
• scheduling of algebra, and others (e.g., scan, group)
operations. are not.
 The Leaves of the Plan: Scans
• Table scan: iterate through the records of
the relation.
• Index scan: go to the index, from there get
the records in the file (when would this be
better?)
• Sorted scan: produce the relation in order.
Implementation depends on relation size.
How do we combine Operations?
• The iterator model. Each operation is implemented
by 3 functions:
– Open: sets up the data structures and performs
initializations
– GetNext: returns the the next tuple of the result.
– Close: ends the operations. Cleans up the data
structures.
• Enables pipelining!
• Contrast with data-driven materialize model.
• Sometimes it’s the same (e.g., sorted scan).
 Implementing Relational
Operations
• We will consider how to implement:
– Selection ( ) Selects a subset of rows from relation.
– Projection (  ) Deletes unwanted columns from
relation.
– Join (  ) Allows us to combine two relations.
– Set-difference Tuples in reln. 1, but not in reln. 2.
– Union Tuples in reln. 1 and in reln. 2.
– Aggregation (SUM, MIN, etc.) and GROUP BY
 Schema for Examples
Purchase (buyer:string, seller: string, product: integer),

Person (name:string, city:string, phone: integer)

• Purchase:
– Each tuple is 40 bytes long, 100 tuples per page, 1000
pages (i.e., 100,000 tuples, 4MB for the entire relation).
• Person:
– Each tuple is 50 bytes long, 80 tuples per page, 500
pages (i.e, 40,000 tuples, 2MB for the entire relation).
 SELECT *
FROM Person R
Simple Selections WHERE R.phone < ‘543%’

• Of the form 
R. attr op value ( R)
• With no index, unsorted: Must essentially scan the whole
relation; cost is M (#pages in R).
• With an index on selection attribute: Use index to find
qualifying data entries, then retrieve corresponding data
records. (Hash index useful only for equality selections.)
• Result size estimation:
(Size of R) * reduction factor.
More on this later.
 Using an Index for Selections
• Cost depends on #qualifying tuples, and clustering.
– Cost of finding qualifying data entries (typically small) plus cost
of retrieving records.
– In example, assuming uniform distribution of phones, about 54%
of tuples qualify (500 pages, 50000 tuples). With a clustered
index, cost is little more than 500 I/Os; if unclustered, up to 50000
I/Os!
• Important refinement for unclustered indexes:
1. Find sort the rid’s of the qualifying data entries.
2. Fetch rids in order. This ensures that each data page is looked at
just once (though # of such pages likely to be higher than with
clustering).
 Two Approaches to General
Selections
• First approach: Find the most selective access path,
retrieve tuples using it, and apply any remaining
terms that don’t match the index:
– Most selective access path: An index or file scan that
we estimate will require the fewest page I/Os.
– Consider city=“seattle AND phone<“543%” :
• A hash index on city can be used; then,
phone<“543%” must be checked for each retrieved
tuple.
• Similarly, a b-tree index on phone could be used;
city=“seattle” must then be checked.
 Intersection of Rids
• Second approach
– Get sets of rids of data records using each matching
index.
– Then intersect these sets of rids.
– Retrieve the records and apply any remaining terms.
 Implementing Projection
SELECT DISTINCT
R.name,
R.phone
• Two parts: FROM Person R
(1) remove unwanted attributes,
(2) remove duplicates from the result.
• Refinements to duplicate removal:
– If an index on a relation contains all wanted
attributes, then we can do an index-only scan.
– If the index contains a subset of the wanted
attributes, you can remove duplicates locally.
 Equality Joins With One Join Column
SELECT *
FROM Person R, Purchase S
WHERE R.name=S.buyer
• R  S is a common operation. The cross product is too

large. Hence, performing R S and then a selection is too
inefficient.
• Assume: M pages in R, pR tuples per page, N pages in S, pS
tuples per page.
– In our examples, R is Person and S is Purchase.
• Cost metric: # of I/Os. We will ignore output costs.
 Discussion
• How would you implement join?
 Simple Nested Loops Join
For each tuple r in R do
for each tuple s in S do
if ri == sj then add <r, s> to result
• For each tuple in the outer relation R, we scan the entire
inner relation S.
– Cost: M + (pR * M) * N = 1000 + 100*1000*500 I/Os: 140
hours!
• Page-oriented Nested Loops join: For each page of R, get
each page of S, and write out matching pairs of tuples <r,
s>, where r is in R-page and S is in S-page.
– Cost: M + M*N = 1000 + 1000*500 (1.4 hours)
 Index Nested Loops Join
foreach tuple r in R do
foreach tuple s in S where ri == sj do
add <r, s> to result
• If there is an index on the join column of one relation (say
S), can make it the inner.
– Cost: M + ( (M*pR) * cost of finding matching S tuples)
• For each R tuple, cost of probing S index is about 1.2 for
hash index, 2-4 for B+ tree. Cost of then finding S tuples
depends on clustering.
– Clustered index: 1 I/O (typical), unclustered: up to 1 I/O per
matching S tuple.
 Examples of Index Nested Loops
• Hash-index on name of Person (as inner):
– Scan Purchase: 1000 page I/Os, 100*1000 tuples.
– For each Person tuple: 1.2 I/Os to get data entry in index, plus 1
I/O to get (the exactly one) matching Person tuple. Total:
220,000 I/Os. (36 minutes)
• Hash-index on buyer of Purchase (as inner):
– Scan Person: 500 page I/Os, 80*500 tuples.
– For each Person tuple: 1.2 I/Os to find index page with data
entries, plus cost of retrieving matching Purchase tuples.
Assuming uniform distribution, 2.5 purchases per buyer (100,000
/ 40,000). Cost of retrieving them is 1 or 2.5 I/Os depending on
clustering.
 Block Nested Loops Join
• Use one page as an input buffer for scanning the
inner S, one page as the output buffer, and use all
remaining pages to hold ``block’’ of outer R.
– For each matching tuple r in R-block, s in S-page, add
<r, s> to result. Then read next R-block, scan S, etc.

R&S Join Result

Hash table for block of R
(k < B-1 pages)
...
... ...
Input buffer for S Output buffer
 Sort-Merge Join (R i=j S)
• Sort R and S on the join column, then scan them to
do a ``merge’’ on the join column.
– Advance scan of R until current R-tuple >= current S
tuple, then advance scan of S until current S-tuple >=
current R tuple; do this until current R tuple = current S
tuple.
– At this point, all R tuples with same value and all S
tuples with same value match; output <r, s> for all pairs
of such tuples.
– Then resume scanning R and S.
 Cost of Sort-Merge Join
• R is scanned once; each S group is scanned
once per matching R tuple.
• Cost: M log M + N log N + (M+N)
– The cost of scanning, M+N, could be M*N
(unlikely!)
• With 35, 100 or 300 buffer pages, both
Person and Purchase can be sorted in 2
passes; total: 7500. (75 seconds).
 Original
Hash-Join Relation OUTPUT
1
Partitions

1
• Partition both relations INPUT 2
using hash fn h: R hash 2
tuples in partition i will ... function
h B-1
only match S tuples in B-1
partition i. Disk B main memory buffers Disk

Partitions
of R & S Join Result
Hash table for partition
 Read in a partition hash Ri (k < B-1 pages)
of R, hash it using fn
h2
h2 (<> h!). Scan
matching partition h2

of S, search for Input buffer

for Si
Output
buffer
matches.
Disk B main memory buffers Disk
 Cost of Hash-Join
• In partitioning phase, read+write both relations; 2(M+N).
In matching phase, read both relations; M+N I/Os.
• In our running example, this is a total of 4500 I/Os. (45
seconds!)
• Sort-Merge Join vs. Hash Join:
– Given a minimum amount of memory both have a cost
of 3(M+N) I/Os. Hash Join superior on this count if
relation sizes differ greatly. Also, Hash Join shown to be
highly parallelizable.
– Sort-Merge less sensitive to data skew; result is sorted.
 How are we doing?
Nested loop 140 hours 50 million
join I/Os
Page nested 1.4 hours 500,000 I/Os
loop
Index nested 36 minutes 220,000 I/Os
loop
Sort-merge 75 seconds 7500 I/O’s
join
Hash join 45 seconds 4500 I/O’s
 Double Pipelined Join (Tukwila)

Hash Join Double Pipelined Hash Join

Partially pipelined: no
output until inner read
Outputs data immediately
Asymmetric (inner vs.
outer) — optimization Symmetric — requires less
requires source behavior source knowledge to optimize
knowledge