Distributed hash table (DHT)

Lecturer: Thanh-Chung Dao

Slides by Viet-Trung Tran
School of Information and Communication Technology
Outline

• Hashing
• Distributed Hash Table
• Chord

2
A Hash Table (hash map)

• A data structure implements an associative array that can

map keys to values.
• searching and insertions are 0(1) in the worse case
• Uses a hash function to compute an index into an array of
buckets or slots from which the correct value can be found.
• index = f(key, array_size)

3
Hash functions

• Crucial for good hash table performance

• Can be difficult to achieve
• WANTED: uniform distribution of hash values
• A non-uniform distribution increases the number of collisions and the
cost of resolving them

4
Hashing for partitioning usecase

• Objective
• Given document X, choose one of k servers to use
• Eg. using modulo hashing
• Number servers 1..k
• Place X on server i = (X mod k)
• Problem? Data may not be uniformly distributed
• Place X on server i = hash (X) mod k
• Problem?
• What happens if a server fails or joins (k à k±1)?
• What is different clients has different estimate of k?
• Answer: All entries get remapped to new nodes!

5
Distributed hash table (DHT)

• Distributed Hash Table (DHT) is similar to hash table but

spread across many hosts
• Interface
• insert(key, value)
• lookup(key)
• Every DHT node supports a single operation:
• Given key as input; route messages to node holding key

6
 DHT: basic idea

K V
K V

K V
K V

K V

K V K V

K V

K V
K V
K V

7
 DHT: basic idea

K V
K V

K V
K V

K V

K V K V

K V

K V
K V
K V

Neighboring nodes are “connected” at the application-level8

 DHT: basic idea

K V
K V

K V
K V

K V

K V K V

K V

K V
K V
K V

Operation: take key as input; route messages to node holding key 9

 DHT: basic idea

K V
K V

K V
K V

K V

K V K V

K V

K V
K V
K V
insert(K1,V1)

Operation: take key as input; route messages to node holding key 10

 DHT: basic idea

K V
K V

K V
K V

K V

K V K V

K V

K V
K V
K V
insert(K1,V1)

Operation: take key as input; route messages to node holding key 11

 DHT: basic idea

(K1,V1) K V
K V

K V
K V

K V

K V K V

K V

K V
K V
K V

Operation: take key as input; route messages to node holding key 12

 DHT: basic idea

K V
K V

K V
K V

K V

K V K V

K V

K V
K V
K V

retrieve (K1)

Operation: take key as input; route messages to node holding key 13

How to design a DHT?

• State Assignment
• What “(key, value) tables” does a node store?
• Network Topology
• How does a node select its neighbors?
• Routing Algorithm:
• Which neighbor to pick while routing to a destination?
• Various DHT algorithms make different choices
• CAN, Chord, Pastry, Tapestry, Plaxton, Viceroy, Kademlia, Skipnet,
Symphony, Koorde, Apocrypha, Land, ORDI …

14
Chord: A scalable peer-to-peer look-up
protocol for internet applications
Credit: University of California, berkely and Max planck institute

15
Outline

• What is Chord?
• Consistent Hashing
• A Simple Key Lookup Algorithm
• Scalable Key Lookup Algorithm
• Node Joins and Stabilization
• Node Failures

16
What is Chord?

• In short: a peer-to-peer lookup system

• Given a key (data item), it maps the key onto a node (peer).
• Uses consistent hashing to assign keys to nodes .
• Solves the problem of locating key in a collection of
distributed nodes.
• Maintains routing information with frequent node arrivals
and departures

17
Consistent hashing

• Consistent hash function assigns each node and key an m-bit

identifier.
• SHA-1 is used as a base hash function.
• A node’s identifier is defined by hashing the node’s IP
address.
• A key identifier is produced by hashing the key (chord doesn’t
define this. Depends on the application).
• ID(node) = hash(IP, Port)
• ID(key) = hash(key)

18
Consistent hashing

• In an m-bit identifier space, there are 2m identifiers.

• Identifiers are ordered on an identifier circle modulo 2m.
• The identifier ring is called Chord ring.
• Key k is assigned to the first node whose identifier is equal to
or follows (the identifier of) k in the identifier space.
• This node is the successor node of key k, denoted by
successor(k).

19
 Consistent hashing – Successor nodes

identifier
node
6
X key
1
0 successor(1) = 1
7 1

identifier
successor(6) = 0 6 6 circle 2 2 successor(2) = 3

5 3
4
2

20
Consistent hashing – Join and departure

• When a node n joins the network, certain keys previously

assigned to n’s successor now become assigned to n.
• When node n leaves the network, all of its assigned keys are
reassigned to n’s successor.

21
Consistent hashing – Node join

keys
5
7

keys
0 1

7 1

keys

6 2

keys
5 3 2

22
Consistent hashing – Node departure

keys
7

keys
0 1

7 1

keys
6 6 2

keys
5 3 2

23
A Simple key lookup

• If each node knows only how to contact its current successor

node on the identifier circle, all node can be visited in linear
order.
• Queries for a given identifier could be passed around the
circle via these successor pointers until they encounter the
node that contains the key.

24
A Simple key lookup

• Pseudo code for finding successor:

// ask node n to find the successor of id
n.find_successor(id)
if (id Î (n, successor])
return successor;
else
// forward the query around the circle
return successor.find_successor(id);

25
A Simple key lookup

• The path taken by a query from node 8 for key 54:

26
Scalable key location

• To accelerate lookups, Chord maintains additional routing

information.
• This additional information is not essential for correctness,
which is achieved as long as each node knows its correct
successor.

27
Scalable key location – Finger tables

• Each node n’ maintains a routing table with up to m entries

(which is in fact the number of bits in identifiers), called finger
table.
• The ith entry in the table at node n contains the identity of
the first node s that succeeds n by at least 2^i-1 on the
identifier circle.
• s = successor(n+2^i-1).
• s is called the ith finger of node n, denoted by n.finger(i)

28
Scalable key location – Finger tables

finger table keys

For. start succ. 6
0+20 1 1
0+21 2 3
0+22 4 0

finger table keys

0 For. start succ. 1

7 1 1+20 2 3
1+21 3 3
1+22 5 0

6 2

finger table keys

5 3 For. start succ. 2
3+20 4 0
4 3+21 5 0
3+22 7 0

29
Scalable key location – Finger tables

• A finger table entry includes both the Chord identifier and the
IP address (and port number) of the relevant node.
• The first finger of n is the immediate successor of n on the
circle.

30
Scalable key location – Example query

• The path a query for key 54 starting at node 8:

31
Scalable key location – A characteristic

• Since each node has finger entries at power of two intervals

around the identifier circle, each node can forward a query at
least halfway along the remaining distance between the node
and the target identifier. From this intuition follows a theorem:
• Theorem: With high probability, the number of nodes that must be
contacted to find a successor in an N-node network is O(logN).

32
Node joins and stabilizations

• The most important thing is the successor pointer.

• If the successor pointer is ensured to be up to date, which is
sufficient to guarantee correctness of lookups, then finger
table can always be verified.
• Each node runs a “stabilization” protocol periodically in the
background to update successor pointer and finger table.

33
Node joins and stabilizations

• “Stabilization” protocol contains 6 functions:

• create()
• join()
• stabilize()
• notify()
• fix_fingers()
• check_predecessor()

34
Node Joins – join()

• When node n first starts, it calls n.join(n’), where n’ is any

known Chord node.
• The join() function asks n’ to find the immediate successor of
n.
• join() does not make the rest of the network aware of n.

35
Node Joins – join()

// create a new Chord ring.

n.create()
predecessor = nil;
successor = n;

// join a Chord ring containing node n’.

n.join(n’)
predecessor = nil;
successor = n’.find_successor(n);

36
Node joins – stabilize()

• Each time node n runs stabilize(), it asks its successor for it’s
predecessor p, and decides whether p should be n’s successor
instead.
• stabilize() notifies node n’s successor of n’s existence, giving
the successor the chance to change its predecessor to n.
• The successor does this only if it knows of no closer
predecessor than n.

37
Node joins – stabilize()

// called periodically. verifies n’s immediate

// successor, and tells the successor about n.
n.stabilize()
x = successor.predecessor;
if (x Î (n, successor))
successor = x;
successor.notify(n);

// n’ thinks it might be our predecessor.

n.notify(n’)
if (predecessor is nil or n’ Î (predecessor, n))
predecessor = n’;

38
Node joins – Join and stabilization

n n joins
ns n predecessor = nil
n n acquires ns as successor via some n’
pred(ns) = n
n n runs stabilize
n n notifies ns being the new predecessor
n ns acquires n as its predecessor

succ(np) = ns
n np runs stabilize
pred(ns) = np

n
n np asks ns for its predecessor (now n)
n np acquires n as its successor
succ(np) = n

n np notifies n
nil n n will acquire np as its predecessor
n all predecessor and successor pointers are now
correct

n fingers still need to be fixed, but old fingers will

still work
np

39
Node Joins – fix_fingers()

• Each node periodically calls fix fingers to make sure its finger
table entries are correct.
• It is how new nodes initialize their finger tables
• It is how existing nodes incorporate new nodes into their
finger tables.

40
Node Joins – fix_fingers()

// called periodically. refreshes finger table entries.

n.fix_fingers()
next = next + 1 ;
if (next > m)
next = 1 ;
finger[next] = find_successor(n + 2next-1);

// checks whether predecessor has failed.

n.check_predecessor()
if (predecessor has failed)
predecessor = nil;

41
Node failures

• Key step in failure recovery is maintaining correct successor

pointers
• To help achieve this, each node maintains a successor-list of
its r nearest successors on the ring. Hence, all r successors
would have to simultaneously fail in order to disrupt the
Chord ring.
• If node n notices that its successor has failed, it replaces it
with the first live entry in the list
• Successor lists are stabilized as follows:
• node n reconciles its list with its successor s by copying s’s successor
list, removing its last entry, and prepending s to it.
• If node n notices that its successor has failed, it replaces it with the
first live entry in its successor list and reconciles its successor list with
its new successor.

42
Chord – The math

• Each node maintains O(logN) state information and lookups

needs O(logN) messages
• Every node is responsible for about K/N keys (N nodes, K keys)
• When a node joins or leaves an N-node network, only O(K/N)
keys change hands (and only to and from joining or leaving
node)

43
Interesting simulation results

• Adding virtual nodes as an indirection layer can significantly

improve load balance.
• The average path length is about ½(logN).
• Maintaining a set of alternate nodes for each finger and route
the queries by selecting the closest node according to
network proximity metric improves routing latency effectively.
• Recursive lookup style is faster iterative style

44
Applications: Time-shared storage

• For nodes with intermittent connectivity (server only

occasionally available)
• Store others‘ data while connected, in return having their
data stored while disconnected
• Data‘s name can be used to identify the live Chord node
(content-based routing)

45
Applications: Chord-based DNS

• DNS provides a lookup service

• keys: host names values: IP adresses
• Chord could hash each host name to a key
• Chord-based DNS:
• no special root servers
• no manual management of routing information
• no naming structure
• can find objects not tied to particular machines

46
What is Chord? – Addressed problems

• Load balance: chord acts as a distributed hash function,

spreading keys evenly over nodes
• Decentralization: chord is fully distributed, no node is more
important than any other, improves robustness
• Scalability: logarithmic growth of lookup costs with the
number of nodes in the network, even very large systems are
feasible
• Availability: chord automatically adjusts its internal tables to
ensure that the node responsible for a key can always be
found
• Flexible naming: chord places no constraints on the structure
of the keys it looks up.

47
Summary

• Simple, powerful protocol

• Only operation: map a key to the responsible node
• Each node maintains information about O(log N) other nodes
• Lookups via O(log N) messages
• Scales well with number of nodes
• Continues to function correctly despite even major changes of
the system

48
Thanks for your attention!

49