Course Topics
Week 1 Week 5
– Introduction to HDFS – HIVE
Week 2 Week 6
– Setting Up Hadoop Cluster – HBASE
Week 3 Week 7
– Map-Reduce Basics, types and formats – ZOOKEEPER
Week 4 Week 8
– PIG – SQOOP
What are we going to learn Today..!
• Problems in the real world
• Traditional RDBMS fallacies
• The advent of HBase
• HBase Architecture
• Hands-on creation, updation of HBase table on shell
• Multiple ways of loading data into HBase (Shell, Jvm-Client, MapReduce, Avro, Thrift,
REST Api)
Problem in real world
Linkedin
Revolutionizing Education
Add Targeting
So, what is common?
• Huge Data
• Fast Random access
• Structured Data
• Variable Schema
• Need of Compression
• Need of Distribution (Sharding)
How Traditional RDBMS will solve
Users Follower
Id User_id
Name Follower_id
Sex type
age
Contd.
Users Connections
Id User_id
Name Connection_id
Sex type
age
Characteristics Of Probable Solution
• Distributed database
• Sorted data
• Sparse data store
• Automatic sharding
History of HBase
2006 Big Table paper published
2006 HBase development starts
2008 Microsoft buys powerset
2010 Facebook’s messaging system
Facebook Messaging System
• Facebook monitored their usage and figured out what the really
needed.
• What they needed was a system that could handle two types of data
patterns:
– A short set of temporal data that tends to be volatile
– An ever-growing set of data that rarely gets accessed
real-time, distributed, linearly scalable, robust, BigData, open-source, key-value, column-oriented
HBase Definition
HBase is a key/value store. Specifically it is a
Sparse, Consistent, Distributed, Multidimensional, Sorted map.
More HBase Implementation
uses HBase to power their Messages
A number of applications including infrastructure
people search rely on HBase internally http://sites.computer.org\debull\A12june\facebo
for data generation. ok.pdf
uses HBase to store document fingerprint for
We use HBase as a real time data detecting near-duplications. We have a
storage and analytics platform. cluster of few nodes that runs HDFS,
mapreduce, and HBase.
uses an HBase cluster containing
uses HBase as a foundation for cloud scale
over a billion anonymized clinical
storage for a variety of applications.
records.
Referred - http://wiki.apache.org/hadoop/Hbase/PoweredBy
Data Model
Versions Of Data
Row key Personal_data demographic
Persons ID Name Address Birth Date Gender
1 Harry BTM layout 1988-10-31 M
2 Dhawan 1956-09-16 M
3 Sana whitefield 1189-12-03 F
….. ….. ….. ….. …..
500,000,000 vineet delhi 1964-01-07 M
Physical storage
Col3(Birth date) ->
1926-10-31
Col1(address) ->Budapest
Row 1(1)
Col3(Gender) -> M
Col1(Name) -> H. Houdini
Col3(Birth date) ->
Row 2 (2) val3
Col5(address) -> D. Copper
Row 3 (3) Col4(Gender) -> val4
Family1(personal data) Family2(Demographic)
How does It look like?
What it means?
Column Family:
Row Key Values
Column Qualifier
• Unique for each row • Less number of families • Various versions of values
• Identifies each row gives faster access are maintained
• Families are fixed column • Scan shows only recent
qualifiers are not version
Three Major Components
Data Distribution
Row
s
Logical View – All rows in a
Region
A1
Null -> A3
A2
A22 Region
A3 A3 -> F34
…..
…..
K4 Region
….. F34 -> K80
….. Region
090 K80 -> 095
table
….. Region
….. 095 -> Null
…..
Z30
Z55 Region Region Region
Server Server Server
HBase Components
Zookeeper
Master /hbase/region
1
/hbase/region
2
…..
RegionServers
…..
memstore
/hbase/region
HDFS HFile WAL
HBase Components
• Table made of regions
• Region – a range of rows stored together
- Single Shard, used for scaling
- Dynamically merge if too big
- merge if too small
• Region servers- serves one or more regions
- A region is served by only one region
• Master server – demon responsible for managing HBase cluster
• HBase stores its data into HDFS
- relies on HDFS’s High Availability and fault tolerance
HBase Storage Architecture
Hbase Storage Simpler
Zookeeper Zookeeper Zookeeper
HBase HDFS
HDFS SNN
Master Namenode
Management Management Management
Node Node Node
Zookeeper Zookeeper Scale Zookeeper
HBase HBase Horizontally HBase
Region Server Region Server N Machines Region Server
Data Node Data Node
Data Node
Different Types of regions
Root/Meta Table
Each row in the ROOT and META tables is approximately 1KB in size.
At the default size of 256MB.
Compactions
Compactions
Time Column
Row key Column “anchor:”
Stamp “contents:”
t12 “<html>…”
t11 “<html>…”
“com.apache.www”
t10 “anchor:apache.com” “APACHE”
t9 “anchor:cnnsi.com” “CNN”
t8 “anchor:my.look.ca” “CNN.com”
“com.cnn.www”
t6 “<html>…”
t5 “<html>…”
t3 “<html>…”
Hstore1
Region Split
Region Splits
Time Column
Row key Column “anchor:”
Stamp “contents:”
t12 “<html>…”
“com.apache.www” t11 “<html>…”
t10 “anchor:apache.com” “APACHE”
t9 “anchor:cnnsi.com” “CNN”
t8 “anchor:my.look.ca” “CNN.com”
“com.cnn.www”
t6 “<html>…”
t5 “<html>…”
t3 “<html>…”
Hstore1
HBase Client API
HBase Client API
Scanner and Filters
Search
Get value from table where key=„com.apache.www‟ AND label=„anchor:apache.com‟
Time
Row key Column “anchor:”
Stamp
t12
“com.apache.www” t11
t10 “anchor:apache.com” “APACHE”
t9 “anchor:cnnsi.com” “CNN”
t8 “anchor:my.look.ca” “CNN.com”
“com.cnn.www” t6
t5
t3
Search
Scanner Select value from table
where, anchor=„cnnsi.com‟
Time
Row key Column “anchor:”
Stamp
t12
“com.apache.www” t11
t10 “anchor:apache.com” “APACHE”
t9 “anchor:cnnsi.com” “CNN”
t8 “anchor:my.look.ca” “CNN.com”
“com.cnn.www” t6
t5
t3
Hbase API
• get(row)
• put(row,Map<column,value>)
• scan(key range, filter)
• increment(row, columns)
• Check and Put, delete etc.
Hbase Shell
• hbase(main):003:0> create 'test', 'cf'
0 row(s) in 1.2200 seconds
• hbase(main):004:0> put 'test', 'row1', 'cf:a', 'value1'
0 row(s) in 0.0560 seconds
• hbase(main):005:0> put 'test', 'row2', 'cf:b', 'value2'
0 row(s) in 0.0370 seconds
• hbase(main):006:0> put 'test', 'row3', 'cf:c', 'value3'
0 row(s) in 0.0450 seconds
Hbase Shell Contd.
• hbase(main):007:0> scan 'test'
ROW COLUMN+CELL
• row1 column=cf:a, timestamp=1288380727188,
value=value1
• row2 column=cf:b, timestamp=1288380738440,
value=value2
• row3 column=cf:c, timestamp=1288380747365,
value=value3
3 row(s) in 0.0590 seconds
Thank You
See You in Class Next Week
