Pig, Hive, and Jaql
IBM Information Management
Cloud Computing Center of Competence
IBM Toronto Lab
Agenda
Overview
Pig
Hive
Jaql
Agenda
Overview
Pig
Hive
Jaql
Similarities of Pig, Hive and Jaql
All translate their respective high-level languages to
MapReduce jobs
All offer significant reductions in program size over
Java
All provide points of extension to cover gaps in
functionality
All provide interoperability with other languages
None support random reads/writes or low-latency
queries
Comparing Pig, Hive, and Jaql
Characteristic
Pig
Hive
Jaql
Developed by
Yahoo!
Facebook
IBM
Language name
Pig Latin
HiveQL
Jaql
Type of language
Data flow
Declarative (SQL
dialect)
Data flow
Data structures it
operates on
Complex, nested
JSON
Schema optional?
Yes
No, but data can
have many
schemas
Relational complete?
Yes
Yes
Turing complete?
Yes when
extended with
Java UDFs
Yes when
extended with
Java UDFs
Yes
Yes
Agenda
Overview
Pig
Hive
Jaql
Pig components
Two Components
Language (called Pig Latin)
Compiler
Pig
Pig Latin
Compiler
Two execution environments
Local (Single JVM)
Distributed (Hadoop cluster)
pig -x local
pig -x mapreduce, or simply pig
Execution Environment
Local
Distributed
Running Pig
Script
pig scriptfile.pig
Grunt (command line)
pig (to launch command line tool)
Embedded
Call in to Pig from Java
Sample code
#pig
grunt> records = LOAD econ_assist.csv
using PigStorage (,)
AS (country:chararray, sum:long);
grunt> grouped = GROUP records BY country;
grunt> thesum
= FOREACH grouped
GENERATE group,
SUM(records, sum);
grunt> DUMP thesum;
Pig Latin
Statements, operations and commands
10
A Pig Latin program is a collection of statements.
A statement is an operation or a command
Example of an operation: LOAD 'statement.txt';
Example of a command: ls *.txt
Logical plan/physical plan
As statement is processed, it is added to logical plan
When a statement such as 'DUMP relation' is reached, logical
plan is compiled to physical plan and executed
Pig Latin statements
UDF Statements
Commands
Hadoop Filesystem (cat, ls, etc.)
Hadoop MapReduce (kill)
Utility (exec, help, quit, run, set)
Diagnostic Operators
11
REGISTER, DEFINE
DESCRIBE, EXPLAIN, ILLUSTRATE
Pig Latin Relational operators
12
Loading and storing (LOAD, STORE, DUMP)
Filtering (FILTER, DISTINCT, FOREACH...GENERATE,
STREAM, SAMPLE)
Grouping and joining (JOIN, COGROUP, GROUP,
CROSS)
Sorting (ORDER, LIMIT)
Combining and splitting (UNION, SPLIT)
Pig Latin Relations and schemata
Result of a relational operator is a relation
A relation is a set of tuples
Relations can be named using an alias
x = LOAD 'sample.txt' AS (id: int, year: int)
DUMP x
alias
(1,1987) tuple
Structure of a relation is a schema
DESCRIBE x
x: {id: int, year: int} schema
13
Pig Latin expressions
Part of a statement containing a relational operator
Categories of expressions:
Constant
Map lookup
Conditional
Functional
14
Field
Cast
Boolean
Flatten
Projection
Arithmetic
Comparison
Pig Latin Data types
Simple types:
int
long
float
double
bytearray
chararray
Complex types:
Tuple
Bag
Map
15
Sequence of fields of any type
Unordered collection of tuples
Set of key-value pairs. Keys must be chararray.
Pig Latin Function types
Eval
Input: One or more expressions
Output: An expression
Example: MAX
Filter
Input: Bag or map
Output: boolean
Example: IsEmpty
16
Pig Latin Function types
Load
Input: Data from external storage
Output: A relation
Example: PigStorage
Store
Input: A relation
Output: Data to external storage
Example: PigStorage
17
Pig Latin User-Defined Functions
Written in Java
Packaged in a JAR file
Register JAR file using the REGISTER statement
Optionally, alias it with DEFINE statement
18
Agenda
19
Overview
Pig
Hive
Jaql
Hive - Configuration
Three ways to configure hive:
hive-site.xml
-
fs.default.name
mapred.job.tracker
Metastore configuration settings
hive hiveconf
Set command in the Hive Shell
20
Running Hive
Hive Shell
Interactive
hive
Script
hive -f myscript
Inline
hive -e 'SELECT * FROM mytable'
21
Hive services
hive --service servicename
where servicename can be:
hiveserver
server for Thrift, JDBC, ODBC clients
hwi
web interface
jar
hadoop jar with Hive jars in classpath
metastore
out of process metastore
22
Sample code
#hive
hive> CREATE TABLE foreign_aid
(country STRING, sum BIGINT)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ,
STORED AS TEXTFILE;
hive> SHOW TABLES;
hive> DESCRIBE foreign_aid;
hive> LOAD DATA INPATH econ_assist.csv
OVERWRITE INTO TABLE foreign_aid;
hive> SELECT * FROM foreign_aid LIMIT 10;
hive> SELECT country, SUM(sum) FROM foreign_aid
GROUP BY country;
23
Hive - Metastore
Stores Hive metadata
Configurations
Embedded
in-process metastore, in-process database
Local
in-process metastore, out-of-process database
Remote
out-of-process metastore, out-of-process
database
24
Hive Schema-On-Read
25
Faster loads into the database (simply copy
or move)
Slower queries
Flexibility multiple schemas for the same
data
Hive Query Language (HiveQL)
SQL dialect
Does not support full SQL92 specification
No support for:
HAVING clause in SELECT
Correlated subqueries
Subqueries outside FROM clauses
Updateable or materialized views
Stored procedures
26
Hive Query Language (HiveQL)
Extensions
MySQL-like extensions
MapReduce extensions
Multi-table insert, MAP, REDUCE, TRANSFORM clauses
Data Types
Simple
TINYINT, SMALLINT, INT, BIGINT, FLOAT, DOUBLE, BOOLEAN, STRING
Complex
ARRAY, MAP, STRUCT
27
Hive Query Language (HiveQL)
Built-in Functions
28
SHOW FUNCTIONS
DESCRIBE FUNCTION
Hive - Tables
Managed CREATE TABLE
External CREATE EXTERNAL TABLE
LOAD File moved into Hive's data warehouse directory
DROP Both metadata and data deleted
LOAD No files moved
DROP Only metadata deleted
Use EXTERNAL when:
29
Sharing data between Hive and other Hadoop applications
You wish to use multiple schemas on the same data
Hive - Partitioning
Can make some queries faster
Divide data based on partition column
Use PARTITION BY clause when creating table
Use PARTITION clause when loading data
SHOW PARTITIONS will show a table's partitions
30
Hive - Bucketing
Can make some queries faster
Supports sampling data
Use CLUSTERED BY clause when creating table
For sorted buckets, use SORTED BY clause
when creating table
To query a sample of your data use the
TABLESAMPLE command which uses bucketing
31
Hive Storage formats
Delimited Text (default)
ROW FORMAT DELIMITED
SerDe Serializer/Deserializer
ROW FORMAT SERDE serdename
e.g. ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy'
Binary SerDe
Row-oriented (Sequence file)
STORED AS SEQUENCEFILE
Column-oriented (RCFile)
STORED AS RCFILE
32
Hive User-Defined Functions
Written in Java
Three UDF types:
UDF
Input: single row, output: single row
UDAF
Input: multiple rows, output: single row
UDTF
Input: single row, output: multiple rows
Register UDF using ADD JAR
Create alias using CREATE TEMPORARY FUNCTION
33
Agenda
34
Overview
Pig
Hive
Jaql
Jaql
A JSON Query Language
JSON = Javascript Object Notation
Running Jaql
35
Jaql Shell
Interactive
jaqlshell
Batch
jaqlshell -b myscript.jaql
Inline
jaqlshell -e jaqlstatement
Modes
Cluster
jaqlshell -c
Minicluster
jaqlshell
Jaql
Query Language
Sources and sinks
e.g. Copy data from a local file to a new file on HDFS
source
sink
read(file(input.json)) -> write(hdfs(output))
36
Core Operators
Filter
Group
Tee
Transform
Join
Sort
Expand
Union
Top
Jaql
Query Language
Variables
= operator binds source output to a variable
e.g. $tweets = read(hdfs(twitterfeed))
Pipes, streams, and consumers
Pipe operator (->) streams data to a consumer
Pipe expects array as input
e.g. $tweets filter $.from_src == 'tweetdeck';
37
$ implicit variable referencing current array value
Jaql
Query Language
38
Categories of Built-in Functions
system
schema
core
xml
hadoop
regex
io
binary
array
date
index
nil
agg
number
string
function
random
record
Jaql
Data Storage
Data store examples
Amazon S3
DB2
HBase
HTTP
JDBC
Local FS
Data format examples
JSON
39
CSV
XML
HDFS
Thank you!
Thank you!
