w it h

n c e
r m a 5 .5
e r fo and
s t p 5 .1
B oo Q L in g
y S it io n
M r t
pa

Giuseppe Maxia
MySQL Community
Team Lead
Sun Microsystems
Wednesday, 13 January 2010
 Who's this guy?

about me
2

Wednesday, 13 January 2010

 Giuseppe Maxia
• a.k.a. The Data Charmer
• MySQL Community Team Lead
• Long time hacking with MySQL features
• Formerly, database consultant, designer, coder.
• A passion for QA
• An even greater passion for open source
• ... and community
• Passionate blogger
• http://datacharmer.blogspot.com 3

Wednesday, 13 January 2010

 MySQL 5.1 GA

Wednesday, 13 January 2010

 MySQL 5.1 GA

Wednesday, 13 January 2010

 Defining the problem

YOUR
NEEDS 6

Wednesday, 13 January 2010

 The problem(s)
• Too much data
• Not enough RAM
• Historical data
• Growing data
• Rotating data

Wednesday, 13 January 2010

 Too much data

data

Wednesday, 13 January 2010

 Not enough RAM

RAM

INDEXES
data INDEXES

Wednesday, 13 January 2010

 Not enough RAM

INDEXES
RAM
data INDEXES

10

Wednesday, 13 January 2010

 MySQL 5.1 partitions

WHAT
11

Wednesday, 13 January 2010

 What is it?
• Logical splitting of tables
• Transparent to user

12

Wednesday, 13 January 2010

 Logical splitting
• No need to create separate tables
• No need to move chunks of data across files

13

Wednesday, 13 January 2010

 MySQL 5.1 partitions

TRANSPARENT SQL
14

Wednesday, 13 January 2010

 COMPARED TO MERGE TABLES

MERGE TABLE • separate tables

• risk of duplicates
• insert in each table
• no constraints

15

Wednesday, 13 January 2010

 COMPARED TO MERGE TABLES

PARTITIONED TABLE • One table

• No risk of duplicates
• insert in one table
• constraints enforced

16

Wednesday, 13 January 2010

 Wait a minute ...
• WHAT THE HELL DOES "LOGICAL SPLIT"
REALLY MEANS?
• LET ME EXPLAIN ...

17

Wednesday, 13 January 2010

 Physical partitioning (1)
• Take a map

18

Wednesday, 13 January 2010

 Physical partitioning (2)
• cut it into pieces

19

Wednesday, 13 January 2010

 Physical partitioning (3)
• What you have, is several different pieces

20

Wednesday, 13 January 2010

 Physical partitioning (4)
• If you want the map back, you need some
application (adhesive tape) and you may get it
wrong

21

Wednesday, 13 January 2010

 Logical partitioning (1)
• Take a map

22

Wednesday, 13 January 2010

 Logical partitioning (2)
• fold it to show the piece you need

23

Wednesday, 13 January 2010

 Logical partitioning (3)
• what you have is still a map, even if you see only
one part.

24

Wednesday, 13 January 2010

 Logical partitioning (4)
• if you unfold the map, you still have the whole thing

25

Wednesday, 13 January 2010

 What partitions can do
• logical split
• but you can also split the data physically
• granular partition (subpartitioning)
• different methods (range, list, hash, key)

26

Wednesday, 13 January 2010

 Partition pruning
1a - unpartitioned table - SINGLE RECORD

select *
from
table_name
where colx =
120

Wednesday, 13 January 2010

 Partition pruning
1b - unpartitioned table - SINGLE RECORD

select *
from
table_name
where colx =
350

Wednesday, 13 January 2010

 Partition pruning
1c - unpartitioned table - RANGE

select *
from
table_name
where colx
between 120
and 230

Wednesday, 13 January 2010

 Partition pruning
2a - table partitioned by colx - SINGLE REC
1-99

100-199
select *
from
200-299 table_name
where colx =
300-399 120
400-499

500-599
Wednesday, 13 January 2010
 Partition pruning
2b - table partitioned by colx - SINGLE REC
1-99

100-199
select *
from
200-299 table_name
where colx =
300-399 350
400-499

500-599
Wednesday, 13 January 2010
 Partition pruning
2c - table partitioned by colx - RANGE
1-99

select *
100-199 from
table_name
200-299
where colx
300-399 between 120
and 230
400-499

500-599
Wednesday, 13 January 2010
 MySQL 5.1 partitions

WHY
33

Wednesday, 13 January 2010

 4 main reasons to use partitions
• To make single inserts and selects faster
• To make range selects faster
• To help split the data across different paths
• To store historical data efficiently
• If you need to delete large chunks of data
INSTANTLY

34

Wednesday, 13 January 2010

 MySQL 5.1 partitions

HOW
35

Wednesday, 13 January 2010

 HOW TO MAKE PARTITIONS

•RTFM ...
• No, seriously, the manual has everything
• But if you absolutely insist ...

36

Wednesday, 13 January 2010

 HOW TO MAKE PARTITIONS
CREATE TABLE t1 (
id int
) ENGINE=InnoDB
# or MyISAM, ARCHIVE
PARTITION BY RANGE (id)
(
PARTITION P1 VALUES LESS THAN (10),
PARTITION P2 VALUES LESS THAN (20)
) 37

Wednesday, 13 January 2010

 HOW TO MAKE PARTITIONS
CREATE TABLE t1 (
id int
) ENGINE=InnoDB
PARTITION BY LIST (id)
(
PARTITION P1 VALUES IN (1,2,4),
PARTITION P2 VALUES IN (3,5,9)
)
38

Wednesday, 13 January 2010

 HOW TO MAKE PARTITIONS
CREATE TABLE t1 (
id int not null primary key
) ENGINE=InnoDB
PARTITION BY HASH (id)
PARTITIONS 10;

39

Wednesday, 13 January 2010

 HOW TO MAKE PARTITIONS
CREATE TABLE t1 (
id int not null primary key
) ENGINE=InnoDB
PARTITION BY KEY ()
PARTITIONS 10;

40

Wednesday, 13 January 2010

 Limitations
• Can partition only by INTEGER columns
• OR you can partition by an expression, which must
return an integer
• Maximum 1024 partitions

• If you have a Unique Key or PK, the partition

column must be part of that key
• No Foreign Key support
• No Fulltext or GIS support
41

Wednesday, 13 January 2010

 PARTITIONING BY DATE
CREATE TABLE t1 (
d date
) ENGINE=InnoDB
PARTITION BY RANGE (year(d))
(
PARTITION P1 VALUES
LESS THAN (1999),
PARTITION P2 VALUES
LESS THAN (2000) 42

)
Wednesday, 13 January 2010
 PARTITIONING BY DATE
CREATE TABLE t1 (
d date
) ENGINE=InnoDB
PARTITION BY RANGE (to_days(d))
(
PARTITION P1 VALUES
LESS THAN (to_days('1999-01-01')),
PARTITION P2 VALUES
LESS THAN (to_days('2000-01-01'))
43

)
Wednesday, 13 January 2010
 MySQL 5.1 partitions

WHEN
44

Wednesday, 13 January 2010

 When to use partitions?
• if you have large tables
• if you know that you will always query for the
partitioning column
• if you have historical tables that you want to purge
quickly
• if your indexes are larger than the available RAM

45

Wednesday, 13 January 2010

 MySQL 5.1 partitions

HANDS
ON 46

Wednesday, 13 January 2010

 components used for testing
• MySQL Sandbox
> created MySQL instances in seconds
> http://launchpad.net/mysql-sandbox

• MySQL Employees Test Database

> has ~ 4 mil records in 6 tables
> http://launchpad.net/test-db

• Command line ability

> fingers
> ingenuity
47

Wednesday, 13 January 2010

 employees test database

48

Wednesday, 13 January 2010

 How many partitions
from information_schema.partitions
+-------+-----------------+----------+
| pname | expr | descr |
+-------+-----------------+----------+
| p01 | year(from_date) | 1985 |
| p02 | year(from_date) | 1986 |

...
| p13 | year(from_date) | 1997 |
| p14 | year(from_date) | 1998 |
| p15 | year(from_date) | 1999 |
| p16 | year(from_date) | 2000 |
...
| p19 | year(from_date) | MAXVALUE |
+-------+-----------------+----------+ 49

Wednesday, 13 January 2010

 How many records
select count(*) from salaries;
+----------+
| count(*) |
+----------+
| 2844047 |
+----------+
1 row in set (0.01 sec)

50

Wednesday, 13 January 2010

 How many records in 1998
not partitioned
select count(*) from salaries
where from_date between
'1998-01-01' and '1998-12-31';
+----------+
| count(*) |
+----------+
| 247489 |
+----------+
1 row in set (1.52 sec)

# NOT PARTITIONED 51

Wednesday, 13 January 2010

 How many records in 1998
PARTITIONED
select count(*) from salaries
where from_date between
'1998-01-01' and '1998-12-31';
+----------+
| count(*) |
+----------+
| 247489 |
+----------+
1 row in set (0.41 sec)

# partition p15
52

Wednesday, 13 January 2010

 How many records in 1999
not partitioned
select count(*) from salaries
where from_date between
'1999-01-01' and '1999-12-31';
+----------+
| count(*) |
+----------+
| 260957 |
+----------+
1 row in set (1.54 sec)

# NOT PARTITIONED
53

Wednesday, 13 January 2010

 How many records in 1999
PARTITIONED
select count(*) from salaries
where from_date between
'1999-01-01' and '1999-12-31';
+----------+
| count(*) |
+----------+
| 260957 |
+----------+
1 row in set (0.17 sec)

# partition p16
54

Wednesday, 13 January 2010

 Deleting 1998 records
not partitioned
delete from salaries where
from_date between '1998-01-01'
and '1998-12-31';
Query OK, 247489 rows affected
(19.13 sec)

# NOT PARTITIONED

55

Wednesday, 13 January 2010

 Deleting 1998 records
partitioned
alter table salaries drop
partition p15;
Query OK, 0 rows affected (1.35
sec)

56

Wednesday, 13 January 2010

 MySQL 5.1 partitions

LEVERAGING
REPLICATION
57

Wednesday, 13 January 2010

 Replication schemes
INNODB INNODB
NOT NOT
MASTER PARTITIONED PARTITIONED

concurrent insert SLAVE

concurrent read

INNODB MyISAM
PARTITIONED PARTITIONED
BY RANGE BY RANGE
SLAVE SLAVE
58
concurrent batch processing large batch processing
Wednesday, 13 January 2010
 Replication schemes
concurrent insert
INNODB
PARTITIONED
MASTER

INNODB
NON
MyISAM PARTITIONED
SLAVE PARTITIONED
SLAVE
59
batch processing concurrent reads
Wednesday, 13 January 2010
 MySQL 5.1 partitions

PITFALLS
60

Wednesday, 13 January 2010

 Expressions
• Partition by function
• Search by column

61

Wednesday, 13 January 2010

 WRONG
PARTITION BY RANGE(year(from_date))

select count(*) from salaries where

year(from_date) = 1998;
+----------+
| count(*) |
+----------+
| 247489 |
+----------+
1 row in set (2.25 sec)
62

Wednesday, 13 January 2010

 RIGHT
PARTITION BY RANGE(year(from_date))

select count(*) from salaries where

from_date between '1998-01-01' and
'1998-12-31';
+----------+
| count(*) |
+----------+
| 247489 |
+----------+
1 row in set (0.46 sec)
63

Wednesday, 13 January 2010

 EXPLAIN
explain partitions select count(*)
from salaries where year(from_date)
= 1998\G
***** 1. row ****
id: 1
select_type: SIMPLE
table: salaries
partitions:
p01,p02,p03,p04,p05,p06,p07,p08,p09
,p10,p11,p12,p13,p14,p15,p16,p17,p1
8,p19
type: index 64

...
Wednesday, 13 January 2010
 EXPLAIN
explain partitions select count(*)
from salaries where from_date
between '1998-01-01' and
'1998-12-31'\G
***** 1. row ****
id: 1
select_type: SIMPLE
table: salaries
partitions: p14,p15
...

65

Wednesday, 13 January 2010

 MySQL 5.1 partitions

BENCHMARKS

66

Wednesday, 13 January 2010

 Partitions benchmarks (laptop)

engine storage
(MB)
innodb 221
myisam 181
archive 74
innodb partitioned (whole) 289
innodb partitioned (file per table) 676
myisam partitioned 182
archive partitioned 72

Wednesday, 13 January 2010

 Benchmarking results (laptop)
engine query year query year
2000 2002
InnoDB 1.25 1.25

MyISAM 1.72 1.73

Archive 2.47 2.45

InnoDB partitioned 0.24 0.10

whole
InnoDB Partitioned 0.45 0.10
(file per table)
MyISAM partitioned 0.18 0.12

Archive partitioned 0.22 0.12

Wednesday, 13 January 2010

 Partitioning storage (huge server)

engine storage
(GB)
innodb (with PK) 330
myisam (with PK) 141
archive 13
innodb partitioned (no PK) 237
myisam partitioned (no PK) 107
archive partitioned 13

Wednesday, 13 January 2010

 Benchmarking results (huge server)
engine 6 month range
InnoDB 4 min 30s
MyISAM 25.03s
Archive 22 min 25s
InnoDB partitioned by month 13.19
MyISAM partitioned by year 6.31
MyISAM partitioned by month 4.45
Archive partitioned by year 16.67
Archive partitioned by month 8.97

Wednesday, 13 January 2010

 MySQL 5.1 partitions

TOOLS
71

Wednesday, 13 January 2010

 The partition helper
• The INFORMATION_SCHEMA.PARTITIONS table
• The partition helper
> http://datacharmer.blogspot.com/2008/12/partition-
helper-improving-usability.html
> A Perl script that creates partitioning statements

• ... anything you are creating right now :)

72

Wednesday, 13 January 2010

 MySQL 5.1 partitions

TIPS
73

Wednesday, 13 January 2010

 TIPS
• For large table ( indexes > available RAM)
> DO NOT USE INDEXES
> PARTITIONS ARE MORE EFFICIENT

74

Wednesday, 13 January 2010

 TIPS
• Before adopting partitions in production, benchmark!
• you can create partitions of different sizes
> in historical tables, you can partition
– current data by day
– recent data by month
– distant past data by year
> ALL IN THE SAME TABLE!
• If you want to enforce a constraint on a integer
column, you may set a partition by list, with only the
values you want to admit.
75

Wednesday, 13 January 2010

 TIPS
• For large historical tables, consider using ARCHIVE
+ partitions
> You see benefits for very large amounts of data ( > 500
MB)
> Very effective when you run statistics
> Almost the same performance of MyISAM
> but 1/10 of storage

76

Wednesday, 13 January 2010

 MySQL 5.5 partitions

NEW
FEATURES
77

Wednesday, 13 January 2010

 MySQL 5.5 enhancements
• PARTITION BY RANGE COLUMNS
• PARTITION BY LIST COLUMNS
• TO_SECONDS

78

Wednesday, 13 January 2010

 R E
MySQL 5.5 enhancements
CREATE TABLE t ( FO
dt date
B E 5.1
)
PARTITION BY RANGE (TO_DAYS(dt))
(
PARTITION p01 VALUES LESS THAN
(TO_DAYS('2007-01-01')),
PARTITION p02 VALUES LESS THAN
(TO_DAYS('2008-01-01')),
PARTITION p03 VALUES LESS THAN
(TO_DAYS('2009-01-01')),
PARTITION p04 VALUES LESS THAN
(MAXVALUE));
79

Wednesday, 13 January 2010

 R E
MySQL 5.5 enhancements
FO
SHOW CREATE TABLE t \G B E 5.1
Table: t
Create Table: CREATE TABLE `t` (
`dt` date DEFAULT NULL
) ENGINE=MyISAM DEFAULT
CHARSET=latin1
/*!50100 PARTITION BY RANGE (TO_DAYS
(dt))
(PARTITION p01 VALUES LESS THAN
(733042) ENGINE = MyISAM,
[…]
80

Wednesday, 13 January 2010

 MySQL 5.5 enhancements
E R
CREATE TABLE t (
FT
dt date
A 5.5
)
PARTITION BY RANGE COLUMNS (dt)
(
PARTITION p01 VALUES LESS THAN
('2007-01-01'),
PARTITION p02 VALUES LESS THAN
('2008-01-01'),
PARTITION p03 VALUES LESS THAN
('2009-01-01'),
PARTITION p04 VALUES LESS THAN
(MAXVALUE)); 81

Wednesday, 13 January 2010

 MySQL 5.5 enhancements
E R
FT
SHOW CREATE TABLE t
Table: t A 5.5
Create Table: CREATE TABLE `t` (
`dt` date DEFAULT NULL
) ENGINE=MyISAM DEFAULT
CHARSET=latin1
/*!50500 PARTITION BY RANGE
COLUMNS(dt)
(PARTITION p01 VALUES LESS THAN
('2007-01-01') ENGINE = MyISAM,
[…]
82

Wednesday, 13 January 2010

 MySQL 5.5 - Multiple columns
CREATE TABLE t (
a int,
b int
)PARTITION BY RANGE COLUMNS (a,b)
(
PARTITION p01 VALUES LESS THAN
(10,1),
PARTITION p02 VALUES LESS THAN
(10,10),
PARTITION p03 VALUES LESS THAN
(10,20),
PARTITION p04 VALUES LESS THAN
(MAXVALUE, MAXVALUE)); 83

Wednesday, 13 January 2010

 partition definition
by range (a,b)
records partition LESS THAN
a b p01 10 10
1 10 p02 10 20
10 9 p03 10 30
10 10 p04 10 MAXVALUE
10 11 p05 MAXVALUE MAXVALUE

Wednesday, 13 January 2010

 partition definition
by range (a,b)
records
partition LESS THAN
a b
p01 10 10
1 10
p02 10 20
10 9 p03 10 30
10 10 p04 10 MAXVALUE
10 11 p05 MAXVALUE MAXVALUE

(1,10) < (10,10) ?

(a < 10)
OR
((a = 10) AND (b < 10))
(1 < 10)
OR TRUE
((1 = 10) AND (10 < 10))
Wednesday, 13 January 2010
 partition definition
by range (a,b)
records partition LESS THAN
a b p01 10 10
1 10 p02 10 20
10 9 p03 10 30
10 10 p04 10 MAXVALUE
10 11 p05 MAXVALUE MAXVALUE

(10,9) < (10,10) ?

(a < 10)
OR
((a = 10) AND (b < 10))
(10 < 10)
OR TRUE
((10 = 10) AND (9 < 10))
Wednesday, 13 January 2010
 partition definition
by range (a,b)
records partition LESS THAN
a b p01 10 10
1 10 p02 10 20
10 9 p03 10 30
10 10 p04 10 MAXVALUE
10 11 p05 MAXVALUE MAXVALUE

(10,10) < (10,10) ?

(a < 10)
OR
((a = 10) AND (b < 10))
(10 < 10)
OR FALSE
((10 = 10) AND (10 < 10))
Wednesday, 13 January 2010
 partition definition
by range (a,b)
records partition LESS THAN
a b p01 10 10
1 10 p02 10 20
10 9 p03 10 30
10 10 p04 10 MAXVALUE
10 11 p05 MAXVALUE MAXVALUE

(10,10) < (10,20) ?

(a < 10)
OR
((a = 10) AND (b < 20))
(10 < 10)
OR TRUE
((10 = 10) AND (10 < 20))
Wednesday, 13 January 2010
CREATE TABLE employees (
emp_no int(11) NOT NULL,
birth_date date NOT NULL,
first_name varchar(14) NOT NULL,
last_name varchar(16) NOT NULL,
gender char(1) DEFAULT NULL,
hire_date date NOT NULL
) ENGINE=MyISAM
PARTITION BY RANGE COLUMNS(gender,hire_date)
(PARTITION p01 VALUES LESS THAN ('F','1990-01-01'),
PARTITION p02 VALUES LESS THAN ('F','2000-01-01'),
PARTITION p03 VALUES LESS THAN ('F',MAXVALUE),
PARTITION p04 VALUES LESS THAN ('M','1990-01-01'),
PARTITION p05 VALUES LESS THAN ('M','2000-01-01'),
PARTITION p06 VALUES LESS THAN ('M',MAXVALUE),
PARTITION p07 VALUES LESS THAN (MAXVALUE,MAXVALUE)

89

Wednesday, 13 January 2010

 MySQL 5.5 enhancements
• TRUNCATE PARTITION
• TO_SECONDS()

90

Wednesday, 13 January 2010

 More new features

COMING
SOON
91

Wednesday, 13 January 2010

 PLANNED FEATURES - DISCUSS
• ALTER TABLE t1 EXCHANGE PARTITION p WITH
TABLE t2

• SELECT * FROM T (PARTITION p) WHERE ...

92

Wednesday, 13 January 2010

 Read more
• articles
http://dev.mysql.com

• official docs
http://dev.mysql.com/doc

• my blog
http://datacharmer.blogspot.com
93

Wednesday, 13 January 2010

 Thank you!

Giuseppe Maxia
http://datacharmer.blogspot.com
[email protected]
Wednesday, 13 January 2010
 MySQL 5.1 partitions

BONUS
SLIDES
95

Wednesday, 13 January 2010

 MySQL 5.1 partitions

ANNOYANCES
96

Wednesday, 13 January 2010

 Annoyances with partitions
• CREATE TABLE STATEMENT hard to read

97

Wednesday, 13 January 2010

 Annoyances with partitions
• hard to read (FIXED!! in 5.1.31)

98

Wednesday, 13 January 2010

 Annoyances with partitions
• CREATE TABLE only keeps the result of the
expression for each partition.
• (you can use the information_schema to ease the
pain in this case)

99

Wednesday, 13 January 2010

 Annoyances with partitions
CREATE TABLE t1 (
d date
) ENGINE=InnoDB
PARTITION BY RANGE (to_days(d))
(
PARTITION P1 VALUES
LESS THAN (to_days('1999-01-01')),
PARTITION P2 VALUES
LESS THAN (to_days('2000-01-01'))
100

)
Wednesday, 13 January 2010
 Annoyances with partitions
CREATE TABLE `t1` (
`d` date DEFAULT NULL
) ENGINE=InnoDB DEFAULT
CHARSET=latin1
/*!50100 PARTITION BY RANGE (to_days
(d))
(PARTITION P1 VALUES LESS THAN
(730120) ENGINE = InnoDB,
PARTITION P2 VALUES LESS THAN
(730485) ENGINE = InnoDB) */
101

Wednesday, 13 January 2010

 Fixing annoyances with partitions
select partition_name part,
partition_expression expr,
partition_description val
from information_schema.partitions
where table_name='t1';
+------+------------+--------+
| part | expr | val |
+------+------------+--------+
| P1 | to_days(d) | 730120 |
| P2 | to_days(d) | 730485 |
102
+------+------------+--------+
Wednesday, 13 January 2010
 fixing annoyances with partitions
select partition_name part ,
partition_expression expr, from_days
(partition_description) val from
information_schema.partitions where
table_name='t1';
+------+------------+------------+
| part | expr | val |
+------+------------+------------+
| P1 | to_days(d) | 1999-01-01 |
| P2 | to_days(d) | 2000-01-01 |
+------+------------+------------+
103

Wednesday, 13 January 2010