Sqlite Internals PDF
Sqlite Internals PDF
SELECT * FROM table1;
Translates into:
Open database file containing table1
Rewind the file
while not at endoffile
read all columns out of current record
return the columns to the caller
advance file to the next record
endwhile
close the file
Imagine what this translates into:
SELECT eqptid, enclosureid
FROM eqpt
WHERE typeid IN (
SELECT typeid FROM typespec
WHERE attrid=(
SELECT attrid FROM attribute
WHERE name='detect_autoactuate'
)
AND value=1
INTERSECT
SELECT typeid FROM typespec
WHERE attrid=(
SELECT attrid FROM attribute
WHERE name='algorithm'
)
AND value IN ('sensor','wetbulb')
)
Or This....
SELECT h.url, h.title, f.url,
(SELECT b.parent
FROM moz_bookmarks b
JOIN moz_bookmarks t ON t.id = b.parent AND t.parent != ?1
WHERE b.type = 1 AND b.fk = h.id
ORDER BY b.lastModified DESC LIMIT 1) AS parent,
(SELECT b.title
FROM moz_bookmarks b
JOIN moz_bookmarks t ON t.id = b.parent AND t.parent != ?1
WHERE b.type = 1 AND b.fk = h.id
ORDER BY b.lastModified DESC LIMIT 1) AS bookmark,
(SELECT GROUP_CONCAT(t.title, ',')
FROM moz_bookmarks b
JOIN moz_bookmarks t ON t.id = b.parent AND t.parent = ?1
WHERE b.type = 1 AND b.fk = h.id) AS tags,
h.visit_count, h.typed, h.frecency
FROM moz_places_temp h
LEFT OUTER JOIN moz_favicons f ON f.id = h.favicon_id
WHERE h.frecency <> 0
UNION ALL
SELECT h.url, h.title, f.url,
(SELECT b.parent
FROM moz_bookmarks b
JOIN moz_bookmarks t ON t.id = b.parent AND t.parent != ?1
WHERE b.type = 1 AND b.fk = h.id
ORDER BY b.lastModified DESC LIMIT 1) AS parent,
(SELECT b.title
FROM moz_bookmarks b
JOIN moz_bookmarks t ON t.id = b.parent AND t.parent != ?1
WHERE b.type = 1 AND b.fk = h.id
ORDER BY b.lastModified DESC LIMIT 1) AS bookmark,
(SELECT GROUP_CONCAT(t.title, ',')
FROM moz_bookmarks b
JOIN moz_bookmarks t ON t.id = b.parent AND t.parent = ?1
WHERE b.type = 1 AND b.fk = h.id) AS tags,
h.visit_count, h.typed, h.frecency
FROM moz_places h
LEFT OUTER JOIN moz_favicons f ON f.id = h.favicon_id
WHERE h.id NOT IN (SELECT id FROM moz_places_temp)
AND h.frecency <> 0
ORDER BY 9 DESC
The Whole Point Of SQL...
● A few lines of SQL generates the equivalent of
hundreds or thousands of lines of procedural
code.
● By adding an index, entirely new procedures
are used without recoding.
● The SQL Query Optimizer is tasked with picking
the algorithm
– so that you, the developer, don't have to
Ins and Outs of SQL
sqlite3_prepare_v2() sqlite3_step()
sqlite3_exec(){
sqlite3_prepare_v2();
while( sqlite3_step()!=DONE ){};
sqlite3_finalize();
}
Architecture Of SQLite
Parser
Code Generator
Virtual Machine
B-Tree
Pager
OS Interface
Architecture Of SQLite
Parser
Code Generator
Virtual Machine
B-Tree
Pager
OS Interface
Compile SQL Prep'ed Run the
SQL into a program Stmt program Result
Parser
Code Generator
Virtual Machine
B-Tree
Pager
OS Interface
Compile SQL Prep'ed Run the
SQL into a program Stmt program Result
Parser
Code Generator
Virtual Machine
B-Tree
Pager
OS Interface
Compile SQL Prep'ed Run the
SQL into a program Stmt program Result
Parser
Code Generator
Virtual Machine
B-Tree
● “Storage Engine”
● Fast, transactional, Pager
addr opcode p1 p2 p3 p4 p5 comment
0 Init 0 12 0 00 Start at 12
1 OpenRead 0 2 0 3 00 root=2 iDb=0; tab
2 Explain 0 0 0 SCAN TABLE tab 00
3 Rewind 0 10 0 00
4 Column 0 0 1 00 r[1]=tab.Fruit
5 Ne 2 9 1 (BINARY) 69 if r[2]!=r[1] goto 9
6 Column 0 2 3 00 r[3]=tab.Price
7 RealAffinity 3 0 0 00
8 ResultRow 3 1 0 00 output=r[3]
9 Next 0 4 0 01
10 Close 0 0 0 00
11 Halt 0 0 0 00
12 Transaction 0 0 1 0 01
13 TableLock 0 2 0 tab 00 iDb=0 root=2 write=0
14 String8 0 2 0 Orange 00 r[2]='Orange'
15 Goto 0 1 0 00
EXPLAIN SELECT price FROM tab WHERE fruit=?1
addr opcode p1 p2 p3 p4 p5 comment
0 Init 0 12 0 00 Start at 12
1 OpenRead 0 2 0 3 00 root=2 iDb=0; tab
2 Explain 0 0 0 SCAN TABLE tab 00
3 Rewind 0 10 0 00
4 Column 0 0 1 00 r[1]=tab.Fruit
5 Ne 2 9 1 (BINARY) 69 if r[2]!=r[1] goto 9
6 Column 0 2 3 00 r[3]=tab.Price
7 RealAffinity 3 0 0 00
8 ResultRow 3 1 0 00 output=r[3]
9 Next 0 4 0 01
10 Close 0 0 0 00
11 Halt 0 0 0 00
12 Transaction 0 0 1 0 01
13 TableLock 0 2 0 tab 00 iDb=0 root=2 write=0
14 Variable 1 2 0 ?1 00 r[2]=parameter(1,?1)
15 Goto 0 1 0 00
Architecture Of SQLite
● Ordered key/value Parser
pairs with unique
Code Generator
keys
Virtual Machine
● O(logN) insert, seek,
and delete B-Tree
Code Generator
Virtual Machine
B-Tree
Pager
OS Interface
Architecture Of SQLite
● Atomic commit and Parser
rollback
Code Generator
● Uniform size pages
Virtual Machine
numbered from 1
● No interpretation of B-Tree
Pager
OS Interface
Multiplex Shim
● Inserted in between Parser
Pager and OS
Code Generator
Interface
Virtual Machine
● Looks like one big file
to the Pager B-Tree
OS Interface
ZIPVFS Shim
● Compresses and Parser
encryptions outbound
Code Generator
● Decrypts and
Virtual Machine
decompresses
inbound B-Tree
Pager
ZIPVFS Shim
Multiplex Shim
OS Interface
Logical View of SQL Table Storage
0 to 127
128 to 16383
16384 to 2097151
2097152 to 268435455
268435456 to 34359738367
34359738368 to 4398046511103
4398046511104 to 562949953421311
562949953421312 to 72057594037927935
Less than 0 or greater than 72057594037927935
Root page
Integer key
Some keys appear more than Non-leaf pages hold only keys
once in the tree.
Between 50 and 8000
keys/page depending
on page size.
Integer key
Page 1
Page 2
Page 3
Page 4
Page 5
Page 6
Page 7
To see how pages are used:
● make showdb
● ./showdb database.db pgidx
Available pages: 1..1146
1: root leaf of table [sqlite_master]
2: root interior node of table [blob]
3: root interior node of index [sqlite_autoindex_blob_1]
4: root interior node of table [delta]
5: root interior node of table [rcvfrom]
6: root leaf of index [sqlite_autoindex_rcvfrom_1]
7: root leaf of table [config]
8: root leaf of index [sqlite_autoindex_config_1]
9: root leaf of table [shun]
10: root leaf of index [sqlite_autoindex_shun_1]
11: root leaf of table [private]
...
264: leaf of table [blob], child 201 of page 2
265: leaf of table [blob], child 202 of page 2
266: overflow 1 from cell 0 of page 268
267: overflow 2 from cell 0 of page 268
268: leaf of table [blob], child 203 of page 2
...
Overflow
Integer key
Binary content
Implications Of Overflow
● Move small and
common fields near
the beginning.
● Put larger and
infrequently accessed
fields at the end.
● Move large BLOBs to
a separate table and
access via join.
Surprising Attributes of Overflow
● Multi-megabyte
BLOBs and strings
work well.
● Faster to store
BLOBs in the
database than directly
on disk for sizes up to
15-150K.
sqlite.org/intern-v-extern-blob.html
Logical View of SQL Index Storage
Record Format
B-Tree
Pager
OS Interface
Key Data
SELECT price FROM tab WHERE fruit='Peach'
1:1
idx2 tab
state rowid rowid fruit state price
CA 5 1 Orange FL 0.85
CA 23 2 Apple NC 0.45
FL 1 4 Peach SC 0.60
FL 18 5 Grape CA 0.80
NC 2 18 Lemon FL 1.25
NC 19 19 Strawberry NC 2.45
SC 4 23 Orange CA 1.05
idx2 tab
state rowid rowid fruit state price
CA 5 1 Orange FL 0.85
CA 23 2 Apple NC 0.45
FL 1 4 Peach SC 0.60
FL 18 5 Grape CA 0.80
NC 2 18 Lemon FL 1.25
NC 19 19 Strawberry NC 2.45
SC 4 23 Orange CA 1.05
CREATE INDEX idx3 ON tab(fruit, state)
idx3 tab
fruit state rowid rowid fruit state price
Apple NC 2 1 Orange FL 0.85
Grape CA 5 2 Apple NC 0.45
Lemon FL 18 4 Peach SC 0.60
Orange CA 23 5 Grape CA 0.80
Orange FL 1 18 Lemon FL 1.25
Peach SC 4 19 Strawberry NC 2.45
Strawberry NC 19 23 Orange CA 1.05
idx5 tab
fruit state price rowid rowid fruit state price
Apple NC 0.45 2 1 Orange FL 0.85
Grape CA 0.80 5 2 Apple NC 0.45
Lemon FL 1.25 18 4 Peach SC 0.60
Orange CA 1.05 23 5 Grape CA 0.80
Orange FL 0.85 1 18 Lemon FL 1.25
Peach SC 0.60 4 19 Strawberry NC 2.45
Strawberry NC 2.45 19 23 Orange CA 1.05
idx2 tab
state rowid rowid fruit state price
CA 5 1 Orange FL 0.85
CA 23 2 Apple NC 0.45
FL 1 4 Peach SC 0.60
FL 18 5 Grape CA 0.80
NC 2 18 Lemon FL 1.25
NC 19 19 Strawberry NC 2.45
SC 4 23 Orange CA 1.05
4
SELECT price FROM tab
WHERE fruit='Orange'
AND state='CA'
fruit rowid
Apple 2
Grape 5
Lemon 18
Orange 1 rowid fruit state price
Orange 23 1 Orange FL 0.85
Peach 4 2 Apple NC 0.45
Strawberry 19 4 Peach SC 0.60
UNION
5 Grape CA 0.80
18 Lemon FL 1.25
state rowid 19 Strawberry NC 2.45
CA 5 23 Orange CA 1.05
CA 23
FL 1
FL 18
NC 2
NC 19
SC 4
CREATE INDEX idx4 ON tab(state,fruit)
idx4 tab
state fruit rowid rowid fruit state price
CA Grape 5 1 Orange FL 0.85
CA Orange 23 2 Apple NC 0.45
FL Orange 1 4 Peach SC 0.60
FL Lemon 18 5 Grape CA 0.80
NC Apple 2 18 Lemon FL 1.25
NC Strawberry 19 19 Strawberry NC 2.45
SC Peach 4 23 Orange CA 1.05
idx4 tab
state fruit rowid rowid fruit state price
CA Grape 5 1 Orange FL 0.85
CA Orange 23 2 Apple NC 0.45
FL Orange 1 4 Peach SC 0.60
FL Lemon 18 5 Grape CA 0.80
NC Apple 2 18 Lemon FL 1.25
NC Strawberry 19 19 Strawberry NC 2.45
SC Peach 4 23 Orange CA 1.05
SELECT * FROM tab ORDER BY fruit
sorter
5 Grape CA 0.80
18 Lemon FL 1.25
19 Strawberry NC 2.45
23 Orange CA 1.05
SELECT * FROM tab ORDER BY rowid
sorter
5 Grape CA 0.80
18 Lemon FL 1.25
19 Strawberry NC 2.45
23 Orange CA 1.05
SELECT * FROM tab ORDER BY fruit DESC
sorter
5 Grape CA 0.80
18 Lemon FL 1.25
19 Strawberry NC 2.45
23 Orange CA 1.05
Key Data
CREATE TABLE tab(
Fruit TEXT,
State TEXT,
Price REAL,
PRIMARY KEY(Fruit, State)
);
With Rowid:
Without Rowid:
fruit state price
Apple NC 0.45
Grape CA 0.80
Lemon FL 1.25
Orange CA 1.05
Orange FL 0.85
Peach SC 0.60
Strawberry NC 2.45
CREATE TABLE tab(
Fruit TEXT,
State TEXT,
Price REAL,
Amt INT,
PRIMARY KEY(Fruit, State)
) WITHOUT ROWID;
CREATE INDEX tab_amt ON tab(amt);