Lecture 17

Ch13: Data Storage Structures

Database Systems
Zubaira Naz
Information Technology University (ITU)
 File Organization
• The database is stored as a collection of files.
• Each file is a sequence of records.
• A record is a sequence of fields.
• One approach:
–assume record size is fixed
–each file has records of one particular type only
–different files are used for different relations
This case is easiest to implement; will consider
variable length records later.
 Fixed-Length Records
• Simple approach:
– Store record i starting from byte n * (i – 1), where n is the size
of each record.
– Record access is simple but records may cross blocks
• Modification: do not allow records to cross block boundaries
 Fixed-Length Records
• Deletion of record i: alternatives:
– move records i + 1, . . ., n to i, . . . , n – 1
– move record n to i
– do not move records, but link all free records on a free list
Record 3 deleted
 Fixed-Length Records
• Deletion of record i: alternatives:
– move records i + 1, . . ., n to i, . . . , n – 1
– move record n to i
– do not move records, but link all free records on a free list
Record 3 deleted and replaced by record 11
 Fixed-Length Records
• Deletion of record i: alternatives:
– move records i + 1, . . ., n to i, . . . , n – 1
– move record n to i
– do not move records, but link all free records on a free list
 Variable-Length Records
• Variable-length records arise in database systems in several ways:
– Storage of multiple record types in a file.
– Record types that allow variable lengths for one or more fields such as
strings (varchar)
– Record types that allow repeating fields (used in some older data
models).
• Attributes are stored in order
• Variable length attributes represented by fixed size (offset, length),
with actual data stored after all fixed length attributes
• Null values represented by null-value bitmap
Variable-Length Records: Slotted Page Structure
Block Header Records

Size # Entries Free Space

Location

End of Free Space

• Slotted page header contains:

– number of record entries
– end of free space in the block
– location and size of each record
• Records can be moved around within a page to keep them
contiguous with no empty space between them; entry in the header
must be updated.
• Pointers should not point directly to record — instead they should
point to the entry for the record in header.
 Storing Large Objects
• E.g., blob/clob types
• Records must be smaller than pages
• Alternatives:
– Store as files in file systems
– Store as files managed by database
– Break into pieces and store in multiple tuples in
separate relation
• PostgreSQL TOAST
Organization of Records in Files
• Heap – record can be placed anywhere in the file where
there is space
• Sequential – store records in sequential order, based on
the value of the search key of each record
• In a multitable clustering file organization records of
several different relations can be stored in the same file
• B+-tree file organization
– Ordered storage even with inserts/deletes
– it supports very efficient access to specific records, based on the
search key.
• Hashing – a hash function computed on search key; the
result specifies in which block of the file the record should
be placed
 Heap File Organization
• Records can be placed anywhere in the file where there is free space
• Records usually do not move once allocated
• Important to be able to efficiently find free space within file
• Free-space map
– Array with 1 entry per block. Each entry is a few bits to a byte, and
records fraction of block that is free
– In example below of 16 blocks, 3 bits per block, value divided by 8
indicates fraction of block that is free

– Can have second-level free-space map

– In example below, each entry stores maximum from 4 entries of first-
level free-space map

• Free space map written to disk periodically, OK to have wrong (old)

values for some entries (will be detected and fixed)
 Sequential File Organization
• Suitable for applications that require sequential processing
of the entire file
• The records in the file are ordered by a search-key
10101 Srinivasan Comp. Sci. 65000
12121 Wu Finance 90000
15151 Mozart Music 40000
22222 Einstein Physics 95000
32343 El Said History 60000
33456 Gold Physics 87000
45565 Katz Comp. Sci. 75000
58583 Califieri History 62000
76543 Singh Finance 80000
76766 Crick Biology 72000
83821 Brandt Comp. Sci. 92000
98345 Kim Elec. Eng. 80000
Sequential File Organization (Cont.)
• Deletion – use pointer chains
• Insertion –locate the position where the record is to be inserted
– if there is free space insert there
– if no free space, insert the record in an overflow block
– In either case, pointer chain must be updated
• Need to reorganize the file 10101 Srinivasan Comp. Sci. 65000
from time to time to restore 12121 Wu Finance 90000
15151 Mozart Music 40000
sequential order
22222 Einstein Physics 95000
32343 El Said History 60000
33456 Gold Physics 87000
45565 Katz Comp. Sci. 75000
58583 Califieri History 62000
76543 Singh Finance 80000
76766 Crick Biology 72000
83821 Brandt Comp. Sci. 92000
98345 Kim Elec. Eng. 80000

32222 Verdi Music 48000

Multitable Clustering File Organization
Store several relations in one file using a multitable clustering
file organization

department

instructor

multitable clustering
of department and
instructor
Multitable Clustering File Organization (cont.)

• good for queries involving department ⨝

instructor, and for queries involving one single
department and its instructors
• bad for queries involving only department
• results in variable size records
• Can add pointer chains to link records of a
particular relation
 Partitioning
• Table partitioning: Records in a relation can be
partitioned into smaller relations that are stored separately
• E.g., transaction relation may be partitioned into
transaction_2018, transaction_2019, etc.
• Queries written on transaction must access records in all
partitions
– Unless query has a selection such as year=2019, in which case
only one partition in needed
• Partitioning
– Reduces costs of some operations such as free space management
– Allows different partitions to be stored on different storage
devices
• E.g., transaction partition for current year on SSD, for older years on
magnetic disk
 Data Dictionary Storage
The Data dictionary (also called system catalog) stores
metadata; that is, data about data, such as

• Information about relations

– names of relations
– names, types and lengths of attributes of each relation
– names and definitions of views
– integrity constraints
• User and accounting information, including passwords
• Statistical and descriptive data
– number of tuples in each relation
• Physical file organization information
– How relation is stored (sequential/hash/…)
– Physical location of relation
• Information about indices (Chapter 14)
 Relational Representation of System Metadata

• Relational Relation_metadata A!ribute_metadata

relation_name relation_name
representation on number_of_a!ributes a!ribute_name
storage_organization domain_type
disk location position
length
• Specialized data
Index_metadata
structures designed index_name
relation_name
for efficient access, index_type
User_metadata
index_a!ributes
in memory user_name
encrypted_password
group
View_metadata
view_name
definition
 Storage Access
• Blocks are units of both storage allocation and data transfer.
• Database system seeks to minimize the number of block transfers
between the disk and memory. We can reduce the number of disk
accesses by keeping as many blocks as possible in main memory.
• Buffer – portion of main memory available to store copies of disk
blocks.
• Buffer manager – subsystem responsible for allocating buffer space
in main memory.
 Buffer Manager
• Programs call on the buffer manager when they need a
block from disk.
– If the block is already in the buffer, buffer manager returns the
address of the block in main memory
– If the block is not in the buffer, the buffer manager
• Allocates space in the buffer for the block
– Replacing (throwing out) some other block, if required, to make space
for the new block.
– Replaced block written back to disk only if it was modified since the
most recent time that it was written to/fetched from the disk.
• Reads the block from the disk to the buffer, and returns the address of
the block in main memory to requester.
 Buffer Manager
• Buffer replacement strategy (details coming up!)
• Pinned block: memory block that is not allowed to be written back to
disk
– Pin done before reading/writing data from a block
– Unpin done when read /write is complete
– Multiple concurrent pin/unpin operations possible
• Keep a pin count, buffer block can be evicted only if pin count = 0
• Shared and exclusive locks on buffer
– Needed to prevent concurrent operations from reading page contents as
they are moved/reorganized, and to ensure only one move/reorganize at a
time
– Readers get shared lock, updates to a block require exclusive lock
– Locking rules:
• Only one process can get exclusive lock at a time
• Shared lock cannot be concurrently with exclusive lock
• Multiple processes may be given shared lock concurrently
 Buffer-Replacement Policies
• Most operating systems replace the block least recently used (LRU
strategy)
– Idea behind LRU – use past pattern of block references as a predictor of
future references
– LRU can be bad for some queries
• Queries have well-defined access patterns (such as sequential scans),
and a database system can use the information in a user’s query to
predict future references
• Toss-immediate strategy – frees the space occupied by a block as soon
as the final tuple of that block has been processed
• Most recently used (MRU) strategy – system must pin the block
currently being processed. After the final tuple of that block has been
processed, the block is unpinned, and it becomes the most recently
used block.
Optimization of Disk Block Access (Cont.)
• Buffer managers support forced output of blocks for the purpose of
recovery (more in Chapter 19)
• Nonvolatile write buffers speed up disk writes by writing blocks to
a non-volatile RAM or flash buffer immediately
– Writes can be reordered to minimize disk arm movement
• Log disk – a disk devoted to writing a sequential log of block
updates
– Used exactly like nonvolatile RAM
• Write to log disk is very fast since no seeks are required
• Journaling file systems write data in-order to NV-RAM or log disk
– Reordering without journaling: risk of corruption of file system
data
 Assignment
1. Explain the concept of data storage structures and their role in database management
systems.
2. Describe the difference between logical and physical storage structures in a database.
3. What challenges arise in maintaining data storage structures as database size grows?
4. Compare and contrast heap files and indexed files in terms of access efficiency.
5. What are the advantages of using sequential files for record storage?
6. How does hashing improve data retrieval in a file organization system?
7. What is the difference between fixed-length and variable-length records?
8. Explain how slotted pages help manage record storage and access in a database.
9. Discuss the impact of record fragmentation on file organization and performance.
10. What is a data dictionary, and why is it essential in a database system?
11. List and describe three types of metadata typically stored in a data dictionary.
12. How does a data dictionary contribute to query optimization in a database?
13. Explain the role of the database buffer in reducing disk I/O operations.
14. What is the least recently used (LRU) replacement policy, and how is it applied in database
buffers?
15. Describe the concept of write-ahead logging (WAL) and its importance for data
consistency.