Database Design and Management

UNIT 2

DATA STRUCTURES, FILE ORGANISATION AND PHYSICAL DATABASE DESIGN

Structure
2.0 2.1 2.2 Objectives Introduction Definitions and Basic Concepts
2.2.1 2.2.2 2.2.3 2.2.4 2.2.5 Why Data Structures Memory Hierarchy RAID Technology Indexes Binary Search

2.3

Data Structures
2.3.1 2.3.2 2.3.3 Linked Lists Inverted Lists B-Trees

2.4

Files and their Organisations

2.4.1 2.4.2 2.4.3 2.4.4 File Storage Concepts Sequential Access Method (SAM) Indexed Sequential Access Method (ISAM) Direct Access Method (DAM)

2.5 2.6 2.7 2.8 2.9

Physical Database Design Summary Answers to Self Check Exercises Keywords References and Further Reading

2.0 OBJECTIVES
After reading this Unit, you will be able to: understand the basic concepts related to data structures and file organisation; comprehend physical storage structures of data and file organisation techniques; and gain an insight into the role the data structures and file organisation play in the overall performance and access efficiency in a database.

2.1 INTRODUCTION
Data structures and file organisation refer to the methods of organising the data in a database. They primarily deal with physical storage of data, which assumes significance in retrieving, storing and re-organising data in a database. Data structures include linked

22

lists, inverted lists, B-trees and hash tables, among others. Data structures can be used to build data files (a data file or a file is a collection of many similar records) and file organisation determines access methods for the file. File organisation (or file structure) is a combination of representations for data in files and of operations for accessing the data. A file structure allows applications to read, write, and modify data. It might also support finding the data that matches some search criteria or reading through the data in some particular order. Data structures and file organisation define the physical design of a database and are critical to its performance.

Data Structures, File Organisation and Physical Database Design

2.2 DEFINITIONS AND BASIC CONCEPTS

Some of the basic concepts related to data structures and file organisation which are essential to provide the necessary background for their understanding are given in the following paragraphs:

2.2.1 Why Data Structures

The key factor in designing data structures and file organisation is the relatively slow speed of hard disks and large amount of time that is required to get information from a disk. All the data structures and file organisation designs focus on minimising disk accesses and maximising the likelihood that the information the user will want is already in the memory. The constraint related to disk access is generally referred to as I/O bottleneck. Accessing information using multiple trips to the disk greatly slows down the access time. Ideally, we should get the information we need with one access to the disk or with as few accesses as possible. Tracking the developments of data structures and file organisation over the years, one finds that early work on files presumed that the files were on tape and access sequential. The cost of sequential access grew in direct proportion to the size of the file. As files grew intolerably large for unaided sequential access, indexes were added to the files. The indexes made it possible to keep a list of keys and pointers in a small file that could be searched more quickly. With the keys and pointers the user had direct access to the large, primary file. However, as the indexes grew, they too became difficult to manage, especially for dynamic files in which the set of keys changes. Then, in the early 1960s the idea of applying tree structures emerged which was refined over the years to provide a solution in the form of the B-tree. Furthermore, hashing technique was developed to provide fast access to files.

2.2.2 Memory Hierarchy

Computer storage media form a memory hierarchy that includes two main categories of storage: Primary storage: Pertains to storage media used by Central Processing Unit (CPU) i.e., the main memory and also the cache memory. The primary storage memory also called RAM (Random Access Memory) provides fast access to data and is volatile i.e., loses its content in case of a power outage. Secondary storage: Includes magnetic disks, optical disks and tapes. Secondary storage memory provides slower access to data than RAM. The memory hierarchy is represented in Fig. 2.1. As one moves down the hierarchy from cache memory, access speed and cost decrease.

23

Database Design and Management

Cache Memory (SRAM)

Main Memory (DRAM)

Flash Memory (EEPROM)

Magnetic Disks (Hard disks & floppies)

Optical Disks (CD-ROM, WORM, CD-R, DVD) Magnetic Tapes

Fig. 2.1: Memory Hierarchy

Static RAM (SRAM) which is cache memory is used by CPU to speed up execution of programmes while Dynamic RAM (DRAM) provides the main work area for CPU. Flash memory which is non-volatile and called EEPROM (Electrically Erasable Programmable Read-Only-Memory) has access speed and performance between DRAM and magnetic disks. CD-ROM (Compact Disk Read-Only-Memory) disks store data optically and are read by a laser. WORM (Write-Once-Read-Memory) disks are used for archiving data and allow data to be written once and read any number of times. DVD (Digital Video Disk) a type of optical disk allows storage of four to fifteen gigabytes of data per disk. Magnetic tapes are used for archiving and back-up storage and are becoming popular as tertiary storage to hold terabytes of data. Juke boxes (optical and tape) are employed to use arrays of CD-ROMs and tapes.

2.2.3 RAID Technology

A major advance in secondary storage technology is represented by RAID (Redundant Array of Inexpensive/Independent Disks) technology. The RAID idea has been developed into an elaborate set of alternative RAID architectures (RAID levels 0 through 6). The main goal of RAID is to even out the widely different rates of performance improvement of disks against those in memory and microprocessors. While RAM capacities have quadrupled every two to three years, disk access times are improving at less than 10 percent per year, and disk transfer rates are improving at roughly 20 percent per year. Though disk capacities are improving at a fast rate, the speed and access time improvements are of much smaller magnitude. The problem of speed and access time is overcome by using a large array of small independent disks acting as a single high-performance logical disk. A concept called data striping is used, which utilises parallelism to improve disk performance. Disk striping exemplifies an important concept that we see more and more in system configuration parallelism. Whenever there is a bottleneck at some point in the system, consider duplicating the source of the bottleneck and configure the system so that several of them operate in parallel. Data striping distributes data transparently over multiple disks to make them appear as a single large, fast disk. Striping improves overall I/O performance by allowing

24

multiple I/Os to be serviced in parallel, thus providing high overall transfer rates. Data striping also accomplishes load balancing among disks. It should be noted that data can be read or written only one block at a time, so a typical transfer contains 512 bytes (block size = 512 bytes). Data striping can be applied at a finer granularity by breaking up a byte of data into bits and spreading the bits to multiple disks. Using bit-level data striping with 8-bit bytes, eight physical disks may be considered as one logical disk with an eight fold increase in data transfer rate. Each disk participates in each I/O request and the total data read per request is eight times. Data striping may also be done at block level which distributes blocks of a file across disks. In addition to improving performance, RAID is also used to improve reliability by storing redundant information on disks. One technique for introducing redundancy is called mirroring. Data is written redundantly to two identical physical disks that are treated as one logical disk. If a disk fails, the other is used until the first is repaired. Thus, RAID technology has contributed significantly in improving the performance and reliability of data storage on disks.

Data Structures, File Organisation and Physical Database Design

2.2.4 Indexes
An index is a file in which each entry (record) consists of a data value together with one or more pointers (physical storage addresses). The data value is a value for some field of the indexed file (the indexed field) and pointers identify records in the indexed file having that value for that field. The concept of indexing is closely linked with the operation of searching. An index (sometimes also referred as a list) can be used in two ways. First, it can be used for sequential access to the indexed file, i.e., access according to the values of the indexed field by imposing an ordering of the indexed file. Second, it can also be used for direct access to individual records in the indexed file on the basis of a given value for that same field. In general, indexing speeds up retrieval but may slow down update. Address/Pointer
LastName Pointer A11 A22 A32 A42 A47 A58 A63 A67 ID LastName FirstName DateJoined

Chauhan Dhote Farrukh Ganguly Hopkin Janardhan Ramachandran Ray

A 67 A 58 A 63 A 22 A 42 A 47 A 32 A 11

1 2 3 4 5 6 7 8

Ray Ganguly Ramachandran Hopkin Janardan Dhote Farrukh Chauhan

Suchanda Akhilesh Muthu Bill Jana Anil Sheikh Rakesh

1/29/96 3/31/96 2/17/96 2/8/96 1/6/96 8/23/96 3/28/96 12/29/96

Index File Fig. 2.2: An Illustration of Indexes

Indexed File

2.2.5 Binary Search

Binary searching is a technique used to substantially lessen the time required to search the indexes of lengthy inverted lists (see 2.3.2). In this technique, the value sought is first compared to the value in the middle of the list. This indicates whether the value sought is
25

Database Design and Management

in the top or bottom half of the list. The value sought is then compared with the middle entry of the appropriate half. This indicates which fourth the value is in. Then the value is compared to the middle of the fourth, and so on until the desired value is found. Thus the binary search keeps splitting the data set in half until it finds the desired value. An example of binary search is shown in Fig. 2.3. To find the entry for Janardan find the middle of the list (Gautam). Janardan is post Gautam so split the second half in half (Kamla). Keep splitting the remainder in half until Janardan is found.

Alexander Bhatnagar Chand Dhani Ejaz Feroze Gautam Hegde Ipshita 4 Janardan

2 Kamla

Fig. 2.3: An Example of Binary Search

Self Check Exercise

1) What is the role of indexes in database search? Write your answer in the space given below. Check your answer with the answers given at the end of the Unit. .................................................................................................................... .................................................................................................................... .................................................................................................................... .................................................................................................................... .................................................................................................................... ....................................................................................................................

Note: i) ii)

2.3 DATA STRUCTURES

The term Data Structure refers to the manner in which relationships between data elements are represented in the computer system. Organisation of indexes, representation of stored fields, physical sequence of stored records, etc., are included in the purview of data structures. Thus, an understanding of data structures is important in gaining an understanding of database management systems.
26

There are three major types of data structures : linked lists (indexes), inverted lists (indexes) and B-trees. These data structures have been explained in the following paragraphs:

2.3.1 Linked Lists

A simple linked list is a chain of pointers embedded in records. It indicates either a record sequence for an attribute other than the primary key or all the records with a common property. With a linked list, any data element can be stored separately. A pointer is then used to link to the next data item. Fig. 2.4 illustrates the basic concept of linked lists. In this example each row of data is stored separately. Then an index is created on the field (key) Last Name. However, each element of the index is stored separately. An index element consists of three parts: the key value, a pointer to the rest of the data of that row, and a pointer to the next index element. To retrieve data sequentially, start at the first element (Chauhan) and follow the link (pointer) to the next element (Dhote). Each element of the index is found by following the link to the next element. The data pointer in each index element provides the entire data row for that key value. The strength of a linked list lies in its ability to easily and rapidly insert and delete data. A67

Data Structures, File Organisation and Physical Database Design

A 22 B 87

Chauhan

B29
A 58

B 29

Dhote

B71
A 63

B 71

B 38

Fig. 2.4: An Illustration of Linked List

2.3.2 Inverted Lists

Inverted lists may be viewed simply as index tables of pointers stored separately from the data records rather than embedded in pointer fields in the stored records themselves. Distinction should be made between nondense and dense lists. In case of a nondense list only a few of the records in the file are part of the list while a dense list is one with a pointer for most or all of the records in the file. Processing for unique secondary keys (those having 1:1 association with primary keys) is somewhat different than those with 1:M associations with primary keys. In the former case, dense indexes are generated while the later gives nondense indexes.

27

Database Design and Management

Examples of inverted lists are given below: List 1 List 2

Company Digital Ford GM Intel Lockheed

Area Computer Auto Auto Computer Aerospace

Index File

Primary Key 1245 1175 1323 1231 1152

Company Symbol DEC F GM INTL L

Indexed File

Primary Key 1245 1175 1323 1231 1152

Fig. 2.5: Dense Inverted Lists

The above lists are dense since there is one-to-one relationship between both company name and primary key and company symbol and primary key. Fig. 2.6 gives an example of non-dense inverted list for area (relationship between area and primary key is one-to-many). Area Aerospace Auto Computer Primary Key 1152 1175, 1323 1231, 1245

Fig. 2.6: Nondense Inverted List

The lists are said to be inverted because company names (or area names) have been alphabetized and the corresponding primary keys have been inverted or rearranged accordingly.

2.3.3

B-Trees

B-trees are a form of data structure based on hierarchies. Some authors claim that the letter B stands for Bayer, the originator while others say it stands for balanced. Btrees are balanced in the sense that all the terminal (bottom) nodes have the same path length to the root (top). Algorithms have been developed for efficiently searching and maintaining B-tree indexes, which have become quite popular for representing both primary and secondary indexes. B-trees provide both sequential and indexed access and are quite flexible. The height of a B-tree is the number of levels in the hierarchy. Each node on the tree contains an index element which has a key value, a pointer to the rest of the data and two link pointers (see Fig.2.7) One link (to the left) points to the elements (nodes) that have lower values while the other link (to the right) points to elements that have a value greater than or equal to the value in the node. The root is the highest node on the tree. The bottom nodes are called leaves because they are at the end of the tree branches.

28

<

Key

Data

Data Structures, File Organisation and Physical Database Design

Dhol Balesh Ashok A B Chauhan E C D

H I

Fig. 2.7: An Illustration of a B-tree

A B-tree is called unbalanced if the terminal nodes (leaves) are not all at the same level i.e., if different terminal nodes are at different depths below the top node. B-trees provide excellent access performance but do not allow a file to be accessed sequentially with efficiency. This problem is overcome by adding a linked list structure at the bottom level of the B-tree. The combination of a B-tree and a sequential linked list is called a B+ tree.

Self Check Exercise

2) Give examples of commercially available database management systems, which use B-trees/ B+ trees. Write your answer in the space given below. Check your answer with the answers given at the end of the Unit. .................................................................................................................... .................................................................................................................... .................................................................................................................... .................................................................................................................... .................................................................................................................... ....................................................................................................................

Note: i) ii)

2.4 FILES AND THEIR ORGANISATIONS

2.4.1 File Storage Concepts
A file is a sequence of records. File organisation refers to physical layout or structure of record occurrences in a file. File organisation determines the way records are stored and accessed. In many cases, all records in a file are of the same record type. If every record in the file has exactly the same size (in bytes), the file is said to be made of fixed-length records. If
29

Database Design and Management

different records in the file have different sizes, the file is said to be made up variablelength records. A file may have variable-length records for several reasons: i) ii) The file records are of the same record type, but one or more fields are of varying sizes (variable-length fields). The file records are of the same record type but one or more fields may have multiple values for individual records. Such a field is called repeating field and a group of values for the field is often called a repeating group. The file records are of the same record type, but one or more fields are optional. The file has records of different record types and hence of varying size (mixed file). This would occur if related records of different types were clustered (placed together) on disk blocks.

iii) iv)

The records of a file must be allocated to disk blocks because a block is a unit of data transfer between disk and memory. The division of a track (on storage medium) into equal sized disk blocks is set by the operating system during disk formatting. The hardware address of a block comprises a surface number, track number and block number. Buffer a contiguous reserved area in main storage that holds one block has also an address. For a read command, the block from disk is copied into the buffer, whereas for a write command the contents of the buffer are copied into the disk block. Sometimes several contiguous blocks, called a cluster, may be transferred as a unit. In such cases buffer size is adjusted to cluster size. When the block size is larger than the record size each block will contain numerous records, while there can be files with large records that cannot fit in one block. In the latter case the records can span more than one block. Here it is worthwhile to note the difference between the terms File Organisation and Access Method. A file organisation refers to the organisation of the data of a file into records, blocks and access structures; this includes the way the records and blocks are placed on the storage medium and interlinked. An access method on the other hand, provides a group of operations such as find, read, modify, delete, etc, that can be applied to a file. In general, it is possible to apply several access methods to a file organisation. Some access methods, though, can be applied only to files organised in certain ways. For example, we cannot apply an indexed access method to a file without an index.

2.4.2 Sequential Access Method (SAM)

In sequential files, records are stored in a predefined order. Their occurrences in a sequential file are usually sorted on the primary key and physically arranged on the storage medium in order by primary key. If only sequential access is required (which is rarely the case), sequential media (magnetic tapes) are suitable and probably the most cost-effective way of processing such files. Direct access devices such as disks may be, but are not necessarily, referenced sequentially. Some types of processing are best done through sequential access, even when direct access devices are used. Sequential access is fast and efficient while dealing with large volumes of data that need to be processed periodically. However, it requires that all new transactions be sorted into proper sequence for sequential access processing. Also, most of the database or file may have to be searched to locate, store, or modify even a small number of data records. Thus, this method is too slow to handle applications requiring immediate updating or responses. Sequential files are generally used for backup or transporting data to a different system. A sequential ASCII file is a popular export/import format that most database systems support.

30

2.4.3 Indexed Sequential Access Method (ISAM)

In indexed sequential files, record occurrences are sorted and stored in order by primary key on a direct access storage device. In addition, a separate table (or file) called an index is maintained on primary key values to give the physical address of each record occurrence. This approach gives (almost) direct access to record occurrences via the index table and sequential access via the way in which the records are laid out on the storage medium. The physical address of a record given by the index file is also called a pointer. The pointer or address can take many forms depending on the operating system and the database one is using. Nowadays, systems use virtual addresses instead of physical addresses. A virtual address could be based on imaginary disk drive layout. The database refers to a base set of tracks and cylinders. The computer then maps these values into actual storage locations. This arrangement is the basis for an approach known as the virtual sequential access method (VSAM). Another common approach is to define a location in terms of its distance from the start of a file (relative address). Virtual or relative addresses are always better than the physical address because of their portability. In case a few records need to be processed quickly, the index is used to directly access the records needed. However, when large numbers of records must be processed periodically, the sequential organisation provided by this method is used. An illustration of access using index file is given in Fig. 2.8. Address Data/Record Data file

Data Structures, File Organisation and Physical Database Design

Key Value

Address/Pointer

Index file

Fig. 2.8: Data Access Using Index File

2.4.4 Direct Access Method (DAM)

When using direct access method, the record occurrences in a file do not have to be arranged in any particular sequence on storage media. However, the computer must keep track of the storage location of each record using a variety of direct organisation methods so that data is retrieved when needed. New transactions data do not have to be sorted, and processing that requires immediate responses or updating is easily handled. In direct access method an algorithm is used to compute the address of a record. The primary key value is the input to the algorithm and the block address of the record is the output. To implement the approach, a portion of the storage space is reserved for the file. This space should be large enough to hold the file plus some allowance for growth. Then the algorithm that generates the appropriate address for a given primary key is devised. The algorithm is commonly called hashing algorithm. The process of converting primary key values into addresses is called key-to-address transformation. More than one logical record usually fits into a block, so we may think of the reserved storage area as being broken into record slots sequentially numbered from 1 to n. These sequential numbers are called relative pointers or relative addresses, because they indicate the position of the record relative to the beginning of the file.

31

Database Design and Management

The objective of the hashing algorithm is to generate relative addresses that disperse the records throughout the reserved storage space in a random but uniform manner. The records can be retrieved very rapidly because the address is computed rather than found through table look-up via indexes stored on a disk file. A collision is said to occur if more than one record maps to the same block. Because one block usually holds several records, collisions are only a problem when the number of records mapping to a block exceeds the blocks capacity. To account for this event, most direct access methods support overflow area for collisions, which is searched sequentially. The hashed key approach is extremely fast since the keys value is immediately converted into a storage location, and data can be retrieved in one pass to the disk. An illustration of direct access method using hashed key is given in Fig. 2.9.

Key Value: 1120

Hashing Algo

Address location for k

A Address location for k

Reserved Overflow A

Fig. 2.9: Direct Access Using Hashed-Key Approach

2.5 PHYSICAL DATABASE DESIGN

Data structures and file organisation play an important role in physical database design (please refer to Unit 3 on Database Management Systems for database architecture and database design issues). The goal of the physical design is to come up with appropriate structuring of data in storage so as to ensure good performance of the database. It is not possible to make meaningful physical design decisions and performance analyses until we know the queries, transactions and applications that are expected to run on the database. The following aspects influence the physical database design: 1) Analysing the database queries and transactions: For each query we should specify: a) b) c)
32

The files that will be accessed by the query. The attributes on which any selection conditions for the query are specified. The attributes on which any join conditions or conditions to link multiple tables for the query are specified. The attributes whose values will be retrieved by the query.

d)

The attributes at b) and c) are candidates for definition of access structures. For each update transaction we should specify: a) b) c) d) The files that will be updated. The type of operation on each file (insert, update or delete). The attributes on which the selection conditions for a delete or update are specified. The attributes whose value will be changed by the update operation.

Data Structures, File Organisation and Physical Database Design

Here the attributes at c) are candidates for access structures and attributes at d) are candidates for avoiding an access structure since modifying them will require updating the access structures. 2) Analysing the expected frequency of queries and transactions: This yields the expected frequency of using each attribute in each file as a selection attribute or a join attribute, over all the queries and transactions. Analysing the time constraint of queries and transactions: Some queries and transactions may have stringent performance constraints with respect to response time. The selection attributes used by queries and transactions with such time constraints become higher priority candidates for primary access structures. Analysing the expected frequencies of update operations : A minimum number of access paths should be specified for a file that is updated frequently, because updating the access paths themselves slows the update operations.

3)

4)

Based on the preceding information one can address the physical database design decisions about indexing. The attributes whose values are required in equality or range conditions (selection operation) and those that are keys or that participate in join conditions (join operation) require access paths. The performance of queries largely depends upon what indexes or hashing schemes exist to expedite the processing of selections and joins. On the other hand, during insert, delete, or update operations, existence of indexes adds to the overhead. The following points may be kept in view while taking decisions for indexing: i) The attribute, which is to be indexed, must be a key or there must be some query that uses that attribute either in selection condition (equality or range of values) or in a join. An index can be made on one or multiple attributes. If multiple attributes from one relation are involved together in several queries, a multiattribute index is warranted. Clustering index (index created on a non-key field i.e., if numerous records in a file can have the same value for the field) can be greatly useful in range queries. If several attributes require range queries relative benefits must be evaluated before deciding which attributes to cluster on. At most one index per table can be a primary or clustering index. RDBMSs generally use B+ trees for indexing. ISAM and hash indexes are also provided in some systems. B+ trees support both equality and range queries on the attribute used as the search key. Hash indexes work well with equality conditions, particularly during joins.

ii) iii)

iv)

2.6 SUMMARY
This Unit covers some of the key issues related to data structures, file organisation and physical database design and provides essential background to facilitate their understanding. The role of data structures and file organisation on the performance of access methods has been explained. RAID technology, binary search and indexes have been discussed to elucidate how access speed can be improved. Typical data structures (linked lists, inverted lists and B-trees) and file organisation techniques (SAM, ISAM and DAM) have been

33

Database Design and Management

dealt with. The factors, which influence the physical database design and decisions on access structures, have been explained. This Unit lays the foundation for understanding the complex key concepts related to the topic.

2.7 ANSWERS TO SELF CHECK EXERCISES

1) Existence of appropriate indexes is critical in data retrieval. In systems where query response time is a major consideration, indexes are used to speed up access. However, presence of indexes tends to slow down updating process. A number of commercially available DBMS have B-tree / B +tree index creation features built into the system i.e., the system automatically generates B+tree indexes for speeding up and optimising queries. ORACLE and SYBASE are examples of relational database management systems, which support B-tree/ B+tree indexes.

2)

2.8 KEYWORDS
Cache Memory Binary Search B-tree B + -tree Hashing : A high speed temporary storage in the CPU for storing parts of a program or data during processing. : A search technique for sorted data. : An indexed data storage method that is efficient for a wide range of data access tasks. : A variation on the B-tree structure that provides sequential access to the data as well as fast-indexed access. : An access mechanism that transforms the search key into a storage address, thereby providing very fast access to stored data. : A sorted list of key values from the original table along with a pointer to the rest of the data in each row. : A logical or physical address of a piece of data . : Redundant Array of Independent Disks. A disk drive system that consists of multiple drives with independent controllers. The goal is to split the data to provide faster access and automatic duplication for error recovery.

Index Pointer RAID

Sequential Access : Access that takes records in order, looking at the first, then the next, and so on.

2.9 REFERENCES AND FURTHER READING

Courtney, James F. and Paradice, David, B. (1988). Database Systems for Management. Toronto: Times Mirror/Mosby College Publishing. Date, C.J. (1989). Introduction to Database Systems. New Delhi: Narosa Publishing House. Elmasri, Ramaz and Navathe, Shaukan, B. (2000). Fundamentals of Database Systems. Asia: Pearson Education. Folk,, Michael, J. [et.al.] (2004). File Structures: An Object-oriented Approach with C++. New Delhi: Pearson Education. Gerald, V. Post (2000). Database Management Systems. New Delhi: Tata McGrawHill.
34

OBrien, James A. (1997). Introduction to Information Systems. Irwin: The McGrawHill Company.