Scribd
Upload
81 views

Google File System

The document summarizes the Google File System (GFS), which is designed to handle large distributed data-intensive applications at Google. It discusses key aspects of GFS including its scalable architecture with a single master and multiple chunkservers, how files are divided into fixed-size chunks, fast read and write capabilities, record append functionality, fault tolerance and recovery mechanisms, and role of the master in managing metadata and balancing the system. Overall, GFS provides a reliable and high-performance distributed file system tailored to Google's unique computing environment and application needs.

Uploaded by

Rakesh Akhileswaran
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
81 views

Google File System

The document summarizes the Google File System (GFS), which is designed to handle large distributed data-intensive applications at Google. It discusses key aspects of GFS including its scalable architecture with a single master and multiple chunkservers, how files are divided into fixed-size chunks, fast read and write capabilities, record append functionality, fault tolerance and recovery mechanisms, and role of the master in managing metadata and balancing the system. Overall, GFS provides a reliable and high-performance distributed file system tailored to Google's unique computing environment and application needs.

Uploaded by

Rakesh Akhileswaran
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 22

RAKESH.

A
Google FILE R7.B
SYSTEM NO:26

GUIDE: P. KRISHNAKUMARAN
THAMPI 1
INTRODUCTION
• GFS is a scalable distributed file system for large
distributed data-intensive applications
• Files are huge by traditional standards. Multi-GB files
are common
• The largest cluster provides hundreds of terabytes of
storage across thousands of disks on over a thousand
machines, and it is concurrently accessed by hundreds of
clients.
• Google File System (GFS), handling these problems that
serve their environment and application, along with
general distributed file system features like scaling,
security, performance, reliability, etc. 2
Design Motivations
• Fault-tolerance and auto-recovery need to be built into
the system.
• It has to serve to queries very fast so problems are often,
inorder to handle everything has to be built in very
efficiently.
• Standard I/O assumptions have to be re-examined.
• Record appends are the prevalent form of writing.
• Google applications and GFS should be codesigned
• Failure, store large files, workload, multiple clients,
sustained bandwidth should be considered.
3
Architecture
• It does not implement standard API
• It consist of single master that maintains metadata and
multiple chunkservers
• Files are divided into fixed size chunks and identified
by globally unique chunkhandler
• master is a single process running on a separate
machine
• chunk is a analogous to a file whose size is 64 MB

4
Contd…
• Master - Chunkserver Communication:
• Master and chunkserver communicate regularly to
obtain state:
• Is chunkserver down?
• Are there disk failures on chunkserver?
• Are any replicas corrupted?
• Which chunk replicas does chunkserver store?
• Master sends instructions to chunkserver:
• Delete existing chunk.
• Create new chunk.
5
6
Chunk
• Important design decision
• Size of a chunk is 64MB which helps in unique
environment of Google since the average file size is
around 100MB.
• Advantages of this are
• Doesn’t need to contact master many times
• It reduces size of metadata in master
• Bigger the size of chunks lesser number of chunks
• Eg: 2MB chunk size,100MB data we have 50 chunks.
and 10MB chunk size,100MB data we have 10
chunks. 7
Metadata
• The master stores three major types of metadata: t he
file and chunk namespaces, the mapping from files to
chunks, and the location of each chunk’s replicas.
• Among these three, the first two types are kept
persistent by keeping the log of mutations to an
operation log stored on the master’s local disk.
• This operation log is also replicated on remote
machines
• The master doesn’t store chunk location information
persistently, instead it asks each chuckservers about
its chunks when it starts up or when a chunkserver
joins the cluster 8
Reading data

9
Writing Data

10
Record Append
Algorithm
• Application originates record append request
• GFS client translates requests and sends it to master
• Master responds with chunk handle and(primary + secondary)
replica locations
• Client pushes write data to all locations
• Primary checks if record fits in specified chunk
• If record doesn’t fit, then the primary:
•Pads the chunk
•Tell secondaries to do the same
•And informs the client
•Client then retries the append with the next chunk
11
Contd..
• If record fits, then the primary:
•Appends the record
•Tells secondaries to do the same
•Receives responses from secondaries
•And sends final response to the client

• Thus the clients can read, write and append records in


parallel

12
Aggregate Throughputs

• Top curves show theoretical limits imposed by our


network topology
• Bottom curves show measured throughputs.
13
Master Operation
• Namespace Management And Locking
• Balancing
• Fault Tolerance
• Garbage Collection

14
Namespace Management &
Locking
• GFS represents its namespace as lookup table
mapping full pathnames to metadata
• Each master operation acquires a set of locks
• Mutation or read on /d1/d2/.../dn/leaf:
• read-locks on: /d1, /d1/d2, /d1/d2/d3......
• read or write lock on: /d1/d2/.../dn/leaf
• Example:
• create /home/user/abc
• read-locks: /home, /home/user
• write-locks: /home/user/abc
15
Balancing
• Creation: New replicas on chunkservers with below
average disk space utilization
• Re-replication: As soon as the number of available
replicas falls below a user-specified goal
• Rebalancing: The master rebalances the replicas
periodically

16
Fault Tolerance
• Fast Recovery: master and chunkservers are designed
to restart and restore state in a few seconds.
• Chunk Replication: across multiple machines, across
multiple racks.
• Master Mechanisms:
• Log of all changes made to metadata.
• Periodic checkpoints of the log.
• Log and checkpoints replicated on multiple
machines.
• Master state is replicated on multiple machines.
• “Shadow” masters for reading data if “real” master
is down
17
GARBAGE
COLLECTION
• Lazy reclamation
• Logs deletion immediately
• Rename to a hidden name
• Remove 3 days later
• Undelete by renaming back
• Regular scan
• For orphaned chunks
• Erase metadata
• Heartbeat message to tell chunk servers to delete
chunks
18
CONCLUSION
• We understand the motivation behind a new
distributed file system.
• GFS clearly demonstrates the qualities essential for the
large-scale processing with commodity class PCs.
• It serves the Google applications a solid backbone that
helps these applications to stand ahead of its
competitors and serving millions of people every
single second!
• GFS not only for their current applications, but also
for anticipated applications too
19
Reference
• [Ghemawat, Gobioff and Leung 2003] Sanjay
Ghemawat, Howard Gobioff and Shun-Tak Leung,
Google File System, ACM SIGOPS Operating
Systems Review, 2003.
• http://en.wikipedia.org/wiki/Google_File_System

20
THANK YOU….

21
22

You might also like