Introduction to Hadoop

Hadoop and its Ecosystem

• Introduction: Hadoop Ecosystem is a platform or a suite which

provides various services to solve the big data problems. It includes
Apache projects and various commercial tools and solutions. There
are four major elements of Hadoop i.e. HDFS, MapReduce, YARN, and
Hadoop Common. Most of the tools or solutions are used to
supplement or support these major elements. All these tools work
collectively to provide services such as absorption, analysis, storage
and maintenance of data etc.
• Following are the components that collectively form a Hadoop ecosystem:

• HDFS: Hadoop Distributed File System

• YARN: Yet Another Resource Negotiator
• MapReduce: Programming based Data Processing
• Spark: In-Memory data processing
• PIG, HIVE: Query based processing of data services
• HBase: NoSQL Database
• Mahout, Spark MLLib: Machine Learning algorithm libraries
• Solar, Lucene: Searching and Indexing
• Zookeeper: Managing cluster
• Oozie: Job Scheduling
Hadoop HDFS
• Data is stored in a distributed manner in HDFS. There are two components of HDFS -
name node and data node. While there is only one name node, there can be multiple
data nodes.
• HDFS is specially designed for storing huge datasets in commodity hardware. An
enterprise version of a server costs roughly $10,000 per terabyte for the full processor. In
case you need to buy 100 of these enterprise version servers, it will go up to a million
dollars.
• Hadoop enables you to use commodity machines as your data nodes. This way, you don’t
have to spend millions of dollars just on your data nodes. However, the name node is
always an enterprise server.
• Features of HDFS
• Provides distributed storage
• Can be implemented on commodity hardware
• Provides data security
• Highly fault-tolerant - If one machine goes down, the data from that machine goes to the
next machine
Hadoop MapReduce

• Hadoop MapReduce is the processing unit of Hadoop. In the

MapReduce approach, the processing is done at the slave nodes, and
the final result is sent to the master node.
• A data containing code is used to process the entire data. This coded
data is usually very small in comparison to the data itself. You only
need to send a few kilobytes worth of code to perform a heavy-duty
process on computers.
`
• The input dataset is first split into chunks of data. In this example, the
input has three lines of text with three separate entities - “bus car
train,” “ship ship train,” “bus ship car.” The dataset is then split into
three chunks, based on these entities, and processed parallelly.
• In the map phase, the data is assigned a key and a value of 1. In this
case, we have one bus, one car, one ship, and one train.
• These key-value pairs are then shuffled and sorted together based on
their keys. At the reduce phase, the aggregation takes place, and the
final output is obtained.
Hadoop YARN

• Hadoop YARN stands for Yet Another Resource Negotiator. It is the

resource management unit of Hadoop and is available as a
component of Hadoop version 2.
• Hadoop YARN acts like an OS to Hadoop. It is a file system that is built
on top of HDFS.
• It is responsible for managing cluster resources to make sure you
don't overload one machine.
• It performs job scheduling to make sure that the jobs are scheduled
in the right place
CON…
CON…
• Suppose a client machine wants to do a query or fetch some code for
data analysis. This job request goes to the resource manager
(Hadoop Yarn), which is responsible for resource allocation and
management.
• In the node section, each of the nodes has its node managers. These
node managers manage the nodes and monitor the resource usage
in the node. The containers contain a collection of physical resources,
which could be RAM, CPU, or hard drives. Whenever a job request
comes in, the app master requests the container from the node
manager. Once the node manager gets the resource, it goes back to
the Resource Manager.
 Introduction to Hadoop Tools

• Hadoop is an open-source framework written in Java that uses lots of

other analytical tools to improve its data analytics operations.
• There is a wide range of analytical tools available in the market that
help Hadoop deal with the astronomical size data efficiently.
• The most famous and widely used tools :
• Sqoop
• Flume
• APACHE Pig
• Oozie
• HBase
Sqoop

• Sqoop is used to transfer data between Hadoop and external

datastores such as relational databases and enterprise data
warehouses. It imports data from external datastores into HDFS, Hive,
and HBase.
• The client machine gathers code, which will then be sent to Sqoop.
The Sqoop then goes to the Task Manager, which in turn connects to
the enterprise data warehouse, documents based systems, and
RDBMS. It can map those tasks into Hadoop
Flume

• Flume is another data collection and ingestion tool, a distributed

service for collecting, aggregating, and moving large amounts of log
data. It ingests online streaming data from social media, logs files,
web server into HDFS.
CON…
• data is taken from various sources, depending on your organization’s
needs. It then goes through the source, channel, and sink. The sink
feature ensures that everything is in sync with the requirements.
Finally, the data is dumped into HDFS.
 Pig

Apache Pig was developed by Yahoo researchers, targeted mainly towards non-
programmers. It was designed with the ability to analyze and process large datasets
without using complex Java codes. It provides a high-level data processing language that
can perform numerous operations without getting bogged down with too many technical
concepts.
It consists of:
1. Pig Latin - This is the language for scripting
2. Pig Latin Compiler - This converts Pig Latin code into executable code
Pig also provides Extract, Transfer, and Load (ETL), and a platform for building data flow.
Did you know that ten lines of Pig Latin script equals approximately 200 lines of
MapReduce job? Pig uses simple, time-efficient steps to analyze datasets
• Pig’s architecture
•
CON…
• Programmers write scripts in Pig Latin to analyze data using Pig.
Grunt Shell is Pig’s interactive shell, used to execute all Pig scripts.
• If the Pig script is written in a script file, the Pig Server executes it.
• The parser checks the syntax of the Pig script, after which the output
will be a DAG (Directed Acyclic Graph). The DAG (logical plan) is
passed to the logical optimizer.
• The compiler converts the DAG into MapReduce jobs.
• The MapReduce jobs are then run by the Execution Engine. The
results are displayed using the “DUMP” statement and stored in
HDFS using the “STORE” statement.
 Oozie

• Oozie is a workflow scheduler system used to manage Hadoop jobs.

It consists of two parts:
1. Workflow engine - This consists of Directed Acyclic Graphs (DAGs),
which specify a sequence of actions to be executed
2. Coordinator engine - The engine is made up of workflow jobs
triggered by time and data availability
• As seen from the flowchart below, the process begins with the
MapReduce jobs. This action can either be successful, or it can end in
an error. If it is successful, the client is notified by an email. If the
action is unsuccessful, the client is similarly notified, and the action is
terminated.
Oozie CON…..
•
HBase
HBase is nothing but a non-relational,
NoSQL distributed, and column-
oriented database. HBase consists of
various tables where each table has
multiple numbers of data rows. These
rows will have multiple numbers of
column family’s, and this column family
will have columns that contain key-
value pairs. HBase works on the top of
HDFS(Hadoop Distributed File System).
We use HBase for searching small size
data from the more massive datasets.
• Features of HBase:
• HBase has Linear and Modular
Scalability
• JAVA API can easily be used for client
access
• Block cache for real time data queries
 Hadoop Distributed File System (HDFS)
The Hadoop Distributed File System (HDFS) is the primary data storage system
used by Hadoop applications. HDFS employs a NameNode and DataNode
architecture to implement a distributed file system that provides high-
performance access to data across highly scalable Hadoop clusters.
Hadoop itself is an open source distributed processing framework that manages
data processing and storage for big data applications.
HDFS is a key part of the many Hadoop ecosystem technologies.
It provides a reliable means for managing pools of big data and supporting
related big data analytics applications
 How Does HDFS work?
HDFS enables the rapid transfer of data between compute nodes.
At its outset, it was closely coupled with MapReduce, a framework for data
processing that filters and divides up work among the nodes in a cluster, and it
organizes and condenses the results into a cohesive answer to a query.
Similarly, when HDFS takes in data, it breaks the information down into separate
blocks and distributes them to different nodes in a cluster.
With HDFS, data is written on the server once, and read and reused numerous times
after that. HDFS has a primary NameNode, which keeps track of where file data is
kept in the cluster.
HDFS also has multiple DataNodes on a commodity hardware cluster -- typically one
per node in a cluster.
The DataNodes are generally organized within the same rack in the data center. Data
is broken down into separate blocks and distributed among the various DataNodes
for storage.
Blocks are also replicated across nodes, enabling highly efficient parallel processing.
The NameNode knows which DataNode contains which blocks and where the
DataNodes reside within the machine cluster. The NameNode also manages
access to the files, including reads, writes, creates, deletes and the data block
replication across the DataNodes.
The NameNode operates in conjunction with the DataNodes. As a result, the
cluster can dynamically adapt to server capacity demand in real time by adding
or subtracting nodes as necessary.
The DataNodes are in constant communication with the NameNode to
determine if the DataNodes need to complete specific tasks. Consequently, the
NameNode is always aware of the status of each DataNode.
If the NameNode realizes that one DataNode isn't working properly, it
can immediately reassign that DataNode's task to a different node
containing the same data block.
DataNodes also communicate with each other, which enables them to
cooperate during normal file operations.
Moreover, the HDFS is designed to be highly fault-tolerant.
The file system replicates -- or copies -- each piece of data multiple
times and distributes the copies to individual nodes, placing at least one
copy on a different server rack than the other copies.
 HDFS architecture, NameNodes and DataNodes
HDFS uses a primary/secondary architecture.
The HDFS cluster's NameNode is the primary server that manages the file system namespace and
controls client access to files.
As the central component of the Hadoop Distributed File System, the NameNode maintains and
manages the file system namespace and provides clients with the right access permissions.
The system's DataNodes manage the storage that's attached to the nodes they run on.
HDFS exposes a file system namespace and enables user data to be stored in files. A file is split into one
or more of the blocks that are stored in a set of DataNodes.
The NameNode performs file system namespace operations, including opening, closing and renaming
files and directories.
The NameNode also governs the mapping of blocks to the DataNodes. The DataNodes serve read and
write requests from the clients of the file system.
In addition, they perform block creation, deletion and replication when the NameNode instructs them
to do so.
HDFS supports a traditional hierarchical file organization.
An application or user can create directories and then store files
inside these directories.
The file system namespace hierarchy is like most other file
systems
-- a user can create, remove, rename or move files from one
directory to another.
The NameNode records any change to the file system
namespace or its properties.
An application can stipulate the number of replicas of a file that
the HDFS should maintain.
The NameNode stores the number of copies of a file, called the
replication factor of that file.
 Features of HDFS
There are several features that make HDFS particularly useful, including:
Data replication. This is used to ensure that the data is always available
and prevents data loss. For example, when a node crashes or there is a
hardware failure, replicated data can be pulled from elsewhere within a
cluster, so processing continues while data is recovered.
Fault tolerance and reliability: HDFS' ability to replicate file blocks and
store them across nodes in a large cluster ensures fault tolerance and
reliability.
High availability: As mentioned earlier, because of replication across
notes, data is available even if the NameNode or a DataNode fails.
Scalability: Because HDFS stores data on various nodes in the cluster, as
requirements increase, a cluster can scale to hundreds of nodes.
High throughput: Because HDFS stores data in a distributed manner, the
data can be processed in parallel on a cluster of nodes. This, plus data
locality (see next bullet), cut the processing time and enable high
throughput.
Data locality: With HDFS, computation happens on the DataNodes
where the data resides, rather than having the data move to where the
computational unit is. By minimizing the distance between the data and
the computing process, this approach decreases network congestion and
boosts a system's overall throughput.
 What are the benefits of using HDFS?
There are five main advantages to using HDFS, including:
Cost effectiveness: The DataNodes that store the data rely on inexpensive off-the-shelf hardware, which
cuts storage costs. Also, because HDFS is open source, there's no licensing fee.
Large data set storage: HDFS stores a variety of data of any size -- from megabytes to petabytes -- and in
any format, including structured and unstructured data.
Fast recovery from hardware failure: HDFS is designed to detect faults and automatically recover on its
own.
Portability: HDFS is portable across all hardware platforms, and it is compatible with several operating
systems, including Windows, Linux and Mac OS/X.
Streaming data access: HDFS is built for high data throughput, which is best for access to streaming data.
 HDFS Data Replication
Data replication is an important part of the HDFS format as it ensures data remains
available if there's a node or hardware failure.
As previously mentioned, the data is divided into blocks and replicated across
numerous nodes in the cluster.
Therefore, when one node goes down, the user can access the data that was on that
node from other machines.
HDFS maintains the replication process at regular intervals.
HDFS use cases and examples
The Hadoop Distributed File System emerged at Yahoo as a part of that company's
online ad placement and search engine requirements.
Like other web-based companies, Yahoo juggled a variety of applications that were
accessed by an increasing number of users, who were creating more and more data.
EBay, Facebook, LinkedIn and Twitter are among the companies that used HDFS to
underpin big data analytics to address requirements similar to Yahoo's.
HDFS has found use beyond meeting ad serving and search engine requirements.
The New York Times used it as part of large-scale image conversions,
Media6Degrees for log processing and machine learning, LiveBet for log storage
and odds analysis, Joost for session analysis, and Fox Audience Network for log
analysis and data mining. HDFS is also at the core of many open source data lakes.
More generally, companies in several industries use HDFS to manage pools of big data, including:
Electric companies: The power industry deploys phasor measurement units (PMUs) throughout their
transmission networks to monitor the health of smart grids.
These high-speed sensors measure current and voltage by amplitude and phase at selected
transmission stations.
These companies analyze PMU data to detect system faults in network segments and adjust the grid
accordingly.
For instance, they might switch to a backup power source or perform a load adjustment. PMU
networks clock thousands of records per second, and consequently, power companies can benefit
from inexpensive, highly available file systems, such as HDFS.
Marketing: Targeted marketing campaigns depend on marketers knowing a lot about their target
audiences.
Marketers can get this information from several sources, including CRM systems, direct mail
responses, point-of-sale systems, Facebook and Twitter.
Because much of this data is unstructured, an HDFS cluster is the most cost-effective place to put data
before analyzing it.
Oil and gas providers: Oil and gas companies deal with a variety of data
formats with very large data sets, including videos, 3D earth models and
machine sensor data. An HDFS cluster can provide a suitable platform
for the big data analytics that's needed.
Research: Analyzing data is a key part of research, so, here again, HDFS
clusters provide a cost-effective way to store, process and analyze large
amounts of data.
 HDFS Commands with Examples and Usage

HDFS is the primary or major component of the Hadoop ecosystem which is

responsible for storing large data sets of structured or unstructured data across
various nodes and thereby maintaining the metadata in the form of log files.
• To check the Hadoop services are up
and running use the following
command:
• jps
ls: This command is used to list all the files.
Use lsr for recursive approach. It is useful when
we want a hierarchy of a folder.
Syntax:
bin/hdfs dfs -ls <path>
Example:
bin/hdfs dfs -ls /
It will print all the directories present in HDFS.
bin directory contains executables so, bin/
hdfs means we want the executables
of hdfs particularly dfs(Distributed File System)
commands.
mkdir: To create a directory. In Hadoop dfs there is no home directory by default. So let’s first create it.
Syntax:
bin/hdfs dfs -mkdir <folder name> creating home directory: hdfs/bin -mkdir /user hdfs/bin -
mkdir /user/username
-> write the username of your computer
Example:
bin/hdfs dfs -mkdir /geeks => '/' means absolute path bin/hdfs dfs -mkdir geeks2 =>
Relative path
-> the folder will be created relative to the home directory.
bin/hdfs dfs -touchz /geeks/myfile.txt

touchz: It creates an empty file.

Syntax:
bin/hdfs dfs -touchz <file_path>
Example: hdfs dfs –touch /geeks/myfile.txt
copyFromLocal (or) put: To copy files/folders from local file system to hdfs store.
This is the most important command. Local filesystem means the files present on the OS.
Syntax:
bin/hdfs dfs -copyFromLocal <local file path> <dest(present on hdfs)>
Example: Let’s suppose we have a file AI.txt on Desktop which we want to copy to
folder geeks present on hdfs.
bin/hdfs dfs -copyFromLocal ../Desktop/AI.txt /geeks (OR) bin/hdfs dfs -put ..
/Desktop/AI.txt /geeks
5 cat: To print file contents.
Syntax:
bin/hdfs dfs -cat <path>
Example:
// print the content of AI.txt present // inside geeks folder. bin/hdfs dfs -cat /geeks/AI.txt ->
copyToLocal (or) get: To copy files/folders from hdfs store to local file system.
Syntax:
bin/hdfs dfs -copyToLocal <<srcfile(on hdfs)> <local file dest>
Example:
bin/hdfs dfs -copyToLocal /geeks ../Desktop/hero (OR) bin/hdfs dfs -get /geeks/myfile.txt ../Desktop/hero
myfile.txt from geeks folder will be copied to folder hero present on Desktop.
moveFromLocal: This command will move file from local to hdfs.
Syntax:
bin/hdfs dfs -moveFromLocal <local src> <dest(on hdfs)>
Example:
bin/hdfs dfs -moveFromLocal ../Desktop/cutAndPaste.txt /geeks
cp: This command is used to copy files within hdfs. Lets copy folder geeks to geeks_copied.
Syntax:
bin/hdfs dfs -cp <src(on hdfs)> <dest(on hdfs)>
Example:
bin/hdfs -cp /geeks /geeks_copied
1. mv: This command is used to move files within hdfs. Lets cut-paste a file myfile.txt from geeks folder
to geeks_copied.
Syntax:
bin/hdfs dfs -mv <src(on hdfs)> <src(on hdfs)>
Example:
bin/hdfs -mv /geeks/myfile.txt /geeks_copied
rmr: This command deletes a file from HDFS recursively. It is very useful command
when you want to delete a
non-empty directory.
Syntax:
bin/hdfs dfs -rmr <filename/directoryName>
Example:
bin/hdfs dfs -rmr /geeks_copied -> It will delete all the content
inside the directory
then the directory itself.