How To Install and Run Hadoop On Windows For Beginners - Data Science Central
How To Install and Run Hadoop On Windows For Beginners - Data Science Central
Sign Up
Sign In
HOME AI ML DL ANALYTICS STATISTICS BIG DATA DATAVIZ HADOOP PODCASTS WEBINARS FORUMS JOBS MEMBERSHIP GROUPS SEARCH
CONTACT
Introduction
Hadoop is a software framework from Apache Software Foundation that is used to store and process Big Data. It has two main components; Hadoop Distributed File
System (HDFS), its storage system and MapReduce, is its data processing framework. Hadoop has the capability to manage large datasets by distributing the dataset into
smaller chunks across multiple machines and performing parallel computation on it .
New Books and Resources for DSC Members
SEE FULL LIST ›
×
Overview of HDFS
Hadoop is an essential component of the Big Data industry as it provides the most reliable storage layer, HDFS, which can scale massively. Companies like Yahoo and
Facebook use HDFS to store their data.
HDFS has a master-slave architecture where the master node is called NameNode and slave node is called DataNode. The NameNode and its DataNodes form a cluster.
NameNode acts like an instructor to DataNode while the DataNodes store the actual data.
source: Hasura
There is another component of Hadoop known as YARN. The idea of Yarn is to manage the resources and schedule/monitor jobs in Hadoop. Yarn has two main
components, Resource Manager and Node Manager. The resource manager has the authority to allocate resources to various applications running in a cluster. The node
manager is responsible for monitoring their resource usage (CPU, memory, disk) and reporting the same to the resource manager.
New Books and Resources for DSC Members
SEE FULL LIST ›
×
source: GeeksforGeeks
Advantages of Hadoop
1. Economical – Hadoop is an open source Apache product, so it is free software. It has hardware cost associated with it. It is cost effective as it uses commodity
hardware that are cheap machines to store its datasets and not any specialized machine.
2. Scalable – Hadoop distributes large data sets across multiple machines of a cluster. New machines can be easily added to the nodes of a cluster and can scale to
thousands of nodes storing thousands of terabytes of data.
3. Fault Tolerance – Hadoop, by default, stores 3 replicas of data across the nodes of a cluster. So if any node goes down, data can be retrieved from other nodes.
4. Fast – Since Hadoop processes distributed data parallelly, it can process large data sets much faster than the traditional systems. It is highly suitable for batch
processing of data.
5. Flexibility – Hadoop can store structured, semi-structured as well as unstructured data. It can accept data in the form of textfile, images, CSV files, XML files, emails,
etc
6. Data Locality – Traditionally, to process the data, the data was fetched from the location it is stored, to the location where the application is submitted; however, in
Hadoop, the processing application goes to the location of data to perform computation. This reduces the delay in processing of data.
7. Compatibility – Most of the emerging big data tools can be easily integrated with Hadoop like Spark. They use Hadoop as a storage platform and work as its
processing system.
1. Standalone Mode – It is the default mode of configuration of Hadoop. It doesn’t use hdfs instead, it uses a local file system for both input and output. It is useful for
debugging and testing.
2. Pseudo-Distributed Mode – It is also called a single node cluster where both NameNode and DataNode resides in the same machine. All the daemons run on the same
machine in this mode. It produces a fully functioning cluster on a single machine.
3. Fully Distributed Mode – Hadoop runs on multiple nodes wherein there are separate nodes for master and slave daemons. The data is distributed among a cluster of
machines providing a production environment.
As a beginner, you might feel reluctant in performing cloud computing which requires subscriptions. While you can install a virtual machine as well in your system, it
requires allocation of a large amount of RAM for it to function smoothly else it would hang constantly.
You can install Hadoop in your system as well which would be a feasible way to learn Hadoop.
p y y y p
New Books
We will be installing single node pseudo-distributed hadoop and
cluster on
SEE FULL LIST
Resources
windows
›
10. for DSC Members
×
Prerequisite: To install Hadoop, you should have Java version 1.8 in your system.
Go this link –
https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downl...
Download the file according to your operating system. Keep the java folder directly under the local disk directory (C:\Java\jdk1.8.0_152) rather than in Program Files
(C:\Program Files\Java\jdk1.8.0_152) as it can create errors afterwards.
After downloading java version 1.8, download hadoop version 3.1 from this link –
https://archive.apache.org/dist/hadoop/common/hadoop-3.1.0/hadoop-3...
Extract it to a folder.
New Books and Resources for DSC Members
SEE FULL LIST ›
×
Create a new user variable. Put the Variable_name as HADOOP_HOME and Variable_value as the path of the bin folder where you extracted hadoop.
New Books and Resources for DSC Members
SEE FULL LIST ›
×
Likewise, create a new user variable with variable name as JAVA_HOME and variable value as the path of the bin folder in the Java directory.
Now we need to set Hadoop bin directory and Java bin directory path in system variable path.
Click on New and add the bin directory path of Hadoop and Java in it.
Configurations
Now we need to edit some files located in the hadoop directory of the etc folder where we installed hadoop. The files that need to be edited have been highlighted.
New Books and Resources for DSC Members
SEE FULL LIST ›
×
1. Edit the file core-site.xml in the hadoop directory. Copy this xml property in the configuration in the file
1 /span>configuration>
2 /span>property>
3 /span>name>fs.defaultFS/span>/name>
4 /span>value>hdfs://localhost:9000</value>
5 /span>/property>
6 /span>/configuration>
New Books and Resources for DSC Members
Create a folder with the name ‘datanode’ and a folder ‘namenode’ in this data directory
SEE FULL LIST ›
×
4. Edit the file hdfs-site.xml and add below property in the configuration
Note: The path of namenode and datanode across value would be the path of the datanode and namenode folders you just created.
1 /span>configuration>
2 /span>property>
3 /span>name>dfs.replication/span>/name>
4 /span>value>1/span>/value>
5 /span>/property>
6 /span>property>
7 /span>name>dfs.namenode.name.dir/span>/name>
8 /span>value>C:\Users\hp\Downloads\hadoop-3.1.0\hadoop-3.1.0\data\namenode/span>/value>
9 /span>/property>
10 /span>property>
11 /span>name>dfs.datanode.data.dir/span>/name>
12 /span>value> C:\Users\hp\Downloads\hadoop-3.1.0\hadoop-3.1.0\data\datanode/span>/value>
13 /span>/property>
14 /span>/configuration>
5. Edit the file yarn-site.xml and add below property in the configuration
1 /span>configuration>
2 /span>property>
3 /span>name>yarn.nodemanager.aux-services/span>/name>
4 /span>value>mapreduce_shuffle/span>/value>
5 /span>/property>
6 /span>property>
7 /span>name>yarn.nodemanager.auxservices.mapreduce.shuffle.class/span>/name>
8 /span>value>org.apache.hadoop.mapred.ShuffleHandler/span>/value>
9 /span>/property>
10 /span>/configuration>
6. Edit hadoop-env.cmd and replace %JAVA_HOME% with the path of the java folder where your jdk 1.8 is installed
Hadoop needs windows OS specific files which does not come with default download of hadoop.
To include those files, replace the bin folder in hadoop directory with the bin folder provided in this github link.
https://github.com/s911415/apache-hadoop-3.1.0-winutils
Download it as zip file. Extract it and copy the bin folder in it. If you want to save the old bin folder, rename it like bin_old and paste the copied bin folder in that directory.
New Books and Resources for DSC Members
SEE FULL LIST ›
×
Since it doesn’t throw error and successfully shows the hadoop version, that means hadoop is successfully installed in the system.
Formatting the NameNode is done once when hadoop is installed and not for running hadoop filesystem, else it will delete all the data inside HDFS. Run this command-
1 hdfs namenode –format
(Note: Make sure you are writing the path as per your system)
1 cd C:\Users\hp\Downloads\hadoop-3.1.0\hadoop-3.1.0\sbin
Two more cmd windows will open for NameNode and DataNode
Two more windows will open, one for yarn resource manager and one for yarn node manager.
Note: Make sure all the 4 Apache Hadoop Distribution windows are up n running. If they are not running, you will see an error or a shutdown message. In that case, you
need to debug the error.
To access information about resource manager current jobs, successful and failed jobs, go to this link in browser-
http://localhost:8088/cluster
New Books and Resources for DSC Members
SEE FULL LIST ›
×
To check the details about the hdfs (namenode and datanode),
http://localhost:9870/
Note: If you are using Hadoop version prior to 3.0.0 – Alpha 1, then use port http://localhost:50070/
I will create a directory named ‘sample’ in my hadoop directory using the following command-
1 hdfs dfs –mkdir /sample
To verify if the directory is created in hdfs, we will use ‘ls’ command which will list the files present in hdfs –
1 hdfs dfs –ls /
Then I will copy a text file named ‘potatoes’ from my local file system to this folder that I just created in hdfs using copyFromLocal command-
1 hdfs dfs -copyFromLocal C:\Users\hp\Downloads\potatoes.txt /sample
To verify if the file is copied to the folder, I will use ‘ls’ command by specifying the folder name which will read the list of files in that folder –
1 hdfs dfs –ls /sample
New Books and Resources for DSC Members
SEE FULL LIST ›
×
To view the contents of the file we copied, I will use cat command-
1 hdfs dfs –cat /sample/potatoes.txt
To Copy file from hdfs to local directory, I will use get command –
1 hdfs dfs -get /sample/potatoes.txt C:\Users\hp\Desktop\priyanka
These were some basic hadoop commands. You can refer to this HDFS commands guide to learn more.
https://hadoop.apache.org/docs/r3.1.0/hadoop-project-dist/hadoop-hd...
Hadoop MapReduce can be used to perform data processing activity. However, it possessed limitations due to which frameworks like Spark and Pig emerged and have
gained popularity. A 200 lines of MapReduce code can be written with less than 10 lines of Pig code. Hadoop has various other components in its ecosystem like Hive,
Sqoop, Oozie, and HBase. You can download these software as well in your windows system to perform data processing operations using cmd.
Follow this link, if you are looking to learn more about data science online.
Oth l
Other popular resources
Follow us: Twitter | Facebook
Views: 21578
Like
1 member likes this
Tweet
Share Facebook
Like 0
Comment
Welcome to
Data Science Central
Sign Up
or Sign In
Or sign in with:
New Books and Resources for DSC Members
SEE FULL LIST ›
×
RESOURCES
VIDEOS
DSC Webinar Series: From Degas to Dashboards: Lessons of the Great Masters
Added by Tim Matteson 1 Comment 1 Like
Add Videos
View All
Badges | Report an Issue | Privacy Policy | Terms of Service