04. Hadoop Installaion (1)
04. Hadoop Installaion (1)
Map-Reduce Programming
by
A K Chakravarthy
Assistant Professor
Department of Information Technology
$ clear
$ sudo addgroup aec_viper_group
$ sudo adduser –ingroup aec_viper_group aec_viper_user
If not
$ sudo apt-get install default –jre
$ sudo apt-get install default –jdk
(Internet connection is must)
$ ssh localhost
$ ssh-keygen -t rsa
$ cat ~/.ssh/id_rsa.pub >>
~/.ssh/authorized_keys $ chmod 0600
~/.ssh/authorized_keys
Just like your Java path, Now we are doing it for Hadoop
path.
Open the file abc.txt, where you have already typed two
lines in that previously
export HADOOP_HOME=/home/acet_viper_user/hadoop-2.7.2
export PATH=$PATH:$HADOOP_HOME/bin
$ source ~/.bashrc
$ echo $JAVA_HOME
$ echo $HADOOP_HOME
$ echo $PATH (It will show both Java path and Hadoop Path)
$ ls –lrt
$ mkdir input_dir
$ cp $HADOOP_HOME/*.txt input_dir
$ cd input_dir
$ ls –lrt
$ cd .. //to come to nitw_viper_user
32 Big Image Data Processing on Hadoop 02/03/25
Step-13: Now, use the following command to execute the
program Jar file name in the
current directory
This is
keyword
just like
java This is
This is output
This is the input directory
file name directory
$ cd output_dir
$ ls –lrt
$ cat part-r-00000
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS=”-Djava.library.path=$HADOOP_HOME/lib”
export PATH=$PATH:$HADOOP_HOME/sbin
$ source ~/.bashrc
You can find all the Hadoop configuration files in the location
“$HADOOP_HOME/etc/hadoop”.
$ cd $HADOOP_HOME/etc/hadoop
$ pwd
$ nano hadoop-env.sh
$ nano core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
$ nano hdfs-site.xml
The hdfs-site.xml file contains information such as the value of
replication data, namenode path, and datanode paths of your local file
systems. It means the place where you want to store the Hadoop
infrastructure.
Let us assume the following data.
dfs.replication (data replication value) = 1
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>file:///home/acet_viper_user/hadoopinfra/hdfs/namenode</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>file:///home/acet_viper_user/hadoopinfra/hdfs/datanode</value>
</property>
</configuration>
$ nano yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
$ cp mapred-site.xml.template mapred-site.xml
$ cd
$ start–dfs.sh
$ start–yarn.sh
$ jps
This is
keyword This is
just like output
java directory
This is
This is the input
file name directory
•Add the ’master’ public SSH key using the following command.
Connect with user nitw_viper_user from the master to the user account
nitw_viper_user on all slaves in 6 different terminals! (UBUNTU Terminals)
/home/hadoop-2.7.2/etc/hadoop directory
nitw_cvhd_user@selab104:~$ cd /home/hadoop-2.7.2/etc/hadoop
nitw_cvhd_user@selab104:~$ sudo gedit masters
Add the following line
secondarymaster
69 Big Image Data Processing on Hadoop 02/03/25
70 Big Image Data Processing on Hadoop 02/03/25
Now, we have to do the same process in
all the reaming 5 systems (I will be doing it
without touching those 5 systems. (i.e.
through ‘ssh’, already connected))
nitw_viper_user@selab105:~$
nitw_viper_user@selab101:~$ nitw_viper_user@selab102:~$
nitw_viper_user@selab103:~$ nitw_viper_user@selab106:~$
/home/hadoop-2.7.2/etc/hadoop directory
nitw_viper_user@selab105:~$
nitw_viper_user@selab101:~$ nitw_viper_user@selab102:~$
nitw_viper_user@selab103:~$ nitw_viper_user@selab106:~$
nitw_viper_user@selab104:~/home/hadoop-2.7.2/etc/hadoop$
sudo gedit core-site.xml
nitw_viper_user@selab105(101,102,103,104,106):~/home/hadoop-2.7.2/
etc/hadoop$ sudo gedit core-site.xml
/home/hadoop-2.7.2/etc/hadoop/core-site.xml (ALL
machines .i.e. Master as well as slave)
Pseudo Mode(core-site.xml) Full Distributed Mode(core-
site.xml)
<property> <property>
<name>fs.default.name</name>
<name>fs.default.name</name> <value>hdfs://master:9000</value>
</property>
<value>hdfs://localhost:9000</val
ue>
</property>
nitw_viper_user@selab105(101,102,103,104,106):~/:~/home/
hadoop-2.7.1/etc/hadoop$ sudo gedit hdfs-site.xml
nitw_cvhd_user@selab104:~/home/hadoop-2.7.2/etc/hadoop$
sudo gedit yarn-site.xml
nitw_cvhd_user@selab105(101,102,103,104,106):~/:~/home/
hadoop-2.7.2/etc/hadoop$ sudo gedit yarn-site.xml
nitw_cvhd_user@selab105(101,102,103,104,106):~/home/hadoop-
2.7.2/etc/hadoop$ sudo gedit mapred-site.xml
nitw_cvhd_user@selab104:~$ bin/start-all.sh
by
Dr. U.S.N. Raju
Asst. Professor, Dept. of CS&E,
N.I.T. Warangal
by
Dr. U.S.N. Raju
Asst. Professor, Dept. of CS&E,
N.I.T. Warangal
Configuring hdfs-site.xml
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
http://localhost:50070/webhdfs/v1/user/nitw_cvhd_user2/input/
input_image.jpg?op=OPEN
108 Big Image Data Processing on Hadoop 02/03/25
109 Big Image Data Processing on Hadoop 02/03/25
Creating 8 Bitmap images for an image in HDFS