Scribd
Upload
45 views

Sqoop Configuration and Installation

This document provides instructions for installing and configuring Sqoop and MySQL connector on Hadoop. The key steps include: 1. Downloading and extracting Sqoop and MySQL connector files. 2. Configuring environment variables and permissions for Sqoop. 3. Editing Hadoop configuration files like mapred-site.xml and yarn-site.xml to configure YARN and MapReduce parameters. 4. Verifying the installation by listing databases using Sqoop with the MySQL connector.

Uploaded by

priyaranjan
Copyright
© © All Rights Reserved
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
45 views

Sqoop Configuration and Installation

This document provides instructions for installing and configuring Sqoop and MySQL connector on Hadoop. The key steps include: 1. Downloading and extracting Sqoop and MySQL connector files. 2. Configuring environment variables and permissions for Sqoop. 3. Editing Hadoop configuration files like mapred-site.xml and yarn-site.xml to configure YARN and MapReduce parameters. 4. Verifying the installation by listing databases using Sqoop with the MySQL connector.

Uploaded by

priyaranjan
Copyright
© © All Rights Reserved
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
You are on page 1/ 2

1.

Download sqoop and mysql connector


2.switch to hduser
3.Extract the tar file
4.copy it to hduser (~/sqoop)
5.directory permission to folder
6.ownership permission to folder
7.open bashrc file
8.copy the code into it (lower part of the file)
9. export SQOOP_HOME=/home/hduser/sqoop
10.export PATH="$SQOOP_HOME/bin":$PATH
11.execute the bashrc file
12.Type: sqoop version

13.cd sqoop
14. type ls
15. cd conf
16. type ls
17.u can see "sqoop-env-template.sh" file
copy that file to same location as sqoop-env.sh
sudo cp sqoop-env-template.sh sqoop-env.sh
18.open that created file
sudo nano sqoop-env.sh
19.under #set path to where bin/hadoop is avilable
you will have an export, remove the # sign infront of export
after = (set hadoop path)
for most of you it will be =/home/hduser/hadoop
20.under #set path to where hadoop-*-core.jar is avilable
you will have an export, remove the # sign infront of export
after = (set hadoop path)
for most of you it will be =/home/hduser/hadoop
21.save and close the file

Now we need to edit hadoop configuration files


-----------------------------------------------------------------

1.cd
2.cd hadoop
3.ls
4.cd etc
5.cd hadoop
6.ls
7.sudo nano mapred-site.xml

a file will be opened delete the


<configuration>........to......</configuration>

and copy this insted of that

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.admin.user.env</name>
<value>HADOOP_MAPRED_HOME=/home/hduser/hadoop</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/home/hduser/hadoop</value>
</property>
</configuration>

8.save and close the file

9.sudo nano yarn-site.xml

a file will be opened delete the already existing configuration part if any
copy the below code and paste there

<configuration>

<!-- Site specific YARN configuration properties -->

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

</configuration>

10.save and close the file


11.cd
12.start all deamons
13.verify with jps
14. if all deamons are working stop the deamons

Installing my sql connector


-----------------------------------------------------
1. mysql-connector copy it to sqoop-->lib

sudo cp sourcepath /home/hduser/sqoop/lib


2.cd sqoop
3.cd lib
4.ls
5.look for mysql-connector
6.cd

Verify the installation


-------------------------------------------------------
sqoop list-databases --connect jdbc:mysql://localhost/ --username root --password
Password@123

You might also like