Scribd
Upload
28 views6 pages

Worksheet 6th

The document describes steps to perform a MapReduce program to analyze weather data and find hot and cold days. It involves downloading weather dataset, creating a Java project in Eclipse, writing Mapper and Reducer classes, exporting to a JAR file, running hadoop daemons, moving data to HDFS, running the JAR to produce output and viewing results.

Uploaded by

Neeraj Kukreti
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
28 views6 pages

Worksheet 6th

The document describes steps to perform a MapReduce program to analyze weather data and find hot and cold days. It involves downloading weather dataset, creating a Java project in Eclipse, writing Mapper and Reducer classes, exporting to a JAR file, running hadoop daemons, moving data to HDFS, running the JAR to produce output and viewing results.

Uploaded by

Neeraj Kukreti
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 6

Experiment No.

6th
Student Name: Neeraj Kukreti UID:22MCC20067
Branch: MCA - CCD Section/Group: 22MCD-1/ Grp A
Semester: IV Date of Performance: 29st Feb 2024
Subject Name: Big Data & Analytics Lab Subject Code: 22CAH-782

1. Aim/Overview of the practical:

MapReduce Program – Weather Data Analysis For Analyzing Hot And Cold Days

2. Code/Steps for practical:

Step1 : Download the dataset from the web, For various cities in different years.

Step2 : Below is the example of our dataset where column 6 and column 7 is showing Maximum and
Minimum temperature, respectively.

Step3 : Make a project in Eclipse with below steps.

Step4 : First Open Eclipse -> then select File -> New -> Java Project ->Name it MyProject -> then select use
an execution environment -> choose JavaSE-1.8 then next -> Finish.
Step5 : below source code to this MyMaxMin java class.

import java.io.IOException;

import java.util.Iterator;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.conf.Configuration;

public class MyMaxMin {

public static class MaxTemperatureMapper extends

Mapper<LongWritable, Text, Text, Text> {

public static final int MISSING = 9999;

@Override

public void map(LongWritable arg0, Text Value, Context context)

throws IOException, InterruptedException {

String line = Value.toString();

if (!(line.length() == 0)) {

String date = line.substring(6, 14);


float temp_Max = Float.parseFloat(line.substring(39, 45).trim());

float temp_Min = Float.parseFloat(line.substring(47, 53).trim());

if (temp_Max > 30.0) {

context.write(new Text("The Day is Hot Day :" + date),

new Text(String.valueOf(temp_Max)));

if (temp_Min < 15) {

context.write(new Text("The Day is Cold Day :" + date),

new Text(String.valueOf(temp_Min)));

public static class MaxTemperatureReducer extends

Reducer<Text, Text, Text, Text> {

public void reduce(Text Key, Iterator<Text> Values, Context context)

throws IOException, InterruptedException {

String temperature = Values.next().toString();

context.write(Key, new Text(temperature));

public static void main(String[] args) throws Exception {

Configuration conf = new Configuration();

Job job = new Job(conf, "weather example");

job.setJarByClass(MyMaxMin.class);
job.setMapOutputKeyClass(Text.class);

job.setMapOutputValueClass(Text.class);

job.setMapperClass(MaxTemperatureMapper.class);

job.setReducerClass(MaxTemperatureReducer.class);

job.setInputFormatClass(TextInputFormat.class);

job.setOutputFormatClass(TextOutputFormat.class);

Path OutputPath = new Path(args[1]);

FileInputFormat.addInputPath(job, new Path(args[0]));

FileOutputFormat.setOutputPath(job, new Path(args[1]));

OutputPath.getFileSystem(conf).delete(OutputPath);

System.exit(job.waitForCompletion(true) ? 0 : 1);

Step6 : Now we add these external jars to our MyProject. Right Click on MyProject -> then select Build Path->
Click on Configure Build Path and select Add External jars…. and add jars from it’s download location then
click -> Apply and Close.

Step7 : Now export the project as jar file. Right-click on MyProject choose Export.. and go to Java -> JAR file
click -> Next and choose your export destination then click -> Next.
Step 4: Start our Hadoop Daemons

start-dfs.sh

start-yarn.sh

Step 5: Move your dataset to the Hadoop HDFS.

Syntax:

hdfs dfs -put /file_path /destination

In below command / shows the root directory of our HDFS.

hdfs dfs -put /home/dikshant/Downloads/CRND0103-2020-AK_Fairbanks_11_NE.txt /

Check the file sent to our HDFS.


hdfs dfs -ls /

Step 6: Now Run your Jar File with below command and produce the output in MyOutput File.

Syntax:

hadoop jar /home/dikshant/Documents/Project.jar /CRND0103-2020-AK_Fairbanks_11_NE.txt /MyOutput

Step7 : Now Move to localhost:50070/, under utilities select Browse the file system and download part-r-00000
in /MyOutput directory to see result.

Learning OutComes
 Understanding MapReduce Architecture:
 Ability to Write MapReduce Programs:

You might also like