UNIT 1

▪ Data continues to be a precise and irreplaceable asset for

enterprises

▪ Data is present internal and outside four walls of the

enterprise

▪ Data present in Homogeneous sources as well as in

heterogeneous sources
Approximate Distribution of Digital Data

Approximate percentage distribution of digital data

▪ Need of the hour is to understand, manage,
process and take the data for analysis to draw
valuable insights

Data -> Information

Information ->Insights
Classification of digital data

1. Unstructured data –
▪ Data which do not conform to a data model or a form
which can not be used by computer program.
▪ 80-90% data is in this form
▪ PPT, Images , Videos, body of an email etc

2. Semi structured data- does not conform to a data model

but has some structure . Not in a form which can be used by
a computer program.
▪ Email, XML, HTML etc
3 . Structured data

▪ Data in organised form

▪ Understandable by a computer program
▪ Eg- Data stored in databases
Structured data
-> When is data structured? –Data conforms to a pre
defined schema/structure

-> Think structured data and think data model

l
-> Context of RDBMS- Conforms to the relational model –
rows/columns
-> Cardinality ratio – Number of rows/columns
-> Degree- Number of columns
Steps
1) Design a relation/table
-> Fields to store -> Data type
2) Constraints- we would like our data to conform
– Unique
- Not Null
– business constraints
– permissible values a column should access
3) Figure
4)Referential Integrity constraints
Sources- When data is structured- leverage on available
RDBMS
Databases such as
Oracle, DB2,
Teradata, MySql,
PostgreSQL, etc

Structured data Spreadsheets

OLTP Systems

Database store operational data generated and

collected by day to day business activities
Ease with Structured Data

Input / Update /
Delete

Security

Ease with Structured data Indexing /

Searching

Scalability

Transaction
Processing
ACID
Atomicity- Transaction happens in its eternity or none of it all

Consistency- If same information is stored at two or more

places they are in complete agreement

Isolation- Resource allocation to transaction happen such that

the transaction gets the impression that it is the only
transaction happens in isolation

Durability- Changes made to the database during transaction

is permanent
Semi structured data

Self describing structure

Example, emails, XML, markup languages like HTML, etc

Features

Does not conform to data model

Use tags to segregate semantic elements
Tags are also used to enforce hierarchies of records and fields within
data
No separation between data and schema
Entities belong to the same class and also grouped together need not
necessarily have same attributes- not necessary same order
Characteristics of Semi-structured Data

Inconsistent Structure

Self-describing
(lable/value
Semi-structured data pairs)
Often Schema information
is blended with data
values
Data objects may have
different attributes not known
beforehand
 XML Extensible MarkUp

Sources Language

XML- popularised by web Other MarkUp Language

services

JSON(JavaScript Object
JSON- transmit data between Semi-Structured Notation)
server and web application. Data

Mongodb store data natively

Unstructured Data

This is the data which does not conform to a data model or is not in
a form which can be used easily by a computer program.

About 80–90% data of an organization is in this format.

Example: memos, chat rooms, PowerPoint presentations,

images, videos, letters, researches, white papers, body of an
email, etc.
Sources of Unstructured Data
Web Pages

Images

Free-Form
Text

Audios
Unstructured data

Videos

Body of
Email

Text
Messages
Chats

Social

Media data

Word
Document
Issues with terminology – Unstructured Data

Structure can be implied despite not

being formerly defined.

Issues with terminology Data with some structure may still be labeled
unstructured if the structure doesn’t help
with processing task at hand

Data may have some structure or may even be

highly structured in ways that are
unanticipated or unannounced.
Dealing with Unstructured Data

Data Mining

Natural Language Processing (NLP)

Dealing with Unstructured Data

Text Analytics

Noisy Text Analytics

Unstructured data

Data Mining
First we deal with large data sets.
Second use methods at the intersection of AI, machine learning and
statistics and database to unearth consistent patterns in large data set
and systematic relation between variables

Association rule mining- Market basket analysis- what goes with what-
bread , cheese
Regression analysis- Predict relationship between two variables –
dependant and independent variable – value to be predicted –
dependant
Collaborative filtering – Predicting user preference based on preferences
of group of users
What is Big Data?

Big data is defined as collections of datasets whose

volume, velocity or variety is so large that it is difficult to
store, manage, process and analyze the data using
traditional databases and data processing tools.

In the recent years, there has been an exponential growth

in the both structured and unstructured data generated by
information technology, industrial, healthcare, Internet of
Things, and other systems
Definition of Big Data

Big data refers to

• datasets whose size is typically beyond the storage capacity
of and also
• complex for traditional database software tools

Big data is anything beyond the human & technical

infrastructure needed to support storage, processing and
analysis.
 Big data is

• high volume, high-velocity and high-variety information

assets that
• demand cost effective, innovative forms of information
processing
• for enhanced insight and decision making.
Part I of the definition:

talks about voluminous data that may have great

variety will require a good speed/pace for
storage, preparation, processing and analysis.

Part II of the definition:

talks about embracing new techniques and
technologies to capture (ingest), store, process,
persist, integrate and visualize the high-volume,
high-velocity, and high-variety data.
Part III of the definition:

talks about deriving deeper, richer and

meaningful insights and then using these
insights to make faster and better
decisions to gain business value and thus a
competitive edge.

Data —> Information —> Actionable

intelligence —> Better decisions
—>Enhanced business value
Below are some key pieces of data from the report:

• Facebook users share nearly 4.16 million pieces of content

• Twitter users send nearly 300,000 tweets
• Instagram users like nearly 1.73 million photos
• YouTube users upload 300 hours of new video content
• Apple users download nearly 51,000 apps
• Skype users make nearly 110,000 new calls
• Amazon receives 4300 new visitors
• Uber passengers take 694 rides
• Netflix subscribers stream nearly 77,000 hours of video
Big Data analytics deals with collection, storage, processing and analysis of this
massive scale data.

Specialized tools and frameworks are required for big data analysis when:

(1) the volume of data involved is so large that it is difficult to store, process
and analyze data on a single machine

(2) the velocity of data is very high and the data needs to be analyzed in
real-time,

(3) there is variety of data involved, which can be structured, unstructured or

semi-structured, and is collected from multiple data sources,

(4) various types of analytics need to be performed to extract value from the
data such as descriptive, diagnostic, predictive and prescriptive analytics
Big data analytics is enabled by several technologies such as cloud computing,
distributed and parallel processing frameworks, non-relational databases,
in-memory computing, for instance.

Some examples of big data are listed as follows:

• Data generated by social networks including text, images, audio and video
data
• Click-stream data generated by web applications such as e-Commerce to
analyze user behavior
• Machine sensor data collected from sensors embedded in industrial and
energy systems
for monitoring their health and detecting failures
• Healthcare data collected in electronic health record (EHR) systems
• Logs generated by web applications
• Stock markets data
Characteristics of Big Data
Volume

Big data is a form of data whose volume is so large that it would not fit on a
single machine therefore specialized tools and frameworks are required to
store process and analyze such data.

The volumes of data generated by modern IT, industrial, healthcare, Internet

of Things, and other systems is growing exponentially driven by the lowering
costs of data storage.

Though there is no fixed threshold for the volume of data to be considered

as big data, however, typically, the term big data is used for massive scale
data that is difficult to store, manage and process using traditional
databases and data processing architectures.
What is Big Data?

Volume

Bits-Bytes-Kilobytes-Megabytes-Gigabytes-Terabytes-Petabytes-Exabytes-Zettabytes-Yottabytes
A Mountain of Data

1 Kilobyte (KB) = 1000 bytes

1 Megabyte (MB) = 1,000,000 bytes
1 Gigabyte (GB) = 1,000,000,000 bytes
1 Terabyte (TB) = 1,000,000,000,000 bytes
1 Petabyte (PB) = 1,000,000,000,000,000 bytes
1 Exabyte (EB) = 1,000,000,000,000,000,000 bytes
1 Zettabyte (ZB) = 1,000,000,000,000,000,000,000 bytes
1 Yottabyte (YB) = 1,000,000,000,000,000,000,000,000 bytes
Where does this data get generated

-Multitude of sources
- XLS, DOC, Youtube videos, Chat conversations, customer feedback CCTV
coverage

1. Typical Internal Data Sources- Data present within an organization firewall

- Data Storage- File systems, SQL, NoSQL..
- Archives- Achieves of scanned documents , paper archives, customer
correspondence records, patient’s health records, Student admission records and so on.

2. External Data Sources- Data residing an organization firewall

- Public Web: Wikepedia, weather, regulatory, census
3. Both internal and external data sources
Velocity
Velocity of data refers to how fast the data is generated.

Data generated by certain sources can arrive at very high velocities, for example,
social media data or sensor data.

Velocity is another important characteristic of big data and the primary reason
for the exponential growth of data.

High velocity of data results in the volume of data accumulated to become very
large, in short span of time.

Some applications can have strict deadlines for data analysis (such as trading or
online fraud detection) and the data needs to be analyzed in real-time.
Velocit
y

Batch Periodic Near real time Real-time processing

Variety

Variety refers to the forms of the data.

Big data comes in different forms such as structured, unstructured or

semi-structured, including text data, image, audio, video and sensor data.

Big data systems need to be flexible enough to handle such variety of data
Veracity

Veracity refers to how accurate is the data.

To extract value from the data, the data needs to be cleaned to remove
noise.

Data-driven applications can reap the benefits of big data only when the data
is meaningful and accurate.

Therefore, cleansing of data is important so that incorrect and faulty data can
be filtered out.
Value

Value of data refers to the usefulness of data for the intended purpose.

The end goal of any big data analytics system is to extract value from
the data.

The value of the data is also related to the veracity or accuracy of the
data.

For some applications value also depends on how fast we are able to
process the data.
Analytics is a broad term that encompasses the processes,
technologies, frameworks and algorithms to extract meaningful
insights from data.

Analytics is this process of extracting and creating information from

raw data by filtering, processing, categorizing, condensing and
contextualizing the data.

This information obtained is then organized and structured to infer

knowledge about the system and/or its users, its environment, and
its operations and progress towards its objectives, thus making the
systems smarter and more efficient.

The choice of the technologies, algorithms, and

frameworks for analytics is driven by the analytics
The goals of the analytics task may be:

(1) to predict something (for example whether a transaction is a

fraud or not, whether it will rain on a particular day, or whether a
tumor is benign or malignant),

(2) to find patterns in the data (for example, finding the top 10
coldest days in the year, finding which pages are visited the most
on a particular website, or finding the most searched celebrity in a
particular year),

(3) finding relationships in the data (for example, finding similar

news articles, finding similar patients in an electronic health record
system
The National Research Council [1] has done a characterization of
computational tasks for massive data analysis (called the seven
“giants").

These computational tasks include:

(1) Basis Statistics, (2) Generalized N-Body Problems, (3) Linear
Algebraic Computations, (4) Graph-Theoretic Computations, (5)
Optimization,
(6) Integration and (7) Alignment Problems.

This characterization of computational tasks aims to provide a taxonomy

of tasks that have proved to be useful in data analysis and grouping
them roughly according to mathematical structure and computational
strategy.
Descriptive Analytics

Descriptive analytics comprises analyzing past data to present it in a

summarized form which can be easily interpreted. Descriptive analytics
aims to answer –
What has happened?

A major portion of analytics done today is descriptive analytics through

use of statistics functions such as counts, maximum, minimum, mean,
topN,percentage, for instance.

These statistics help in describing patterns in the data and present the
data in a summarized form.

For example, computing the total number of likes for a particular post,
computing the average monthly rainfall or finding the average number
Diagnostic Analytics

Diagnostic analytics comprises analysis of past data to diagnose the

reasons as to why certain events happened.

Diagnostic analytics aims to answer - Why did it happen?

Let us consider an example of a system that collects and analyzes sensor

data from machines for monitoring their health and predicting failures.

While descriptive analytics can be useful for summarizing the data by

computing various statistics (such as mean, minimum, maximum,
variance, or top-N), diagnostic analytics can provide more insights into
why certain a fault has occurred based on the patterns in the sensor
data for previous faults.
Predictive Analytics

Predictive analytics comprises predicting the occurrence of an event or

the likely outcome of an event or forecasting the future values using
prediction models.

Predictive analytics aims to answer - What is likely to happen?

For example, predictive analytics can be used for predicting when a fault
will occur in a machine, predicting whether a tumor is benign or
malignant, predicting the occurrence of natural emergency (events
such as forest fires or river floods) or forecasting the pollution levels.

Predictive Analytics is done using predictive models which are trained by

existing data. These models learn patterns and trends from the existing
data and predict the occurrence of an event
Prescriptive Analytics
While predictive analytics uses prediction models to predict the
likely outcome of an event, prescriptive analytics uses multiple
prediction models to predict various outcomes and the best
course of action for each outcome.

Prescriptive analytics aims to answer - What can we do to make

it happen?

Prescriptive Analytics can predict the possible outcomes based

on the current choice of actions.
Prescriptive analytics can be used to prescribe the best
medicine for treatment of a patient based on the outcomes of
various medicines for similar patients.
Another example of prescriptive analytics would be to suggest the
best mobile data plan for a customer based on the customer’s
browsing patterns
 Domain Specific Examples of Big Data
The applications of big data span a wide range of domains including (but not
limited to) homes, cities, environment, energy systems, retail, logistics, industry,
agriculture, Internet of Things, and healthcare.

Various applications of big data for each of these domains are:

Web
• Web Analytics: Web analytics deals with collection and analysis of data on
the user visits on websites and cloud applications.

Analysis of this data can give insights about the user engagement and tracking
the performance of online advertisement campaigns.

For collecting data on user visits, two approaches are used.

In the first approach, user visits are logged on the web server which collects data
such as the date and time of visit, resource requested, user’s IP address, HTTP
status code, for instance.

The second approach, called page tagging, uses a JavaScript which is embedded
in the web page.

The benefit of the page tagging approach is that it facilitates real-time data
collection and analysis.
This approach allows third party services, which do not have access to the web
server (serving the website) to collect and process the data.

These specialized analytics service providers (such as Google Analytics) are offer
advanced analytics and summarized reports. user sessions, page visits, top
entry and exit pages, bounce rate, most visited page
Performance Monitoring: Multi-tier web and cloud applications such as such as
• e-Commerce,
• Business-to-Business,
• Health care, Banking and Financial,
• Retail and Social Networking applications

can experience rapid changes in their workloads.

Provisioning and capacity planning is a challenging task for complex multi-tier

applications since each class of applications has different deployment
configurations with web servers, application servers and database servers.
For performance monitoring, various types of tests can be performed such as

• load tests (which evaluate the performance of the system with multiple users
and workload levels )
• Stress test etc

Big data systems can be used to analyze the data generated by such
tests, to predict application performance under heavy workloads
and identify bottlenecks in the system so that failures can be
prevented.
• Ad Targeting & Analytics:

Search and display advertisements are the two most widely used approaches
for Internet advertising.

In search advertising, users are displayed advertisements ("ads"), along with

the search results, as they search for specific keywords on a search engine.

Advertisers can create ads using the advertising networks provided by the
search engines or social media networks.

These ads are setup for specific keywords which are related to the product or
service being advertised.

Users searching for these keywords are shown ads along with the search
results.
Display advertising, is another form of Internet advertising, in which the ads are
displayed within websites, videos and mobile applications who participate in
the advertising network

The ad-network matches these ads against the content on the website, video or
mobile application and places the ads.

The most commonly used compensation method for Internet ads is

Pay-per-click (PPC), in which the advertisers pay each time a user clicks on an
advertisement.

Advertising networks use big data systems for matching and placing
advertisements and generating advertisement statistics reports.
• Advertises can use big data tools for tracking the performance of
advertisements, optimizing the bids for pay-per-click advertising, tracking
which keywords link the most to the advertising landing page

• Content Recommendation: Content delivery applications that serve content

(such as music and video streaming applications), collect various types of
data such as user search patterns and browsing history, history of content
consumed, and user ratings.

Such applications can leverage big data systems for recommending new
content to the users based on the user preferences and interests.
Financial
• Credit Risk Modeling: Banking and Financial institutions use credit risk
modeling to score credit applications and predict if a borrower will default or
not in the future.

Credit risk models are created from the customer data that includes, credit
scores obtained from credit bureaus, credit history, account balance data,
account transactions data and spending patterns of the customer.

Big data systems can help in computing credit risk scores of a large number of
customers on a regular basis.

These frameworks can be used to build credit risk models by analysis of

customer data
•Fraud Detection: Banking and Financial institutions can leverage big data
systems for detecting frauds such as credit card frauds, money laundering
and insurance claim frauds.

Real-time big data analytics frameworks can help in analyzing data from
disparate sources and label transactions in real-time
• Healthcare

The healthcare ecosystem consists of numerous entities including healthcare

providers (primary care physicians, specialists, or hospitals), payers
(government, private health insurance companies, employers), pharmaceutical,
device and medical service companies, IT solutions and services firms, and
patients.

The process of provisioning healthcare involves massive healthcare data that

exists in different forms (structured or unstructured), is stored in disparate
data sources (such as relational databases, or file servers) and in many different
formats.

To promote more coordination of care across the multiple providers involved

with patients, their clinical information is increasingly aggregated from diverse
sources into Electronic Health Record (EHR) systems.
EHRs capture and store information on patient health and provider actions
including individual-level laboratory results, diagnostic, treatment, and
demographic data.

Though the primary use of EHRs is to maintain all medical data for an individual
patient and to provide efficient access to the stored data at the point of care,
EHRs can be the source for valuable aggregated information about overall
patient populations.

Big data systems can be used for data collection from different stakeholders
(patients, doctors, payers, physicians, specialists, etc) and disparate data sources.

Big data analytics systems allow massive scale clinical data analytics and
facilitate development of more efficient healthcare applications, improve the
accuracy of predictions and help in timely decision making
• Internet of Things
Internet of Things (IoT) refers to things that have unique identities and are
connected to the Internet.

The "Things" in IoT are the devices which can perform remote sensing, actuating
and monitoring.

IoT devices can exchange data with other connected devices and applications
(directly or indirectly), or collect data from other devices and process the data.

IoT systems can leverage big data technologies for storage and analysis of data.
IoT applications that can benefit from big data system
• Intrusion Detection
• Smart Parking
• Smart Roads
• Structural Health Monitoring
• Smart Irrigation
Intrusion Detection: Intrusion detection systems use security cameras and
sensors (such as PIR sensors and door sensors) to detect intrusions and raise
alerts.

Smart Parkings: Smart parkings make the search for parking space easier and
convenient for drivers. Smart parkings are powered by IoT systems that detect
the number of empty parking slots and send the information over the
Internet to smart parking application back-ends

Smart Roads: Smart roads equipped with sensors can provide information on
driving conditions, travel time estimates and alerts in case of poor driving
conditions, traffic congestions and accidents

.
Structural Health Monitoring: Structural Health Monitoring systems use a
network of sensors to monitor the vibration levels in the structures such as
bridges and buildings. The data collected from these sensors is analyzed to
assess the health of the structures

Smart Irrigation: Smart irrigation systems can improve crop yields while
saving water. Smart irrigation systems - use IoT devices with soil moisture
sensors - determine the amount of moisture in the soil and release the flow
of water -when the moisture levels go below a predefined threshold.
Environment
Environment monitoring systems generate high velocity and high volume data.

Accurate and timely analysis of such data can help in understanding the current
status of the environment and also predicting environmental trends.

Weather Monitoring : Weather monitoring systems can collect data from a

number of sensor attached. This data can then be analyzed and visualized for
monitoring weather and generating weather alerts

Air Pollution Monitoring: Air pollution monitoring systems can monitor emission
of harmful gases (CO2, CO, NO, or NO2) by factories and automobiles using
gaseous and meterological sensor
Noise Pollution Monitoring: Due to growing urban development, noise levels in
cities have increased and even become alarmingly high in some cities
Urban noise maps can help the policy makers in urban planning and making
policies to control noise levels near residential areas, schools and parks

Forest Fire Detection: There can be different causes of forest fires including
lightening, human negligence, volcanic eruptions and sparks from rock falls
Forest fire detection systems use a number of monitoring nodes deployed at
different locations in a forest.
River Floods Detection: River flood monitoring system use a number of sensor
nodes that monitor the water level (using ultrasonic sensors) and flow rate
(using the flow velocity sensors).

Big data systems can be used to collect and analyze data from a number of
such sensor nodes and raise alerts when a rapid increase in water level and
flow rate is detected.
Logistics & Transportation
• Real-time Fleet Tracking: Vehicle fleet tracking systems use GPS
technology to track the locations of the vehicles in real-time.
Big data systems can be used to aggregate and analyze vehicle locations
and routes data for detecting bottlenecks in the supply chain such as
traffic congestions on routes, assignment and generation of alternative
routes

Shipment Monitoring:monitoring the conditions inside containers- food

spoilage

Remote Vehicle Diagnostics: Remote vehicle diagnostic systems can detect

faults in the vehicles or warn of impending faults.
Route Generation & Scheduling: Modern transportation systems are driven by
data collected from multiple sources which is processed to provide new
services to the stakeholders. such as advanced route guidance, dynamic vehicle
routing, anticipating customer demands for pickup and delivery problem

Hyper-local Delivery: Hyper-local delivery platforms are being increasingly used

by businesses such as restaurants and grocery stores to expand their reach.
These platforms allow customers to order products (such as grocery and food
items) using web and mobile applications and the products are sourced from
local stores

Cab/Taxi Aggregators: On-demand transport technology aggregators (or

cab/taxi aggregators) allow customers to book cabs using web or mobile
applications and the requests are routed to nearest available cabs
Industry
• Machine Diagnosis & Prognosis: Machine prognosis refers to predicting the
performance of a machine by analyzing the data on the current operating
conditions and the deviations from the normal operating conditions.
Machine diagnosis refers to determining the cause of a machine fault.
Industrial machines have a large number of components that must function
correctly for the machine to perform its operations. Sensors in machines can
monitor the operating conditions

Risk Analysis of Industrial Operations: In many industries, there are strict

requirements on the environment conditions and equipment working conditions
Harmful and toxic gases such as carbon monoxide (CO), nitrogen monoxide (NO),
Nitrogen Dioxide (NO2), for instance, can cause serious health problems. Gas
monitoring systems can help in monitoring the indoor air quality using various gas
Production Planning and Control: Production planning and control systems
measure various parameters of production processes and control the entire
production process in real-time
Retail
Retailers can use big data systems for boosting sales, increasing profitability
and improving customer satisfaction.

Inventory Management: Inventory management -increasingly important in the

recent years with the growing competition. While over-stocking of products can
result in additional storage expenses and risk -under-stocking can lead to loss
of revenue.

Tags attached to the products allow them to be tracked in real-time so that

the inventory levels can be determined accurately and products which are low
Customer Recommendations: Big data systems can be used to analyze the
customer data (such as demographic data, shopping history, or customer
feedback) and predict the customer preferences

Store Layout Optimization: Big data systems can help in analyzing the data on
customer shopping patterns and customer feedback to optimize the store
layouts

Forecasting Demand: Due to a large number of products, seasonal variations

in demands and changing trends and customer preferences, retailers find it
difficult to forecast demand and sales volumes. Big data systems can be used
to analyze the customer purchase patterns and predict demand and sale
volumes
Case Study: Weather Data Analysis

Using the big data stack for analysis of weather data

To come up with a selection of the tools and frameworks from the

big data stack that can be used for weather data analysis, let us first
come up with the analytics flow for the application
Data Collection

Let us assume, we have multiple weather monitoring stations or

end-nodes equipped with temperature, humidity, wind, and
pressure sensors.

To collect and ingest streaming sensor data generated by the

weather monitoring stations, we can use a publish-subscribe
messaging framework to ingest data for real-time analysis within the
Big Data stack and

Source-Sink connector to ingest data into a distributed filesystem

for batch analysis.
Data Preparation

Since the weather data received from different monitoring stations

can have missing values, use different units and have different
formats, we may need to prepare data for analysis by cleaning,
wrangling, normalizing and filtering the data
Analysis Types

The choice of the analysis types is driven by the requirements of the

application.
Let us say, we want our weather analysis application

• to aggregate data on various timescales (minute, hourly, daily or

monthly)

• to determine the mean, maximum and minimum readings for

temperature, humidity, wind and pressure.
to support interactive querying for exploring the data, for example,
queries such as: finding the day with the lowest temperature in each
month of a year, finding the top-10 most wet days in the year, for
instance.

These type of analysis come under the basic statistics category.

Next, we also want the application to make predictions of certain

weather events, for example, predict the occurrence of fog or haze. For
such an analysis, we would require a classification model.

Additionally, if we want to predict values (such as the amount of

rainfall), we would require a regression model
Analysis Modes

Based on the analysis types determined the previous step, we

know that the analysis modes required for the application will be
batch, real-time and interactive.

Visualizations

The front end application for visualizing the analysis results would
be dynamic and interactive.
Mapping Analysis Flow to Big Data Stack
Now that we have the analytics flow for the application, let us map the selections at each step of the flow to the big data
stack
Figure shows a subset of the components of the big data stack based
on the analytics flow.

To collect and ingest streaming sensor data generated by the

weather monitoring stations, we can use a publish-subscribe
messaging framework such as Apache Kafka (for real-time analysis
within the Big Data stack).

Each weather station publishes the sensor data to Kafka

Real-time analysis frameworks such as Storm and Spark Streaming

can receive data from Kafka for processing
For batch analysis, we can use a source-sink connector such as Flume
to move the data to HDFS.

Once the data is in HDFS, we can use batch processing frameworks

such as Hadoop-MapReduce

While the batch and real-time processing frameworks are useful

when the analysis requirements and goals are known upfront,
interactive querying tools can be useful for exploring the data.
We can use interactive querying framework such as Spark SQL,
which can query the data in HDFS for interactive queries.

For presenting the results of batch and real-time analysis, a

NoSQL database such as DynamoDB can be used as a serving
database.

For developing web applications and displaying the analysis

results we can use a web framework such as Django.