IET - Chapter 2
IET - Chapter 2
Activity 2.1
What is data science? Can you describe the role of
data in emerging technology?
What are data and information?
2
What is big data?
An Overview of Data Science ….Cont’d
Data science is:
A multi-disciplinary field that uses scientific methods,
processes, algorithms and systems to extract knowledge
and insights from structured, semi-structured and
unstructured data.
is much more than simply analyzing data.
It offers a range of roles and requires a range of skills.
3
An Overview of Data Science ….Cont’d
• Let’s consider this idea by thinking about some of the data
supermarket:
When you get to the store, you use your data as a reminder
6
What are Data and Information?
Data
Is a representation of facts, concepts, or instructions in a
formalized manner.
is represented with the help of characters such as alphabets
(A-Z, a-z), digits (0-9) or special characters (+, -, /, *, <,>, =,
etc.).
Whereas information
is the processed data on which decisions and actions are
based.
is data that has been processed into a form that is
meaningful.
is interpreted data; created from organized, structured, and
7
processed data in a particular context.
Data Processing Cycle
8
Figure 2.1 Data Processing Cycle
Data Processing Cycle….Cont’d
• Input − data is prepared in some convenient form for
processing. The form will depend on the processing machine.
• Processing − data is changed to produce data in a more
useful form.
• Output − the result of the proceeding processing step is
collected. The particular form of the output data depends on
the use of the data.
Activity 2.3
Discuss the main differences between data and information with
examples.
Can we process data manually using a pencil and paper? Discuss the
differences with data processing using the computer. 9
Data types and their representation
• A data type is simply an attribute of data that tells the compiler
or interpreter how the programmer intends to use the data.
Data types from Computer programming perspective
Common data types include:
Integers(int)- is used to store whole numbers
Booleans(bool)- is used to represent either true or false.
Characters(char)- is used to store a single character
Floating-point numbers(float)- is used to store real
numbers
Alphanumeric strings(string)- used to store a combination 10
11
14
Activity 2.4
Discuss data types from programing and analytics
perspectives.
Compare metadata with structured, unstructured and
semi-structured data
Given at least one example of structured, unstructured
and semi-structured data types
15
Data value Chain
• is introduced to describe the information flow
within a big data system as a series of steps needed
to generate value and useful insights from data.
• The Big Data Value Chain identifies the following
key high-level activities:
16
Data Acquisition
• process of gathering, filtering, and cleaning data before it is
put in a data warehouse on which data analysis can be carried
out.
• is one of the major big data challenges in terms of infrastructure
requirements.
Data Analysis
• is concerned with making the raw data acquired amenable to
use in decision-making.
• involves exploring, transforming, and modeling data with the
goal of highlighting relevant data, synthesizing and extracting
useful hidden information with high potential from a business
point of view.
17
• Related areas include data mining, business intelligence, and
machine learning.
Data Curation
• active management of data over its life cycle to
ensure it meets the necessary data quality
requirements.
• It can be categorized into different activities such
as content creation, selection, classification,
transformation, validation, and preservation.
• is performed by expert curators.
• The curators have the responsibility of ensuring
that data are trustworthy, discoverable, 18
and almost unique, solution to the storage paradigm for nearly 40 years.
regard to schema changes and the performance and fault tolerance when
• NoSQL technologies have been designed with the scalability goal in mind
20
Activity 2.5
Which information flow step in the data value chain you
think is labor-intensive? Why?
What are the different data types and their value chain?
21
Basic concepts of big data
Big data:
• is a blanket term for the non-traditional strategies and
22
Basic concepts of big data …
• is the term for a collection of data sets so large and complex
process or store.
24
Clustered Computing and Hadoop Ecosystem
Clustered Computing
individual computers are often inadequate for handling the
data at most stages.
To better address this issue computer clusters are a better fit.
Big data clustering software combines the resources of many
smaller machines, seeking to provide a number of benefits:
• Resource Pooling: Combining the available storage space
• High Availability: fault tolerance and availability
guarantees
• Easy Scalability: Clusters make it easy to scale horizontally
• This means the system can react to changes in resource
requirements without expanding the physical resources on
a machine.
25
Clustered Computing… cont’d
26
Activity 2.6
List and discuss the characteristics of big data
Describe the big data life cycle. Which step you think most
useful and why?
List and describe each technology or tool used in the big data
life cycle.
Discuss the three methods of computing over a large dataset.
27
Hadoop and its Ecosystem
Hadoop:
• is an open-source framework.
• intended to make interaction with big data easier.
• allows for the distributed processing of large datasets across
clusters of computers.
• It is inspired by a technical document published by Google.
The four key characteristics of Hadoop are:
• Economical: ordinary computers can be used for data
processing.
• Reliable: stores copies of the data on different machines
and is resistant to hardware failure.
• Scalable: A few extra nodes help in scaling up the
framework.
28
• Flexible: store as much structured and unstructured data
as you need to and decide to use them later.
Hadoop and its Ecosystem …. Cont’d
• Hadoop has an ecosystem that has evolved from its four core
components: data management, access, processing, and storage.
• It comprises the following components and many others:
HDFS: Hadoop Distributed File System
YARN: Yet Another Resource Negotiator
MapReduce: Programming based Data Processing
Spark: In-Memory data processing
PIG, HIVE: Query-based processing of data services
HBase: NoSQL Database
Mahout, Spark MLLib: Machine Learning algorithm
Libraries
Solar, Lucene: Searching and Indexing
Zookeeper: Managing cluster 29
Oozie: Job Scheduling
Figure 2.5 Hadoop Ecosystem
30
Activity 2.5
Students in a group shall discuss the purpose of each Hadoop Ecosystem components?
Big Data Life Cycle with Hadoop
Ingesting data into the system
first stage of Big Data
The data is ingested or transferred to Hadoop from various sources
Sqoop transfers data from RDBMS to HDFS, whereas Flume transfers event
data.
Processing the data in storage
the data is stored and processed.
is stored in the distributed file system, HDFS, and the NoSQL distributed
data, HBase.
Spark and MapReduce perform data processing.
Computing and analyzing data
data is analyzed by processing frameworks such as Pig, Hive, and Impala.
Pig converts the data using a map and reduce and then analyzes it.
Hive is also based on the map and reduce programming and is most
suitable for structured data.
Visualizing the results
is performed by tools such as Hue and Cloudera Search. 31
In this stage, the analyzed data can be accessed by users.
Chapter Two Review Questions (Assignment 10%)
1. Define data science; what are the roles of a data scientist?