1.

INTRODUCTION TO BIGDATA PLATFORM

1.1 Introduction Data

1.1.1 Data and Information

 Data are plain facts.
 The word "data" is plural for "datum."
 Data is nothing but facts and statistics stored or free flowing over a network, generally
it's raw and unprocessed.
 When data are processed, organized, structured or presented in a given context so as to
make them useful, they are called Information.
 It is not enough to have data (such as statistics on the economy).
 Data themselves are fairly useless, but when these data are interpreted and processed to
determine its true meaning, they becomes useful and can be called Information.

For example: When you visit any website, they might store you IP address, that is data,
in return they might add a cookie in your browser, marking you that you visited the
website, that is data, your name, it's data, your age, it's data.

• What is Data?
– The quantities, characters, or symbols on which operations are performed by a
computer,
– which may be stored and transmitted in the form of electrical signals and
– recorded on magnetic, optical, or mechanical recording media.

• 3 Actions on Data
– Capture
– Transform
– Store

BigData

• Big Data may well be the Next Big Thing in the IT world.
• Big data burst upon the scene in the first decade of the 21st century.

1
 • The first organizations to embrace it were online and startup firms.
• Firms like Google, eBay, LinkedIn, and Facebook were built around big data from the
beginning.
• Like many new information technologies,
• big data can bring about dramatic cost reductions,
• substantial improvements in the time required to perform a computing task, or
• new product and service offerings.

• Walmart handles more than 1 million customer transactions every hour.

• Facebook handles 40 billion photos from its user base.
• Decoding the human genome originally took 10years to process; now it can be achieved
in one week.

• What is Big Data?

– Big Data is also data but with a huge size.
– Big Data is a term used to describe a collection of data that is huge in size and yet
growing exponentially with time.
– In short, such data is so large and complex that none of the traditional
data management tools are able to store it or process it efficiently.

No single definition; here is from Wikipedia:

• Big data is the term for
– a collection of data sets so large and complex that it becomes difficult to process
using on-hand database management tools or traditional data processing
applications.

Examples of Bigdata

• Following are some the examples of Big Data-

– The New York Stock Exchange generates about one terabyte of new trade data
per day.
– Other examples of Big Data generation includes
• stock exchanges,
• social media sites,
• jet engines,
• etc.

2
Types Of Big Data

• Big Data could be found in three forms:

1. Structured
2. Unstructured
3. Semi-structured

What is Structured Data?

• Any data that can be stored, accessed and processed in the form of fixed format is
termed as a 'structured' data.
• Developed techniques for working with such kind of data (where the format is well
known in advance) and also deriving value out of it.
• Foreseeing issues of today :
– when a size of such data grows to a huge extent, typical sizes are being in the rage
of multiple zetta bytes.
• Do you know?
• 1021 bytes equal to 1 zettabyte or one billion terabytes forms a zettabyte.
– That is why the name Big Data is given and imagine the challenges involved in
its storage and processing?
• Do you know?
– Data stored in a relational database management system is one example of
a 'structured' data.

• An 'Employee' table in a database is an example of Structured Data:

Employee_ID Employee_Name Gender Department Salary_In_lacs

2365 Rajesh Kulkarni Male Finance 650000

3398 Pratibha Joshi Female Admin 650000

7465 Shushil Roy Male Admin 500000

7500 Shubhojit Das Male Finance 500000

7699 Priya Sane Female Finance 550000

3
Unstructured Data
• Any data with unknown form or the structure is classified as unstructured data.
• In addition to the size being huge,
– un-structured data poses multiple challenges in terms of its processing for
deriving value out of it.
– A typical example of unstructured data is
• a heterogeneous data source containing a combination of simple text files,
images, videos etc.
• Now day organizations have wealth of data available with them but unfortunately,
– they don't know how to derive value out of it since this data is in its raw form or
unstructured format.

• Example of Unstructured data

– The output returned by 'Google Search'

Semi-structured Data

• Semi-structured data can contain both the forms of data.

• Semi-structured data as a structured in form
– but it is actually not defined with e.g. a table definition in relational DBMS.
• Example of semi-structured data is
– a data represented in an XML file.

• Personal data stored in an XML file.

<rec>
<name>Prashant Rao</name>
<sex>Male</sex>
<age>35</age>
</rec>
<rec>
<name>Seema R.</name>
<sex>Female</sex>
<age>41</age>

4
</rec>
<rec>
<name>Satish Mane</name>
<sex>Male</sex>
<age>29</age>
</rec>
<rec>
<name>Subrato Roy</name>
<sex>Male</sex>
<age>26</age>
</rec>
<rec>
<name>Jeremiah J.</name>
<sex>Male</sex>
<age>35</age></rec>

Characteristics of BD OR 3Vs of Big Data

• Three Characteristics of Big Data V3s:

1) Volume
➢ Data quantity
2) Velocity
➢ Data Speed
3) Variety
➢ Data Types

Growth of Big Data

5
Storing Big Data

• Analyzing your data characteristics

– Selecting data sources for analysis
– Eliminating redundant data
– Establishing the role of NoSQL
• Overview of Big Data stores
– Data models: key value, graph, document, column-family
– Hadoop Distributed File System (HDFS)
– Hbase
– Hive

Processing Big Data

• Integrating disparate data stores
– Mapping data to the programming framework
– Connecting and extracting data from storage
– Transforming data for processing
– Subdividing data in preparation for Hadoop MapReduce
• Employing Hadoop MapReduce
6
 – Creating the components of Hadoop MapReduce jobs
– Distributing data processing across server farms
– Executing Hadoop MapReduce jobs
– Monitoring the progress of job flows

Why Big Data?

• Growth of Big Data is needed

– Increase of storage capacities
– Increase of processing power
– Availability of data(different data types)
– Every day we create 2.5 quintillion bytes of data; 90% of the data in the world
today has been created in the last two years alone

• Huge storage needs in Real Time Applications

– FB generates 10TB daily
– Twitter generates 7TB of data Daily
– IBM claims 90% of today’s stored data was generated in just the last two years.

How Is Big Data Different?

1) Automatically generated by a machine (e.g. Sensor embedded in an engine)

2) Typically, an entirely new source of data(e.g. Use of the internet)
3) Not designed to be friendly(e.g. Text streams)
4) May not have much values
– Need to focus on the important part

Big Data sources

• Users
• Application
• Systems
• Sensors

7
 Moved to

• Large and growing files (Big data files)

Risks of Big Data

• Will be so overwhelmed
– Need the right people and solve the right problems
• Costs escalate too fast
– Isn’t necessary to capture 100%
• Many sources of big data is privacy
– self-regulation
– Legal regulation

Leading Technology Vendors

Example Vendors Commonality

IBM – Netezza • MPP architectures

EMC – Greenplum • Commodity Hardware

Oracle – Exadata • RDBMS based

• Full SQL compliance

8
1.1.2 Basics of Bigdata Platform

• Big Data platform is IT solution which combines several Big Data tools and utilities into
one packaged solution for managing and analyzing Big Data.

• Big data platform is a type of IT solution that combines the features and capabilities of
several big data application and utilities within a single solution.
• It is an enterprise class IT platform that enables organization in developing, deploying,
operating and managing a big data infrastructure /environment.

What is Big Data Platform?

• Big Data Platform is integrated IT solution for Big Data management which combines
several software systems, software tools and hardware to provide easy to use tools
system to enterprises.
• It is a single one-stop solution for all Big Data needs of an enterprise irrespective of size
and data volume. Big Data Platform is enterprise class IT solution for developing,
deploying and managing Big Data.
• There are several Open source and commercial Big Data Platform in the market with
varied features which can be used in Big Data environment.
• Big data platform is a type of IT solution that combines the features and capabilities of
several big data application and utilities within a single solution.
• It is an enterprise class IT platform that enables organization in developing,
deploying, operating and managing a big data infrastructure /environment.
• Big data platform generally consists of big data storage, servers, database, big data
management, business intelligence and other big data management utilities
• It also supports custom development, querying and integration with other systems.
• The primary benefit behind a big data platform is to reduce the complexity of multiple
vendors/ solutions into a one cohesive solution.
• Big data platform is also delivered through cloud where the provider provides an all-
inclusive big data solutions and services.

1.1.2.2 Features of Big Data Platform

Here are most important features of any good Big Data Analytics Platform:

9
 a) Big Data platform should be able to accommodate new platforms and tool based on the
business requirement. Because business needs can change due to new technologies or
due to change in business process.
b) It should support linear scale-out
c) It should have capability for rapid deployment
d) It should support variety of data format
e) Platform should provide data analysis and reporting tools
f) It should provide real-time data analysis software
g) It should have tools for searching the data through large data sets

Big data is a term for data sets that are so large or complex that traditional data processing
applications are inadequate.

Challenges include
• Analysis,
• Capture,
• Data Curation,
• Search,
• Sharing,
• Storage,
• Transfer,
• Visualization,
• Querying,
• Updating

Information Privacy.
• The term often refers simply to the use of predictive analytics or certain other
advancedmethods to extract value from data, and seldom to a particular size of data set.

• ACCURACY in big data may lead to more confident decision making, and better
decisions can result in greater operational efficiency, cost reduction and reduced risk.
• Big data usually includes data sets with sizes beyond the ability of commonly used
• software tools to capture, curate, manage, and process data within a tolerable elapsed
• time. Big data "size" is a constantly moving target.
• Big data requires a set of techniques and technologies with new forms of integration to
• reveal insights from datasets that are diverse, complex, and of a massive scale
10
1.1.2.3 List of BigData Platforms
a) Hadoop
b) Cloudera
c) Amazon Web Services
d) Hortonworks
e) MapR
f) IBM Open Platform
g) Microsoft HDInsight
h) Intel Distribution for Apache Hadoop
i) Datastax Enterprise Analytics
j) Teradata Enterprise Access for Hadoop
k) Pivotal HD

a) Hadoop
What is Hadoop?
• Hadoop is open-source, Java based programming framework and server software which
is used to save and analyze data with the help of 100s or even 1000s of commodity
servers in a clustered environment.
• Hadoop is designed to storage and process large datasets extremely fast and in fault
tolerant way.
• Hadoop uses HDFS (Hadoop File System) for storing data on cluster of commodity
computers. If any server goes down it know how to replicate the data and there is no loss
of data even in hardware failure.
• Hadoop is Apache sponsored project and it consists of many software packages which
runs on the top of the Apache Hadoop system.
• Top Hadoop based Commercial Big Data Analytics Platform
• Hadoop provides set of tools and software for making the backbone of the Big Data
analytics system.
• Hadoop ecosystem provides necessary tools and software for handling and analyzing
Big Data.
• On the top of the Hadoop system many applications can be developed and plugged-in to
provide ideal solution for Big Data needs.

11
b) Cloudera

• Cloudera is one of the first commercial Hadoop based Big Data Analytics
Platform offering Big Data solution.
• Its product range includes Cloudera Analytic DB, Cloudera Operational DB, Cloudera
Data Science & Engineering and Cloudera Essentials.
• All these products are based on the Apache Hadoop and provides real-time processing
and analytics of massive data sets.

Website: https://www.cloudera.com

c) Amazon Web Services

• Amazon is offering Hadoop environment in cloud as part of its Amazon Web Services
package.
• AWS Hadoop solution is hosted solution which runs on Amazon’s Elastic Cloud
Compute and Simple Storage Service (S3).
• Enterprises can use the Amazon AWS to run their Big Data processing analytics in the
cloud environment.
• Amazon EMR allows companies to setup and easily scale Apache Hadoop, Spark,
HBase, Presto, Hive, and other Big Data Frameworks using its cloud hosting
environment.
Website: https://aws.amazon.com/emr/

d) Hortonworks
• Hortonworks is using 100% open-source software without any propriety software.
Hortonworks were the one who first integrated support for Apache HCatalog.
• The Hortonworks is a Big Data company based in California.
• This company is developing and supports application for Apache Hadoop.
Hortonworks Hadoop distribution is 100% open source and its enterprise ready with following
features:

• Centralized management and configuration of clusters

• Security and data governance are built in feature of the system
• Centralized security administration across the system
Website: https://hortonworks.com/

12
e) MapR
• MapR is another Big Data platform which us using the Unix file system for handling
data.
• It is not using HDFS and this system is easy to learn anyone familiar with the Unix
system.
• This solution integrates Hadoop, Spark, and Apache Drill with a real-time data
processing feature.
• Website: https://mapr.com

f) IBM Open Platform

• IBM also offers Big Data Platform which is based on the Hadoop eco-system software.
• IBM well knows company in software and data computing.
It uses the latest Hadoop software and provides following features (IBM Open Platform
Features):

• Based on 100% Open source software

• Native support for rolling Hadoop upgrades
• Support for long running applications within YEARN.
• Support for heterogeneous storage which includes HDFS for in-memory and SSD in
addition to HDD
• Native support for Spark, developers can use Java, Python and Scala to written program
• Platform includes Ambari, which is a best tool for provisioning, managing & monitoring
Apache Hadoop clusters
• IBM Open Platform includes all the software of Hadoop ecosystem e.g. HDFS, YARN,
MapReduce, Ambari, Hbase, Hive, Oozie, Parquet, Parquet Format, Pig, Snappy, Solr,
Spark, Sqoop, Zookeeper, Open JDK, Knox, Slider
• Developer can download the trial Docker Image or Native installer for testing and
learning the system
• Application is well supported by IBM technology team

Website: https://www.ibm.com/analytics/us/en/technology/hadoop/

13
g) Microsoft HDInsight
• The Microsoft HDInsight is also based on the Hadoop distribution and it’s a commercial
Big Data platform from Microsoft.
• Microsoft is software giant which is into development of windows operating system for
Desktop users and Server users.
• This is the big Hadoop distribution offering which runs on the Windows and Azure
environment.
• It offer customized, optimized open source Hadoop based analytics clusters which uses
Spark, Hive, MapReduce, HBase, Strom, Kafka and R Server which runs on the Hadoop
system on windows/Azure environment.

Website: https://azure.microsoft.com/en-in/services/hdinsight/

h) Distribution for Apache Hadoop

• Intel also offers its package distribution of Hadoop software which includes company’s
Graph builder and Analytics toolkit.
• This distribution can be purchased with various channel partners and come with support
and yearly subscription.

Website: http://www.intel.com/content/www/us/en/software/intel-distribution-for-apache-
hadoop-software-solutions.html

i) Datastax Enterprise Analytics

• Datastax Enterprise Analytics is another play in the Big Data Analytics platform which
offers its own distribution which is based on Apache Cassandra database management
system which runs on the top of Apache Hadoop installation.
• It also included propriety system with a dashboard which is used for security
management, searching data, dashboard for viewing various details and visualization
engine.
• It can handle analysis of 10 million data points every second, so it’s a powerful system.
Features:

• It provides powerful indexing, search, analytics and graph functionality into the Big
Data system
• It supports advanced indexing and searching features

14
 • It comes with powerful integrated analytics system
• It provides multi-model support into the platform. It supports key-value, tabular,
JSON/Document and graph data formats. Powerful search features enables the users to
get required data in real-time
Website: http://www.datastax.com/

j) Teradata Enterprise Access for Hadoop

• Teradata Enterprise Access for Hadoop is another player into Big Data Platform and it
offers package Hadoop distribution which again based on Hortonworks distribution.
• Teradata Enterprise Access for Hadoop offers Hardware and software in its Big Data
solution which can be used by enterprise to process its data sets.
Company offers:

• Teradata
• Teradata Aster and
• Hadoop
as part of its package solution.

Website: http://www.teradata.com

k) Pivotal HD
Pivotal HD offers is another Hadoop distribution with includes includes database tools
Greenplum and analytics platform Gemfire.
Features:

• It can be installed on-premise and in public clouds

• This system is based on the open source software
• It supports data evolution within the 3 years subscription period.
Indian railways, BMW, China Citic Bank and many other big players are using this distribution
of Big Data Platform.

15
Website: https://pivotal.io/

1.1.3 Open Source Big Data Platform

There are various open-source Big Data Platform which can be used for Big Data handling and
data analytics in real-time environment.
Both small and Big Enterprise can use these tools for managing their enterprise data for getting
best value from their enterprise data.

i) Apache Hadoop
• Apache Hadoop is Big Data platform and software package which is Apache sponsored
project.
• Under Apache Hadoop project various other software is being developed which runs on
the top of Hadoop system to provide enterprise grade data management and analytics
solutions to enterprise.
• Apache Hadoop is open-source, distributed file system which provides data processing
and analysis engine for analyzing large set of data.
• Hadoop can run on Windows, Linux and OS X operating systems, but it is mostly used
on Ubunut and other Linux variants.

ii) MapReduce
• The MapReduce engine was originally written by Google and this is the system which
enables the developers to write program which can run in parallel on 100 or even 1000s
of computer nodes to process vast data sets.
• After processing all the job on the different nodes it comes the results and return it to the
program which executed the MapReduce job.
• This software is platform independent and runs on the top of Hadoop ecosystem. It can
process tremendous data at very high speed in Big Data environment.

iii) GridGain
• GridGain is another software system for parallel processing of data just like MapRedue.
GridGain is an alternative of Apache MapReduce.
16
 • GridGain is used for the processing of in-memory data and its is based on Apache Iginte
framework.
• GridGain is compatable with the Hadoop HDFS and runs on the top of Hadoop
ecosystem.
• Then enterprise version of GridGain can be purchased from official website of
GridGain. While free version can be downloaded from GitHub repository.

Website: https://www.gridgain.com/

iv) HPCC Systems

• HPCC Systems stands for "high performance computing cluster” and this system is
developed by LexisNexis Risk Solutions.
• According to the company this software is much faster than Hadoop and can be used in
the cloud environment.
• HPCC Systems is developed in C++ and compiled into binary code for distribution.
• HPCC Systems is open-source, massive parallel processing system which is installed in
cluster to process data in real-time.
• It requires Linux operating system and runs on the commodity servers connected with
high-speed network.
• It is scalable from one node to 1000s of nodes to provide performance and scalability.
• Website: https://hpccsystems.com/

v) Apache Storm
• Apache Storm is a software for real-time computing and distributed processing.
• Its free and open-source software developed at Apache Software foundation. It’s a real-
time, parallel processing engine.
• Apache Storm is highly scalable, fault-tolerant which supports almost all the
programming language.

vi) Apache Strom can be used in:

• Realtime analytics
• Online machine learning

17
 • Continuous computation
• Distributed RPC
• ETL
• And all other places where real-time processing is required.
Apache Strom is used by Yahoo, Twitter, Spotify, Yelp, Flipboard and many other data giants.

Website: http://storm.apache.org/

vii) Apache Spark

• Apache Spark is software that runs on the top of Hadoop and provides API for real-time,
in-memory processing and analysis of large set of stored in the HDFS.
• It stores the data into memory for faster processing.
• Apache Spark runs program 100 times faster in-memory and 10 times faster on disk as
compared to the MapRedue.
• Apache Spark is here to faster the processing and analysis of big data sets in Big Data
environment.
• Apache Spark is being adopted very fast by the business to analyze their data set to get
real value of their data.
• Website: http://spark.apache.org/

viii) SAMOA
• SAMOA stands for Scalable Advanced Massive Online Analysis,
• It’s a system for mining the Big Data streams.
• SAMOA is open-source software distributed at GitHub, which can be used as distributed
machine learning framework also.
• Website: https://github.com/yahoo/samoa
Thus, the Big Data industry is growing very fast in 2017 and companies are fast moving
their data to Big Data Platform. There is huge requirement of Big Data in the job market;
many companies are providing training and certifications in Big Data technologies.
*********************

18
1.2. CHALLENGES OF CONVENTIONAL SYSTEMS

1.2.1 Introduction to Conventional Systems

What is Conventional System?

Conventional Systems.
• The system consists of one or more zones each having either manually operated call
points or automatic detection devices, or a combination of both.
• Big data is huge amount of data which is beyond the processing capacity
ofconventional data base systems to manage and analyze the data in a specific time
interval.

Difference between conventional computing and intelligent computing

• The conventional computing functions logically with a set of rules and calculations
while the neural computing can function via images, pictures, and concepts.
• Conventional computing is often unable to manage the variability of data obtained in the
real world.
• On the other hand, neural computing, like our own brains, is well suited to situations that
have no clear algorithmic solutions and are able to manage noisy imprecise data. This
allows them to excel in those areas that conventional computing often finds difficult.

1.2.2 Comparison of Big Data with Conventional Data

Big Data Conventional Data

Huge data sets Data set size in control.
Unstructured data such as text, video, Normally structured data such as numbers
and audio. and categories, but it can take other forms
as well.

Hard-to-perform queries and analysis Relatively easy-to-perform queries and

analysis.
Needs a new methodology for analysis. Data analysis can be achieved by using
conventional methods.
Need tools such as Hadoop, Hive, Tools such as SQL, SAS, R, and Excel
Hbase, Pig, Sqoop, and so on. alone may be sufficient.

19
The aggregated or sampled or filtered data. Raw transactional data.

Used for reporting, basic analysis, and Used for reporting, advanced analysis, and
text mining. Advanced analytics is only in predictive modeling .
a starting stage in big data.
Big data analysis needs both Analytical skills are sufficient for
programming skills (such as Java) and conventional data; advanced analysis tools
analytical skills to perform analysis. don’t require expert programing skills.

Petabytes/exabytes of data. Millions/billions of accounts.

Billions/trillions of transactions. Megabytes/gigabytes of data.

Thousands/millions of accounts. Millions of transactions

Generated by big financial institutions, Generated by small enterprises and small

Facebook, Google, Amazon, eBay, banks.
Walmart, and so on.

1.2.2 List of challenges of Conventional Systems

The following list of challenges has been dominating in the case Conventional systems in real
time scenarios:

1) Uncertainty of Data Management Landscape

2) The Big Data Talent Gap
3) The talent gap that exists in the industry Getting data into the big data platform
4) Need for synchronization across data sources
5) Getting important insights through the use of Big data analytics

1) Uncertainty of Data Management Landscape:

• Because big data is continuously expanding, there are new companies and technologies
that are being developed everyday.

20
 • A big challenge for companies is to find out which technology works bests for them
without the introduction of new risks and problems.

2) The Big Data Talent Gap:

• While Big Data is a growing field, there are very few experts available in this field.
• This is because Big data is a complex field and people who understand the complexity
and intricate nature of this field are far few and between.

3) The talent gap that exists in the industry Getting data into the big data platform:
• Data is increasing every single day. This means that companies have to tackle limitless
amount of data on a regular basis.
• The scale and variety of data that is available today can overwhelm any data practitioner
and that is why it is important to make data accessibility simple and convenient for
brand mangers and owners.

4) Need for synchronization across data sources:

• As data sets become more diverse, there is a need to incorporate them into an analytical
platform.
• If this is ignored, it can create gaps and lead to wrong insights and messages.

5) Getting important insights through the use of Big data analytics:

• It is important that companies gain proper insights from big data analytics and it is
important that the correct department has access to this information.
• A major challenge in the big data analytics is bridging this gap in an effective fashion.

Other Three challenges of Conventional systems

Three Challenges That big data face.

1. Data
2. Process
3. Management

1. Data Challenges
21
Volume

1. The volume of data, especially machine-generated data, is exploding,

2. how fast that data is growing every year, withnew sources of data that are emerging.
3. For example, in the year 2000, 800,000petabytes (PB) of data were stored in the world, and it
is expected to reach 35 zetta bytes (ZB) by2020 (according to IBM).

Social media plays a key role: Twitter generates 7+ terabytes (TB) of data every day. Facebook,
10 TB.
• Mobile devices play a key role as well, as there were estimated 6 billion mobile phones in
2011.
• The challenge is how to deal with the size of Big Data.

Variety, Combining Multiple Data Sets

• More than 80% of today’s information is unstructured and it is typically too big to manage
effectively.
• Today, companies are looking to leverage a lot more•data from a wider variety of sources both
inside and outside the organization.
• Things like documents, contracts, machine data, sensor data, social media, health records,
emails, etc. The list is endless really.

Variety•A lot of this data is unstructured, or has a complex structure that’s hard to represent in
rows and columns.

2. Processing

• More than 80% of today’s information isunstructured and it is typically too big to
manage effectively.

• Today, companies are looking to leverage a lot more data from a wider variety of
sources both inside and outside the organization.

• Things like documents, contracts, machine data, sensor data, social media, health
records, emails, etc. The list is endless really.

22
3. Management

• A lot of this data is unstructured, or has acomplex structure that’s hard to represent in
rows and columns.

Big Data Challenges

– The challenges include capture, duration, storage, search, sharing, transfer,

– analysis, and visualization.

• Big Data is trend to larger data sets

• due to the additional information derivable from analysis of a single large set of related
data,
– as compared to separate smaller sets with the same total amount of data, allowing
correlations to be found to
• "spot business trends, determine quality of research, prevent diseases, link
legal citations, combat crime, and determine real-time roadway traffic
conditions.”

Challenges of Big Data

The following are the five most important challenges of the Big Data

a) Meeting the need for speed

In today’s hypercompetitive business environment, companies not only have to find and
analyze the relevant data they need, they must find it quickly.

b) Visualization helps organizations perform analyses and make decisions much more
rapidly, but the challenge is going through the sheer volumes of data and accessing the
level of detail needed, all at a high speed.

c) The challenge only grows as the degree of granularity increases. One possible solution
is hardware. Some vendors are using increased memory and powerful parallel
processing to crunch large volumes of data extremely quickly
23
d) Understanding the data
• It takes a lot of understanding to get data in the RIGHT SHAPE so that you can use
• visualization as part of data analysis.

d) Addressing data quality

• Even if you can find and analyze data quickly and put it in the proper context for the
• audience that will be consuming the information, the value of data for DECISION-
MAKING PURPOSES will be jeopardized if the data is not accurate or timely.
This is a challenge with any data analysis.

e) Displaying meaningful results

• Plotting points on a graph for analysis becomes difficult when dealing with extremely
• large amounts of information or a variety of categories of information.
• For example, imagine you have 10 billion rows of retail SKU data that you’re trying to
• compare. The user trying to view 10 billion plots on the screen will have a hard time
• seeing so many data points.
• . By grouping the data together, or “binning,” you can more effectively visualize the
data.

f) Dealing with outliers

• The graphical representations of data made possible by visualization can communicate
• trends and outliers much faster than tables containing numbers and text.
• Users can easily spot issues that need attention simply by glancing at a chart. Outliers
typically represent about 1 to 5 percent of data, but when you’re working with massive
amounts of data, viewing 1 to 5 percent of the data is rather difficult
• We can also bin the results to both view the distribution of data and see the outliers.
• While outliers may not be representative of the data, they may also reveal previously
• unseen and potentially valuable insights.

• Visual analytics enables organizations to take raw data and present it in a meaningful
way that generates the most value. However, when used with big data, visualization is
bound to lead to some challenges.
**************

24
1.3. INTELLIGENT DATA ANALYSIS

1.3.1 INTRODUCTION TO INTELLIGENT DATA ANALYSIS (IDA)

Intelligent Data Analysis (IDA) is one of the hot issues in the field of
artificial intelligence and information.

What is Intelligent Data Analysis (IDA)?

IDA is

… an interdisciplinary study concerned with the effective analysis of data;

… used for extracting useful information from large quantities of online data; extracting
desirable knowledge or interesting patterns from existing databases;

➢ the distillation of information that has been collected, classified, organized, integrated,
abstracted and value-added;

➢ at a level of abstraction higher than the data, and information on which it is based and
can be used to deduce new information and new knowledge;

➢ usually in the context of human expertise used in solving problems.

➢ the distillation of information that has been collected, classified, organized, integrated,
abstracted and value-added;

➢ at a level of abstraction higher than the data, and information on which it is based and
can be used to deduce new information and new knowledge;

➢ usually in the context of human expertise used in solving problems.

Goal:

Goal of Intelligent data analysis is to extract useful knowledge, the process demands a
combination of extraction, analysis, conversion, classification, organization, reasoning, and so
on.
25
1,3,2 Uses / Benefits of IDA

Intelligent Data Analysis provides a forum for the examination of issues related to the research
and applications of Artificial Intelligence techniques in data analysis across a variety of
disciplines and the techniques include (but are not limited to):

The benefit areas are:

• Data Visualization
• Data pre-processing (fusion, editing, transformation, filtering, sampling)
• Data Engineering
• Database mining techniques, tools and applications
• Use of domain knowledge in data analysis
• Big Data applications
• Evolutionary algorithms
• Machine Learning(ML)
• Neural nets
• Fuzzy logic
• Statistical pattern recognition
• Knowledge Filtering and
• Post-processing

1.4 NATURE OF DATA

1.4.1 INTRODUCTION
Data

• Data is a set of values of qualitative or quantitative variables; restated, pieces

of data are individual pieces of information.
• Data is measured, collected and reported, and analyzed, whereupon it can be visualized
using graphs or images.

Properties of Data
For examining the properties of data, reference to the various definitions of data.
26
Reference to these definitions reveals that following are the properties of data:
a) Amenability of use
b) Clarity
c) Accuracy
d) Essence
e) Aggregation
f) Compression
g) Refinement

.
a) Amenability of use: From the dictionary meaning of data, it is learnt that data are
facts used in deciding something. In short, data are meant to be used as a base for
arriving at definitive conclusions.

b) Clarity: Data are a crystallized presentation. Without clarity, the meaning desired to be
communicated will remain hidden.

c) Accuracy: Data should be real, complete and accurate. Accuracy is thus, an essential
property of data.

d) Essence: A large quantity of data is collected and they have to be Compressed and
refined. Data so refined can present the essence or derived qualitative value, of the
matter.

e) Aggregation: Aggregation is cumulating or adding up.

f) Compression: Large amounts of data are always compressed to make them more
meaningful. Compress data to a manageable size. Graphs and charts are some
examples of compressed data.

g) Refinement: Data require processing or refinement. When refined, they are capable of
leading to conclusions or even generalizations. Conclusions can be drawn only when
data are processed or refined.

27
1.6 ANALYSIS AND REPORTING

1.6.1 INTRODUCTION TO ANALYSIS AND REPORTING

What is Analysis?
• The process of exploring data and reports
– in order to extract meaningful insights,
– which can be used to better understand and improve business performance.

• What is Reporting ?
• Reporting is
– “the process of organizing data
– into informational summaries
– in order to monitor how different areas of a business are performing.”

1.6.2 COMPARING ANALYSIS WITH REPORTING

• Reporting is “the process of organizing data into informational summaries in order to

monitor how different areas of a business are performing.”
• Measuring core metrics and presenting them — whether in an email, a slidedeck,
or online dashboard — falls under this category.
• Analytics is “the process of exploring data and reports in order to extract meaningful
insights, which can be used to better understand and improve business performance.”
• Reporting helps companies to monitor their online business and be alerted to when data
falls outside of expected ranges.
• Good reporting
• should raise questions about the business from its end users.
• The goal of analysis is
• to answer questions by interpreting the data at a deeper level and providing
actionable recommendations.

• A firm may be focused on the general area of analytics (strategy, implementation,

reporting, etc.)
– but not necessarily on the specific aspect of analysis.
• It’s almost like some organizations run out of gas after the initial set-up-related
activities and don’t make it to the analysis stage
28
A reporting activity deliberately proposes Analysis activity.

1.6.3 CONTRAST BETWEEN ANALYSIS AND REPORTING

The basis differences between Analysis and Reporting are as follows:

Analysis Reporting
Provides what is needed Provides what is asked for
Is typically customized Is Typically standardized
Involves a person Does not involve a person
Is extremely flexible Is fairly Inflexible

• Reporting translates raw data into information.

• Analysis transforms data and information into insights.
• reporting shows you what is happening
• while analysis focuses on explaining why it is happening and what you can do about it.

• Reports are like Robots n monitor and alter you and whereas analysis is like parents -
can figure out what is going on (hungry, dirty diaper, no pacifier, , teething, tired, ear
infection, etc).
• Reporting and analysis can go hand-in-hand:
• Reporting provides no limited context about what is happening in the data. Context is
critical to good analysis.
• Reporting translates a raw data into information
• Reporting usually raises a question – What is happening ?
• Analysis transforms the data into insights - Why is it happening ? What you can do
about it?

Thus, Analysis and Reporting is synonym to each other with respect their need and utilizing
29
in the needy context.

*****************

30