Unit 1

Digital data:

Digital data is the electronic representation of information in a format or language that

machines can read and understand. In more technical terms, digital data is a binary format
of information that's converted into a machine-readable digital format.

Today, data undoubtedly is an invaluable asset of any enterprise (big or small). Even
though professionals work with data all the time, the understanding, management and
analysis of data from heterogeneous sources remains a serious challenge.

Digital Data in Big Data

Big data involves vast amounts of digital data, but in a wide variety of formats and
gathered at mind-boggling speeds. Different types of digital data, including batch or
streaming, can be collected and processed for consumption by machines or people,
through big data integration.

Digital data Units:

1 Bit = Binary Digit

·8 Bits = 1 Byte

1024 Bytes = 1 Kilobyte

1024 Kilobytes = 1 Megabyte

1024 Megabytes = 1 Gigabyte

1024 Gigabytes = 1 Terabyte

 1024 Terabytes = 1 Petabyte

1024 Petabytes = 1 Exabyte

1024 Exabytes = 1 Zettabyte

1024 Zettabytes = 1 Yottabyte

1024 Yottabytes = 1 Brontobyte

1024 Brontobytes = 1 Geopbyte

Big Data and Thick Data

 Thick data is qualitative information (such as observations, feelings, and reactions)

that offers insight into consumers’ daily emotional life. Thick data is difficult to
quantify since it seeks to discover people’s emotions, stories, and representations of
the environment they live in. It’s only collected in tiny amounts.

 Big data is quantitative in nature—that is, it works with numbers and figures. Big data
is distinguished by the fact that it is collected on such a huge scale and continues to
increase exponentially over time.

Example of Big Data

1.The New York Stock Exchange is an example of Big Data that generates about one
terabyte of new trade data per day.

2. Social Media: The statistic shows that 500+terabytes of new data get ingested into the
databases of social media site Facebook, every day. This data is mainly generated in
terms of photo and video uploads, message exchanges, putting comments etc.
 3. A single Jet engine can generate 10+terabytes of data in 30 minutes of flight time.
With many thousand flights per day, generation of data reaches up to many Petabytes.

Types of Big Data

Now that we are on track with what is big data, let’s have a look at the types of big data:

 Structured
 Unstructured
 Semi-structured

Structured

 Structured is one of the types of big data. It is the data that can be processed, stored,
and retrieved in a fixed format.
 It refers to highly organized information that can be readily and seamlessly stored and
accessed from a database by simple search engine algorithms.
 For instance, the employee table in a company database will be structured as the
employee details, their job positions, their salaries, etc., will be present in an
organized manner.

Sources of Structured Data

 Unstructured Data

 Unstructured data refers to the data that lacks any specific form or structure
whatsoever.
 This makes it very difficult and time-consuming to process and analyze unstructured
data.
 Email is an example of unstructured data.

Semi-structured Data

 Semi-structured data does not conform to any data model i.e. it is difficult to
determine the meaning of data neither can data be stored in rows and columns as in a
database but semi-structured data has tags and markers which help to group data and
describe how data is stored, giving some metadata but it is not sufficient for
management and automation of data.
 Similar entities in the data are grouped and organized in a hierarchy. The attributes or
the properties within a group may or may not be the same.
 For example two addresses may or may not contain the same number of properties.

History of big data innovation

 In 1663, first serious statistical data analysis was done, when information regarding
impact of bubonic plague such as death rates and other health records were published.
 In 1865, Richard Millar Devens presented the phrase “Business Intelligence” (BI).
Business intelligence uses technology to gather and analyze data, translate it into useful
information, and act on it “before the competition.” Richard Millar describe the case of
Sir Henry Furnese, a banker, profited from information by gathering and acting on it
before his competition

 The beginning of data processing is marked by the invention of punch card tabulating
machine in 1884.

 Nicole Tesla, in 1926, predicted that human will have access to large warehouse of data
through an instrument that could be carried in the pocket.

 In 1959, the term machine learning was coined.

 In the early 2000s, the emergence of the internet and the proliferation of digital devices
led to a massive increase in the amount of data being generated and collected. This, in
turn, created a need for new tools and technologies to store, process, and analyze the data.

 In 2004, Google introduced a new technology called MapReduce, which allowed large-
scale data processing on distributed systems using commodity hardware.

 In 2005, Big Data was labelled by Roger Mougalas as he referred to a large set of data
that, at the time, was almost impossible to manage and process using the traditional
business intelligence tools available.

 In 2006, Hadoop was created to handle big data. Hadoop was based on an open-sourced
software framework called Nutch and was merged with Google’s MapReduce.

 Over the next decade, Big Data technologies continued to evolve, with the development
of NoSQL databases, in-memory computing, and cloud computing, among other
 advancements. These technologies enabled organizations to store, process, and analyze
massive amounts of data, leading to new insights and opportunities for innovation.

 Today, Big Data is a critical component of many industries, including healthcare, finance,
retail, and manufacturing. The rise of artificial intelligence and machine learning has
further accelerated the growth of Big Data, as these technologies require large volumes of
high-quality data to train and improve their models.

 One of the biggest challenges in dealing with Big Data is how to effectively store,
manage, and analyze such vast amounts of information. This requires specialized
software and hardware tools, as well as skilled data scientists and analysts who are able to
extract insights and make sense of the data.

Drivers for Big Data

There are number of reasons that contribute to the rapid increase in importance of big
data. They are called drivers of big data.

Six main business drivers can be identified:

1. The digitization of society;

2. The reduction of technology costs;

3. Connectivity through cloud computing;

4. Increased knowledge about data science;

5. Social media applications;

6. The upcoming Internet-of-Things (IoT).

1. The digitization of society

 Big Data is largely consumer driven and consumer oriented. Most of the data in the world
is generated by consumers, who are nowadays ‘always-on’. Most people now spend 4-6
hours per day consuming and generating data through a variety of devices and (social)
applications.

With every click, swipe or message, new data is created in a database somewhere around
the world. Because everyone now has a smartphone in their pocket, the data creation
sums to incomprehensible amounts. Some studies estimate that 60% of data was
generated within the last two years, which is a good indication of the rate with which
society has digitized.

2. The reduction of technology costs

Technology related to collecting and processing massive quantities of diverse (high

variety) data has become increasingly more affordable. The costs of data storage and
processors keep declining, making it possible for small businesses and individuals to
become involved with Big Data.

3. Connectivity through cloud computing

Cloud computing environments have made it possible to quickly scale up or scale down
IT infrastructure and facilitate a pay-as-you-go model. This means that organizations that
want to process massive quantities of data (and thus have large storage and processing
requirements) do not have to invest in large quantities of IT infrastructure.

Instead, they can license the storage and processing capacity they need and only pay for
the amounts they actually used. As a result, most of Big Data solutions leverage the
possibilities of cloud computing to deliver their solutions to enterprises.

4. Increased knowledge about data science

In the last decade, the term data science and data scientist have become tremendously
popular. The demand for data scientist (and similar job titles) has increased
 tremendously and many people have actively become engaged in the domain of data
science.

5. Social media applications

Everyone understands the impact that social media has on daily life. However, in the
study of Big Data, social media plays a role of paramount importance. Not only because
of the sheer volume of data that is produced everyday through platforms such as Twitter,
Facebook, LinkedIn and Instagram, but also because social media provides nearly real-
time data about human behavior.

Social media data provides insights into the behaviors, preferences and opinions of ‘the
public’ on a scale that has never been known before. Due to this, it is immensely valuable
to anyone who is able to derive meaning from these large quantities of data. Social media
data can be used to identify customer preferences for product development, target new
customers for future purchases, or even target potential voters in elections. Social media
data might even be considered one of the most important business drivers of Big Data.

6. The upcoming internet of things (IoT)

The Internet of things (IoT) is the network of physical devices, vehicles, home appliances
and other items embedded with electronics, software, sensors, actuators, and network
connectivity which enables these objects to connect and exchange data. It is increasingly
gaining popularity as consumer goods providers start including ‘smart’ sensors in
household appliances.

Big Data architecture and characteristics

Big data architecture is a comprehensive solution to deal with an enormous amount of

data. It details the blueprint for providing solutions and infrastructure for dealing with big
data based on a company’s demands. It clearly defines the components, layers, and
methods of communication.
 A reference architecture is a document or set of documents to which a other project
manager or interested party can refer to for best practices.

The objective of reference architecture is to create an open standard, one that every
company can use for their benefit. The National Institute of Standards and Technology,
One of the leading organizations in the development of standard, has developed such a
reference architecture: the NIST Big Data Reference Architecture.

1. Data Sources

The data sources involve all those golden sources from where the data extraction pipeline
is built and therefore this can be said to be the starting point of the big data pipeline.

The examples include:

(i) Datastores of applications such as the ones like relational databases

(ii) The files which are produced by a number of applications and are majorly a
part of static file systems such as web-based server files generating logs.
 (iii) IoT devices and other real time-based data sources.

2. Data Storage

This includes the data which is managed for the batch built operations and is stored in the
file stores which are distributed in nature and are also capable of holding large volumes
of different format backed big files. It is called the data lake. This generally forms the
part where our Hadoop storage such as HDFS, Microsoft Azure, AWS, GCP storages are
provided along with blob containers.

3. Batch Processing

All the data is segregated into different categories or chunks which makes use of long-
running jobs used to filter and aggregate and also prepare data o processed state for
analysis. These jobs usually make use of sources, process them and provide the output of
the processed files to the new files. The batch processing is done in various ways by
making use of Hive jobs or U-SQL based jobs or by making use of Sqoop or Pig along
with the custom map reducer jobs which are generally written in any one of the Java or
Scala or any other language such as Python.

4. Real Time-Based Message Ingestion

This includes, in contrast with the batch processing, all those real-time streaming systems
which cater to the data being generated sequentially and in a fixed pattern. This is often a
simple data mart or store responsible for all the incoming messages which are dropped
inside the folder necessarily used for data processing. There are, however, majority of
solutions that require the need of a message-based ingestion store which acts as a
message buffer and also supports the scale based processing, provides a comparatively
reliable delivery along with other messaging queuing semantics. The options include
those like Apache Kafka, Apache Flume, Event hubs from Azure, etc.

5. Stream Processing
 There is a slight difference between the real-time message ingestion and stream
processing. The former takes into consideration the ingested data which is collected at
first and then is used as a publish-subscribe kind of a tool. Stream processing, on the
other hand, is used to handle all that streaming data which is occurring in windows or
streams and then writes the data to the output sink. This includes Apache Spark, Apache
Flink, Storm, etc.

6. Analytics-Based Data store

This is the data store that is used for analytical purposes and therefore the already
processed data is then queried and analyzed by using analytics tools that can correspond
to the BI solutions. The data can also be presented with the help of a NoSQL data
warehouse technology like HBase or any interactive use of hive database which can
provide the metadata abstraction in the data store. Tools include Hive, Spark SQL,
Hbase, etc.

7. Reporting and Analysis

The insights have to be generated on the processed data and that is effectively done by
the reporting and analysis tools which makes use of their embedded technology and
solution to generate useful graphs, analysis, and insights helpful to the businesses. Tools
include Cognos, Hyperion, etc.

8. Orchestration

It consist of data related operations that are repetitive in nature and are also encapsulated
in the workflows which can transform the source data and also move data across sources
as well as sinks and load in stores and push into analytical units. Examples include
Sqoop, oozie, data factory, etc.

Major roles in reference architecture are:

1. System Orchestrator
2. Data Provider
 3. Big Data Application Provider
4 Big Data Framework Provider
5. Data Consumer.

 System Orchestrator is the automated arrangement, coordination, and management of

computer system, middleware and services. It ensures that different applications, data and
infrastructure components of Big Data environments work together. It is done by making
use of workflows and automation.

 The Data Provider role introduces new data or information feeds into the Big Data system
for discovery, access, and transformation. The data can originate from different sources,
such as human generated data (social media)-sensory data or third-party systems.

 The Big Data Application Provider is the component that contains the business logic and
functionality that is necessary to transform the data into the desired results. The common
objective of this component is to extract value from the input data.

 The Big Data Framework Provider has the resources and services that can be used by
Data Application Provider. It provides the core infrastructure. It includes how data is
stored and processed based on designs that are optimized to Big Data environments.

The Big Data framework Provider can be further sub-divided into the following sub-roles:
1. infrastructure: networking, computing and storage
2. Platforms: data organization and distribution
3. Processing: computing and analytic

 The Data Consumer uses the interfaces or services provided by the Big Data Application
Provider to get access to the information of interest.

The activities associated with the Data Consumer role are

1. Search and Retrieve

 2. Download
3. Analyze Locally
4. Reporting
5. Visualization of Data

5 Vs of Big Data

The 5 V's are defined as follows:

1. Velocity is the speed at which the data is created and how fast it moves.

2. Volume is the amount of data qualifying as big data.

3. Value is the value the data provides.

4. Variety is the diversity that exists in the types of data.

5. Veracity is the data's quality and accuracy.

Velocity

 Velocity refers to how quickly data is generated and how fast it moves. This is an
important aspect for organizations that need their data to flow quickly, so it's available at
the right times to make the best business decisions possible.
 An organization that uses big data will have a large and continuous flow of data that's
being created and sent to its end destination. Data could flow from sources such as
machines, networks, smartphones or social media.
 Velocity applies to the speed at which this information arrives -- for example, how many
social media posts per day are ingested -- as well as the speed at which it needs to be
digested and analyzed -- often quickly and sometimes in near real time.
 As an example, in healthcare, many medical devices today are designed to monitor
patients and collect data. From in-hospital medical equipment to wearable devices,
collected data needs to be sent to its destination and analyzed quickly.

In some cases, however, it might be better to have a limited set of collected data than to collect
more data than an organization can handle -- because this can lead to slower data velocities.
Volume

 Volume refers to the amount of data that exists. Volume is like the base of big data, as it's
the initial size and amount of data that's collected. If the volume of data is large enough,
it can be considered big data. However, what's considered to be big data is relative and
will change depending on the available computing power that's on the market.
 For example, a company that operates hundreds of stores across several states generates
millions of transactions per day. This qualifies as big data, and the average number of
total transactions per day across stores represents its volume.

Value

 Value refers to the benefits that big data can provide, and it relates directly to what
organizations can do with that collected data. Being able to pull value from big data is a
requirement, as the value of big data increases significantly depending on the insights that
can be gained from it.
 Organizations can use big data tools to gather and analyze the data, but how they derive
value from that data should be unique to them.
 Tools like Apache Hadoop can help organizations store, clean and rapidly process this
massive amount of data.
 A great example of big data value can be found in the gathering of individual customer
data. When a company can profile its customers, it can personalize their experience
in marketing and sales, improving the efficiency of contacts and garnering greater
customer satisfaction.

Variety

 Variety refers to the diversity of data types. An organization might obtain data from
several data sources, which might vary in value. Data can come from sources in and
outside an enterprise as well. The challenge in variety concerns the standardization and
distribution of all data being collected.
 Collected data can be unstructured, semi-structured or structured. Unstructured data is
data that's unorganized and comes in different files or formats. Typically, unstructured
data isn't a good fit because it doesn't fit into conventional data models. Semi-structured
data is data that hasn't been organized into a specialized repository but has associated
 information, such as metadata. This makes it easier to process than unstructured data.
Structured data, meanwhile, is data that has been organized into a formatted repository.
This means the data is made more addressable for effective data processing and analysis.
 Example could be found in a company that gathers a variety of data about its customers.
This can include structured data from transactions or unstructured social media posts and
feedback form. Much of this might arrive in the form of raw data, requiring cleaning
before processing.

Veracity

 Veracity refers to the quality, accuracy, integrity and credibility of data. Gathered data
could have missing pieces, might be inaccurate or might not be able to provide real,
valuable insight. Veracity, overall, refers to the level of trust there is in the collected data.
 Data can sometimes become messy and difficult to use. A large amount of data can cause
more confusion if it's incomplete. For example, in the medical field, if data about what
drugs a patient is taking is incomplete, the patient's life could be endangered.

Both value and veracity help define the quality and insights gathered from data.

Importance of Big Data

1. Cost Savings

Big Data bring cost-saving benefits to businesses when they have to store large amounts
of data. These tools help organizations in identifying more effective ways of doing
business.

2. Time-Saving

Real-time in-memory analytics helps companies to collect data from various sources.
Tools like Hadoop help them to analyze data immediately thus helping in making quick
decisions based on the learnings.
 3. Understand the market conditions

Big Data analysis helps businesses to get a better understanding of market situations.

For example, analysis of customer purchasing behavior helps companies to identify the
products sold most and thus produces those products accordingly. This helps companies
to get ahead of their competitors.

4. Social Media Listening

Companies can perform sentiment analysis using Big Data tools. These enable them to
get feedback about their company, that is, who is saying what about the company.

Companies can use Big data tools to improve their online presence.

5. Boost Customer Acquisition and Retention

Customers are a vital asset on which any business depends on. No single business can
achieve its success without building a robust customer base. But even with a solid
customer base, the companies can’t ignore the competition in the market.

6. The driver of Innovations and Product Development

Big data makes companies capable to innovate and redevelop their products.

Application of Big Data

1. Tracking Customer Spending Habit, Shopping Behavior: In big retails store (like
Amazon, Walmart, Big Bazar etc.) management team has to keep data of customer’s
spending habit (in which product customer spent, in which brand they wish to spent,
how frequently they spent), shopping behavior, customer’s most liked product (so that
 they can keep those products in the store). Which product is being searched/sold most,
based on that data, production/collection rate of that product get fixed.

2. Recommendation: By tracking customer spending habit, shopping behavior, Big retails

store provide a recommendation to the customer. E-commerce site like Amazon,
Walmart, Flipkart does product recommendation. They track what product a customer
is searching, based on that data they recommend that type of product to that customer.

YouTube also shows recommend video based on user’s previous liked, watched video
type. Based on the content of a video, the user is watching, relevant advertisement is
shown during video running.

3. Smart Traffic System: Data about the condition of the traffic of different road,
collected through camera kept beside the road, at entry and exit point of the city, GPS
device placed in the vehicle (Ola, Uber cab, etc.). All such data are analyzed and jam-
free or less jam way, less time taking ways are recommended. Such a way smart traffic
system can be built in the city by Big data analysis. One more profit is fuel
consumption can be reduced.

4. Secure Air Traffic System: At various places of flight (like propeller etc) sensors
present. These sensors capture data like the speed of flight, moisture, temperature, other
environmental condition. Based on such data analysis, an environmental parameter
within flight are set up and varied.

By analyzing flight’s machine-generated data, it can be estimated how long the machine
can operate flawlessly when it to be replaced/repaired.

5. Auto Driving Car: Big data analysis helps drive a car without human interpretation. In
the various spot of car camera, a sensor placed, that gather data like the size of the
surrounding car, obstacle, distance from those, etc. These data are being analyzed, then
various calculation like how many angles to rotate, what should be speed, when to stop,
etc carried out. These calculations help to take action automatically.
6. Virtual Personal Assistant Tool: Big data analysis helps virtual personal assistant tool
(like Siri in Apple Device, Cortana in Windows, Google Assistant in Android) to
provide the answer of the various question asked by users. This tool tracks the location
of the user, their local time, season, other data related to question asked, etc. Analyzing
all such data, it provides an answer.

7. IoT:

Manufacturing company install IOT sensor into machines to collect operational data.
Analyzing such data, it can be predicted how long machine will work without any
problem when it requires repairing so that company can take action before the situation
when machine facing a lot of issues or gets totally down. Thus, the cost to replace the
whole machine can be saved.

 In the Healthcare field, Big data is providing a significant contribution. Using big data
tool, data regarding patient experience is collected and is used by doctors to give better
treatment. IoT device can sense a symptom of probable coming disease in the human
body and prevent it from giving advance treatment. IoT Sensor placed near-patient,
new-born baby constantly keeps track of various health condition like heart bit rate,
blood presser, etc. Whenever any parameter crosses the safe limit, an alarm sent to a
doctor, so that they can take step remotely very soon.

8. Education Sector: Online educational course conducting organizations utilize big data
to search candidate, interested in that course. If someone searches for YouTube tutorial
video on a subject, then online or offline course provider organization on that subject
send ad online to that person about their course.

9. Energy Sector: Smart electric meter read consumed power every 15 minutes and sends
this read data to the server, where data analyzed and it can be estimated what is the time
in a day when the power load is less throughout the city. By this system manufacturing
 unit or housekeeper are suggested the time when they should drive their heavy machine
in the night time when power load less to enjoy less electricity bill.

10. Media and Entertainment Sector: Media and entertainment service providing
company like Netflix, Amazon Prime, Spotify do analysis on data collected from their
users. Data like what type of video, music users are watching, listening most, how long
users are s pending on site, etc are collected and analyzed to set the next business
strategy.

Big Data features – security, compliance, auditing and protection

 Security and privacy issues are magnified by velocity, volume, and variety of Big data,
such as large scale cloud infrastructure, diversity of data sources and formats, streaming
nature of data acquisition, and high volume inter-cloud migration.
 Traditional security mechanisms are in-adequate
 Streaming data demands ultra-fast response time from any security and privacy solution.

The sheer size of Big Data brings with it a major security challenge. Proper security entails more
than keeping the bad guys out; it also means backing up data and protecting data from
corruption.

 Data access: data can be protected if you eliminate access to the data! Not pragmatic so
we opt to control access.
 Data availability: controlling where the data are stored and how it is distributed; more
control position you better to protect the data.
 Performance: encryption and other measures can improve security but they carry a
processing burden that can severely affect the system performance.
 Adequate security becomes a strategic balancing act among the above concerns. With
planning, logic, and observations, security becomes manageable. Effectively protecting
data while allowing access to the authorized users and systems.

Pragmatic Steps to Securing Big Data:

 • First get rid of data that are no longer needed. If not possible to destroy then the
information should be securely archived and kept offline

• A real challenge is to decide which data is needed? As value can be found in unexpected
places. For example, activity logs represent a risk but logs can be used to determine scale,
use, and efficiency of big data analytics

• There is no easy answer to the above question, and it becomes a case of choosing the lesser
of two evils.

Classifying Data:

• Protecting data is much easier if data is classified into categories, e.g., internal email
between colleagues is different from financial report, etc.

• Simple classification can be: financial, HR, sales, inventory, and communications.

• Once organizations better understand their data, they can take important steps to segregate
the information and that makes it easier to employ security measures like encryption and
monitoring more manageable.

Big Data and Compliance:

• Compliance has major effect on how Big Data is protected, stored, accessed, and archived.
• Big Data is not easily handled by RDBMS; this means it is harder to understand how
compliance affects the data.
• Big Data is transforming the storage and access paradigm to a new world of horizontally
scaling, unstructured databases, which are more suited to solve old business problems with
analytics.

• New data types and methodologies are still expected to meet the legislative requirements
expected by compliance laws
  Preventing compliance from becoming the next Big Data nightmare is going to be the job
of security professionals.
 Health care is a good example of Big Data compliance challenge, i.e., different data types
and vast rate of data from different devices, etc.
 NoSQL is evolving as the new data management approach to unstructured data. No need
for federating multiple RDBMS. Clustered single NoSQL database and being deployed in
the cloud.
 Unfortunately, most data stores in the NoSQL world (i.e., Hadoop, Cassandra and
MongoDB) do not incorporate sufficient data security tools to provide what is needed.
 Big Data changed few things: For example network security developers spent a great deal
of time and money on perimeter-based security mechanisms (e.g., firewalls) but that cannot
prevent unauthorized access to data once a criminal/hacker has entered the network

Big data privacy and ethics

Privacy

Big data privacy is protecting individuals' personal and sensitive data when it comes to
collecting, storing, processing, and analyzing large amounts of data. Following are some
important aspects of big data privacy:

Informed consent

When it comes to big data privacy, informed consent is the foundation. Organizations need to
ask individuals' permission before they collect their data. With informed consent, people
know exactly what their data is being used for, how it's being used, and what the
consequences could be. By giving clear explanations and letting people choose how they
want to use their data, organizations can create trust and respect for people's privacy.

Protecting individual identity

Protecting individual identity is of paramount importance. There are two techniques used to
protect individual identity: anonymization and de-identification. Anonymization means
 removing or encrypting personal information (PII) so that individuals cannot be found in
the dataset. De-identification goes beyond anonymization by transforming data in ways
that prevent re-identification. These techniques enable organizations to gain insights
while protecting privacy.

Data integrity and confidentiality

Data integrity and confidentiality are two of the most important aspects of data security.
Without them, unauthorized access to data, data breaches, and cyber threats are at an all-
time high. That’s why it’s essential for organizations to implement strong security
measures, such as encryption, security access controls, and periodic security audits. Data
integrity and confidentiality help organizations build trust with their users and promote
responsible data management.

Purpose limitation and data minimization

Big data privacy and ethics call for the principle of purpose limitation. Data should only be
used for specified, authorized purposes and should not be reused without permission from
the user. Additionally, data minimization involves collecting and retaining only the
minimum amount of data necessary for the intended purpose, reducing privacy risks and
potential harm.

Transparency and accountability

One of the most important ways to build trust with users is through transparency in data
practices. Individuals' data collection, data usage, and data sharing should all be clearly
defined by organizations. Accountability for data management and privacy compliance
reinforce ethical data management.
Control and autonomy

Privacy and ethics require organizations to respect individual rights. Individuals are entitled
to access, update, and erase their data. Organizations should provide easy mechanisms for
users to exercise these rights and maintain control and autonomy over their data.

Ethics

Big data ethics refers to the ethical and responsible decisions that are made when collecting,
processing, analyzing, and deploying large and complex data sets. The following are
some important aspects of the big data ethics:

Fairness and bias mitigation

One of the most important aspects of big data analytics is ensuring that data is collected and
analyzed in a way that is fair and free of bias and discrimination. Organizations should be
aware of how bias can exist and how to reduce it so they can make ethical choices and
make sure everyone is treated equally.

Data governance and impact assessment

Ethical data management is best achieved when data governance frameworks are in place. By
setting up procedures, organization s encourage responsible use of data. Privacy impact
assessments help identify and address privacy concerns before they escalate.

Ownership

In the world of big data privacy, when we refer to data ownership we mean who
can control the data and who can benefit from the collected data. In reference to the two
terms: control and benefit, individuals should own their personal data. They should have
control over how their personal data is collected, used and shared. Organizations that
collect and process large amounts of data should view themselves as custodians of data.
Organizations should responsibly manage data while respecting individuals’ rights.
Challenges of conventional data

Fundamental Challenges

 How to store large volume, variety and high speed data

Conventional system technology

 CPU capacity, RAM speed is comparatively less.

 Parallel programming is costly.

Process challenges

 Capturing, aligning and transforming data for analysis is time consuming and costly.
 Output understanding, data visualization and display issues.

Management Challenges

 Data privacy, ethics, security challenges.

Analysis Vs reporting

Analytics Reporting

Analytics is the method of

Reporting is an action that includes all the
examining and analyzing
needed information and data and is put
summarized data to make business
together in an organized way.
decisions.

Questioning the data, understanding Identifying business events, gathering the

it, investigating it, and presenting it required information, organizing,
to the end users are all part of summarizing, and presenting existing data are
 analytics. all part of reporting.

The purpose of analytics is to draw The purpose of reporting is to organize the

conclusions based on data. data into meaningful information.

Analytics is used by data analysts, Reporting is provided to the appropriate

scientists, and business people to business leaders to perform effectively and
make effective decisions. efficiently within a firm.

Intelligent data analysis

Intelligent analytics (IA) is a science and technology that collects, organizes, and analyzes
big data, big information, big knowledge, and big intelligence as well as big wisdom in
order to discover and visualize patterns, knowledge, and intelligence as well as other
information within big data, information, knowledge, and intelligence using big analytics,
artificial intelligence (AI), and intelligent systems.

Intelligent analytics benefits for enterprise

For business functions, easy-to-use, intelligent analytics tools become a necessity. This is
particularly true in IT service management operations such as planning, providing,
managing, and upgrading end-user IT services.

Modern data analytic tools

1. APACHE Hadoop

It’s a Java-based open-source platform that is being used to store and process big data. It is
built on a cluster system that allows the system to process data efficiently and let the data run
parallel. It can process both structured and unstructured data from one server to multiple
computers. Hadoop also offers cross-platform support for its users. Today, it is the best big
data analytic tool and is popularly used by many tech giants such as Amazon, Microsoft, IBM,
etc.

Features:

 Free to use and offers an efficient storage solution for businesses.

 Offers quick access via HDFS (Hadoop Distributed File System).
 Highly flexible and can be easily implemented with MySQL, and JSON.
 Highly scalable as it can distribute a large amount of data in small segments.
 It works on small commodity hardware like JBOD or a bunch of disks.

2. Cassandra

APACHE Cassandra is an open-source NoSQL distributed database that is used to fetch large
amounts of data. It’s one of the most popular tools for data analytics and has been praised by
many tech companies due to its high scalability and availability without compromising speed
and performance. It is capable of delivering thousands of operations every second and can
handle petabytes of resources with almost zero downtime. It was created by Facebook back in
2008 and was published publicly.

Features of APACHE Cassandra:

 Data Storage Flexibility: It supports all forms of data i.e. structured, unstructured, semi-
structured, and allows users to change as per their needs.
 Data Distribution System: Easy to distribute data with the help of replicating data on
multiple data centers.
 Fast Processing: Cassandra has been designed to run on efficient commodity hardware and
also offers fast storage and data processing.
 Fault-tolerance: The moment, if any node fails, it will be replaced without any delay.

3. Qubole

It’s an open-source big data tool that helps in fetching data in a value of chain using ad-hoc
analysis in machine learning. Qubole is a data lake platform that offers end-to-end service with
reduced time and effort which are required in moving data pipelines. It is capable of
configuring multi-cloud services such as AWS, Azure, and Google Cloud. Besides, it also
helps in lowering the cost of cloud computing by 50%.
Features of Qubole:

 Supports ETL process: It allows companies to migrate data from multiple sources in one
place.
 Real-time Insight: It monitors user’s systems and allows them to view real-time insights
 Predictive Analysis: Qubole offers predictive analysis so that companies can take actions
accordingly for targeting more acquisitions.
 Advanced Security System: To protect users’ data in the cloud, Qubole uses an advanced
security system and also ensures to protect any future breaches. Besides, it also allows
encrypting cloud data from any potential threat.

4. Xplenty

It is a data analytic tool for building a data pipeline by using minimal codes in it. It offers a
wide range of solutions for sales, marketing, and support. With the help of its interactive
graphical interface, it provides solutions for ETL, ELT, etc. The best part of using Xplenty is its
low investment in hardware & software and its offers support via email, chat, telephonic and
virtual meetings. Xplenty is a platform to process data for analytics over the cloud and
segregates all the data together.

Features of Xplenty:

 Rest API: A user can possibly do anything by implementing Rest API

 Flexibility: Data can be sent, and pulled to databases, warehouses, and salesforce.
 Data Security: It offers SSL/TSL encryption and the platform is capable of verifying
algorithms and certificates regularly.
 Deployment: It offers integration apps for both cloud & in-house and supports deployment
to integrate apps over the cloud.

5. Spark

APACHE Spark is another framework that is used to process data and perform numerous tasks
on a large scale. It is also used to process data via multiple computers with the help of
distributing tools. It is widely used among data analysts as it offers easy-to-use APIs that
provide easy data pulling methods and it is capable of handling multi-petabytes of data as
well. Recently, Spark made a record of processing 100 terabytes of data in just 23
minutes which broke the previous world record of Hadoop (71 minutes). This is the reason
why big tech giants are moving towards spark now and is highly suitable for ML and AI today.

Features of APACHE Spark:

 Ease of use: It allows users to run in their preferred language. (JAVA, Python, etc.)
 Real-time Processing: Spark can handle real-time streaming via Spark Streaming
 Flexible: It can run on, Mesos, Kubernetes, or the cloud.

6. Mongo DB

Came in limelight in 2010, is a free, open-source platform and a document-oriented (NoSQL)

database that is used to store a high volume of data. It uses collections and documents for
storage and its document consists of key-value pairs which are considered a basic unit
of Mongo DB. It is so popular among developers due to its availability for multi-programming
languages such as Python, Jscript, and Ruby.

Features of Mongo DB:

 Written in C++: It’s a schema-less DB and can hold varieties of documents inside.
 Simplifies Stack: With the help of mongo, a user can easily store files without any
disturbance in the stack.
 Master-Slave Replication: It can write/read data from the master and can be called back for
backup.

7. Apache Storm

A storm is a robust, user-friendly tool used for data analytics, especially in small companies.
The best part about the storm is that it has no language barrier (programming) in it and can
support any of them. It was designed to handle a pool of large data in fault-tolerance and
horizontally scalable methods. When we talk about real-time data processing, Storm leads the
chart because of its distributed real-time big data processing system, due to which today many
tech giants are using APACHE Storm in their system. Some of the most notable names are
Twitter, Zendesk, NaviSite, etc.

Features of Storm:

 Data Processing: Storm process the data even if the node gets disconnected
 Highly Scalable: It keeps the momentum of performance even if the load increases
 Fast: The speed of APACHE Storm is impeccable and can process up to 1 million
messages of 100 bytes on a single node.

8. SAS

Today it is one of the best tools for creating statistical modeling used by data analysts. By
using SAS, a data scientist can mine, manage, extract or update data in different variants from
different sources. Statistical Analytical System or SAS allows a user to access the data in any
format (SAS tables or Excel worksheets). Besides that it also offers a cloud platform for
business analytics called SAS Viya and also to get a strong grip on AI & ML, they have
introduced new tools and products.

Features of SAS:

 Flexible Programming Language: It offers easy-to-learn syntax and has also vast libraries
which make it suitable for non-programmers
 Vast Data Format: It provides support for many programming languages which also
include SQL and carries the ability to read data from any format.
 Encryption: It provides end-to-end security with a feature called SAS/SECURE.
9. Data Pine

Datapine is an analytical used for BI and was founded back in 2012 (Berlin, Germany). In a
short period of time, it has gained much popularity in a number of countries and it’s mainly
used for data extraction (for small-medium companies fetching data for close monitoring).
With the help of its enhanced UI design, anyone can visit and check the data as per their
requirement and offer in 4 different price brackets, starting from $249 per month. They do
offer dashboards by functions, industry, and platform.
Features of Datapine:

 Automation: To cut down the manual chase, datapine offers a wide array of AI assistant
and BI tools.
 Predictive Tool: datapine provides forecasting/predictive analytics by using historical and
current data, it derives the future outcome.
 Add on: It also offers intuitive widgets, visual analytics & discovery, ad hoc reporting,
etc.

10. Rapid Miner

It’s a fully automated visual workflow design tool used for data analytics. It’s a no-code
platform and users aren’t required to code for segregating data. Today, it is being heavily used
in many industries such as ed-tech, training, research, etc. Though it’s an open-source platform
but has a limitation of adding 10000 data rows and a single logical processor. With the help
of Rapid Miner, one can easily deploy their ML models to the web or mobile (only when the
user interface is ready to collect real-time figures).

Features of Rapid Miner:

 Accessibility: It allows users to access 40+ types of files (SAS, ARFF, etc.) via URL
 Storage: Users can access cloud storage facilities such as AWS and dropbox
 Data validation: Rapid miner enables the visual display of multiple results in history for
better evaluation.