Q1. What is Big Data?

Requirements, constantly updated to cope with the three

Challenges and Benefits? Big data can be Vs. 2. Organizations do not have enough
defined as very volumes of data available at skilled data professionals who can
various sources, in varying degrees of understand and work with big data and big
complexity, generated at different speed data tools. Benefits :: Improved customer
i.e. velocities and varying degrees of service. Business can utilize outside
ambiguity, which cannot be processed intelligence while taking decision.
using traditional technologies, processing Reducing maintenance costs. Re-develop
methods, algorithms or any commercial your products : Big data can also help you
off-the-shelf solutions. • 'Big data' is a term understand how others perceive your
used to describe collection of data that is products so that you can adapt them or
huge in size and yet growing exponentially your marketing, if need be. Early
with time. In short, such a data is so large identification of risk to the
and complex that none of the traditional product/services, if any. Better operational
data management tools are able to store it efficiency.
or process it efficiently. The processing of
Q2. Explain current analytics architecture
big data begins with the raw data that isn't
with suitable diagram? Data sources: This
aggregated or organized and is most often
is where the data comes from. It can be
impossible to store in the memory of a
from a variety of sources, such as
single computer. • Big data processing is a
transactional systems, operational
set of techniques or programming models
databases, social media, and sensors. *
to access large-scale data to extract useful
Data integration: This is where the data is
information for supporting and providing
collected and combined from different
decisions. Hadoop is the open-source
sources. This can be done using a variety of
implementation of MapReduce and is
tools, such as ETL (Extract, Transform,
widely used for big data processing.
Load) tools and data warehouses. * Data
Requirements :: 1. Volume :Size of data to
storage: This is where the data is stored for
be processed is large-it needs to be broken
analysis. It can be stored in a variety of
into manageable chunks. Data needs to be
ways, such as in a data warehouse, a data
processed in parallel across multiple
lake, or a cloud-based data store. * Data
systems. Data needs to be processed
analysis: This is where the data is analyzed
across several program modules
to find patterns and trends. This can be
simultaneously. 2. Velocity : • Data needs
done using a variety of tools, such as
to be processed at streaming speeds during
statistical software, machine learning
data collection. • Data needs to be
algorithms, and business intelligence (BI)
processed for multiple acquisition points.
tools. * Data visualization: This is where the
3. Variety : Data of different formats needs
data is presented in a way that is easy to
to be processed. Data of different types
understand. This can be done using a
needs to be processed. Data of different
variety of tools, such as BI tools, data
structures needs to be processed. Data
visualization software, and dashboards.
from different regions needs to be
The current analytics architecture is a
processed. Challenges :: 1. Big data is
complex and ever-evolving landscape.
growing exponentially and existing data
There are a variety of tools and
management solutions have to be
technologies available to help aggregators : • The entities which process
organizations collect, store, analyze, and collected data from the first layer make
visualize data. The key to success is to them understandable. They give them
choose the right tools and technologies for additional value to prepare them for the
your specific needs. Here are some of the handing over process. Now the data is
key trends in current analytics architecture: ready to be offered on the market. •
* The move to the cloud: More and more Typically, one of these data aggregators can
organizations are moving their analytics transform and package the data as
workloads to the cloud. This is due to the products to sell to list brokers that might
many benefits of the cloud, such as want to generate marketing lists of people
scalability, cost-effectiveness, and agility. * who may be good targets for specific ad
The rise of big data: The amount of data campaigns.4. Data users and buyers : • The
that organizations are generating is entities represent a group of the final layer
growing exponentially. This is driving the from the big data ecosystem. This group
need for new analytics technologies that has the final benefits from the collected
can handle big data. * The increasing use of and aggregated data offered by the data
artificial intelligence (AI): AI is being used in aggregators. • Data users may wan to to
a variety of ways to improve analytics, such track or prepare for natural disaster by
as automating data preparation, identifying which areas a hurricane will
identifying patterns and trends, and affect first hand. It can be observed by
making predictions. tracking tweets it or discussing it in social
media.
Q3. Draw and explain big data ecosystem?
1. Data device .Data device and sensor
network gather data from various locations
and continuously generate new data. •
Example of data devices : Playing games,
smart phone and retail shopping. • Sensor
data : Growing network of sensor devices
generate data based on monitoring
environmental conditions, such as
temperature, sound, pressure, power
water levels etc. 2. Data collectors : • Data
collectors : It includes samples entities that
collect data from the device and user. Retail
stores tracking the path a customer takes
through their store while pushing a
shopping cart with an RFID chip so they can
gauge which products get the most foot
traffic using geospatial data collected from
the RFID chips. Data collectors example :
Government, retail stores, cable TV
provider. Cable TV provider which tracks
the shows that a person watches. 3. Data
Q4. Sources of Big Data:: [Link] media : reporting and 81 tasks and solve many of
Social media is one of the biggest the problems that proliferating
contributors to the flood of data we have spreadsheets introduce, such as hitch of
today. Facebook generates around 500+ the multiple versions of a spreadsheet is
terabytes of data everyday in the form of correct. From an analyst perspective, EDW
content generated by the users like status and 81 solve problems related to data
messages, photos and video uploads, accuracy and availability. Advantages : 1.
messages, comments etc. [Link] Data is preserved and archived. 2. Data
exchange : Data generated by stock isolation allows for easier and faster data
exchanges is also in terabytes per day. Most reporting. 3. Data administrators have
of this data is the trade data of users and easier time tracking problems. 4. There is
companies. [Link] industry : A single jet value to storing and analysing data.
engine can generate around 10 terabytes Disadvantages : 1. Growing data sets could
of data during a 30 minute flight. [Link] slow down systems. 2. A system crash could
data : Online or offline surveys conducted affect all the data. 3. Unauthorized users
on various topics which typically have can access all sensitive data more easily
hundreds and thousands of responses and than if it was distributed across several
need to be processed for analysis and locations.
visualization by creating a cluster of
Q6. Data Analytics Lifecycle : The Data
population and their associated responses.
analytic lifecycle is designed for Big Data
5. Compliance data : Many organizations
problems and data science projects. The
like healthcare, hospitals, life sciences,
cycle is iterative to represent real project.
finance etc, has to file compliance reports.
To address the distinct requirements for
[Link] repository is also known as a data performing analysis on Big Data, step – by
library or data archive. This is a general – step methodology is needed to organize
term to refer to a data set isolated to be the activities and tasks involved with
mined for data reporting and analysis. The acquiring, processing, analyzing, and
data repository is a large database repurposing data Phase 1: Discovery – The
infrastructure, several databases that data science team learn and investigate the
collect, manage and store data sets for data problem. Develop context and
analysis, sharing and reporting. 1. understanding. Come to know about data
Spreadsheets : • Spreadsheets enabled sources needed and available for the
business user to crate simple logic on data project. The team formulates initial
structured in rows and columns and create hypothesis that can be later tested with
their own analyses of business problems. • data. Phase 2: Data Preparation – Steps to
Database administrator training is not explore, preprocess, and condition data
required to create spreadsheets : They can prior to modelling and analysis. It requires
be set up to do many things quickly and the presence of an analytic sandbox, the
independently of Information Technology team execute, load, and transform, to get
(IT) groups. • Spreadsheets are easy to data into the sandbox. Data preparation
share and en users have control over the tasks are likely to be performed multiple
logic involved. 2. Enterprise Data times and not in predefined order. Several
Warehouse (EDWs) are critical for tools commonly used for this phase are –
Hadoop, Alpine Miner, Open Refine, etc. the Dataset: The dataset is typically split
Phase 3: Model Planning – Team explores into two parts: a training set and a
data to learn about relationships between testing/validation set. The training set is
variables and subsequently, selects key used to train the model, while the
variables and the most suitable models. In testing/validation set is used to evaluate
this phase, data science team develop data the model's performance. 4. Model
sets for training, testing, and production Selection and Configuration: We select a
purposes. 4: Model Building – Team machine learning algorithm suitable for
develops datasets for testing, training, and our task, such as a Naive Bayes classifier or
production purposes. Team also considers a Support Vector Machine (SVM). We also
whether its existing tools will suffice for configure any hyperparameters associated
running the models or if they need more with the chosen algorithm, such as the
robust environment for executing models. regularization parameter or the number of
Phase 5: Communication Results – After hidden layers in a neural network. 5.
executing model team need to compare Training the Model: The training process
outcomes of modeling to criteria involves feeding the prepared data into the
established for success and failure. Team model and adjusting its internal
considers how best to articulate findings parameters to minimize the difference
and outcomes to various team members between the predicted outputs and the
and stakeholders, taking into account true labels. The model learns from the data
warning, assumptions. Phase 6: through an optimization algorithm, such as
Operationalize – The team communicates gradient descent, which updates the
benefits of project more broadly and sets model's parameters iteratively. 6. Model
up pilot project to deploy work in Evaluation: Once the model has been
controlled way before broadening the work trained, we evaluate its performance on
to full enterprise of users. the testing/validation set. Various metrics,
such as accuracy, precision, recall, or F1
Q7. Explain mode building phase with
score, can be used to assess how well the
example? 1. Data Preprocessing: In this
model generalizes to unseen data. 7.
step, we clean and prepare the data for
Iterative Refinement: If the model's
training. This may involve removing
performance is unsatisfactory, we can
unnecessary characters, converting text to
make adjustments to the preprocessing
lowercase, removing stop words, and
steps, feature extraction techniques, or
tokenizing the text into individual words or
model configuration to improve its
phrases. 2. Feature Extraction: To train the
accuracy. This iterative process of refining
model, we need to convert the text data
the model and retraining it continues until
into a numerical representation that the
we achieve the desired performance. 8.
model can understand. This process is
Deployment: After the model has been
called feature extraction. One common
trained and evaluated, it can be deployed
technique is using the Bag-of-Words
in a real-world setting, where it can classify
model, where each unique word in the
new, unseen emails as either "spam" or
dataset becomes a feature, and the
"not spam" based on its learned patterns.
presence or absence of each word is
represented by a binary value. 3. Splitting
Q8. Why communication is important in and Validation: During the model
data analytics lifecycle project? [Link] development phase, communication is
Project Goals: Effective communication necessary between data scientists,
helps in clearly understanding and defining statisticians, and domain experts. It helps
the project goals. It involves discussions in understanding the underlying
with stakeholders, data scientists, and assumptions, selecting appropriate
analysts to align everyone's understanding algorithms, and validating the models
of the project objectives, expectations, and against the business problem. Effective
desired outcomes. This ensures that the communication ensures that the models
project is focused and on track from the reflect the real-world scenario and
beginning. 2. Gathering Requirements: generate meaningful insights.
Communication plays a vital role in
Q9. Essential Python Libraries? 1. NumPy
gathering requirements for the data
(Numerical Python): NumPy is a
analytics project. By actively engaging with
fundamental library for numerical
stakeholders and subject matter experts,
computing in Python. It provides powerful
data analysts can gain a deep
mathematical and array operations,
understanding of the business problem or
enabling efficient manipulation and
question to be addressed. This helps in
computation on large multidimensional
designing appropriate data collection
arrays. NumPy is widely used in scientific
strategies, determining relevant variables,
and data analysis applications and forms
and identifying the desired insights. 3. Data
the foundation for other libraries like
Collection and Integration:
Pandas and SciPy. 2. Pandas: Pandas is a
Communication is essential during the data
data manipulation and analysis library built
collection phase. It involves coordinating
on top of NumPy. It provides easy-to-use
with data sources, such as databases, APIs,
data structures, such as DataFrame and
or external vendors, to ensure the
Series, for handling structured data.
necessary data is collected, shared, and
Pandas offers a rich set of functions for
integrated properly. Clear communication
data cleaning, preprocessing, merging,
helps to address any issues or challenges
reshaping, and aggregating data. It is
that may arise during data acquisition,
extensively used for data wrangling tasks
ensuring the data is accurate, complete,
and exploratory data analysis in Python. 3.
and meets the project's needs. 4. Data
SciPy (Scientific Python): SciPy is a library
Preprocessing and Cleaning: In the data
that provides a wide range of scientific and
preprocessing phase, communication is
numerical algorithms. It builds on top of
critical to understanding the context of the
NumPy and includes modules for
data and the potential challenges it may
optimization, interpolation, integration,
present. By communicating with domain
linear algebra, signal and image processing,
experts and data providers, data analysts
statistics, and more. SciPy is designed to be
can gain insights into data quality issues,
used in scientific and engineering
missing values, outliers, and other
applications and provides efficient and
anomalies. This collaborative
reliable implementations of various
communication helps in making informed
mathematical algorithms. 4. scikit-learn:
decisions on how to handle and clean the
Scikit-learn is a popular machine learning
data effectively. 5. Model Development
library in Python. It provides a consistent techniques include: - Scaling and
interface and implementations for a wide normalization: Scaling ensures that
range of machine learning algorithms, different variables are on a similar scale,
including classification, regression, preventing certain variables from
clustering, dimensionality reduction, and dominating the analysis due to their larger
model evaluation. Scikit-learn is designed magnitudes. Normalization rescales data to
to be user-friendly and provides tools for have a specific range, such as between 0
data preprocessing, model selection, and 1, making it easier to compare and
hyperparameter tuning, and performance interpret. - Logarithmic transformation:
evaluation. It is widely used for building Log transformation is used to reduce the
and deploying machine learning models in skewness of data with a long-tailed
various domains. distribution. It can help stabilize the
variance and make the data more normally
Q10. What is data Preprocessing? Explain
distributed, which is often useful for
i) Removing Duplicates ii)Handling missing
statistical analysis. - Feature engineering:
values iii) Data Transformation in short?
Feature engineering involves creating new
Data preprocessing is a crucial step in the
features or transforming existing features
data analysis pipeline. It involves cleaning
based on domain knowledge. This can
and transforming raw data into a format
include combining variables, creating
suitable for analysis. Here are brief
interaction terms, or deriving new
explanations of three important data
variables that capture meaningful patterns
preprocessing tasks: 1. Removing
or relationships in the data.
Duplicates: Removing duplicates refers to
the process of identifying and eliminating
identical or redundant data entries from a
dataset. Duplicates can skew analysis
results and introduce biases. By removing
duplicates, you ensure that each data point
is unique, preventing any misleading or
redundant information from affecting your
analysis. 2. Handling Missing Values:
Missing values are gaps or empty entries in
a dataset. They can occur due to various
reasons, such as data collection errors,
equipment failures, or participant non-
responses. Handling missing values is
crucial to avoid biased or inaccurate
analysis. Techniques for handling missing
values include: Deleting rows or columns -
Imputation 3. Data Transformation: Data
transformation involves modifying the
original data to improve its quality,
distribution, or representation for analysis.
Some common data transformation
Q11. Explain Data Analytics types answers questions like "What should we do
i)Descriptive ii)Predictive iii)Prescriptive?? to achieve a specific goal?" or "What
1. Descriptive Analytics: Descriptive actions will optimize a particular
analytics focuses on understanding and outcome?" This type of analytics helps in
summarizing historical data to gain insights decision-making by considering various
into what has happened in the past. It constraints, objectives, and scenarios.
involves analyzing data to uncover Prescriptive analytics is often used in
patterns, trends, and relationships within complex problem-solving and decision
the dataset. Descriptive analytics answers optimization applications.
questions like "What happened?" and
Q12. What is association rules and explain
"What are the key characteristics or
apriori algorithm?? Association rules are a
metrics of the data?" It often involves
type of analysis in data mining that seeks to
techniques such as data visualization, data
discover interesting relationships or
aggregation, and basic statistical analysis.
associations between items in a dataset. It
Descriptive analytics provides a foundation
aims to identify patterns or co-occurrences
for understanding the current state of
among different items or variables. The
affairs and can be used for reporting and
Apriori algorithm is a popular algorithm
exploratory data analysis. 2. Predictive
used for mining association rules. It is
Analytics: Predictive analytics goes beyond
based on the concept of frequent itemsets,
descriptive analytics by utilizing historical
which are sets of items that frequently
data to make predictions and forecasts
occur together in the dataset. The
about future events or outcomes. It
algorithm follows a two-step process:
involves building models and algorithms to
finding frequent itemsets and generating
identify patterns in historical data and
association rules. 1. Finding Frequent
apply them to predict unknown or future
Itemsets: The Apriori algorithm begins by
data points. Predictive analytics answers
identifying frequent itemsets, which are
questions like "What is likely to happen in
sets of items that occur together with a
the future?" or "What will be the outcome
frequency greater than or equal to a
of a particular event?" Techniques used in
specified minimum support threshold. The
predictive analytics include regression
algorithm works in an iterative manner: - In
analysis, time series forecasting, machine
the first iteration, the algorithm scans the
learning algorithms, and data mining.
dataset to identify individual items and
Predictive analytics is valuable for making
their frequencies. - Then, it generates
informed decisions and anticipating future
candidate itemsets of length two by
trends or outcomes. 3. Prescriptive
combining the frequent items found in the
Analytics: Prescriptive analytics takes
previous step. - For each candidate
predictive analytics a step further by not
itemset, the algorithm scans the dataset
only predicting future events but also
again to determine its support or
providing recommendations on the actions
frequency. - The itemsets that meet or
to take to achieve desired outcomes. It
exceed the minimum support threshold
involves leveraging optimization and
are considered frequent. - The process
simulation techniques to determine the
continues iteratively, generating candidate
best course of action based on the
itemsets of increasing length and checking
predicted outcomes. Prescriptive analytics
their support until no more frequent 1000 = 0.1 or 10% 2. Confidence:
itemsets can be found. 2. Generating Confidence measures the reliability or
Association Rules: Once the frequent strength of a rule by assessing how often
itemsets have been identified, the Apriori the consequent appears in transactions
algorithm generates association rules containing the antecedent. The formula for
based on these itemsets. An association confidence is: Confidence(A → B) =
rule is an implication of the form "If X, then (Support(A ∪ B)) / (Support(A)) For
Y," where X and Y are itemsets. The example, if we have a rule "bread → milk"
algorithm generates rules by considering and we observe that the "bread and milk"
different combinations of items within the itemset appears in 100 transactions, while
frequent itemsets and evaluating their the "bread" itemset appears in 400
confidence and support. - Confidence: transactions, the confidence of the rule
Confidence measures the likelihood that would be: Confidence(bread → milk) =
the rule is true. It is calculated as the (Support(bread and milk)) /
support of the combined itemset (X ∪ Y) (Support(bread)) = 100 / 400 = 0.25 or 25%
divided by the support of the antecedent 3. Lift: Lift measures the strength of
(X). - Support: Support measures the association between the antecedent and
frequency of occurrence of the itemset in consequent, taking into account their
the dataset. - The algorithm generates individual occurrences. It compares the
association rules by considering frequent observed support of the rule to what would
itemsets and selecting those with be expected if the antecedent and
confidence greater than or equal to a consequent were independent. The
specified minimum confidence threshold. - formula for lift is: Lift(A → B) = (Support(A
Additional measures such as lift, which ∪ B)) / (Support(A) * Support(B)) For
quantifies the strength of the relationship example, if we have a rule "bread → milk"
between the antecedent and consequent, and the support for "bread and milk" is
can also be used to evaluate the rules. 100, while the support for "bread" is 400
and "milk" is 300, the lift of the rule would
[Link] Support, Confidence and Lift
be: Lift(bread → milk) = (Support(bread
with example? 1. Support: Support
and milk)) / (Support(bread) *
measures the frequency or prevalence of
Support(milk)) = 100 / (400 * 300) ≈ 0.008.
an itemset in a dataset. It indicates how
frequently an itemset appears in the
transactions. The formula for support is:
Support(A) = (Number of transactions
containing itemset A) / (Total number of
transactions) For example, consider a
dataset of customer transactions in a
grocery store. Let's say we are interested in
finding association rules for two items:
"bread" and "milk". If the "bread and milk"
itemset appears in 100 out of 1000
transactions, the support for the itemset
would be: Support(bread and milk) = 100 /
Q14. What is classification ? Explain Bayes highest posterior probability as the
'theorem. Explain Naive Bayes' predicted class label.
classifier.?? Classification is a machine
Q15. Explain following decision tree
learning technique that involves assigning
algorithms : i) ID3 Algorithm ii) C4.5 iii)
predefined labels or categories to data
CART ?? i) ID3 Algorithm: The ID3 (Iterative
based on its features or attributes. The goal
Dichotomiser 3) algorithm is one of the
of classification is to build a model that can
early decision tree algorithms developed
generalize from a labeled training dataset
by Ross Quinlan. It follows a top-down,
to accurately predict the class labels of
greedy approach to construct the decision
unseen or future instances. Naive Bayes
tree. Here are some key features of the ID3
classifier is a probabilistic machine learning
algorithm: - Attribute Selection: ID3 selects
algorithm based on Bayes' theorem. It
attributes based on the information gain
assumes that the features are conditionally
criterion. It calculates the information gain
independent of each other given the class.
for each attribute and chooses the
This assumption is known as the "naive"
attribute with the highest information gain
assumption and simplifies the computation
as the splitting attribute at each node. ii)
of probabilities. The Naive Bayes classifier
C4.5 Algorithm: C4.5 is an extension of the
works as follows: 1. Training: The algorithm
ID3 algorithm and was also developed by
learns the probability distributions of the
Ross Quinlan. It addresses some limitations
features for each class from a labelled
of ID3 and introduces several
training dataset. It calculates the prior
enhancements. Here are some key features
probability of each class and the
of the C4.5 algorithm: - Handling
conditional probabilities of the features
Continuous Attributes: C4.5 can handle
given each class. 2. Prediction: Given a new
both discrete and continuous attributes. It
instance with observed features, the
uses binary splits for continuous attributes
classifier applies Bayes' theorem to
and selects the best split point based on
calculate the posterior probability of each
information gain or gain ratio. - Handling
class given the features. The class with the
Missing Values: C4.5 can handle missing
highest posterior probability is assigned as
values in the dataset by estimating their
the predicted class label. The key steps in
values during attribute selection and
the Naive Bayes classification algorithm are
classification. iii) CART Algorithm: CART
as follows: - Calculate the prior
(Classification and Regression Trees) is
probabilities of each class based on the
another decision tree algorithm that is
relative frequencies in the training data.
widely used. It was introduced by Leo
Calculate the conditional probabilities of
Breiman and Jerome Friedman. Here are
the features given each class using
some key features of the CART algorithm: -
techniques such as the Gaussian,
Binary Tree: CART constructs binary
multinomial, or Bernoulli distributions,
decision trees, meaning that each internal
depending on the nature of the features. -
node has two branches representing a
Multiply the prior probability with the
binary decision based on a feature. - Gini
product of the conditional probabilities for
Index: CART uses the Gini index as the
each feature. - Normalize the probabilities
criterion for attribute selection.
to obtain the posterior probabilities using
Bayes' theorem. - Assign the class with the
[Link] the following with their with one independent variable is: y = mx +
significance : i) Entropy ii) Information b - "y" represents the dependent variable
gain iii) Gain ratio?? i) Entropy: Entropy is or the target variable. - "x" represents the
a measure of impurity or randomness in a independent variable or the predictor
set of examples or data. In the context of variable. - "m" is the slope of the line,
decision trees, entropy is used to quantify indicating the change in the dependent
the uncertainty or disorder in a node. It variable per unit change in the
helps in determining the best attribute to independent variable. - "b" is the y-
split the data and create decision tree intercept, representing the value of the
nodes. The formula for entropy is typically dependent variable when the independent
based on the proportion of examples variable is zero. Linear regression is used
belonging to each class in the given node. for continuous target variables and is
ii) Information Gain: Information gain is a primarily focused on predicting and
metric used to determine the importance understanding the relationship between
or usefulness of an attribute in a decision variables. It can be extended to multiple
tree. It quantifies the reduction in entropy linear regression when there are multiple
or uncertainty achieved by splitting the independent variables. 2. Logistic
data based on a particular attribute. The Regression: Logistic regression is a
information gain is calculated by taking the classification technique used to predict
difference between the entropy of the categorical outcomes. Unlike linear
parent node and the weighted average of regression, which predicts continuous
entropies of the child nodes created by the values, logistic regression models the
attribute split.. iii) Gain Ratio: The gain probability of an event occurring. The
ratio is an enhancement of information logistic regression model uses the logistic
gain that addresses the bias towards function (also known as the sigmoid
attributes with a large number of outcomes function) to map the input variables to a
or values. It normalizes the information probability value between 0 and 1. The
gain by taking into account the intrinsic equation for logistic regression is: p = 1 / (1
information associated with the attribute, + e^(-z)) - "p" represents the probability of
known as the split information. The gain the event occurring. - "z" is the linear
ratio is calculated by dividing the combination of the independent variables
information gain by the split information. and their coefficients: z = b0 + b1x1 + b2x2
+ ... + bnxn .The coefficients (b0, b1, b2, ...,
[Link] linear and logistic
bn) are estimated using maximum
regression?? 1. Linear Regression: Linear
likelihood estimation or other optimization
regression is a statistical modelling
techniques. These coefficients determine
technique used to establish a relationship
the impact of each independent variable
between a dependent variable and one or
on the probability of the event occurring.
more independent variables. It assumes a
Logistic regression is widely used in binary
linear relationship between the variables
classification problems, where the
and aims to find the best-fitting straight
dependent variable has two categories. It
line that minimizes the distance between
can also be extended to multinomial
the predicted values and the actual values.
logistic regression for problems with more
The equation for a simple linear regression
than two categories.
Q18. What is clustering? Explain its regions of lower density. One of the
Types?? Clustering is a machine learning prominent density-based clustering
technique used to group similar data points algorithms is DBSCAN (Density-Based
together based on their inherent patterns Spatial Clustering of Applications with
or similarities. It helps in identifying Noise). DBSCAN defines clusters as dense
structures or categories within a dataset regions of data points that are separated by
without prior knowledge of the groups. i) areas of lower density. It starts by
Hierarchical Clustering: Hierarchical randomly selecting a data point and
clustering is a clustering technique that expands the cluster by adding nearby
builds nested clusters by iteratively points that have a sufficient number of
merging or dividing clusters. It creates a neighbors within a specified radius. The
hierarchy of clusters, where each cluster process continues until no more points can
can be either a single data point or a group be added, and unvisited points are labeled
of similar data points. There are two main as noise or outliers.
types of hierarchical clustering:
Q19. What is K-means clustering
agglomerative and divisive. Agglomerative
algorithm? Working? Advantages and
clustering starts with each data point as a
Disadvantages? The k-means clustering
separate cluster and then merges the most
algorithm is a popular unsupervised
similar clusters iteratively until a single
machine learning technique used for
cluster is formed. Divisive clustering, on the
partitioning a dataset into k distinct
other hand, starts with all data points in
clusters. It aims to minimize the within-
one cluster and then recursively splits it
cluster variance or sum of squared
into smaller clusters until each cluster
distances between data points and their
contains only one data point. ii)
cluster centroids. The k-means algorithm
Partitioning Clustering: Partitioning
aims to optimize the clustering by
clustering is a technique where the dataset
minimizing the within-cluster variance,
is divided into non-overlapping subsets or
which is often measured using the sum of
partitions. Each partition represents a
squared distances. It seeks to find
cluster, and the goal is to optimize a certain
centroids that are representative of the
criterion, usually the distance between
data points within each cluster while
data points within the same cluster. The
minimizing the distances between the data
most popular algorithm for partitioning
points and their assigned centroids. how
clustering is the k-means algorithm. It
the k-means algorithm works: 1.
starts by randomly initializing k cluster
Initialization: Choose k initial cluster
centers and assigns each data point to the
centroids randomly or based on some
nearest center. It then updates the cluster
heuristics. 2. Assignment: Assign each data
centers based on the mean of the data
point to the nearest centroid based on a
points assigned to each cluster and repeats
distance metric, typically Euclidean
the process until convergence. iii) Density-
distance. 3. Update: Recalculate the
Based Clustering: Density-based clustering
centroids based on the mean of the data
aims to discover clusters of arbitrary shape
points assigned to each cluster. 4. Iteration:
in the data by identifying regions of high-
Repeat the assignment and update steps
density. The key idea is that clusters are
until convergence or a maximum number
areas of higher density separated by
of iterations is reached. Convergence is Components of Time Series Analysis–
reached when the centroids no longer [Link] [Link] Variations [Link]
change significantly or when the Variations Random [Link] Movements.
assignment of data points to clusters
[Link] is Classification in detail?
remains the same. Advantages of the k-
Explain i)Binary ii)Multiclass ? Use cases
means clustering algorithm: 1. Simplicity:
of classification ?? Classification is a
The k-means algorithm is relatively simple
machine learning technique that involves
and computationally efficient, making it
categorizing data into predefined classes or
suitable for large datasets. 2. Scalability: It
categories based on input features. It is a
can handle large datasets with a moderate
supervised learning approach where the
number of clusters and dimensions. 3.
algorithm learns from labeled training data
Interpretable Results: The resulting
to make predictions on unseen data points.
clusters are represented by their centroids,
i) Binary Classification: Binary
which can be easily interpreted.
classification is a type of classification
Disadvantages of the k-means clustering
problem where the task is to assign data
algorithm: 1. Sensitivity to Initial Centroids:
points to one of two possible classes or
The choice of initial centroids can impact
categories. For example, determining
the final clustering result. Different
whether an email is spam or not spam,
initializations may lead to different
predicting whether a customer will churn
outcomes. 2. Dependency on Number of
or not churn, or classifying whether a
Clusters: The number of clusters (k) needs
tumor is malignant or benign. Binary
to be specified in advance, which may
classification algorithms learn to draw a
require prior knowledge or trial-and-error.
decision boundary that separates the two
3. Sensitive to Outliers: Outliers or noisy
classes based on the provided features. ii)
data can significantly affect the cluster
Multiclass Classification: Multiclass
centroids and lead to suboptimal clustering
classification involves categorizing data
results.
points into three or more mutually
[Link] is Time Series Analysis? In order exclusive classes or categories. It can be
to evaluate the performance of a company, thought of as an extension of binary
its past can be compared with the present classification. Examples of multiclass
data. When comparisons of past and classification tasks include classifying
present data are done, the process is images of animals into different species or
known as Time Series Analysis. Time series categorizing news articles into various
are stretched over a period of time rather topics. Multiclass classification algorithms
than being confined to a shorter time learn to differentiate between multiple
period. Time series analysis draws its classes and assign the most appropriate
important because it can help predict the class label to each data point. Use Cases of
future. Depending on the past and future Classification: 1. Sentiment Analysis 2.
trends, time series are able to predict the Fraud Detection 3. Disease Diagnosis 4.
future. Time series analysis is helpful in Image Recognition 5. Document
financial planning as it offers insight into Classification 6. Customer Churn Prediction
the future data depending on the present 7. Credit Risk Assessment
and past data of performance.
Q22. Explain Text analysis with i)POS commonly used for applications like
ii)Lemmatization iii)Stemming?? i) Part-of- information retrieval, search engines, and
Speech (POS) Tagging: POS tagging is the text mining where speed is crucial.
process of assigning grammatical tags to
[Link] Performance
each word in a text corpus, indicating their
Measure?? Classification performance
syntactic role within a sentence. These tags
measures are used to evaluate the
represent the part of speech of each word,
performance of a classification model by
such as noun, verb, adjective, adverb, etc.
comparing its predictions to the actual
POS tagging helps in understanding the
known values of the target variable. These
grammatical structure of a sentence and is
measures provide insights into the
used as a fundamental step in many natural
accuracy, reliability, and effectiveness of
language processing tasks, such as named
the classification model. 1. Accuracy:
entity recognition, sentiment analysis, and
Accuracy is the most basic and widely used
machine translation. ii) Lemmatization:
performance measure. It calculates the
Lemmatization is the process of reducing
proportion of correctly classified instances
words to their base or dictionary form,
(both true positives and true negatives) out
known as the lemma. It aims to convert
of the total number of instances. While
different inflected forms of a word into a
accuracy is easy to interpret, it may not be
common base form for meaningful
suitable for imbalanced datasets where the
analysis. For example, the lemma of the
classes are not evenly represented. 2.
words "running," "ran," and "runs" would
Recall: Recall calculates the proportion of
be "run." Lemmatization takes into account
correctly predicted positive instances (true
the word's morphological analysis,
positives) out of all actual positive
considering factors such as tense, gender,
instances (true positives + false negatives).
and number. It helps in improving text
It assesses the model's ability to capture all
analysis accuracy by grouping together
positive instances and is particularly
words with the same meaning.
important when the cost of false negatives
Lemmatization is useful in various
is high, such as in disease detection. 3.
applications, including information
Confusion Matrix: A confusion matrix
retrieval, text classification, and topic
provides a detailed breakdown of the
modeling. iii) Stemming: Stemming is a
model's predictions. It presents the
simpler text normalization technique
number of true positives, true negatives,
compared to lemmatization. It involves
false positives, and false negatives. It is
reducing words to their root or base form,
useful for analyzing the types of errors the
called the stem, by removing suffixes or
model is making and gaining a deeper
prefixes. Stemming algorithms apply
understanding of its performance.
heuristics without considering the context
or grammar of words. For example, Q24. A confusion matrix is a table that
stemming would convert "running," "ran," provides a detailed breakdown of the
and "runs" to the common stem "run." predictions made by a classification model.
While stemming can be computationally It presents the number of true positives
faster than lemmatization, it may generate (TP), true negatives (TN), false positives
stems that are not actual words or may not (FP), and false negatives (FN). Each row of
capture the precise meaning. Stemming is the matrix represents the instances in a
predicted class, while each column identify patterns, communicate findings,
represents the instances in an actual class. and make informed decisions. Challenges
The confusion matrix allows for a in Big Data Visualization →1. Scalability:
comprehensive analysis of a model's Big data often consists of massive datasets
performance and the types of errors it is that exceed the processing and memory
making. A confusion matrix allows you to capabilities of traditional visualization
derive various performance metrics for tools. Visualizing large-scale datasets
evaluating the model, such as accuracy, requires scalable and distributed
precision, recall, and specificity. Here's how computing frameworks to handle the data
these metrics are calculated based on the processing and rendering efficiently. 2.
values in the confusion matrix: - Accuracy: Performance: The complexity and volume
It measures the overall correctness of the of big data can significantly impact the
model's predictions and is calculated as (TP performance of visualization processes.
+ TN) / (TP + TN + FP + FN). - Precision: It Visualization techniques need to be
assesses the proportion of correctly optimized to handle large datasets and
predicted positive instances out of all deliver real-time or near-real-time
instances predicted as positive and is performance. 3. Data Variety and
calculated as TP / (TP + FP). Precision Complexity: Big data comes in various
focuses on the model's ability to avoid false forms, including structured, unstructured,
positives. - Recall (Sensitivity or True and semi-structured data. Visualizing
Positive Rate): It measures the proportion heterogeneous data sources and handling
of correctly predicted positive instances the complexity of diverse data formats and
out of all actual positive instances and is structures can be challenging. 4. Data
calculated as TP / (TP + FN). Recall assesses Integration: Big data visualization often
the model's ability to capture all positive involves integrating data from multiple
instances. - Specificity: It measures the sources or data streams. Handling data
proportion of correctly predicted negative integration complexities, ensuring data
instances out of all actual negative consistency, and maintaining data quality
instances and is calculated as TN / (TN + become significant challenges in the
FP). Specificity indicates the model's ability visualization process. 5. Visualization
to correctly identify negative instances. Design: Designing effective visualizations
for big data requires careful consideration
[Link] is data visualization? Also
of the information density, data
Challenges in Big data visualization?? Data
abstraction, and interactivity. Choosing
visualization refers to the representation of
appropriate visualization techniques and
data in a visual or graphical form to
representations to convey meaningful
facilitate the understanding of patterns,
insights can be challenging due to the
trends, and insights hidden within the data.
complexity and size of the data.
It involves the use of visual elements, such
as charts, graphs, maps, and infographics,
to present complex data in a more
accessible and intuitive way. By leveraging
the power of visualization, data analysts
and decision-makers can explore data,
Q26. Types of visualization Network visualization, also known as graph
i)Multidimentionl:2D ii)Temporal visualization, focuses on visualizing the
iii)Hierarchicaliv)Network?? i) relationships and connections between
Multidimensional (2D) Visualization: entities or nodes. It is used to analyze and
Multidimensional visualization, also known understand complex networks, such as
as 2D visualization, aims to represent data social networks, transportation networks,
that consists of multiple attributes or or computer networks. Network
dimensions. It involves plotting data points visualizations commonly use node-link
on a two-dimensional plane, where each diagrams, force-directed layouts, or matrix-
axis represents a specific attribute or based representations to depict nodes as
dimension. Common examples of entities and edges as connections between
multidimensional visualizations include them. Network visualization allows for the
scatter plots, bubble charts, and parallel identification of central nodes, clusters,
coordinate plots. This type of visualization and patterns of connectivity within the
allows for the exploration and network.
understanding of relationships and
Q27. Data Visualization Techniques ??i)
patterns between multiple variables
Line Graph: A line graph is a type of
simultaneously. ii) Temporal Visualization:
visualization that represents data points as
Temporal visualization focuses on
connected line segments. It is commonly
visualizing data that varies over time. It is
used to display trends and patterns over a
used to analyze and understand patterns,
continuous interval or time series. In a line
trends, and changes in data across different
graph, the x-axis typically represents the
time periods. Examples of temporal
independent variable (e.g., time) and the y-
visualizations include line charts, area
axis represents the dependent variable
charts, and heatmaps that display data
(e.g., quantity). Line graphs are effective in
over time. Temporal visualization is widely
showing changes over time, identifying
used in various fields, such as financial
relationships between variables, and
analysis, climate studies, and social media
comparing multiple series of data. ii) Pie
analysis, to examine temporal patterns and
Chart: A pie chart is a circular visualization
make time-based predictions. iii)
divided into slices, where each slice
Hierarchical Visualization: Hierarchical
represents a category or a proportion of a
visualization represents data in a
whole. The size of each slice corresponds to
hierarchical or tree-like structure, where
the relative proportion of the data it
entities or elements are organized based
represents. Pie charts are useful for
on hierarchical relationships. It visualizes
displaying categorical data and visualizing
the relationships and dependencies
the composition or distribution of different
between different levels of a hierarchy.
categories. They are commonly used to
Tree maps, sunburst charts, and
present percentages or proportions and
dendrograms are commonly used
provide a quick overview of relative
hierarchical visualizations. They help in
magnitudes. iii) Venn Diagram: A Venn
understanding the composition, structure,
diagram is a visualization technique that
and clustering of hierarchical data, such as
uses overlapping circles or shapes to show
organizational structures, file systems, or
the logical relationships between different
taxonomies. iv) Network Visualization:
sets or groups. Each circle represents a set, data preparation and exploration
and the overlapping regions represent the capabilities along with visualization. iii)
intersections between sets. Venn diagrams JasperReports: JasperReports is an open-
are helpful in illustrating commonalities source reporting library that allows
and differences among sets, identifying developers to embed sophisticated
overlaps, and understanding relationships reporting and visualization capabilities into
between elements. iv) Scatter Diagram: A their applications. It provides a wide range
scatter diagram, also known as a scatter of chart types, including bar charts, line
plot, is a two-dimensional visualization that charts, pie charts, and more, to visualize
represents data points as individual dots on Big Data. JasperReports supports multiple
a Cartesian plane. It displays the data sources and offers advanced features
relationship between two numerical such as drill-down, interactive filtering, and
variables, with one variable plotted on the report scheduling. It is highly customizable
x-axis and the other on the y-axis. Scatter and widely used in enterprise applications
plots are useful for identifying patterns, for Big Data reporting and visualization. iv)
correlations, and clusters in data. They help Tableau: Tableau is a popular and powerful
in understanding the relationship between data visualization tool that offers a range of
variables, detecting outliers, and features to visualize and analyze Big Data.
visualizing the distribution of data points. It provides an intuitive drag-and-drop
interface that allows users to create
Q28. Big Data Visualization Tools? i)
interactive dashboards, charts, maps, and
Pentaho: Pentaho is a comprehensive data
other visualizations without the need for
integration and business analytics platform
coding. Tableau can connect to various
that includes robust visualization
data sources, including Big Data platforms
capabilities. It offers interactive and
like Hadoop, and provides advanced
customizable dashboards, charts, and
analytics capabilities. It offers features like
reports for visualizing Big Data. Pentaho
data blending, real-time data visualization,
supports a wide range of data sources and
and sharing and collaboration options.
allows users to create visually appealing
visualizations to explore and analyze large
datasets. It provides features for data
blending, interactive filtering, and drill-
down capabilities to dive deeper into the
data. ii) Datameer: Datameer is a Big Data
analytics and visualization platform that
enables users to explore, transform, and
visualize large datasets without the need
for coding. It offers a visually intuitive
interface that allows users to build
interactive visualizations, such as charts,
graphs, and maps, to gain insights from Big
Data. Datameer integrates with various
data sources, including Hadoop, NoSQL
databases, and cloud storage, and provides
Q29. Explain Tableau in Detail?? Tableau is Tableau Server or Tableau Online, enabling
a widely used data visualization and colleagues and stakeholders to access and
business intelligence tool that helps interact with the visualizations through a
organizations transform raw data into web browser or mobile device. Tableau
interactive and meaningful visualizations, also allows for embedding visualizations
dashboards, and reports. It provides a user- into websites or sharing them as static files.
friendly and intuitive interface, making it 5. Scalability and Performance: Tableau is
accessible to both technical and non- designed to handle large datasets and
technical users. Tableau is widely used in provides efficient data processing and
various industries and functions, including rendering capabilities. It can leverage in-
business intelligence, data analytics, memory technology for faster data analysis
finance, marketing, healthcare, and more. and supports data extraction and live
It empowers users to visually explore and connectivity options.
analyze data, derive insights, and make
Q30. Hadoop Ecosystem?? The Hadoop
data-driven decisions. The flexibility,
ecosystem refers to a collection of open-
interactivity, and rich visualization options
source software frameworks and tools built
of Tableau make it a popular choice for
around the Apache Hadoop platform.
organizations seeking to unleash the power
Hadoop provides a scalable and distributed
of their data. Features of Tableau: 1. Data
computing framework for processing and
Connection and Integration: Tableau can
storing large volumes of data across
connect to various data sources, including
clusters of commodity hardware. The
databases, spreadsheets, cloud services,
ecosystem expands on the core capabilities
and Big Data platforms like Hadoop. It
of Hadoop and includes various
supports both structured and unstructured
components that enhance its functionality
data and provides options for data
and address different aspects of Big Data
blending and data preparation. 2. Drag-
processing. Here are some key components
and-Drop Interface: Tableau offers a drag-
of the Hadoop ecosystem: 1. Hadoop
and-drop interface, allowing users to easily
Distributed File System (HDFS): HDFS is a
create visualizations without the need for
distributed file system that allows for the
coding or complex queries. Users can
storage and retrieval of data across
simply drag data fields onto the canvas and
multiple machines in a Hadoop cluster. It
choose from a wide range of visualization
provides fault tolerance, high availability,
types. 3. Interactive Visualizations:
and scalability for handling large datasets.
Tableau enables users to create interactive
2. MapReduce: MapReduce is a
visualizations that can be explored and
programming model and processing
manipulated in real-time. Users can filter,
framework for distributed computing in
sort, drill down, and highlight data points
Hadoop. It allows developers to write
to uncover insights and answer ad-hoc
parallelizable code that is executed across
questions. This interactivity promotes a
the nodes of a Hadoop cluster. MapReduce
data-driven and exploratory approach to
enables the efficient processing of large-
analysis. 4. Sharing and Collaboration:
scale data by dividing tasks into map and
Tableau provides options for sharing
reduce phases. 3. Apache Hive: Hive is a
visualizations, dashboards, and reports
data warehousing framework built on top
with others. Users can publish their work to
of Hadoop that provides a SQL-like query parallel and distributed processing of data
language called HiveQL. It allows users to in a fault-tolerant manner. Components Of
query and analyze data stored in Hadoop HDFS: 1. NameNode: The NameNode is
using familiar SQL syntax and supports data the master node in the HDFS cluster and
summarization, ad-hoc queries, and data acts as the central metadata repository. It
analysis. 4. Apache Pig: Pig is a high-level stores the file system namespace, including
scripting language and data flow platform information about files, directories, and
designed for parallel processing of large their corresponding block locations. The
datasets in Hadoop. It simplifies the NameNode maintains the file system tree
development of complex data processing and coordinates access to data blocks. It
tasks by providing a procedural language keeps track of the location of data blocks
called Pig Latin. 5. Apache Spark: Spark is a across the DataNodes. 2. DataNode:
fast and general-purpose cluster DataNodes are worker nodes in the HDFS
computing framework that provides in- cluster responsible for storing and
memory data processing capabilities. It managing the actual data blocks. Each
offers a wide range of APIs and libraries for DataNode manages the data blocks of the
batch processing, real-time streaming, local storage devices attached to it. They
machine learning, and graph processing. communicate with the NameNode and
Spark can be seamlessly integrated with report the list of data blocks they store.
Hadoop and provides enhanced DataNodes handle read and write requests
performance for data processing tasks. 6. from clients, replicate data blocks, and
Apache HBase: HBase is a distributed, perform data block recovery in case of
scalable, and non-relational database built failures. 3. Block: Data in HDFS is stored in
on top of Hadoop. It provides random and blocks, typically with a default size of 128
real-time read/write access to large MB or 256 MB. Files are split into these
volumes of structured data, making it fixed-size blocks, and each block is
suitable for use cases that require low- independently replicated across multiple
latency data access. DataNodes for fault tolerance. The block
size provides advantages for efficient data
Q31. Explain HDFS Architecture ?? The
storage and processing in a distributed
Hadoop Distributed File System (HDFS) is
environment. 4. Secondary NameNode:
the primary storage system used by
The Secondary NameNode is a helper node
Hadoop for storing and managing large
that periodically checkpoints the metadata
volumes of data across a cluster of
of the HDFS cluster. It does not act as a
machines. The architecture of HDFS is
backup or failover for the primary
designed to provide fault tolerance, high
NameNode. Instead, it helps the
availability, and scalability. The HDFS
NameNode by combining the edit log and
architecture follows a master-slave design,
file system image to create a new
where the NameNode acts as the master
checkpoint. This process reduces the
node responsible for managing metadata,
startup time of the NameNode in case of
and the DataNodes act as slave nodes
failures.
responsible for storing and managing data
blocks. The architecture is optimized for
large-scale data storage and enables
Q32. MapReduce?? MapReduce is a transformation and analysis tasks without
programming model and processing the need for complex MapReduce code. Pig
framework designed for parallel and Latin scripts are compiled into MapReduce
distributed processing of large-scale data jobs and executed on a Hadoop cluster. Pig
sets across a cluster of computers. It was provides a rich set of operators and
introduced by Google and popularized by functions for data manipulation, joining,
Apache Hadoop, which implemented the filtering, and aggregation. It simplifies the
MapReduce framework as part of its process of writing data processing tasks
ecosystem. The MapReduce framework and enables faster development and
handles the distribution of tasks across the prototyping. ii) Hive: Hive is a data
cluster, manages the communication warehousing framework built on top of
between the nodes, and ensures fault Hadoop that provides a SQL-like query
tolerance. It automatically splits input data language called HiveQL. It allows users to
into chunks, assigns Map and Reduce tasks query and analyze large datasets stored in
to available worker nodes, and handles Hadoop using a familiar SQL syntax. Hive
node failures by reassigning tasks to other translates HiveQL queries into MapReduce
nodes. Benefits of MapReduce: - jobs or other execution engines compatible
Scalability: MapReduce enables the with Hadoop. It provides schema-on-read
processing of large volumes of data by capabilities, meaning that the data
distributing the work across multiple nodes structure and schema can be defined at the
in a cluster, providing scalability as the time of querying rather than during data
cluster size increases. - Fault Tolerance: ingestion. Hive supports various data
MapReduce handles node failures and formats, including CSV, JSON, Parquet, and
automatically reassigns tasks to ensure ORC. It is commonly used for data
fault tolerance. If a node fails, its tasks are exploration, ad-hoc querying, and business
reassigned to other nodes, minimizing the intelligence tasks on large-scale datasets.
impact on overall performance. - Data iii) HBase: HBase is a distributed, scalable,
Locality: MapReduce optimizes and non-relational database built on top of
performance by scheduling tasks on nodes Hadoop. It provides random and real-time
that already contain the required input read/write access to large volumes of
data. This reduces network traffic and structured data. HBase is designed to
improves processing efficiency. - Parallel handle high-velocity and high-volume data
Processing: MapReduce allows for parallel with low-latency requirements. It offers a
processing of data, as the Map and Reduce flexible data model similar to Google's
tasks can be executed independently on Bigtable, where data is organized into rows,
different nodes. This enables faster data columns, and column families. HBase
processing and analysis. supports automatic sharding and
replication for fault tolerance and high
Q33. Pig, Hive, HBase and Mahout ?? → i)
availability. It is suitable for use cases that
Pig: Pig is a high-level scripting language
require low-latency access to vast amounts
and data flow platform built on top of
of structured data, such as sensor data,
Apache Hadoop. It provides a simplified
time series data, and real-time analytics. iv)
and expressive language called Pig Latin,
Mahout: Mahout is a scalable machine
which allows users to write data
learning and data mining library built on
top of Hadoop. It provides a set of clusters using different colors, shapes, or
algorithms and tools for building and sizes, allowing users to identify distinct
deploying machine learning models on groups and understand the underlying
large datasets. Mahout includes algorithms patterns and structures in the data. iii)
for classification, clustering, Regression: Regression analysis is a
recommendation, and collaborative statistical technique used to model and
filtering. It leverages the parallel analyze the relationship between a
processing capabilities of Hadoop to dependent variable and one or more
efficiently train models and process large- independent variables. It helps understand
scale data. Mahout supports integration how the value of the dependent variable
with other Hadoop ecosystem components changes as the independent variables vary.
like HDFS and MapReduce, making it easy In big data visualization, regression analysis
to incorporate machine learning tasks into can be used to create scatter plots, trend
Hadoop workflows. It is widely used for lines, or heat maps that illustrate the
building predictive models, personalization relationship between variables. It enables
systems, and recommendation engines in the visualization of trends, correlations,
Big Data environments. and predictive patterns in the data,
allowing for insights into the impact of
Q34. Analytical Techniques in big data
independent variables on the dependent
visualization?? → i) Classification:
variable. iv) Association: Association
Classification is an analytical technique
analysis, also known as market basket
used in big data visualization to categorize
analysis, is a technique used to discover
data into predefined classes or groups
interesting relationships or associations
based on a set of features or attributes. It
between items in a dataset. It identifies
involves building a classification model
frequent itemsets and generates
using machine learning algorithms that can
association rules that describe the co-
assign new data points to the appropriate
occurrence patterns among different
class. In the context of visualization,
items. In big data visualization, association
classification can be used to label data
analysis can be represented using
points, assign colors or shapes to different
visualizations like network diagrams or
categories, and enable the visual
chord diagrams, showing the relationships
exploration of patterns and relationships
between items or entities. It helps uncover
between classes. ii) Clustering: Clustering
hidden associations, dependencies, and
is an analytical technique that groups
patterns in the data, enabling businesses to
similar data points together based on their
make data-driven decisions, such as cross-
intrinsic characteristics or similarities. It
selling or personalized recommendations.
aims to discover natural clusters or
patterns within the data without prior
knowledge of the class labels. Clustering
algorithms partition data points into
clusters such that points within the same
cluster are more similar to each other than
to those in other clusters. Big data
visualization techniques can represent