Statistical Learning
Feature Extraction
Feature extraction is a machine learning and data analysis process
that transforms raw data into numerical features.
Feature extraction refers to the process of transforming raw data into
numerical features that can be processed while preserving the
information in the original data set.
Feature extraction is a process used in machine learning to reduce
the number of resources needed for processing without losing
important or relevant information.
Why is Feature Extraction
Important?
Reduction of Computational Cost: By reducing the dimensionality of the
data, machine learning algorithms can run more quickly.
Improved Performance: Algorithms often perform better with a reduced
number of features. This is because noise and irrelevant details are removed,
allowing the algorithm to focus on the most important aspects of the data.
Prevention of Overfitting: With too many features, models can become
overfitted to the training data, meaning they may not generalize well to new,
unseen data. Feature extraction helps to prevent this by simplifying the model.
Better Understanding of Data: Extracting and selecting important features
can provide insights into the underlying processes that generated the data.
Principal Component Analysis
Principal Component Analysis is an unsupervised learning algorithm
that is used for the dimensionality(or features) reduction in machine
learning.
Dimensionality reduction in machine learning refers to the process of
reducing the number of random variables (or features) under
consideration.
This reduction can be achieved by obtaining a set of principal
variables.
Dimensionality reduction can be used for feature selection, feature
extraction, or a combination of the two.
Principal Component Analysis (PCA) is an unsupervised learning
algorithm technique used to examine the interrelations among a set
of variables. It is also known as a general factor analysis
Principal Component Analysis
Principal Component Analysis (PCA) is a statistical procedure that
uses an orthogonal transformation that converts a set of correlated
variables to a set of uncorrelated variables.
Set of Correlated Variables: PCA is typically applied to a dataset
containing multiple correlated variables (features). The goal of PCA is
to find a new set of variables (principal components) that are linear
combinations of the original variables but are uncorrelated with each
other.
Uncorrelated Variables: The principal components derived from
PCA are uncorrelated, which means that the covariance between any
pair of principal components is zero.
The PCA algorithm is based on some mathematical concepts such as:
• Variance and Covariance
• Eigenvalues and Eigen vectors
STEP 1: Standardization
STEP 2: Covariance Matrix
Computation
Step 3: Compute Eigenvalues and
Eigenvectors of Covariance Matrix to Identify
Principal Components
Singular Value Decomposition(SVD)
The singular value decomposition(SVD) of a matrix is a factorization
of that matrix into three matrices.
Singular-value decomposition is also one of the popular
dimensionality reduction techniques.
It is the matrix-factorization method of linear algebra, and it is widely
used in different applications such as feature selection, visualization,
noise reduction, and many more.
Intensity Level of the image(Which is applicable in Discrete signal)
Since an image is contiguous, the values of most pixels depend on
the pixels around them.
Application of SVD
Image Recovery
Image Compression
Feature Selection
While developing the machine learning model, only a few variables in
the dataset are useful for building the model, and the rest features
are either redundant or irrelevant.
If we input the dataset with all these redundant and irrelevant
features, it may negatively impact and reduce the overall
performance and accuracy of the model.
A feature is an attribute that has an impact on a problem or is useful
for the problem, and choosing the important features for the model is
known as feature selection.
The main difference between them is that feature selection is about
selecting the subset of the original feature set, whereas feature
extraction creates new features.
Feature selection is a way of reducing the input variable for the model
by using only relevant data to reduce overfitting in the model.
It is a process of automatically or manually selecting the
subset of most appropriate and relevant features to be used
in model building.
Need for Feature Selection
Before implementing any technique, it is really important to
understand, need for the technique and so for the Feature Selection.
As we know, in machine learning, it is necessary to provide a pre-
processed and good input dataset in order to get better outcomes.
We collect a huge amount of data to train our model and help it to
learn better. Generally, the dataset consists of noisy data, irrelevant
data, and some part of useful data. Moreover, the huge amount of
data also slows down the training process of the model, and with
noise and irrelevant data, the model may not predict and perform
well.
Benefits of using feature selection in
machine learning:
• It helps in avoiding the curse of dimensionality.
• It helps in the simplification of the model so that it can be
easily interpreted by the researchers.
• It reduces the training time.
• It reduces overfitting hence enhance the generalization.
Feature Selection Techniques
Wrapper Method
In wrapper methods, different subsets of features are evaluated by
training a model for each subset, and then the performance is
compared and the right combination is chosen.
Some techniques of wrapper
methods:
Forward selection
Backward elimination
Exhaustive Feature Selection
Recursive Feature Elimination
Forward Selection
• Starting from Scratch: Begin with an empty set of features and
iteratively add one feature at a time.
• Model Evaluation: At each step, train and evaluate the machine
learning model using the selected features.
• Stopping Criterion: Continue until a predefined stopping criterion is
met, such as a maximum number of features or a significant drop in
performance.
Backward Elimination
• Starting with Everything: Start with all available features.
• Iterative Removal: In each iteration, remove the least important
feature and evaluate the model.
• Stopping Criterion: Continue until a stopping condition is met.
Exhaustive Search
• Exploring All Possibilities: Evaluate all possible combinations of
features, which ensures finding the best subset for model
performance.
• Computational Cost: This can be computationally expensive,
especially with a large number of features.
Recursive Feature Elimination (RFE)
• Ranking Features: Start with all features and rank them based on
their importance or contribution to the model.
• Iterative Removal: In each iteration, remove the least important
feature(s).
• Stopping Criterion: Continue until a desired number of features is
reached.
Filter Method
These methods are generally used while doing the pre-processing
step. These methods select features from the dataset irrespective of
the use of any machine learning algorithm.
In terms of computation, they are very fast and inexpensive and are
very good for removing duplicated, correlated, redundant features
but these methods do not remove multicollinearity.
Techniques
Information Gain
Chi-square test
Fisher’s Score
Correlation Coefficient
Information Gain
It is defined as the amount of information provided by the feature for
identifying the target value and measures reduction in the entropy
values. Information gain of each attribute is calculated considering
the target values for feature selection.
In the context of information theory and machine learning, entropy
is a measure of uncertainty or randomness associated with a random
variable.
Chi-square test
Chi-square method (X2) is generally used to test the relationship
between categorical variables. It compares the observed values from
different attributes of the dataset to its expected value.
Fisher’s Score
Fisher’s Score selects each feature independently according to their
scores under Fisher criterion leading to a suboptimal set of features.
The larger the Fisher’s score is, the better is the selected feature.
Correlation Coefficient
Pearson’s Correlation Coefficient is a measure of quantifying the
association between the two continuous variables and the direction of
the relationship with its values ranging from -1 to 1.
Embedded Method
Embedded methods combine the advantageous aspects of both Filter
and Wrapper methods.
Embedded methods combined the advantages of both filter and
wrapper methods by considering the interaction of features along
with low computational cost.
These are fast processing methods similar to the filter method but
more accurate than the filter method.
These methods are also iterative, which evaluates each iteration, and
optimally finds the most important features that contribute the most
to training in a particular iteration. :
Some techniques of embedded
methods
Regularization
Random Forest Importance
Regularization
Regularization adds a penalty term to different parameters of the
machine learning model to avoid overfitting in the model.
This penalty term is added to the coefficients; hence it shrinks some
coefficients to zero. Those features with zero coefficients can be
removed from the dataset.
The types of regularization techniques are L1 Regularization (Lasso
Regularization) or d L2 regularization(Ridge Regularization)
Random Forest Importance
Different tree-based methods of feature selection help us with feature
importance to provide a way of selecting features. Here, feature
importance specifies which feature has more importance in model
building or has a great impact on the target variable.
Random Forest is a tree-based method, which is a type of bagging
algorithm that aggregates a different number of decision trees.
Evaluating ML Algo and Model
Selection
Model evaluation is the process that uses some metrics that help us
to analyze the performance of the model.
Accuracy
Accuracy: Accuracy is defined as the ratio of the number of correct
predictions to the total number of predictions.
accuracy_score
Precision
Precision is the ratio of true positives to the summation of true
positives and false positives. It basically analyses the positive
predictions.
The drawback of Precision is that it does not consider the True
Negatives and False Negatives.
precision_score()
Recall
Recall is the ratio of true positives to the summation of true positives
and false negatives. It basically analyses the number of correct
positive samples.
recall_score()
Confusion Matrix
The confusion matrix is a representation of the accuracy of the
classification model.
True Positive: - This is the number of times the model predicted an
independent variable as positive when the actual value was positive.
False Positive: - This is the number of times the model predicted an
independent variable as positive when the actual value was negative.
False Negative: - This is the number of times the model predicted
an independent variable as negative when the actual value was
positive.
True Negative: - This is the number of times the model predicted an
independent variable as negative when the actual value was
negative.
