3.

Assignment 3
What is convex function. Write the criteria for convex function in the mathematical
form.
A convex function is a mathematical function that satisfies a specific property related to
the shape of its graph. A function is considered convex if, for any two points within its
domain, the line segment connecting those two points lies above or on the graph of the
function.
Mathematically, a function f(x) defined on a convex set S is convex if, for all ( ) and in S
x1 x2

and for all in the interval ([0, 1]), the following inequality holds:
(λ)

In simpler terms, this inequality states that if you take two points, (x_1) and , within the
x2

domain of the function, and any point along the line segment that connects them (
), the value of the function at that point must be less than or equal to the
λx 1 + (1 − λ)x 2

weighted average of the function values at and , with weights and

x1 x2 λ (1 − λ), respectively.
This criterion implies that a convex function has a graph that curves upward and doesn't have
any "dips" or "bumps" between any two points within its domain. Common examples of
convex functions include linear functions, quadratic functions with positive leading
coefficients, and exponential functions.
Convex functions have important properties and are commonly encountered in optimization
problems, where the goal is to find the minimum (or maximum) of a function subject to
certain constraints. Convex optimization problems have well-defined solutions and can be
efficiently solved, making convexity a crucial concept in various fields, including machine
learning, economics, and engineering.
A function is strictly convex if the line segment connecting any two points on the graph of
f (x)lies strictly above the graph (excluding endpoints)
Convex Condition
f (θx + (1 − θ)y) ≤ θf (x) + (1 − θ)f (y)

for all θ ∈ (0, 1)

Functions that are convex functions, have a global minimum value that may be present at
multiple points in the graph of the function.
Observe how the minimum value of 0 for the function is present at multiple values for the
input value
Strictly Convex Condition
f (θx + (1 − θ)y < θf (x) + (1 − θ)f (y)

for all θ ∈ (0, 1)

Write the KKT Conditions for constrained optimization problem.

The Karush-Kuhn-Tucker (KKT) conditions are necessary conditions for an optimization
problem with inequality and equality constraints. These conditions are used to determine
whether a particular set of candidate solutions satisfies the constraints and optimality
conditions. Here are the KKT conditions for a general constrained optimization problem:
Consider the following optimization problem:
Minimize:
f (x)

Subject to:
g i (x) ≤ 0, i = 1, 2, … , m

h j (x) = 0, j = 1, 2, … , p

Where:
is the vector of variables to be optimized.
x

is the objective function to be minimized.

f (x)

are inequality constraints.

g i (x)

are equality constraints.

h j (x)

First derivative tests for a solution to be optimal, provide that some regularity conditions
are fulfilled.
It generalizes the method of Lagrange multipliers, which allows only equality constraints.
Stationary:
m p
∇ x L(x, λ) = ∇f (x) + ∑ λ i ∇g i (x) + ∑ ν j ∇h j (x) = 0
i=1 j=1

This condition ensures that the gradient of the Lagrangian with respect to (x) is zero.
Primal Feasibility
∗
h j (x ) = 0, f or j = 1, … , ℓ

∗
g i (x ) ≤ 0, f or i = 1, … , m

Dual Feasibility
λ i ≥ 0, f or i = 1,....,m

Complementary Slackness
m ∗
∑ λ i g i (x ) = 0
i=1

These KKT conditions together provide necessary conditions for optimality in

constrained optimization problems. Satisfying these conditions indicates that a
candidate solution could be a local optimum. However, the KKT conditions do not
x

guarantee global optimality in non-convex problems; additional analysis may be

required.
Explain Stochastic gradient method and its advantage over steepest gradient
method.
Stochastic Gradient Descent (SGD) is an optimization technique commonly used in
machine learning for training models, especially in scenarios with large datasets. Its primary
advantage over the steepest gradient method (also known as Batch Gradient Descent) lies in
its efficiency and ability to handle big data. Here's a brief explanation of SGD and its
advantage:
Stochastic Gradient Descent (SGD):
Working Principle: In SGD, the model's parameters are updated iteratively, but instead
of computing the gradient of the loss function using the entire dataset (as in Batch
Gradient Descent), SGD computes the gradient using only one randomly selected data
point (or a small random subset called a mini-batch) at each iteration.
 Advantage: SGD's main advantage is its speed and efficiency. By using only a subset of
the data at each iteration, it processes data much faster, making it suitable for large
datasets. It also converges faster because it frequently updates the model parameters.
Stochastic Nature: Because SGD uses random data points, it introduces noise into the
optimization process. While this noise can make the convergence path noisy, it can also
help the algorithm escape local minima and find better solutions, especially in non-
convex optimization problems.
Online Learning: SGD is well-suited for online learning scenarios, where the model is
updated continuously as new data becomes available.
Advantage of SGD Over Steepest Gradient Method (Batch Gradient Descent):
Efficiency: Batch Gradient Descent computes gradients using the entire dataset, which
can be computationally expensive and slow for large datasets. In contrast, SGD uses a
small random subset of data, making it much faster.
Parallelization: SGD is highly parallelizable, allowing it to take advantage of multi-core
processors and distributed computing, further speeding up training.
Escape Local Minima: The stochastic nature of SGD helps it escape local minima more
effectively, as the noise introduced by random data points can push the optimization
process out of local minima, resulting in potentially better solutions.
Regularization Effect: The noise in SGD can act as implicit regularization, preventing
overfitting and improving the generalization ability of the model.
In summary, SGD's main advantage over the steepest gradient method (Batch Gradient
Descent) is its efficiency, scalability, and ability to escape local minima more effectively. It is
particularly well-suited for training machine learning models on large datasets and in online
learning scenarios.
What is linear regression, and how is it used in machine learning? Explain the
difference between simple linear regression and multiple linear regression.
Linear Regression is a supervised machine learning algorithm used for modeling the
relationship between a dependent variable (target) and one or more independent variables
(features or predictors) by fitting a linear equation to the observed data. Its primary goal is to
establish a linear relationship that can be used to make predictions or understand the
relationship between variables.
Here's an overview of how linear regression works and its two main variants: Simple Linear
Regression and Multiple Linear Regression.
1. Simple Linear Regression:
Objective: Simple Linear Regression aims to establish a linear relationship between a
single independent variable (predictor) and a dependent variable (target).
Equation: The linear regression equation for simple linear regression is typically
expressed as:
 y = mx + b

'y' is the dependent variable (target).

'x' is the independent variable (predictor).
'm' is the slope or coefficient, representing the change in 'y' for a unit change in 'x.'
'b' is the intercept, representing the value of 'y' when 'x' is zero.
Goal: The goal of simple linear regression is to find the best-fitting line (a straight line)
that minimizes the sum of squared differences between the observed values and the
predicted values.
Use Cases: Simple linear regression is suitable when there is a clear linear relationship
between two variables, such as predicting house prices based on the number of
bedrooms.
2. Multiple Linear Regression:
Objective: Multiple Linear Regression extends the concept of linear regression to
multiple independent variables. It models the relationship between the dependent
variable and two or more independent variables.
Equation: The multiple linear regression equation is an extension of the simple linear
regression equation:

y = b0 + b1*x1 + b2*x2 + ... + bn*xn

'y' is the dependent variable (target).

'x1', 'x2', ..., 'xn' are the independent variables (predictors).
'b0' is the intercept, representing the value of 'y' when all predictors are zero.
'b1', 'b2', ..., 'bn' are the coefficients for each predictor, indicating their respective
impact on 'y.'
Goal: Similar to simple linear regression, the goal is to find the best-fitting hyperplane (a
multi-dimensional straight line) that minimizes the sum of squared differences between
the observed values and the predicted values.
Use Cases: Multiple Linear Regression is used when there are multiple factors
influencing the dependent variable. For example, predicting a car's price based on
factors like mileage, age, and engine size.
Key Differences:
Number of Predictors: Simple linear regression involves a single predictor variable,
while multiple linear regression involves two or more predictor variables.
Equation Complexity: Simple linear regression has a simpler linear equation with one
coefficient, while multiple linear regression has a more complex linear equation with
multiple coefficients.
Model Complexity: Multiple linear regression can capture more complex relationships
between the dependent variable and multiple predictors, allowing for a better fit to real-
world data.
 Use Case: Simple linear regression is suitable when you're analyzing the relationship
between two variables, whereas multiple linear regression is appropriate when multiple
predictors influence the outcome.
Explain the k-Nearest Neighbors (KNN) algorithm and its basic working principle ?
The k-Nearest Neighbors (KNN) algorithm is a simple yet powerful supervised machine
learning algorithm used for both classification and regression tasks. It is a non-parametric
and instance-based algorithm, meaning it doesn't make assumptions about the underlying
data distribution and makes predictions based on the similarity between data points. Here's
an explanation of how the KNN algorithm works:
Basic Working Principle of KNN:
1. Data Representation: The first step in KNN is to represent your data as data points in a
multi-dimensional space, where each feature of your data becomes a dimension. For
example, if you have a dataset with two features (e.g., height and weight of individuals),
you can represent each data point as a point in a 2D space.
2. Choose a Value for 'k': You need to choose a positive integer value 'k,' which represents
the number of nearest neighbors to consider when making predictions. The choice of 'k'
is a crucial parameter in KNN and can significantly affect the algorithm's performance.
3. Distance Metric: You also need to select a distance metric (e.g., Euclidean distance,
Manhattan distance, etc.) to measure the similarity or distance between data points in
the feature space. The choice of distance metric depends on the nature of your data and
the problem you're trying to solve.
4. Prediction for Classification:
For classification tasks, when you want to predict the class label of a new data point,
KNN works as follows:
Calculate the distance between the new data point and all other data points in the
dataset using the chosen distance metric.
Select the 'k' nearest data points (i.e., those with the smallest distances) to the new
point.
Count the frequency of each class label among these 'k' neighbors.
Assign the class label to the new data point based on the majority class among the
neighbors (i.e., the class with the highest frequency).
5. Prediction for Regression:
For regression tasks, when you want to predict a continuous value for a new data
point, KNN works as follows:
Calculate the distance between the new data point and all other data points in the
dataset.
Select the 'k' nearest data points.
Calculate the average (or weighted average) of the target values of these 'k'
neighbors.
Assign this average value as the prediction for the new data point.
What is the impact of choosing different values of 'k' in the KNN algorithm?
 1. In the k-Nearest Neighbors (KNN) algorithm, the value of 'k' plays a crucial role in
determining the algorithm's behavior and performance. It controls the number of
nearest neighbors considered when making predictions. Choosing different values of 'k'
can have various impacts on the algorithm:
1. Classification Accuracy: The choice of 'k' can significantly impact the accuracy of
the KNN classifier. Smaller values of 'k' (e.g., k=1) can result in a more flexible model
that closely fits the training data but may be sensitive to noise. Larger values of 'k'
(e.g., k=10) can lead to a smoother decision boundary but may oversmooth and
generalize too much.
2. Overfitting and Underfitting: Small values of 'k' are prone to overfitting, meaning
the model may capture noise in the training data. On the other hand, large values of
'k' can lead to underfitting, where the model may oversimplify and fail to capture
important patterns in the data.
3. Bias-Variance Trade-off: The choice of 'k' is part of the bias-variance trade-off in
machine learning. Smaller 'k' values tend to have low bias and high variance, while
larger 'k' values have higher bias and lower variance. It's essential to strike a balance
that minimizes overall error.
4. Computational Complexity: Smaller 'k' values require less computation because
you're considering fewer neighbors. However, larger 'k' values involve more
neighbors and can be computationally expensive, especially for large datasets.
5. Robustness to Noisy Data: A larger 'k' can make the KNN algorithm more robust
to noisy data because it considers a broader set of neighbors, reducing the impact
of outliers or mislabeled data points.
6. Smoothness of Decision Boundary: Smaller 'k' values tend to produce decision
boundaries that are more jagged and follow the training data closely. Larger 'k'
values create smoother decision boundaries, which can be beneficial when the data
has complex or noisy patterns.
7. Imbalanced Data: The choice of 'k' can also affect the performance of KNN on
imbalanced datasets. Smaller 'k' values can be influenced more by the majority
class, while larger 'k' values can give more weight to the minority class.
8. Cross-Validation: Cross-validation techniques, such as k-fold cross-validation, can
help determine the optimal 'k' value by assessing the model's performance across
different values of 'k' on the validation data.
In practice, selecting the right 'k' value often involves experimentation and validation. You can
try different 'k' values and use techniques like cross-validation to assess their impact on
model performance. The choice of 'k' should be based on the specific characteristics of your
dataset and the trade-off between bias and variance that best suits your problem.
What is a Support Vector Machine (SVM), and what is its primary objective in
classification
1. SVM sees every feature vector as a point in a high-dimensional space. The algorithm
puts all the feature vectors on an imaginary n-dimensional plane (n being the number of
features in the dataset) and draws an imaginary n-dimensional line (a hyperplane) that
separates the examples with positive labels from examples with negative labels.
2. The boundary separating the examples is known as the decision boundary.
 3. The equation of the hyperplane is given by two parameters, a real-valued vector w of the
same dimensionality as our input feature vector x, and a real number b like this:
wx − b = 0

where is
wx w
(1)
x
(1)
+ w
(2)
x
(2)
+ w
(3)
x
(3)
+ w
(4)
x
(4)
. . . . +w
(D)
x
(D)
and D is the number of
dimensions of the feature vector x.

Numerical Question
Given a fixed area A of cardboard, Find the dimension of the cardboard box with maximum
volume.
Determine if any of the following functions is convex.