Regression
Regression
Module1_2
Regression
[email protected]
Topics to be covered
𝑓(𝑥)=𝑏𝑜+𝑏1𝑥
Our goal is to calculate the optimal values of the predicted weights 𝑏₀ and 𝑏₁ that minimize
the sum of squared residuals (SSR) and determine the estimated regression function.
Linear Re(lationship)
you assume a linear relationship between 𝑦 and 𝐱:
𝑝
𝑓 𝑥 = 𝛽0 + 𝛽𝑗 𝑥𝑗
𝑗=1
i.e., E 𝑦 𝑥 is Linear
Where,
It rains only if it’s a little humid and does not rain if it’s windy,
hot or freezing.
We also quantify the model’s performance using metrics like Accuracy, Mean
Squared Error(MSE), F1-Score, etc and try to improve these metrics.
A supervised Machine Learning model aims to train itself on the input variables(X) in
such a way that the predicted values(Y) are as close to the actual values as possible.
This difference between the actual values and predicted values is the error and it is
used to evaluate the model. The error for any supervised Machine Learning algorithm
comprises of 3 parts:
Bias error
Variance error
The noise (irreducible that we cannot eliminate)
What is Bias?
In the simplest terms, Bias is the difference between the Predicted Value and the
Expected Value. To explain further, the model makes certain assumptions when it
trains on the data provided. When it is introduced to the testing/validation data,
these assumptions may not always be correct.
Mathematically, let the input variables be X and a target variable Y. We map the
relationship between the two using a function f.
2. Shrinkage/Ridge Regression
3. Derived Inputs
Evaluating an Estimator: Bias and
Variance
Tuning Model Complexity:
Bias/Variance Dilemma
The Expected Value
A Didactic Example
Bias Variance Trade-off
Under fitting/Over fitting
Here, you have a given set of input-output (or 𝑥-𝑦) pairs, represented by green circles.
These are your observations. Remember that 𝑦 can only be 0 or 1.
For example, the leftmost green circle has the input 𝑥 = 0 and the actual output 𝑦 = 0. The
rightmost observation has 𝑥 = 9 and 𝑦 = 1.
Single-Variate Logistic Regression
• Logistic regression finds the weights 𝑏₀ and 𝑏₁ that
correspond to the maximum log-likelihood function (LLF).
• These weights define the
𝑓(𝑥) = 𝑏₀ + 𝑏₁𝑥,
which is the dashed black line.
They also define the predicted probability
𝑝(𝑥)=1/(1+𝑒𝑥𝑝(−𝑓(𝑥)))p(x)=1/(1+exp(−𝑓(𝑥))),
shown here as the full black line.
In this case, the threshold 𝑝(𝑥) = 0.5 and 𝑓(𝑥) = 0 corresponds
to the value of 𝑥 slightly higher than 3. This value is the limit
between the inputs with the predicted outputs of 0 and 1.