Logistic Regression with

Newton Method
- Arsh Sharan (2K20/EE/56)
- Aryan Mittal (2K20/EE/62)
- Asif Akhtar (2K20/EE/68)
 Logistic Regression

Logistic regression is a statistical analysis method used to predict a data value based on prior
observations of a data set.

A logistic regression model predicts a dependent data variable by analyzing the relationship between
one or more existing independent variables. For example, a logistic regression could be used to predict
whether a political candidate will win or lose an election or whether a high school student will be
admitted to a particular college.

Logistic regression is one of the most commonly used machine learning algorithms for binary
classification problems, which are problems with two class values, including predictions such as “this or
that,” “yes or no” and “A or B.”
 Uses of Logistic Regression

● Logistic regression has become particularly popular in online advertising, enabling marketers to
predict the likelihood of specific website users who will click on a particular advertisement as a yes or
no percentage.

● Healthcare, to identify risk factors for diseases and plan preventive measures.

● Insurance, to predict the chance that a policy holder will die before the term of the policy expires
based on certain criteria, such as gender, age and physical examination.

● Banking, to predict the chances that a loan applicant will default on a loan or not, based on annual
income, past defaults and past debts.
 Newton - Raphson Method
The Newton-Raphson method (also known as Newton's method) is a way to quickly find a good approximation for the
root of a real-valued function f(x) = 0. It uses the idea that a continuous and differentiable function can be
approximated by a straight line tangent to it.

Suppose you need to find the root of a continuous, differentiable function f(x) and you know the root you are looking
for is near the point x = x0. Then Newton's method tells us that a better approximation for the root is :

This process may be repeated as many times as necessary to get the desired accuracy. In general, for any x value xn ,the
next value is given by

1
Geometrical Representation
 Problem Statement
The dataset is publically available on the Kaggle website, and it is from an ongoing cardiovascular study on residents of
the town of Framingham, Massachusetts. The classification goal is to predict whether the patient has 10-year risk of
future coronary heart disease (CHD). The dataset provides the patients’ information. It includes over 4,000 records and
15 attributes. Variables Each attribute is a potential risk factor. There are both demographic, behavioral and medical risk
factors.

We want to predict whether a patient will have 10-year risk of future coronary heart disease (CHD) based on his current
medical reports.
 Hypothesis Function
Our problem requires a binary classifier and we want to solve a linear combination of features and weights. We also
need to transform this linear combination by wrapping it in a function that is smooth, and has its range defined as [0, 1]
(because we are trying to map our linear combination to a binary output of 0 or 1!).

The logistic function, or sigmoid function, accomplishes all of these things:

h(x) = 1/(1 + e^-z), z = w1(x) + w2

=x

(x

0
 Cost Function
The cost function represents optimization objective i.e. we create a cost function and minimize it so that we can
develop an accurate model with minimum error.

For logistic regression, the Cost function is defined as:

=x

(x

0
 Gradient Descent
Gradient descent is an iterative optimization algorithm, which finds the minimum of a differentiable function. In this
process, we try different values and update them to reach the optimal ones, minimizing the output.

As already explained, we’re using the sigmoid function as the hypothesis function in logistic regression.
Assume we have a total of ‘n’ features. In this case, we have parameters for the vector. To minimize our cost function, we
need to run the gradient descent on each parameter :

Furthermore, we need to update each parameter simultaneously for each iteration. In other words, we need to loop
through the parameters in vector .
To complete the algorithm :
Geometrical Representation
 Optimisation Technique - Newton Method
Our ultimate aim is to minimize the cost function. One way to do this is by using the Gradient Descent technique which
we have discussed before , but it takes several iterations to get desired results. We want something which can minimize
the cost function in just a few iterations. We can achieve this by using Newton-Raphson Method.
Newton-Raphson method is generally used for finding root of an equation but it can be also used to minimize a function
if we know the second derivative of the function (which we can find out for our problem).
We minimize the cost function by finding its gradient (first derivative) and hessian (second derivative) and putting the
values in the formula given below :

Where , thata → parameters, H → Hessian and del(l) → Gradient

● Dividing the dataset → We divided the dataset into 2 parts randomly for training and
testing purposes.

● Implementing Newton Method and Gradient Descent algorithms → We wrote code

● Comparing Results → We found the accuracy achieved from both the method and
also compared the the number of iteration used in both the methods.
 Comparing the Algorithms
● In terms of accuracy, there was not much of a difference . We got an accuracy of
84.76 % from Newton Method and an accuracy of 84.61 % from the Gradient
Descent Method.
● In terms of number of iteration required, we see a huge difference.

Newton Method Gradient Descent