Open in app Get started

Published in Towards Data Science

SAGAR SHARMA Follow

Sep 6, 2017 · 5 min read · Listen

Save

Activation Functions in Neural Networks

Sigmoid, tanh, Softmax, ReLU, Leaky ReLU EXPLAINED !!!

What is Activation Function?

It’s just a thing function that you use to get the output of node. It is also known as Transfer
Function.

Why we use Activation functions with Neural Networks?

It is used to determine the output of neural network like yes or no. It maps the resulting
 Open in app Get started

1. Linear Activation Function

2. Non-linear Activation Functions

FYI: The Cheat sheet is given below.

Linear or Identity Activation Function

As you can see the function is a line or linear. Therefore, the output of the functions will
not be confined between any range.

Fig: Linear Activation Function

Equation : f(x) = x
 Open in app Get started

Non-linear Activation Function

The Nonlinear Activation Functions are the most used activation functions.
Nonlinearity helps to makes the graph look something like this

Fig: Non-linear Activation Function

It makes it easy for the model to generalize or adapt with variety of data and to
differentiate between the output.

The main terminologies needed to understand for nonlinear functions are:

Derivative or Differential: Change in y-axis w.r.t. change in x-axis.It is also known as

slope.

Monotonic function: A function which is either entirely non-increasing or non-

decreasing.
 Open in app Get started

Fig: Sigmoid Function

The main reason why we use sigmoid function is because it exists between (0 to 1).
Therefore, it is especially used for models where we have to predict the probability as
an output.Since probability of anything exists only between the range of 0 and 1,
sigmoid is the right choice.

The function is differentiable.That means, we can find the slope of the sigmoid curve
at any two points.

The function is monotonic but function’s derivative is not.

The logistic sigmoid function can cause a neural network to get stuck at the training
time.

The softmax function is a more generalized logistic activation function which is used
for multiclass classification.

2. Tanh or hyperbolic tangent Activation Function

 Open in app Get started

Fig: tanh v/s Logistic Sigmoid

The advantage is that the negative inputs will be mapped strongly negative and the
zero inputs will be mapped near zero in the tanh graph.

The function is differentiable.

The function is monotonic while its derivative is not monotonic.

The tanh function is mainly used classification between two classes.

Both tanh and logistic sigmoid activation functions are used in feed-forward nets.

3. ReLU (Rectified Linear Unit) Activation Function

The ReLU is the most used activation function in the world right now.Since, it is used in
almost all the convolutional neural networks or deep learning.
 Open in app Get started

Fig: ReLU v/s Logistic Sigmoid

As you can see, the ReLU is half rectified (from bottom). f(z) is zero when z is less than
zero and f(z) is equal to z when z is above or equal to zero.

Range: [ 0 to infinity)

The function and its derivative both are monotonic.

But the issue is that all the negative values become zero immediately which decreases
the ability of the model to fit or train from the data properly. That means any negative
input given to the ReLU activation function turns the value into zero immediately in the
graph, which in turns affects the resulting graph by not mapping the negative values
appropriately.

4. Leaky ReLU
It is an attempt to solve the dying ReLU problem
 Open in app Get started

Fig : ReLU v/s Leaky ReLU

Can you see the Leak?

The leak helps to increase the range of the ReLU function. Usually, the value of a is 0.01
or so.

When a is not 0.01 then it is called Randomized ReLU.

Therefore the range of the Leaky ReLU is (-infinity to infinity).

Both Leaky and Randomized ReLU functions are monotonic in nature. Also, their
derivatives also monotonic in nature.

Why derivative/differentiation is used ?

When updating the curve, to know in which direction

and how much to change or update the curve
depending upon the slope.That is why we use
differentiation in almost every part of Machine
Learning and Deep Learning.
 Open in app Get started

Fig: Activation Function Cheetsheet

 Open in app Get started

Fig: Derivative of Activation Functions

 Open in app Get started

If you liked it

So, follow me on Medium, LinkedIn to see similar posts.

Any comments or if you have any questions, write them in the comment.

Clap it! Share it! Follow Me!

Previous stories you will love:

What is Linear Regression and How does it work? - theffork

It is a method used for predicting future values by finding a linear
pattern in the previously given data. The linear…
theffork.com

What the Hell is “Tensor” in “TensorFlow”?

I didn’t know it…
hackernoon.com
 Open in app Get started
Epoch vs Batch Size vs Iterations
Know your code…
towardsdatascience.com

12.4K 49
Monte Carlo Tree Search
MCTS For Every Data Science Enthusiast
towardsdatascience.com

Sign up for The Variable

By Towards Data Science

Every Thursday, the Variable delivers the very best of Towards Data Science: from hands-on tutorials
and cutting-edge research to original features you don't want to miss. Take a look.

Get this newsletter

On CPU with Inception-v3(In seconds)

towardsdatascience.com

How to Send Emails using Python

Design Professional Mails using Flask!
medium.com