8) Linear Regression II Lesson

Regularization in Machine Learning

3 min to complete · By Gilad Gressel

You have just learned how about the regularization term for linear regression. While regularization is most commonly used to refer to the mathematical term that you add to our cost function in order to penalize overfitting, it can also just refer to any strategy that combats overfitting. This can be confusing, because if you google "machine learning regularization" you will probably find a few pages like this one.

What is Regularization

Regularization (mathematics) which is a great page but entirely mathematical. It leads you to believe that regularization is only used to refer to adding a penalty in linear regression (or different mathematical formulations). I actually wish this was true, and I think it used to be true that regularization was used to refer to this strict mathematical term. These days in machine learning, though, the word "regularization" refers to anything that is done to combat overfitting.

You can read this in the popular deeplearningbook where they define regularization as "Regularization is any modification you make to a learning algorithm that is intended to reduce its generalization error but not its training error."

So there you have it -- regularization in machine learning today means anything that you do (within an algorithm) to try and stop overfitting.

Summary: Regularization in Machine Learning

  • Regularization is added to functions to penalize overfitting
  • There are different ways of using the word regularization
  • In machine learning regularization means anything that stops overfitting