

No Free Hunch
(http://blog.kaggle.com/)

 (HTTP://BLOG.KAGGLE.COM)  A KAGGLE MASTER EXPLAINS GRADIENT BOOSTING

 (HTTP://BLOG.KAGGLE.COM/2017/01/26/OPEN-DATA-SPOTLIGHT-THE-GLOBAL-TERRORISM-DATABASE/) 
(HTTP://BLOG.KAGGLE.COM/2017/01/12/SANTANDER-PRODUCT-RECOMMENDATION-COMPETITION-2ND-PLACE-
WINNERS-SOLUTION-WRITE-UP-TOM-VAN-DE-WIELE/)

55
A Kaggle Master Explains Gradient Boosting (http://b
Ben Gorman (http://blog.kaggle.com/author/bengorman/) | 01.23.2017
kaggle-

master-
This tutorial was originally posted here (https://gormanalysis.com/gradient-boosting-explained/) on Ben's blog,
GormAnalysis (https://gormanalysis.com/). explains

gradien
If linear regression was a Toyota Camry, then gradient boosting would be a UH-60 Blackhawk
Helicopter. A particular implementation of gradient boosting, XGBoost boostin
(https://github.com/dmlc/xgboost), is consistently used to win machine learning competitions on Kaggle
(https://www.kaggle.com/). Unfortunately many practitioners (including my former self) use it as a black
box. It’s also been butchered to death by a host of drive-by data scientists’ blogs. As such, the purpose
of this article is to lay the groundwork for classical gradient boosting, intuitively and comprehensively.
Motivation
We’ll start with a simple example. We want to predict a person’s age based on whether they play video
games, enjoy gardening, and their preference on wearing hats. Our objective is to minimize squared
error. We have these nine training samples to build our model.

PersonID Age LikesGardening PlaysVideoGames LikesHats

1 13 FALSE TRUE TRUE

2 14 FALSE TRUE FALSE

3 15 FALSE TRUE FALSE

4 25 TRUE TRUE TRUE

5 35 FALSE TRUE TRUE

6 49 TRUE FALSE FALSE

7 68 TRUE TRUE TRUE

8 71 TRUE FALSE FALSE

9 73 TRUE FALSE TRUE

Intuitively, we might expect

– The people who like gardening are probably older
– The people who like video games are probably younger
– LikesHats is probably just random noise

We can do a quick and dirty inspection of the data to check these assumptions:

Feature FALSE TRUE

LikesGardening {13, 14, 15, 35} {25, 49, 68, 71, 73}

PlaysVideoGames {49, 71, 73} {13, 14, 15, 25, 35, 68}

LikesHats {14, 15, 49, 71} {13, 25, 35, 68, 73}

Now let’s model the data with a regression tree. To start, we’ll require that terminal nodes have at least
three samples. With this in mind, the regression tree will make its rst and last split on LikesGardening.
This is nice, but it’s missing valuable information from the feature LikesVideoGames. Let’s try letting
terminal nodes have 2 samples.

Here we pick up some information from PlaysVideoGames but we also pick up information from
LikesHats – a good indication that we’re over tting and our tree is splitting random noise.
Here in lies the drawback to using a single decision/regression tree – it fails to include predictive
power from multiple, overlapping regions of the feature space. Suppose we measure the training
errors from our rst tree.

PersonID Age Tree1 Prediction Tree1 Residual

1 13 19.25 -6.25

2 14 19.25 -5.25

3 15 19.25 -4.25

4 25 57.2 -32.2

5 35 19.25 15.75

6 49 57.2 -8.2

7 68 57.2 10.8

8 71 57.2 13.8

9 73 57.2 15.8

Now we can t a second regression tree to the residuals of the rst tree.
Notice that this tree does not include LikesHats even though our over tted regression tree above did.
The reason is because this regression tree is able to consider LikesHats and PlaysVideoGames with
respect to all the training samples, contrary to our over t regression tree which only considered each
feature inside a small region of the input space, thus allowing random noise to select LikesHats as a
splitting feature.

Now we can improve the predictions from our rst tree by adding the “error-correcting” predictions
from this tree.

Tree1 Tree1 Tree2 Combined Final

PersonID Age Prediction Residual Prediction Prediction Residual

1 13 19.25 -6.25 -3.567 15.68 2.683

2 14 19.25 -5.25 -3.567 15.68 1.683

3 15 19.25 -4.25 -3.567 15.68 0.6833

4 25 57.2 -32.2 -3.567 53.63 28.63

5 35 19.25 15.75 -3.567 15.68 -19.32

6 49 57.2 -8.2 7.133 64.33 15.33

7 68 57.2 10.8 -3.567 53.63 -14.37

8 71 57.2 13.8 7.133 64.33 -6.667

9 73 57.2 15.8 7.133 64.33 -8.667

Tree1 SSE Combined SSE

1994 1765

Gradient Boosting – Draft 1

Inspired by the idea above, we create our rst (naive) formalization of gradient boosting. In pseudocode

1. Fit a model to the data,

2. Fit a model to the residuals,
3. Create a new model,

It’s not hard to see how we can generalize this idea by inserting more models that correct the errors of
the previous model. Speci cally,

where is an initial model t to

Since we initialize the procedure by tting , our task at each step is to nd .

Stop. Notice something. is just a “model”. Nothing in our de nition requires it to be a tree-based
model. This is one of the broader concepts and advantages to gradient boosting. It’s really just a
framework for iteratively improving any weak learner. So in theory, a well coded gradient boosting
module would allow you to “plug in” various classes of weak learners at your disposal. In practice
however, is almost always a tree based learner, so for now it’s ne to interpret as a regression
tree like the one in our example.

Gradient Boosting – Draft 2

Now we’ll tweak our model to conform to most gradient boosting implementations – we’ll initialize the
model with a single prediction value. Since our task (for now) is to minimize squared error, we’ll initialize
with the mean of the training target values.

Then we can de ne each subsequent recursively, just like before

, for

where comes from a class of base learners (e.g. regression trees).

At this point you might be wondering how to select the best value for the model’s hyper-parameter .
In other words, how many times should we iterate the residual-correction procedure until we decide
upon a nal model, ? This is best answered by testing di erent values of via cross-validation
(https://en.wikipedia.org/wiki/Cross-validation_(statistics)).

Gradient Boosting – Draft 3

Up until now we’ve been building a model that minimizes squared error, but what if we wanted to
minimize absolute error? We could alter our base model (regression tree) to minimize absolute error,
but this has a couple drawbacks..

1. Depending on the size of the data this could be very computationally expensive. (Each considered
split would need to search for a median.)
2. It ruins our “plug-in” system. We’d only be able to plug in weak learns that support the objective
function(s) we want to use.

Instead we’re going to do something much niftier. Recall our example problem. To determine , we
start by choosing a minimizer for absolute error. This’ll be . Now we can measure the
residuals, .

PersonID Age F0 Residual0

1 13 35 -22
 PersonID Age F0 Residual0

2 14 35 -21

3 15 35 -20

4 25 35 -10

5 35 35 0

6 49 35 14

7 68 35 33

8 71 35 36

9 73 35 38

Consider the rst and fourth training samples. They have residuals of -22 and -10 respectively. Now
suppose we’re able to make each prediction 1 unit closer to its target. Respective squared error
reductions would be 43 and 19, while respective absolute error reductions would be 1 and 1. So a
regression tree, which by default minimizes squared error, will focus heavily on reducing the residual of
the rst training sample. But if we want to minimize absolute error, moving each prediction one unit
closer to the target produces an equal reduction in the cost function. With this in mind, suppose that
instead of training on the residuals of , we instead train on the gradient of the loss function, with
respect to the prediction values produced by . Essentially, we’ll train on the cost reduction for each
sample if the predicted value were to become one unit closer to the observed value. In the case of
absolute error, will simply consider the sign of every residual (as apposed to squared error which
would consider the magnitude of every residual). After samples in are grouped into leaves, an average
gradient can be calculated and then scaled by some factor, , so that minimizes the loss function for the
samples in each leaf. (Note that in practice, a di erent factor is chosen for each leaf.)

Gradient Descent
Let’s formalize this idea using the concept of gradient descent
(https://en.wikipedia.org/wiki/Gradient_descent). Consider a di erentiable function we want to
minimize. For example,

The goal here is to nd the pair that minimizes . Notice, you can interpret this function as calculating
the squared error for two data points, 15 and 25 given two prediction values, and (but with a
multiplier to make the math work out nicely). Although we can minimize this function directly, gradient
descent will let us minimize more complicated loss functions that we can’t minimize directly.

Initialization Steps:
Number of iteration steps
Starting point
Step size

For iteration to :
1. Calculate the gradient of at the point
2. “Step” in the direction of greatest descent (the negative gradient) with step size . That is,

If is small and is su ciently large, will be the location of ‘s minimum value.

A few ways we can improve this framework:

– Instead of iterating a xed number of times, we can iterate until the next iteration produces
su ciently small improvement.
– Instead of stepping a xed magnitude for each step, we can use something like line search
(https://en.wikipedia.org/wiki/Line_search) smartly choose step sizes.

If you’re struggling with this part, just google gradient descent (https://www.google.com/webhp?
sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=gradient%20descent). It’s been explained many times in
many ways.

Leveraging Gradient Descent

Now we can use gradient descent for our gradient boosting model. The objective function we want to
minimize is . Our starting point is . For iteration , we compute the gradient of with respect to . Then we
t a weak learner to the gradient components. In the case of a regression tree, leaf nodes produce an
average gradient among samples with similar features. For each leaf, we step in the direction of the
average gradient (using line search to determine the step magnitude). The result is . Then we can repeat
the process until we have .

Take a second to stand in awe of what we just did. We modi ed our gradient boosting algorithm so that
it works with any di erentiable loss function. (This is the part that gets butchered by a lot of gradient
boosting explanations.) Let’s clean up the ideas above and reformulate our gradient boosting model
once again.

Initialize the model with a constant value:

For m = 1 to M:
Compute pseudo residuals,
Fit base learner, to pseudo residuals
Compute step magnitude multiplier . (In the case of tree models, compute a di erent for every leaf.)
Update

In case you want to check your understanding so far, our current gradient boosting applied to our
sample problem for both squared error and absolute error objectives yields the following results.
Squared Error
Age F0 PseudoResidual0 h0 gamma0 F1 PseudoResidual1 h1 gamma1 F2

13 40.33 -27.33 -21.08 1 19.25 -6.25 -3.567 1 15.6

14 40.33 -26.33 -21.08 1 19.25 -5.25 -3.567 1 15.6

15 40.33 -25.33 -21.08 1 19.25 -4.25 -3.567 1 15.6

25 40.33 -15.33 16.87 1 57.2 -32.2 -3.567 1 53.6

35 40.33 -5.333 -21.08 1 19.25 15.75 -3.567 1 15.6

49 40.33 8.667 16.87 1 57.2 -8.2 7.133 1 64.3

68 40.33 27.67 16.87 1 57.2 10.8 -3.567 1 53.6

71 40.33 30.67 16.87 1 57.2 13.8 7.133 1 64.3

73 40.33 32.67 16.87 1 57.2 15.8 7.133 1 64.3

Absolute Error
Age F0 PseudoResidual0 h0 gamma0 F1 PseudoResidual1 h1 gamma1 F2

13 35 -1 -1 20.5 14.5 -1 -0.3333 0.75 14.25

14 35 -1 -1 20.5 14.5 -1 -0.3333 0.75 14.25

15 35 -1 -1 20.5 14.5 1 -0.3333 0.75 14.25

25 35 -1 0.6 55 68 -1 -0.3333 0.75 67.75

35 35 -1 -1 20.5 14.5 1 -0.3333 0.75 14.25

49 35 1 0.6 55 68 -1 0.3333 9 71
 Age F0 PseudoResidual0 h0 gamma0 F1 PseudoResidual1 h1 gamma1 F2

68 35 1 0.6 55 68 -1 -0.3333 0.75 67.75

71 35 1 0.6 55 68 1 0.3333 9 71

73 35 1 0.6 55 68 1 0.3333 9 71

Gradient Boosting – Draft 4

Here we introduce something called shrinkage
(https://en.wikipedia.org/wiki/Gradient_boosting#Shrinkage).
(https://en.wikipedia.org/wiki/The_Hamptons_(Seinfeld)) The concept is fairly simple. For each gradient
step, the step magnitude is multiplied by a factor between 0 and 1 called a learning rate. In other words,
each gradient step is shrunken by some factor. The current Wikipedia excerpt
on shrinkage doesn’t mention why shrinkage is e ective – it just says that
shrinkage appears to be empirically e ective. My personal take is that it
causes sample-predictions to slowly converge toward observed values. As this
slow convergence occurs, samples that get closer to their target end up being
grouped together into larger and larger leaves (due to xed tree size
parameters), resulting in a natural regularization e ect.

Gradient Boosting – Draft 5

Last up – row sampling and column sampling. Most gradient boosting algorithms provide the ability to
sample the data rows and columns before each boosting iteration. This technique is usually e ective
because it results in more di erent tree splits, which means more overall information for the model. To
get a better intuition for why this is true, check out my post on Random Forest
(https://gormanalysis.com/random-forest-from-top-to-bottom/), which employs the same random
sampling technique. Alas we have our nal gradient boosting framework.

Gradient Boosting in Practice

Gradient boosting in incredibly e ective in practice. Perhaps the most popular implementation,
XGBoost (https://github.com/dmlc/xgboost), is used in a number of winning Kaggle solutions. XGBoost
employs a number of tricks that make it faster and more accurate than traditional gradient boosting
(particularly 2nd-order gradient descent) so I’ll encourage you to try it out and read Tianqi Chen’s paper
about the algorithm (http://www.kdd.org/kdd2016/papers/ les/rfp0697-chenAemb.pdf). With that said,
a new competitor, LightGBM (https://github.com/Microsoft/LightGBM) from Microsoft, is gaining
signi cant traction.

What else can it do? Although I presented gradient boosting as a regression model, it’s also very
e ective as a classi cation and ranking model. As long as you have a di erentiable loss function for the
algorithm to minimize, you’re good to go. The logistic function
(https://en.wikipedia.org/wiki/Logistic_function) is typically used for binary classi cation and the softmax
function (https://en.wikipedia.org/wiki/Softmax_function) is often used for multi-class classi cation.

I leave you with a quote from my fellow Kaggler Mike Kim (https://www.kaggle.com/mikeskim).

My only goal is to gradient boost over myself of yesterday. And to

repeat this everyday with an unconquerable spirit.
Bio
(https://www.kaggle.com/ben519)I’m Ben Gorman
(https://www.kaggle.com/ben519) – math nerd and data science enthusiast based
in the New Orleans area. I spent roughly ve years as the Senior Data Analyst for
Strategic Comp
(http://www.greatamericaninsurancegroup.com/Insurance/Strategic-
Comp/Pages/default.aspx) before starting GormAnalysis
(https://gormanalysis.com/). I love talking about data science, so never hesitate to shoot me an email if
you have questions: [email protected] (mailto:[email protected]). As of
September 2016, I’m a Kaggle Master ranked in the top 1% of competitors world-wide.

Share this:

 (http://blog.kaggle.com/2017/01/23/a-kaggle-master-explains-gradient-boosting/?share=twitter&nb=1)

 (http://blog.kaggle.com/2017/01/23/a-kaggle-master-explains-gradient-boosting/?share=facebook&nb=1)
159

 (http://blog.kaggle.com/2017/01/23/a-kaggle-master-explains-gradient-boosting/?share=google-plus-1&nb=1)

TUTORIAL (HTTP://BLOG.KAGGLE.COM/TAG/TUTORIAL/)

XGBOOST (HTTP://BLOG.KAGGLE.COM/TAG/XGBOOST/)

33 Comments No Free Hunch 

1 Login

 Recommend 28 t Tweet f Share Sort by Best

Join the discussion…

LOG IN WITH OR SIGN UP WITH DISQUS ?

Name

ben • a year ago

Thank you for the great article! I am hung up on the "Final Residual," however. Why did you
subtract the combined prediction from the actual age, when you subtracted actual age from
Tree1 prediction and Tree2 prediction? I don't understand why you are reversing the order of
the arithmetic here.
8△ ▽ • Reply • Share ›

Yohan Obd • 2 years ago

Thank you very much for this article !
I have one question however If after the first model the next ones are trained on the error of
I have one question however. If after the first model, the next ones are trained on the error of
the one before, that means that to "feed" those models you need to have the true value in
order to compute the error to use for the model. However once you start working on data
outside of your train set, you don't have the true value anymore, it is then impossible to
compute the error. Should the model 2 to m be rather fed with the prediction of the previous
one instead of its error ?
1△ ▽ • Reply • Share ›

ldmtwo • 2 years ago

Looking at the table below tree 2, how are the two trees combined? Take row 1 for instance.
We have regression trees that produce 19.25 and -3.567? Is tree 2 trying to predict the error
or what? I'm lost. Thanks.
PersonID, Age, Tree1 Prediction, Tree1 Residual, Tree2 Prediction, Combined Prediction,
Final Residual
1, 13, 19.25, -6.25, -3.567, 15.68, 2.683
1△ ▽ • Reply • Share ›

ldmtwo > ldmtwo • a year ago

IDK It wasn't clear before, but to answer my question: each residual R in the earlier
steps is made by 1) get the prediction for a base model, 2) with a 2nd model, predict
the individual errors (residuals) that the 1st model will have, and 3) adjust base
predictions with the residual. The first model would be fit with inputs X and labels Y.
The 2nd model would be fit with inputs X and labels R.
△ ▽ • Reply • Share ›

Sijo Vm • 2 years ago

For personid 9 and age = 73, playing video games was false. However, in your overfit tree
diagram, it falls in true. Please correct it and edit the document since it is very confusing.
Thank you .
1△ ▽ • Reply • Share ›

Ben Gorman > Sijo Vm • 2 years ago

Good catch. Fixing this now!
EDIT: Fixed.
△ ▽ • Reply • Share ›

Rok Resnik • 10 months ago

Great article!

There's just one minor nitpick in this part, where thing get a bit confusing - at least for me:

> "They have F_0 residuals of -22 and -10 respectively. ... Respective squared error
reductions would be 43 and 19,..."

If you define squared error without 1/2 in front of the sum, this is indeed true (i.e. if you
change prediction 35 (for age 13) to 34, you get (35-13)^2 - (34-13)^2 = 43), but that means
that the residual (y - F(x)) would have to be 2*(y - F(x)), if you want it to be the gradient of the
loss function... and that means that the aforementioned residual (-22) should be -44, right?
I'm just trying to visualize what this all means in terms of gradients in function space and all
that, and it's perfectly possible that I blew it...
△ ▽ • Reply • Share ›

ananiask8 • a year ago

Can you elaborate on this one:

"My personal take is that it causes sample-predictions to slowly converge toward observed
values. As this slow convergence occurs, samples that get closer to their target end up
being grouped together into larger and larger leaves (due to fixed tree size parameters),
resulting in a natural regularization effect."

I'm not following unto which values exactly they converge... what are the observed values?
Why does this grouping effect help regularization? Is it that samples that are intrinsically
better explained via some sort of feature, eventually end up in a leaf described by it? Or
something different?
△ ▽ • Reply • Share ›

ST22 Bangalore • a year ago

Thank you Ben! This is awesome.
△ ▽ • Reply • Share ›

Octavian Tuchila • a year ago

Great article!

I just don't understand one thing:

How do you get numerical predictions from Tree 1, like `19.25` or `57.2`?

Tree1, because it's a decision tree, should be a binary classifier, right?

So it should return make predictions such as `True` or `False`, correct?
△ ▽ • Reply • Share ›

zekeriya > Octavian Tuchila • a year ago

İt is decision tree for regression problem. And you should calculate average age for
each terminal node.
mean(13,14,15,35) = 19.25
mean(25,49,68,71,73) = 57.2
△ ▽ • Reply • Share ›

Sean • a year ago

Excellent article,

Any chance you could (here or in another article) give us a sense of what the prediction step
for new examples looks like? Is it an average or weighted average of the stump models? Or
would we only use model 2? Something entirely different? thanks --Sean
△ ▽ • Reply • Share ›

Chetan Bhat • a year ago

Hey Ben, great article! I've used this write-up, along with other reference material in tandem
to "boost" my own understanding of Gradient Boosting.

In the section "leveraging gradient descent" where you present a summary table to check
understanding (for squared error), shouldn't the "PseudoResidual0" column contain
gradients of residuals rather than the residuals themselves? Right now this column contains
residuals only i.e. difference between age and F0. Or am I missing something?
△ ▽ • Reply • Share ›

Ben Gorman > Chetan Bhat • a year ago

Thanks! In the case of squared error, the gradients of residuals = residuals.
(derivative of 0.5(yhat - y)^2 w.r.t. yhat is yhat - y)
△ ▽ • Reply • Share ›

Chetan Bhat > Ben Gorman • a year ago

Ah! of course. Thank you :)
△ ▽ • Reply • Share ›

Shantanu Gangal • a year ago

@Ben Gorman Thanks for writing this fantastic article.
I don't follow how the absolute error loss function is differentiable though. Maybe I am
missing a simple thing -- but isn't the function non-differentiable at 0?
△ ▽ • Reply • Share ›

Ben Gorman > Shantanu Gangal • a year ago

Thanks Shantanu. We could define the derivative = 0 at x = 0 to get around this
scenario (which is unlikely).
△ ▽ • Reply • Share ›

Kumar Rajendran • a year ago

>What else can it do? Although I presented gradient boosting as a regression model, it’s
also very effective as a classification and ranking model. As long as you have a differentiable
loss function for the algorithm to minimize, you’re good to go. The logistic function is
typically used for binary classification

It will be cool if we you can say what the logistic function is used for. Is it to convert the
prediction to a probability?
△ ▽ • Reply • Share ›

ldmtwo • a year ago

When updating predictions using a learning rate, would that be done as
1) p(i+1) = p(i) + r(i)*lr, which is a simple reduction,
or 2) p(i+1) = p(i) * (1-lr) + (p(i)+r(i))*lr, which is exponential moving average?

lr=learning rate, r=residual, p=predictions

△ ▽ • Reply • Share ›

b k • a year ago
In Draft '3', I question in following sentence "They have F_0 residuals of -22 and -10
respectively. Now suppose we’re able to make each prediction 1 unit closer to its target.
Respective squared error reductions would be 43 and 19, while respective absolute error
reductions would be 1 and 1". Can anyone help me in understanding on values 43,19 are
arrived?
△ ▽ • Reply • Share ›

Lee XA > b k • a year ago

make each prediction closer to target , F_0 residuals will be -22 -> -21, -10 -> -9 .
MSE will be 22**2 - 21**2 = 43 and 10**2 - 9**2 = 81
△ ▽ • Reply • Share ›

António Góis • 2 years ago

First of all, thanks for this awesome explanation. I found it very easy to read, and I think I
finally grasped the concept of gradient boosting. All the other explanations I've seen just hit
you with the formulas without giving an intuition.
I just didn't get your comparison with random forest in "Draft 5". The point in using only
some samples per tree and only some features per node, in random forests, is that you'll
have a lot of trees voting for the final decision and you want diversity among those trees
(correct me if I'm wrong here).
From what I understood of gradient boosting, you have a tree with only one node/feature
voting for the direction of the gradient in each step, and you don't reuse the same feature in
a future step. If you don't have several trees voting in parallel to decide one gradient
direction, whats the point of limiting the features+samples that a tree has access to?

Probably I got something wrong here, but I don't know what...

Thanks a lot for the article and for your attention,
António
△ ▽ • Reply • Share ›

disqus_pu6Sx8tZjX • 2 years ago

Excellent article, light on jargon and great explanations. I wish I came upon articles like this
one when I was learning about gradient boosting, I would have learned it 10x faster.
△ ▽ • Reply • Share ›

suraj • 2 years ago

how did you get the values for gamma0 and gamma1 in the absolute error table?
△ ▽ • Reply • Share ›

Agzamov Rustam • 2 years ago

Is it correct that F_2(x) = F_1(x) + F_2(x) (step 3 in draft 1)? May be it should be F_2(x) =
F_1(x) + h_1(x)?
△ ▽ • Reply • Share ›

Ben Gorman > Agzamov Rustam • 2 years ago

Whoops, that's a mistake. Thanks for catching. Will try to get it fixed ASAP.
EDIT: Fixed.
△ ▽ • Reply • Share ›
 Vladislavs Dovgalecs • 2 years ago
Yes, UH-60 Blackhawk Helicopter is *way* much faster than Toyota Camry but it is also
super expensive and very costly in maintenance :)
△ ▽ • Reply • Share ›

Carlos Aguayo > Vladislavs Dovgalecs • 2 years ago

"Don't miss the forest for the trees" ;)
△ ▽ • Reply • Share ›

huts • 2 years ago

Thanks for the awesome article Ben! I only had a general view of how boosting worked, but
this makes it very clear for gradient boosting, though I did struggle to understand the
paragraph "Gradient Boosting - draft 3". I think the fact that the gradient descent is only
used to get residuals and NOT to optimize parameters could be stressed even more (I was
misleaded by searching for those parameters being optimized with gradient descent, but I
was being confused by the "classic" use of gradient descent in other models, such as in
neural nets).

There is, however, one point that I did not get when you are giving the estimating process
for the squared error and the absolute error loss functions. For the squared error, h0 and h1
forecasts are given by the mean of the samples contained in each leaf node. However, I did
not get how you got the h0 and h1 forecasts for the absolute error function. Besides, the
way gamma0 and gamma1 are estimated for the absolute error loss function is still obscure
(though I understand from the line search article from wikipedia that we are looking for the
step that minimizes the loss function given the gradient in one point).

That would be great if you could explicit those last details so we could understand the
whole picture of your article ;-)

Thanks again, and continue the great work!

△ ▽ • Reply • Share ›

Walter > huts • 2 years ago

Huts,

The way that h0 and h1 are computed in the absolute error loss function example is
again by the mean of the samples contained in each leaf node. The problem is that
the first decision tree has a drawing mistake. The left leaf (LikesGardening==F)
should contain the values {-1,-1,-1,-1} corresponding to PersonID {1,2,3,5}. The
second leaf (LikesGardening==T) should contain the values {-1,1,1,1,1}
corresponding to PersonID {4,6,7,8,9}. Now, buy computing the mean in leaf1: (-1-1-
1-1)/4 = -1 and for leaf2: (-1+1+1+1+1)/5=0 6 This values coincide with the values

 
(https://www.facebook.com/kaggle) (https://twitter.com/kaggle)