Chapter 8

Nonlinear Regression
Functions
Nonlinear Regression Functions
(SW Chapter 8)

 Everything so far has been linear in the X’s

 But the linear approximation is not always a good one
 The multiple regression framework can be extended to handle
regression functions that are nonlinear in one or more X.

Outline
1. Nonlinear regression functions – general comments
2. Nonlinear functions of one variable
3. Nonlinear functions of two variables: interactions

2
The TestScore – STR relation looks
linear (maybe)…

3
But the TestScore – Income relation
looks nonlinear...

4
Nonlinear Regression Population Regression
Functions – General Ideas (SW Section 8.1)
If a relation between Y and X is nonlinear:

 The effect on Y of a change in X depends on the value of X –

that is, the marginal effect of X is not constant
 A linear regression is mis-specified – the functional form is
wrong
 The estimator of the effect on Y of X is biased – it needn’t
even be right on average.
 The solution to this is to estimate a regression function that is
nonlinear in X
5
The general nonlinear population
regression function
Yi = f(X1i, X2i,…, Xki) + ui, i = 1,…, n

Assumptions
1. E(ui| X1i,X2i,…,Xki) = 0 (same); implies that f is the
conditional expectation of Y given the X’s.
2. (X1i,…,Xki,Yi) are i.i.d. (same).
3. Big outliers are rare (same idea; the precise mathematical
condition depends on the specific f).
4. No perfect multicollinearity (same idea; the precise statement
depends on the specific f).

6
Nonlinear Functions of a Single
Independent Variable (SW Section 8.2)
We’ll look at two complementary approaches:

1. Polynomials in X
The population regression function is approximated by a
quadratic, cubic, or higher-degree polynomial

2. Logarithmic transformations
 Y and/or X is transformed by taking its logarithm
 this gives a “percentages” interpretation that makes sense
in many applications
7
1. Polynomials in X
Approximate the population regression function by a polynomial:

Yi = 0 + 1Xi + 2 X i2 +…+ r X ir + ui

 This is just the linear multiple regression model – except that

the regressors are powers of X!
 Estimation, hypothesis testing, etc. proceeds as in the
multiple regression model using OLS
 The coefficients are difficult to interpret, but the regression
function itself is interpretable
8
Example: the TestScore – Income
relation
Incomei = average district income in the ith district
(thousands of dollars per capita)

Quadratic specification:

TestScorei = 0 + 1Incomei + 2(Incomei)2 + ui

Cubic specification:

TestScorei = 0 + 1Incomei + 2(Incomei)2

+ 3(Incomei)3 + ui
9
Estimation of the quadratic
specification in STATA
generate avginc2 = avginc*avginc; Create a new regressor
reg testscr avginc avginc2, r;

Regression with robust standard errors Number of obs = 420

F( 2, 417) = 428.52
Prob > F = 0.0000
R-squared = 0.5562
Root MSE = 12.724

------------------------------------------------------------------------------
| Robust
testscr | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
avginc | 3.850995 .2680941 14.36 0.000 3.32401 4.377979
avginc2 | -.0423085 .0047803 -8.85 0.000 -.051705 -.0329119
_cons | 607.3017 2.901754 209.29 0.000 601.5978 613.0056
------------------------------------------------------------------------------

Test the null hypothesis of linearity against the alternative that

the regression function is a quadratic….
10
Interpreting the estimated
regression function:
(a) Plot the predicted values

TestScore = 607.3 + 3.85Incomei – 0.0423(Incomei)2
(2.9) (0.27) (0.0048)

11
Interpreting the estimated
regression function, ctd:
(b) Compute “effects” for different values of X


TestScore = 607.3 + 3.85Incomei – 0.0423(Incomei)2
(2.9) (0.27) (0.0048)

Predicted change in TestScore for a change in income from

$5,000 per capita to $6,000 per capita:


TestScore = 607.3 + 3.85 6 – 0.0423 62

– (607.3 + 3.85 5 – 0.0423 52)

= 3.4 12

TestScore = 607.3 + 3.85Income i – 0.0423(Income i) 2

Predicted “effects” for different values of X:

Change in Income ($1000 per capita) 

TestScore
from 5 to 6 3.4
from 25 to 26 1.7
from 45 to 46 0.0

The “effect” of a change in income is greater at low than high

income levels (perhaps, a declining marginal benefit of an
increase in school budgets?)

13
Estimation of a cubic specification
in STATA
gen avginc3 = avginc*avginc2; Create the cubic regressor
reg testscr avginc avginc2 avginc3, r;

Regression with robust standard errors Number of obs = 420

F( 3, 416) = 270.18
Prob > F = 0.0000
R-squared = 0.5584
Root MSE = 12.707

------------------------------------------------------------------------------
| Robust

testscr | Coef. Std. Err. t P>|t| [95% Conf. Interval]

-------------+----------------------------------------------------------------
avginc | 5.018677 .7073505 7.10 0.000 3.628251 6.409104
avginc2 | -.0958052 .0289537 -3.31 0.001 -.1527191 -.0388913
avginc3 | .0006855 .0003471 1.98 0.049 3.27e-06 .0013677
_cons | 600.079 5.102062 117.61 0.000 590.0499 610.108
------------------------------------------------------------------------------

14
Testing the null hypothesis of linearity, against the alternative
that the population regression is quadratic and/or cubic, that is, it
is a polynomial of degree up to 3:

H0: pop’n coefficients on Income2 and Income3 = 0

H1: at least one of these coefficients is nonzero.

test avginc2 avginc3; Execute the test command after running the regression

( 1) avginc2 = 0.0
( 2) avginc3 = 0.0

F( 2, 416) = 37.69
Prob > F = 0.0000

The hypothesis that the population regression is linear is rejected

at the 1% significance level against the alternative that it is a
polynomial of degree up to 3.
15
Summary: polynomial regression
functions
Yi =  0 + 1Xi + 2 X i2 +…+ r X ir + ui
 Estimation: by OLS after defining new regressors
 Coefficients have complicated interpretations
 To interpret the estimated regression function:
 plot predicted values as a function of x
 compute predicted Y/X at different values of x
 Hypotheses concerning degree r can be tested by t- and F-
tests on the appropriate (blocks of) variable(s).
 Choice of degree r
 plot the data; t- and F-tests, check sensitivity of estimated
effects; judgment.
16
The three log regression
specifications:

Case Population regression function

I. linear-log Yi = 0 + 1ln(Xi) + ui
II. log-linear ln(Yi) = 0 + 1Xi + ui
III. log-log ln(Yi) = 0 + 1ln(Xi) + ui

 The interpretation of the slope coefficient differs in each case.

 The interpretation is found by applying the general “before
and after” rule: “figure out the change in Y for a given change
in X.”
17
I. Linear-log population regression
function
Y = 0 + 1ln(X) (b)

Now change X: Y + Y = 0 + 1ln(X + X) (a)

Subtract (a) – (b): Y = 1[ln(X + X) – ln(X)]

X
now ln(X + X) – ln(X)  ,
X
X
so Y  1
X
Y
or 1  (small X)
X / X 18
Linear-log case, continued
Yi = 0 + 1ln(Xi) + ui

for small X,

Y
1 
X / X

X
Now 100 = percentage change in X, so a 1% increase in X
X
(multiplying X by 1.01) is associated with a .011 change in Y.
(1% increase in X  .01 increase in ln(X)
 .011 increase in Y)
19
Example: TestScore vs. ln(Income)
 First defining the new regressor, ln(Income)
 The model is now linear in ln(Income), so the linear-log model
can be estimated by OLS:


TestScore = 557.8 + 36.42 ln(Incomei)
(3.8) (1.40)

so a 1% increase in Income is associated with an increase in

TestScore of 0.36 points on the test.
 Standard errors, confidence intervals, R2 – all the usual tools of
regression apply here.
 How does this compare to the cubic model?
20
The linear-log and cubic regression
functions

21
II. Log-linear population regression
function
ln(Y) = 0 + 1X (b)

Now change X: ln(Y + Y) = 0 + 1(X + X) (a)

Subtract (a) – (b): ln(Y + Y) – ln(Y) = 1X

Y
so  1X
Y
Y / Y
or 1  (small X)
X
22
Log-linear case, continued
ln(Yi) = 0 + 1Xi + ui

Y / Y
for small X, 1 
X
Y
 Now 100 = percentage change in Y, so a change in X by
Y
one unit (X = 1) is associated with a 100 1% change in Y.
 1 unit increase in X   1 increase in ln(Y)
 100 1% increase in Y
 Note: What are the units of ui and the SER?
 fractional (proportional) deviations

23
III. Log-log population regression
function
ln(Yi) = 0 + 1ln(Xi) + ui (b)

Now change X: ln(Y + Y) = 0 + 1ln(X + X) (a)

Subtract: ln(Y + Y) – ln(Y) = 1[ln(X + X) – ln(X)]

Y X
so  1
Y X
Y / Y
or 1  (small X)
X / X
24
Log-log case, continued
ln(Yi) = 0 + 1ln(Xi) + ui

for small X,

Y / Y
1 
X / X
Y X
Now 100 = percentage change in Y, and 100 =
Y X
percentage change in X, so a 1% change in X is associated with
a 1% change in Y.
 In the log-log specification, 1 has the interpretation of an
elasticity.
25
Example: ln( TestScore) vs. ln( Income)
 First defining a new dependent variable, ln(TestScore), and the
new regressor, ln(Income)
 The model is now a linear regression of ln(TestScore) against
ln(Income), which can be estimated by OLS:


ln(TestScore) = 6.336 + 0.0554 ln(Incomei)
(0.006) (0.0021)

An 1% increase in Income is associated with an increase of

.0554% in TestScore (Income up by a factor of 1.01,
TestScore up by a factor of 1.000554)
26
The log-linear and log-log specifications:

27
Summary: Logarithmic
transformations
 Three cases, differing in whether Y and/or X is transformed
by taking logarithms.
 The regression is linear in the new variable(s) ln(Y) and/or
ln(X), and the coefficients can be estimated by OLS.
 Hypothesis tests and confidence intervals are now
implemented and interpreted “as usual.”
 The interpretation of 1 differs from case to case.
 Choice of specification should be guided by judgment (which
interpretation makes the most sense in your application?),
tests, and plotting predicted values
28
Interactions Between Independent
Variables (SW Section 8.3)
 Perhaps a class size reduction is more effective in some
circumstances than in others…
 Perhaps smaller classes help more if there are many English
learners, who need individual attention
TestScore
 That is, might depend on PctEL
STR
Y
 More generally, might depend on X2
X 1
 How to model such “interactions” between X1 and X2?
 We first consider binary X’s, then continuous X’s
29
(a) Interactions between two binary
variables
Yi = 0 + 1D1i + 2D2i + ui

 D1i, D2i are binary

 1 is the effect of changing D1=0 to D1=1. In this specification,
this effect doesn’t depend on the value of D2.
 To allow the effect of changing D1 to depend on D2, include the
“interaction term” D1i D2i as a regressor:

Yi = 0 + 1D1i + 2D2i + 3(D1i D2i) + ui

30
Interpreting the coefficients
Yi = 0 + 1D1i + 2D2i + 3(D1i D2i) + ui

General rule: compare the various cases

E(Yi|D1i=0, D2i=d2) = 0 + 2d2 (b)
E(Yi|D1i=1, D2i=d2) = 0 + 1 + 2d2 + 3d2 (a)

subtract (a) – (b):

E(Yi|D1i=1, D2i=d2) – E(Yi|D1i=0, D2i=d2) = 1 + 3d2

 The effect of D1 depends on d2 (what we wanted)

 3 = increment to the effect of D1, when D2 = 1
31
Example: TestScore, STR, English
learners
Let
1 if STR  20  1 if PctEL  l0
HiSTR =  and HiEL = 
0 if STR  20 0 if PctEL  10


TestScore = 664.1 – 18.2HiEL – 1.9HiSTR – 3.5(HiSTR HiEL)
(1.4) (2.3) (1.9) (3.1)

 “Effect” of HiSTR when HiEL = 0 is –1.9

 “Effect” of HiSTR when HiEL = 1 is –1.9 – 3.5 = –5.4
 Class size reduction is estimated to have a bigger effect when
the percent of English learners is large
 This interaction isn’t statistically significant: t = 3.5/3.1
32
(b) Interactions between continuous
and binary variables
Yi = 0 + 1Di + 2Xi + ui

 Di is binary, X is continuous
 As specified above, the effect on Y of X (holding constant D) =
2, which does not depend on D
 To allow the effect of X to depend on D, include the
“interaction term” Di Xi as a regressor:

Yi = 0 + 1Di + 2Xi + 3(Di Xi) + ui

33
Binary-continuous interactions: the
two regression lines
Yi = 0 + 1Di + 2Xi + 3(Di Xi) + ui

Observations with Di= 0 (the “D = 0” group):

Yi = 0 + 2Xi + ui The D=0 regression line

Observations with Di= 1 (the “D = 1” group):

Yi = 0 + 1 + 2Xi + 3Xi + ui
= (0+1) + (2+3)Xi + ui The D=1 regression line
34
Binary-continuous interactions, ctd.

35
Interpreting the coefficients
Yi = 0 + 1Di + 2Xi + 3(Di Xi) + ui

General rule: compare the various cases

Y = 0 + 1D + 2X + 3(D X) (b)
Now change X:
Y + Y = 0 + 1D + 2(X+X) + 3[D (X+X)] (a)
subtract (a) – (b):
Y
Y = 2X + 3DX or = 2 + 3D
X
 The effect of X depends on D (what we wanted)
 3 = increment to the effect of X, when D = 1
36
Example: TestScore, STR, HiEL
(=1 if PctEL  10)

TestScore = 682.2 – 0.97STR + 5.6HiEL – 1.28(STR HiEL)
(11.9) (0.59) (19.5) (0.97)

 When HiEL = 0:

TestScore = 682.2 – 0.97STR
 When HiEL = 1,

TestScore = 682.2 – 0.97STR + 5.6 – 1.28STR
= 687.8 – 2.25STR
 Two regression lines: one for each HiSTR group.
 Class size reduction is estimated to have a larger effect when
the percent of English learners is large. 37
Example, ctd: Testing hypotheses

TestScore = 682.2 – 0.97STR + 5.6HiEL – 1.28(STR HiEL)
(11.9) (0.59) (19.5) (0.97)
 The two regression lines have the same slope  the
coefficient on STRHiEL is zero: t = –1.28/0.97 = –1.32
 The two regression lines have the same intercept  the
coefficient on HiEL is zero: t = –5.6/19.5 = 0.29
 The two regression lines are the same  population
coefficient on HiEL = 0 and population coefficient on
STR HiEL = 0: F = 89.94 (p-value < .001) !!
 We reject the joint hypothesis but neither individual
hypothesis (how can this be?)
38
(c) Interactions between two
continuous variables
Yi = 0 + 1X1i + 2X2i + ui

 X1, X2 are continuous

 As specified, the effect of X1 doesn’t depend on X2
 As specified, the effect of X2 doesn’t depend on X1
 To allow the effect of X1 to depend on X2, include the
“interaction term” X1i X2i as a regressor:

Yi = 0 + 1X1i + 2X2i + 3(X1i X2i) + ui

39
Interpreting the coefficients:
Yi = 0 + 1X1i + 2X2i + 3(X1i X2i) + ui

General rule: compare the various cases

Y = 0 + 1X1 + 2X2 + 3(X1 X2) (b)
Now change X1:
Y+ Y = 0 + 1(X1+X1) + 2X2 + 3[(X1+X1) X2] (a)
subtract (a) – (b):
Y
Y = 1X1 + 3X2X1 or = 1 + 3X2
X 1
 The effect of X1 depends on X2 (what we wanted)
 3 = increment to the effect of X1 from a unit change in X2
40
Example: TestScore, STR, PctEL

TestScore = 686.3 – 1.12STR – 0.67PctEL + .0012(STR PctEL),
(11.8) (0.59) (0.37) (0.019)

The estimated effect of class size reduction is nonlinear because

the size of the effect itself depends on PctEL:
TestScore
= –1.12 + .0012PctEL
STR
PctEL TestScore
STR
0 –1.12
20% –1.12+.0012 20 = –1.10
41
Example, ctd: hypothesis tests

TestScore = 686.3 – 1.12STR – 0.67PctEL + .0012(STR PctEL),
(11.8) (0.59) (0.37) (0.019)

 Does population coefficient on STR PctEL = 0?

t = .0012/.019 = .06  can’t reject null at 5% level
 Does population coefficient on STR = 0?
t = –1.12/0.59 = –1.90  can’t reject null at 5% level
 Do the coefficients on both STR and STR PctEL = 0?
F = 3.89 (p-value = .021)  reject null at 5% level(!!) (Why?
high but imperfect multicollinearity)
42
Application: Nonlinear Effects on
Test Scores of the Student-Teacher
Ratio (SW Section 8.4)
Nonlinear specifications let us examine more nuanced questions
about the Test score – STR relation, such as:

1. Are there nonlinear effects of class size reduction on test

scores? (Does a reduction from 35 to 30 have same effect as
a reduction from 20 to 15?)
2. Are there nonlinear interactions between PctEL and STR?
(Are small classes more effective when there are many
English learners?)

43
Strategy for Question #1 (different
effects for different STR?)
 Estimate linear and nonlinear functions of STR, holding constant
relevant demographic variables
 PctEL
 Income (remember the nonlinear TestScore-Income relation!)
 LunchPCT (fraction on free/subsidized lunch)
 See whether adding the nonlinear terms makes an “economically
important” quantitative difference (“economic” or “real-world”
importance is different than statistically significant)
 Test for whether the nonlinear terms are significant

44
Strategy for Question #2
(interactions between PctEL and
STR?)
 Estimate linear and nonlinear functions of STR, interacted with
PctEL.
 If the specification is nonlinear (with STR, STR2, STR3), then you
need to add interactions with all the terms so that the entire
functional form can be different, depending on the level of
PctEL.
 We will use a binary-continuous interaction specification by
adding HiELSTR, HiELSTR2, and HiELSTR3.

45
46
Tests of joint hypotheses:

47