Slides by

JOHN
LOUCKS
St. Edwards
University

2008 Thomson South-Western. All Rights Reserved 1

 Chapter 16
Regression Analysis: Model Building
General Linear Model
Determining When to Add or Delete
Variables
Variable-Selection
Procedures
Multiple Regression Approach to Experimental
Design
Autocorrelation and the Durbin-Watson Test

2008 Thomson South-Western. All Rights Reserved 2

 General Linear Model

Models in which the parameters (0, 1, . . . , p ) all

have exponents of one are called linear models.
A general linear model involving p
independent variables is
y 0 1z1 2z2 L pzp

Each of the independent variables z is a

function of x1, x2,..., xk (the variables for which
data have been collected).

2008 Thomson South-Western. All Rights Reserved 3

 General Linear Model

The simplest case is when we have collected

data for just one variable x1 and want to
estimate y by using a straight-line relationship.
In this case z = x1.
This model is 1called a simple first-order model
with one predictor variable.

y 0 1x1

2008 Thomson South-Western. All Rights Reserved 4

 Modeling Curvilinear Relationships

To account for a curvilinear relationship, we

2
might set z1 = x1xand
1 z2 = .
This model is called a second-order model with
one predictor variable.

y 0 1x1 2x12

2008 Thomson South-Western. All Rights Reserved 5

 Interaction

If the original data set consists of observations

for y and two independent variables x1 and x2
we might develop a second-order model with
two predictor variables.
y 0 1x1 2x2 3x12 4x22 5x1x2

In this model, the variable z5 = x1x2 is added to

account for the potential effects of the two
variables acting together.
This type of effect is called interaction.

2008 Thomson South-Western. All Rights Reserved 6

 Transformations Involving the Dependent
Variable
Often the problem of nonconstant variance can
be
corrected by transforming the dependent
variable to a
Most statistical packages provide the ability to
different scale.
apply
logarithmic transformations using either the
base-10
Another approach, called a reciprocal transformation, is
(common
to use 1/y aslog) or the base
the dependent e = 2.71828...
variable instead of y.
(natural log).

2008 Thomson South-Western. All Rights Reserved 7

 Nonlinear Models That Are Intrinsically
Linear
Models in which the parameters (0, 1, . . . , p )
have exponents other than one are called
nonlinear models
In some cases we. can perform a transformation
of variables that will enable us to use
regression analysis with the general linear
model.
The exponential model involves the regression
equation:
E(y) 0 x1

We can transform this nonlinear model to a linear

model by taking the logarithm of both sides.

2008 Thomson South-Western. All Rights Reserved 8

 Determining When to Add or Delete
Variables
To test whether the addition of x2 to a model
involving x1 (or the deletion of x2 from a model
involving x1 and x2) is statistically significant we
can
The perform
F Test is an F Test
based on. a determination of the
amount of reduction in the error sum of squares
resulting from adding one or more independent
variables to the model.
(SSE(reduced)-SSE(full))/ number of extra terms
F
MSE(full)

(SSE(x1 )-SSE(x1 ,x2 ))/ 1

F
(SSE(x1 , x2 ))/ (n p 1)

2008 Thomson South-Western. All Rights Reserved 9

 Determining When to Add or Delete
Variables
The p value criterion can also be used to
determine whether it is advantageous to add
one or more dependent variables to a multiple
regression model.
The p value associated with the computed F
statistic can be compared to the level of
significance .
It is difficult to determine the p value directly
from the tables of the F distribution, but
computer software packages, such as Minitab
or Excel, provide the pvalue.

2008 Thomson South-Western. All Rights Reserved 10

 Variable Selection Procedures

Stepwise Iterative; one independent

Regression variable at a time is added or
Forward Selection deleted based on the F statistic
Backward Different subsets of the
Best-Subsets
Elimination independent variables
Regression
are evaluated

The first 3 procedures are

heuristics.
There is no guarantee that the
best model will be found.

2008 Thomson South-Western. All Rights Reserved 11

 Variable Selection: Stepwise Regression

At each iteration, the first consideration is to

see whether the least significant variable
currently in the model can be removed
because its F value is less than the user-
specified or default
If no variable can beAlpha to remove
removed, .
the procedure
checks to see whether the most significant
variable not in the model can be added
because its F value is greater than the user-
specified or default
If no variable can beAlpha to enter
removed and. no variable
can be added, the procedure stops.

2008 Thomson South-Western. All Rights Reserved 12

 Variable Selection: Forward Selection

This procedure is similar to stepwise

regression, but does not permit a variable to
be deleted.
This forward-selection procedure starts with no
independent variables.
It adds variables one at a time as long as a
significant reduction in the error sum of
squares (SSE) can be achieved.

2008 Thomson South-Western. All Rights Reserved 13

 Variable Selection: Backward Elimination

This procedure begins with a model that

includes all the independent variables the
modeler wants considered.
It then attempts to delete one variable at a
time by determining whether the least
significant variable currently in the model can
be removed because its p-value is less than
the
Once user-specified or been
a variable has default value. from the
removed
model it cannot reenter at a subsequent step.

2008 Thomson South-Western. All Rights Reserved 14

 Variable Selection: Backward Elimination

Example: Clarksville Homes

Tony Zamora, a real estate investor, has
just
moved to Clarksville and wants to
learn about the citys residential
real estate market. Tony has ran-
domly selected 25 house-for-sale
listings from the Sunday news-
paper and collected the data
partially listed on an upcoming
slide.

2008 Thomson South-Western. All Rights Reserved 15

 Variable Selection: Backward Elimination

Example: Clarksville Homes

Develop, using the backward elimination
procedure, a multiple regression
model to predict the selling price
of a house in Clarksville.

2008 Thomson South-Western. All Rights Reserved 16

 Variable Selection: Backward Elimination

Partial Data

Note: Rows 10-26 are not shown.

2008 Thomson South-Western. All Rights Reserved 17

 Variable Selection: Backward Elimination

Regression Output

Greates
Variable t
to be p-value
removed > .05
2008 Thomson South-Western. All Rights Reserved 18
 Variable Selection: Backward Elimination

Cars (garage size) is the independent variable

with the highest p-value (.697) > .05.
Cars variable is removed from the
model.
Multiple regression is performed again on
the remaining independent variables.

2008 Thomson South-Western. All Rights Reserved 19

 Variable Selection: Backward Elimination

Regression Output

Greates
Variable t
to be p-value
removed > .05

2008 Thomson South-Western. All Rights Reserved 20

 Variable Selection: Backward Elimination

Bedrooms is the independent variable with the

highest p-value (.281) > .05.
Bedrooms variable is removed from the
model.
Multiple regression is performed again on
the remaining independent variables.

2008 Thomson South-Western. All Rights Reserved 21

 Variable Selection: Backward Elimination

Regression Output

Greates
Variable t
to be p-value
removed > .05

2008 Thomson South-Western. All Rights Reserved 22

 Variable Selection: Backward Elimination

Bathrooms is the independent variable with

the highest p-value (.110) > .05.
Bathrooms variable is removed from the
model.
Multiple regression is performed again on
the remaining independent variable.

2008 Thomson South-Western. All Rights Reserved 23

 Variable Selection: Backward Elimination

Regression Output

Greates
t
p-value
is < .05

2008 Thomson South-Western. All Rights Reserved 24

 Variable Selection: Backward Elimination

House size is the only independent variable

remaining in the model.
The estimated regression equation is:

y 9.8669 11.3383(House Size)

2008 Thomson South-Western. All Rights Reserved 25

 Variable Selection: Best-Subsets
Regression
The three preceding procedures are one-
variable-at-a-time methods offering no
guarantee that the best model for a given
number of variables
Some software will be
packages found.best-subsets
include
regression that enables the user to find, given
a specified number of independent variables,
the best regression model.
Minitab output identifies the two best one-
variable estimated regression equations, the
two best two-variable equation, and so on.

2008 Thomson South-Western. All Rights Reserved 26

 Variable Selection: Best-Subsets
Regression
Example: PGA Tour Data
The Professional Golfers Association keeps a
variety of statistics regarding
performance measures. Data
include the average driving
distance, percentage of drives
that land in the fairway, percent-
age of greens hit in regulation,
average number of putts, percentage
of sand saves, and average score.

2008 Thomson South-Western. All Rights Reserved 27

 Variable-Selection Procedures

Variable Names and Definitions

Drive: average length of a drive in yards

Fair: percentage of drives that land in the fairw
Green: percentage of greens hit in regulation (a par-3
green is hit in regulation if the players first
shot lands on the green)
Putt: average number of putts for greens that
been hit in regulation
Sand: percentage of sand saves (landing in a sand
trap and still scoring par or better)
Score: average score for an 18-hole round

2008 Thomson South-Western. All Rights Reserved 28

 Variable-Selection Procedures

Sample Data

Drive Fair Green Putt Sand

Score
277.6 .681 .667 1.768 .550 69.10
259.6 .691 .665 1.810 .536 71.09
269.1 .657 .649 1.747 .472 70.12
267.0 .689 .673 1.763 .672 69.88
267.3 .581 .637 1.781 .521 70.71
255.6 .778 .674 1.791 .455 69.76
272.9 .615 .667 1.780 .476 70.19
265.4 .718 .699 1.790 .551 69.73

2008 Thomson South-Western. All Rights Reserved 29

 Variable-Selection Procedures

Sample Correlation Coefficients

Score Drive Fair Green Putt

Drive -.154
Fair -.427 -.679
Green -.556 -.045 .421
Putt .258 -.139 .101 .354
Sand -.278 -.024 .265 .083 -.296
-.29

2008 Thomson South-Western. All Rights Reserved 30

 Multiple Regression Approach to
Experimental Design
The use of dummy variables in a multiple
regression equation can provide another
approach to solving analysis of variance and
experimental design problems.
We will use the results of multiple regression to
perform the ANOVA test on the difference in
the means of three populations.

2008 Thomson South-Western. All Rights Reserved 31

 Multiple Regression Approach to
Experimental Design
Example: Reed Manufacturing
Janet Reed would like to know if
there is any significant difference in
the mean number of hours worked per
week for the department managers
at her three manufacturing plants
(in Buffalo, Pittsburgh, and Detroit).

2008 Thomson South-Western. All Rights Reserved 32

 Multiple Regression Approach to
Experimental Design
Example: Reed Manufacturing
A simple random sample of five
managers from each of the three plants
was taken and the number of hours
worked by each manager for the
previous week is shown on the next
slide.

2008 Thomson South-Western. All Rights Reserved 33

 Multiple Regression Approach to
Experimental Design

Plant 1 Plant 2 Plant 3

Observation Buffalo Pittsburgh Detroit
1 48 73 51
2 54 63 63
3 57 66 61
4 54 64 54
5 62 74 56
Sample Mean 55 68 57
Sample Variance 26.0 26.5 24.5

2008 Thomson South-Western. All Rights Reserved 34

 Multiple Regression Approach to
Experimental Design
We begin by defining two dummy variables, A
and B, that will indicate the plant from which
each sample observation was selected.
In general, if there are k populations, we need
to define k 1 dummy variables.

A = 0, B = 0 if observation is from Buffalo

A=plant
1, B = 0 if observation is from
A=Pittsburgh
0, B = 1 plant
if observation is from
Detroit plant

2008 Thomson South-Western. All Rights Reserved 35

 Multiple Regression Approach to
Experimental Design
Input Data

Plant 1 Plant 2 Plant 3

Buffalo Pittsburgh Detroit
A B y A B y A B y
0 0 48 1 0 73 0 1 51
0 0 54 1 0 63 0 1 63
0 0 57 1 0 66 0 1 61
0 0 54 1 0 64 0 1 54
0 0 62 1 0 74 0 1 56

2008 Thomson South-Western. All Rights Reserved 36

 Multiple Regression Approach to
Experimental Design

E(y) = expected number of hours worked

= 0 + 1 A + 2 B

For Buffalo: E(y) = 0 + 1(0) + 2(0) =

For0Pittsburgh: E(y) = 0 + 1(1) + 2(0) = 0
1
For+Detroit: E(y) = 0 + 1(0) + 2(1) =
0 + 2

2008 Thomson South-Western. All Rights Reserved 37

 Multiple Regression Approach to
Experimental Design

Excel produced the regression equation:

y = 55 +13A + 2B

Plant Estimate of E(y)

Buffalo b0 = 55
Pittsburgh b0 + b1 = 55 + 13 = 68
Detroit b0 + b2 = 55 + 2 = 57

2008 Thomson South-Western. All Rights Reserved 38

 Multiple Regression Approach to
Experimental Design
Next, we observe that if there is no difference
in the means:
E(y) for the Pittsburgh plant E(y) for the Buffalo plant = 0
E(y) for the Detroit plant E(y) for the Buffalo plant = 0

2008 Thomson South-Western. All Rights Reserved 39

 Multiple Regression Approach to
Experimental Design
Because 0 equals E(y) for the Buffalo plant and
0 + 1 equals E(y) for the Pittsburgh
plant, the first difference is equal to (0 + 1) - 0
= 1 .
Because 0 + 2 equals E(y) for the Detroit
plant, the second difference is equal to (0 +

We 0 = 2conclude
2) -would . that there is no difference
in the three means if 1 = 0 and 2 = 0.

2008 Thomson South-Western. All Rights Reserved 40

 Multiple Regression Approach to
Experimental Design
The null hypothesis for a test of the difference
of means is
H0: 1 = 2 =
0
To test this null hypothesis, we must compare
the value of MSR/MSE to the critical value from
an F distribution with the appropriate
numerator and denominator degrees of
freedom.

2008 Thomson South-Western. All Rights Reserved 41

 Multiple Regression Approach to
Experimental Design
ANOVA Table Produced by Excel

Source of Sum of Degrees of Mean

Variation Squares Freedom Squares F p
Regression 490 2 245 9.55 .003
Error 308 12 25.667
Total 798 14

2008 Thomson South-Western. All Rights Reserved 42

 Multiple Regression Approach to
Experimental Design
At a .05 level of significance, the critical value
of F with k 1 = 3 1 = 2 numerator d.f.
and nT k = 15 3 = 12 denominator d.f. is
3.89.
Because the observed value of F (9.55) is
greater than the critical value of 3.89, we reject
the null hypothesis.
Alternatively, we reject the null hypothesis
because the p-value of .003 < = .05.

2008 Thomson South-Western. All Rights Reserved 43

 Autocorrelation and the Durbin-Watson
Test
Often, the data used for regression studies in
business and economics are collected over
time.
It is not uncommon for the value of y at one
time period to be related to the value of y at
previous time periods.
In this case, we say autocorrelation (or serial
correlation) is present in the data.

2008 Thomson South-Western. All Rights Reserved 44

 Autocorrelation and the Durbin-Watson
Test
With positive autocorrelation, we expect a
positive residual in one period to be followed
by a positive residual in the next period.
With positive autocorrelation, we expect a
negative residual in one period to be followed
by a negative residual in the next period.
With negative autocorrelation, we expect a
positive residual in one period to be followed
by a negative residual in the next period, then a
positive residual, and so on.

2008 Thomson South-Western. All Rights Reserved 45

 Autocorrelation and the Durbin-Watson
Test
When autocorrelation is present, one of the
regression assumptions is violated: the error
terms are not independent.
When autocorrelation is present, serious errors
can be made in performing tests of
significance based upon the assumed
regression model.
The Durbin-Watson statistic can be used to
detect first-order autocorrelation.

2008 Thomson South-Western. All Rights Reserved 46

 Autocorrelation and the Durbin-Watson
Test
Durbin-Watson Test Statistic
The statistic ranges in value from zero to
four.
If successive values of the residuals are
close
together (positive autocorrelation is
present),
If successive values are far apart
the statistic will be small.
(negative
autocorrelation is present), the
statistic will
Abevalue of two indicates no
large.
autocorrelation.

2008 Thomson South-Western. All Rights Reserved 47

 Autocorrelation and the Durbin-Watson
Test
Suppose the values of (residuals) are not
independent but are related in the following
manner: t = t-1 + zt

where is a parameter with an absolute value

less than
one and zt is a normally and independently
Wedistributed
see that if = 0, the error terms are not
random
related. variable2 with a mean of zero and
The of . test uses the residuals to
Durbin-Watson
variance
determine whether = 0.

2008 Thomson South-Western. All Rights Reserved 48

 Autocorrelation and the Durbin-Watson
Test
The null hypothesis always is:
H0 : 0 there is no autocorrelation
The alternative hypothesis is:
Ha : 0 to test for positive autocorrelation

Ha : 0 to test for negative autocorrelation

Ha : 0 to test for positive or negative

autocorrelation

2008 Thomson South-Western. All Rights Reserved 49

 Autocorrelation and the Durbin-Watson
Test
A Sample Of Critical Values For The
Durbin-Watson Test For Autocorrelation
Significance Points of dL and dU: =
.05
Number of Independent Variables
1 2 3 4 5
n dL dU dL dU dL dU dU dU dU dU
1.0 1.3 0.9 1.5 0.8 1.7 0.6 1.9 0.5 2.2
15
8 6 5 4 2 5 9 7 6 1
1.1 1.3 0.9 1.5 0.8 1.7 0.7 1.9 0.6 2.1
16
0 7 8 4 6 3 4 3 2 5
1.1 1.3 1.0 1.5 0.9 1.7 0.7 1.9 0.6 2.1
17
3 8 2 4 0 1 8 0 7 0
2008 Thomson South-Western. All Rights Reserved
2008 Thomson South-Western. All Rights Reserved 50
 Autocorrelation and the Durbin-Watson
Test
Positive Incon- No evidence of
autocor- clusive positive autocorrelation
relation
0 dL dU 2 4-dU 4-dL 4

No evidence of Incon- Negative

negative autocorrelation clusive autocor-
relation
0 dL dU 2 4-dU 4-dL 4

Positive Incon- No evidence of Incon- Negative

autocor- clusive autocorrelation clusive autocor-
relation relation
0 dL dU 2 4-dU 4-dL 4

2008 Thomson South-Western. All Rights Reserved 51

 Chapter 16
Regression Analysis: Model Building
General Linear Model
Determining When to Add or Delete
Variables
Variable-Selection
Procedures
Multiple Regression Approach to
Experimental Design
Autocorrelation and the
Durbin-Watson Test

2008 Thomson South-Western. All Rights Reserved 52