What is a Math/Stats Model?

1. Often Describe Relationship between Variables

2. Types
- Deterministic Models (no randomness)

- Probabilistic Models (with randomness)

EPI 809/Spring 2008 1

Regression Models

EPI 809/Spring 2008 2

Types of
Probabilistic Models

Probabilistic
Models

Regression Correlation Other

Models Models Models

EPI 809/Spring 2008 3

Regression Models

Relationship between one dependent variable and explanatory

variable(s)
Use equation to set up relationship
Numerical Dependent (Response) Variable
1 or More Numerical or Categorical Independent (Explanatory) Variables
Used Mainly for Prediction & Estimation

EPI 809/Spring 2008 4

Regression Modeling Steps

1. Hypothesize Deterministic Component

Estimate Unknown Parameters
2. Specify Probability Distribution of Random Error Term
Estimate Standard Deviation of Error
3. Evaluate the fitted Model
4. Use Model for Prediction & Estimation

EPI 809/Spring 2008 5

Model Specification

EPI 809/Spring 2008 6

Specifying the deterministic
component

1. Define the dependent variable and independent variable

2. Hypothesize Nature of Relationship

Expected Effects (i.e., Coefficients Signs)
Functional Form (Linear or Non-Linear)
Interactions

EPI 809/Spring 2008 7

Types of
Regression Models

EPI 809/Spring 2008 8

Types of
Regression Models

Regression
Models

EPI 809/Spring 2008 9

Types of
Regression Models

1 Explanatory Regression
Variable Models

Simple

EPI 809/Spring 2008 10

Types of
Regression Models

1 Explanatory Regression 2+ Explanatory

Variable Models Variables

Simple Multiple

EPI 809/Spring 2008 11

Types of
Regression Models

1 Explanatory Regression 2+ Explanatory

Variable Models Variables

Simple Multiple

Linear

EPI 809/Spring 2008 12

Types of
Regression Models

1 Explanatory Regression 2+ Explanatory

Variable Models Variables

Simple Multiple

Non-
Linear
Linear

EPI 809/Spring 2008 13

Types of
Regression Models

1 Explanatory Regression 2+ Explanatory

Variable Models Variables

Simple Multiple

Non-
Linear Linear
Linear

EPI 809/Spring 2008 14

Types of
Regression Models

1 Explanatory Regression 2+ Explanatory

Variable Models Variables

Simple Multiple

Non- Non-
Linear Linear
Linear Linear

EPI 809/Spring 2008 15

Linear Regression Model

EPI 809/Spring 2008 16

Types of
Regression Models

1 Explanatory Regression 2+ Explanatory

Variable Models Variables

Simple Multiple

Non- Non-
Linear Linear
Linear Linear

EPI 809/Spring 2008 17

Linear Equations

Y
Y = mX + b
Change
m = Slope in Y
Change in X
b = Y-intercept
X

1984-1994 T/Maker Co.

EPI 809/Spring 2008 18
Linear Regression Model

1. Relationship Between Variables Is a Linear Function

Population Population Random

Y-Intercept Slope Error

Yi 0 1X i i
Dependent Independent (Explanatory)
(Response) Variable
Variable (e.g., Years s. serocon.)
(e.g., CD+ c.)
Population & Sample Regression Models

EPI 809/Spring 2008 20

Population & Sample Regression Models

Population

EPI 809/Spring 2008 21
Population & Sample Regression Models

Population

Unknown
Relationship
Yi 0 1X i i



EPI 809/Spring 2008 22
Population & Sample Regression Models

Population Random Sample

Unknown
Relationship
Yi 0 1X i i




EPI 809/Spring 2008 23
Population & Sample Regression Models

Population Random Sample

Unknown

Yi 0 1X i i
Relationship
Yi 0 1X i i




EPI 809/Spring 2008 24
 Population Linear Regression Model

Y Yi 0 1X i i Observedv
alue

i = Random error

E Y 0 1 X i

X
Observed value
EPI 809/Spring 2008 25
Sample Linear Regression Model

Y Yi 0 1X i i

^
i = Random
error
Unsampled
observation
Yi 0 1X i
X
Observed value
EPI 809/Spring 2008 26
 Keep in mind that the regressand Y and the regressor X themselves may be
nonlinear.
 look at Table 2.1. Keeping the value of income X fixed, say, at $80, we
draw at random a family and observe its weekly family consumption
expenditure Y as, say, $60. Still keeping X at $80, we draw at random
another family and observe its Y value as $75. In each of these
drawings (i.e., repeated sampling), the value of X is fixed at $80. We
can repeat this process for all the X values shown in Table 2.1.
This means that our regression analysis is conditional regression
analysis, that is, conditional on the given values of the regressor(s) X.
 As shown in Figure 3.3, each Y population corresponding to a given X
is distributed around its mean value with some Y values above the
mean and some below it. the mean value of these deviations
corresponding to any given X should be zero.
Note that the assumption E(ui | Xi) = 0 implies that E(Yi | Xi) = 1 +
2Xi.
E(ui | Xi) = 0
 Technically, (3.2.2) represents the assumption of homoscedasticity, or equal
(homo) spread (scedasticity) or equal variance. Stated differently, (3.2.2)
means that the Y populations corresponding to various X values have the
same variance.
Put simply, the variation around the regression line (which is the line of
average relationship between Y and X) is the same across the X values; it
neither increases or decreases as X varies
 In Figure 3.5, where the conditional variance of the Y population
varies with X. This situation is known as heteroscedasticity, or unequal
spread, or variance. Symbolically, in this situation (3.2.2) can be
written as
var (ui | Xi) = 2i (3.2.3)
Figure 3.5. shows that, var (u| X1) < var (u| X2), . . . , < var (u| Xi).
Therefore, the likelihood is that the Y observations coming from the
population with X = X1 would be closer to the PRF than those coming
from populations corresponding to X = X2, X = X3, and so on. In short,
not all Y values corresponding to the various Xs will be equally
reliable, reliability being judged by how closely or distantly the Y
values are distributed around their means, that is, the points on the
PRF.
 The disturbances ui and uj are uncorrelated, i.e., no serial correlation. This
means that, given Xi , the deviations of any two Y values from their mean
value do not exhibit patterns. In Figure 3.6a, the us are positively correlated,
a positive u followed by a positive u or a negative u followed by a negative u.
In Figure 3.6b, the us are negatively correlated, a positive u followed by a
negative u and vice versa. If the disturbances follow systematic patterns,
Figure 3.6a and b, there is auto- or serial correlation. Figure 3.6c shows that
there is no systematic pattern to the us, thus indicating zero correlation.
 The disturbance u and explanatory variable X are uncorrelated. The PRF
assumes that X and u (which may represent the influence of all the omitted
variables) have separate (and additive) influence on Y. But if X and u are
correlated, it is not possible to assess their individual effects on Y. Thus, if X
and u are positively correlated, X increases when u increases and it decreases
when u decreases. Similarly, if X and u are negatively correlated, X increases
when u decreases and it decreases when u increases. In either case, it is
difficult to isolate the influence of X and u on Y.
 In the hypothetical example of Table 3.1, imagine that we had only the first
pair of observations on Y and X (4 and 1). From this single observation there
is no way to estimate the two unknowns, 1 and 2. We need at least two pairs
of observations to estimate the two unknowns
 This assumption too is not so innocuous as it looks. Look at Eq. (3.1.6). If
all the X values are identical, then Xi = X and the denominator of that
equation will be zero, making it impossible to estimate 2 and therefore 1.
Looking at our family consumption expenditure example in Chapter 2, if
there is very little variation in family income, we will not be able to explain
much of the variation in the consumption expenditure.
 An econometric investigation begins with the specification of the
econometric model underlying the phenomenon of interest. Some important
questions that arise in the specification of the model include the following:
(1) What variables should be included in the model?
(2) What is the functional form of the model? Is it linear in the parameters,
the variables, or both?
(3) What are the probabilistic assumptions made about the Yi , the Xi, and
the ui entering the model?
 Suppose we choose the following two models to depict the underlying
relationship between the rate of change of money wages and the
unemployment rate:
Yi = 1 + 2Xi + ui (3.2.7)
Yi = 1 + 2 (1/Xi ) + ui (3.2.8)
where Yi = the rate of change of money wages, and Xi = the unemployment
rate. The regression model (3.2.7) is linear both in the parameters and the
variables, whereas (3.2.8) is linear in the parameters (hence a linear
regression model by our definition) but nonlinear in the variable X. Now
consider Figure 3.7.
If model (3.2.8) is the correct or the true model, fitting the model
(3.2.7) to the scatterpoints shown in Figure 3.7 will give us wrong
predictions.
Unfortunately, in practice one rarely knows the correct variables to include
in the model or the correct functional form of the model or the correct
probabilistic assumptions about the variables entering the model for the
theory underlying the particular investigation may not be strong or robust
 We will discuss this assumption in Chapter 7, where we discuss multiple
regression models.
 PRECISION OR STANDARD ERRORS OF LEAST-
SQUARES ESTIMATES

The least-squares estimates are a function of the sample data. But since the
data change from sample to sample, the estimates will change. Therefore,
what is needed is some measure of reliability or precision of the estimators
1 and 2. In statistics the precision of an estimate is measured by its
standard error (se), which can be obtained as follows:
 2 is the constant or homoscedastic variance of ui of Assumption 4.
2 itself is estimated by the following formula:

where 2 is the OLS estimator of the true but unknown 2 and where the
expression n2 is known as the number of degrees of freedom (df), is
the residual sum of squares (RSS). Once is known, 2 can be easily
computed.

Compared with Eq. (3.1.2), Eq. (3.3.6) is easy to use, for it does not require
computing ui for each observation.
 Since

an alternative expression for computing is

In passing, note that the positive square root of 2

is known as the standard error of estimate or the standard error of the

regression (se). It is simply the standard deviation of the Y values about the
estimated regression line and is often used as a summary measure of the
goodness of fit of the estimated regression line.
 Note the following features of the variances (and therefore the standard
errors) of 1 and 2.

1. The variance of 2 is directly proportional to 2 but inversely proportional

to x2i . That is, given 2, the larger the variation in the X values, the smaller
the variance of 2 and hence the greater the precision with which 2 can be
estimated.

2. The variance of 1 is directly proportional to 2 and X2i but inversely

proportional to x2i and the sample size n.
 3. Since 1 and 2 are estimators, they will not only vary from sample to
sample but in a given sample they are likely to be dependent on each other,
this dependence being measured by the covariance between them.

Since var (2) is always positive, as is the variance of any variable, the nature
of the covariance between 1 and 2 depends on the sign of X . If X is
positive, then as the formula shows, the covariance will be negative. Thus, if
the slope coefficient 2 is overestimated (i.e., the slope is too steep), the
intercept coefficient 1 will be underestimated (i.e., the intercept will be too
small).
 PROPERTIES OF LEAST-SQUARES ESTIMATORS:
THE GAUSSMARKOV THEOREM

To understand this theorem, we need to consider the best linear

unbiasedness property of an estimator. An estimator, say the OLS estimator
2, is said to be a best linear unbiased estimator (BLUE) of 2 if the
following hold:
1. It is linear, that is, a linear function of a random variable, such as the
dependent variable Y in the regression model.
2. It is unbiased, that is, its average or expected value, E(2), is equal to the
true value, 2.
3. It has minimum variance in the class of all such linear unbiased
estimators; an unbiased estimator with the least variance is known as an
efficient estimator.
 What all this means can be explained with the aid of Figure 3.8. In Figure
3.8(a) we have shown the sampling distribution of the OLS estimator 2, that
is, the distribution of the values taken by 2 in repeated sampling experiment.
For convenience we have assumed 2 to be distributed symmetrically. As the
figure shows, the mean of the 2 values, E(2), is equal to the true 2. In this
situation we say that 2 is an unbiased estimator of 2. In Figure 3.8(b) we
have shown the sampling distribution of 2, an alternative estimator of 2
obtained by using another (i.e., other than OLS) method.
 For convenience, assume that *2, like 2, is unbiased, that is, its average or
expected value is equal to 2. Assume further that both 2 and *2 are linear
estimators, that is, they are linear functions of Y. Which estimator, 2 or *2,
would you choose? To answer this question, superimpose the two figures, as
in Figure 3.8(c). It is obvious that although both 2 and *2 are unbiased
the distribution of *2 is more diffused or widespread around the mean
value than the distribution of 2. In other words, the variance of *2 is
larger than the variance of 2.
Now given two estimators that are both linear and unbiased, one would
choose the estimator with the smaller variance because it is more likely to
be close to 2 than the alternative estimator. In short, one would choose the
BLUE estimator.
 THE COEFFICIENT OF DETERMINATION r2:
A MEASURE OF GOODNESS OF FIT

We now consider the goodness of fit of the fitted regression line to a set of
data; that is, we shall find out how well the sample regression line fits the
data. The coefficient of determination r2 (two-variable case) or R2 (multiple
regression) is a summary measure that tells how well the sample regression
line fits the data.
Consider a heuristic explanation of r2 in terms of a graphical device, known
as the Venn diagram shown in Figure 3.9.
In this figure the circle Y represents variation in the dependent variable Y and
the circle X represents variation in the explanatory variable X. The overlap of
the two circles indicates the extent to which the variation in Y is explained
by the variation in X.
 To compute this r2, we proceed as follows: Recall that
Yi = Yi +ui (2.6.3)
or in the deviation form
yi = yi + ui (3.5.1)
where use is made of (3.1.13) and (3.1.14). Squaring (3.5.1) on both sides
and summing over the sample, we obtain

Since = 0 and yi = 2xi .

 The various sums of squares appearing in (3.5.2) can be described as
follows: = total variation of the actual Y values about their
sample mean, which may be called the total sum of squares (TSS).
= variation of the estimated Y
values about their mean (Y = Y), which appropriately may be called the
sum of squares due to/or explained by regression, or simply the explained
sum of squares (ESS). = residual or unexplained variation of the Y
values about the regression line, or simply the residual sum of squares (RSS).
Thus, (3.5.2) is
TSS = ESS + RSS (3.5.3)
and shows that the total variation in the observed Y values about their mean
value can be partitioned into two parts, one attributable to the regression
line and the other to random forces because not all actual Y observations lie
on the fitted line. Geometrically, we have Figure 3.10
 The quantity r2 thus defined is known as the (sample) coefficient of
determination and is the most commonly used measure of the goodness of fit
of a regression line. Verbally, r2 measures the proportion or percentage of the
total variation in Y explained by the regression model.
Two properties of r2 may be noted:
1. It is a nonnegative quantity.
2. Its limits are 0 r2 1. An r2 of 1 means a perfect fit, that is, Y i = Yi for
each i. On the other hand, an r2 of zero means that there is no relationship
between the regressand and the regressor whatsoever (i.e., 2 = 0). In this
case, as (3.1.9) shows, Yi = 1 = Y, that is, the best prediction of any Y value
is simply its mean value. In this situation therefore the regression line will
be horizontal to the X axis.
Although r2 can be computed directly from its definition given in (3.5.5), it
can be obtained more quickly from the following formula:
 Some of the properties of r are as follows (see Figure 3.11):
1. It can be positive or negative,
2. It lies between the limits of 1 and +1; that is, 1 r 1.
3. It is symmetrical in nature; that is, the coefficient of correlation between X
and Y(rXY) is the same as that between Y and X(rYX).
4. It is independent of the origin and scale; that is, if we define X*i = aXi + C
and Y*i = bYi + d, where a > 0, b > 0, and c and d are constants, then r
between X* and Y* is the same as that between the original variables X and Y.
5. If X and Y are statistically independent, the correlation coefficient between
them is zero; but if r = 0, it does not mean that two variables are
independent.
6. It is a measure of linear association or linear dependence only; it has no
meaning for describing nonlinear relations.
7. Although it is a measure of linear association between two variables, it
does not necessarily imply any cause-and-effect relationship.