B.

Com Finance

Financial Econometrics and data analysis

Linear Regression Models

Lectured by

Dr. Edson Mbedzi

Department of Finance & Fiscal Sciences

1
 Session outline
At the end of this session, students must be able to:

• Define linear regression models.

• Understand and differentiate various linear regression models; single and multiple regressions.

• The basis of OLS

• Assumptions of linear regression models and how they are tested.

• Apply linear regression models using selected computer software packages and interpret the
results.

• Present the results academically.

 Linear Regression
•It is one of the most widely known econometric modelling technique in quantitative data
analysis.

•Linear regression is concerned with the describing and evaluating the relationship
between a given variable and one or more other variables.

•In this technique, the dependent variable is continuous, independent variable(s) can
be continuous or discrete, and nature of regression line is linear.

•Linear Regression establishes a relationship between dependent variable (Y) and one
or more independent variables (X) using a best fit straight line (also known as
regression line).
 Classical Linear Regression Model
 In econometrics, there is need to examine the relationship
between two or more financial variables.
 The relationship between variables can be explored by
a. Constructing scatter diagrams,
b. Building a linear regression model
c. Calculating a correlation coefficient
Scatter plots
 It is a graph that shows the relationship between
the observations for two data series in two
dimensions.
 The pattern of data is indicative of the type of
relationship between your two variables
Positive relationship
Negative relationship
No relationship
Positive relationship
Negative relationship

Reliability

Age of Car
No relation
 Types of linear regression models
•There are two types of linear regression models.

1. Simple regression model

• Denoted byYi    xi  i where Y is the dependent variable and small x is the only factor
influencing or causing Y to change.

• E.g. Analysing how asset returns vary with changes in the level of market risk. Here asset
returns are expected to be affected or influenced to change by one factor only i.e. market risk.

2. Multiple regression model

Yi   X i  i
• Denoted by where Y is the dependent variable and big X is a vector of a
number of factors influencing changes in Y.

• E.g. Evaluating how Share price of a company is influenced by company size, market risk,
sector sensitive risk and inward foreign direct investment to the sector. Here share
price of a listed company is expected to be affected or influenced to change by a
number of factors. This is more realistic in real practice, so multiple regression has
more value in evaluating such a finance models.
THE SIMPLE REGRESSION MODEL
 Regression analysis is concerned with describing and
evaluating the relationship between a given
variable (explained or dependent variable) and one
or more other variables (explanatory or independent
variables).
 In statistical modelling, regression analysis is a
statistical process for estimating the relationships
among variables.
 Explained variable is denoted by y and explanatory
variable by x.
 Regression is an attempt to explain the variation in a
dependent variable using the variation in
independent variables.
 Regression is thus an explanation of causation.
 If the independent variable(s) sufficiently explain the
variation in the dependent variable, the model can
be used for prediction.
Sources of Errors in Regression

 The question which remains unanswered is why

should we add an error term? What are the sources
of the error term u in the equation?
 Technically, u is known as the stochastic disturbance
or stochastic error term.
 It is a surrogate or proxy for all the omitted or
neglected variables that may affect Y but are not (or
cannot be) included in the regression model.
Sources of Errors in Regression
1. Unpredictable element of randomness in
human response
 If y = profitability of a company and x = sales and
expenditure distributions of the company. There is an
unpredictable element of randomness in each
company’s profitability.
 The company does not operate like a machine, the
sales and its expenditure distributions fluctuate from
time to time.
2. Effect of Omitted Variables
 In our example x is not the only variable influencing
y.
 The company risk, market size, economic structural
changes, etc can affect the variable y.
 The u is a catch ball for the effects of all the
variables, some of which may not even be
quantifiable and some of which may not even be
identifiable.
3. Measurement Error in y
 It refers to the measurement error in in the company
profitability.
 The argument is we cannot measure it accurately
 For now lets assume that there is measurement
error in y but not in x.
The Gauss-Markov Theorem
 Under assumptions, OLS is the Best Linear Unbiased
Estimator (BLUE) of the population parameters.
 Best = smallest variance
 It’s reassuring to know that, under assumptions, you
cannot find a better estimator than OLS.
 If one or several of these assumptions fail, OLS is no
longer BLUE
Regression Line

 We need also to draw a fitted regression line that

best fits the collection of (x-y) data points.
 A better procedure is to find the best straight line
using a criterion that minimises the sum of squared
distances (errors) from the points to the line as
measured by Y direction.
 It is possible to use the general equation for a straight
line to get the line that best ‘fits’ the data.
 The researcher would then be seeking to find the
values of the parameters or coefficients, α and β,
which would place the line as close as possible to all
of the data points taken together.
 The most common method used to fit a line to the
data is known as Ordinary Least Squares (OLS).
 Ordinary Least Squares (OLS) or linear least
squares is a method for estimating the unknown
parameters in a linear regression model, with the goal
of minimizing the sum of the squares of the
differences between the observed responses in the
given dataset and those predicted by a linear function
of a set of explanatory variables.
 This line is known as the least squares line or fitted
regression line.
DERIVING THE ORDINARY LEAST
SQUARES ESTIMATES
 Linear regression also known as linear least squares,
computes a line that best fit the observations.
 The method of least squares requires that we should
choose as estimates of

 Thus the predictions must be based on parameters-

estimated values and testing is based on estimated
values in relation to hypothesized population values.
 Linear regression parameters estimation
 of
•Linear regression estimates the values and ̂
using ̂ and estimated as follows.

ˆ
   
xi  x yi  y  and forˆ  y  ˆ x

 xi  x
2

Linear regression model generates a straight line which summarises the values of all data
points for corresponding values of y and x.

Yi   X i  i
 Linear regression model based OLS method
•The method used to fit the data into the straight line is called the Ordinary Least
Squares (OLS) method.

•OLS takes each vertical distance from each data point to the line, squares it and
minimise the total sum of the areas of the squares – hence the ordinary least squares.

•This is similar to minimising the sum of all the areas of the squares (distance) from each
of the data point to the line from either sides of it.

•The whole idea is to make total sum of

the area of the squares (distances) above the

line equal to total sum of the area of the
Squares (distances) below the line. Then the
Straight line represent the data.
 OLS error term
•Let yi denote the actual data point for one observation, then,
ŷi denotes the fitted value
for that observation on the regression line.

•In other words,ŷi is the value yfor

i which the regression model have predicted on the
fitted line.

i
• denotes the residual, the difference

between the actual value of Y and its fitted

( yi  yˆ i )
value on the line i.e. .
 i2
The whole idea is to minimise sum of so
i
that the total sum of approximates zero.
 In other words, for the given value of x of this
observation t, ˆyt is the value for y which the model
would have predicted.
 Note that a hat (ˆ) over a variable or parameter is
used to denote a value estimated by a model.
 Finally, let ˆut denote the residual, which is the
difference between the actual value of y and the
value fitted by the model for this data point -- i.e. (yt −
ˆyt).
 The distance of the data points from the fitted line is
the residual – which is the difference between the
actual value of the dependent variable and the
predicted value.
 The method minimises collectively the vertical
distances from the data points to the fitted line (y-y
hat).
 To guess is cheap. To guess wrongly is expensive -
Chinese proverb
 The reason that the sum of the squared distances is
minimised - finding the sum of ˆut that is as close to
zero as possible – some points will lie above the line
while others lie below it.
 When the sum to be made as close to zero as
possible is formed, the points above the line would
count as positive values, while those below would
count as negatives.
 Any fitted line that goes through the mean of the
observations (i.e. ¯x, ¯y) would set the sum of the ˆut
to zero.
Testing Validity of the Model
 In simple linear regression, the validity of the model is
tested by Coefficient of Determination
 If Coefficient of Determination is high, it shows the
model is very good.
 This is what is called the R-Squared.
 Coefficient of Determination
 The coefficient of determination measures the
percentage of variability in Y that can be explained
through knowledge of the variability in the independent
variable X -

 The more of the variance in Y you can explain, the more

powerful your model
 Calculate Coefficient of Determination using previous
example.
 Linear regression model assumptions
•There are five assumptions for the linear regression model.

Technical notation Interpretation

E ( i ) 0 The errors have a zero mean – no outliers.

var(i )  2   The variance of the mean is constant and finite over all values of xi - linearlity
cov(i ,  j ) 0 The errors are linearly independent of one another – no multicollinearity.

con( i , xi ) 0 There is no relationship between the errors and corresponding values

x – error independence
i
i ~ N (0,  )2
The errors are normally distributed – normality test.

•NB: one rule of thumb in linear regression is that sample size must be at least 15 cases
per one independent variable.
Testing & interpreting linear regression assumptions from SPSS output.

1. Linearity – The P-P plots show the distribution of data points close to the perfect line that
diagonal cuts through the square of the plots. Any points away from it suggest problems with
linearity.
2. Normality – the histogram must show the bell shaped curve of the normal distribution of the
residuals of the dependent variable.
3. Multicollinearity - Tests whether individual independent variables are highly related to one
another thereby representing the same effect on the dependent and thus inflate the results of
the model. It is tested via either the tolerance level or the Vector Inflation Factor (VIF) levels.
Tolerance value must be between 0 and 1, anything below 0.2 is unacceptable, above 0.2 but
less than 0.5 is moderate, up to 1 is very good. VIF is the inverse of the tolerance score. VIF
below 3 is great, above 3 but less than 5 is moderate, up to 10 is not a good sign while
anything above 10 is unacceptable.
4. Outliers – cases with outliers must be removed from the data before your final analysis and
such changes must be explained in your presentation. Tested using either the mahalanodis or
cooks distances. Cooks distance must be between 0 and 1, anything above 1 indicates
presence of outliers and must be removed or treated as missing.
5. Independence of errors – Durbin-Watson static tests autocorrelation in residuals. It ranges
between 0 and 4. Values between 1.5 and 2.5 mean non-autocorrelation, a value towards 0
indicates positive autocorrelation and a value towards 4 indicate negative autocorrelation.
Addressing normality problem - Transformation of quantitative data

When data do not meet all the normality assumption, it means linear regression cannot be
used and data has to be 1) transformed first to normalise it or bring it near normality or 2)
use nonparametric models.

• If mean = median = mode, the sample shows a perfectly normal distribution

• If mean < median < mode, the sample shows a negatively skewed distribution
• If mean > median > mode, the sample shows a positively skewed distribution

Data can be transformed in three ways; logarithm, square root or the reciprocal of the
Addressing outlier, multicollinearity and linearity problems – By elimination

1. Outliers – Tested using either the mahalanodis or cooks distances. Cooks distance
must be between 0 and 1, anything above 1 indicates presence of outliers. Cases
with values more than 1 must be removed or treated as missing data.

2. Linearity – The P-P plots must show the distribution of data points close to the
perfect line that diagonal cuts through the square of the plots. Any points away
from it suggest problems with linearity and must be excluded.

3. Multicollinearity - It is tested via either the tolerance level or the Vector Inflation
Factor (VIF) levels. Tolerance value must be between 0 and 1, anything below 0.2 is
unacceptable, above 0.2 but less than 0.5 is moderate, up close to 1 is very good.
VIF is the inverse of the tolerance score. VIF below 3 is great, above 3 but less than
5 is moderate, up to 10 is not a good sign while anything above 10 is unacceptable.
When two independent variables are highly correlated, one must be
removed from the analysis.

NB: Whatever you do as part of data cleaning must be explained in your

methodology section and the testing results must be presented as well.
 Example of a multiple linear regression
A mortgage finance institution is facing a number of cases relating to differences between the true
selling price on defaulted house loans and the value of the mortgage at purchase price resulting in
foreclosure loses of repossessed houses. To address this problem, finance specialists in the
mortgage division decided to model the true selling prices using historical information from their
past clients in their data base on property characteristics. The information they have is on:
1. current value of the property (denoted as “price”),
2. Size of the house in square metres (denoted as “size”)
3. Size of the land area of the property in square metres (denoted as “Land size”)
4. Age of the property in years (denoted as “age’).

They want to model or produce an unbiased estimate of the current price for any property in future
based on its size, land size and its age, thus Price is the DV while three IVs are size, land size
and age of property. Since all these variables are continuous, multiple linear regression is the most
appropriate method. Based on data is in the SPSS file, the aim of this exercise is to:
5. Test assumptions of multiple linear regression
6. Apply the model.
7. Present and interpret the results.
 Presenting and interpreting results
• Results section of your project/dissertation/thesis/research paper must cover the following:
1. Descriptive statistics
• Before presenting your main results, it is important first to understand your data. So, present simple
descriptive statistics of the data as indicator of the expected results as well as the nature and scope of your
data.

• It is important to understand your data.

• As you discuss the descriptive statistics, comment on the following.

i. Common trends outlined on the data
ii. Categories with most and least influence on the dependent.
iii. The relationship between these results and theory.– explain the main trends and main facts of the data.

2. Assumptions diagnosis – discuss all the relevant assumptions tests and provide statistics
showing how they are met.
3. Results – present results only, do not interpret.
4. Discussion of results – interpret the results and discuss variance or agreement with
literature or theory.
 Academic writing structure…
Chapter Four: DATA ANALYSIS AND INTERPRETATION OF RESULTS
Introduction: introduce and state the main methods used and the main classes of the results presented
Main Body: must cover.
• Descriptive statistics – state the results and explain what they mean in general and in relation to literature.
• Diagnostic tests (explain how good and significant the tests are, if they applicable to your research).
• Main results - Present main results objective by objective (if quantitative, state statistical significance of
each result and if qualitative support by the examples give verbatim by respondents or frequency of the
responses)

• Results discussion - interpret the results and show how they link to literature and theory.
• Chapter Summary: summarise main results and introduce the next chapter

NB: All headings and subheadings must clearly reflect what is presented in the results or what the results are
all about. This is important for the reader to easily link the results to the particular problem or objective of the
study addressed by those findings. Headings must be informative enough to the reader in all chapters.

Example 1.
Poor heading : Descriptive statistics
Good heading: Summary Statistics of factors affecting the price of properties.

Example 2:
Poor heading: linear regression results.
Good heading: The effects of property characteristics on price of properties.
 Factors affecting prices of houses
4.0 Introduction
This section presents results on factors that influence prices of properties. The data
consists of 65 properties in selected suburbs in Bulawayo and capture the price of
property, its age, size of the land where located and the size of the house itself. The
results were analysed using multiple linear regression model. Three sets of results are
presented; first are the descriptive statistics describing the nature and scope of the data,
followed by tests on assumptions of multiple linear regression model and the main
results. The last section discusses the results.

4.1 Descriptive statistics on property prices and factors affecting price

It is important to understand the pattern of the sample data in descriptive terms and the
extent to which these descriptions are in conformity with literature. For that purpose,
descriptive statistics on nature and scope of the prices of houses and each of the
independent variables are presented in the following sections.
An assessment of the prices of houses by the different characteristics of properties
shows that prices vary significantly as a results of these factors (See Table 1). Houses
sell on average for $82 000 with some standard deviation of $41 000. These prices vary
depending on the size of the house, its age and the size of the land area where it is
located and these average 188 square metres, 8 years in age and 1 550 square metres
in size respectively. These variations are somewhat not surprising because similar
evidences are also well documented in some of the literature (Nkuah, Tanyeh and Kala
2013). Literature reveals that prices of houses are positively related to these physical
characteristics of the house.
Table 1: Descriptive statistics on factors affecting property
prices
Mean Std. Deviation N

Selling price ($000) 82.18 41.322 65

House size in square meters 1550.43 1219.102 65

Age in years 7.71 3.991 65

Land size in square meters 187.78 63.385 65

4.3 Multiple regression assumption tests

The regression model as whole is correctly specified and fits the data well as the F-
statistics is significant (Table 2). In addition, the adjusted R square indicates that about
82% of the variations in house prices is explained by the changes in the three selected
independent variables (Table 3) which gives the model a high explanatory power. The
model does not seem to have any multicollinearity problems as evidenced by the low
values of the VIF. Residual do not suffer from autocorrelation given the moderate value
of the Durbin Watson statistic. Finally, the data reflects the presence of normality of the
residuals of the dependent variable with a linear characteristic (See Appendices 1 and
2) and with no case with cooks distance greater than 1, means there are no outliers
observed in the data.
Table 2: ANOVA Table 3: Model Summary
Model Sum of df Mean F Sig. Mode R R Adjusted Std. Error Durbin-
Squares Square l Squar R Square of the Watson
Regressio 89695.40 3 29898.4 93.13 .000b e Estimate
n 7
1 Residual 19584.38 61 321.06 .906a .821 .812 17.918 1.682
1
109279.79 64
Total

b. Predictors: (Constant), Land size in square meters, Age in

years, House size in square meters
Appendix 1: Normality of the residual of house prices Appendix 2: Linearity of
house prices
4.4 Factors affecting prices of houses

The results show that only the size of the house and land size where the house is
located are important factors affecting price of a house. However, the age of the house
seems not to affect prices of houses. Based on this results, it means that prices of
houses can only be modelled using land size and house size only (Table 4).

Table 4: Factors affecting house prices

Variables Coefficients Tolerance VIF

House size in square meters 0.162*** 0.454 2.201

Age in years 0.071 0.990 1.010

Land size in square meters 0.248*** 0.454 2.203

* (**) [***] = Significant at 10% (5%) and [1%]

As an example, if a house is located in a 300 square meters area and its built-up area is
90 square meters in size, its price would approximately be $50 760.00
(300x0.162)+(90x0.024).
Notes on interpreting and presentation of linear regression results from SPSS
output.

1. Model fit – results from regression must be used only if the data fit in the model well. The
significance of the model fit must have p-value of less than 0.05. You find this from the ANOVA
table. If this condition is satisfied, we conclude that the model fits the data well and we can go
ahead and interpret coefficients of independent variables. Another way to test the goodness of the
model is to look at the R squared. This tells us how much of the changes in the dependent variable
are a result of/explained by the selected independent variables. The higher is this value the better
is the model. This is useful especially where stepwise method is used. R Squared is used in simple
linear regression while adjusted R Squared is preferred in multiple linear regression.

2. Coefficients of the independent variables are presented in two ways.

a) Unstandardised coefficients – as the name indicates these are the true coefficients in their
ordinary measurements. They estimate the outcome of the dependent variable based on actual
values of the independent variables in question. These are the coefficients put in the equation
to estimate or model the value of the dependent variable given any values of the independent
variables.

b) Standardised coefficients – in this case, coefficients of different independent variables are

converted into a uniform scale measure so that you can easily compare which one is superior in
influencing changes in the dependent variable. The independent variable with the highest value
is the most influential one and the one with the least value has the least contribution. If you
have many independent variables, you can use this method to remove independent variables
Hierarchical regression modelling in SPSS
There are two methods to do this – aim to select the most important IVs that affect
the DV.
1. Step wise methods.
• When running regression, you click on Statistics.
• Then click on method which gives you options
a) Forwards – start with only the IV with the most contribution (R Squared) to the overall model
in the first model, and then add the next most contribution IVs one by one in subsequent
models until all IVs are added. So, changes in R squares will be decreasing from model to
model.
b) Backward - start with all IVs and then remove IVs with the least contribution (R Squared) to
the overall model and then only remain with the most important IVs.
c) Stepwise – A combination of both forward and backward methods. Starts like a forward
regression and then remove the least important the SPSS runs several regressions each
with different number of IVs from the highest to the lowest by order of importance. I.e. if
you have 3 independent variables in your analysis, it will produce three sets of results, the
results for all the three, the best two and the best of one. you then decide which one you
want to use by looking at all three.

2. Block method
a) In this case you enter independent variables in their box in sets of selected independent
variables, each as a separate block in ascending or descending order. The aim is to compare the
R squared and significance levels for the different blocks to identify which block gives a better