The Coefficient Stability Test

Rus’an Nasrudin

Sep 28, 2021

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 1 / 21

1 The Oster test

2 Theory and Implementation

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 2 / 21

 The Oster test

Table of Contents

1 The Oster test

2 Theory and Implementation

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 3 / 21

 The Oster test

What we do in testing the selection-on-observables?

First is to report the coefficient stability as we see in the example

using Table 3.2.1 above.
Claiming causal effect after inclusion of covariates, however, is not
always enough.
The reason was, one can raise a question whether the true
relationship between endogenous regressor and the unobservable(s)
can be recovered fully by the relationship between the endogenous
and the included observables?

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 4 / 21

 The Oster test

Oster (2017) is the first to address the issue, and to provide a formal
test (report) about the unobservable selection and coefficient stability.

Emily Oster
Unobservable Selection and Coefficient Stability: Theory and Evidence

Journal of Business and Economic Statistics 37(2): 187-204 (2019)

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 5 / 21

 The Oster test

Goodness of fit

Before we proceed, recall that regression, or in particular OLS estimate

divides a value into two:
yi = ŷi + ε̂i (1)
Define that the total sum squares (SST), the explained sum of squares
(SSE) and the residual sum of squares (SSR) as follows:
n
X
SST ≡ (yi − ȳi )2
i
n
X
SSE ≡ (ŷi − ȳi )2
i
2
SSR ≡ ε̂i

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 6 / 21

 The Oster test

Interpretation

SST is a measure of the total sample variation in the yi , that is how

spread they are in the sample. If we divide it with n − 1 we get
sample variance.
SSE is a measure of the sample variation in ŷi , and
SSR is a measure of variation in ε̂i .
The total variation identity is

SST = SSE + SSR

(we can easily proof this right?)

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 7 / 21

 The Oster test

R-squared

From here, we have a way of measuring how well the explanatory or

independent variable(s) explain the dependent variable. The ratio of the
explained variation compared to the total variation is:

R2 ≡ SSE/SST = 1 − SSR/SST

We know R2 as “R-squared”.

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 8 / 21

 The Oster test

What is ’Oster test’ about?

Many economics articles use coefficient stability on endogenous

regressor to argue for a causal treatment effect in the presence of
imperfect controls.
Oster (2017) argues that this approach is not perfect for limiting the
bias, especially when we have a low power control variable(s)
(relevant but weak).
Coefficient movement is only meaningful when the R-squared also
moves significantly, if not an omitted variable bias remains.
What to do? we need to report the bounds of the coefficient for the
true treatment effect on certain assumptions on what we know about
the maximum R-squared and δ.
δ is the relative degree of selection on observed and unobserved
variable.

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 9 / 21

 The Oster test

Illustrative example of bias from imperfect control

Assume that the model that determines wage is given by:

Y = βS + W + C
Y is wage, S is education, W and C are orthogonal components of
ability which relate to S in the same way.
Assume that variance of W is larger than C.

Quality of observed control Without control With control

High variance 0.202 [0.004] 0.002 [0.990]
Low variance 0.202 [0.004] 0.200 [0.013]

Note: assume that true β = 0. Numbers are β̂ and R2 in bracket.

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 10 / 21

 The Oster test

Key feature of what so called “Oster-test”

The quality of the included covariate(s) is diagnosed by how much of

the variance in the outcome is explained by its inclusion or how much
the R2 moves when they are introduced.
Omitted variable bias is proportion to coefficient movements, only if
such movements are scaled by the change in R2 when the covariates
are included.

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 11 / 21

 Theory and Implementation

Table of Contents

1 The Oster test

2 Theory and Implementation

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 12 / 21

 Theory and Implementation

General setup and definitions

Consider the regression model

Y = βX + Ψω ◦ + W2 + 

Where X is the (scalar) treatment, ω ◦ is a vector of the observed control,

W2 is unobserved. Define that W1 = Ψω ◦ and each element of ω ◦ are
orthogonal to each other.

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 13 / 21

 Theory and Implementation

Proportional selection relationship

We define the proportional selection relationship among observables and

unobservables as:
σ1X σ2X
δ 2 = 2
σ1 σ2
Where σiX = cov(Wi , X) and σi2 = var(Wi ) for i ∈ {1, 2} and δ is the
coefficient of proportionality. δ = 1 implies an equal selection relationship,
i.e. unobservable and observables are equally related to the treatment.

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 14 / 21

 Theory and Implementation

The auxiliary regressions

Suppose (denote R2 as R):

From Y ∼ X we get β̇ and Ṙ
From Y ∼ X, W1 , we get β̄ and R̄
and from a hypothetical regression of Y ∼ X, W1 , W2 we have Rmax
And denote that
λ̂ω◦ |X is the coefficient of regression of each ω ◦ on X.
λ̂W2 |X is the coefficient of regression of the unobservable of W2 on X
(hypothetical).
λ̂W2 |X,ω◦ is the coefficient of regression of W2 on X and ω ◦
The population analog of these coefficients are the ones without hat.

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 15 / 21

 Theory and Implementation

The probability limits of βs

Using a standard omitted variable bias formula, we have

J
X
β̇ →
−b β + Ψ◦i λωi◦ |X + λW2 |X
i=1
β̄ →
−b β + λW2 |X,ω◦

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 16 / 21

 Theory and Implementation

The approximation

The asymptotic bias on b̂eta is λW2 |X,ω◦ .

Under the assumption of equal proportion for all element coefficient on ω ◦ ,
β̂ can be recovered from regression of Y on X and W1 .
Given this, the bias is equal to

σ22 σ1X
Π= 2
σ1X
σ12 (σX
2 −
σ12
)

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 17 / 21

 Theory and Implementation

The approximation..

Theorem 2.1 (Approximation of β ∗ )

 Rmax − R̄
β ∗ ≈ β̄ − δ β̇ − β̄

(2)
R̄ − Ṙ
β ∗ is an approximation of the bias-adjusted treatment effect. With
Rmax and δ are external information.

See the proof in section ’3.2 Restricter Estimator’ of Oster (2019).

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 18 / 21

 Theory and Implementation

Coefficient stability reporting

Implementation
Set both Rmax and δ to report sets of bounds for β, that is set of
values of β ∗ .
I Oster (2017) provides rule of thumbs: (1) Rmax = 1.3R̄. 90% of
randomised article survives and only 45% non-randomised articles
survives. And,
I An average δ from a ’constructed’ data gives value of 0.545 which
would be appropriate in many settings.
Report the value of δ with the assumption of Rmax and β = 0
Report Ṙ to R̄ movements.

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 19 / 21

 Theory and Implementation

Stata Example
We can implement the coefficient stability reporting, calculate δ and β ∗
given our assumption about Rmax with Stata command psacalc (Oster,
2013). It is available from ssc.
eststo eq1a: reg lwage educ
estat ovtest
local ov0=r(p)
eststo eq1: reg lwage educ exper expersq
local R2max1=e(r2)*1.3
estat ovtest
local ov1=r(p)
psacalc delta educ, rmax(‘R2max1’) beta(0)
local A=r(delta)
psacalc beta educ, delta(0.545) rmax(‘R2max1’)
estadd scalar Treatment_effect ‘r(beta)’: eq1
estadd scalar delta ‘A’: eq1
estadd scalar OVtest ‘ov1’ : eq1

etc..

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 20 / 21

 Theory and Implementation

Stata example..

Dependent var: log(wage)

(no control) (+ind) (+parent) (+husband) (+location)
edu 0.109*** 0.107*** 0.120*** 0.128*** 0.128***
(0.014) (0.014) (0.016) (0.019) (0.019)
Ṙ, R̄ 0.118 0.157 0.163 0.164 0.166
β∗ 0.106 0.128 0.150 0.148
δ 16.443 2.053 1.026 1.050
OVtest (p-value) 0.030 0.348 0.431 0.401 0.418

Rus’an Nasrudin The Coefficient Stability Test Sep 28, 2021 21 / 21