Lecture 4

Methods of estimation

-
Method of Moments
-

Maximum Likelihood
-

M-estimators

Assymptotic properties
-
 Summary of Yesterdayind parametrica
Po
-
Xe

(Po(dx)
·
Xn

(b(x)((dx)
, ...,

·
(X , F ,
(P0)oe dominating *
measure

In (0 ;
·
X1 ... Xn) = Co(Xi) - Likelihood
=C
Score
·
Sn (8; X1, ...,
No fogLn10 ; X1 Fisher
Xn) =
Xn) , . .

>
-
·
In (8) =
Covo (Sn(8 ; X1, Xn)) Inform ..., .

Et =

g(t) => Var(F) gll()Tog(t)

b >
-
Unbiased estimator under assumptions Cramer-Rao
 All this works not
only ind
in model

In general : we observe ZEZ

e g
. .

z
E
=
= xxxx xx
(X1 , Xn)
-- ,
...

[
Stat model : (E ,
F
, (Q)be
Now Qo is the distribution of z
and not of each individual Xi !
Likelihood : 2(0 ; 2) bo(z)
=

where lo(z)y(dz) = Qu(dz)

[Everything else is the same ! ]
Example : Assume that we observe

z = ((Xz Y1) (Xn Yn)

Fixed
design
... Linear
regression
, ,

where : YER
,
X : E IRP

Yi =
BXi + Ei with Ed N(a
Let's suppose that CXDE and
B are

some fixed vectors

Then
,
the only E
unknow
parameter isCBEIRP
~

= f )3
(Yi BYx
2n(p ; z)
-

exp 2
Method of Moments

Let X- ,
An ~ Ro
...,

1) .
Pick T1
, ...,
Ta : X -R
f
Compute by hand ej(t) To [Tj(X)]
2) Solve the
following system
; Tj((i) =
ej(t)
and obtain as this solution
Intuition

=Fo[t(n)andT (
Since ej(d)
by LLN
I=
j
;

KJVGE
1

Then since

(n! C
(Ye) ej(y)
-

I =

expect a
-j
could
we

ej(t)
Thus if the
, mapping

0 -(b , (d)
,
- -

, (b)
to one the we could
is one ,

hope that - o

[One to one is
importanthave
, because
otherwise we could had

(e1(0) , ..., ea(8)) = (e+ (61), ... Ca(81)]

[To be one-to-one reed
we
a such funcy
where d = dim(k)
Example : X , . ... Xn ~exp(x) E
- EM
+

E[X1 =; #x[XY] = 2

=
T(x) = x

= ↓ Xi =
* =
= X :

-
T() = x

=>
Xi =
*
* ↑
= x
 i N(p 5)
Example : X1, ..
,
Xn ,
=

(M 2)
,

# o[Xi =

M , Eo[Xi] =
M2
+ G2
2
Tr(x) = x
,
Tr(x) = x

E -EriXi 2) = iX -=M X :

E =

(y , 2)
 Assymptotic normality (waving hands, after knowing
some math)
Say - FR and we use T(x) = xc then
,

if e(o) =
Eo[T(Xe)] is invertible
,
there exists h sit hoe-Coh =
Identity
let In =
EX :
and
M
= lo[X1]

h(kn)
=>
E
=

O =
h(M)
Delta method would tell us that

in ( -
) = in (h(n) h(y)) 0)V /0, [h(m)] )
-

Varf(X1)
=
where r =
So we have shown that

in (f -0) =0 NIo , [h(y)) -2)

CLT like theorems
Actually most
derived
are via that
type
of argument
Maximum Likelihood Estimator
Observation Z in (E E)
,

Model (Q) of

MLE 2 (0 e
argmax
E
Fundamental fact
X-logx isincreasing
a

This ,

MLE Earg max log 2(oiz)

E ④
or

qMLE eargmin[-log2(fiz)]
Important
I simplification iid
in model

In (0 ; X1 -Xn) lo(Xi)
,
=
Then iid
in model
,

eMLE
#
eargmin
· E
E- +N logfo(xi)
,

Why did I add hf

-

Hilogfo(xi) -Eo[loglo(X2)
 Let's return to the linear
regression
Yi =
B Xi + E:
,
E: N(o e) ,

Xi -

fixed known

B
-
fixed unknown

In (B ; (4z Xc)
,
.

.,
(2 , Xa) = exp/
"2
=

() exp C-5 (Y : -

XB

it i
2

-Flog In (p ; (Y1X1), ..,

(niXa) = (Yi xiB
-

+ Constant .
Thus in this model

1 MLE

B Earg min
P
↓ (Yi xi
-

B >
-

least-squares
1 MLE

the estimator !
P is

Question for homework : In the same

model
,
what is MLE
is Ei it Laplacell)
Density of Laplace is f(x) = <exp f- ( ) *
Projperties of MLE

Det We that estimator E

:
say an

is weakly consistent if
P
-

Theorem Under certain technical tions

:
assual
↑

- (see Lecture notes)

Mc is
weakly consistent
Question : Eo[qMIty -O O ?
-

t P
In general no
,
since QMLE
>-

* -
+
but ,
if we can ensure that #I +
O
then
yes (remember our exercise)
MEC E if
In
particular ,
since ,

④ then III
is
compact ,
EK

Under these
type of Conditions
MLE is
asymptotically unbiased
What about ?
a
symptotic variance

We need CLT !

rom : Under certain technical conditions

(EMLE -O) 0 N(0

[11(0)] )
=

↑
Fisher information
again
(Dingue ,
non )
?
Ideas of the
proof (Why the hell
Fisher inf
?)
Again this not the formal proof, pops up

but given your expertise

in measure
theory
,
I believe in !
you
- = R
/
= MLE (S)(b ; x) = log(b(x)
Eargmax[tE log Co(Xi ,

=>
↓ ! S1 ( , xi) I Sm (8 =
,
X1 ..,

-
f() =
f(0) + f'()( 0
for some between and
 Hence - f(8) = 0

M(E (f() f(t)

0)
o
-

-
=
I
- I
f()
Since
f (0) = ↓ Sn(0 ; X
;X)/n
n( -
8) =
-
-

Numerator's
8 Sn(EiX)/
story
Recall that Eo[Sn/fiX)] =
0 Un-#[S1/8'X]O

In Sn10 , X) =
in (t S= (fixi) -

Eo[Silt ; xi))) 0LCLT

NI0 VardS1(x))
,
Demenator's story

· Sn(8, X) =
S( ; xi)

By weak
consistency
***
Moreover NOEQ
<
* Eo[S1/t": X)
t Se(fixi)
LLN
by
**
=>
t S, (iXi) (S1/0 ; X1)]
Continuous mapping - Slutsky tell us

-
in (8-8) >
- -

Eo[oS1 (0 ; X)]
·
Clo, Varo(s (ix)
-

last
from Lecture
But
(oix)] =-I1(f)
E
Eo[
Pl
S,

In(0) = Vare(Si(f ; Xe)

=>

)
- E( -

0) - ((0 :
What this result tell us

vender some conditions

#[MLE
E
- O

ind to N(0 ,
(a)
Cramer-Rao : Est] =

=> Varo() n le
efficien !
MLE is
asymptotically
That's it ? I finished Math .
Stat ?

everythingis
Is it a science where
own ?

: a symptotic properties
are nice

but in reality we have a fixed

sample of
n data -o modern
stats
 large
A family of estimators

M-estimators

Def :
Let 4 : X-R

Er Eargmax 24(0 ; X )
of

dof learning
and butter machine
Which
If to ? No only
pick answer
,

rules of thumbs
and experience

But certainly want

, we

· -
argmax
Eo[ 4/0 ; x1)])
of

as it is the case for 410; X1) =

Loglo()
rd
I that 3 port of the
suspect in
I hear more abt it
will
course you
e
o

e
-

.
=