Introduction to Probability

Lecture 10: Estimators (Part I)

Mateja Jamnik, Thomas Sauerwald

University of Cambridge, Department of Computer Science and Technology

email: {mateja.jamnik,thomas.sauerwald}@cl.cam.ac.uk

Easter 2024
Outline

Introduction

Defining and Analysing Estimators

More Examples

Intro to Probability Introduction 2

Introduction

Setting: We can take random samples in the form of i.i.d. random vari-
ables X1 , X2 , . . . , Xn from an unknown distribution.

Taking enough samples allows us to estimate the mean (WLLN, CLT)

Using indicator variables, we can estimate P [ X ≤ a ] for any a ∈ R
in principle we can reconstruct the entire distribution

Intro to Probability Introduction 3

Introduction

Setting: We can take random samples in the form of i.i.d. random vari-
ables X1 , X2 , . . . , Xn from an unknown distribution.

Taking enough samples allows us to estimate the mean (WLLN, CLT)

Using indicator variables, we can estimate P [ X ≤ a ] for any a ∈ R
in principle we can reconstruct the entire distribution

How can we estimate the variance or other parameters?

estimator
How can we measure the accuracy of an estimator?
bias (this lecture) and mean-squared error (next lecture)

Intro to Probability Introduction 3

Introduction

Setting: We can take random samples in the form of i.i.d. random vari-
ables X1 , X2 , . . . , Xn from an unknown distribution.

Taking enough samples allows us to estimate the mean (WLLN, CLT)

Using indicator variables, we can estimate P [ X ≤ a ] for any a ∈ R
in principle we can reconstruct the entire distribution

How can we estimate the variance or other parameters?

estimator
How can we measure the accuracy of an estimator?
bias (this lecture) and mean-squared error (next lecture)

Physical Experiments:
Measurement = Quantity of Interest + Measurement Error

Intro to Probability Introduction 3

Introduction

Setting: We can take random samples in the form of i.i.d. random vari-
ables X1 , X2 , . . . , Xn from an unknown distribution.

Taking enough samples allows us to estimate the mean (WLLN, CLT)

Using indicator variables, we can estimate P [ X ≤ a ] for any a ∈ R
in principle we can reconstruct the entire distribution

How can we estimate the variance or other parameters?

estimator
How can we measure the accuracy of an estimator?
bias (this lecture) and mean-squared error (next lecture)
expe
ctatio
n
Physical Experiments:
Measurement = Quantity of Interest + Measurement Error

Intro to Probability Introduction 3

Introduction

Setting: We can take random samples in the form of i.i.d. random vari-
ables X1 , X2 , . . . , Xn from an unknown distribution.

Taking enough samples allows us to estimate the mean (WLLN, CLT)

Using indicator variables, we can estimate P [ X ≤ a ] for any a ∈ R
in principle we can reconstruct the entire distribution

How can we estimate the variance or other parameters?

estimator
How can we measure the accuracy of an estimator?
bias (this lecture) and mean-squared error (next lecture)
expe
ctatio variance
n
Physical Experiments:
Measurement = Quantity of Interest + Measurement Error

Intro to Probability Introduction 3

Introduction

Setting: We can take random samples in the form of i.i.d. random vari-
ables X1 , X2 , . . . , Xn from an unknown distribution.

Taking enough samples allows us to estimate the mean (WLLN, CLT)

Using indicator variables, we can estimate P [ X ≤ a ] for any a ∈ R
in principle we can reconstruct the entire distribution

How can we estimate the variance or other parameters?

estimator
How can we measure the accuracy of an estimator?
bias (this lecture) and mean-squared error (next lecture)
expe
ctatio variance
n
Physical Experiments:
Measurement = Quantity of Interest + Measurement Error

Intro to Probability Introduction 3

Empirical Distribution Functions
Definition of Empirical Distribution Function (Empirical CDF)
Let X1 , X2 , . . . , Xn be i.i.d. samples, and F be the corresponding distribution
function. For any a ∈ R, define
number of Xi ∈ (−∞, a]
Fn (a) := .
n

Intro to Probability Introduction 4

Empirical Distribution Functions
Definition of Empirical Distribution Function (Empirical CDF)
Let X1 , X2 , . . . , Xn be i.i.d. samples, and F be the corresponding distribution
function. For any a ∈ R, define
number of Xi ∈ (−∞, a]
Fn (a) := .
n

Remark
The Weak Law of Large Numbers implies that for any  > 0 and a ∈ R,

lim P [ |Fn (a) − F (a)| >  ] = 0.

n→∞

Intro to Probability Introduction 4

Empirical Distribution Functions
Definition of Empirical Distribution Function (Empirical CDF)
Let X1 , X2 , . . . , Xn be i.i.d. samples, and F be the corresponding distribution
function. For any a ∈ R, define
number of Xi ∈ (−∞, a]
Fn (a) := .
n

Remark
The Weak Law of Large Numbers implies that for any  > 0 and a ∈ R,

lim P [ |Fn (a) − F (a)| >  ] = 0.

n→∞

Thus by taking enough samples, we can estimate the

entire distribution (including its expectation and variance).

Intro to Probability Introduction 4

Empirical Distribution Functions (Example 1/2)

Example 1
Consider throwing an unbiased dice 8 times, and let the realisation be:

(x1 , x2 , . . . , x8 ) = (4, 1, 4, 3, 1, 6, 4, 1).

What is the Empirical Distribution Function F8 (a)?

Answer

Intro to Probability Introduction 5

Empirical Distribution Functions (Example 1/2)

Example 1
Consider throwing an unbiased dice 8 times, and let the realisation be:

(x1 , x2 , . . . , x8 ) = (4, 1, 4, 3, 1, 6, 4, 1).

What is the Empirical Distribution Function F8 (a)?

Answer

1
7/8
6/8
5/8
F8 (a)

4/8
3/8
2/8
1/8
0 a
1 2 3 4 5 6

Intro to Probability Introduction 5

Empirical Distribution Functions (Example 1/2)

Example 1
Consider throwing an unbiased dice 8 times, and let the realisation be:

(x1 , x2 , . . . , x8 ) = (4, 1, 4, 3, 1, 6, 4, 1).

What is the Empirical Distribution Function F8 (a)?

Answer

1
7/8 5/6
6/8
5/8 4/6
F8 (a)
F (a)

4/8 3/6
3/8 2/6
2/8
1/8 1/6
0 a
1 2 3 4 5 6

Intro to Probability Introduction 5

Empirical Distribution Functions (Example 2/2)

Source: Modern Introduction to Statistics

Figure: Empirical Distribution Functions of samples from a Normal Distribution N (5, 4)

(n = 20 left, n = 200 right)

Intro to Probability Introduction 6

An Example of an Estimation Problem
Scenario
Consider the packages arriving at a network server.

Source: Wikipedia

Intro to Probability Introduction 7

An Example of an Estimation Problem
Scenario
Consider the packages arriving at a network server.
We might be interested in:

Source: Wikipedia

Intro to Probability Introduction 7

An Example of an Estimation Problem
Scenario
Consider the packages arriving at a network server.
We might be interested in:
1. number of packets that arrive within a “typical” minute

Source: Wikipedia

Intro to Probability Introduction 7

An Example of an Estimation Problem
Scenario
Consider the packages arriving at a network server.
We might be interested in:
1. number of packets that arrive within a “typical” minute

minutes
0 1 2 3 4 5 6

Source: Wikipedia

Intro to Probability Introduction 7

An Example of an Estimation Problem
Scenario
Consider the packages arriving at a network server.
We might be interested in:
1. number of packets that arrive within a “typical” minute

minutes
0 1 2 3 4 5 6

Source: Wikipedia

Intro to Probability Introduction 7

An Example of an Estimation Problem
Scenario
Consider the packages arriving at a network server.
We might be interested in:
1. number of packets that arrive within a “typical” minute
2. percentage of minutes during which no packets arrive

minutes
0 1 2 3 4 5 6

Source: Wikipedia

Intro to Probability Introduction 7

An Example of an Estimation Problem
Scenario
Consider the packages arriving at a network server.
We might be interested in:
1. number of packets that arrive within a “typical” minute
2. percentage of minutes during which no packets arrive

minutes
0 1 2 3 4 5 6

Source: Wikipedia

Intro to Probability Introduction 7

An Example of an Estimation Problem
Scenario
Consider the packages arriving at a network server.
We might be interested in:
1. number of packets that arrive within a “typical” minute
2. percentage of minutes during which no packets arrive

minutes
0 1 2 3 4 5 6

Source: Wikipedia

Intro to Probability Introduction 7

An Example of an Estimation Problem
Scenario
Consider the packages arriving at a network server.
We might be interested in:
1. number of packets that arrive within a “typical” minute
2. percentage of minutes during which no packets arrive
If arrivals occur at random time number of arrivals during one
minute follows a Poisson distribution with unknown parameter λ

minutes
0 1 2 3 4 5 6

Source: Wikipedia

Intro to Probability Introduction 7

An Example of an Estimation Problem
Scenario
Consider the packages arriving at a network server.
We might be interested in:
1. number of packets that arrive within a “typical” minute
2. percentage of minutes during which no packets arrive
If arrivals occur at random time number of arrivals during one
minute follows a Poisson distribution with unknown parameter λ

minutes
0 1 2 3 4 5 6

X ∼ Poi(λ)

Source: Wikipedia

Intro to Probability Introduction 7

An Example of an Estimation Problem
Scenario
Consider the packages arriving at a network server.
We might be interested in:
1. number of packets that arrive within a “typical” minute
2. percentage of minutes during which no packets arrive
If arrivals occur at random time number of arrivals during one
minute follows a Poisson distribution with unknown parameter λ

minutes
0 1 2 3 4 5 6

X ∼ Poi(λ)

λk
P [ X = k ] = e−λ ·
Source: Wikipedia
k!

Intro to Probability Introduction 7

An Example of an Estimation Problem
Scenario
Consider the packages arriving at a network server.
We might be interested in:
1. number of packets that arrive within a “typical” minute
2. percentage of minutes during which no packets arrive
If arrivals occur at random time number of arrivals during one
minute follows a Poisson distribution with unknown parameter λ

minutes
0 1 2 3 4 5 6

X ∼ Poi(λ)

λk
P [ X = k ] = e−λ ·
Source: Wikipedia
k!

Intro to Probability Introduction 7

An Example of an Estimation Problem
Scenario
Consider the packages arriving at a network server.
We might be interested in:
1. number of packets that arrive within a “typical” minute
2. percentage of minutes during which no packets arrive
If arrivals occur at random time number of arrivals during one
minute follows a Poisson distribution with unknown parameter λ

∆ ∼ Exp(λ)

minutes
0 1 2 3 4 5 6

X ∼ Poi(λ)

λk
P [ X = k ] = e−λ ·
Source: Wikipedia
k!

Intro to Probability Introduction 7

An Example of an Estimation Problem
Scenario
Consider the packages arriving at a network server.
We might be interested in:
1. number of packets that arrive within a “typical” minute
2. percentage of minutes during which no packets arrive
If arrivals occur at random time number of arrivals during one
minute follows a Poisson distribution with unknown parameter λ

Waiting Time (Lecture 5, Slide 22)

∆ ∼ Exp(λ)

minutes
0 1 2 3 4 5 6

X ∼ Poi(λ)

λk
P [ X = k ] = e−λ ·
Source: Wikipedia
k!

Intro to Probability Introduction 7

An Example of an Estimation Problem
Scenario
Consider the packages arriving at a network server.
We might be interested in: Estimator for λ
1. number of packets that arrive within a “typical” minute
2. percentage of minutes during which no packets arrive
If arrivals occur at random time number of arrivals during one
minute follows a Poisson distribution with unknown parameter λ

Waiting Time (Lecture 5, Slide 22)

∆ ∼ Exp(λ)

minutes
0 1 2 3 4 5 6

X ∼ Poi(λ)

λk
P [ X = k ] = e−λ ·
Source: Wikipedia
k!

Intro to Probability Introduction 7

An Example of an Estimation Problem
Scenario
Consider the packages arriving at a network server.
We might be interested in: Estimator for λ
1. number of packets that arrive within a “typical” minute
2. percentage of minutes during which no packets arrive Estimator for e−λ
If arrivals occur at random time number of arrivals during one
minute follows a Poisson distribution with unknown parameter λ

Waiting Time (Lecture 5, Slide 22)

∆ ∼ Exp(λ)

minutes
0 1 2 3 4 5 6

X ∼ Poi(λ)

λk
P [ X = k ] = e−λ ·
Source: Wikipedia
k!

Intro to Probability Introduction 7

Estimator

Definition of Estimator
An estimate is a value t that only depends on the dataset x1 , x2 , . . . , xn , i.e.,

t = h(x1 , x2 , . . . , xn ).

Intro to Probability Introduction 8

Estimator

Definition of Estimator
An estimate is a value t that only depends on the dataset x1 , x2 , . . . , xn , i.e.,

t = h(x1 , x2 , . . . , xn ).

Then t is a realisation of the random variable

T = h(X1 , X2 , . . . , Xn ),

which is called estimator.

Intro to Probability Introduction 8

Estimator

Definition of Estimator
An estimate is a value t that only depends on the dataset x1 , x2 , . . . , xn , i.e.,

t = h(x1 , x2 , . . . , xn ).

Then t is a realisation of the random variable

T = h(X1 , X2 , . . . , Xn ),

which is called estimator.

Questions:

Intro to Probability Introduction 8

Estimator

Definition of Estimator
An estimate is a value t that only depends on the dataset x1 , x2 , . . . , xn , i.e.,

t = h(x1 , x2 , . . . , xn ).

Then t is a realisation of the random variable

T = h(X1 , X2 , . . . , Xn ),

which is called estimator.

Questions:
What makes an estimator suitable? unbiased (later: MSE)
Does an unbiased estimator always exist? How to compute it?
If there are several unbiased estimators, which one to choose?

Intro to Probability Introduction 8

Outline

Introduction

Defining and Analysing Estimators

More Examples

Intro to Probability Defining and Analysing Estimators 9

Example: Arrival of Packets (1/3)

Samples: Given X1 , X2 , . . . , Xn i.i.d., Xi ∼ Pois(λ)

Meaning: Xi is the number of packets arriving in minute i

Example 2

Estimate λ by using the sample mean X n .

Answer

Intro to Probability Defining and Analysing Estimators 10

Example: Arrival of Packets (1/3)

Samples: Given X1 , X2 , . . . , Xn i.i.d., Xi ∼ Pois(λ)

Meaning: Xi is the number of packets arriving in minute i

Example 2

Estimate λ by using the sample mean X n .

Answer

We have
X1 + X2 + · · · + Xn
X n := ,
n
h i
and E X n = E [ X1 ] = λ.

Intro to Probability Defining and Analysing Estimators 10

Example: Arrival of Packets (1/3)

Samples: Given X1 , X2 , . . . , Xn i.i.d., Xi ∼ Pois(λ)

Meaning: Xi is the number of packets arriving in minute i

Example 2

Estimate λ by using the sample mean X n .

Answer

We have
X1 + X2 + · · · + Xn
X n := ,
n
h i
and E X n = E [ X1 ] = λ. This suggests the estimator:

h(X1 , X2 , . . . , Xn ) := X n .

Intro to Probability Defining and Analysing Estimators 10

Example: Arrival of Packets (1/3)

Samples: Given X1 , X2 , . . . , Xn i.i.d., Xi ∼ Pois(λ)

Meaning: Xi is the number of packets arriving in minute i

Example 2

Estimate λ by using the sample mean X n .

Answer

We have
X1 + X2 + · · · + Xn
X n := ,
n
h i
and E X n = E [ X1 ] = λ. This suggests the estimator:

h(X1 , X2 , . . . , Xn ) := X n .

Applying the Weak Law of Large Numbers:

h i
lim P X n − λ >  = 0 for any  > 0.
n→∞

Intro to Probability Defining and Analysing Estimators 10

Example: Arrival of Packets (2/3)

Example 3a

Estimate the probability of zero arrivals, which is e−λ .

Answer

Intro to Probability Defining and Analysing Estimators 11

Example: Arrival of Packets (2/3)

Example 3a

Estimate the probability of zero arrivals, which is e−λ .

Answer

Let X1 , X2 , . . . , Xn be the n samples. Let

Yi := 1Xi =0 .

Intro to Probability Defining and Analysing Estimators 11

Example: Arrival of Packets (2/3)

Example 3a

Estimate the probability of zero arrivals, which is e−λ .

Answer

Let X1 , X2 , . . . , Xn be the n samples. Let

Yi := 1Xi =0 .

Then
E [ Yi ] = P [ Xi = 0 ] = e−λ ,

Intro to Probability Defining and Analysing Estimators 11

Example: Arrival of Packets (2/3)

Example 3a

Estimate the probability of zero arrivals, which is e−λ .

Answer

Let X1 , X2 , . . . , Xn be the n samples. Let

Yi := 1Xi =0 .

Then
E [ Yi ] = P [ Xi = 0 ] = e−λ ,
and thus we can define an estimator by
Y1 + Y2 + · · · + Yn
h1 (X1 , X2 , . . . , Xn ) := = Yn .
n

Intro to Probability Defining and Analysing Estimators 11

Example: Arrival of Packets (3/3)

Suppose we get the sample (x1 , x2 , x3 ) = (50, 100, 0)

Intro to Probability Defining and Analysing Estimators 12

Example: Arrival of Packets (3/3)

Suppose we get the sample (x1 , x2 , x3 ) = (50, 100, 0)

1
Then (y1 , y2 , y3 ) = (0, 0, 1), and h1 (x1 , x2 , x3 ) = 3

Intro to Probability Defining and Analysing Estimators 12

Example: Arrival of Packets (3/3)

Suppose we get the sample (x1 , x2 , x3 ) = (50, 100, 0)

1
Then (y1 , y2 , y3 ) = (0, 0, 1), and h1 (x1 , x2 , x3 ) = 3
This seems too small! Also note that for the samples
(x1 , x2 , x3 ) = (1, 1, 0), our estimator would give the same estimate

Intro to Probability Defining and Analysing Estimators 12

Example: Arrival of Packets (3/3)

Suppose we get the sample (x1 , x2 , x3 ) = (50, 100, 0)

1
Then (y1 , y2 , y3 ) = (0, 0, 1), and h1 (x1 , x2 , x3 ) = 3
This seems too small! Also note that for the samples
(x1 , x2 , x3 ) = (1, 1, 0), our estimator would give the same estimate

Example 3b

Estimate the probability of zero arrivals e−λ based on X n .

Answer

Intro to Probability Defining and Analysing Estimators 12

Example: Arrival of Packets (3/3)

Suppose we get the sample (x1 , x2 , x3 ) = (50, 100, 0)

1
Then (y1 , y2 , y3 ) = (0, 0, 1), and h1 (x1 , x2 , x3 ) = 3
This seems too small! Also note that for the samples
(x1 , x2 , x3 ) = (1, 1, 0), our estimator would give the same estimate

Example 3b

Estimate the probability of zero arrivals e−λ based on X n .

Answer
Pn h i
Xi
We saw that X n = n
satisfies E X n = E [ X1 ] = λ.
i=1

Recall by the Weak Law of Large Numbers:

Intro to Probability Defining and Analysing Estimators 12

Example: Arrival of Packets (3/3)

Suppose we get the sample (x1 , x2 , x3 ) = (50, 100, 0)

1
Then (y1 , y2 , y3 ) = (0, 0, 1), and h1 (x1 , x2 , x3 ) = 3
This seems too small! Also note that for the samples
(x1 , x2 , x3 ) = (1, 1, 0), our estimator would give the same estimate

Example 3b

Estimate the probability of zero arrivals e−λ based on X n .

Answer
Pn h i
Xi
We saw that X n = n
satisfies E X n = E [ X1 ] = λ.
i=1

Recall by the Weak Law of Large Numbers:

h i
lim P X n − λ >  = 0 for any  > 0.
n→∞

Intro to Probability Defining and Analysing Estimators 12

Example: Arrival of Packets (3/3)

Suppose we get the sample (x1 , x2 , x3 ) = (50, 100, 0)

1
Then (y1 , y2 , y3 ) = (0, 0, 1), and h1 (x1 , x2 , x3 ) = 3
This seems too small! Also note that for the samples
(x1 , x2 , x3 ) = (1, 1, 0), our estimator would give the same estimate

Example 3b

Estimate the probability of zero arrivals e−λ based on X n .

Answer
Pn h i
Xi
We saw that X n = n
satisfies E X n = E [ X1 ] = λ.
i=1

Recall by the Weak Law of Large Numbers:

h i
lim P X n − λ >  = 0 for any  > 0.
n→∞

This suggests to estimate e−λ by e−X n . Hence our estimator is

h2 (X1 , X2 , . . . , Xn ) := e−X n .

Intro to Probability Defining and Analysing Estimators 12

Behaviour of the Estimators

Suppose we have n = 30 and we want to estimate e−λ

Consider the two estimators h1 (X1 , . . . , Xn ) and h2 (X1 , . . . , Xn ).

Intro to Probability Defining and Analysing Estimators 13

Behaviour of the Estimators

Suppose we have n = 30 and we want to estimate e−λ

Consider the two estimators h1 (X1 , . . . , Xn ) and h2 (X1 , . . . , Xn ).

How good are these two estimators?

Intro to Probability Defining and Analysing Estimators 13

Behaviour of the Estimators

Suppose we have n = 30 and we want to estimate e−λ

Consider the two estimators h1 (X1 , . . . , Xn ) and h2 (X1 , . . . , Xn ).

How good are these two estimators?

1
⇒ The first estimator can only attain values 0, , 2 ,...,1
30 30
−1/30 −2/30
⇒ The second estimator can only attain values 1, e ,e ,...

Intro to Probability Defining and Analysing Estimators 13

Behaviour of the Estimators

Suppose we have n = 30 and we want to estimate e−λ

Consider the two estimators h1 (X1 , . . . , Xn ) and h2 (X1 , . . . , Xn ).

How good are these two estimators?

1
⇒ The first estimator can only attain values 0, , 2 ,...,1
30 30
−1/30 −2/30
⇒ The second estimator can only attain values 1, e ,e ,...

For most values of λ, both estimators will never return

the exact value of e−λ on the basis of 30 observations.

Intro to Probability Defining and Analysing Estimators 13

Simulation of the two Estimators

The unknown parameter is p = e−λ = 0.1 (i.e., λ = ln 10 ≈ 2.30 . . .)

Intro to Probability Defining and Analysing Estimators 14

Simulation of the two Estimators

The unknown parameter is p = e−λ = 0.1 (i.e., λ = ln 10 ≈ 2.30 . . .)

We consider n = 30 minutes and compute h1 and h2
We repeat this 500 times and draw a frequency histogram
(h1 = Y n left, h2 = e−X n right)

Intro to Probability Defining and Analysing Estimators 14

Simulation of the two Estimators

The unknown parameter is p = e−λ = 0.1 (i.e., λ = ln 10 ≈ 2.30 . . .)

We consider n = 30 minutes and compute h1 and h2
We repeat this 500 times and draw a frequency histogram
(h1 = Y n left, h2 = e−X n right)
h1 h2

Source: Modern Introduction to Statistics

Intro to Probability Defining and Analysing Estimators 14

Simulation of the two Estimators

The unknown parameter is p = e−λ = 0.1 (i.e., λ = ln 10 ≈ 2.30 . . .)

We consider n = 30 minutes and compute h1 and h2
We repeat this 500 times and draw a frequency histogram
(h1 = Y n left, h2 = e−X n right)
h1 h2

Source: Modern Introduction to Statistics

Both estimators concentrate around the true value 0.1, but

the second estimator appears to be more concentrated.

Intro to Probability Defining and Analysing Estimators 14

Unbiased Estimators and Bias

Definition
An estimator T is called an unbiased estimator for
the parameter θ if

E [ T ] = θ,

irrespective of the value θ.

Intro to Probability Defining and Analysing Estimators 15

Unbiased Estimators and Bias

Definition
An estimator T is called an unbiased estimator for
the parameter θ if

E [ T ] = θ,

irrespective of the value θ. The bias is defined as

E[T ] − θ = E[T − θ].

Intro to Probability Defining and Analysing Estimators 15

Unbiased Estimators and Bias

Definition
An estimator T is called an unbiased estimator for
the parameter θ if

E [ T ] = θ,

irrespective of the value θ. The bias is defined as

E[T ] − θ = E[T − θ].

Source: Edwin Leuven (Point Estimation)

Intro to Probability Defining and Analysing Estimators 15

Unbiased Estimators and Bias

Definition
An estimator T is called an unbiased estimator for
the parameter θ if

E [ T ] = θ,

irrespective of the value θ. The bias is defined as

E[T ] − θ = E[T − θ].

Source: Edwin Leuven (Point Estimation)

Which of the two estimators h1 , h2 are unbiased?

Intro to Probability Defining and Analysing Estimators 15

Analysis of the Bias of the First Estimator

Example 4a
Y1 +Y2 +···+Yn
Is h1 (X1 , X2 , . . . , Xn ) = n
an unbiased estimator for e−λ ?
Answer

Intro to Probability Defining and Analysing Estimators 16

Analysis of the Bias of the First Estimator

Example 4a
Y1 +Y2 +···+Yn
Is h1 (X1 , X2 , . . . , Xn ) = n
an unbiased estimator for e−λ ?
Answer

Recall we defined Yi := 1Xi =0 .

Intro to Probability Defining and Analysing Estimators 16

Analysis of the Bias of the First Estimator

Example 4a
Y1 +Y2 +···+Yn
Is h1 (X1 , X2 , . . . , Xn ) = n
an unbiased estimator for e−λ ?
Answer

Recall we defined Yi := 1Xi =0 . Yes, because:

E [ h1 (X1 , X2 , . . . , Xn ) ]

Intro to Probability Defining and Analysing Estimators 16

Analysis of the Bias of the First Estimator

Example 4a
Y1 +Y2 +···+Yn
Is h1 (X1 , X2 , . . . , Xn ) = n
an unbiased estimator for e−λ ?
Answer

Recall we defined Yi := 1Xi =0 . Yes, because:

n · E [ Y1 ]
E [ h1 (X1 , X2 , . . . , Xn ) ] =
n

Intro to Probability Defining and Analysing Estimators 16

Analysis of the Bias of the First Estimator

Example 4a
Y1 +Y2 +···+Yn
Is h1 (X1 , X2 , . . . , Xn ) = n
an unbiased estimator for e−λ ?
Answer

Recall we defined Yi := 1Xi =0 . Yes, because:

n · E [ Y1 ]
E [ h1 (X1 , X2 , . . . , Xn ) ] =
n
= P [ X1 = 0 ]

Intro to Probability Defining and Analysing Estimators 16

Analysis of the Bias of the First Estimator

Example 4a
Y1 +Y2 +···+Yn
Is h1 (X1 , X2 , . . . , Xn ) = n
an unbiased estimator for e−λ ?
Answer

Recall we defined Yi := 1Xi =0 . Yes, because:

n · E [ Y1 ]
E [ h1 (X1 , X2 , . . . , Xn ) ] =
n
= P [ X1 = 0 ]
= e−λ .

Intro to Probability Defining and Analysing Estimators 16

Bias of the Second Estimator (and Jensen’s Inequality)
Example 4b

Is h2 (X1 , X2 , . . . , Xn ) = e−X n an unbiased estimator for e−λ ?

Answer

Intro to Probability Defining and Analysing Estimators 17

Bias of the Second Estimator (and Jensen’s Inequality)
Example 4b

Is h2 (X1 , X2 , . . . , Xn ) = e−X n an unbiased estimator for e−λ ?

Answer

2
≥ E [ X ]2 )
 
No! (recall: E X

Intro to Probability Defining and Analysing Estimators 17

Bias of the Second Estimator (and Jensen’s Inequality)
Example 4b

Is h2 (X1 , X2 , . . . , Xn ) = e−X n an unbiased estimator for e−λ ?

Answer

2
≥ E [ X ]2 )
 
No! (recall: E X

Jensen’s Inequality
For any random variable X , and any convex function g : R → R, we have

E [ g(X ) ] ≥ g(E [ X ]).

If g is strictly convex and X is not constant, then the inequality is strict.

Intro to Probability Defining and Analysing Estimators 17

Bias of the Second Estimator (and Jensen’s Inequality)
Example 4b

Is h2 (X1 , X2 , . . . , Xn ) = e−X n an unbiased estimator for e−λ ?

Answer

2
≥ E [ X ]2 )
 
No! (recall: E X

λg(a) + (1 − λ)g(b) ≥ g(λa + (1 − λ)b)

Jensen’s Inequality
For any random variable X , and any convex function g : R → R, we have

E [ g(X ) ] ≥ g(E [ X ]).

If g is strictly convex and X is not constant, then the inequality is strict.

Intro to Probability Defining and Analysing Estimators 17

Bias of the Second Estimator (and Jensen’s Inequality)
Example 4b

Is h2 (X1 , X2 , . . . , Xn ) = e−X n an unbiased estimator for e−λ ?

Answer

2
≥ E [ X ]2 )
 
No! (recall: E X
We have
h i
E e−X n > e−E[ X n ] = e−λ

λg(a) + (1 − λ)g(b) ≥ g(λa + (1 − λ)b)

Jensen’s Inequality
For any random variable X , and any convex function g : R → R, we have

E [ g(X ) ] ≥ g(E [ X ]).

If g is strictly convex and X is not constant, then the inequality is strict.

Intro to Probability Defining and Analysing Estimators 17

Bias of the Second Estimator (and Jensen’s Inequality)
Example 4b

Is h2 (X1 , X2 , . . . , Xn ) = e−X n an unbiased estimator for e−λ ?

Answer

2
≥ E [ X ]2 )
 
No! (recall: E X
We have
h i
E e−X n > e−E[ X n ] = e−λ

This follows by Jensen’s inequality, and the inequality is strict since

g : z 7→ e−z is strictly convex and X n is not constant.

λg(a) + (1 − λ)g(b) ≥ g(λa + (1 − λ)b)

Jensen’s Inequality
For any random variable X , and any convex function g : R → R, we have

E [ g(X ) ] ≥ g(E [ X ]).

If g is strictly convex and X is not constant, then the inequality is strict.

Intro to Probability Defining and Analysing Estimators 17

Bias of the Second Estimator (and Jensen’s Inequality)
Example 4b

Is h2 (X1 , X2 , . . . , Xn ) = e−X n an unbiased estimator for e−λ ?

Answer

2
≥ E [ X ]2 )
 
No! (recall: E X
We have
h i
E e−X n > e−E[ X n ] = e−λ

This follows by Jensen’s inequality, and the inequality is strict since

g : z 7→ e−z is strictly convex and X n is not constant.
Thus h2 (X1 , X2 , . . . , Xn ) is not unbiased – it has positive bias.
λg(a) + (1 − λ)g(b) ≥ g(λa + (1 − λ)b)
Jensen’s Inequality
For any random variable X , and any convex function g : R → R, we have

E [ g(X ) ] ≥ g(E [ X ]).

If g is strictly convex and X is not constant, then the inequality is strict.

Intro to Probability Defining and Analysing Estimators 17

 Asymptotic Bias of the Second Estimator (non-examinable)
Example 4c
n→∞
E [ h2 (X1 , . . . , Xn ) ] −→ e−λ (hence it is asymptotically unbiased).
Answer

Recall h2 (X1 , . . . , Xn ) = e−X n . For any 0 ≤ k ≤ n,

" n #
h i X
−k /n
P h2 (X1 , . . . , Xn ) = e =P Xi = k = P [ Z = k ] ,
i=1

where Z ∼ Pois(n · λ) (since Pois(λ1 ) + Pois(λ2 ) = Pois(λ1 + λ2 ))

h i e−nλ · (nλ)k
⇒ P h2 (X1 , . . . , Xn ) = e−k /n =
k!
∞
X (nλk ) −k /n
⇒ E [ h2 (X1 , . . . , Xn ) ] = e−nλ · ·e
k!
k =0
∞
By LOTUS −1/n X −1/n (nλe−1/n )k
= e−nλ · enλe e−nλe ·
k!
k =0
−1/n
= e−nλ·(1−e )
·1
n→∞ 2
since ex = 1 + x + O(x 2 ) for small x ≈ e−nλ·(1−1+1/n+O(1/n ))
= e−λ+O(λ/n) .
Hence in the limit, the positive bias of h2 diminishes.
Intro to Probability Defining and Analysing Estimators 18
Outline

Introduction

Defining and Analysing Estimators

More Examples

Intro to Probability More Examples 19

Unbiased Estimator for Expectation and Variance

Unbiased Estimators for Expectation and Variance

Let X1 , X2 , . . . , Xn be identically distributed samples from a distribution
with finite expectation µ and finite variance σ 2 .

Intro to Probability More Examples 20

Unbiased Estimator for Expectation and Variance

Unbiased Estimators for Expectation and Variance

Let X1 , X2 , . . . , Xn be identically distributed samples from a distribution
with finite expectation µ and finite variance σ 2 .
Then
X1 + X2 + · · · + Xn
X n :=
n
is an unbiased estimator for µ.

Intro to Probability More Examples 20

Unbiased Estimator for Expectation and Variance

Unbiased Estimators for Expectation and Variance

Let X1 , X2 , . . . , Xn be identically distributed samples from a distribution
with finite expectation µ and finite variance σ 2 .
Then
X1 + X2 + · · · + Xn
X n :=
n
is an unbiased estimator for µ.
Furthermore, for n ≥ 2,
n 
1 X 2
Sn = Sn (X1 , . . . , Xn ) := · Xi − X n
n−1
i=1

2
is an unbiased estimator for σ .

Intro to Probability More Examples 20

 Example 5

We need to prove: E [ Sn ] = σ 2 .
Answer

Multiplying by n − 1 yields:
X n  2
(n − 1) · Sn = Xi − X n
i=1
n 
X 2
= Xi − µ + µ − X n
i=1

Intro to Probability More Examples 21

 Example 5

We need to prove: E [ Sn ] = σ 2 .
Answer

Multiplying by n − 1 yields:
X n  2
(n − 1) · Sn = Xi − X n
i=1
n 
X 2
= Xi − µ + µ − X n
i=1
n n  2 n  
2
X X X
= (Xi − µ) + Xn − µ − 2 (Xi − µ) X n − µ
i=1 i=1 i=1

Intro to Probability More Examples 21

 Example 5

We need to prove: E [ Sn ] = σ 2 .
Answer

Multiplying by n − 1 yields:
X n  2
(n − 1) · Sn = Xi − X n
i=1
n 
X 2
= Xi − µ + µ − X n
i=1
n n  2 n  
2
X X X
= (Xi − µ) + Xn − µ − 2 (Xi − µ) X n − µ
i=1 i=1 i=1
n  2    
2
X
= (Xi − µ) + n X n − µ − 2 X n − µ · n · X n − µ
i=1

Intro to Probability More Examples 21

 Example 5

We need to prove: E [ Sn ] = σ 2 .
Answer

Multiplying by n − 1 yields:
X n  2
(n − 1) · Sn = Xi − X n
i=1
n 
X 2
= Xi − µ + µ − X n
i=1
n n  2 n  
2
X X X
= (Xi − µ) + Xn − µ − 2 (Xi − µ) X n − µ
i=1 i=1 i=1
n  2    
2
X
= (Xi − µ) + n X n − µ − 2 X n − µ · n · X n − µ
i=1
n  2
2
X
= (Xi − µ) − n X n − µ .
i=1

Intro to Probability More Examples 21

 Example 5

We need to prove: E [ Sn ] = σ 2 .
Answer

Multiplying by n − 1 yields:
X n  2
(n − 1) · Sn = Xi − X n
i=1
n 
X 2
= Xi − µ + µ − X n
i=1
n n  2 n  
2
X X X
= (Xi − µ) + Xn − µ − 2 (Xi − µ) X n − µ
i=1 i=1 i=1
n  2    
2
X
= (Xi − µ) + n X n − µ − 2 X n − µ · n · X n − µ
i=1
n  2
2
X
= (Xi − µ) − n X n − µ .
i=1

Let us now take expectations:

Intro to Probability More Examples 21

 Example 5

We need to prove: E [ Sn ] = σ 2 .
Answer

Multiplying by n − 1 yields:
X n  2
(n − 1) · Sn = Xi − X n
i=1
n 
X 2
= Xi − µ + µ − X n
i=1
n n  2 n  
2
X X X
= (Xi − µ) + Xn − µ − 2 (Xi − µ) X n − µ
i=1 i=1 i=1
n  2    
2
X
= (Xi − µ) + n X n − µ − 2 X n − µ · n · X n − µ
i=1
n  2
2
X
= (Xi − µ) − n X n − µ .
i=1

Let us now take expectations:

(n − 1) · E [ Sn ] =

Intro to Probability More Examples 21

 Example 5

We need to prove: E [ Sn ] = σ 2 .
Answer

Multiplying by n − 1 yields:
X n  2
(n − 1) · Sn = Xi − X n
i=1
n 
X 2
= Xi − µ + µ − X n
i=1
n n  2 n  
2
X X X
= (Xi − µ) + Xn − µ − 2 (Xi − µ) X n − µ
i=1 i=1 i=1
n  2    
2
X
= (Xi − µ) + n X n − µ − 2 X n − µ · n · X n − µ
i=1
n  2
2
X
= (Xi − µ) − n X n − µ .
i=1

Let us now take expectations:

n h i  2 
2
X
(n − 1) · E [ Sn ] = E (Xi − µ) − n · E Xn − µ
i=1

Intro to Probability More Examples 21

 Example 5

We need to prove: E [ Sn ] = σ 2 .
Answer

Multiplying by n − 1 yields:
X n  2
(n − 1) · Sn = Xi − X n
i=1
n 
X 2
= Xi − µ + µ − X n
i=1
n n  2 n  
2
X X X
= (Xi − µ) + Xn − µ − 2 (Xi − µ) X n − µ
i=1 i=1 i=1
n  2    
2
X
= (Xi − µ) + n X n − µ − 2 X n − µ · n · X n − µ
i=1
n  2
2
X
= (Xi − µ) − n X n − µ .
i=1 h i h i
Let us now take expectations:
By Lec. 8, Slide 21: E (X − µ)2 = V X n = σ 2 /n
n
n h i  2 
2
X
(n − 1) · E [ Sn ] = E (Xi − µ) − n · E Xn − µ
i=1

Intro to Probability More Examples 21

 Example 5

We need to prove: E [ Sn ] = σ 2 .
Answer

Multiplying by n − 1 yields:
X n  2
(n − 1) · Sn = Xi − X n
i=1
n 
X 2
= Xi − µ + µ − X n
i=1
n n  2 n  
2
X X X
= (Xi − µ) + Xn − µ − 2 (Xi − µ) X n − µ
i=1 i=1 i=1
n  2    
2
X
= (Xi − µ) + n X n − µ − 2 X n − µ · n · X n − µ
i=1
n  2
2
X
= (Xi − µ) − n X n − µ .
i=1 h i h i
Let us now take expectations:
By Lec. 8, Slide 21: E (X − µ)2 = V X n = σ 2 /n
n
n h i  2 
2
X
(n − 1) · E [ Sn ] = E (Xi − µ) − n · E Xn − µ
i=1
2 2
= n · σ − n · σ /n

Intro to Probability More Examples 21

 Example 5

We need to prove: E [ Sn ] = σ 2 .
Answer

Multiplying by n − 1 yields:
X n  2
(n − 1) · Sn = Xi − X n
i=1
n 
X 2
= Xi − µ + µ − X n
i=1
n n  2 n  
2
X X X
= (Xi − µ) + Xn − µ − 2 (Xi − µ) X n − µ
i=1 i=1 i=1
n  2    
2
X
= (Xi − µ) + n X n − µ − 2 X n − µ · n · X n − µ
i=1
n  2
2
X
= (Xi − µ) − n X n − µ .
i=1 h i h i
Let us now take expectations:
By Lec. 8, Slide 21: E (X − µ)2 = V X n = σ 2 /n
n
n h i  2 
2
X
(n − 1) · E [ Sn ] = E (Xi − µ) − n · E Xn − µ
i=1
2 2
= n · σ − n · σ /n
2
= (n − 1) · σ .

Intro to Probability More Examples 21

An Unbiased Estimator may not always exist
Example 6
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist
Example 6
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

First a simpler proof which exploits that p might be arbitrarily small

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist
Example 6
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

First a simpler proof which exploits that p might be arbitrarily small

Intuition: By making p smaller and smaller, we force max0≤k ≤n T (k ),
k ∈ {0, 1, . . . , n} to become bigger and bigger

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist
Example 6
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

First a simpler proof which exploits that p might be arbitrarily small

Intuition: By making p smaller and smaller, we force max0≤k ≤n T (k ),
k ∈ {0, 1, . . . , n} to become bigger and bigger
Formal Argument:

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist
Example 6
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

First a simpler proof which exploits that p might be arbitrarily small

Intuition: By making p smaller and smaller, we force max0≤k ≤n T (k ),
k ∈ {0, 1, . . . , n} to become bigger and bigger
Formal Argument:
Fix any estimator T (X )

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist
Example 6
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

First a simpler proof which exploits that p might be arbitrarily small

Intuition: By making p smaller and smaller, we force max0≤k ≤n T (k ),
k ∈ {0, 1, . . . , n} to become bigger and bigger
Formal Argument:
Fix any estimator T (X )
Define M := max0≤k ≤n T (k ).

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist
Example 6
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

First a simpler proof which exploits that p might be arbitrarily small

Intuition: By making p smaller and smaller, we force max0≤k ≤n T (k ),
k ∈ {0, 1, . . . , n} to become bigger and bigger
Formal Argument:
Fix any estimator T (X )
Define M := max0≤k ≤n T (k ). Then,
n  
X n k
E [ T (X ) ] = p (1 − p)n−k · T (k )
k
k =0
n  
X n k
≤M· p (1 − p)n−k = M.
k
k =0

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist
Example 6
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

First a simpler proof which exploits that p might be arbitrarily small

Intuition: By making p smaller and smaller, we force max0≤k ≤n T (k ),
k ∈ {0, 1, . . . , n} to become bigger and bigger
Formal Argument:
Fix any estimator T (X )
Define M := max0≤k ≤n T (k ). Then,
n  
X n k
E [ T (X ) ] = p (1 − p)n−k · T (k )
k
k =0
n  
X n k
≤M· p (1 − p)n−k = M.
k
k =0

1
Hence this estimator does not work for p < M
, since then
E [ T (X ) ] ≤ M < p1 (negative bias!)

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist
Example 6
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

First a simpler proof which exploits that p might be arbitrarily small

Intuition: By making p smaller and smaller, we force max0≤k ≤n T (k ),
k ∈ {0, 1, . . . , n} to become bigger and bigger
Formal Argument:
Fix any estimator T (X )
Define M := max0≤k ≤n T (k ). Then,
n  
X n k
E [ T (X ) ] = p (1 − p)n−k · T (k )
k
k =0
n  
X n k
≤M· p (1 − p)n−k = M.
k
k =0

1
Hence this estimator does not work for p < M
, since then
E [ T (X ) ] ≤ M < p1 (negative bias!)
The next proof will work even if p ∈ [a, b] for 0 < a < b ≤ 1.

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist (cntd. - non-examinable)
Example 6 (cntd.)
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist (cntd. - non-examinable)
Example 6 (cntd.)
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

Suppose there exists an unbiased estimator with E [ T (X ) ] = 1/p.

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist (cntd. - non-examinable)
Example 6 (cntd.)
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

Suppose there exists an unbiased estimator with E [ T (X ) ] = 1/p.

Then
1 = p · E [ T (X ) ]

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist (cntd. - non-examinable)
Example 6 (cntd.)
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

Suppose there exists an unbiased estimator with E [ T (X ) ] = 1/p.

Then
1 = p · E [ T (X ) ]
n
X
=p· P [ X = k ] · T (k )
k =0

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist (cntd. - non-examinable)
Example 6 (cntd.)
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

Suppose there exists an unbiased estimator with E [ T (X ) ] = 1/p.

Then
1 = p · E [ T (X ) ]
n
X
=p· P [ X = k ] · T (k )
k =0
n
!
X n k
=p· p · (1 − p)n−k · T (k )
k
k =0

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist (cntd. - non-examinable)
Example 6 (cntd.)
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

Suppose there exists an unbiased estimator with E [ T (X ) ] = 1/p.

Then
1 = p · E [ T (X ) ]
n
X
=p· P [ X = k ] · T (k )
k =0
n
!
X n k
=p· p · (1 − p)n−k · T (k )
k
k =0

Last term is a polynomial of degree n + 1 with constant term zero

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist (cntd. - non-examinable)
Example 6 (cntd.)
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

Suppose there exists an unbiased estimator with E [ T (X ) ] = 1/p.

Then
1 = p · E [ T (X ) ]
n
X
=p· P [ X = k ] · T (k )
k =0
n
!
X n k
=p· p · (1 − p)n−k · T (k )
k
k =0

Last term is a polynomial of degree n + 1 with constant term zero

⇒ p · E [ T (X ) ] − 1 is a (non-zero) polynomial of degree ≤ n + 1

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist (cntd. - non-examinable)
Example 6 (cntd.)
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

Suppose there exists an unbiased estimator with E [ T (X ) ] = 1/p.

Then
1 = p · E [ T (X ) ]
n
X
=p· P [ X = k ] · T (k )
k =0
n
!
X n k
=p· p · (1 − p)n−k · T (k )
k
k =0

Last term is a polynomial of degree n + 1 with constant term zero

⇒ p · E [ T (X ) ] − 1 is a (non-zero) polynomial of degree ≤ n + 1
⇒ this polynomial has at most n + 1 roots

Intro to Probability More Examples 22

An Unbiased Estimator may not always exist (cntd. - non-examinable)
Example 6 (cntd.)
Suppose that we have one sample X ∼ Bin(n, p), where 0 < p < 1 is
unknown but n is known. Prove there is no unbiased estimator for 1/p.
Answer

Suppose there exists an unbiased estimator with E [ T (X ) ] = 1/p.

Then
1 = p · E [ T (X ) ]
n
X
=p· P [ X = k ] · T (k )
k =0
n
!
X n k
=p· p · (1 − p)n−k · T (k )
k
k =0

Last term is a polynomial of degree n + 1 with constant term zero

⇒ p · E [ T (X ) ] − 1 is a (non-zero) polynomial of degree ≤ n + 1
⇒ this polynomial has at most n + 1 roots
⇒ E [ T (X ) ] can be equal to 1/p for at most n + 1 values of p, and
thus cannot be an unbiased.

Intro to Probability More Examples 22