Point Estimation

Population: In Statistics, population is an aggregate of objects, animate or inanimate, under

study. The population may be finite or infinite.
Sample: A part or a finite subset of population is called a sample and the number of units in the
sample is called the sample size.
Parameter: The specific characteristics of the population such as population mean (µ), population
variance (σ 2 ) are referred as parameters.
Statistic: It is a function of sample observations, for example, sample mean (x̄), sample variance
(s2 ) are known as statistics.
Here in the theory of point estimation, we consider that the population under study is described by
a probability density function (pdf) or probability mass function (pmf), say, f (x|θ). The knowledge
of parameter(s) θ yields the knowledge of entire population but the problem of statistical parametric
inference is that θ is unknown. In order to estimate this θ, we resort to take a random sample from
the population and infer about the unknown parameter(s) θ. It may also happen that instead of
θ, our interest is to find an estimator for a function of θ, say, g(θ).

Definition 1. Estimator: Any function of the random sample which is used to estimate the
unknown value of the given parametric function g(θ) is called an estimator. If X = X1 , . . . , Xn is
a random sample from a population with common distribution function Fθ , a function t(X) used
for estimating g(θ) is known as an estimator. Let x = x1 , · · · , xn be a realization of X. Then, t(x)
is called an estimate.

For example, in estimating the average height of male students in a class, we may use the sample
mean X̄ as an estimator. Now, if a random sample of size 20 has a sample mean 170cm, then
170cm is an estimate of the average height of male students of that class.

Parameter Space: The set of all possible values of a parameter(s) is called parameter space. It
is denoted by Θ.

Desirable Criteria for Estimators

Given the sample, one may have multiple estimators to estimate the parametric function. For
example, to estimate the population average, one may use sample mean/sample median/sample
mode. So, in order to choose among the estimators, we should have certain desirable criteria which
the estimator to be used should meet. Two such criteria unbiasedness and consistency are discussed
as follows.

Definition 2. Unbiasedness: Let X1 , · · · , Xn be a random sample from a population with prob-

ability distribution Pθ , θ ∈ Θ. An estimator t(X), X = X1 , · · · , Xn is said to be unbiased for
estimating g(θ), if
Eθ (t(X)) = g(θ), ∀θ ∈ Θ. (1)

If for some θ ∈ Θ, we have

Eθ (t(X)) = g(θ) + bn (θ),
then, bn (θ) is called bias of t. If bn (θ) > 0, ∀θ, then t is said to overestimate g(θ). On the other
hand if bn (θ) < 0, ∀θ, then t is an underestimator of g(θ).

Definition 3. An estimator t(X) is said to be asymptotically unbiased estimator of θ if

lim bn (θ) = 0, ∀θ ∈ Θ. (2)

n→∞

1
Definition 4. The quantity Eθ (t(X) − θ)2 is called the mean square error (MSE) of t(X) about θ.

M SE(t(X)) = V ar(t(X)) + (bn (θ))2 .

If t is unbiased for θ, M SE(t) reduces to V ar(t).

Example 5. Let X1 , · · · , Xn be a random sample from binomial distribution with parameters n and
p, where, n is known and 0 ≤ p ≤ 1. Find unbiased estimators for a) p, the population proportion,
b) p2 c) Variance of X.

Solution: a) Given that X follows binomial(n, p), n is known and p, the population proportion is
unknown. Let t(X) = Xn , the sample proportion. Now,
 
X np
E(t(X)) = E = = p.
n n

Thus, the sample proportion is an unbiased estimator of population proportion.

b) We can compute that
E(X(X − 1)) = n(n − 1)p2 (3)
Hence, X(X−1) 2
n(n−1) is an unbiased estimator for p .
2
 − p) = n(p− p ).
c)Since, Var(X) = np(1
X(X−1) X(n−X)
Therefore, t(X) = n X n − n(n−1) = n−1 is an unbiased estimator of Variance of X.

Example 6. Let X1 , · · · , Xn be a random sample from the population

(
e−(x−θ) , x > θ
f (x, θ) =
0, otherwise
.
Is X̄ unbiased for θ?

Solution: Note that Z ∞

E(X) = xe−(x−θ) = θ + 1,
0

so that E(X̄) = E(X) = θ + 1. Thus, X̄ is a biased estimator for θ. However, E(X̄ − 1) = θ.

Remark 7. 1. The unbiased estimator need not be unique. For example, let X1 , · · · , Xn be
a random sample form Poisson distribution with parameter λ, λ > 0. Then, t1 (X) = X̄,
t2 (X) = Xi , t3 (X) = X1 +2X
3
2
are some unbiased estimators for λ.

2. If E(X) exists, then the sample mean is an unbiased estimator of the population mean.
1 Pn
3. Let E(X 2 ) exists, i.e. V ar(X) = σ 2 exists. Then, S 2 = n−1 2
i=1 (Xi − X̄) is unbiased for
σ 2 .(Prove!)

4. Unbiased estimators may not always exist. For example, X follows binomial distribution with
parameters n and p. Then, there exists no unbiased estimator for pn+1 .(Prove!)

5. Unbiased estimators may not be reasonable always. They may be absurd. For example t(X) =
(−2)X is an absurd unbiased estimator for e−3λ , where, X follows Poisson distribution with
parameter λ. (Why?)

2
It is intuitively clear that for tn (X)(= t(X)) to be a good estimator the difference tn − θ should
be as small as possible. However, tn is a random variable and has its own sampling distribution
whose range may be infinitely large. Therefore, it would be sufficient if the sampling distribution
of tn becomes more and more concentrated around θ as the sample size n increases. This means
that for each fixed θ ∈ Θ, the probability

Pθ [|Tn − θ| ≤ ]

for any given (> 0) should be an increasing function of n. This idea leads to the concept of
consistency as a criterion of a good estimator.

Definition 8. Consistency: A statistic t or rather a sequence {tn } is said to be consistent for θ if

tn converges in probability to θ (tn → θ) as n → ∞ for each fixed θ ∈ Θ. Thus, tn is said to be
consistent if for every fixed θ ∈ Θ and every pair of positive quantities  and η, however, small, it
is possible to find an n0 , depending on  and η, such that

Pθ [|tn − θ| < ] > 1 − η,

whenever n ≥ n0 (, η).

If such statistics are used, the accuracy of the estimate increases with the increase in the value
of n. It is to be noted that consistency is a large sample property as it is concerned with the
behavior of an estimator as the sample size becomes infinitely large.

Example 9. Let X1 , · · · , Xn be a random sample from a population with mean µ and variance σ 2 .
Then,
V ariance(X̄) σ2
P (|X̄ − µ| > ) ≤ = → 0 as n → ∞.
2 n2
Hence, X̄ is consistent for µ.

Example 10. Let X1 , · · · , Xn be a sequence of independently and identically distributed (iid) ran-
dom variables with mean µ, then by weak law of large numbers (WLLN), X̄ is consistent for µ.

Example 11. Let {Xn } be a sequence of iid random variables with pdf
(
e−(x−θ) , x > θ
f (x, θ) =
0, otherwise

Show that X(1) = min Xi is a consistent estimator of θ.

Solution: The pdf of X(1) is

!n−1
Z ∞
−(x(1) −θ) −(x−θ)dx
g(x(1) ) = ne e = ne−n(x(1) )−θ ,
x(1)

for x(1) > θ and g(x(1) ) = 0 otherwise.

Now, P (|X(1) − θ| < ) = P (θ < X(1) < θ + ) = 1 − e−n → 1 as n → ∞.
Hence, X(1) is consistent for θ.

Remark 12. 1. If population mean exists, sample mean is consistent for the population mean.

2. The consistent estimator may not be unique. For example, if tn is consistent for θ, then,
n n+2
n+1 tn , n+4 tn are all consistent for θ.

3
Theorem 13. Let {tn } be a sequence of estimates such that for every θ ∈ Θ, the expectation and
variance of tn exist and E(tn ) = θn → θ and V (tn ) → 0 as n → ∞. Then, tn is consistent for θ.

Proof. We have, by Chebyshev’s inequality

E(tn − θ)2 V (tn ) + (E(tn ) − θ)2

P [|tn − θ| > ] < = →0 as n → ∞.
2 2

Theorem 14. If t is consistent for θ and h is a continuous function of θ. Then, h(t) is consistent
for h(θ).

Exercise 15. 1. Let X1 , . . . , Xn be random sample from uniform distribution

(
1
, 0<x<θ
f (x) = θ .
0, otherwise

2. Let X1 , . . . , Xn be a random sample from the uniform disitribution

(
1
, 0≤x≤θ
f (x) = θ .
0, otherwise

Examine the consistencies of the estimators T1 = max Xi , T2 = (n+1) min Xi , T3 = min Xi +

max Xi , T4 = 2X̄ for estimating θ.