Scribd
Upload
861 views3 pages

Assignment 11: Introduction To Machine Learning Prof. B. Ravindran

This document contains an assignment on machine learning concepts including: 1. Finding the maximum likelihood estimate of the parameter λ for samples from an exponential distribution. The correct MLE is the average of the samples. 2. Estimating the number of independent parameters needed for a Gaussian distribution, which is p(p+1)/2 due to the symmetric covariance matrix. 3. Finding the MLE of the parameter λ for a Poisson distribution, which is the average of the samples. 4. EM algorithm minimizes the negative log-likelihood when estimating parameters for a Gaussian mixture model.

Uploaded by

puranam Satya
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
861 views3 pages

Assignment 11: Introduction To Machine Learning Prof. B. Ravindran

This document contains an assignment on machine learning concepts including: 1. Finding the maximum likelihood estimate of the parameter λ for samples from an exponential distribution. The correct MLE is the average of the samples. 2. Estimating the number of independent parameters needed for a Gaussian distribution, which is p(p+1)/2 due to the symmetric covariance matrix. 3. Finding the MLE of the parameter λ for a Poisson distribution, which is the average of the samples. 4. EM algorithm minimizes the negative log-likelihood when estimating parameters for a Gaussian mixture model.

Uploaded by

puranam Satya
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

Assignment 11

Introduction to Machine Learning


Prof. B. Ravindran
1. Given n samples x1 , x2 , . . . , xN drawn independently from an Exponential distribution un-
known parameter λ, find the MLE of λ.
Pn
(a) λM LE = i=1 xi
Pn
(b) λM LE = n i=1 xi
Pn
xi
(c) λM LE = n
i=1

(d) λM LE = Pnn
i=1 xi

(e) λM LE = Pn−1
n
i=1 xi
Pn
i=1 xi
(f) λM LE = n−1

Sol. (d)

n
Y n
Y Pn
L(λ, x1 , . . . , xn ) = f (xi , λ) = λe−λx = λn e−λ i=1 xi

i=1 i=1

Pn
d ln λn e−λ i=1 xi

d ln (L(λ, x1 , . . . , xn ))
=
dλ dλ Pn
d ln (n ln(λ) − λ i=1 xi )
=
dλ
n
n X
= − xi
λ i=1

Set the above term to zero to obtain MLE of λ


n
λ= P
n
xi
i=1

2. (2 marks) Suppose we are trying to model a p dimensional Gaussian distribution. What is the
actual number of independent parameters that need to be estimated?

(a) 2
(b) p
(c) 2p
(d) p(p + 1)
(e) p(p + 1)/2
(f) p(p + 3)/2

1
Sol. (f)
Explanation Mean vector has p parameters. The covariance matrix is symmetric (p × p) and
hence has p p+1
2 independent parameters.

3. Given n samples x1 , x2 , . . . , xN drawn independently from a Poisson distribution unknown


parameter λ, find the MLE of λ.
Pn
(a) λM LE = i=1 xi
Pn
(b) λM LE = n i=1 xi
Pn
xi
(c) λM LE = i=1
n
(d) λM LE = Pnn
i=1 xi

(e) λM LE = Pn−1
n
i=1 xi
Pn
i=1 xi
(f) λM LE = n−1

Sol. (c)
Write the likelihood:
Y λxi e−nλ
l(λ; x) = e−λ = λx1 +x2 +···+xn
i
xi ! x1 !x2 ! · · · xn !

Take the log and differentiate the log-likelihood with respect to λ and set it to 0.
4. During parameter estimation for a GMM model using data X, which of the following quantities
are you minimizing (directly or indirectly)?
(a) Log-likelihood
(b) Negative Log-likelihood
(c) Cross-entropy
(d) Residual Sum of Squares (RSS)
Sol. (b)

5. (2 marks) In Gaussian Mixture Models, πi are the mixing coefficients. Select the correct
conditions that the mixing coefficients need to satisfy for a valid GMM model.
(a) 0 ≤ πi ≤ 1∀i
(b) −1 ≤ πi ≤ 1∀i
P
(c) i πi = 1
P
(d) i πi need not be bounded

Sol. (a), (c)

2
6. (2 marks) Expectation-Maximization, or the EM algorithm, consists of two steps - E step and
the M-step. Using the following notation, select the correct set of equations used at each step
of the algorithm.
Notation.
X Known/Given variables/data
Z Hidden/Unknown variables
θ Total set of parameters to be learned
θk Values of all the parameters after stage k
Q(, ) The Q-function as described in the lectures
(a) E − EZ|X,θm−1 [log(P r(X, Z|θ))]
(b) E − EZ|X,θ [log(P r(X, Z|θm ))]
P
(c) M − argmaxθ Z P r(Z|X, θm−1 ) · log(P r(X, Z|θ))
(d) M − argmaxθ Q(θ, θm−1 )
(e) M − argmaxθ Q(θ, θm−2 )
Sol. (a), (c), (d)

You might also like