1.5.2.md

Solution to Review Question

by Qiang Gao, updated at May 15, 2017

---

Chapter 1 Finite-Sample Properties of OLS

Section 5 Relation to Maximum Likelihood

...

Review Question 1.5.2 (Maximizing joint log likelihood)

Consider maximizing (the log of) the joint likelihood

$$ f_{ \mathbf{y}, \mathbf{X} } ( \mathbf{y}, \mathbf{X}; \tilde{ \boldsymbol{ \zeta } } ) = f_{ \mathbf{y} \mid \mathbf{X} } ( \mathbf{y} \mid \mathbf{X}; \tilde{ \boldsymbol{ \theta } } ) \cdot f_{ \mathbf{X} } ( \mathbf{X} ; \tilde{ \boldsymbol{ \psi } } ) \tag{1.5.2} $$

for the classical regression model, where $$ \tilde{ \boldsymbol{ \theta } } = ( \tilde{ \boldsymbol{ \beta } }, \tilde{ \sigma }^2 )'$$ and $$ \log f_{ \mathbf{y} \mid \mathbf{X} } ( \mathbf{y} \mid \mathbf{X}; \tilde{ \boldsymbol{ \theta } } ) $$ is given by

$$ \log L ( \tilde{ \boldsymbol{ \beta } }, \tilde{ \sigma }^2 ) =

• \frac{n}{2} \log ( 2 \pi ) - \frac{n}{2} \log ( \tilde{ \sigma }^2 ) - \frac{1}{ 2 \tilde{ \sigma }^2 } ( \mathbf{y} - \mathbf{X} \tilde{ \boldsymbol{ \beta } } )' ( \mathbf{y} - \mathbf{X} \tilde{ \boldsymbol{ \beta } } ). \tag{1.5.5} $$

You would parameterize the marginal likelihood $$ f( \mathbf{X} ; \tilde{ \boldsymbol{ \psi } } ) $$ and take the log of (1.5.2) to obtain the objective function to be maximized over $$ \boldsymbol{ \zeta } \equiv ( \boldsymbol{ \theta }', \boldsymbol{ \psi }' )' $$. (a) What is the ML estimator of $$ \boldsymbol{ \theta } \equiv ( \boldsymbol{ \beta }', \sigma^2 )' $$? (b) Derive the Cramer-Rao bound for $$ \boldsymbol{ \beta } $$.

Solution

(a) Taking log of (1.5.2),

$$ \begin{align} \log f_{ \mathbf{y}, \mathbf{X} } ( \mathbf{y}, \mathbf{X}; \tilde{ \boldsymbol{ \zeta } } ) = \log f_{ \mathbf{y} \mid \mathbf{X} } ( \mathbf{y} \mid \mathbf{X}; \tilde{ \boldsymbol{ \theta } } ) + \log f_{ \mathbf{X} } ( \mathbf{X} ; \tilde{ \boldsymbol{ \psi } } ). \tag{1} \end{align} $$

The ML estimator of $$ \boldsymbol{ \theta } $$ maximizing (1) will be exactly the same as that of maximizing $$ \log f_{ \mathbf{y} \mid \mathbf{X} } ( \mathbf{y} \mid \mathbf{X}; \tilde{ \boldsymbol{ \theta } } ) $$, because $$ \tilde{ \boldsymbol{ \theta } } $$ does not appear in $$ \log f_{ \mathbf{X} } ( \mathbf{X} ; \tilde{ \boldsymbol{ \psi } } ) $$, the first-order conditions of the two maximization will be the same.

(b) By the information matrix equality,

$$ \begin{align} \mathbf{I} ( \boldsymbol{ \zeta } ) & =

• \mathrm{E} \left[ \frac{ \partial^2 \log L ( \boldsymbol{ \zeta } ) } { \partial \tilde{ \boldsymbol{ \zeta } } , \partial \tilde{ \boldsymbol{ \zeta } }' } \right] \ & =
• \mathrm{E} \left[ \begin{matrix} \frac{ \partial^2 \log L ( \boldsymbol{ \theta }, \boldsymbol{ \psi } ) } { \partial \tilde{ \boldsymbol{ \theta } } , \partial \tilde{ \boldsymbol{ \theta } }' } & \frac{ \partial^2 \log L ( \boldsymbol{ \theta }, \boldsymbol{ \psi } ) } { \partial \tilde{ \boldsymbol{ \theta } } , \partial \tilde{ \boldsymbol{ \psi } }' } \ \frac{ \partial^2 \log L ( \boldsymbol{ \theta }, \boldsymbol{ \psi } ) } { \partial \tilde{ \boldsymbol{ \psi } } , \partial \tilde{ \boldsymbol{ \theta } }' } & \frac{ \partial^2 \log L ( \boldsymbol{ \theta }, \boldsymbol{ \psi } ) } { \partial \tilde{ \boldsymbol{ \psi } } , \partial \tilde{ \boldsymbol{ \psi } }' } \end{matrix} \right] \ & =
• \mathrm{E} \left[ \begin{matrix} \frac{ \partial^2 \log L ( \boldsymbol{ \theta }, \boldsymbol{ \psi } ) } { \partial \tilde{ \boldsymbol{ \theta } } , \partial \tilde{ \boldsymbol{ \theta } }' } & \mathbf{0} \ \mathbf{0} & \frac{ \partial^2 \log L ( \boldsymbol{ \theta }, \boldsymbol{ \psi } ) } { \partial \tilde{ \boldsymbol{ \psi } } , \partial \tilde{ \boldsymbol{ \psi } }' } \end{matrix} \right], \end{align} $$

since $$ \partial^2 \log L ( \boldsymbol{ \theta }, \boldsymbol{ \psi } ) / ( \partial \tilde{ \boldsymbol{ \theta } } , \partial \tilde{ \boldsymbol{ \psi } }' ) = \mathbf{0} $$. Thus the information matrix $$ \mathbf{I} ( \boldsymbol{ \zeta } ) $$ is block diagonal with its first block corresponding to $$ \boldsymbol{ \theta } $$ and the second block corresponding to $$ \boldsymbol{ \psi } $$. Its inverse is also block diagonal, with its first block being the inverse of

$$

• \mathrm{E} \left[ \frac{ \partial^2 \log L ( \boldsymbol{ \theta }, \boldsymbol{ \psi } ) } { \partial \tilde{ \boldsymbol{ \theta } } , \partial \tilde{ \boldsymbol{ \theta } }' } \right] =
• \mathrm{E} \left[ \frac{ \partial^2 \log L ( \boldsymbol{ \theta } ) } { \partial \tilde{ \boldsymbol{ \theta } } , \partial \tilde{ \boldsymbol{ \theta } }' } \right]. $$

So the Cramer-Rao bound for $$ \boldsymbol{ \theta } $$ is the negative of the inverse of the expected value of (1.5.12) in the text. The expectation, however, is over $$ \mathbf{y} $$ and $$ \mathbf{X} $$ because here the density is a joint density. Therefore, the Cramer-Rao bound for $$ \boldsymbol{ \beta } $$ is $$ \sigma^2 [ \mathrm{E} ( \mathbf{X}' \mathbf{X} ) ]^{-1} $$.

---

Copyright ©2017 by Qiang Gao