by Qiang Gao, updated at May 15, 2017
...
Consider maximizing (the log of) the joint likelihood
for the classical regression model, where $$ \tilde{ \boldsymbol{ \theta } } = ( \tilde{ \boldsymbol{ \beta } }, \tilde{ \sigma }^2 )'$$ and $$ \log f_{ \mathbf{y} \mid \mathbf{X} } ( \mathbf{y} \mid \mathbf{X}; \tilde{ \boldsymbol{ \theta } } ) $$ is given by
$$ \log L ( \tilde{ \boldsymbol{ \beta } }, \tilde{ \sigma }^2 ) =
- \frac{n}{2} \log ( 2 \pi ) - \frac{n}{2} \log ( \tilde{ \sigma }^2 ) - \frac{1}{ 2 \tilde{ \sigma }^2 } ( \mathbf{y} - \mathbf{X} \tilde{ \boldsymbol{ \beta } } )' ( \mathbf{y} - \mathbf{X} \tilde{ \boldsymbol{ \beta } } ). \tag{1.5.5} $$
You would parameterize the marginal likelihood $$ f( \mathbf{X} ; \tilde{ \boldsymbol{ \psi } } ) $$ and take the log of (1.5.2) to obtain the objective function to be maximized over $$ \boldsymbol{ \zeta } \equiv ( \boldsymbol{ \theta }', \boldsymbol{ \psi }' )'
(a) Taking log of (1.5.2),
The ML estimator of $$ \boldsymbol{ \theta } $$ maximizing (1) will be exactly the same as that of maximizing $$ \log f_{ \mathbf{y} \mid \mathbf{X} } ( \mathbf{y} \mid \mathbf{X}; \tilde{ \boldsymbol{ \theta } } )
(b) By the information matrix equality,
$$ \begin{align} \mathbf{I} ( \boldsymbol{ \zeta } ) & =
- \mathrm{E} \left[ \frac{ \partial^2 \log L ( \boldsymbol{ \zeta } ) } { \partial \tilde{ \boldsymbol{ \zeta } } , \partial \tilde{ \boldsymbol{ \zeta } }' } \right] \ & =
- \mathrm{E} \left[ \begin{matrix} \frac{ \partial^2 \log L ( \boldsymbol{ \theta }, \boldsymbol{ \psi } ) } { \partial \tilde{ \boldsymbol{ \theta } } , \partial \tilde{ \boldsymbol{ \theta } }' } & \frac{ \partial^2 \log L ( \boldsymbol{ \theta }, \boldsymbol{ \psi } ) } { \partial \tilde{ \boldsymbol{ \theta } } , \partial \tilde{ \boldsymbol{ \psi } }' } \ \frac{ \partial^2 \log L ( \boldsymbol{ \theta }, \boldsymbol{ \psi } ) } { \partial \tilde{ \boldsymbol{ \psi } } , \partial \tilde{ \boldsymbol{ \theta } }' } & \frac{ \partial^2 \log L ( \boldsymbol{ \theta }, \boldsymbol{ \psi } ) } { \partial \tilde{ \boldsymbol{ \psi } } , \partial \tilde{ \boldsymbol{ \psi } }' } \end{matrix} \right] \ & =
- \mathrm{E} \left[ \begin{matrix} \frac{ \partial^2 \log L ( \boldsymbol{ \theta }, \boldsymbol{ \psi } ) } { \partial \tilde{ \boldsymbol{ \theta } } , \partial \tilde{ \boldsymbol{ \theta } }' } & \mathbf{0} \ \mathbf{0} & \frac{ \partial^2 \log L ( \boldsymbol{ \theta }, \boldsymbol{ \psi } ) } { \partial \tilde{ \boldsymbol{ \psi } } , \partial \tilde{ \boldsymbol{ \psi } }' } \end{matrix} \right], \end{align} $$
since $$ \partial^2 \log L ( \boldsymbol{ \theta }, \boldsymbol{ \psi } ) / ( \partial \tilde{ \boldsymbol{ \theta } } ,
\partial \tilde{ \boldsymbol{ \psi } }' ) = \mathbf{0}
- \mathrm{E} \left[ \frac{ \partial^2 \log L ( \boldsymbol{ \theta }, \boldsymbol{ \psi } ) } { \partial \tilde{ \boldsymbol{ \theta } } , \partial \tilde{ \boldsymbol{ \theta } }' } \right] =
- \mathrm{E} \left[ \frac{ \partial^2 \log L ( \boldsymbol{ \theta } ) } { \partial \tilde{ \boldsymbol{ \theta } } , \partial \tilde{ \boldsymbol{ \theta } }' } \right]. $$
So the Cramer-Rao bound for $$ \boldsymbol{ \theta } $$ is the negative of the inverse of the expected value of (1.5.12) in the text. The expectation, however, is over $$ \mathbf{y} $$ and $$ \mathbf{X} $$ because here the density is a joint density. Therefore, the Cramer-Rao bound for $$ \boldsymbol{ \beta } $$ is $$ \sigma^2 [ \mathrm{E} ( \mathbf{X}' \mathbf{X} ) ]^{-1} $$.
Copyright ©2017 by Qiang Gao