Losgistic Regression
In general linear regression, the values of dependent variable
It is also worth noticing that the change in output of the lienar combination in logistic regression is equal to the log of odds ratio, since:
where
Maximum Likelihood Estimation
The likelihood function is defined for the logsitic model is:
The second partial derivative of the log-likelihood function with respect to
Maximum A Posteriori Estimation
The MLE approach is a frequentist approach that does not take into account any prior knowledge about the parameters. On the other hand, the Maximum A Posteriori (MAP) estimation incorporates prior knowledge about the parameters through the prior distribution
The following assumptions are made in this project for MAP estimation:
- The coefficients
$\boldsymbol{\beta}$ follow a non-informative multivariate normal distribution as the prior distribution (zero mean and large variance). - The coefficients
$\boldsymbol{\beta}$ are independent (diagonal covariance matrix).
The same technique can be applied to MAP estimation to acquire the
where
Posterior Distribution Estimation
In MCMC algorithms, the Monte Carlo simulation describes the process of using a Markov chain to produce random samples that can be used for estimation (e.g. samples of logistic regression parameters as the parameters themselves are treated as random variables) by applying the law of large numbers. Markov chain models the probability of transitions between states, and the stationary state of a Markov chain can be treated as a probability distribution. Running a Markov chain after it has reached the stationary state can then be considered as a process of random sampling from the stationary distribution.
One sufficient but not necessary condition to ensure the stationary distribution exists is the “detailed balance” condition, which states that the probability of transitioning from the
where
- Initialize a empty list samples
- Define initial states
$\boldsymbol{\beta}$ and add it to the samples - If sample size is not reached, propose the next candidate
$\boldsymbol{\beta'}$ be the randomly select a sample from the proposal distribution$MVN(\boldsymbol{\beta}, \boldsymbol{\sigma}^{2}I)$ . If sample size reached, stop. - Calculate the acceptance probability. Let
$A(\boldsymbol{\beta} | \boldsymbol{\beta'}) = min(1,\frac{\textit{ll}(\boldsymbol{\beta'})}{\textit{ll}(\boldsymbol{\beta})})$ . Let$u$ be a random sample from a uniform distribution$U(0,1)$ . If$u <= A(\boldsymbol{\beta} | \boldsymbol{\beta'})$ , accept$\boldsymbol{\beta'}$ and add$\boldsymbol{\beta'}$ to the samples list, then go to step 2 and propose the next candidate from$MVN(\boldsymbol{\beta'}, \boldsymbol{\sigma}^{2}I)$ ; if$u > A(\boldsymbol{\beta} | \boldsymbol{\beta'})$ , reject$\boldsymbol{\beta'}$ and add$\boldsymbol{\beta}$ to the samples list, then go to step 2 and propose the next candidate from$MVN(\boldsymbol{\beta}, \boldsymbol{\sigma}^{2}I)$ . After running the Metropolis algorithm, the initial states that did not reach stationary distribution (burn-in state) will be discarded. Then the rest of the samples will be used to estimate the expectation and variance of the stationary distribution, which is the posterior distribution in the Bayesian regression.