I am studying how to construct a 95% confidence interval for odds ratio from the coefficients obtained in the logistic regression. So, considering the logistic regression model,

$$ \log\left(\frac{p}{1 - p}\right) = \alpha + \beta x \newcommand{\var}{\rm Var} \newcommand{\se}{\rm SE} $$

such that $x = 0$ for control group and $x = 1$ for case group.

I have already read that the simplest way is to construct a 95% CI for $\beta$ then we applied the exponential function, that is,

$$ \hat{\beta} \pm 1.96\times \se(\hat{\beta}) \rightarrow \exp\{\hat{\beta} \pm 1.96\times \se(\hat{\beta})\} $$

My questions are:

  1. What is the theoretical reason that justifies this procedure? I know $\mbox{odds ratio} = \exp\{\beta\}$ and maximum likelihood estimators are invariant. However, I do not know the connection among these elements.

  2. Should the delta method produce the same 95% confidence interval as the previous procedure? Using the delta method,

    $$\exp\{\hat{\beta}\} \dot{\sim} N(\beta,\ \exp\{\beta\}^2 \var(\hat{\beta}))$$


    $$\exp\{\hat{\beta}\} \pm 1.96\times \sqrt{\exp\{\beta\}^2 \var(\hat{\beta})}$$

    If not, which is the best procedure?

  • 1
    $\begingroup$ I like bootstrap for CI as well, if I have parameter values or training data of sufficient size. $\endgroup$ Apr 10 '16 at 1:32
  • 2
    $\begingroup$ There is a better way of doing this, see stats.stackexchange.com/questions/5304/… for details $\endgroup$
    – mdewey
    May 10 '16 at 13:28
  1. The justification for the procedure is the asymptotic normality of the MLE for $\beta$ and results from arguments involving the Central Limit Theorem.

  2. The Delta method comes from a linear (i.e first order Taylor) expansion of the function around the MLE. Subsequently we appeal to the asymptotic normality and unbiasedness of the MLE.

Asymptotically both give the same answer. But practically, you would favor the one which looks more closely normal. In this example, I would favor the first one because the latter is likely to be less symmetric.


A comparison of confidence intervals methods on an example from ISL

The book "Introduction to Statistical Learning" by Tibshirani, James, Hastie provides an example on page 267 of confidence intervals for polynomial logistic regression degree 4 on the wage data. Quoting the book:

We model the binary event $wage>250$ using logistic regression with a degree-4 polynomial. The fitted posterior probability of wage exceeding $250,000 is shown in blue, along with an estimated 95 % confidence interval.

Below is a quick recap of two methods for constructing such intervals as well as comments on how to implement them from scratch

Wald / Endpoint transformation intervals

  • Compute the upper and lower bounds of the confidence interval for the linear combination $x^T\beta$ (using the Wald CI)
  • Apply a monotonic transformation to the endpoints $F(x^T\beta)$ to obtain the probabilities.

Since $Pr(x^T\beta) = F(x^T\beta)$ is a monotonic transformation of $x^T\beta$

$$ [Pr(x^T\beta)_L \leq Pr(x^T\beta) \leq Pr(x^T\beta)_U] = [F(x^T\beta)_L \leq F(x^T\beta) \leq F(x^T\beta)_U] $$

Concretely this means computing $\beta^Tx \pm z^* SE(\beta^Tx)$ and then applying the logit transform to the result to get the lower and upper bounds:

$$[\frac{e^{x^T\beta - z^* SE(x^T\beta)}}{1 + e^{x^T\beta - z^* SE(x^T\beta)}}, \frac{e^{x^T\beta + z^* SE(x^T\beta)}}{1 + e^{x^T\beta + z^* SE(x^T\beta)}},] $$

Computing the standard error

Maximum Likelihood theory tells us that the approximate variance of $x^T\beta$ can be calculated using the covariance matrix $\Sigma$ of the regression coefficients using

$$ Var(x^T\beta) = x^T \Sigma x$$

Define the design matrix $X$ and the matrix $V$ as

$$\textbf{X = }\begin{bmatrix} 1 & x_{1,1} & \ldots & x_{1,p} \\ 1 & x_{2,1} & \ldots & x_{2,p} \\ \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n,1} & \ldots & x_{n,p} \end{bmatrix} \ \ \ \ \textbf{V = } \begin{bmatrix} \hat{\pi}_{1}(1 - \hat{\pi}_{1}) & 0 & \ldots & 0 \\ 0 & \hat{\pi}_{2}(1 - \hat{\pi}_{2}) & \ldots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \ldots & \hat{\pi}_{n}(1 - \hat{\pi}_{n}) \end{bmatrix}$$

where $x_{i,j}$ is the value of the $j$th variable for the $i$th observations and $\hat{\pi}_{i}$ represents the predicted probability for observation $i$.

The covariance matrix can then be found as: $\Sigma = \textbf{(X}^{T}\textbf{V}\textbf{X)}^{-1}$ and the standard error as $SE(x^T\beta) = \sqrt{Var(x^T\beta)}$

The 95% confidence intervals for the predicted probability can then be plotted as

enter image description here

Delta method confidence intervals

The approach is to compute the variance of a linear approximation of the function $F$ and use this to construct large sample confidence intervals.

$$ \text{Var}[F\mathbf{(x^T \hat \beta)}] \approx \nabla F^T \ \Sigma \ \nabla F $$

Where $\nabla$ is the gradient and $ \Sigma$ the estimated covariance matrix. Note that in one dimension:

$$\frac{\partial F(x\beta)}{\partial \beta} = \frac{\partial F(x\beta)}{\partial x\beta} \frac{\partial x\beta}{\partial \beta} = x f(x\beta)$$

Where $f$ is the derivative of $F$. This generalizes in the multivariate case

$$ \text{Var}[F\mathbf{(x^T \hat \beta)}] \approx f^T \ \mathbf{x^T} \ \Sigma \ \mathbf{x} \ f $$

In our case F is the logistic function (which we will denote $\pi(x^T\beta)$) whose derivative is

$$ \pi'(x^T\beta) = \pi (x^T\beta) (1 - \pi (x^T\beta) ) $$

We can now construct a confidence interval using the variance computed above.

$$ C.I. = [Pr(x\hat \beta) - z^* \sqrt{\text{Var}[ \pi(x \hat \beta) ]} \leq Pr(x\hat \beta) + z^* \sqrt{\text{Var}[ \pi(x \hat \beta) ]} ]$$

In vector form for the multivariate case

$$ C.I. = \mathbf{[\pi(x^T\hat \beta) \pm z^* \sqrt{ \left(\pi(x^T \hat \beta) (1 - \pi(x^T \hat \beta) ) \right)^T x^T \ \ \text{Var}[ \hat \beta] \ \ x \ \ \pi(x^T \hat \beta) (1 - \pi(x^T \hat \beta) ) ]}}$$

  • Note that $\mathbf{x}$ represent a single data point in $\mathbb{R}^{p+1}$, i.e. a single row of the design matrix $X$

enter image description here

A open ended conclusion

A look at the Normal QQ plots for both the probabilities and the negative log odds show that neither are normally distributed. Could this explain the difference ?

enter image description here



For most purposes the simplest way is probably best, as discussed in the context of a log transform on this page. Think about your dependent variable as being analyzed in the logit scale, with statistical tests performed and confidence intervals (CI) defined on that logit scale. The back transformation to odds ratio is simply to put those results into a scale that a reader might more readily grasp. This is also done, for example, in Cox survival analysis, where the regression coefficients (and the 95% CI) are exponentiated to obtain hazard ratios and their CI.


Your Answer

