Demystified: double robustness with nuisance parameters estimated at rate n-to-the-1/4
Judith J. Lok




Demystified: double robustness with nuisance parameters estimated at rate n-to-the-1/4
    Judith J. Lok
 Department of Mathematics and Statistics, Boston University
 jjlok@bu.edu
    
============================================================================================





§ ABSTRACT


Have you also been wondering what is this thing with double robustness and nuisance parameters estimated at rate n^1/4? It turns out that to understand this phenomenon one just needs the Middle Value Theorem (or a Taylor expansion) and some smoothness conditions. This note explains why under some fairly simple conditions, as long as the nuisance parameter θ∈ℝ^k is estimated at rate n^1/4 or faster, 1. the resulting variance of the estimator of the parameter of interest ψ∈ℝ^d does not depend on how the nuisance parameter θ is estimated, and 2. the sandwich estimator of the variance of ψ̂ ignoring estimation of θ is consistent.



§ INTRODUCTION


It is not uncommon that an estimator for a parameter ψ depends on nuisance parameters θ. In such settings, θ is often estimated in a first step. Some estimators for ψ are doubly robust: they depend on two nuisance parameters θ_1 and θ_2, and are consistent if one of the nuisance parameters θ_1 or θ_2 is consistently estimated, but not necessarily both.

Double robustness has been shown to often improve precision, and several efficient estimators that depend on more than one nuisance parameter have been shown to be doubly robust. Examples of this include doubly robust estimation of means from observational data (<cit.>), doubly robust estimation of (coarse) Structural Nested Mean Models (<cit.>), and multiply robust estimation of indirect and direct effects (<cit.>). The orthogonal moment functions from <cit.> are locally doubly robust (see their equation (2.4)), but beyond the scope of this note.

In order to obtain the efficiency gain from double robustness, it is advantageous to use flexible models to estimate θ. Flexible methods do not always estimate θ at rate √(n) (e.g., <cit.>). Fortunately, it often suffices to estimate θ at rate n^1/4 in order to obtain the efficiency gain, and if this is achieved, the variance of the resulting estimator ψ̂ does not depend on how the nuisance parameter θ is estimated. 

It does not take much more than the Mean Value Theorem (or a Taylor expansion) to understand this phenomenon. This note shows how this works for estimators ψ̂ based on smooth unbiased estimating equations. 



§ SETTING AND NOTATION


Henceforth, ψ^*∈ℝ^d is the true parameter of interest and θ^*∈ℝ^k is the true nuisance parameter. ψ̂ solves

    ℙ_n U(ψ,θ̂)=0,

where ℙ_n denotes the empirical average over i=1,…,n independent identically distributed observations, with

    EU(ψ^*,θ^*)=0

and U of dimension k, the dimension of ψ. Examples include Maximum Likelihood Estimation settings where ψ̂ solves the score equations, but this so-called Z-estimation is much more general; see e.g. <cit.>. 

Such ψ̂ is doubly robust if with θ=(θ_1,θ_2), ψ̂ solves unbiased estimating equations if θ_1 is consistently estimated and if θ_2 is consistently estimated, and not necessarily both; that is,

    EU(ψ^*,θ^*_1,θ_2)=0       and       EU(ψ^*,θ_1,θ^*_2)=0

for every θ_1 and θ_2.

This note assumes that θ is estimated at rate n^1/4 or faster:

    n^1/4(θ̂-θ^*)=O_P(1).




§ REGULARITY CONDITIONS


Throughout, this note assumes that regularity conditions hold so that U(ψ,θ) and EU(ψ,θ) depend smoothly enough on (ψ,θ). It also assumes that the order of differentiation and integration with respect to θ can be changed, so it is assumed that the support of the distribution of the observations does not depend on θ. 

It is also assumed that ψ is uniquely identified by equation (<ref>), so that

    .∂/∂ψ|_ψ^*EU(ψ,θ^*)

has an inverse. To simplify the exposition, it is assumed that it has already been proven that ψ̂ converges in probability to ψ^*.



§ DERIVATIONS BASED ON TAYLOR EXPANSIONS


Double robustness implies that

    E.∂/∂θ_p|_θ^* U_q(ψ^*,θ)=0,

where U_q is the qth component of U, 1≤ q≤ k∈ℕ.
This follows for the derivative with respect to θ_1 by taking the derivative with respect to θ_1 of EU(ψ^*,θ_1,θ_2^*), which equals zero because of equation (<ref>). Notice that this assumes that the support of the observations does not depend on θ, so that differentiation with respect to θ and integration can be interchanged. The same reasoning works for the derivative with respect to θ_2.

After estimating θ resulting in θ̂, ψ̂ solves equation (<ref>):

    0   =   ℙ_n U(ψ̂,θ̂)
       =   ℙ_n U(ψ^*,θ^*)
    +
    (.∂/∂(ψ,θ)|_(ψ̃,θ̃)ℙ_n U(ψ,θ))
    ([ ψ̂-ψ^*; θ̂-θ^* ])
       =   ℙ_n U(ψ^*,θ^*)
    +
    (.∂/∂ψ|_(ψ̃,θ̃)ℙ_n U(ψ,θ))
    (ψ̂-ψ^*)
          +(.∂/∂θ|_(ψ̃,θ̃)ℙ_n U(ψ,θ))(θ̂-θ^*)
       =   ℙ_n U(ψ^*,θ^*)
    +
    (.∂/∂ψ|_ψ̃ℙ_n U(ψ,θ̃))
    (ψ̂-ψ^*)
          +(.∂/∂θ|_θ̃ℙ_n U(ψ̃,θ))(θ̂-θ^*)

for some (ψ̃,θ̃) between (ψ̂,θ̂) and (ψ^*,θ^*), possibly different in each row (from the Middle Value Theorem applied to each entry in the vector separately). Equation (<ref>) implies that

    (ψ̂-ψ^*)=
          (.∂/∂ψ|_ψ̃ℙ_n U(ψ,θ̃))^-1(ℙ_n U(ψ^*,θ^*)-.∂/∂θ|_θ̃ℙ_n U(ψ̃,θ)(θ̂-θ^*)).

The derivations below show that if equation (<ref>) holds,
the last term in equation (<ref>) multiplied by √(n) converges in probability to zero.

First, we show that equation (<ref>) implies that

    n^1/4(ψ̂-ψ^*)
    →^P0.

Notice that as usual (see for example <cit.> Lemma A.6.1), under the usual regularity conditions (mainly differentiability conditions), since (ψ̃,θ̃)→^P (ψ^*,θ^*),

    .∂/∂ψ|_ψ̃ℙ_n U(ψ,θ̃)→^P E.∂/∂ψ|_ψ^*U(ψ,θ^*)

and

    .∂/∂θ|_θ̃ℙ_n U(ψ̃,θ)→^P E.∂/∂θ|_θ^* U(ψ^*,θ)=0,

where the equality follows from the double robustness equation (<ref>).
Combining with equation (<ref>), it follows that the last term in equation (<ref>) multiplied by n^1/4 converges in probability to zero. Combining with the Central Limit Theorem on ℙ_n U(ψ^*,θ^*), equation (<ref>) shows that equation (<ref>) implies equation (<ref>).

To show that the last term in equation (<ref>) multiplied by √(n) converges in probability to zero, we next consider each

    n^1/4.∂/∂θ_p|_θ̃ℙ_n U_q(ψ̃,θ)

separately, were U_q is the qth component of U. We show that the quantity in equation (<ref>) converges in probability to zero when equation (<ref>) holds.

    .∂/∂θ_p|_θ̃ℙ_n U_q(ψ̃,θ)
    =.∂/∂θ_p|_θ^*ℙ_n U_q(ψ̃,θ)+.∂/∂θ|_θ̇∂/∂θ_pℙ_n U_q(ψ̃,θ)(θ̃-θ^*)

because of the Middle Value Theorem, for some θ̇ between θ̃ and θ^*.
As usual, under the usual regularity conditions, since (ψ̃,θ̇)→^P (ψ^*,θ^*),

    .∂/∂θ|_θ̇∂/∂θ_pℙ_n U_q(ψ̃,θ)
    →^P E.∂/∂θ|_θ^*∂/∂θ_p U_q(ψ^*,θ)=0,

where the equality follows from the same reasoning as equation (<ref>).
Combining equations (<ref>) and (<ref>)  implies that n^1/4 times the last term in equation (<ref>) converges in probability to zero. 

For the first term on the right hand side of equation (<ref>), because of the Middle Value Theorem,

    .∂/∂θ_p|_θ^*ℙ_n U_q(ψ̃,θ)
       =   .∂/∂θ_p|_θ^*ℙ_n U_q(ψ^*,θ)+(.∂/∂ψ|_ψ̇.∂/∂θ_p|_θ^*ℙ_n U_q(ψ,θ))(ψ̃-ψ^*),

for some ψ̇ between ψ̃ and ψ, possibly different in each row.
As usual, under the usual regularity conditions,

    .∂/∂ψ|_ψ̇.∂/∂θ_p|_θ^*ℙ_n U_q(ψ,θ)→^P E.∂/∂ψ|_ψ^*.∂/∂θ_p|_θ^* U_q(ψ,θ).

Moreover, from equation (<ref>), the Central Limit Theorem implies that

    √(n).∂/∂θ_p|_θ^*ℙ_n U_q(ψ^*,θ)→^ D N(0,E((.∂/∂θ_p|_θ^*U_q(ψ^*,θ))^2)).

Combining equations (<ref>), (<ref>), (<ref>), and (<ref>) leads to

    n^1/4.∂/∂θ_p|_θ^*ℙ_n U_q(ψ̃,θ)→^P0.

Combining with equations (<ref>), (<ref>), and (<ref>), it follows that

    n^1/4.∂/∂θ_p|_θ̃ℙ_n U_q(ψ̃,θ)→^P 0.


Combining equations (<ref>), (<ref>), and (<ref>), it follows that √(n) times the last term in equation (<ref>) converges in probability to zero. 



§ CONCLUSION


It follows that if equation (<ref>) and the usual regularity conditions from Section <ref> hold, for ψ̂ of the form of Section <ref> equation (<ref>),

    √(n)(ψ̂-ψ^*)
       =   (.∂/∂ψ|_ψ̃ℙ_n U(ψ,θ̃))^-1√(n)ℙ_n U(ψ^*,θ^*)+o_P(1)
       →^ D    N(0,V(ψ^*,θ^*)),

from the Central Limit Theorem, equation (<ref>), and Slutsky's Theorem, with

    V(ψ^*,θ^*)=(E.∂/∂ψ|_ψ^*U(ψ,θ^*))^-1
    E(U(ψ^*,θ^*)^2)(E.∂/∂ψ|_ψ^*U(ψ,θ^*))^-1⊤.

That is, estimating θ leads to the same variance as plugging in the true but usually unknown θ, and the sandwich estimator for the variance of ψ̂ ignoring estimation of θ is consistent, all provided that ψ̂ is doubly robust and θ is estimated at rate n^1/4 or faster.

chicago


