# Complete Subset Averaging with Many Instruments††thanks: We would like to
thank Graham Elliott, Ryo Okui and the seminar participants at Seoul National
University, ISNPS 2018, CESG 2018, MEG 2018, BK21plus International Conference
on Econometrics for helpful comments. Shin gratefully acknowledges financial
support received from the Social Sciences and Humanities Research Council of
Canada (SSHRC-435-2018-0275) and from Australia Research Council
(DP170100987). This work was made possible by the facilities of the Shared
Hierarchical Academic Research Computing Network (SHARCNET:www.sharcnet.ca)
and Compute/Calcul Canada.

Seojeong Lee
University of New South Wales  Address: Level 4 Business School building, UNSW
Sydney, NSW 2052, Australia. E-mail<EMAIL_ADDRESS>Youngki Shin
McMaster University  Address: 1280 Main St. W., Hamilton, ON L8S 4L8, Canada.
Email<EMAIL_ADDRESS>
###### Abstract

We propose a two-stage least squares (2SLS) estimator whose first stage is the
equal-weight average over a complete subset with $k$ instruments among $K$
available, which we call the complete subset averaging (CSA) 2SLS. The
approximate mean squared error (MSE) is derived as a function of the subset
size $k$ by the Nagar (1959) expansion. The CSA-2SLS estimator is obtained by
choosing $k$ minimizing the sample counterpart of the approximate MSE. We show
that this method achieves asymptotic optimality among the class of estimators
with different subset sizes. A feature of equal-weight averaging is that all
the instruments are used. To deal with averaging over irrelevant instruments,
we generalize the approximate MSE under the presence of a possibly growing set
of irrelevant instruments, which suggests to choose a smaller $k$ than
otherwise. An extensive simulation experiment shows potentially huge
improvement in the bias and the MSE by using the CSA-2SLS when instruments are
correlated with each other and there exists large endogeneity. As an empirical
illustration, we estimate the logistic demand function in Berry, Levinsohn,
and Pakes (1995) and find the estimated coefficient value is better supported
by economic theory than other IV estimators.

Keywords: two-stage least squares, endogeneity, model averaging, equal-weight.
JEL Classification: C13, C14, C26.

##  1 Introduction

Instrumental variables (IV) estimators are commonly used to estimate the
parameters associated with endogenous variables in economic models. The two-
stage least squares (2SLS) is the most popular IV estimator for linear
regression models with endogenous regressors. While the 2SLS estimator is
usually applied to just-identified models where the number of instruments is
the same as the number of endogenous variables, there are many applications
where more instruments are used than the number of endogenous variables (over-
identified). In particular, when the instrument set is a large set of dummy
variables or is constructed by interacting the original instruments with
exogenous variables, the total number of instruments can be quite large. For
example, Angrist and Krueger (1991) use as many as 180 instruments for one
endogenous variable to get a tighter confidence interval for the structural
parameter than using a smaller set of instruments.

Although using a large set of instruments can improve efficiency, there is a
trade-off in terms of increased bias in the point estimate (Bekker, 1994).
Motivated by the trade-off relationship, Donald and Newey (2001) propose to
select the number of instruments by minimizing the approximate mean squared
error (MSE) of IV estimators. Kuersteiner and Okui (2010) propose an IV
estimator that applies the model averaging approach of Hansen (2007) in the
first stage and show that the selected weights attain optimality in the sense
of Li (1987). Okui (2011) proposes to average the first stage using shrinkage
to obtain the shrinkage IV estimators.

In this paper we propose a 2SLS estimator whose first stage is the equal-
weight average over a complete subset with $k$ instruments among the total of
$K$ instruments. We call this estimator the complete subset averaging (CSA)
2SLS estimator. Our approach differs from the important existing work in the
many instruments literature: Unlike Donald and Newey (2001), the CSA-2SLS is
based on model averaging in the first-stage rather than model selection and
does not require ordering of the instruments; Unlike Kuersteiner and Okui
(2010), it does not require weight estimation; Unlike Okui (2011), it does not
require to specify the main set of instruments a priori.

The main theoretical contribution of this paper is three-fold. First, we
derive the approximate MSE for our CSA-2SLS estimator by the Nagar (1959)
expansion. It is technically challenging due to the fact that the average of
non-nested projection matrices is not idempotent. In contrast, the existing
literature usually assume nested models. The derived formula shows the bias-
variance trade-off and some interesting features which will be discussed in
detail in the following sections. Second, we generalize the approximate MSE
formula when irrelevant instruments exist whose number grows as the sample
size increases. The generalized formula shows a penalty term for the subset
size $k$ which increases with the proportion of irrelevant instruments. This
suggests that the choice of a smaller $k$ than otherwise would be desirable in
the presence of irrelevant instruments. Third, we prove that the CSA-2SLS
estimator with the subset size minimizing the sample approximate MSE is
asymptotically optimal in the sense that it attains the lowest possible MSE
among the class of the CSA-2SLS estimators with different subset sizes. Our
optimality proof is based on Li (1987) and Whittle (1960).

Our approach is motivated by the following observations. First, a set of
instruments in economic applications is usually correlated with each other,
often by construction. The model selection approach by Donald and Newey (2001)
would work well if the instruments are uncorrelated with each other and only a
fraction of instruments matters (provided that the ordering is correct). Thus,
we believe that the model averaging approach in the first-stage is more
appropriate. Second, in the model averaging, estimating weights can cause
finite sample efficiency loss, especially when the dimension of the weight
vector is large. In the forecasting literature it is not surprising to see
equal-weight averaging can outperform other sophisticated optimal weighting
schemes, e.g., see Clemen (1989), Stock and Watson (2004), and Smith and
Wallis (2009). Bootstrap aggregating (known as bagging; Breiman (1996)) is
another example of equal-weight model averaging and is a popular method in the
machine learning literature.

It is worth emphasizing the important work by Elliott, Gargano, and Timmermann
(2013, 2015) who propose the equal-weight complete subset regressions in the
forecasting context. They demonstrate an excellent performance of the complete
subset regression relative to competing methods such as ridge regression, the
Lasso, the Elastic Net, bagging, and the Bayesian model averaging. We build on
their idea of the complete subset regression to provide a formal theoretical
justification of the CSA-2SLS estimator with extensive Monte Carlo
simulations. Indeed, we find that the CSA-2SLS exhibits potentially huge gains
in terms of the bias and the MSE relative to the existing methods especially
when the instruments are correlated and there exists large endogeneity.

There are two limitations to be noted. First, conditionally homoskedastic
errors are assumed in the derivation of the approximate MSE, similarly in
Donald and Newey (2001), Kuersteiner and Okui (2010), Hansen (2007), and Okui
(2011). This is required to obtain the explicit order of the higher-order
terms in the expansion of the MSE, which allows one-to-one comparison with the
existing literature. Donald, Imbens, and Newey (2009) derive the approximate
MSE under heteroskedasticity for the efficient generalized method of moment
(GMM) and the generalized empirical likelihood (GEL) estimators at the expense
of more complicated expressions. Also note that the 2SLS is no longer
efficient under heteroskedasticity.

Second, we focus on the 2SLS estimator because of its utmost popularity among
applied researchers. On the other hand, the limited information maximum
likelihood (LIML) estimator has gained a considerable attention recently
mainly due to its theoretical advantage over the 2SLS having a smaller bias
under the many instruments asymptotics (Donald and Newey (2001)). Our
simulation result shows that the CSA-2SLS estimator exhibits very small bias
across different specifications of the data-generating processes. Thus,
although our analysis can be extended to the $k$-class estimators, including
the LIML and the bias-corrected 2SLS, we expect that improvements would be
smaller compared to the 2SLS. To maintain our focus on the new averaging
method, we defer these extensions to future research.

Finally, we summarize related literature. The model averaging approach becomes
prevalent in the econometrics literature. Hansen (2007) shows that the weight
choice based on the Mallows criterion achieves optimality. Hansen and Racine
(2012) propose the jackknife model averaging which allows heteroskedasticity.
Ando and Li (2014) present a model averaging for high-dimensional regression.
Ando and Li (2017) and Zhang, Yu, Zou, and Liang (2016) consider the class of
generalized linear models to show the optimality under the Kullback-Leibler
loss function. Zhang and Yu (2018) propose a model averaging in the spatial
autoregressive models. Kitagawa and Muris (2016) propose the propensity score
model averaging estimator for the average treatment effects for treated. Lee
and Zhou (2015) propose a model averaging approach over complete subsets in
the second stage IV regression.

Our approach is different from the many weak instruments asymptotics in Chao
and Swanson (2005), Stock and Yogo (2005), Han and Phillips (2006), Andrews
and Stock (2007), and Hansen, Hausman, and Newey (2008), where the first-stage
coefficients are modeled to converge to zero as the sample size grows.
Alternative estimators under hetoroskedasticity and many instruments are
proposed by Hausman, Lewis, Menzel, and Newey (2011) and Hausman, Newey,
Woutersen, Chao, and Swanson (2012). Kuersteiner (2012) extends the instrument
selection criteria of Donald and Newey (2001) to the time series setting and
proposes a GMM estimator using lags as instruments. Kang (2018) derives the
approximate MSE of IV estimators with locally invalid instruments. Antoine and
Lavergne (2014) and Escanciano (2017) propose estimators free from choice
variables by adopting the continuum of unconditional moment condition in the
first stage.

The remainder of the paper is organized as follows. Section 2 describes the
model and proposes the CSA-2SLS estimator. Section 3 derives the approximate
MSE formula of the estimator and investigates its properties. We also show the
asymptotic optimality result as well as the implementation procedure. Section
4 extends the result by allowing that the number of irrelevant instruments
increases. Section 5 studies the finite sample properties via Monte Carlo
simulations and Section 6 provides an empirical illustration. The appendices
contain the proofs and additional simulation results.

##  2 Model and Estimator

We follow the setup of Donald and Newey (2001) and Kuersteiner and Okui
(2010). The model is

$\displaystyle y_{i}$ $\displaystyle=$ $\displaystyle
Y_{i}^{\prime}\beta_{y}+x_{1i}^{\prime}\beta_{x}+\varepsilon_{i}=X_{i}^{\prime}\beta+\varepsilon_{i},$
(1) $\displaystyle X_{i}$ $\displaystyle=$
$\displaystyle\left(\begin{array}[]{c}Y_{i}\\\
x_{1i}\end{array}\right)=f(z_{i})+u_{i}=\left(\begin{array}[]{c}E[Y_{i}|z_{i}]\\\
x_{1i}\end{array}\right)+\left(\begin{array}[]{c}\eta_{i}\\\
0\end{array}\right),~{}~{}i=1,\ldots,N,$ (8)

where $y_{i}$ is a scalar outcome variable, $Y_{i}$ is a $d_{1}\times 1$
vector of endogenous variables, $x_{1i}$ is a $d_{2}\times 1$ vector of
included exogenous variables, $z_{i}$ is a vector of exogenous variables
(including $x_{1i}$), $\varepsilon_{i}$ and $u_{i}$ are unobserved random
variables with finite second moments which do not depend on $z_{i}$, and
$f(\cdot)$ is an unknown function of $z$. Let $f_{i}=f(z_{i})$ and
$d=d_{1}+d_{2}$. The second equation represents a nonparametric reduced form
relationship between $Y_{i}$ and the exogenous variables $z_{i}$, with
$E[\eta_{i}|z_{i}]=0$ by construction. Define the $N\times 1$ vectors
$y=(y_{1},\ldots,y_{N})^{\prime}$,
$\varepsilon=(\varepsilon_{1},\ldots,\varepsilon_{N})^{\prime}$, and the
$N\times d$ matrices $X=(X_{1},\ldots,X_{N})^{\prime}$,
$f=(f_{1},\ldots,f_{N})^{\prime}$, and $u=(u_{1},\ldots,u_{N})^{\prime}$.

The set of instruments has the form
$Z_{K,i}\equiv(\psi_{1}(z_{i}),\ldots,\psi_{K}(z_{i}),x_{1i})^{\prime}$, where
$\psi_{k}$’s are functions of $z_{i}$ such that $Z_{K,i}$ is a
$(K(N)+d_{2})\times 1$ vector of instruments. The total number of instruments
$K(N)$ increases as $N\rightarrow\infty$ but we suppress the dependency on $N$
and write $K$ unless we need to express the dependence of $K$ on $N$
explicitly. Define its matrix version as
$Z_{K}=(Z_{K,1},\ldots,Z_{K,N})^{\prime}$. To define the complete subset
averaging estimator, consider a subset of $k$ excluded instruments. To avoid
confusion, we will use “instruments” to refer to excluded instruments
throughout the paper. The number of subsets with $k$ instruments is

$\binom{K}{k}=\frac{K!}{k!(K-k)!}.$

A complete subset with size $k$ is the collection of all these subsets. Let
$M(K,k)=\binom{K}{k}$, which is the number of models given $K$ and $k$. For
brevity, the dependence of $M$ on $K$ and $k$ will be suppressed unless it is
necessary. For any model $m$ with $k$ instruments, let $Z_{m,i}^{k}$ be an
$(k+d_{2})\times 1$ vector of subset instruments including $x_{1i}$ where
$d_{1}\leq k\leq K$ and $Z_{m}^{k}=(Z_{m,1}^{k},\ldots,Z_{m,N}^{k})^{\prime}$
be an $N\times(k+d_{2})$ matrix for $m=1,\ldots,M$. For each $m$, the first
stage equation can be rewritten as

$X_{i}=\Pi_{m}^{k^{\prime}}Z_{m,i}^{k}+u_{m,i}^{k},~{}~{}i=1,\ldots,N,$ (9)

or equivalently,

$X=Z_{m}^{k}\Pi_{m}^{k}+u_{m}^{k},$ (10)

where $\Pi_{m}^{k}$ is the $(k+d_{2})\times d$ dimensional projection
coefficient matrix for model $m$ with $k$ instruments, $u_{m,i}^{k}$ is the
projection error, and $u_{m}^{k}=(u_{m,1}^{k},\ldots,u_{m,N}^{k})^{\prime}$.
The projection coefficient matrix is estimated by

$\widehat{\Pi}_{m}^{k}=(Z_{m}^{k^{\prime}}Z_{m}^{k})^{-1}Z_{m}^{k^{\prime}}X,$
(11)

and the average fitted value of $X$ over complete subset $k$ becomes

$\widehat{X}=\frac{1}{M}\sum_{m=1}^{M}Z_{m}^{k}\widehat{\Pi}_{m}^{k}=\frac{1}{M}\sum_{m=1}^{M}Z_{m}^{k}(Z_{m}^{k^{\prime}}Z_{m}^{k})^{-1}Z_{m}^{k^{\prime}}X\equiv\frac{1}{M}\sum_{m=1}^{M}P_{m}^{k}X\equiv
P^{k}X,$ (12)

where
$P_{m}^{k}=Z_{m}^{k}(Z_{m}^{k^{\prime}}Z_{m}^{k})^{-1}Z_{m}^{k^{\prime}}$ and
$P^{k}=\frac{1}{M}\sum_{m=1}^{M}P_{m}^{k}$. The complete subset averaging
(CSA) 2SLS estimator with a given $k$ is now defined as

$\widehat{\beta}=(\widehat{X}^{\prime}X)^{-1}\widehat{X}^{\prime}y=(X^{\prime}P^{k}X)^{-1}X^{\prime}P^{k}y.$
(13)

The matrix $P^{k}$ is the average of projection matrices. We call this matrix
as the complete subset averaging $P$ (CSA-$P$) matrix. Note that the CSA-$P$
matrix is symmetric but not idempotent in general.

Therefore, we can estimate the model by the CSA-2SLS estimator for a given
$k$. We will deliberate on the choice of $k$ in the next section and close
this section by characterizing the CSA-2SLS estimator and the CSA-$P$ matrix
in a broader context.

The CSA-2SLS estimator can be interpreted as the minimizer of the average of
2SLS criterion functions. For each subset of instruments $Z_{m,i}^{k}$, the
corresponding moment condition is

$E[Z_{m,i}^{k}\varepsilon_{i}]=E[Z_{m,i}^{k}(y_{i}-X_{i}^{\prime}\beta)]=0.$
(14)

The standard 2SLS estimator given the moment condition (14) minimizes

$(y-X\beta)^{\prime}Z_{m}^{k}\left(Z_{m}^{k^{\prime}}Z_{m}^{k}\right)^{-1}Z_{m}^{k^{\prime}}(y-X\beta).$
(15)

This equation is the GMM criterion with the weight matrix
$(Z_{m}^{k^{\prime}}Z_{m}^{k})^{-1}$. Conventional model average estimators
are based on a weighted average of $\widehat{\beta}^{k}_{m}$, the minimizer of
(15), over different models. For this type of model averaging estimators, see
Hansen (2007) for the OLS estimator and see Lee and Zhou (2015) for the 2SLS
estimator.

In contrast, the CSA-2SLS estimator minimizes the average of (15) over
different models directly:

$\widehat{\beta}=\operatorname*{arg\,min}_{\beta}~{}\sum_{m=1}^{M}\left((y-X\beta)^{\prime}Z_{m}^{k}\left(Z_{m}^{k^{\prime}}Z_{m}^{k}\right)^{-1}Z_{m}^{k^{\prime}}(y-X\beta)\right).$
(16)

The optimal model averaging 2SLS estimator of Kuersteiner and Okui (2010) can
be interpreted similarly as the minimizer of the average of 2SLS criterion
functions with data-dependent weights.

There are special cases when the CSA-2SLS estimator coincides exactly with the
2SLS estimator using all the instruments. The two estimators are the same if
(i) $k=K$ and (ii) the instruments are mutually orthogonal. We elaborate on
the second case as it is not trivial. Note that the projection matrix of the
orthogonal instruments is equal to the sum of the projection matrices of each
of the instruments. Thus, the projection matrix of the 2SLS estimator,
$P_{all}$, becomes

$\displaystyle
P_{all}=\widetilde{P}^{1}_{1}+\widetilde{P}^{1}_{2}+\cdots+\widetilde{P}^{1}_{K},$

where $\widetilde{P}^{1}_{j}$ for $j=1,...,K$ is the projection matrix based
on the orthogonal instrument $j$ with the subset size 1. Now consider the
CSA-$P$ matrix with subset size $k$ constructed from the orthogonal
instruments:

$\displaystyle P^{k}\equiv\frac{1}{M(K,k)}\left(P^{k}_{1}+\cdots
P^{k}_{M}\right)=\frac{\binom{K-1}{k-1}}{M(K,k)}\left(\widetilde{P}^{1}_{1}+\cdots+\widetilde{P}^{1}_{K}\right)=\frac{k}{K}P_{all}.$

Since the $k/K$’s in the numerator and the denominator are cancelled out in
Equation (13), the CSA-2SLS estimator becomes identical to the 2SLS estimator
for any $k$.

The identity result does not hold in general when instruments are correlated,
although the correlated instruments can be always orthogonalized without
affecting the column space they span. We illustrate this point by a simple
example with $K=2$ and $k=1$. Let $(Z_{1},Z_{2})$ be the vector of instruments
and $(\widetilde{Z}_{1},\widetilde{Z}_{2})$ be the corresponding vector of
orthogonalized instruments. Then,

$\displaystyle
P^{1}=\frac{1}{2}\left(P^{1}_{1}+P^{1}_{2}\right)\neq\frac{1}{2}\left(\widetilde{P}^{1}_{1}+\widetilde{P}^{1}_{2}\right)\equiv\frac{1}{2}\widetilde{P}^{2}=\frac{1}{2}P^{2},$

where $\widetilde{P}$ denotes a generic projection matrix of the
orthogonalized instruments. Therefore, the CSA-2SLS estimator using $P^{1}$
does not give the same estimate as that using $\frac{1}{2}P^{2}$, the 2SLS
estimator.

##  3 Subset Size Choice and Optimality

In this section we derive the approximate MSE of the CSA-2SLS estimator by the
Nagar (1959) expansion collecting the leading terms that depend on $k$. The
subset size $k$ is chosen to minimize the sample counterpart of the
approximate MSE whose formula is provided. Finally we prove the optimality of
the chosen subset size in the sense of Li (1987).

###  3.1 Approximate MSE

In our analysis, the CSA-$P$ matrix $P^{k}$ plays an important role. Since
$P^{k}$ is not idempotent unless $k=K(N)$, we cannot directly apply the
existing technique in Donald and Newey (2001) or Kuersteiner and Okui (2010)
to our analysis. We first list regularity conditions. Let
$\|A\|=\sqrt{\text{tr}(A^{\prime}A)}$ denote the Frobenius norm for a matrix
$A$.

######  Assumption 1.

  1. (i)

$\\{y_{i},X_{i},z_{i}\\}$ are i.i.d. with finite fourth moment and
$E[\varepsilon_{i}^{2}]=\sigma_{\varepsilon}^{2}>0$.

  2. (ii)

$E[\varepsilon_{i}|z_{i}]=0$ and $E[u_{i}|z_{i}]=0$.

  3. (iii)

Let $u_{ia}$ be the $a$th element of $u_{i}$. Then
$E[\varepsilon_{i}^{r}u_{ia}^{s}|z_{i}]$ are constant and bounded for all $a$
and all $r,s\geq 0$ and $r+s\leq 4$.

  4. (iv)

$f_{i}$ is bounded.

  5. (v)

$\overline{H}=Ef_{i}f_{i}^{\prime}$ exists and is nonsingular.

  6. (vi)

The excluded exogenous instruments $\psi_{1}(z_{i}),\cdots,\psi_{K}(z_{i})$
are not mutually orthogonal.

  7. (vii)

For each $k\geq d$ and all $m=1,\ldots,M$, $Z_{m}^{k^{\prime}}Z_{m}^{k}$ is
nonsingular with probability approaching one.

  8. (viii)

Let $P_{ii}^{k}$ denote the $i$th diagonal element of $P^{k}$. Then
$\max_{i\leq N}P_{ii}^{k}\xrightarrow{p}0$ as $N\rightarrow\infty$.

######  Assumption 2.

For each $k$ there exists $\Pi_{m}^{k}$ such that

$\frac{1}{M}\sum_{m=1}^{M}E\|f(z_{i})-\Pi_{m}^{k^{\prime}}Z_{m,i}^{k}\|^{2}\rightarrow
0$

as $k\rightarrow\infty$.

Assumption 1 collects standard moment and identification conditions similar to
those of Donald and Newey (2001) and Kuersteiner and Okui (2010), except for
Assumption 1(vi), which excludes the trivial special case that the CSA-2SLS is
identical to the 2SLS with all of the instruments regardless of $k$.
Assumption 2 requires that there be a sequence of complete subsets
approximating the unknown reduced form $f(z_{i})$ arbitrarily well for large
enough $k$. The condition allows some models in a complete subset with $k$
instruments approximate $f(z_{i})$ poorly but the proportion of those models
becomes negligible as $k$ increases. Note that the condition coincides with
Assumption 2(ii) of Donald and Newey (2001) if the sequence, $\\{k\\}$, is set
to $k=K(N)$.

Define $H=f^{\prime}f/N$. The MSE can be written as

$\displaystyle N(\widehat{\beta}-\beta)(\widehat{\beta}-\beta)^{\prime}$
$\displaystyle=$ $\displaystyle\widehat{Q}(k)+\widehat{r}(k),$ (17)
$\displaystyle E\left[\widehat{Q}(k)|Z\right]$ $\displaystyle=$
$\displaystyle\sigma_{\varepsilon}^{2}H^{-1}+S(k)+T(k),$ (18)

where $\widehat{r}(k)$ and $T(k)$ are terms of smaller order in probability
than those in $S(k)$. The term $\sigma_{\varepsilon}^{2}H^{-1}$ is the first-
order asymptotic variance under homoskedasticity. Theorem 1 provides the
formula of $S(k)$.

######  Theorem 1.

If Assumptions 1-2 are satisfied, $k^{2}/N\rightarrow 0$, and
$E[u_{i}\varepsilon_{i}|z_{i}]=\sigma_{u\varepsilon}\neq 0$, then, for the
CSA-2SLS estimator, the equations (17)-(18) are satisfied with

$S(k)=H^{-1}\left[\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}\frac{k^{2}}{N}+\sigma_{\varepsilon}^{2}\frac{f^{\prime}(I-P^{k})(I-P_{f})(I-P^{k})f}{N}\right]H^{-1},$
(19)

where $P_{f}=f(f^{\prime}f)^{-1}f^{\prime}$.

The first term of $S(k)$ (ignoring pre- and post-multiplied $H^{-1}$),
$\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}k^{2}/N$, corresponds to
the bias and is similar to that of Donald and Newey (2001). The second term is
different from the usual higher-order variance in similar expansions in the
literature and thus deserves attention.

Let $V(k)=f^{\prime}(I-P^{k})(I-P_{f})(I-P^{k})f/N$. In Donald and Newey
(2001), the higher-order variance term, which takes the form of
$f^{\prime}(I-P^{K})f/N$ where $P^{K}$ is the projection matrix consisting of
$K$ instruments, decreases with $K$. This gives the bias-variance trade-off in
their expression. In contrast, $V(k)$ has a different form from the usual
higher-order variance (see the term, $(I-P_{f})$, in the middle) and the
monotone decrease with respect to $k$ is not always guaranteed. Since
$I-P_{f}$ is idempotent, we can write

$V(k)=\frac{u^{k^{\prime}}(I-P_{f})(I-P_{f})u^{k}}{N}$ (20)

where $u^{k}\equiv(I-P^{k})f$. There exist two factors that force to move
$V(k)$ into opposite directions as $k$ varies. First, the norm of $u^{k}$
decreases as $k$ increases since $P^{k^{*}}$ for $k^{*}>k$ is the average of
the projection matrices on the space spanned by a larger set of instruments.
This is the effect of decreasing higher-order variance with respect to $k$ as
in Donald and Newey (2001). Second, $P^{k}$ can cause less shrinkage for
$P^{k}f$ for a larger $k$, which makes the norm of $(I-P_{f})u^{k}$ larger.
Note that

$(I-P_{f})u^{k}=(I-P_{f})(f-P^{k}f)=f-P_{f}f-P^{k}f+P_{f}P^{k}f=-(I-P_{f})P^{k}f$
(21)

and that $P^{k}f$ with less shrinkage lies farther away from the space spanned
by $f$ (see, e.g. Figure 1). Therefore, if the second factor dominates the
first then $V(k)$ increases along with $k$. In Example 1 and Figure 1 below,
we provide a simple example.

  * Example 1

Illustrated in Figure 1. Let $N=3$ and $K=2$. Suppose that
$f=(2,2,1)^{\prime}$, $Z_{1}=(1,0,0)^{\prime}$, $Z_{2}=(0,1,0)^{\prime}$, and
$Z=[Z_{1},Z_{2}]$. Since
$P^{1}=(Z_{1}(Z_{1}^{\prime}Z_{1})^{-1}Z_{1}^{\prime}+Z_{2}(Z_{2}^{\prime}Z_{2})^{-1}Z_{2}^{\prime})/2$
and $P^{2}=Z(Z^{\prime}Z)^{-1}Z^{\prime}$, we calculate
$u^{1}=(I-P^{1})f=(1,1,1)$ and $u^{2}=(I-P^{2})f=(0,0,1)^{\prime}$. Thus,
$\|u^{2}\|<\|u^{1}\|$. However, it is clear from the figure below that $u^{1}$
is closer to $f$ than $u^{2}$. Indeed, we have $\|(I-P_{f})u^{1}\|\approx
0.47$ and $\|(I-P_{f})u^{2}\|\approx 0.94$.

Figure 1: Graphical Illustration of Example 1

$f_{1}=f_{2}$$f_{3}$$f=(2,2,1)$$P^{2}f=(2,2,0)$$u^{2}$$P^{1}f=(1,1,0)$$u^{1}$$u^{2}$$u^{1}$$(I-P_{f})u^{1}$$(I-P_{f})u^{2}$

[ht]

  * •

Note: The horizontal axis represents the $45^{\circ}$ line on the coordinate
plane of $(f_{1},f_{2})$. Note that the length of the vector $(I-P_{f})u^{k}$
increases as $k$ increases from 1 to 2 in this example.

If there are more than one endogenous variable, then we choose $k$ that
minimizes a linear combination of the MSE,
$S_{\lambda}(k)\equiv\lambda^{\prime}S(k)\lambda$ for a user-specified
$\lambda$. However, for a model containing only one endogenous variable, the
choice of $\lambda$ is irrelevant. To see this, first observe that
$\sigma_{u\varepsilon}=\sigma_{\eta\varepsilon}e_{1}$ where $e_{1}$ is the
first unit vector. In addition, the components of $f$ that are included in the
main equation lie on the column space spanned by the instruments (including
the included exogenous variables) so that

$\displaystyle f^{\prime}(I-P^{k})(I-P^{k})f$
$\displaystyle=\overline{Y}^{\prime}(I-P^{k})(I-P^{k})\overline{Y}e_{1}e_{1}^{\prime},$
$\displaystyle f^{\prime}(I-P^{k})P_{f}(I-P^{k})f$
$\displaystyle=\left(\overline{Y}^{\prime}(I-P^{k})\overline{Y}\right)^{2}\left(e_{1}^{\prime}H^{-1}e_{1}\right)\frac{1}{N}e_{1}e_{1}^{\prime},$

where $\overline{Y}=(E[Y_{1}|z_{1}],\ldots,E[Y_{N}|z_{N}])^{\prime}$. Thus we
can write

$\lambda^{\prime}S(k)\lambda=\left(\lambda^{\prime}H^{-1}e_{1}\right)^{2}\left[\sigma_{\eta\varepsilon}^{2}\frac{k^{2}}{N}+\sigma_{\varepsilon}^{2}\left(\frac{\overline{Y}^{\prime}(I-P^{k})(I-P^{k})\overline{Y}}{N}-\left(\frac{\overline{Y}^{\prime}(I-P^{k})\overline{Y}}{N}\right)^{2}e_{1}^{\prime}H^{-1}e_{1}\right)\right],$

which shows that the minimization problem with a single endogenous variable
does not depend on $\lambda$.

###  3.2 Implementation and Optimality

We first introduce notation to construct the sample counterpart of the
approximate MSE. Let $\widetilde{\beta}$ be a preliminary estimator that is
fixed across different values of $k$ and
$\widetilde{\varepsilon}=y-X\widetilde{\beta}$. Let $\widetilde{f}$ be an
estimate of $f$. The residual matrix is denoted by
$\widetilde{u}=X-\widetilde{f}$ and
$\widetilde{u}=(u_{1},u_{2},\ldots,u_{N})^{\prime}$ where $\widetilde{u}_{i}$
is a $d\times 1$ vector. Define
$\widetilde{H}=\widetilde{f}^{\prime}\widetilde{f}/N$,
$\widetilde{\sigma}_{\varepsilon}^{2}=\widetilde{\varepsilon}^{\prime}\widetilde{\varepsilon}/N$,
$\widetilde{\sigma}_{u\varepsilon}=\widetilde{u}^{\prime}\widetilde{\varepsilon}/N$,
$\widetilde{\sigma}_{\lambda\varepsilon}=\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}\widetilde{\sigma}_{u\varepsilon}$,
and $\widetilde{\Sigma}_{u}=\widetilde{u}^{\prime}\widetilde{u}/N$. The
feasible criterion function for $S_{\lambda}(k)$ is defined as below:

$\displaystyle\widehat{S}_{\lambda}(k)$
$\displaystyle=\widetilde{\sigma}_{\lambda\varepsilon}^{2}\frac{k^{2}}{N}+\widetilde{\sigma}_{\varepsilon}^{2}\left[\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}\widetilde{e}_{f}^{k}\widetilde{H}^{-1}\widetilde{\lambda}-\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}\widetilde{\xi}_{f}^{k}\widetilde{H}^{-1}\widetilde{\xi}_{f}^{k}\widetilde{H}^{-1}\widetilde{\lambda}\right],$

where

$\displaystyle\widetilde{e}_{f}^{k}$
$\displaystyle=\frac{X^{\prime}(I-P^{k})^{2}X}{N}+\widetilde{\Sigma}_{u}\left(\frac{2k-\text{tr}((P^{k})^{2})}{N}\right),$
$\displaystyle\widetilde{\xi}_{f}^{k}$
$\displaystyle=\frac{X^{\prime}(I-P^{k})X}{N}+\widetilde{\Sigma}_{u}\frac{k}{N}-\widetilde{\Sigma}_{u},$
$\displaystyle\widetilde{\sigma}^{2}_{\lambda\varepsilon}$
$\displaystyle=(\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}\widetilde{\sigma}_{u\varepsilon})^{2},$
$\displaystyle\widetilde{\Sigma}_{u}$
$\displaystyle=\frac{\widetilde{u}^{\prime}\widetilde{u}}{N}.$

Then, we choose $\widehat{k}$ as a minimizer of the feasible approximate MSE
function, $\widehat{S}_{\lambda}(k)$. Given $\widehat{k}$, we can compute the
CSA-2SLS estimator defined in (13).

We remark on two practical issues in implementation. First, our theory
requires the preliminary estimator $\widetilde{\beta}$ be consistent and it is
simply achieved by 2SLS estimation with any valid IVs. However, the
performance of the estimator in a finite sample might rely on the choice.
Following Donald and Newey (2001) and Kuersteiner and Okui (2010), we propose
to choose the set of IVs based on the first stage Mallows criterion, which
brings satisfactory performance results in the extensive simulations
experiments reported in Section 5. Second, it could be infeasible to estimate
$P^{k}$ when $K$ is too large. Note that the number of complete subsets (all
the subsets with different $k$’s) grows exponentially, $2^{K}-1$. To deal with
this computational issue, we propose to use a subsampling method as follows.
Let $\mathcal{R}^{k}$ be a class of $R$ subsets with $k$ elements that are
randomly selected from $\\{1,\ldots,M\\}$. Then, the subsampled CSA projection
matrix is defined as $\breve{P}=R^{-1}\sum_{r\in\mathcal{R}^{k}}P^{k}_{r}$,
where $P^{k}_{r}$ is a projection matrix using the set of IVs indexed by $r$.
Table 3 in Section 5 shows simulation results that the random subsampling
method works well with a feasible size of $R$.

We finalize this subsection by proving that the CSA-2SLS estimator achieves
the asymptotic optimality in the sense that it minimizes the MSE among the
class of CSA-2SLS estimators with different complete subset sizes. We collect
additional regularity conditions below.

######  Assumption 3.

Assume that

  1. (i)

$\widetilde{\sigma}^{2}_{\varepsilon}\xrightarrow{p}\sigma^{2}_{\varepsilon}$,
$\widetilde{\sigma}^{2}_{u\varepsilon}\xrightarrow{p}\sigma^{2}_{u\varepsilon}$,
$\widetilde{\lambda}\xrightarrow{p}\lambda$,
$\widetilde{H}\xrightarrow{p}\overline{H}$, and
$\lambda^{\prime}\overline{H}^{-1}\sigma_{u\varepsilon}\neq 0$.

  2. (ii)

$\lim_{N\rightarrow\infty}\sum_{k=1}^{K(N)}\left(NS_{\lambda}(k)\right)^{-1}=0$
almost surely in $Z$.

  3. (iii)

There exist a constant $\phi\in(0,1/2)$ such that
$\|\widetilde{\Sigma}_{u}-\Sigma_{u}\|=O_{p}(N^{-1/2+\phi}S_{\lambda}(k)^{\phi})$
where $\Sigma_{u}=E[u_{i}u_{i}^{\prime}|z_{i}]$.

Assumptions 3 (i) and (ii) are similar to Assumptions 4–5 in Donald and Newey
(2001). Assumption 3 (i) is a high-level assumption on the consistency of the
preliminary estimators. Assumption 3 (ii) is a standard assumption in the
model selection or model averaging literature (see, e.g. Assumption (A.3) in
Li (1987)). This condition excludes the case that $f$ is perfectly explained
by a finite number of instruments. Assumption 3 (iii) is a mild requirement on
the convergence rate of $\widetilde{f}$ that the standard series estimator
easily satisfies. For example, consider a series estimator with
$\widetilde{k}$ b-spline bases as in Newey (1997), where
$\sup_{z}\|f(z)-\Pi^{\widetilde{k}\prime}_{0}Z_{\widetilde{k}}(z)\|=O_{p}(\widetilde{k}^{-\alpha})$
where $\alpha=1$ and
$Z_{\widetilde{k}}(z)\equiv(\psi_{1}(z),\ldots,\psi_{K}(z),x_{1})^{\prime}$.
By choosing the optimal rate for $\widetilde{k}$, we get the rate for the
error variance, $\|\widetilde{\Sigma}_{u}-\Sigma_{u}\|=O_{p}(N^{-1/3})$. Since
$S_{\lambda}(k)=o_{p}(1)$, we can write $S_{\lambda}(k)=O_{p}(N^{-c})$ for
some $c>0$. Then, Assumption 3 (iii) requires
$\|\widetilde{\Sigma}_{u}-\Sigma_{u}\|=O_{p}(N^{-1/2+\phi-c})$. Comparing this
rate with $O_{p}(N^{-1/3})$ above, we can find $\phi>1/6+c$.

The following result shows that the proposed estimator is optimal among the
class of the CSA-2SLS estimators.

######  Theorem 2.

Under Assumptions 1, 2, and 3,

$\displaystyle\frac{S_{\lambda}(\widehat{k})}{\min_{k}S_{\lambda}(k)}\xrightarrow{p}1.$
(22)

##  4 Irrelevant Instruments

The higher-order MSE expansion in the previous section assumes an increasing
sequence of instruments where the instruments are strong enough for
$\sqrt{N}$-consistency and the existence of higher-order terms. This implies
that the concentration parameter, which measures the strength of the
instruments, increases at the rate of $N$. An important feature of the
CSA-2SLS is that no instrument is excluded either in finite sample or
asymptotically due to equal-weight averaging. Although irrelevant instruments
can be excluded in principle by a good pre-screening method, it is important
to investigate whether the CSA-2SLS estimator and the MSE expansion are still
valid if irrelevant instruments happen to be included in the averaging. In
contrast, irrelevant instruments can be excluded by zero or negative weight in
Kuersteiner and Okui (2010).

In this section, we derive the MSE assuming that a set of the instruments is
irrelevant and the set may grow as $N$ and $K$ increase. This makes the
concentration parameter become smaller in level than the case with only
relevant instruments for given $N$ and $K$. By comparing the MSEs with and
without irrelevant instruments we can analyze the effect of irrelevant
instruments on the MSE. Under the many weak instruments asymptotics of Chao
and Swanson (2005), the growth rate of the concentration parameter can be
slower than $N$. We do not take this approach because assuming a slower growth
rate of the concentration parameter would break down $\sqrt{N}$-consistency of
the 2SLS estimator and the validity of the higher-order expansion.

For the sake of simplicity, we assume that there is no exogenous variables,
$x_{1i}$, so the model simplifies to

$\displaystyle y_{i}$ $\displaystyle=$ $\displaystyle
X_{i}^{\prime}\beta+\varepsilon_{i},$ (23) $\displaystyle X_{i}$
$\displaystyle=$ $\displaystyle E[X_{i}|z_{i}]+u_{i},~{}~{}i=1,\ldots,N,$ (24)

where $X_{i}$ is a $d\times 1$ vector of endogenous variables and $z_{i}$ is a
vector of exogenous variables. We set $f(z_{i})=E[X_{i}|z_{i}]$. We divide
$z_{i}$ into the relevant and irrelevant ones, $z_{1i}$ and $z_{2i}$ and make
the following definition:

  * Definition 1

A vector of instruments $z_{2i}$ is irrelevant if
$E[X_{i}|z_{1i},z_{2i}]=E[X_{i}|z_{1i}]$.

Since mean independence implies uncorrelatedness, this definition implies that
$f(z_{i})=f(z_{1i})$ is uncorrelated with the set of instruments
$Z_{i}=(\psi_{1}(z_{2i}),\ldots,\psi_{K_{2}}(z_{2i}))$ where $Z_{i}$ is a
$K_{2}\times 1$ vector and $K_{2}<K$. Write $f_{i}=f(z_{1i})$ and assume
$Ef_{i}=0$ without loss of generality. Then Definition 1 implies
$0=Ef_{i}Z_{i}^{\prime}$ and under Assumption 1 the CLT holds

$\frac{1}{\sqrt{N}}\sum_{i=1}^{N}f_{i}\psi_{k}(z_{2i})^{\prime}=O_{p}(1)$ (25)

for $k=1,\ldots,K_{2}$.

Let $\mathcal{M}(K,k)=\left\\{1,2,\ldots,M(K,k)\right\\}$ be the index set for
all subsets with $k$ instruments and let $\mathcal{M}_{1}(K,k)$ and
$\mathcal{M}_{2}(K,k)$ be the index sets for subsets with at least $d$
relevant instrument and those with less than $d$, respectively. By
construction, $\mathcal{M}_{1}(K,k)\cup\mathcal{M}_{2}(K,k)=\mathcal{M}(K,k)$.
Let $M_{1}(K,k)$ and $M_{2}(K,k)$ be the number of elements in the subset
$\mathcal{M}_{1}(K,k)$ and $\mathcal{M}_{2}(K,k)$, respectively. For brevity,
we suppress the dependence of $K$ and $k$ and write them as
$M_{1},M_{2},\mathcal{M}_{1}$, and $\mathcal{M}_{2}$, hereafter. Define

$\displaystyle\widehat{P}^{k}$ $\displaystyle=$
$\displaystyle\frac{1}{M_{1}}\sum_{m\in\mathcal{M}_{1}}P^{k}_{m},$ (26)
$\displaystyle\widetilde{P}^{k}$ $\displaystyle=$
$\displaystyle\frac{1}{M_{2}}\sum_{m\in\mathcal{M}_{2}}P^{k}_{m}$ (27)

so that

$P^{k}=\frac{M_{1}}{M}\widehat{P}^{k}+\frac{M_{2}}{M}\widetilde{P}^{k}.$

We make assumptions that will replace Assumption 2.

######  Assumption 4.

  1. (i)

$\mathcal{M}_{1}(K,k)$ is nonempty for all $k$ and $K$ and $M_{1}/M\rightarrow
C$ for some constant $0<C<1$.

  2. (ii)

For each $k$ there exists $\Pi_{m}^{k}$ such that as $N,k\rightarrow\infty$,

$\frac{1}{M_{1}}\sum_{m\in\mathcal{M}_{1}}E\|f(z_{i})-\Pi_{m}^{k^{\prime}}Z_{m,i}^{k}\|^{2}\rightarrow
0.$

Assumption 4 (i) allows for an increasing sequence of irrelevant instruments.
However the number of irrelevant instruments cannot dominate that of the
relevant instruments asymptotically. Assumption 4 (ii) is a version of
Assumption 2 with relevant instruments only.

######  Theorem 3.

If Assumptions 1 and 4 are satisfied, $k^{2}/N\rightarrow 0$, and
$\sigma_{u\varepsilon}\neq 0$, then for the CSA-2SLS estimator the equations
(17)-(18) are satisfied with

$S(k)=H^{-1}\left[\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}\left(\frac{M}{M_{1}}\right)^{2}\frac{k^{2}}{N}+\sigma_{\varepsilon}^{2}\frac{f^{\prime}(I-\widehat{P}^{k})(I-P_{f})(I-\widehat{P}^{k})f}{N}\right]H^{-1},$

where $P_{f}=f(f^{\prime}f)^{-1}f^{\prime}$.

The MSE formula shows that the presence of irrelevant instruments inflates the
bias term by $(M/M_{1})^{2}$. This implies that a larger penalty should be
imposed on a larger $k$ and the optimal $k$ that minimizes the MSE under the
presence of irrelevant instruments will be smaller than otherwise. Although
this MSE cannot be estimated in practice, a practical recommendation is to
choose a smaller $k$ when the presence of irrelevant instruments is suspected.
Elliott, Gargano, and Timmermann (2013) similarly recommend choosing a small
$k$ in regression models when regressors are weak. In Section 5, we provide
simulation evidence that the choice of $k=1$ works well across various data
generating processes.

##  5 Simulation

We investigate the finite sample properties of the CSA-2SLS estimator by
conducting Monte Carlo simulation studies111The replication R codes for both
the Monte Carlo experiments and empirical applications are available at
https://github.com/yshin12/ls-csa.. We consider the following simulation
design:

$\displaystyle y_{i}$ $\displaystyle=\beta_{0}+\beta_{1}Y_{i}+\varepsilon_{i}$
(28) $\displaystyle Y_{i}$
$\displaystyle=\pi^{\prime}Z_{i}+u_{i},~{}~{}i=1,\ldots,N,$ (29)

where $Y_{i}$ is a scalar, $(\beta_{0},\beta_{1})$ is set to be (0, 0.1),
$\beta_{1}$ is the parameter of interest, and $Z_{i}\sim\text{i.i.d.
}N(0,\Sigma_{z})$. The diagonal terms of $\Sigma_{z}$ are ones and off-
diagonal terms are $\rho_{z}$’s. Thus, $\rho_{z}$ denotes the correlation
between instruments. With $\rho_{z}=0$, the simulation design is the same with
that of Donald and Newey (2001) and Kuersteiner and Okui (2010). The error
terms $(\varepsilon_{i},u_{i})$ are i.i.d. over $i$, bivariate normal with
variances 1 and covariance $\sigma_{u\varepsilon}$. Note that
$\sigma_{u\varepsilon}$ denotes the severity of endogeneity.

We set the parameter value for $\pi$ while controlling the explanatory power
of instruments. As instruments are possibly correlated to each other, the
theoretical first stage R-squared now becomes

$R_{f}^{2}=\frac{\pi^{\prime}E[Z_{i}Z_{i}^{\prime}]\pi}{\pi^{\prime}E[Z_{i}Z_{i}^{\prime}]\pi+1}=\frac{\sum_{k=1}^{K}\pi_{k}^{2}+\sum_{k=1}^{K}\sum_{j\neq
k}\pi_{k}\pi_{j}\rho_{z}}{\sum_{k=1}^{K}\pi_{k}^{2}+\sum_{k=1}^{K}\sum_{j\neq
k}\pi_{k}\pi_{j}\rho_{z}+1},$ (30)

where $\pi_{k}$ is the $k$th element of the $k\times 1$ vector $\pi$ and $K$
is the total number of instruments. Thus, we can set the value of $\pi$ given
$R_{f}^{2}$ and $\rho_{z}$ by solving Equation (30). Specifically, we consider
the following three designs:

Flat Signal:
$\displaystyle\pi_{k}=\sqrt{\frac{R_{f}^{2}}{(K+K(K-1)\rho_{z})(1-R_{f}^{2})}}$
Decreasing Signal:
$\displaystyle\pi_{k}=C_{D}\left(1-\frac{k}{K+1}\right)^{4}$ Half-zero Signal:
$\displaystyle\pi_{k}=\left\\{\begin{array}[]{ll}0,&\text{ for }k\leq K/2\\\
C_{H}\left(1-\frac{k-K/2}{K/2+1}\right)^{4},&\text{ for }k>K/2\\\
\end{array}\right.$ (33)

where $C_{D}$ and $C_{H}$ are defined as

$C_{D}=\sqrt{\frac{R_{f}^{2}}{1-R_{f}^{2}}\cdot\frac{1}{\sum_{k=1}^{K}\left(1-\frac{k}{K+1}\right)^{8}+\sum_{k=1}^{K}\sum_{j\neq
k}\left(1-\frac{k}{K+1}\right)^{4}\left(1-\frac{j}{K+1}\right)^{4}\rho_{z}}}$

and

$\displaystyle
C_{H}=\sqrt{\frac{R_{f}^{2}}{1-R_{f}^{2}}\cdot\frac{1}{\sum_{k=K/2+1}^{K}\left(1-\frac{k-K/2}{K/2+1}\right)^{8}+\sum_{k=K/2+1}^{K}\sum_{j\neq
k,j=K/2+1}\left(1-\frac{k-K/2}{K/2+1}\right)^{4}\left(1-\frac{j-K/2}{K/2+1}\right)^{4}\rho_{z}}}.$

The number of observations and the number of instruments are set
$(N,K)=(100,20),(1000,30)$ and the first stage R-squared is set
$R_{f}^{2}=0.01,0.1$. We report results with $\rho_{z}=0.5$ (moderate
correlation among instruments) and $\sigma_{u\varepsilon}=0.9$ (large
endogeneity). We choose these values because our CSA-2SLS estimator is
expected to perform well with a nonzero $\rho_{z}$ and IV estimators generally
perform better than OLS when there is severe endogeneity. Results with
$\rho_{z}=0$ (independent instruments) and $\sigma_{u\varepsilon}=0.1$ (low
endogeneity) are collected in the supplementary appendix.

In addition to the CSA-2SLS (denoted by CSA hereafter) estimator, we also
estimate the model by OLS, 2SLS with full instruments, the optimal 2SLS by
Donald and Newey (2001) denoted by DN, and the model average estimator by
Kuersteiner and Okui (2010) denoted by KO and compare their
performance.222Since we build up the idea of CSA based on 2SLS, we focus on
the comparison with similar 2SLS type estimators in various situations. We
leave it for future research to develop an CSA estimator in different classes
(e.g. LIML or JIVE) and compare the performance with other types of
estimators. We apply the Mallows criterion for the preliminary estimates
required for DN, KO, and CSA. For each $k$, we use 1,000 random draws from
complete subsets and estimate the average projection matrix in CSA when the
complete subset size is bigger than 1,000. The simulation results are based on
1,000 replications of each design.

Figure 2 compares the mean bias and the mean squared error of each estimator.
The result is quite striking: CSA outperforms all other estimation methods in
terms of mean bias and MSE and the gain is huge. This result holds for all six
different designs on the structure of $\pi$ and the signal structure and
across different sample sizes.

Tables 1-2 also show median bias, median absolute deviation (MAD), range,
coverage of the 95% asymptotic confidence interval333Heteroskedasticity-robust
standard errors given the choice of $k$ (CSA), $K$ (DN), and optimal weight
(KO) are used., and the mean and median number of the subset size choice
$\widehat{k}$. Median bias and mean bias differ more when the distribution of
the estimator is skewed. DN shows some difference but for other estimators
mean and median bias are quite similar. CSA still dominates other estimators
in terms of median bias. KO shows a smaller MAD than CSA but given KO’s larger
MSE this suggests that KO often gives outliers which may be due to weight
estimation. KO also exhibits a smaller range than CSA, which shows that
smaller bias of CSA comes at a cost of having larger variance compared to KO.
Lastly, the coverage of CSA is close to the nominal level across difference
specifications but the coverage of KO can be far from the nominal level
especially when the instrument signal is weak ($R_{f}^{2}=0.01$)444We
construct the confidence intervals by assuming the instrument/weight/subset
selection in each method is correct. See Appendix B for details..

When there is little endogeneity ($\sigma_{u\varepsilon}=0.1$) and/or the
instruments are independent ($\rho_{z}=0$), the performance of CSA is quite
similar to KO. The performance of DN is less satisfactory because it often
exhibits a very large MSE. The results are collected in the supplementary
appendix.

Figure 2: $\sigma_{u\varepsilon}=0.9$ (high endogeneity), $\rho_{z}=0.5$
(moderate correlation among $z$)

$N=100$, $K=20$
Mean Bias

Mean Squared Error

$N=1000$, $K=30$
Mean Bias

Mean Squared Error

[ht]

Table 1: $N=100$, $K=20$, $\sigma_{u\varepsilon}=0.9$ (high endogeneity),
$\rho_{z}=0.5$ (moderate correlation among $z$)

| MSE | Bias | MAD | Median Bias | Range | Coverage |  Mean($\widehat{k}$) |  Med($\widehat{k}$)
---|---|---|---|---|---|---|---|---
$R_{f}^{2}=0.01$ (weak IV signal)
$\pi_{0}:$ flat |  |  |  |  |  |
OLS | 0.667 | 0.815 | 0.036 | 0.815 | 0.133 | 0.000 | – | –
2SLS | 0.358 | 0.586 | 0.081 | 0.587 | 0.304 | 0.015 | – | –
DN | 1.196 | 0.031 | 0.205 | 0.154 | 1.079 | 0.813 | 1.625 | 1.000
KO | 0.237 | 0.463 | 0.102 | 0.456 | 0.391 | 0.159 | – | –
CSA | 0.076 | 0.048 | 0.146 | 0.098 | 0.621 | 0.885 | 1.130 | 1.000
CSA.1 | 0.085 | 0.028 | 0.169 | 0.078 | 0.640 | 0.896 | 1.000 | 1.000
$\pi_{0}:$ decreasing |  |  |  |  |  |
OLS | 0.795 | 0.891 | 0.032 | 0.889 | 0.120 | 0.000 | – | –
2SLS | 0.746 | 0.856 | 0.074 | 0.861 | 0.288 | 0.000 | – | –
DN | 45.785 | 0.833 | 0.342 | 0.642 | 2.317 | 0.608 | 2.216 | 1.000
KO | 0.706 | 0.828 | 0.091 | 0.835 | 0.355 | 0.002 | – | –
CSA | 0.561 | 0.673 | 0.206 | 0.662 | 0.751 | 0.328 | 3.424 | 1.000
CSA.1 | 0.545 | 0.637 | 0.199 | 0.594 | 0.949 | 0.377 | 1.000 | 1.000
$\pi_{0}:$ half-zero |  |  |  |  |  |
OLS | 0.796 | 0.891 | 0.032 | 0.889 | 0.121 | 0.000 | – | –
2SLS | 0.748 | 0.857 | 0.073 | 0.863 | 0.290 | 0.001 | – | –
DN | 130.901 | 0.425 | 0.348 | 0.759 | 2.230 | 0.589 | 2.339 | 1.000
KO | 0.726 | 0.841 | 0.088 | 0.845 | 0.338 | 0.002 | – | –
CSA | 0.595 | 0.699 | 0.203 | 0.693 | 0.763 | 0.295 | 4.104 | 1.000
CSA.1 | 0.564 | 0.657 | 0.195 | 0.618 | 0.928 | 0.335 | 1.000 | 1.000
$R_{f}^{2}=0.1$ (strong IV signal)
$\pi_{0}:$ flat |  |  |  |  |  |
OLS | 0.178 | 0.418 | 0.038 | 0.419 | 0.139 | 0.000 | – | –
2SLS | 0.022 | 0.126 | 0.048 | 0.130 | 0.198 | 0.583 | – | –
DN | 0.011 | 0.035 | 0.060 | 0.046 | 0.239 | 0.906 | 3.728 | 4.000
KO | 0.013 | 0.079 | 0.051 | 0.087 | 0.212 | 0.787 | – | –
CSA | 0.009 | 0.001 | 0.056 | 0.012 | 0.229 | 0.941 | 1.037 | 1.000
CSA.1 | 0.009 | -0.003 | 0.062 | 0.006 | 0.234 | 0.945 | 1.000 | 1.000
$\pi_{0}:$ decreasing |  |  |  |  |  |
OLS | 0.660 | 0.811 | 0.034 | 0.811 | 0.133 | 0.000 | – | –
2SLS | 0.346 | 0.576 | 0.083 | 0.575 | 0.305 | 0.014 | – | –
DN | 1.891 | -0.029 | 0.188 | 0.120 | 0.924 | 0.838 | 1.520 | 1.000
KO | 0.202 | 0.422 | 0.102 | 0.416 | 0.384 | 0.225 | – | –
CSA | 0.075 | 0.048 | 0.144 | 0.099 | 0.617 | 0.885 | 1.114 | 1.000
CSA.1 | 0.086 | 0.031 | 0.168 | 0.081 | 0.640 | 0.894 | 1.000 | 1.000
$\pi_{0}:$ half-zero |  |  |  |  |  |
OLS | 0.662 | 0.812 | 0.036 | 0.811 | 0.138 | 0.000 | – | –
2SLS | 0.349 | 0.578 | 0.082 | 0.579 | 0.302 | 0.016 | – | –
DN | 1.262 | 0.038 | 0.231 | 0.183 | 1.224 | 0.778 | 1.740 | 1.000
KO | 0.258 | 0.486 | 0.100 | 0.484 | 0.376 | 0.126 | – | –
CSA | 0.080 | 0.068 | 0.148 | 0.112 | 0.642 | 0.874 | 1.209 | 1.000
CSA.1 | 0.089 | 0.043 | 0.169 | 0.087 | 0.641 | 0.888 | 1.000 | 1.000

  * •

Note: We report mean squared errors (MSE), mean biases (Bias), median absolute
deviations (MAD), median biases (Median Bias), 10-90% ranges of the estimator
(Range), coverages for the 95% confidence interval (Coverage), means of
$\widehat{k}$ and medians of $\widehat{k}$. For estimators DN, KO, and CSA, we
apply the Mallows criterion for the preliminary estimator. We set $k=1$ for
CSA.1.

[ht]

Table 2: $N=1000$, $K=30$, $\sigma_{u\varepsilon}=0.9$ (high endogeneity),
$\rho_{z}=0.5$ (moderate correlation among $z$)

| MSE | Bias | MAD | Median Bias | Range | Coverage |  Mean($\widehat{k}$) |  Med($\widehat{k}$)
---|---|---|---|---|---|---|---|---
$R_{f}^{2}=0.01$ (weak IV signal)
$\pi_{0}:$ flat |  |  |  |  |  |
OLS | 0.605 | 0.778 | 0.011 | 0.778 | 0.041 | 0.000 | – | –
2SLS | 0.023 | 0.139 | 0.041 | 0.144 | 0.160 | 0.407 | – | –
DN | 0.008 | 0.029 | 0.057 | 0.037 | 0.219 | 0.894 | 3.862 | 4.000
KO | 0.009 | 0.060 | 0.051 | 0.065 | 0.188 | 0.813 | – | –
CSA | 0.006 | 0.002 | 0.056 | 0.006 | 0.205 | 0.940 | 1.014 | 1.000
CSA.1 | 0.006 | 0.001 | 0.055 | 0.003 | 0.204 | 0.946 | 1.000 | 1.000
$\pi_{0}:$ decreasing |  |  |  |  |  |
OLS | 0.793 | 0.890 | 0.009 | 0.891 | 0.036 | 0.000 | – | –
2SLS | 0.456 | 0.669 | 0.058 | 0.670 | 0.225 | 0.000 | – | –
DN | 12.594 | -0.089 | 0.202 | 0.111 | 0.934 | 0.843 | 1.362 | 1.000
KO | 0.231 | 0.463 | 0.078 | 0.466 | 0.302 | 0.113 | – | –
CSA | 0.075 | 0.051 | 0.169 | 0.095 | 0.653 | 0.876 | 1.091 | 1.000
CSA.1 | 0.072 | 0.042 | 0.168 | 0.083 | 0.646 | 0.885 | 1.000 | 1.000
$\pi_{0}:$ half-zero |  |  |  |  |  |
OLS | 0.793 | 0.890 | 0.009 | 0.891 | 0.037 | 0.000 | – | –
2SLS | 0.456 | 0.669 | 0.059 | 0.670 | 0.235 | 0.000 | – | –
DN | 6.671 | -0.102 | 0.235 | 0.143 | 1.147 | 0.822 | 1.436 | 1.000
KO | 0.288 | 0.521 | 0.081 | 0.522 | 0.309 | 0.057 | – | –
CSA | 0.079 | 0.075 | 0.170 | 0.112 | 0.659 | 0.855 | 1.314 | 1.000
CSA.1 | 0.073 | 0.051 | 0.171 | 0.090 | 0.659 | 0.882 | 1.000 | 1.000
$R_{f}^{2}=0.1$ (strong IV signal)
$\pi_{0}:$ flat |  |  |  |  |  |
OLS | 0.109 | 0.330 | 0.011 | 0.330 | 0.041 | 0.000 | – | –
2SLS | 0.001 | 0.015 | 0.017 | 0.016 | 0.060 | 0.889 | – | –
DN | 0.001 | 0.007 | 0.018 | 0.008 | 0.062 | 0.931 | 9.710 | 10.000
KO | 0.001 | 0.010 | 0.017 | 0.010 | 0.062 | 0.918 | – | –
CSA | 0.001 | 0.001 | 0.017 | 0.001 | 0.061 | 0.951 | 1.016 | 1.000
CSA.1 | 0.001 | 0.000 | 0.017 | -0.000 | 0.061 | 0.953 | 1.000 | 1.000
$\pi_{0}:$ decreasing |  |  |  |  |  |
OLS | 0.655 | 0.809 | 0.010 | 0.810 | 0.041 | 0.000 | – | –
2SLS | 0.039 | 0.185 | 0.044 | 0.190 | 0.177 | 0.282 | – | –
DN | 0.010 | 0.029 | 0.065 | 0.037 | 0.246 | 0.897 | 3.259 | 3.000
KO | 0.012 | 0.068 | 0.057 | 0.074 | 0.215 | 0.809 | – | –
CSA | 0.010 | 0.003 | 0.068 | 0.009 | 0.249 | 0.938 | 1.000 | 1.000
CSA.1 | 0.009 | 0.001 | 0.066 | 0.005 | 0.245 | 0.943 | 1.000 | 1.000
$\pi_{0}:$ half-zero |  |  |  |  |  |
OLS | 0.655 | 0.809 | 0.011 | 0.810 | 0.040 | 0.000 | – | –
2SLS | 0.039 | 0.186 | 0.045 | 0.192 | 0.170 | 0.283 | – | –
DN | 0.014 | 0.038 | 0.073 | 0.048 | 0.293 | 0.880 | 3.064 | 3.000
KO | 0.016 | 0.097 | 0.057 | 0.100 | 0.214 | 0.724 | – | –
CSA | 0.010 | 0.003 | 0.070 | 0.009 | 0.254 | 0.938 | 1.006 | 1.000
CSA.1 | 0.010 | 0.001 | 0.069 | 0.005 | 0.257 | 0.941 | 1.000 | 1.000

  * •

Note: See the Note below Table 1 for details.

[ht]

Table 3: Comparison of CSA for Different Random Draws, $R$
$N=100$, $K=20$, $\sigma_{u\varepsilon}=0.9$ (high endogeneity),
$\rho_{z}=0.5$ (moderate correlation among $z$), $R_{f}^{2}=0.01$ (weak IV
signal), $\pi_{0}:$ flat

| MSE | Bias | MAD | Median Bias | Range | Coverage | Mean($\widehat{k}$) | Med($\widehat{k}$)
---|---|---|---|---|---|---|---|---
$R=all$ | 0.082 | 0.037 | 0.168 | 0.082 | 0.635 | 0.890 | 1.107 | 1.000
$R=1,000$ | 0.076 | 0.048 | 0.146 | 0.098 | 0.621 | 0.885 | 1.130 | 1.000
$R=500$ | 0.087 | 0.043 | 0.156 | 0.092 | 0.658 | 0.879 | 1.112 | 1.000
$R=250$ | 0.075 | 0.037 | 0.151 | 0.086 | 0.643 | 0.889 | 1.072 | 1.000

  * •

Note: $R$ is the number of random sampling from the complete subset for each
$k$. When $R=all$, the CSA projection matrix is calculated by using all
complete subsets. See the note below Table 1 for other details.

##  6 Empirical Illustration

In this section we illustrate our CSA-2SLS by estimating a logistic demand
function for automobiles in Berry, Levinsohn, and Pakes (1995). The model
specification is

$\displaystyle\log(S_{it})-\log(S_{0t})$
$\displaystyle=\alpha_{0}P_{it}+X_{it}^{\prime}\beta_{0}+\varepsilon_{it},$
$\displaystyle P_{it}$
$\displaystyle=Z_{it}^{\prime}\delta_{0}+X_{it}^{\prime}\gamma_{0}+u_{it},$

where $S_{it}$ is the market share of the product $i$ in market $t$ with
product 0 denoting the outside option, $P_{it}$ is the endogenous price
variable, $x_{it}$ is a vector of included exogenous variables, and $Z_{it}$
is a vector of instruments. The parameter of interest is $\alpha_{0}$ from
which we can calculate the price elasticity of demand. We first estimate the
model by using the same set of regressors and instruments used by Berry,
Levinsohn, and Pakes (1995). The regressors vector $X_{it}$ includes 5
variables: a constant, an air conditioning dummy, horsepower divided by
weight, miles per dollar, and vehicle size. The original instrument vector
$Z_{it}$ includes 10 variables and is constructed by the characteristics of
other car models. We also consider an extended design by adopting 48
instruments and 24 regressors constructed by Chernozhukov, Hansen, and
Spindler (2015). We presume that all instruments are valid and relevant. Based
on the previous simulation results, we used the Mallows criterion for
selecting IVs for preliminary estimates and set $R=1,000$ when calculating the
CSA-P matrix.

Table 4 summarizes the estimation results. As in the simulations studies, we
estimate the model using five different methods: OLS, 2SLS, DN, KO, and CSA.
For each design we report the optimal choice of $k$ in DN and CSA, the
estimate of $\alpha$, and the heteroskedasticity and cluster robust standard
errors of $\alpha$ given that we have chosen the correct model for $k$ or the
optimal weight of KO. Finally, we report the number of products whose price
elasticity of demand is inelastic, which is computed by the following formula:

$\displaystyle\sum_{i,t}1\\{|\widehat{\alpha}\times
P_{it}\times(1-S_{it})|<1\\},$

where $1\\{A\\}$ is an indicator function equal to 1 if $A$ holds.

It is interesting to note that the estimation result of CSA contrasts sharply
with those of other estimation methods in the extended design. Recall that the
economic theory predicts elastic demand in this market. Similar to the result
in Berry, Levinsohn, and Pakes (1995), the OLS estimate is biased towards zero
and makes 1,405 out of 2,217 products (63%) have inelastic own price demand.
The 2SLS estimator mitigates the bias but not enough. Still, 874 (41%)
products show inelastic own price elasticity. It is interesting to note that
DN and KO are not particularly better than 2SLS in this empirical example
although they are supposed to correct the bias caused by many instruments. The
estimation result of DN comes close to 2SLS by choosing most instruments, 47
out of 48, and that of KO coincides exactly with 2SLS by putting the whole
weight to the largest set of instruments. On the contrary, only 7 products
(0.3%) have inelastic demand according to the estimation result of CSA. The
$\alpha$ estimate by CSA is about twice as large as those by other estimators
in absolute term. Since the bias caused by many instruments is towards the OLS
estimates, this results can be viewed as a correction for the many instrument
bias. However, the standard error of CSA is larger than others and there is a
potential trade-off between the bias and the variance. Finally, the original
design has less number of instruments and the bias correction by CSA is not as
large as in the extended design.

In sum, 2SLS with all the available instruments suffers from many instruments
bias in this application. DN and KO do not correct the bias enough to make the
estimation results consistent with the prediction by the economic theory. In
contrast, the CSA point estimate reduces the bias substantially in the
extended design. Therefore, it is worthwhile to estimate a model with CSA and
to compare the result with other existing methods when the model contains many
instruments.

[ht]

Table 4: Logistic Demand Function for Automobiles

| $\hat{k}$ | $\widehat{\alpha}$ | # of Inelastic |  | $\hat{k}$ | $\widehat{\alpha}$ | # of Inelastic
---|---|---|---|---|---|---|---
| Demand |  | Demand
| Original Design |  | Extended Design
OLS | – | -0.0886 | 1502 |  | – | -0.0991 | 1405
|  | (0.0114) |  |  |  | (0.0124) |
2SLS | – | -0.1357 | 746 |  | – | -0.1273 | 874
|  | (0.0464) |  |  |  | (0.0246) |
DN | 10 | -0.1357 | 746 |  | 47 | -0.1271 | 876
|  | (0.0464) |  |  |  | (0.0245) |
KO | – | -0.1357 | 746 |  | – | -0.1273 | 874
|  | (0.0464) |  |  |  | (0.0246) |
CSA | 9 | -0.1426 | 659 |  | 1 | -0.2515 | 7
|  | (0.0491) |  |  |  | (0.0871) |

  * •

Note: The original design uses 5 regressors and 10 instruments and the
extended design does 24 regressors and 48 instruments. The sample size is
2,217. The heteroskedasticity and cluster robust standard errors of
$\widehat{\alpha}$ are provided inside parentheses.

## References

  * Ando and Li (2014) Ando, T. and K.-C. Li (2014).  A model-averaging approach for high-dimensional regression.  Journal of the American Statistical Association 109(505), 254–265.
  * Ando and Li (2017) Ando, T. and K.-C. Li (2017).  A weight-relaxed model averaging approach for high-dimensional generalized linear models.  The Annals of Statistics 45(6), 2654–2679.
  * Andrews and Stock (2007) Andrews, D. W. and J. H. Stock (2007).  Testing with many weak instruments.  Journal of Econometrics 138(1), 24–46.
  * Angrist and Krueger (1991) Angrist, J. D. and A. B. Krueger (1991).  Does compulsory school attendance affect schooling and earnings?  The Quarterly Journal of Economics 106(4), 979–1014.
  * Antoine and Lavergne (2014) Antoine, B. and P. Lavergne (2014).  Conditional moment models under semi-strong identification.  Journal of Econometrics 182(1), 59–69.
  * Berry et al. (1995) Berry, S., J. Levinsohn, and A. Pakes (1995).  Automobile prices in market equilibrium.  Econometrica: Journal of the Econometric Society, 841–890.
  * Breiman (1996) Breiman, L. (1996).  Bagging predictors.  Machine Learning 24(2), 123–140.
  * Chao and Swanson (2005) Chao, J. C. and N. R. Swanson (2005).  Consistent estimation with a large number of weak instruments.  Econometrica 73(5), 1673–1692.
  * Chernozhukov et al. (2018) Chernozhukov, V., D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins (2018).  Double/debiased machine learning for treatment and structural parameters.  The Econometrics Journal 21(1), C1–C68.
  * Chernozhukov et al. (2015) Chernozhukov, V., C. Hansen, and M. Spindler (2015).  Post-selection and post-regularization inference in linear models with many controls and instruments.  American Economic Review 105(5), 486–90.
  * Clemen (1989) Clemen, R. T. (1989).  Combining forecasts: A review and annotated bibliography.  International Journal of Forecasting 5(4), 559–583.
  * Donald et al. (2009) Donald, S. G., G. W. Imbens, and W. K. Newey (2009).  Choosing instrumental variables in conditional moment restriction models.  Journal of Econometrics 152(1), 28–36.
  * Donald and Newey (2001) Donald, S. G. and W. K. Newey (2001).  Choosing the number of instruments.  Econometrica 69(5), 1161–1191.
  * Elliott et al. (2013) Elliott, G., A. Gargano, and A. Timmermann (2013).  Complete subset regressions.  Journal of Econometrics 177(2), 357–373.
  * Elliott et al. (2015) Elliott, G., A. Gargano, and A. Timmermann (2015).  Complete subset regressions with large-dimensional sets of predictors.  Journal of Economic Dynamics and Control 54, 86–110.
  * Escanciano (2017) Escanciano, J. C. (2017).  A simple and robust estimator for linear regression models with strictly exogenous instruments.  The Econometrics Journal 21(1), 36–54.
  * Furuta (1987) Furuta, T. (1987).  $A\geq B\geq 0$ assures $(B^{r}A^{p}B^{r})^{1/q}\geq B^{(p+2r)/q}$ for $r\geq 0$, $p\geq 0$, $q\geq 1$ with $(1+2r)q\geq p+2r$.  Proceedings of the American Mathematical Society, 85–88.
  * Han and Phillips (2006) Han, C. and P. C. Phillips (2006).  GMM with many moment conditions.  Econometrica 74(1), 147–192.
  * Hansen (2007) Hansen, B. E. (2007).  Least squares model averaging.  Econometrica 75(4), 1175–1189.
  * Hansen and Lee (2019) Hansen, B. E. and S. Lee (2019).  Asymptotic theory for clustered samples.  Journal of Econometrics 210(2), 268–290.
  * Hansen and Racine (2012) Hansen, B. E. and J. S. Racine (2012).  Jackknife model averaging.  Journal of Econometrics 167(1), 38–46.
  * Hansen et al. (2008) Hansen, C., J. Hausman, and W. Newey (2008).  Estimation with many instrumental variables.  Journal of Business & Economic Statistics 26(4), 398–422.
  * Hausman et al. (2011) Hausman, J., R. Lewis, K. Menzel, and W. Newey (2011).  Properties of the CUE estimator and a modification with moments.  Journal of Econometrics 165(1), 45–57.
  * Hausman et al. (2012) Hausman, J. A., W. K. Newey, T. Woutersen, J. C. Chao, and N. R. Swanson (2012).  Instrumental variable estimation with heteroskedasticity and many instruments.  Quantitative Economics 3(2), 211–255.
  * Kang (2018) Kang, B. (2018).  Higher order approximation of IV estimators with invalid instruments.  Working paper.
  * Kitagawa and Muris (2016) Kitagawa, T. and C. Muris (2016).  Model averaging in semiparametric estimation of treatment effects.  Journal of Econometrics 193(1), 271–289.
  * Kuersteiner and Okui (2010) Kuersteiner, G. and R. Okui (2010).  Constructing optimal instruments by first-stage prediction averaging.  Econometrica 78(2), 697–718.
  * Kuersteiner (2012) Kuersteiner, G. M. (2012).  Kernel-weighted GMM estimators for linear time series models.  Journal of Econometrics 170(2), 399–421.
  * Lee and Zhou (2015) Lee, Y. and Y. Zhou (2015).  Averaged instrumental variables estimators.  Working paper.
  * Li (1987) Li, K.-C. (1987).  Asymptotic optimality for $C_{p}$, $C_{L}$, cross-validation and generalized cross-validation: discrete index set.  The Annals of Statistics, 958–975.
  * Nagar (1959) Nagar, A. L. (1959).  The bias and moment matrix of the general k-class estimators of the parameters in simultaneous equations.  Econometrica: Journal of the Econometric Society, 575–595.
  * Newey (1997) Newey, W. K. (1997).  Convergence rates and asymptotic normality for series estimators.  Journal of Econometrics 79(1), 147–168.
  * Okui (2011) Okui, R. (2011).  Instrumental variable estimation in the presence of many moment conditions.  Journal of Econometrics 165(1), 70–86.
  * Patel and Toda (1979) Patel, R. and M. Toda (1979).  Trace inequalities involving hermitian matrices.  Linear Algebra and Its Applications 23, 13–20.
  * Smith and Wallis (2009) Smith, J. and K. F. Wallis (2009).  A simple explanation of the forecast combination puzzle.  Oxford Bulletin of Economics and Statistics 71(3), 331–355.
  * Stock and Yogo (2005) Stock, J. and M. Yogo (2005).  Asymptotic distributions of instrumental variables statistics with many instruments.  Identification and inference for econometric models: Essays in honor of Thomas Rothenberg, 109–120.
  * Stock and Watson (2004) Stock, J. H. and M. W. Watson (2004).  Combination forecasts of output growth in a seven-country data set.  Journal of Forecasting 23(6), 405–430.
  * Wager and Athey (2018) Wager, S. and S. Athey (2018).  Estimation and inference of heterogeneous treatment effects using random forests.  Journal of the American Statistical Association 113(523), 1228–1242.
  * Whittle (1960) Whittle, P. (1960).  Bounds for the moments of linear and quadratic forms in independent variables.  Theory of Probability & Its Applications 5(3), 302–305.
  * Zhang et al. (2016) Zhang, X., D. Yu, G. Zou, and H. Liang (2016).  Optimal model averaging estimation for generalized linear models and generalized linear mixed-effects models.  Journal of the American Statistical Association 111(516), 1775–1790.
  * Zhang and Yu (2018) Zhang, X. and J. Yu (2018).  Spatial weights matrix selection and model averaging for spatial autoregressive models.  Journal of Econometrics 203(1), 1–18.

## Appendices

##  Appendix A Proofs and Lemmas

Let $e_{f}^{k}=f^{\prime}(I-P^{k})^{2}f/N$,
$\xi_{f}^{k}=f^{\prime}(I-P^{k})f/N$, and $\Delta_{k}=\text{tr}(e_{f}^{k})$.
In addition, let $\widehat{e}_{f}^{k}=f^{\prime}(I-\widehat{P}^{k})^{2}f/N$,
$\widehat{\xi}_{f}^{k}=f^{\prime}(I-\widehat{P}^{k})f/N$, and
$\widehat{\Delta}_{k}=\text{tr}(\widehat{e}_{f}^{k})$.

###  A.1 Proof of Theorem 1

The complete subset averaging 2SLS estimator is

$\sqrt{N}\left(\widehat{\beta}-\beta\right)=\left(\frac{X^{\prime}P^{k}X}{N}\right)^{-1}\frac{X^{\prime}P^{k}\varepsilon}{\sqrt{N}}.$

By expanding

$\displaystyle\frac{X^{\prime}P^{k}X}{N}$ $\displaystyle=$
$\displaystyle\frac{f^{\prime}f}{N}-\frac{f^{\prime}(I-P^{k})f}{N}+\frac{u^{\prime}f+f^{\prime}u}{N}+\frac{u^{\prime}P^{k}u}{N}-\frac{u^{\prime}(I-P^{k})f+f^{\prime}(I-P^{k})u}{N}$
(34) $\displaystyle\frac{X^{\prime}P^{k}\varepsilon}{\sqrt{N}}$
$\displaystyle=$
$\displaystyle\frac{f^{\prime}\varepsilon}{\sqrt{N}}-\frac{f^{\prime}(I-P^{k})\varepsilon}{\sqrt{N}}+\frac{u^{\prime}P^{k}\varepsilon}{\sqrt{N}}$
(35)

we can write

$\sqrt{N}\left(\widehat{\beta}-\beta\right)=\widehat{H}^{-1}\widehat{h},$ (36)

where

$\displaystyle\widehat{H}$ $\displaystyle=$ $\displaystyle
H+T_{1}^{H}+T_{2}^{H}+Z^{H},$ $\displaystyle H$ $\displaystyle=$
$\displaystyle\frac{f^{\prime}f}{N},$ $\displaystyle T_{1}^{H}$
$\displaystyle=$ $\displaystyle-\frac{f^{\prime}(I-{P}^{k})f}{N},$
$\displaystyle T_{2}^{H}$ $\displaystyle=$
$\displaystyle\frac{u^{\prime}f+f^{\prime}u}{N},$ $\displaystyle Z^{H}$
$\displaystyle=$
$\displaystyle-\frac{u^{\prime}(I-{P}^{k})f+f^{\prime}(I-{P}^{k})u}{N}+\frac{u^{\prime}P^{k}u}{N},$

and

$\displaystyle\widehat{h}$ $\displaystyle=$ $\displaystyle
h+T_{1}^{h}+T_{2}^{h}$ $\displaystyle h$ $\displaystyle=$
$\displaystyle\frac{f^{\prime}\varepsilon}{\sqrt{N}},$ $\displaystyle
T_{1}^{h}$ $\displaystyle=$
$\displaystyle-\frac{f^{\prime}(I-{P}^{k})\varepsilon}{\sqrt{N}},$
$\displaystyle T_{2}^{h}$ $\displaystyle=$
$\displaystyle\frac{u^{\prime}P^{k}\varepsilon}{\sqrt{N}}.$

Based on Lemmas 1-2, we specify the convergence rate of each term. By Lemma
2(ii), $T_{1}^{H}=O_{p}(\Delta_{k}^{1/2})$. By CLT,
$T_{2}^{H}=O_{p}(1/\sqrt{N})$. By Lemmas 2(iii)-(v),

$Z^{H}=O_{p}\left(\frac{\Delta_{k}^{1/2}}{\sqrt{N}}\right)+O_{p}\left(\frac{k}{N}\right)=o_{p}\left(\frac{k^{2}}{N}+\Delta_{k}\right).$
(37)

By Lemma 2(ii), $T_{1}^{h}=O_{p}(\Delta_{k}^{1/2})$. By Lemma 2(iv),
$T_{2}^{h}=O_{p}(k/\sqrt{N})$. By WLLN, $H\xrightarrow{p}Ef_{i}f_{i}^{\prime}$
and thus $H=O_{p}(1)$. By CLT, $h=O_{p}(1)$. By Assumption 1.6 and the
continuous mapping theorem, $\widehat{H}^{-1}=O_{p}(1)$. Let
$T^{h}=T_{1}^{h}+T_{2}^{h}$ and $T^{H}=T_{1}^{H}+T_{2}^{H}$.

We show that

$S(k)=H^{-1}\left[\frac{k^{2}}{N}\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}+\sigma_{\varepsilon}^{2}e_{f}^{k}-\sigma_{\varepsilon}^{2}\xi_{f}^{k}H^{-1}\xi_{f}^{k}\right]H^{-1}.$
(38)

By Markov inequality and the trace inequality
$\text{tr}(AB)\leq\text{tr}(A)\text{tr}(B)$ for positive semi-definite
matrices $A$ and $B$ (e.g. Patel and Toda (1979)),

$\displaystyle\frac{k^{2}}{N}H^{-1}\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}H^{-1}$
$\displaystyle=$ $\displaystyle O_{p}\left(\frac{k^{2}}{N}\right),$ (39)
$\displaystyle\sigma_{\varepsilon}^{2}H^{-1}e_{f}^{k}H^{-1}$ $\displaystyle=$
$\displaystyle O_{p}(\Delta_{k}),$ (40)
$\displaystyle\sigma_{\varepsilon}^{2}H^{-1}\xi_{f}^{k}H^{-1}\xi_{f}^{k}H^{-1}$
$\displaystyle=$ $\displaystyle O_{p}(\Delta_{k}).$ (41)

The proof proceed by showing that $\widehat{r}(k)+T(k)=o_{p}(\rho_{k,N})$ as
$k,N\rightarrow\infty$ where $\rho_{k,N}$ is the lower order (the slower)
between $k^{2}/N$ and $\Delta_{k}$.

Our expansion is a non-trivial extension of Donald and Newey (2001) and
Kuersteiner and Okui (2010) because we need to specify additional terms that
are supposed to be small in those papers. This is due to the fact that our
$P^{k}$ matrix not being idempotent. In particular, Lemma 1 of Donald and
Newey (2001) cannot be applied because $\|T^{H}\|\cdot\|T^{h}\|$ is not small.
In addition, Lemma A.1 of Kuersteiner and Okui (2010) cannot be applied
because $\|T^{H}\|^{2}$ is not small.

We use the following expansion. Let $\widehat{H}=\widetilde{H}+Z^{H}$ and
$\widetilde{H}=H+T^{H}$. Using

$\displaystyle\widehat{H}^{-1}$ $\displaystyle=$
$\displaystyle\widetilde{H}^{-1}-\widehat{H}^{-1}(\widehat{H}-\widetilde{H})\widetilde{H}^{-1}$
(42)

we can write

$\widehat{H}^{-1}\widehat{h}=\widetilde{H}^{-1}\widehat{h}+o_{p}(\rho_{k,N})$
(43)

because $Z^{H}=o_{p}(\rho_{k,N})$, $\widehat{H}^{-1}=O_{p}(1)$, and
$\widetilde{H}^{-1}=O_{p}(1)$. Furthermore, using

$\displaystyle\widetilde{H}^{-1}$ $\displaystyle=$ $\displaystyle
H^{-1}-\widetilde{H}^{-1}(\widetilde{H}-H)H^{-1},$ (44)

and that $\|T_{2}^{H}\|^{2}=O_{p}(1/N)=o_{p}(\rho_{k,N})$,
$\|T_{1}^{H}\|\cdot\|T_{2}^{H}\|=O_{p}((\Delta_{k}/N)^{1/2})=o_{p}(\rho_{k,N})$
by Lemma 2(v), and
$\|T_{1}^{H}\|^{3}=O_{p}(\Delta_{k}^{3/2})=o_{p}(\Delta_{k})$ by Lemma 2(i),
(43) can be further expanded as

$\widehat{H}^{-1}\widehat{h}=H^{-1}\widehat{h}-H^{-1}T^{H}H^{-1}\widehat{h}+H^{-1}T^{H}_{1}H^{-1}T^{H}_{1}H^{-1}\widehat{h}+o_{p}(\rho_{k,N}).$
(45)

Thus,

$\displaystyle\widehat{H}^{-1}\widehat{h}\widehat{h}^{\prime}\widehat{H}^{-1}$
$\displaystyle=$ $\displaystyle H^{-1}\widehat{h}\widehat{h}^{\prime}H^{-1}$
$\displaystyle-H^{-1}\widehat{h}\widehat{h}^{\prime}H^{-1}{T^{H}}^{\prime}H^{-1}-H^{-1}T^{H}H^{-1}\widehat{h}\widehat{h}^{\prime}H^{-1}$
$\displaystyle+H^{-1}\widehat{h}\widehat{h}^{\prime}H^{-1}{T_{1}^{H}}^{\prime}H^{-1}{T_{1}^{H}}^{\prime}H^{-1}+H^{-1}T_{1}^{H}H^{-1}T_{1}^{H}H^{-1}\widehat{h}\widehat{h}^{\prime}H^{-1}$
$\displaystyle+H^{-1}T^{H}_{1}H^{-1}\widehat{h}\widehat{h}^{\prime}H^{-1}{T^{H}_{1}}^{\prime}H^{-1}$
$\displaystyle+o_{p}(\rho_{k,N}).$

The higher-order terms in the MSE are obtained by taking the conditional
expectation on both sides of the above expansion.

First, take (A.1). We derive the conditional expectation of
$\widehat{h}\widehat{h}^{\prime}=hh^{\prime}+hT^{h^{\prime}}+T_{h}h^{\prime}+T^{h}T^{h^{\prime}}$.
By Lemma 2,

$\displaystyle E[hh^{\prime}|z]$ $\displaystyle=$ $\displaystyle
E\left[\frac{f^{\prime}\varepsilon\varepsilon^{\prime}f}{N}|z\right]=\sigma_{\varepsilon}^{2}H,$
$\displaystyle E[T_{1}^{h}T_{1}^{h^{\prime}}|z]$ $\displaystyle=$
$\displaystyle\sigma_{\varepsilon}^{2}\frac{f^{\prime}(I-{P}^{k})^{2}f}{N}=\sigma_{\varepsilon}^{2}e_{f}^{k},$
$\displaystyle E[T_{2}^{h}T_{2}^{h^{\prime}}|z]$ $\displaystyle=$
$\displaystyle\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}\frac{k^{2}}{N}+O_{p}\left(\frac{k}{N}\right)=\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}\frac{k^{2}}{N}+o_{p}\left(\frac{k^{2}}{N}\right)=\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}\frac{k^{2}}{N}+o_{p}(\rho_{k,N}),$
$\displaystyle E[hT_{1}^{h^{\prime}}|z]$ $\displaystyle=$
$\displaystyle-\sigma_{\varepsilon}^{2}\frac{f^{\prime}(I-{P}^{k})f}{N}=-\sigma_{\varepsilon}^{2}\xi_{f}^{k},$
$\displaystyle E[hT_{2}^{h^{\prime}}|z]$ $\displaystyle=$ $\displaystyle
E\left[\frac{f^{\prime}\varepsilon\varepsilon^{\prime}P^{k}u}{N}|z\right]=O_{p}\left(\frac{k}{N}\right)=o_{p}\left(\frac{k^{2}}{N}\right)=o_{p}(\rho_{k,N}),$
$\displaystyle E[T_{1}^{h}T_{2}^{h^{\prime}}|z]$ $\displaystyle=$
$\displaystyle-E\left[\frac{f^{\prime}(I-P^{k})\varepsilon\varepsilon^{\prime}P^{k}u}{N}|z\right]=o_{p}\left(\sqrt{k}\frac{\Delta_{k}^{1/2}}{\sqrt{N}}\right)=o_{p}\left(\frac{k}{N}+\Delta_{k}\right)=o_{p}(\rho_{k,N}).$
(50)

In (50) the third equality holds by the inequality $\sqrt{xy}\leq 2^{-1}(x+y)$
with $x=k/N$ and $y=\Delta_{k}$. Thus,we have

$E[\widehat{h}\widehat{h}^{\prime}|z]=\sigma_{\varepsilon}^{2}H+\sigma_{\varepsilon}^{2}e_{f}^{k}+\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}\frac{k^{2}}{N}-2\sigma_{\varepsilon}^{2}\xi_{f}^{k}+o_{p}(\rho_{k,N}).$
(51)

Next, take (A.1). Since $\|T_{1}^{h}\|\cdot\|T_{2}^{H}\|=o_{p}(\rho_{k,N})$,
$\|T_{2}^{h}\|\cdot\|T_{2}^{H}\|=o_{p}(\rho_{k,N})$,
$\|T^{h}\|^{2}\cdot\|T^{H}\|=o_{p}(\rho_{k,N})$,

$\displaystyle\widehat{h}\widehat{h}^{\prime}H^{-1}T^{H^{\prime}}$
$\displaystyle=$ $\displaystyle
hh^{\prime}H^{-1}T_{1}^{H^{\prime}}+hh^{\prime}H^{-1}T_{2}^{H^{\prime}}+hT_{1}^{h^{\prime}}H^{-1}T_{1}^{H^{\prime}}+hT_{2}^{h^{\prime}}H^{-1}T_{1}^{H^{\prime}}$
(52)
$\displaystyle+T_{1}^{h}h^{\prime}H^{-1}T_{1}^{H^{\prime}}+T_{2}^{h}h^{\prime}H^{-1}T_{1}^{H^{\prime}}+o_{p}(\rho_{k,N}).$

By Lemma 2,

$\displaystyle E[hh^{\prime}H^{-1}T_{1}^{H^{\prime}}|z]$ $\displaystyle=$
$\displaystyle-E\left[\frac{f^{\prime}\varepsilon\varepsilon^{\prime}f}{N}H^{-1}\xi_{f}^{k}|z\right]=-\sigma_{\varepsilon}^{2}\xi_{f}^{k},$
$\displaystyle E[hh^{\prime}H^{-1}T_{2}^{H^{\prime}}|z]$ $\displaystyle=$
$\displaystyle
E[hh^{\prime}H^{-1}u^{\prime}f/N|z]+E[hh^{\prime}H^{-1}f^{\prime}u/N|z]=O_{p}\left(1/N\right)=o_{p}(\rho_{k,N}),$
$\displaystyle E[hT_{1}^{h^{\prime}}H^{-1}T_{1}^{H^{\prime}}|z]$
$\displaystyle=$ $\displaystyle
E\left[\frac{f^{\prime}\varepsilon\varepsilon^{\prime}(I-{P}^{k})f}{N}H^{-1}\xi_{f}^{k}|z\right]=\sigma_{\varepsilon}^{2}\xi_{f}^{k}H^{-1}\xi_{f}^{k},$
$\displaystyle E[T_{1}^{h}h^{\prime}H^{-1}T_{1}^{H^{\prime}}|z]$
$\displaystyle=$ $\displaystyle
E\left[\frac{f^{\prime}(I-{P}^{k})\varepsilon\varepsilon^{\prime}f}{N}H^{-1}\xi_{f}^{k}|z\right]=\sigma_{\varepsilon}^{2}\xi_{f}^{k}H^{-1}\xi_{f}^{k},$
$\displaystyle E[hT_{2}^{h^{\prime}}H^{-1}T_{1}^{H^{\prime}}|z]$
$\displaystyle=$ $\displaystyle
E\left[\frac{f^{\prime}\varepsilon\varepsilon^{\prime}P^{k}u}{N}H^{-1}\xi_{f}^{k}|z\right]=O_{p}(\Delta_{k}^{1/2}k/N)=o_{p}(\rho_{k,N}),$
$\displaystyle E[T_{2}^{h}h^{\prime}H^{-1}T_{1}^{H^{\prime}}|z]$
$\displaystyle=$ $\displaystyle E\left[\frac{u^{\prime}P^{k}\varepsilon
f^{\prime}\varepsilon}{N}H^{-1}\xi_{f}^{k}|z\right]=o_{p}(\rho_{k,N}).$

Thus,

$E[\widehat{h}\widehat{h}^{\prime}H^{-1}T^{H^{\prime}}|z]=-\sigma_{\varepsilon}^{2}\xi_{f}^{k}+2\sigma_{\varepsilon}^{2}\xi_{f}^{k}H^{-1}\xi_{f}^{k}+o_{p}(\rho_{k,N}).$
(53)

By symmetry,

$E[T^{H}H^{-1}\widehat{h}\widehat{h}^{\prime}|z]=-\sigma_{\varepsilon}^{2}\xi_{f}^{k}+2\sigma_{\varepsilon}^{2}\xi_{f}^{k}H^{-1}\xi_{f}^{k}+o_{p}(\rho_{k,N}).$
(54)

Finally, take (A.1) and (A.1). Since $\|T_{1}^{H}\|^{2}=O_{p}(\Delta_{k})$,

$\displaystyle\widehat{h}\widehat{h}^{\prime}H^{-1}{T_{1}^{H}}^{\prime}H^{-1}{T_{1}^{H}}^{\prime}$
$\displaystyle=$ $\displaystyle
hh^{\prime}H^{-1}{T_{1}^{H}}^{\prime}H^{-1}{T_{1}^{H}}^{\prime}+o_{p}(\rho_{k,N}),$
(55) $\displaystyle
T^{H}_{1}H^{-1}\widehat{h}\widehat{h}^{\prime}H^{-1}{T^{H}_{1}}^{\prime}$
$\displaystyle=$ $\displaystyle
T^{H}_{1}H^{-1}hh^{\prime}H^{-1}{T^{H}_{1}}^{\prime}+o_{p}(\rho_{k,N}).$ (56)

Thus,

$\displaystyle
E[hh^{\prime}H^{-1}{T_{1}^{H}}^{\prime}H^{-1}{T_{1}^{H}}^{\prime}|z]$
$\displaystyle=$ $\displaystyle
E\left[\frac{f^{\prime}\varepsilon\varepsilon^{\prime}f}{N}H^{-1}\xi_{f}^{k}H^{-1}\xi_{f}^{k}|z\right]=\sigma_{\varepsilon}^{2}\xi_{f}^{k}H^{-1}\xi_{f}^{k},$
(57) $\displaystyle E[T_{1}^{H}H^{-1}T_{1}^{H}H^{-1}hh^{\prime}|z]$
$\displaystyle=$
$\displaystyle\sigma_{\varepsilon}^{2}\xi_{f}^{k}H^{-1}\xi_{f}^{k},$ (58)
$\displaystyle E[T^{H}_{1}H^{-1}hh^{\prime}H^{-1}{T^{H}_{1}}^{\prime}|z]$
$\displaystyle=$
$\displaystyle\sigma_{\varepsilon}^{2}\xi_{f}^{k}H^{-1}\xi_{f}^{k}.$ (59)

Combining the results together, we get

$\displaystyle\widehat{H}^{-1}\widehat{h}\widehat{h}^{\prime}\widehat{H}^{-1}$
$\displaystyle=$ $\displaystyle
H^{-1}\left(\sigma_{\varepsilon}^{2}H+\sigma_{\varepsilon}^{2}e_{f}^{k}+\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}\frac{k^{2}}{N}-2\sigma_{\varepsilon}^{2}\xi_{f}^{k}+2\sigma_{\varepsilon}^{2}\xi_{f}^{k}\right.$
$\displaystyle\left.-4\sigma_{\varepsilon}^{2}\xi_{f}^{k}H^{-1}\xi_{f}^{k}+3\sigma_{\varepsilon}^{2}\xi_{f}^{k}H^{-1}\xi_{f}^{k}+o_{p}(\rho_{k,N})\right)H^{-1}$
$\displaystyle=$ $\displaystyle
H^{-1}\left(\sigma_{\varepsilon}^{2}H+\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}\frac{k^{2}}{N}+\sigma_{\varepsilon}^{2}e_{f}^{k}-\sigma_{\varepsilon}^{2}\xi_{f}^{k}H^{-1}\xi_{f}^{k}\right)H^{-1}+o_{p}(\rho_{k,N}).$

The desired result is established by noting that

$\displaystyle e_{f}^{k}-\xi_{f}^{k}H^{-1}\xi_{f}^{k}$ $\displaystyle=$
$\displaystyle\frac{f^{\prime}(I-P^{k})^{2}f}{N}-\frac{f^{\prime}(I-P^{k})f}{N}\left(\frac{f^{\prime}f}{N}\right)^{-1}\frac{f^{\prime}(I-P^{k})f}{N}$
(61) $\displaystyle=$
$\displaystyle\frac{f^{\prime}(I-P^{k})(I-P_{f})(I-P^{k})f}{N}$ (62)

where $P_{f}=f(f^{\prime}f)^{-1}f^{\prime}$. ∎

###  A.2 Proof of Theorem 2

In the following proof, let $0<C<\infty$ be a generic constant.

Recall that

$\displaystyle S_{\lambda}(k)$
$\displaystyle=\lambda^{\prime}H^{-1}\left[\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}\frac{k^{2}}{N}+\sigma_{\varepsilon}^{2}\frac{f^{\prime}(I-P^{k})(I-P_{f})(I-P^{k})f}{N}\right]H^{-1}\lambda$
$\displaystyle=\sigma_{\lambda\varepsilon}^{2}\frac{k^{2}}{N}+\sigma_{\varepsilon}^{2}\left[\lambda^{\prime}H^{-1}e_{f}^{k}H^{-1}\lambda-\lambda^{\prime}H^{-1}\xi_{f}^{k}H^{-1}\xi_{f}^{k}H^{-1}\lambda\right],$

where
$\sigma_{\lambda\varepsilon}=\lambda^{\prime}H^{-1}\sigma_{u\varepsilon}$. The
feasible criterion is obtained by replacing the unknown population quantities
$\sigma_{\lambda\varepsilon},\sigma_{\varepsilon}$, $\lambda$, and $f$ with
their sample counterparts, which come from the preliminary estimates. Note
that these preliminary estimators do not depend on $k$. Then, we can rewrite
$\widehat{S}_{\lambda}(k)$ as

$\displaystyle\widehat{S}_{\lambda}(k)$
$\displaystyle=\widetilde{\sigma}_{\lambda\varepsilon}^{2}\frac{k^{2}}{N}+\widetilde{\sigma}_{\varepsilon}^{2}\left[\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}\widetilde{e}_{f}^{k}\widetilde{H}^{-1}\widetilde{\lambda}-\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}\widetilde{\xi}_{f}^{k}\widetilde{H}^{-1}\widetilde{\xi}_{f}^{k}\widetilde{H}^{-1}\widetilde{\lambda}\right]$

where

$\displaystyle\widetilde{e}_{f}^{k}$
$\displaystyle=\frac{X^{\prime}(I-P^{k})^{2}X}{N}+\widetilde{\Sigma}_{u}\left(\frac{2k-\text{tr}((P^{k})^{2})}{N}\right),$
$\displaystyle\widetilde{\xi}_{f}^{k}$
$\displaystyle=\frac{X^{\prime}(I-P^{k})X}{N}+\widetilde{\Sigma}_{u}\frac{k}{N}-\widetilde{\Sigma}_{u},$
$\displaystyle\widetilde{\sigma}^{2}_{\lambda\varepsilon}$
$\displaystyle=(\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}\widetilde{\sigma}_{u\varepsilon})^{2},$
$\displaystyle\widetilde{\Sigma}_{u}$
$\displaystyle=\frac{\widetilde{u}^{\prime}\widetilde{u}}{N}.$

Note that if we use the above expression, an additional term
$\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}\widetilde{\Sigma}_{u}\widetilde{H}^{-1}\widetilde{\Sigma}_{u}\widetilde{H}^{-1}\widetilde{\lambda}$
will be added to $\widehat{S}_{\lambda}(k)$ defined in the main text. However,
the additional term does not depend on $k$ and is irrelevant to get
$\widehat{k}$. By Lemma A.9 in Donald and Newey (2001), it suffices to show

$\sup_{k\in\mathcal{K}_{N}}\frac{\left|\widehat{S}_{\lambda}(k)-S_{\lambda}(k)\right|}{S_{\lambda}(k)}\stackrel{{\scriptstyle
p}}{{\rightarrow}}0.$ (63)

We first define

$\displaystyle V_{f}^{k}$
$\displaystyle=\lambda^{\prime}H^{-1}e_{f}^{k}H^{-1}\lambda-\lambda^{\prime}H^{-1}\xi_{f}^{k}H^{-1}\xi_{f}^{k}H^{-1}\lambda,$
$\displaystyle\widetilde{V}_{f}^{k}$
$\displaystyle=\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}\widetilde{e}_{f}^{k}\widetilde{H}^{-1}\widetilde{\lambda}-\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}\widetilde{\xi}_{f}^{k}\widetilde{H}^{-1}\widetilde{\xi}_{f}^{k}\widetilde{H}^{-1}\widetilde{\lambda},$

and rewrite the LHS of (63) as

$\displaystyle\frac{\left|\widehat{S}_{\lambda}(k)-S_{\lambda}(k)\right|}{S_{\lambda}(k)}$
$\displaystyle=\frac{\left|(\widetilde{\sigma}^{2}_{\lambda\varepsilon}-{\sigma}^{2}_{\lambda\varepsilon})k^{2}/N+(\widetilde{\sigma}^{2}_{\varepsilon}-\sigma^{2}_{\varepsilon})V_{f}^{k}+\widetilde{\sigma}^{2}_{\varepsilon}(\widetilde{V}_{f}^{k}-V_{f}^{k})\right|}{S_{\lambda}(k)}$
$\displaystyle\leq\left|\widetilde{\sigma}^{2}_{\lambda\varepsilon}-{\sigma}^{2}_{\lambda\varepsilon}\right|\frac{k^{2}/N}{S_{\lambda}(k)}+\left|\widetilde{\sigma}_{\varepsilon}^{2}-\sigma_{\varepsilon}^{2}\right|\frac{|V_{f}(k)|}{S_{\lambda}(k)}+\widetilde{\sigma}_{\varepsilon}^{2}\frac{\left|\widetilde{V}_{f}^{k}-V_{f}^{k}\right|}{S_{\lambda}(k)}.$
(64)

Since $\sigma_{\lambda\varepsilon}\neq 0$ and $\sigma^{2}_{\varepsilon}\neq
0$, it holds that

$\frac{k^{2}/N}{S_{\lambda}(k)}=O_{p}(1)~{}~{}\text{and}~{}~{}\frac{|V_{f}(k)|}{S_{\lambda}(k)}=O_{p}(1)$

uniformly over all $k$. Thus, consistency of
$\widetilde{\sigma}_{\lambda\varepsilon}^{2}$ and
$\widetilde{\sigma}^{2}_{\varepsilon}$ implies that the first two terms in
(64) are $o_{p}(1)$ uniformly over $k$.

Let $\mathcal{K}_{N}=\\{1,\ldots,K(N)\\}$. Since
$\widetilde{\sigma}^{2}_{\varepsilon}=O_{p}(1)$ by Assumption 3, it remains to
show that

$\sup_{k\in\mathcal{K}_{N}}\frac{\left|\widetilde{V}_{f}^{k}-V_{f}^{k}\right|}{S_{\lambda}(k)}\stackrel{{\scriptstyle
p}}{{\rightarrow}}0.$ (65)

By the triangle inequality

$\displaystyle\frac{\left|\widetilde{V}_{f}^{k}-V_{f}^{k}\right|}{S_{\lambda}(k)}\leq$
$\displaystyle\frac{\left|\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}\widetilde{e}_{f}^{k}\widetilde{H}^{-1}\widetilde{\lambda}-\lambda^{\prime}H^{-1}e_{f}^{k}H^{-1}\lambda\right|}{S_{\lambda}(k)}$
(66)
$\displaystyle+\frac{\left|\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}\widetilde{\xi}_{f}^{k}\widetilde{H}^{-1}\widetilde{\xi}_{f}^{k}\widetilde{H}^{-1}\widetilde{\lambda}-\lambda^{\prime}H^{-1}\xi_{f}^{k}H^{-1}\xi_{f}^{k}H^{-1}\lambda\right|}{S_{\lambda}(k)}.$
(67)

We first show the uniform convergence of the RHS of (66). Expanding
$\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}\widetilde{e}_{f}^{k}\widetilde{H}^{-1}\widetilde{\lambda}$
and applying the triangle inequality, we get

$\displaystyle\left|\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}\widetilde{e}_{f}^{k}\widetilde{H}^{-1}\widetilde{\lambda}-\lambda^{\prime}H^{-1}e_{f}^{k}H^{-1}\lambda\right|$
$\displaystyle\leq\left|\lambda^{\prime}H^{-1}e_{f}^{k}(\widetilde{H}^{-1}\widetilde{\lambda}-H^{-1}\lambda)\right|+\left|\lambda^{\prime}H^{-1}(\widetilde{e}_{f}^{k}-e_{f}^{k})H^{-1}\lambda\right|+\left|\lambda^{\prime}H^{-1}(\widetilde{e}_{f}^{k}-e_{f}^{k})(\widetilde{H}^{-1}\widetilde{\lambda}-H^{-1}\lambda)\right|$
$\displaystyle\hskip
10.0pt+\left|(\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}-\lambda^{\prime}H^{-1})e_{f}^{k}H^{-1}\lambda\right|+\left|(\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}-\lambda^{\prime}H^{-1})e_{f}^{k}(\widetilde{H}^{-1}\widetilde{\lambda}-H^{-1}\lambda)\right|$
$\displaystyle\hskip
10.0pt+\left|(\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}-\lambda^{\prime}H^{-1})(\widetilde{e}_{f}^{k}-e_{f}^{k})H^{-1}\lambda\right|+\left|(\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}-\lambda^{\prime}H^{-1})(\widetilde{e}_{f}^{k}-e_{f}^{k})(\widetilde{H}^{-1}\widetilde{\lambda}-H^{-1}\lambda)\right|.$

Since $\|\lambda^{\prime}H^{-1}\|=O_{p}(1)$,
$\|\widetilde{\lambda}^{\prime}\widetilde{H}^{-1}-\lambda^{\prime}H^{-1}\|=o_{p}(1)$,
and $\|e_{f}^{k}\|/S_{\lambda}(k)=O_{p}(1)$ uniformly over $k$, it is enough
to show that

$\displaystyle\sup_{k\in\mathcal{K}_{N}}\frac{\left\|\widetilde{e}_{f}^{k}-e_{f}^{k}\right\|}{S_{\lambda}(k)}=o_{p}(1)$

for the uniform convergence of the RHS of (66). Since the dimension of
$e_{f}^{k}$ is fixed, we abuse notation and use
$\widetilde{e}_{f}^{k}-e_{f}^{k}$ for the maximum element of the $d\times d$
matrix. Let $\breve{e}_{f}^{k}:=\tilde{e}_{f}^{k}-N^{-1}u^{\prime}u$. Since
$N^{-1}u^{\prime}u$ does not depend on $k$, we can prove the uniform
convergence for $\breve{e}_{f}^{k}$ instead of $\widetilde{e}_{f}^{k}$.
Recentering each term and applying the triangle inequality, we have

$\displaystyle\frac{\left|\breve{e}_{f}^{k}-e_{f}^{k}\right|}{S_{\lambda}(k)}$
$\displaystyle\leq\frac{2\left|f^{\prime}(I-P^{k})^{2}u\right|}{NS_{\lambda}(k)}+\frac{2\left|u^{\prime}P^{k}u-\Sigma_{u}k\right|}{NS_{\lambda}(k)}+\frac{\left|u^{\prime}(P^{k})^{2}u-\Sigma_{u}\text{tr}((P^{k})^{2})\right|}{NS_{\lambda}(k)}$
$\displaystyle\hskip
10.0pt+(\widetilde{\Sigma}_{u}-\Sigma_{u})\left(\frac{-2N^{-1}k+N^{-1}\text{tr}((P^{k})^{2})}{S_{\lambda}(k)}\right)$
$\displaystyle\equiv\text{I.1}+\text{I.2}+\text{I.3}+\text{I.4}.$

We show that terms I.1–I.4 converge to zero in probability uniformly over $k$.

We first look at I.1. Given any $\delta>0$, it holds almost surely for $z$
that

$\displaystyle\Pr\left(\sup_{k\in\mathcal{K}_{N}}\frac{\left|f^{\prime}(I-P^{k})^{2}u\right|}{NS_{\lambda}(k)}>\delta|z\right)$
$\displaystyle\leq\sum_{k=1}^{K(N)}\frac{E\left[\left|f^{\prime}(I-P^{k})^{2}u\right|^{2}|z\right]}{\delta^{2}\left(NS_{\lambda}(k)\right)^{2}}$
$\displaystyle\leq\frac{C}{\delta^{2}}\sum_{k=1}^{K(N)}\frac{f^{\prime}(I-P^{k})^{4}f}{\left(NS_{\lambda}(k)\right)^{2}}$
$\displaystyle\leq\frac{C}{\delta^{2}}\sum_{k=1}^{K(N)}\frac{(NS_{\lambda}(k))}{\left(NS_{\lambda}(k)\right)^{2}}$
$\displaystyle=\frac{C}{\delta^{2}}\sum_{k=1}^{K(N)}{(NS_{\lambda}(k))^{-1}}\xrightarrow{p}0.$

The first inequality holds by the Markov inequality and the second inequality
holds by Theorem 2 (7) in Whittle (1960). The third inequality holds by Lemma
4 and from the definition of $S_{\lambda}(k)$. The final convergence result
comes from Assumption 3.

We next take our attention to I.2. It holds almost surely for $z$ that

$\displaystyle\Pr\left(\sum_{k\in\mathcal{K}_{N}}\frac{|u^{\prime}P^{k}u-\Sigma_{u}k|}{NS_{\lambda}(k)}>\delta|z\right)$
$\displaystyle\leq\sum_{k=1}^{K(N)}\frac{E\left[|u^{\prime}P^{k}u-\Sigma_{u}\text{tr}(P^{k})|^{2}|z\right]}{\delta^{2}\left(NS_{\lambda}(k)\right)^{2}}$
$\displaystyle\leq\frac{C}{\delta^{2}}\sum_{k=1}^{K(N)}\frac{\text{tr}((P^{k})^{2})}{\left(NS_{\lambda}(k)\right)^{2}}$
$\displaystyle\leq\frac{C}{\delta^{2}}\sum_{k=1}^{K(N)}\frac{NS_{\lambda}(k)}{\left(NS_{\lambda}(k)\right)^{2}}$
$\displaystyle=\frac{C}{\delta^{2}}\sum_{k=1}^{K(N)}{(NS_{\lambda}(k))^{-1}}\xrightarrow{p}0.$

The first and the second inequalities hold by the Markov inequality and by
Theorem 2 (8) in Whittle (1960), respectively. Third third holds by Lemma 1
(ii) and the definition of $S_{\lambda}(k)$.

Similarly, we can show the uniform convergence of I.3 as follows:

$\displaystyle\Pr\left(\sup_{k\in\mathcal{K}_{N}}\frac{|u^{\prime}(P^{k})^{2}u-\Sigma_{u}\text{tr}((P^{k})^{2})|}{NS_{\lambda}(k)}>\delta|z\right)$
$\displaystyle\leq\sum_{k=1}^{K(N)}\frac{E\left[|u^{\prime}(P^{k})^{2}u-\Sigma_{u}\text{tr}((P^{k})^{2})|^{2}|z\right]}{\delta^{2}\left(NS_{\lambda}(k)\right)^{2}}$
$\displaystyle\leq\frac{C}{\delta^{2}}\sum_{k=1}^{K(N)}\frac{\text{tr}((P^{k})^{4})}{\left(NS_{\lambda}(k)\right)^{2}}$
$\displaystyle\leq\frac{C}{\delta^{2}}\sum_{k=1}^{K(N)}\frac{NS_{\lambda}(k)}{\left(NS_{\lambda}(k)\right)^{2}}$
$\displaystyle=\frac{C}{\delta^{2}}\sum_{k=1}^{K(N)}{(NS_{\lambda}(k))^{-1}}\xrightarrow{p}0.$

The third inequality holds by Lemma 1 (ii)–(iii) and the definition of
$S_{\lambda}(k)$.

The uniform convergence of I.4 immediately follow from
$({-2N^{-1}k+N^{-1}\text{tr}((P^{k})^{2})})/{S_{\lambda}(k)}=O_{p}(1)$
uniformly over $k$ and $\widetilde{\Sigma}_{u}-\Sigma_{u}=o_{p}(1)$.

We now show the uniform convergence of (67). For the same arguments above, it
is enough to show that

$\displaystyle\sup_{k\in\mathcal{K}_{N}}\frac{\left|\widetilde{\xi}_{f}^{k}\widetilde{H}^{-1}\widetilde{\xi}_{f}^{k}-\xi_{f}^{k}H^{-1}\xi_{f}^{k}\right|}{S_{\lambda}(k)}=o_{p}(1).$
(68)

We expand $\widetilde{\xi}_{f}^{k}\widetilde{H}^{-1}\widetilde{\xi}_{f}^{k}$
and apply the triangular inequality to get

$\displaystyle\left|\widetilde{\xi}_{f}^{k}\widetilde{H}^{-1}\widetilde{\xi}_{f}^{k}-\xi_{f}^{k}H^{-1}\xi_{f}^{k}\right|$
$\displaystyle\leq\left|\xi_{f}^{k}H^{-1}(\widetilde{\xi}_{f}^{k}-\xi_{f}^{k})\right|+\left|\xi_{f}^{k}(\widetilde{H}^{-1}-H^{-1})\xi_{f}^{k}\right|+\left|\xi_{f}^{k}(\widetilde{H}^{-1}-H^{-1})(\widetilde{\xi}_{f}^{k}-\xi_{f}^{k})\right|$
$\displaystyle\hskip
10.0pt+\left|(\widetilde{\xi}_{f}^{k}-\xi_{f}^{k})H^{-1}\xi_{f}^{k}\right|+\left|(\widetilde{\xi}_{f}^{k}-\xi_{f}^{k})H^{-1}(\widetilde{\xi}_{f}^{k}-\xi_{f}^{k})\right|$
$\displaystyle\hskip
10.0pt+\left|(\widetilde{\xi}_{f}^{k}-\xi_{f}^{k})(\widetilde{H}^{-1}-H^{-1})\xi_{f}^{k}\right|+\left|(\widetilde{\xi}_{f}^{k}-\xi_{f}^{k})(\widetilde{H}^{-1}-H^{-1})(\widetilde{\xi}_{f}^{k}-\xi_{f}^{k})\right|.$

Note again that $\|H^{-1}\|=O_{p}(1)$,
$\|\widetilde{H}^{-1}-H^{-1}\|=o_{p}(1)$, and
$\xi_{f}^{k}/\sqrt{S_{\lambda}(k)}=O_{p}(1)$ uniformly over $k$. Therefore,
the uniform convergence in (68) is implied by

$\displaystyle\sup_{k\in\mathcal{K}_{N}}\frac{|\widetilde{\xi}_{f}^{k}-\xi_{f}^{k}|}{\sqrt{S_{\lambda}(k)}}=o_{p}(1).$
(69)

Recentering each term and applying the triangle inequality, we have

$\displaystyle\frac{|\widetilde{\xi}_{f}^{k}-\xi_{f}^{k}|}{\sqrt{S_{\lambda}(k)}}$
$\displaystyle=\frac{2|f^{\prime}(I-P^{k})u|}{N\sqrt{S_{\lambda}(k)}}+\frac{|N^{-1}u^{\prime}u-\Sigma_{u}|}{\sqrt{S_{\lambda}(k)}}+\frac{|u^{\prime}P^{k}u-\Sigma_{u}k|}{N\sqrt{S_{\lambda}(k)}}+\frac{|\widetilde{\Sigma}_{u}-\Sigma_{u}|}{\sqrt{S_{\lambda}(k)}}+|\widetilde{\Sigma}_{u}-\Sigma_{u}|\frac{k/N}{\sqrt{S_{\lambda}(k)}}$
$\displaystyle\equiv\mbox{II.1}+\mbox{II.2}+\mbox{II.3}+\mbox{II.4}+\mbox{II.5}$

We first show the uniform convergence of II.1:

$\displaystyle\Pr\left(\sup_{k\in\mathcal{K}_{N}}\frac{|f^{\prime}(I-P^{k})u|}{N\sqrt{S_{\lambda}(k)}}>\delta|z\right)$
$\displaystyle\leq\sum_{k=1}^{K(N)}\frac{E\left[\left|f^{\prime}(I-P^{k})u\right|^{2}|z\right]}{\delta^{2}N^{2}S_{\lambda}(k)}$
$\displaystyle\leq\frac{C}{\delta^{2}}\sum_{k=1}^{K(N)}\frac{f^{\prime}(I-P^{k})^{2}f}{N^{2}S_{\lambda}(k)}$
$\displaystyle\leq\frac{C}{\delta^{2}}\sum_{k=1}^{K(N)}\frac{NS_{\lambda}(k)}{N^{2}S_{\lambda}(k)}$
$\displaystyle\leq\frac{C}{\delta^{2}}\sum_{k=1}^{K(N)}\frac{1}{N}\rightarrow
0.$

The same arguments above apply to the first four inequalities. The final
convergence holds from $K(N)^{2}/N\rightarrow 0$.

We next show the uniform convergence of II.2:

$\displaystyle\sup_{k\in\mathcal{K}_{N}}\frac{|N^{-1}u^{\prime}u-\Sigma_{u}|}{\sqrt{S_{\lambda}(k)}}$
$\displaystyle\leq\sup_{k\in\mathcal{K}_{N}}\frac{1}{\sqrt{S_{\lambda}(k)}}O_{p}\left(\frac{1}{\sqrt{N}}\right)$
$\displaystyle\leq\sup_{k\in\mathcal{K}_{N}}\frac{1}{\sqrt{NS_{\lambda}(k)}}O_{p}\left(1\right)\xrightarrow{p}0,$

where the first inequality holds from the central limit theorem and the second
inequality holds from Assumption 3 (ii),
$\sum_{k=1}^{K(N)}(NS_{\lambda}(k))^{-1}\rightarrow 0$.

We next show the uniform convergence of II.3, which holds by the Markov
inequality and the Whittle inequality:

$\displaystyle\Pr\left(\sup_{k\in\mathcal{K}_{N}}\frac{|u^{\prime}P^{k}u-\Sigma_{u}k|}{N\sqrt{S_{\lambda}(k)}}>\delta|z\right)$
$\displaystyle\leq\sum_{k=1}^{K(N)}\frac{E\left[|u^{\prime}P^{k}u-\Sigma_{u}\text{tr}(P^{k})|^{2}|z\right]}{\delta^{2}N^{2}S_{\lambda}(k)}$
$\displaystyle\leq\frac{C}{\delta^{2}}\sum_{k=1}^{K(N)}\frac{\text{tr}((P^{k})^{2})}{N^{2}S_{\lambda}(k)}$
$\displaystyle\leq\frac{C}{\delta^{2}}\sum_{k=1}^{K(N)}\frac{NS_{\lambda}(k)}{N^{2}S_{\lambda}(k)}$
$\displaystyle=\frac{C}{\delta^{2}}\sum_{k=1}^{K(N)}\frac{1}{N}\rightarrow 0.$

We next show the uniform convergence of II.4:

$\displaystyle\sup_{k\in\mathcal{K}_{N}}\frac{|\widetilde{\Sigma}_{u}-\Sigma_{u}|}{\sqrt{S_{\lambda}(k)}}$
$\displaystyle=\sup_{k\in\mathcal{K}_{N}}\frac{O_{p}(N^{-1/2+\phi}S_{\lambda}(k)^{\phi})}{\sqrt{S_{\lambda}(k)}}$
$\displaystyle=\sup_{k\in\mathcal{K}_{N}}\frac{(NS_{\lambda}(k))^{\phi}}{\sqrt{NS_{\lambda}(k)}}\cdot
O_{p}(1)$
$\displaystyle=\sup_{k\in\mathcal{K}_{N}}(NS_{\lambda}(k))^{\phi-\frac{1}{2}}\cdot
O_{p}(1)\xrightarrow{p}0$

The first equality holds from Assumption 3 (iii). The final convergence is
implied from Assumption 3 (ii) and from $\phi\in(0,1/2)$.

The uniform convergence of II.5 follows from
$|\widetilde{\Sigma}_{u}-\Sigma_{u}|=o_{p}(1)$ and
$(k/N)/\sqrt{S_{\lambda}(k)}=O_{p}(1)$ uniformly over $k$. ∎

###  A.3 Proof of Theorem 3

Let $\widehat{e}_{f}^{k}=f^{\prime}(I-\widehat{P}^{k})^{2}f/N$,
$\widehat{\xi}_{f}^{k}=f^{\prime}(I-\widehat{P}^{k})f/N$, and
$\widehat{\Delta}_{k}=\text{tr}(\widehat{e}_{f}^{k})$. Decompose

$\displaystyle\frac{f^{\prime}(I-P^{k})f}{N}$ $\displaystyle=$
$\displaystyle\frac{M_{1}}{M}\frac{f^{\prime}(I-\widehat{P}^{k})f}{N}+\frac{M_{2}}{M}\frac{f^{\prime}(I-\widetilde{P}^{k})f}{N}$
$\displaystyle=$
$\displaystyle\frac{M_{1}}{M}\frac{f^{\prime}(I-\widehat{P}^{k})f}{N}+\frac{M_{2}}{M}\frac{f^{\prime}f}{N}-\frac{M_{2}}{M}\frac{f^{\prime}\widetilde{P}^{k}f}{N}$

and similarly

$\displaystyle\frac{f^{\prime}(I-P^{k})u}{N}$ $\displaystyle=$
$\displaystyle\frac{M_{1}}{M}\frac{f^{\prime}(I-\widehat{P}^{k})u}{N}+\frac{M_{2}}{M}\frac{f^{\prime}u}{N}-\frac{M_{2}}{M}\frac{f^{\prime}\widetilde{P}^{k}u}{N},$
$\displaystyle\frac{f^{\prime}(I-P^{k})\varepsilon}{\sqrt{N}}$
$\displaystyle=$
$\displaystyle\frac{M_{1}}{M}\frac{f^{\prime}(I-\widehat{P}^{k})\varepsilon}{\sqrt{N}}+\frac{M_{2}}{M}\frac{f^{\prime}\varepsilon}{\sqrt{N}}-\frac{M_{2}}{M}\frac{f^{\prime}\widetilde{P}^{k}\varepsilon}{N}.$

Now the expansion (34) and (35) can be further written as

$\displaystyle\frac{X^{\prime}P^{k}X}{N}$ $\displaystyle=$
$\displaystyle\frac{f^{\prime}f}{N}-\frac{f^{\prime}(I-P^{k})f}{N}+\frac{u^{\prime}f+f^{\prime}u}{N}+\frac{u^{\prime}P^{k}u}{N}-\frac{u^{\prime}(I-P^{k})f+f^{\prime}(I-P^{k})u}{N}$
$\displaystyle=$
$\displaystyle\frac{M_{1}}{M}\left(\frac{f^{\prime}f}{N}-\frac{f^{\prime}(I-\widehat{P}^{k})f}{N}+\frac{u^{\prime}f+f^{\prime}u}{N}+\frac{M}{M_{1}}\frac{u^{\prime}P^{k}u}{N}\right.$
$\displaystyle\left.-\frac{u^{\prime}(I-\widehat{P}^{k})f+f^{\prime}(I-\widehat{P}^{k})u}{N}+\frac{M_{2}}{M_{1}}\frac{f^{\prime}\widetilde{P}^{k}f+f^{\prime}\widetilde{P}^{k}u+u^{\prime}\widetilde{P}^{k}f}{N}\right)$

and

$\displaystyle\frac{X^{\prime}P^{k}\varepsilon}{\sqrt{N}}$ $\displaystyle=$
$\displaystyle\frac{f^{\prime}\varepsilon}{\sqrt{N}}-\frac{f^{\prime}(I-P^{k})\varepsilon}{\sqrt{N}}+\frac{u^{\prime}P^{k}\varepsilon}{\sqrt{N}}$
$\displaystyle=$
$\displaystyle\frac{M_{1}}{M}\left(\frac{f^{\prime}\varepsilon}{\sqrt{N}}-\frac{f^{\prime}(I-\widehat{P}^{k})\varepsilon}{\sqrt{N}}+\frac{M}{M_{1}}\frac{u^{\prime}P^{k}\varepsilon}{\sqrt{N}}+\frac{M_{2}}{M_{1}}\frac{f^{\prime}\widetilde{P}^{k}\varepsilon}{\sqrt{N}}\right).$

Therefore,

$\sqrt{N}(\hat{\beta}-\beta_{0})=\left(\frac{X^{\prime}P^{k}X}{N}\right)^{-1}\frac{X^{\prime}P^{k}\varepsilon}{\sqrt{N}}=\widehat{H}^{-1}\widehat{h},$

where

$\displaystyle\widehat{H}$ $\displaystyle=$ $\displaystyle
H+T_{1}^{H}+T_{2}^{H}+Z^{H},$ $\displaystyle H$ $\displaystyle=$
$\displaystyle\frac{f^{\prime}f}{N},$ $\displaystyle T_{1}^{H}$
$\displaystyle=$ $\displaystyle-\frac{f^{\prime}(I-\widehat{P}^{k})f}{N},$
$\displaystyle T_{2}^{H}$ $\displaystyle=$
$\displaystyle\frac{u^{\prime}f+f^{\prime}u}{N},$ $\displaystyle Z^{H}$
$\displaystyle=$
$\displaystyle-\frac{u^{\prime}(I-\widehat{P}^{k})f+f^{\prime}(I-\widehat{P}^{k})u}{N}+\frac{M}{M_{1}}\frac{u^{\prime}P^{k}u}{N}+\frac{M_{2}}{M_{1}}\frac{f^{\prime}\widetilde{P}^{k}f+f^{\prime}\widetilde{P}^{k}u+u^{\prime}\widetilde{P}^{k}f}{N},$

and

$\displaystyle\widehat{h}$ $\displaystyle=$ $\displaystyle
h+T_{1}^{h}+T_{2}^{h}+T_{3}^{h},$ $\displaystyle h$ $\displaystyle=$
$\displaystyle\frac{f^{\prime}\varepsilon}{\sqrt{N}},$ $\displaystyle
T_{1}^{h}$ $\displaystyle=$
$\displaystyle-\frac{f^{\prime}(I-\widehat{P}^{k})\varepsilon}{\sqrt{N}},$
$\displaystyle T_{2}^{h}$ $\displaystyle=$
$\displaystyle\frac{M}{M_{1}}\frac{u^{\prime}P^{k}\varepsilon}{\sqrt{N}},$
$\displaystyle T_{3}^{h}$ $\displaystyle=$
$\displaystyle\frac{M_{2}}{M_{1}}\frac{f^{\prime}\widetilde{P}^{k}\varepsilon}{\sqrt{N}}.$

These expansions simplifies to those in the proof of Theorem 1 when all the
instruments are strong so that $M_{2}=0$ and $M=M_{1}$. Since $0<M_{1}/M\leq
1$ for all $k$ and $K$ and no restriction is imposed on the growth rate of
$M_{2}$ and $M_{1}$ except for $M_{2}/M_{1}\rightarrow(1-C)/C$, we treat
$M/M_{1}$ and $M_{2}/M_{1}$ as constants in the remainder of the proof.

We first specify the convergence rate of each term. First, Lemma 1 holds with
$\widehat{P}^{k}$ and $\widetilde{P}^{k}$ because we do not use the instrument
strength in the proof at all. Second, Lemma 2 holds with $\widehat{P}^{k}$,
$\widehat{\Delta}_{k}$, and $\widehat{\xi}_{f}^{k}$ because $\widehat{P}^{k}$
is the average of projection matrices consist of strong instruments, which is
assumed for $P^{k}$ in Lemma 2. Thus, the convergence rates of $T_{1}^{H}$,
$T_{2}^{H}$, $T_{1}^{h}$, and $T_{2}^{h}$ are the same with those in the proof
of Theorem 1. As is similar to the proof of Theorem 1, let $\rho_{k,N}$ be the
lower (slower) order between $k^{2}/N$ and $\widehat{\Delta}_{k}$ so that the
higher order terms can be conveniently written as $o_{p}(\rho_{k,N})$.
Precisely speaking, $\rho_{k,N}$ may be different from that defined in the
proof of Theorem 1 because $\Delta_{k}\neq\widehat{\Delta}_{k}$. Nevertheless,
for the sake of simplicity we use the same notation.

The convergence rates of $Z^{H}$ and $T_{3}^{h}$ can be found by Lemma 3 (i),
which proves the convergence rate of $f^{\prime}\widetilde{P}^{k}f$,
$f^{\prime}\widetilde{P}^{k}u$, and $f^{\prime}\widetilde{P}^{k}\varepsilon$.
Combined with Lemmas 2 (iii)-(v),

$Z^{H}=O_{p}\left(\frac{\widehat{\Delta}_{k}^{1/2}}{\sqrt{N}}\right)+O_{p}\left(\frac{k}{N}\right)+O_{p}\left(\frac{k}{N}\right)=o_{p}(\rho_{k,N}).$
(70)

Thus, the expansion arguments (42)-(A.1) hold with $\widehat{\rho}_{k,N}$. It
is sufficient to derive the conditional expectations of (A.1)-(A.1) with
$T^{h}=T_{1}^{h}+T_{2}^{h}+T_{3}^{h}$ where
$T_{1}^{h}=O_{p}(\widehat{\Delta}_{k}^{1/2})$, $T_{2}^{h}=O_{p}(k/\sqrt{N})$,
and $T_{3}^{h}=O_{p}(k/\sqrt{N})$.

Now it is sufficient to derive the conditional expectations of (A.1)-(A.1)
with $T^{h}=T_{1}^{h}+T_{2}^{h}+T_{3}^{h}$. This can be done by checking the
convergence rate of terms including $T_{3}^{h}$ and multiplying $M/M_{1}$ to
those terms with $T_{2}^{h}$ whenever it appears in the formula. First take
(A.1). By Lemma 3, the terms that include $T_{3}^{h}$ are

$\displaystyle E[hT_{3}^{h^{\prime}}|z]$
$\displaystyle=\frac{M_{2}}{M_{1}}E\left[\frac{f^{\prime}\varepsilon\varepsilon^{\prime}\widetilde{P}^{k}f}{N}|z\right]=\sigma_{\varepsilon}^{2}\frac{M_{2}}{M_{1}}O_{p}\left(\frac{k}{N}\right)=o_{p}(\rho_{k,N}),$
(71) $\displaystyle E[T_{1}^{h}T_{3}^{h^{\prime}}|z]$
$\displaystyle=-\frac{M_{2}}{M_{1}}E\left[\frac{f^{\prime}(I-\widehat{P}^{k})\varepsilon\varepsilon^{\prime}\widetilde{P}^{k}f}{N}|z\right]=-\sigma_{\varepsilon}^{2}\frac{M_{2}}{M_{1}}\frac{f^{\prime}(I-\widehat{P}^{k})\widetilde{P}^{k}f}{N}=o_{p}(\rho_{k,N}),$
(72) $\displaystyle E[T_{2}^{h}T_{3}^{h^{\prime}}|z]$
$\displaystyle=\frac{MM_{2}}{M_{1}^{2}}E\left[\frac{u^{\prime}P^{k}\varepsilon\varepsilon^{\prime}\widetilde{P}^{k}f}{N}|z\right]=\frac{MM_{2}}{M_{1}^{2}}o_{p}\left(\frac{k}{N}\right)=o_{p}(\rho_{k,N}),$
(73) $\displaystyle E[T_{3}^{h}T_{3}^{h^{\prime}}|z]$
$\displaystyle=\left(\frac{M_{2}}{M_{1}}\right)^{2}E\left[\frac{f^{\prime}\widetilde{P}^{k}\varepsilon\varepsilon^{\prime}\widetilde{P}^{k}f}{N}|z\right]=\sigma_{\varepsilon}^{2}\left(\frac{M_{2}}{M_{1}}\right)^{2}\frac{f^{\prime}\widetilde{P}^{k}\widetilde{P}^{k}f}{N}=o_{p}(\rho_{k,N}),$
(74)

where (72) holds by the inequality $\sqrt{xy}\leq 2^{-1}(x+y)$ and

$O_{p}\left(\widehat{\Delta}_{k}^{1/2}\sqrt{\frac{k}{N}}\right)=O_{p}\left(\widehat{\Delta}_{k}^{1/2}\sqrt{\frac{k^{2}}{N}}\right)\frac{1}{\sqrt{k}}\leq
O_{p}\left(\widehat{\Delta}_{k}+\frac{k^{2}}{N}\right)o(1)=o_{p}(\rho_{k,N}).$
(75)

These terms are added to the right-hand side of (51). Thus, we have

$E[\widehat{h}\widehat{h}^{\prime}|z]=\sigma_{\varepsilon}^{2}H+\sigma_{\varepsilon}^{2}\widehat{e}_{f}^{k}+\left(\frac{M}{M_{1}}\right)^{2}\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}\frac{k^{2}}{N}-2\sigma_{\varepsilon}^{2}\widehat{\xi}_{f}^{k}+o_{p}(\rho_{k,N}).$
(76)

Next take (A.1). Since $\|T_{3}^{h}\|\cdot\|T_{2}^{H}\|=o_{p}(\rho_{k,N})$ and
$\|T^{h}\|^{2}\cdot\|T^{H}\|=o_{p}(\rho_{k,N})$, (52) becomes

$\displaystyle\widehat{h}\widehat{h}^{\prime}H^{-1}T^{H^{\prime}}$
$\displaystyle=$ $\displaystyle
hh^{\prime}H^{-1}T_{1}^{H^{\prime}}+hh^{\prime}H^{-1}T_{2}^{H^{\prime}}+hT_{1}^{h^{\prime}}H^{-1}T_{1}^{H^{\prime}}+hT_{2}^{h^{\prime}}H^{-1}T_{1}^{H^{\prime}}+hT_{3}^{h^{\prime}}H^{-1}T_{1}^{H^{\prime}}$
(77)
$\displaystyle+T_{1}^{h}h^{\prime}H^{-1}T_{1}^{H^{\prime}}+T_{2}^{h}h^{\prime}H^{-1}T_{1}^{H^{\prime}}+T_{3}^{h}h^{\prime}H^{-1}T_{1}^{H^{\prime}}+o_{p}(\rho_{k,N}).$

The term that contains $T_{3}^{h}$ is

$\displaystyle E[hT_{3}^{h^{\prime}}H^{-1}T_{1}^{H^{\prime}}|z]$
$\displaystyle=$
$\displaystyle-\frac{M_{2}}{M_{1}}E\left[\frac{f^{\prime}\varepsilon\varepsilon^{\prime}\widetilde{P}^{k}f}{N}H^{-1}\widehat{\xi}_{f}^{k}|z\right]=-\frac{M_{2}}{M_{1}}\sigma_{\varepsilon}^{2}O_{p}\left(\frac{k}{N}\widehat{\Delta}_{k}^{1/2}\right)=o_{p}(\rho_{k,N}).$

Thus, by symmetry

$\displaystyle E[\widehat{h}\widehat{h}^{\prime}H^{-1}T^{H^{\prime}}|z]$
$\displaystyle=-\sigma_{\varepsilon}^{2}\widehat{\xi}_{f}^{k}+2\sigma_{\varepsilon}^{2}\widehat{\xi}_{f}^{k}H^{-1}\widehat{\xi}_{f}^{k}+o_{p}(\rho_{k,N})$
(78) $\displaystyle=E[T^{H}H^{-1}\widehat{h}\widehat{h}^{\prime}|z].$ (79)

The remaining terms of (A.1) and (A.1) are identical to those in the proof of
Theorem 1.

Combining the results together, we find that the addition of $T_{3}^{h}$ does
not change the formula except for those terms with $T_{2}^{h}$ where $M/M_{1}$
is multiplied. Thus,

$\displaystyle\widehat{H}^{-1}\widehat{h}\widehat{h}^{\prime}\widehat{H}^{-1}$
$\displaystyle=$ $\displaystyle
H^{-1}\left(\sigma_{\varepsilon}^{2}H+\left(\frac{M}{M_{1}}\right)^{2}\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}\frac{k^{2}}{N}+\sigma_{\varepsilon}^{2}\widehat{e}_{f}^{k}-\sigma_{\varepsilon}^{2}\widehat{\xi}_{f}^{k}H^{-1}\widehat{\xi}_{f}^{k}\right)H^{-1}+o_{p}(\rho_{k,N}).$

This proves Theorem 3. ∎

###  A.4 Lemmas

######  Lemma 1.

Under Assumption 1, the followings hold for all $k\geq d$.

  1. (i)

$\text{tr}(P^{k})=k$,

  2. (ii)

$\text{tr}((P^{k})^{2})\leq k$,

  3. (iii)

$\text{tr}((P^{k})^{s+1})\leq\text{tr}((P^{k})^{s})$ for all positive integer
$s$,

  4. (iv)

$\sum_{i}(P^{k}_{ii})^{2}=o_{p}(k)$,

  5. (v)

$\sum_{i\neq j}P^{k}_{ii}P^{k}_{jj}=k^{2}+o_{p}(k)$,

  6. (vi)

$\sum_{i\neq j}P^{k}_{ij}P^{k}_{ij}=\text{tr}((P^{k})^{2})+o_{p}(k)=O_{p}(k)$,

Proof of Lemma 1: (i)

$\text{tr}(P^{k})=\text{tr}\left(\frac{1}{M}\sum_{m=1}^{M}P_{m}^{k}\right)=\frac{1}{M}\sum_{m=1}^{M}\text{tr}(P_{m}^{k})=k.$

(ii) Since $P_{m}^{k}$ is positive semidefinite, so are $P^{k}$ and $I-P^{k}$.
From

$\text{tr}(P^{k})-\text{tr}((P^{k})^{2})=\text{tr}(P^{k}(I-P^{k}))\geq 0,$

it follows that $\text{tr}((P^{k})^{2})\leq k$.

(iii) Since $(P^{k})^{s}$ is symmetric, $(P^{k})^{s}$ is positive
semidefinite. Thus,

$\text{tr}((P^{k})^{s})-\text{tr}((P^{k})^{s+1})=\text{tr}((P^{k})^{s}(I-P^{k}))\geq
0.$

(iv)

$\displaystyle\sum_{i}\left(P^{k}_{ii}\right)^{2}$ $\displaystyle\leq$
$\displaystyle\max_{i}P^{k}_{ii}\cdot\sum_{i}P^{k}_{ii}=\max_{i}P^{k}_{ii}\cdot\text{tr}(P^{k})=k\max_{i}P_{ii}^{K}=o_{p}(k)$

by Lemma 1 (i) and Assumption 1 (viii).

(v) By Lemma 1 (i) and Lemma 1 (iii),

$\sum_{i\neq
j}P^{k}_{ii}P^{k}_{jj}=\sum_{i}P^{k}_{ii}\sum_{j}P^{k}_{jj}-\sum_{i}\left(P^{k}_{ii}\right)^{2}=k^{2}+o_{p}(k).$

(vi) By Lemma 1 (iii) and symmetry of $P^{k}$,

$\displaystyle\sum_{i\neq j}P^{k}_{ij}P^{k}_{ij}$ $\displaystyle=$
$\displaystyle\sum_{i}\sum_{j}P^{k}_{ij}P^{k}_{ij}-\sum_{i}\left(P^{k}_{ii}\right)^{2}=\text{tr}((P^{k})^{2})+o_{p}(k)=O_{p}(k).$

The last equality comes from Lemma 1 (ii).

∎

######  Lemma 2.

Under Assumptions 1 and 2, the followings hold for all $k\geq d$.

  1. (i)

$\Delta_{k}=o_{p}(1)$,

  2. (ii)

$\xi_{f}^{k}=O_{p}(\Delta_{k}^{1/2})$,

  3. (iii)

$f^{\prime}(I-P^{k})\varepsilon/\sqrt{N}=O_{p}(\Delta_{k}^{1/2})$ and
$f^{\prime}(I-P^{k})u/N=O_{p}(\Delta_{k}^{1/2}/\sqrt{N})$,

  4. (iv)

$u^{\prime}P^{k}\varepsilon=O_{p}(k)$ and $u^{\prime}P^{k}u=O_{p}(k)$,

  5. (v)

$\Delta_{k}^{1/2}/\sqrt{N}=o_{p}\left(k/N+\Delta_{k}\right)$,

  6. (vi)

$E[u^{\prime}P^{k}\varepsilon\varepsilon^{\prime}P^{k}u|z]=\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}k^{2}+O_{p}(k)$,

  7. (vii)

$E[f^{\prime}\varepsilon\varepsilon^{\prime}P^{k}u|z]=\sum_{i}^{N}f_{i}P_{ii}^{k}E[\varepsilon_{i}^{2}u_{i}^{\prime}|z_{i}]=O_{p}(k)$,

  8. (viii)

$E[f^{\prime}(I-P^{k})\varepsilon\varepsilon^{\prime}P^{k}u/N|z]=o_{p}\left(\Delta_{k}^{1/2}\sqrt{k}/\sqrt{N}\right)$,

  9. (ix)

$E[hh^{\prime}H^{-1}u^{\prime}f|z]=\sum_{i=1}^{N}f_{i}f_{i}^{\prime}H^{-1}E[\varepsilon_{i}^{2}u_{i}|z_{i}]f_{i}^{\prime}/N^{2}=O_{p}(1/N)$.

Proof of Lemma 2:  (i) By the matrix version of the Cauchy-Schwarz inequality
(Corollary 9.3.9. of Bernstein (2009)) and Jensen’s inequality,

$\displaystyle\text{tr}\left(\frac{f^{\prime}(I-P^{k})^{2}f}{N}\right)$
$\displaystyle=$
$\displaystyle\frac{1}{M^{2}}\sum_{m=1}^{M}\sum_{l=1}^{M}\text{tr}\left(\frac{f^{\prime}(I-P^{k}_{m})(I-P^{k}_{l})f}{N}\right)$
$\displaystyle\leq$
$\displaystyle\frac{1}{M}\sum_{m=1}^{M}\sqrt{\text{tr}\left(\frac{f^{\prime}(I-P^{k}_{m})f}{N}\right)}\frac{1}{M}\sum_{l=1}^{M}\sqrt{\text{tr}\left(\frac{f^{\prime}(I-P^{k}_{l})f}{N}\right)}$
$\displaystyle\leq$
$\displaystyle\text{tr}\left(\frac{f^{\prime}(I-P^{k})f}{N}\right).$

Since $\Delta_{k}\geq 0$, it suffices to show

$\text{tr}\left(\frac{f^{\prime}(I-P^{k})f}{N}\right)=o_{p}(1).$ (80)

Using $(I-P_{m}^{k})Z_{m}^{k}\Pi_{m}^{k}=0$ and the fact that $P^{k}_{m}$ and
$I-P_{m}^{k}$ are positive semi-definite,

$\displaystyle
E\left[\text{tr}\left(\frac{f^{\prime}(I-P^{k})f}{N}\right)\right]$
$\displaystyle=$
$\displaystyle\frac{1}{MN}\sum_{m=1}^{M}E\left[\text{tr}\left(f^{\prime}(I-P_{m}^{k})f\right)\right]$
$\displaystyle=$
$\displaystyle\frac{1}{MN}\sum_{m=1}^{M}E\left[\text{tr}\left((f-Z_{m}^{k}\Pi_{m}^{k})^{\prime}(I-P_{m}^{k})(f-Z_{m}^{k}\Pi_{m}^{k})\right)\right]$
$\displaystyle\leq$
$\displaystyle\frac{1}{MN}\sum_{m=1}^{M}E\left[\text{tr}\left((f-Z_{m}^{k}\Pi_{m}^{k})^{\prime}(f-Z_{m}^{k}\Pi_{m}^{k})\right)\right]$
$\displaystyle=$
$\displaystyle\frac{1}{M}\sum_{m=1}^{M}E\|f(z_{i})-\Pi_{m}^{k^{\prime}}Z_{m,i}^{k}\|^{2}\rightarrow
0$

as $k\rightarrow\infty$ by Assumptions 1 (i) and 2. By Markov inequality, (80)
is shown.

(ii) By the trace inequality, the norm equivalence, the fact that
$\text{tr}(A^{2})\leq\left(\text{tr}(A)\right)^{2}$ for a positive semi-
definite matrix $A$, and $\text{tr}(H)=O_{p}(1)$,

$\displaystyle\left\|\xi_{f}^{k}\right\|^{2}$ $\displaystyle=$
$\displaystyle\text{tr}\left(\frac{(I-P^{k})ff^{\prime}(I-P^{k})}{N}\frac{ff^{\prime}}{N}\right)$
(81) $\displaystyle\leq$
$\displaystyle\left\|\frac{(I-P^{k})ff^{\prime}(I-P^{k})}{N}\right\|\text{tr}(H)$
(82) $\displaystyle\leq$
$\displaystyle\sqrt{\text{tr}\left(e_{f}^{k}e_{f}^{k}\right)}O_{p}(1)$ (83)
$\displaystyle\leq$ $\displaystyle\Delta_{k}\cdot O_{p}(1).$ (84)

Note that $\text{tr}(H)=O_{p}(1)$ by the Markov inequality under Assumptions 1
(i) and 1 (v).

(iii) To show $f^{\prime}(I-P^{k})u/\sqrt{N}=O_{p}(\Delta_{k}^{1/2})$, it
suffices to show it holds for each column vector of $u$, which corresponds to
each element of $u_{i}$. We use the fact that the expectation and trace are
linear operators,
$E[\varepsilon\varepsilon^{\prime}|z]=\sigma_{\varepsilon}^{2}I$,
$E[u_{a}u_{a}^{\prime}|z]=\sigma^{2}_{u,a}I$ where $u_{a}$ is any column
vector of $u$, and the trace inequality to get

$\displaystyle
E\left\|\Delta_{k}^{-1/2}\frac{f^{\prime}(I-P^{k})\varepsilon}{\sqrt{N}}\right\|^{2}$
$\displaystyle=$ $\displaystyle
E\left[\Delta_{k}^{-1}E\left[\left.\text{tr}\left(\frac{f^{\prime}(I-P^{k})\varepsilon\varepsilon^{\prime}(I-P^{k})f}{N}\right)\right|z\right]\right]=\sigma_{\varepsilon}^{2},$
$\displaystyle
E\left\|\Delta_{k}^{-1/2}\frac{f^{\prime}(I-P^{k})u_{a}}{\sqrt{N}}\right\|^{2}$
$\displaystyle=$ $\displaystyle
E\left[\Delta_{k}^{-1}E\left[\left.\text{tr}\left(\frac{f^{\prime}(I-P^{k})u_{a}u_{a}^{\prime}(I-P^{k})f}{N}\right)\right|z\right]\right]=\sigma_{u,a}^{2}.$

By the Markov inequality, for $a>0$,

$\displaystyle
P\left(\left\|\Delta_{k}^{-1/2}\frac{f^{\prime}(I-P^{k})\varepsilon}{\sqrt{N}}\right\|\geq
a\right)$ $\displaystyle=$ $\displaystyle
P\left(\left\|\Delta_{k}^{-1/2}\frac{f^{\prime}(I-P^{k})\varepsilon}{\sqrt{N}}\right\|^{2}\geq
a^{2}\right)$ $\displaystyle\leq$
$\displaystyle\frac{E\left\|\Delta_{k}^{-1/2}\frac{f^{\prime}(I-P^{k})\varepsilon}{\sqrt{N}}\right\|^{2}}{a^{2}}=\frac{\sigma_{\varepsilon}^{2}}{a^{2}}$

and similarly

$\displaystyle
P\left(\left\|\Delta_{k}^{-1/2}\frac{f^{\prime}(I-P^{k})u_{a}}{\sqrt{N}}\right\|\geq
a\right)$ $\displaystyle\leq$ $\displaystyle\frac{\sigma_{u,a}^{2}}{a^{2}}.$

Since $\sigma_{\varepsilon}^{2}$ and $\Sigma_{u}$ are finite, the desired
conclusion follows by taking $a\rightarrow\infty$.

(iv) Since

$\displaystyle E[u^{\prime}P^{k}\varepsilon|z]$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{N}P_{ii}^{k}E[u_{i}\varepsilon_{i}|z_{i}]=\sigma_{u\varepsilon}k,$
(85) $\displaystyle E[u^{\prime}P^{k}u|z]$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{N}P_{ii}^{k}E[u_{i}u_{i}|z_{i}]+\sum_{i\neq
j}P_{ij}^{k}E[u_{i}u_{j}^{\prime}|z_{i},z_{j}]=\Sigma_{u}k,$ (86)

the statement of the Lemma follows by the Markov inequality.

(v) This is Lemma A.3 (vi) of Donald and Newey (2001).

(vi) By the same argument of the proof of Lemma A.3 (iv) of Donald and Newey
(2001) and using our Lemma 1,

$\displaystyle E[u^{\prime}P^{k}\varepsilon\varepsilon^{\prime}P^{k}u|z]$
$\displaystyle=$
$\displaystyle\sum_{i}(P_{ii}^{k})^{2}E[\varepsilon_{i}^{2}u_{i}u_{i}^{\prime}|z_{i}]+\sum_{i\neq
j}P_{ii}^{k}P_{jj}^{k}E[u_{i}\varepsilon_{i}|z_{i}]E[\varepsilon_{j}u_{j}^{\prime}|z_{j}]$
$\displaystyle+\sum_{i\neq
j}(P_{ij}^{k})^{2}E[u_{i}\varepsilon_{i}|z_{i}]E[\varepsilon_{j}u_{j}^{\prime}|z_{j}]+\sum_{i\neq
j}P_{ij}^{k}P_{ji}^{k}E[u_{i}u_{i}^{\prime}|z_{i}]E[\varepsilon_{j}^{2}|z_{j}]$
$\displaystyle=$ $\displaystyle
o_{p}(k)+(k^{2}+o_{p}(k))\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}+O_{p}(k)$
$\displaystyle=$
$\displaystyle\sigma_{u\varepsilon}\sigma_{u\varepsilon}^{\prime}k^{2}+O_{p}(k).$

(vii) This is Lemma A.3 (v) of Donald and Newey (2001).

(viii) The proof proceeds using a similar argument of the proof of Lemma A.3
(viii) of Donald and Newey (2001). Let $Q^{k}=I-P^{k}$. Note that $Q^{k}$ is
not idempotent. For some $a$ and $b$, let $f_{i,a}=f_{a}(z_{i})$ and
$\mu_{i,b}^{k}=E[\varepsilon_{i}^{2}u_{ib}|z_{i}]P^{k}_{ii}$. Let $f_{a}$ and
$\mu_{b}^{k}$ be stacked matrices over $i=1,\ldots,N$. Then by the Cauchy-
Schwarz inequality the absolute value of the $(a,b)$th element of
$E[f^{\prime}(I-P^{k})\varepsilon\varepsilon^{\prime}P^{k}u|z]$ satisfies

$\displaystyle\left|E\left[\sum_{i,j,l,m}f_{i,a}Q^{k}_{ij}\varepsilon_{j}\varepsilon_{l}P^{k}_{lm}u_{mb}|z\right]\right|$
$\displaystyle=$
$\displaystyle\left|\sum_{i,j}f_{i,a}Q^{k}_{ij}E\left[\varepsilon_{j}^{2}u_{jb}|z_{j}\right]P^{k}_{jj}\right|$
$\displaystyle=$
$\displaystyle\left|f_{a}^{\prime}Q^{k}\mu_{b}^{k}\right|\leq\left(f_{a}^{\prime}Q^{k}Q^{k}f_{a}\right)^{1/2}\cdot\left(\mu_{b}^{k^{\prime}}\mu_{b}^{k}\right)^{1/2}.$

Now $f_{a}^{\prime}Q^{k}Q^{k}f_{a}/N=O_{p}(\Delta_{k})$ by the definition of
$\Delta_{k}$. In addition, for some constant $0<C<\infty$,

$\displaystyle\mu_{b}^{k^{\prime}}\mu_{b}^{k}=\sum_{i=1}^{N}E[\varepsilon_{i}^{2}u_{ib}^{\prime}|z_{i}]\left(P^{k}_{ii}\right)^{2}E[\varepsilon_{i}^{2}u_{ib}|z_{i}]\leq
C\sum_{i=1}^{N}\left(P^{k}_{ii}\right)^{2}=o_{p}(k)$

by Assumption 1 (iii) and Lemma 2 (iii). Combining these results, the desired
conclusion follows.

(ix) This is Lemma A.3 (vii) of Donald and Newey (2001).
∎

######  Lemma 3.

Under Assumptions 1 and 4, the followings hold for all $k\geq d$.

  1. (i)

$f^{\prime}\widetilde{P}^{k}f=O_{p}(k)$,
$f^{\prime}\widetilde{P}^{k}u=O_{p}(k)$, and
$f^{\prime}\widetilde{P}^{k}\varepsilon=O_{p}(k)$,

  2. (ii)

$f^{\prime}\widetilde{P}^{k}\widetilde{P}^{k}f=O_{p}(k)$,

  3. (iii)

$f^{\prime}(I-\widehat{P}^{k})\widetilde{P}^{k}f/N=O_{p}\left(\widehat{\Delta}_{k}^{1/2}\sqrt{k}/\sqrt{N}\right)$,

  4. (iv)

$E[u^{\prime}P^{k}\varepsilon\varepsilon^{\prime}\widetilde{P}^{k}f|z]=o_{p}(k)$.

Proof of Lemma 3:  (i) Since $\widetilde{P}^{k}$ is an average of projection
matrices consist of irrelevant instruments, it is sufficient to verify the
convergence rate of $f^{\prime}P_{m}^{k}f$, $f^{\prime}P_{m}^{k}u$, and
$f^{\prime}P_{m}^{k}\varepsilon$ for $m\in\mathcal{M}_{2}$. Since
$P_{m}^{k}=Z_{m}^{k}\left(Z_{m}^{k^{\prime}}Z_{m}^{k}\right)^{-1}Z_{m}^{k^{\prime}}$
is the projection matrix, $f^{\prime}P_{m}^{k}$ lies on the column space
spanned by each of the $k$ instrument in
$Z_{m}^{k}=(z_{m}^{1},\cdots,z_{m}^{k})$. This column space can be
equivalently spanned by a set of orthogonalized instrument vectors we can
write

$f^{\prime}P_{m}^{k}f=\frac{1}{\sqrt{N}}f^{\prime}\widetilde{Z}_{m}^{k}\left(\frac{1}{N}\widetilde{Z}_{m}^{k^{\prime}}\widetilde{Z}_{m}^{k}\right)^{-1}\frac{1}{\sqrt{N}}\widetilde{Z}_{m}^{k^{\prime}}f$
(87)

where
$\widetilde{Z}_{m}^{k}=(\widetilde{z}_{m}^{1},\cdots,\widetilde{z}_{m}^{k})$
is the $N\times k$ orthogonalized instruments matrix. Since
$\frac{1}{N}\widetilde{Z}_{m}^{k^{\prime}}\widetilde{Z}_{m}^{k}$ is a diagonal
matrix, we can write

$\displaystyle\left(\frac{1}{N}\widetilde{Z}_{m}^{k^{\prime}}\widetilde{Z}_{m}^{k}\right)^{-1}$
$\displaystyle=\text{diag}\left(\left(\frac{1}{N}\widetilde{z}_{m}^{k,1^{\prime}}\widetilde{z}_{m}^{k,1}\right)^{-1},\left(\frac{1}{N}\widetilde{z}_{m}^{k,2^{\prime}}\widetilde{z}_{m}^{k,2}\right)^{-1},\cdots,\left(\frac{1}{N}\widetilde{z}_{m}^{k,k^{\prime}}\widetilde{z}_{m}^{k,k}\right)^{-1}\right).$

Write $\widetilde{v}_{m}^{k,j}\equiv
N^{-1}\widetilde{z}_{m}^{k,j^{\prime}}\widetilde{z}_{m}^{k,j}$. Then
$\widetilde{v}_{m}^{k,j}=O_{p}(1)$ and is nonsingular. Using this and the fact
that $N^{-1/2}f^{\prime}\widetilde{z}_{m}^{k,j}=O_{p}(1)$ by CLT, we can show

$\displaystyle f^{\prime}P_{m}^{k}f$
$\displaystyle=\frac{1}{\sqrt{N}}f^{\prime}\widetilde{Z}_{m}^{k}\left(\frac{1}{N}\widetilde{Z}_{m}^{k^{\prime}}\widetilde{Z}_{m}^{k}\right)^{-1}\frac{1}{\sqrt{N}}\widetilde{Z}_{m}^{k^{\prime}}f$
$\displaystyle=\sum_{j=1}^{k}\left(\frac{1}{\sqrt{N}}f^{\prime}\widetilde{z}_{m}^{k,j}\right)\left(\frac{1}{\sqrt{N}}\widetilde{z}_{m}^{k,j^{\prime}}f\right)\frac{1}{\widetilde{v}_{m}^{k,j}}$
$\displaystyle=O_{p}(k).$

By a similar argument using
$N^{-1/2}u^{\prime}\widetilde{z}_{m}^{k,j}=O_{p}(1)$ and
$N^{-1/2}\varepsilon^{\prime}\widetilde{z}_{m}^{k,j}=O_{p}(1)$, which hold by
CLT, we can show that $f^{\prime}\widetilde{P}^{k}u=O_{p}(k)$ and
$f^{\prime}\widetilde{P}^{k}\varepsilon=O_{p}(k)$ .

(ii) Let $f_{a}$ be any column of $f$, $N\times d$ matrix. By the Cauchy-
Schwarz and Jensen’s inequalities,

$\displaystyle f_{a}^{\prime}\widetilde{P}^{k}\widetilde{P}^{k}f_{b}$
$\displaystyle=\frac{1}{M_{2}^{2}}\sum_{m\in\mathcal{M}_{2}}\sum_{l\in\mathcal{M}_{2}}f_{a}^{\prime}P_{m}^{k}P_{l}^{k}f_{b}$
$\displaystyle\leq\frac{1}{M_{2}^{2}}\sum_{m\in\mathcal{M}_{2}}\sum_{l\in\mathcal{M}_{2}}\left(f_{a}^{\prime}P_{m}^{k}f_{a}\right)^{1/2}\left(f_{b}^{\prime}P_{l}^{k}f_{b}\right)^{1/2}$
$\displaystyle\leq\left(f_{a}^{\prime}\widetilde{P}^{k}f_{a}\right)^{1/2}\left(f_{b}^{\prime}\widetilde{P}^{k}f_{b}\right)^{1/2}$
$\displaystyle=O_{p}(k).$

(iii) By the Cauchy-Schwarz and Jensen’s inequalities,

$\displaystyle\frac{f_{a}^{\prime}(I-\widehat{P}^{k})\widetilde{P}^{k}f_{b}}{N}$
$\displaystyle=\frac{1}{M_{1}M_{2}}\sum_{m\in\mathcal{M}_{1}}\sum_{l\in\mathcal{M}_{2}}\frac{f_{a}^{\prime}(I-P_{m}^{k})P_{l}^{k}f_{b}}{N}$
$\displaystyle\leq\frac{1}{M_{1}M_{2}}\sum_{m\in\mathcal{M}_{1}}\sum_{l\in\mathcal{M}_{2}}\left(\frac{f_{a}^{\prime}(I-P_{m}^{k})f_{a}}{N}\right)^{1/2}\left(\frac{f_{b}^{\prime}P_{l}^{k}f_{b}}{N}\right)^{1/2}$
$\displaystyle\leq\left(\frac{f_{a}^{\prime}(I-\widehat{P}^{k})f_{a}}{N}\right)^{1/2}\left(\frac{f_{b}^{\prime}\widetilde{P}^{k}f_{b}}{N}\right)^{1/2}$
$\displaystyle=O_{p}\left(\widehat{\Delta}_{k}^{1/2}\right)O_{p}\left(\sqrt{\frac{k}{N}}\right).$

(iv) The proof proceeds using the same argument with that of Lemma 2 (viii) by
replacing $Q^{k}$ with $\widetilde{P}^{k}$. ∎

######  Lemma 4.

Under Assumption 1, for all $k\geq d$,

$(I-P^{k})^{4}\leq(I-P^{k})^{3}\leq(I-P^{k})^{2}\leq I-P^{k}.$

Proof of Lemma 4:  Since $P_{m}^{k}$ and $I-P_{m}^{k}$ for $m=1,.,,,M$ are
idempotent, they are both positive semi-definite. Thus, $P^{k}$ and $I-P^{k}$
are also positive semi-definite. Since $P^{k}(I-P^{k})$ is symmetric (and thus
is normal), $P^{k}(I-P^{k})\geq 0$. From this, we deduce that

$0\leq(I-P^{k})^{2}\leq I-P^{k}.$ (88)

By Theorem 1(i) of Furuta (1987) with $A=I-P^{k}$, $B=(I-P^{k})^{2}$, $p=q=4$,
and $r=1$, we have

$(I-P^{k})^{3}\leq(I-P^{k})^{2},$ (89)

and with $p=q=2$ and $r=1$, we have

$(I-P^{k})^{4}\leq(I-P^{k})^{3}.$ (90)

Since $(I-P^{k})^{4}\geq 0$ this proves the lemma. ∎

##  Appendix B Inference with CSA-2SLS

In this section we briefly discuss the inference procedure of the CSA-2SLS
estimator. If the bias is the major concern of an estimation problem, then the
bias-minimizing CSA-2SLS can be obtained by setting $k=1$. In other words, we
calculate the equal-weighted average of the fitted values of the endogenous
variable using only one instrument at a time and use that averaged fitted
value as the instrument in the second stage. Since the choice of $k$ is not
data-dependent, the researcher can proceed the standard inference procedure.

More generally, we could use sample-splitting method to make inference with
the optimal $\widehat{k}$. For example, a half of the sample is used to obtain
$\widehat{k}$ and the other half is used to estimate $\widehat{\beta}$ given
$\widehat{k}$. This method is popular in machine-learning literature where
various machine-learning methods are used for model selection or averaging
before inference. Wager and Athey (2018) use sample-splitting to allow
asymptotic inference with causal forests. Chernozhukov, Chetverikov, Demirer,
Duflo, Hansen, Newey, and Robins (2018) use cross-fitting to remove bias
arising from the machine-learning estimates of nonparametric functions.

Either with a fixed $k$ or an optimal $\widehat{k}$ by sample-splitting,
suppose that the CSA-2SLS point estimate is obtained. The next task is to
calculate the standard error, which is robust to heteroskedasticity and
clustering. This is important because many empirical studies report
heteroskedasticity-and-cluster robust standard errors as a measure of
estimation uncertainty.

To present the standard error formula robust to heteroskedasticity and
clustering, we first introduce some definitions and notations for clustered
data. Let $G$ be the number of clusters. The number of observations in each
cluster is $N_{g}$ for $g=1,\ldots,G$. We assume that the clusters are
independent but allow for arbitrary dependence within the cluster. Let
$y_{g}$, $X_{g}$, and $P^{k}_{g}$ be the $N_{g}\times 1$, $N_{g}\times d$, and
$N_{g}\times N$ submatrix of $P^{k}$ corresponding the $g$th cluster,
respectively. Define $\widehat{\varepsilon}_{g}=y_{g}-X_{g}\widehat{\beta}$.
For i.i.d. sampling, set $N_{g}=1$ and $G=N$. Let $\Sigma$ be the covariance
matrix of $\sqrt{N}(\widehat{\beta}-\beta)$ under the standard large $N$ and
fixed $k$ asymptotics. Hansen and Lee (2019) provide sufficient conditions for
consistency, asymptotic normality, and consistency of variance estimators.

A covariance matrix estimator robust to heteroskedasticity and clustering is
given by

$\widehat{\Sigma}=N\left(X^{\prime}P^{k}X\right)^{-1}\sum_{g=1}^{G}X^{\prime}P^{k^{\prime}}_{g}\widehat{\varepsilon}_{g}\widehat{\varepsilon}_{g}^{\prime}P^{k}_{g}X\left(X^{\prime}P^{k}X\right)^{-1}.$
(91)

The standard error can be obtained by taking the diagonal elements of
$\sqrt{\widehat{\Sigma}/N}$.

##  Appendix C Simulation Results with $\sigma_{u\varepsilon}=0.1$ (low
endogeneity) and/or $\rho_{z}=0$ (no correlation among instruments)

[ht]

Table 5: $N=100$, $K=20$, $\sigma_{u\varepsilon}=0.1$, $\rho_{z}=0$

| MSE | Bias | MAD | Median Bias | Range | Coverage |  Mean($\widehat{k}$) |  Med($\widehat{k}$)
---|---|---|---|---|---|---|---|---
$R_{f}^{2}=0.01$ (weak IV signal)
$\pi_{0}:$ flat |  |  |  |  |  |
OLS | 0.020 | 0.102 | 0.068 | 0.103 | 0.257 | 0.802 | – | –
2SLS | 0.063 | 0.093 | 0.146 | 0.091 | 0.609 | 0.926 | – | –
DN | 9.452 | 0.016 | 0.440 | 0.091 | 2.531 | 0.989 | 3.930 | 2.000
KO | 0.071 | 0.096 | 0.157 | 0.089 | 0.630 | 0.922 | – | –
CSA | 0.069 | 0.094 | 0.151 | 0.084 | 0.631 | 0.931 | 5.120 | 1.000
CSA.1 | 0.071 | 0.092 | 0.166 | 0.095 | 0.632 | 0.936 | 1.000 | 1.000
$\pi_{0}:$ decreasing |  |  |  |  |  |
OLS | 0.020 | 0.102 | 0.068 | 0.104 | 0.260 | 0.800 | – | –
2SLS | 0.063 | 0.094 | 0.148 | 0.086 | 0.588 | 0.925 | – | –
DN | 7.038 | 0.122 | 0.386 | 0.076 | 2.291 | 0.980 | 3.994 | 2.000
KO | 0.071 | 0.092 | 0.157 | 0.085 | 0.637 | 0.919 | – | –
CSA | 0.069 | 0.095 | 0.151 | 0.088 | 0.635 | 0.922 | 4.711 | 1.000
CSA.1 | 0.070 | 0.091 | 0.171 | 0.092 | 0.616 | 0.932 | 1.000 | 1.000
$\pi_{0}:$ half-zero |  |  |  |  |  |
OLS | 0.020 | 0.102 | 0.068 | 0.106 | 0.262 | 0.799 | – | –
2SLS | 0.064 | 0.093 | 0.154 | 0.086 | 0.593 | 0.925 | – | –
DN | 138.551 | 0.198 | 0.470 | 0.100 | 2.666 | 0.987 | 3.985 | 2.000
KO | 0.072 | 0.095 | 0.163 | 0.090 | 0.613 | 0.924 | – | –
CSA | 0.071 | 0.093 | 0.164 | 0.086 | 0.625 | 0.924 | 5.169 | 1.000
CSA.1 | 0.072 | 0.092 | 0.163 | 0.085 | 0.645 | 0.924 | 1.000 | 1.000
$R_{f}^{2}=0.1$ (strong IV signal)
$\pi_{0}:$ flat |  |  |  |  |  |
OLS | 0.017 | 0.093 | 0.066 | 0.095 | 0.242 | 0.810 | – | –
2SLS | 0.040 | 0.063 | 0.119 | 0.067 | 0.484 | 0.924 | – | –
DN | 3.776 | -0.059 | 0.238 | 0.059 | 1.316 | 0.965 | 7.648 | 5.000
KO | 0.044 | 0.062 | 0.123 | 0.066 | 0.500 | 0.924 | – | –
CSA | 0.044 | 0.061 | 0.122 | 0.057 | 0.510 | 0.935 | 4.234 | 1.000
CSA.1 | 0.043 | 0.057 | 0.134 | 0.060 | 0.511 | 0.942 | 1.000 | 1.000
$\pi_{0}:$ decreasing |  |  |  |  |  |
OLS | 0.018 | 0.093 | 0.065 | 0.093 | 0.250 | 0.812 | – | –
2SLS | 0.039 | 0.062 | 0.120 | 0.058 | 0.475 | 0.928 | – | –
DN | 0.149 | 0.032 | 0.182 | 0.037 | 0.779 | 0.954 | 6.582 | 5.000
KO | 0.044 | 0.056 | 0.128 | 0.059 | 0.510 | 0.922 | – | –
CSA | 0.043 | 0.058 | 0.128 | 0.057 | 0.507 | 0.935 | 2.482 | 1.000
CSA.1 | 0.044 | 0.058 | 0.137 | 0.055 | 0.511 | 0.930 | 1.000 | 1.000
$\pi_{0}:$ half-zero |  |  |  |  |  |
OLS | 0.018 | 0.093 | 0.064 | 0.095 | 0.245 | 0.819 | – | –
2SLS | 0.041 | 0.062 | 0.123 | 0.057 | 0.493 | 0.925 | – | –
DN | 45.521 | -0.151 | 0.248 | 0.074 | 1.674 | 0.969 | 8.197 | 11.000
KO | 0.044 | 0.061 | 0.122 | 0.057 | 0.525 | 0.924 | – | –
CSA | 0.045 | 0.061 | 0.132 | 0.053 | 0.518 | 0.935 | 3.796 | 1.000
CSA.1 | 0.045 | 0.060 | 0.134 | 0.060 | 0.512 | 0.930 | 1.000 | 1.000

  * •

Note: See the Note below Table 1 for details.

[ht]

Table 6: $N=100$, $K=20$, $\sigma_{u\varepsilon}=0.9$, $\rho_{z}=0$

| MSE | Bias | MAD | Median Bias | Range | Coverage |  Mean($\widehat{k}$) |  Med($\widehat{k}$)
---|---|---|---|---|---|---|---|---
$R_{f}^{2}=0.01$ (weak IV signal)
$\pi_{0}:$ flat |  |  |  |  |  |
OLS | 0.795 | 0.890 | 0.032 | 0.889 | 0.119 | 0.000 | – | –
2SLS | 0.744 | 0.855 | 0.074 | 0.860 | 0.290 | 0.001 | – | –
DN | 29.064 | 1.063 | 0.293 | 0.875 | 1.928 | 0.548 | 2.438 | 1.000
KO | 0.744 | 0.854 | 0.080 | 0.856 | 0.304 | 0.001 | – | –
CSA | 0.738 | 0.850 | 0.080 | 0.855 | 0.305 | 0.001 | 4.942 | 1.000
CSA.1 | 0.736 | 0.849 | 0.079 | 0.847 | 0.306 | 0.000 | 1.000 | 1.000
$\pi_{0}:$ decreasing |  |  |  |  |  |
OLS | 0.795 | 0.891 | 0.033 | 0.889 | 0.121 | 0.000 | – | –
2SLS | 0.746 | 0.856 | 0.075 | 0.860 | 0.286 | 0.000 | – | –
DN | 28.107 | 0.668 | 0.327 | 0.745 | 2.227 | 0.573 | 2.402 | 1.000
KO | 0.720 | 0.837 | 0.086 | 0.841 | 0.345 | 0.002 | – | –
CSA | 0.739 | 0.851 | 0.083 | 0.859 | 0.305 | 0.000 | 3.988 | 1.000
CSA.1 | 0.734 | 0.849 | 0.076 | 0.849 | 0.297 | 0.001 | 1.000 | 1.000
$\pi_{0}:$ half-zero |  |  |  |  |  |
OLS | 0.795 | 0.890 | 0.032 | 0.890 | 0.120 | 0.000 | – | –
2SLS | 0.746 | 0.856 | 0.070 | 0.860 | 0.289 | 0.001 | – | –
DN | 24.579 | 0.818 | 0.267 | 0.886 | 1.984 | 0.549 | 2.474 | 1.000
KO | 0.747 | 0.856 | 0.074 | 0.858 | 0.300 | 0.001 | – | –
CSA | 0.739 | 0.851 | 0.077 | 0.856 | 0.308 | 0.001 | 5.273 | 1.000
CSA.1 | 0.735 | 0.849 | 0.081 | 0.844 | 0.308 | 0.000 | 1.000 | 1.000
$R_{f}^{2}=0.1$ (strong IV signal)
$\pi_{0}:$ flat |  |  |  |  |  |
OLS | 0.660 | 0.811 | 0.035 | 0.811 | 0.137 | 0.000 | – | –
2SLS | 0.345 | 0.575 | 0.081 | 0.580 | 0.316 | 0.017 | – | –
DN | 69.855 | 0.294 | 0.274 | 0.576 | 1.832 | 0.606 | 2.917 | 1.000
KO | 0.341 | 0.570 | 0.085 | 0.573 | 0.324 | 0.027 | – | –
CSA | 0.317 | 0.545 | 0.096 | 0.550 | 0.353 | 0.047 | 3.344 | 1.000
CSA.1 | 0.305 | 0.536 | 0.084 | 0.539 | 0.336 | 0.036 | 1.000 | 1.000
$\pi_{0}:$ decreasing |  |  |  |  |  |
OLS | 0.660 | 0.811 | 0.035 | 0.810 | 0.133 | 0.000 | – | –
2SLS | 0.346 | 0.576 | 0.082 | 0.578 | 0.311 | 0.016 | – | –
DN | 1482.839 | -1.215 | 0.206 | 0.188 | 1.061 | 0.785 | 1.738 | 1.000
KO | 0.234 | 0.459 | 0.101 | 0.457 | 0.373 | 0.147 | – | –
CSA | 0.313 | 0.542 | 0.093 | 0.540 | 0.344 | 0.050 | 1.089 | 1.000
CSA.1 | 0.306 | 0.537 | 0.087 | 0.541 | 0.329 | 0.040 | 1.000 | 1.000
$\pi_{0}:$ half-zero |  |  |  |  |  |
OLS | 0.660 | 0.811 | 0.036 | 0.812 | 0.130 | 0.000 | – | –
2SLS | 0.344 | 0.574 | 0.077 | 0.582 | 0.299 | 0.016 | – | –
DN | 21640.609 | -3.751 | 0.342 | 0.758 | 2.068 | 0.604 | 2.559 | 1.000
KO | 0.331 | 0.561 | 0.085 | 0.570 | 0.332 | 0.031 | – | –
CSA | 0.315 | 0.545 | 0.087 | 0.546 | 0.338 | 0.047 | 3.640 | 1.000
CSA.1 | 0.303 | 0.535 | 0.084 | 0.530 | 0.333 | 0.046 | 1.000 | 1.000

  * •

Note: See the Note below Table 1 for details.

[ht]

Table 7: $N=100$, $K=20$, $\sigma_{u\varepsilon}=0.1$, $\rho_{z}=0.5$

| MSE | Bias | MAD | Median Bias | Range | Coverage |  Mean($\widehat{k}$) |  Med($\widehat{k}$)
---|---|---|---|---|---|---|---|---
$R_{f}^{2}=0.01$ (weak IV signal)
$\pi_{0}:$ flat |  |  |  |  |  |
OLS | 0.018 | 0.093 | 0.065 | 0.094 | 0.250 | 0.810 | – | –
2SLS | 0.040 | 0.064 | 0.126 | 0.062 | 0.479 | 0.927 | – | –
DN | 0.186 | 0.012 | 0.190 | 0.041 | 0.818 | 0.957 | 6.200 | 4.000
KO | 0.044 | 0.056 | 0.133 | 0.058 | 0.518 | 0.922 | – | –
CSA | 0.089 | 0.008 | 0.174 | 0.032 | 0.735 | 0.962 | 2.526 | 1.000
CSA.1 | 0.099 | -0.005 | 0.192 | 0.001 | 0.761 | 0.974 | 1.000 | 1.000
$\pi_{0}:$ decreasing |  |  |  |  |  |
OLS | 0.020 | 0.102 | 0.067 | 0.104 | 0.265 | 0.800 | – | –
2SLS | 0.063 | 0.094 | 0.152 | 0.085 | 0.582 | 0.925 | – | –
DN | 3190.705 | 1.309 | 0.397 | 0.086 | 2.247 | 0.983 | 3.935 | 2.000
KO | 0.071 | 0.092 | 0.159 | 0.086 | 0.635 | 0.920 | – | –
CSA | 0.183 | 0.079 | 0.225 | 0.080 | 1.013 | 0.973 | 4.617 | 1.000
CSA.1 | 0.226 | 0.065 | 0.290 | 0.061 | 1.150 | 0.987 | 1.000 | 1.000
$\pi_{0}:$ half-zero |  |  |  |  |  |
OLS | 0.020 | 0.102 | 0.067 | 0.105 | 0.262 | 0.800 | – | –
2SLS | 0.063 | 0.093 | 0.152 | 0.084 | 0.595 | 0.927 | – | –