[@Zhang09-multistage] tracks the progress from one stage $\ell-1$ to the next stage $\ell$ using a bound on 2-norm parameter estimate, while in the current proof we track the progress using the set of variables that differ significantly from the true variables. Moreover, in [@Zhang09-multistage], we compare the current estimated parameter to the true parameter $\bar{{{\mathbf w}}}$, which is sufficient for parameter estimation. However, in order to establish feature selection result of this paper, it is necessary to compare the current estimated parameter to the least squares solution $\tilde{{{\mathbf w}}}$ within the true feature set $\bar{F}$ as defined below in (\[eq:ls\]). These subtle technical differences mean that many details in the proofs presented below differ from that of [@Zhang09-multistage].

Auxiliary lemmas
----------------

We first introduce some definitions. Consider the positive semi-definite matrix $A= n^{-1} X^\top X \in {{\mathbb R}}^{d \times d}$. Given $s,k \geq 1$ such that $s + k \leq d$. Let $I,J$ be disjoint subsets of $\{1,\ldots,d\}$ with $k$ and $s$ elements respectively. Let $A_{I,I} \in R^{k \times k}$ be the restriction of $A$ to indices $I$, $A_{I,J} \in R^{k \times s}$ be the restriction of $A$ to indices $I$ on the left and $J$ on the right. Similarly 