# Tuning Parameter Selection for Penalized Estimation via $R^{2}$

Julia C. Holter  Jonathan W. Stallrich

###### Abstract

The tuning parameter selection strategy for penalized estimation is crucial to
identify a model that is both interpretable and predictive. However, popular
strategies (e.g., minimizing average squared prediction error via cross-
validation) tend to select models with more predictors than necessary. This
paper proposes a simple, yet powerful cross-validation strategy based on
maximizing squared correlations between the observed and predicted values,
rather than minimizing squared error loss for the purposes of support
recovery. The strategy can be applied to all penalized least-squares
estimators and we show that, under certain conditions, the metric implicitly
performs a bias adjustment. Specific attention is given to the Lasso
estimator, in which our strategy is closely related to the Relaxed Lasso
estimator. We demonstrate our approach on a functional variable selection
problem to identify optimal placement of surface electromyogram sensors to
control a robotic hand prosthesis.

###### keywords:

Cross-Validation , Model/Variable Selection , Functional Data , Relaxed Lasso

††journal: Computational Statistics and Data Analysis

[inst1]organization=Department of Statistics,addressline=North Carolina State
University, city=Raleigh, state=NC, postcode=27695, country=USA

##  1 Introduction

Many statistical problems aim to build a predictive model from a large set of
potential predictor variables. Variable selection is often performed to select
a predictive model that depends on as few predictor variables as possible. For
example, [1] discussed an important functional variable selection problem to
develop a prosethesis controller (PC) for a robotic hand. Electromyogram (EMG)
signals from surface sensors placed on the residual forearm muscles of an
amputee were input into a PC and translated into movement of the robotic hand.
For able-bodied subjects, it is known that certain movements are caused by
contractions of only a few muscles, implying a predictive PC requires a few
strategically-placed EMG sensors.

This paper concerns problems that are well approximated by a sparse linear
model:

$\textbf{y}_{n\times 1}=\textbf{X}_{n\times p}\boldsymbol{\beta}_{p\times
1}^{*}+\boldsymbol{\epsilon}_{n\times 1}$ (1)

where $E(\boldsymbol{\epsilon})=\textbf{0}$,
$V(\boldsymbol{\epsilon})=\boldsymbol{\Sigma}$,
${\boldsymbol{\beta}^{*}}^{T}=(\beta_{1}^{*},...,\beta_{p^{*}}^{*},0,...,0)$,
and $p^{*}$ is the number of important variables. Without loss of generality,
assume y and predictor variables, X, are centered and the diagonals of
$\textbf{X}^{T}\textbf{X}$ equal $n$. Let
$\mathcal{M}^{*}=\\{j\,:\,\beta_{j}^{*}\neq 0\\}$ denote the support of
$\boldsymbol{\beta}^{*}$. A predictive model’s estimate for
$\boldsymbol{\beta}^{*}$, denoted $\hat{\boldsymbol{\beta}}$, will ideally
also have support $\mathcal{M}^{*}$ and will be close to
$\boldsymbol{\beta}^{*}$ in some other sense, such as
$||\hat{\boldsymbol{\beta}}-\boldsymbol{\beta}^{*}||_{2}^{2}=\sum_{j}(\hat{\beta}_{j}-\beta_{j}^{*})^{2}$.

For high-dimensional data like that in [1], simultaneous support recovery and
parameter estimation can be performed via penalized estimation. A penalized
estimator is represented generally by
$\hat{\boldsymbol{\beta}}_{\lambda}=\operatorname*{arg\,min}L(\boldsymbol{\beta})+P_{\lambda}(\boldsymbol{\beta})$
where $L(\boldsymbol{\beta})$ is a loss function comparing y to its predicted
values $\textbf{X}\boldsymbol{\beta}$, and $P_{\lambda}(\boldsymbol{\beta})$
is a penalty function that depends on tuning parameter(s) $\lambda\geq 0$.
Henceforth we let
$L(\boldsymbol{\beta})=(2n)^{-1}||\textbf{y}-\textbf{X}\boldsymbol{\beta}||_{2}^{2}$.
Penalty functions can take myriad forms but we are interested in those that
increase as $\boldsymbol{\beta}$ moves away from 0. The Lasso [2] penalty,
$P_{\lambda}(\boldsymbol{\beta})=\lambda||\boldsymbol{\beta}||_{1}=\lambda\sum_{j=1}^{p}|\beta_{j}|$,
is one such penalty that can force estimates to equal 0, thereby performing
simultaneous variable selection and estimation. For such sparsity-inducing
estimators, we are interested in comparing the estimated support,
$\mathcal{M}_{\lambda}=\\{j\,:\,\hat{\beta}_{\lambda,j}\neq 0\\}$, to
$\mathcal{M}^{*}$.

The chosen $\lambda$ balances the importance of minimizing
$P_{\lambda}(\boldsymbol{\beta})$ relative to $L(\boldsymbol{\beta})$, so it
is recommended to explore the tuning parameter space to identify an “optimal”
value. Potential criteria for an optimal value include identifying a
$\hat{\boldsymbol{\beta}}_{\lambda}$ that minimizes
$||\hat{\boldsymbol{\beta}}_{\lambda}-\boldsymbol{\beta}^{*}||_{2}^{2}$,
minimizes
$||\textbf{X}\hat{\boldsymbol{\beta}}_{\lambda}-\textbf{X}\boldsymbol{\beta}^{*}||_{2}^{2}$,
or has $\mathcal{M}_{\lambda}=\mathcal{M}^{*}$. The latter criterion is
referred to as support recovery and is the primary focus of this paper. Even
if a $\lambda$ exists where $\mathcal{M}_{\lambda}=\mathcal{M}^{*}$, there is
no guarantee that we will be able to correctly identify it. Popular
approaches, such as minimizing information criteria [3, 4] or minimizing
average squared prediction error from $K$-fold cross validation often choose a
$\lambda$ that overselects the number of important variables [5, 6], i.e.,
$\mathcal{M}^{*}\subset\mathcal{M}_{\lambda}$. Post-selection inference
techniques [7, 8] and multi-stage modifications [9, 10] can correct for this
overselection, albeit with added computations and assumptions.

This paper proposes a new $K$-fold cross validation strategy that assesses the
predictive quality of $\hat{\boldsymbol{\beta}}_{\lambda}$ by maximizing
average squared prediction correlation rather than minimizing average squared
prediction error. The scale invariance of correlation reduces the impact of
the potential shrinkage when estimating large $\beta^{*}_{j}$ that would
burgeon squared prediction error. To demonstrate, we simulated a dataset
$\\{\textbf{y},\textbf{X}\\}$ with $n=p=100$, $p^{*}=5$, and $\sigma^{2}=1$,
and generated the entire solution path of the Lasso. The elements of X were
independently sampled from a standard Normal distribution and then
appropriately centered and scaled. The active coefficients in
$\boldsymbol{\beta}^{*}$ were $\\{2.13,1.81,-2.46,-1.89,-2.51\\}$. Many
$\lambda$ values had $\mathcal{M}_{\lambda}=\mathcal{M}^{*}$, so a wide range
were optimal for support recovery. Figure 1 plots y against their in-sample
predictions, $\hat{\textbf{y}}_{\lambda}$, under four different $\lambda$
values, and summarizes the models’ number of false positives (FP), average
squared prediction error (APE) from 10-fold CV, and in-sample squared
prediction correlation ($R^{2}$). Figure 1(a) corresponds to the $\lambda$
determined by a cross validation one-standard-error rule (CV 1SE) and Figure
1(b) corresponds to the $\lambda$ that minimizes
$||\textbf{X}\boldsymbol{\beta}^{*}-\textbf{X}\hat{\boldsymbol{\beta}}_{\lambda}||_{2}$
(Min PB). Among the $\lambda$ having $\mathcal{M}_{\lambda}=\mathcal{M}^{*}$,
we consider the largest and smallest in Figures 1(c) and 1(d), respectively.

(a) CV 1SE

(b) Min PB

(c) Max $\lambda$, $\mathcal{M}_{\lambda}=\mathcal{M}^{*}$

(d) Min $\lambda$, $\mathcal{M}_{\lambda}=\mathcal{M}^{*}$

Figure 1: In-sample observations ($y_{i}$) versus predictions ($\hat{y}_{i}$)
for Lasso estimator at four values of $\lambda$ for a data set with $n=100$,
$p=100$, and $p^{*}=5$. We also report average prediction error (APE) under
10-fold CV, in-sample squared prediction correlation ($R^{2}$), and the number
of false positives (FP). The minimum APE plus-or-minus one standard error is
$0.2844\pm 0.0262$ and all estimates have
$\mathcal{M}^{*}\subseteq\mathcal{M}_{\lambda}$.

All models shown in Figure 1 include the 5 important variables. Both the CV
1SE and Min PB models have small APE and large $R^{2}$, but many false
positives. The model in Figure 1(c) has no false positives, but has relatively
large APE. The model under Figure 1(d) also has no false positives and its in-
sample $R^{2}$ approximately equals that of the Min PB estimate. This
motivates a tuning parameter selection strategy based on an $R^{2}$ metric
rather than APE to compromise between prediction and variable selection.

After justifying the $R^{2}$ metric, we highlight and investigate an
equivalence between the metric and a multiplicative adjustment on
$\hat{\boldsymbol{\beta}}_{\lambda}$, referred to here as the
$\alpha$-modification. We argue that for $\hat{\boldsymbol{\beta}}_{\lambda}$
with certain statistical properties, the adjustment can reduce the bias of
$\hat{\boldsymbol{\beta}}_{\lambda}$ thereby improving its predictive
potential. We go on to study the $\alpha$-modification for the Lasso,
highlighting its similarities to the Nonnegative Garrote [11] and Relaxed
Lasso[10]. Unlike these two methods, the $\alpha$-modification can be applied
to any penalized least-squares estimator, including non-convex penalties,
without additional computational complexity.

This paper is organized as follows. In Section 2 we review classes of
penalized estimators, popular tuning parameter selection strategies, and post-
selection inference methods. Section 3 justifies the value in the $R^{2}$
metric and provides statistical properties for a general class of penalized
estimators. Finite-sample properties are then derived for the
$\alpha$-modification for the Lasso in Section 4. Section 5 presents a
simulation study of the new approaches and Section 6 applies our new methods
to the EMG data of [1]. Section 7 provides a discussion on the implications of
our new framework for evaluating model fit and propose avenues of future
research. Proofs of all results may be found in the Supplementary Materials.

##  2 Background

###  2.1 Classes of Penalized Estimators

Consider the class of penalties
$P_{\lambda}(\boldsymbol{\beta})=||\boldsymbol{\beta}||_{q}^{q}=\sum_{j}|\beta_{j}|^{q}$
, $q>0$, corresponding to the so-called bridge estimators [12]. It has been
shown [13] that for $q\leq 1$ and large enough $\lambda$, the penalized
estimate will have some $\hat{\beta}_{\lambda,j}=0$, yielding a continuous
approach to the intractable exploration of all submodels. Under appropriate
regularity conditions, the limiting distributions of such
$\hat{\beta}_{\lambda,j}$ whose corresponding $\beta_{j}^{*}=0$ can have
positive probability mass at 0 when $p$ is fixed, meaning that they are
capable of support recovery. The result has been shown to hold as $p$ and $n$
grow to infinity, under certain growth rate conditions [14]. However, the
large $\lambda$ necessary for this to occur may cause
$|\hat{\beta}_{\lambda,j}|<<|\beta^{*}_{j}|$ for $j\in\mathcal{M}^{*}$,
thereby inflating criteria commonly used for tuning parameter selection. A
$\lambda$ yielding support recovery may then be ignored by popular tuning
parameter selection strategies.

To address this shrinkage problem, the Smoothly Clipped Absolute Deviation
(SCAD) penalty [15] and the Minimax Concave penalty (MCP) give a continuous,
nearly unbiased method of penalized estimation [16]. The SCAD estimator has
been shown to be support recovery consistent (i.e.,
$P(\mathcal{M}_{\lambda_{n}}=\mathcal{M}^{*})\to 1$ as $n\to\infty$) for
appropriately chosen $\lambda$ [15]. MCP shares a similar result but again
under certain conditions on $\lambda$ [16]. These penalties are less likely to
have $|\hat{\beta}_{\lambda,j}|<<|\beta^{*}_{j}|$ but their tuning parameter
selection problem is further complicated by having to explore a
multidimensional space.

Another approach to prevent $|\hat{\beta}_{\lambda,j}|<<|\beta^{*}_{j}|$ is to
adjust bridge penalties. The Ridge penalty ($q=2$) cannot shrink any
coefficient estimates to exactly zero, so [17] recently proposed
$P_{\lambda}(\boldsymbol{\beta})=\sum_{j=1}^{p}\lambda_{j}\beta_{j}^{2},$ to
allow for this behavior. The Adaptive Lasso [9] is a similar adjustment but
for the Lasso penalty. It follows a two-stage process: first
$\hat{\boldsymbol{\beta}}$—a consistent estimator for $\boldsymbol{\beta}^{*}$
such as Ordinary Least Squares (OLS)—is calculated. Then, with $\gamma>0$,
Adaptive Lasso estimates are found using the penalty
$P_{\lambda}(\boldsymbol{\beta})=\lambda\sum_{j=1}^{p}|\hat{\beta}_{j}|^{-\gamma}\lvert\beta_{j}\rvert.$
When $\gamma=1$ and $\hat{\boldsymbol{\beta}}=\hat{\boldsymbol{\beta}}_{OLS}$,
the objective function reduces to the Nonnegative Garotte [11]. This approach
is support recovery consistent when $\lambda_{n}/\sqrt{n}\rightarrow 0$ and
$\lambda_{n}n^{(\gamma-1)/2}\rightarrow\infty$ as $n\rightarrow\infty$. The
Adaptive Lasso is easily generalized to non-Lasso penalties, however the
selection of $\gamma$ and consistent estimation of $\boldsymbol{\beta}^{*}$
for high dimensional data can be difficult to establish.

The Relaxed Lasso [10] minimizes the objective function:

$\frac{1}{2n}||\textbf{y}-\textbf{X}(\boldsymbol{\beta}\circ\textbf{1}_{\mathcal{M}_{\lambda}})||_{2}^{2}+\lambda\phi||\boldsymbol{\beta}||_{1}\
.\ $ (2)

where $\phi\in(0,1]$ and
$\boldsymbol{\beta}\circ\textbf{1}_{\mathcal{M}_{\lambda}}$ is the Hadamard
product of $\boldsymbol{\beta}$ with the support vector under
$\mathcal{M}_{\lambda}$. In simulations, the Relaxed Lasso returns sparse
models with low bias. Moreover, the expected value of the loss function of the
Relaxed Lasso converges to 0 faster than the Lasso when $p$ increases quickly
relative to $n$, meaning that Relaxed Lasso estimates tend to be closer to
$\boldsymbol{\beta}^{*}$ for smaller $n$ than the traditional Lasso. This
result is again achieved assuming that $\lambda$ is sufficiently large. The
Relaxed Lasso is computationally efficient to calculate but its extensions to
more complicated penalties can be computationally intensive, and to our
knowledge has not been well-studied.

The Group Lasso [18] penalty assumes
$E(y_{i})=\sum_{j=1}^{p}\textbf{x}_{ij}^{T}\boldsymbol{\beta}_{j}$ where each
$\textbf{x}_{ij}$ is a $d_{j}\times 1$ vector corresponding with the $i$th
observation of the $j$th covariate group and shrinks coefficients at the group
level. This class of models includes general additive models (GAMs) and
functional linear models. GAMs [19] have the form
$y_{i}=\sum_{i=1}^{p}f_{j}(\textbf{x}_{ij})+\epsilon_{i}$ where the $f_{j}$’s
are functions of one or more covariates. Each $f(\cdot)$ is commonly
approximated by a pre-specified basis expansion, such as B-splines, so that
estimation of $f(\cdot)$ is equivalent to estimating the corresponding group
of basis coefficients. The functional linear model [20], an example of which
may be found in Section 6, has functional covariates $x_{ij}(t)$ on domain
$\mathcal{T}$ and models $y_{i}=\sum_{i=1}^{p}\int
x_{ij}(t)\beta_{j}(t)dt+\epsilon_{i}$. The model is often approximated by
imposing a basis expansion of the $\beta_{j}(\cdot)$. The Group Lasso penalty
for such models is
$P_{\lambda}(\boldsymbol{\beta})=\sum_{j=1}^{p}\lambda_{j}\lvert\lvert\boldsymbol{\beta}_{j}\rvert\rvert_{K_{j}}$
where
$\lVert\textbf{z}\rVert_{K}=(\textbf{z}^{T}\textbf{K}\textbf{z})^{\frac{1}{2}}$
and $\textbf{K}_{1},...,\textbf{K}_{J}$ are known positive definite matrices.
The group of coefficient estimates, $\hat{\boldsymbol{\beta}}_{\lambda,j}$,
are then either all zero or all nonzero. For the linear model, and under some
regularity conditions, [21] proved that the Group Lasso is support recovery
consistent as long as $\sqrt{n}\lambda_{j}\rightarrow\infty$ for all
$j\notin\mathcal{M}^{*}$.

###  2.2 Tuning Parameter Selection Strategies

Desirable statistical properties of penalized estimators hold under certain
$\lambda$ values, making tuning parameter selection a pivotal step in the
analysis. One popular approach is to choose $\lambda$ that minimizes an
information criterion
$IC(\lambda)=-2\log{L(\hat{\boldsymbol{\beta}}_{\lambda})}+h(k_{\lambda})$
where $L(\hat{\boldsymbol{\beta}}_{\lambda})$ is the likelihood of the data
under $\hat{\boldsymbol{\beta}}_{\lambda}$ and $h(\cdot)$ is a penalty to
prevent overselection based on the $k_{\lambda}=|\mathcal{M}_{\lambda}|$. Two
well-known criteria are AIC [3], where $h(k_{\lambda})=2k_{\lambda}$ , and BIC
[4], where $h(k_{\lambda})=k_{\lambda}\textrm{log}(n)$. AIC often overselects,
particularly for small sample sizes, so a corrected AIC (AICc), where
$h(k_{\lambda})=2k_{\lambda}+\frac{2k_{\lambda}^{2}+2k_{\lambda}}{n-k_{\lambda}-1}$
is recommended [22]. BIC, unlike AIC, is support recovery consistent when
$\epsilon_{i}\overset{iid}{\sim}N(0,\sigma^{2})$ [23]. For Gaussian errors,
$L(\hat{\boldsymbol{\beta}}_{\lambda})$ involves $\sigma^{2}$ and for unknown
$\sigma^{2}$, $-2\log{L(\hat{\boldsymbol{\beta}}_{\lambda})}\propto
n\textrm{log}\big{(}\hat{\sigma}^{2}_{\lambda}\big{)}$ where
$\hat{\sigma}^{2}_{\lambda}$ is the estimated model variance at $\lambda$.
This can lead to overselection when $\hat{\sigma}^{2}_{\lambda}<1$ [24]. When
possible, $\sigma^{2}$ is substituted with $\hat{\sigma}^{2}$ from a presumed
low bias model [6], but this may be challenging to identify for high-
dimensional data. The Extended Regularized Information Criterion (ERIC), where
$h(k_{\lambda})=2\nu
k_{\lambda}\textrm{log}(n\hat{\sigma}_{\lambda}^{2}/\lambda)$ and $\nu>0$, was
proposed by [25] specifically for tuning parameter selection of penalized
estimators. ERIC outperformed popular tuning parameter selection approaches in
their simulations for the Adaptive Lasso, but the choice of $\nu$ is
subjective and determines the balance between fit and sparsity.

Cross Validation (CV), is the process of splitting data into training and
validation sets, in which the models are fitted on the training sets and
overfitting is assessed by predicting observations in the validation sets. CV
takes many forms, but $K$-fold CV [26, 27, 28] is arguably the most common. In
$K$-fold CV, the data are partitioned into $K$ sets, or folds, of size $n_{k}$
each. For each $\lambda$, $K$ sets of estimates are generated using $K-1$ of
the $K$ folds and predictions are generated for the remaining fold, denoted
$\hat{\textbf{y}}_{\lambda,k}$. Prediction error is calculated for each
$\lambda$ and fold as
$\frac{1}{n_{k}}||\textbf{y}_{k}-\hat{\textbf{y}}_{\lambda,k}||_{2}^{2}$ and
is averaged across the $K$ folds to give the average prediction error (APE)
for each $\lambda$. The $\lambda$ with the minimum APE is selected, or a 1SE
rule—which chooses the largest $\lambda$ within one standard error of the
minimum APE—is implemented. The use of a 1SE rule is most common for a one-
dimensional $\lambda$, but [1] proposed a multidimensional extension where the
$\hat{\boldsymbol{\beta}}_{\lambda}$ chosen lies withing one standard error of
the minimum and also minimizes some penalty function. $K$-fold CV still has a
tendency towards overselection, even with a 1SE rule [29].

Generalized Cross Validation (GCV) is an efficient alternative to $n$-fold
Cross Validation [30, 31]. GCV is appropriate when the estimation procedure
admits linear predictions $\hat{\textbf{y}}=\textbf{S}\textbf{y}$ for some
matrix S [6]. For example, in Ridge Regression,
$\textbf{S}=\textbf{X}((\textbf{X}^{T}\textbf{X})^{-1}+\lambda\textbf{I})\textbf{X}^{T}$.
GCV minimizes a function that divides
$L(\hat{\boldsymbol{\beta}}_{\lambda})=||\textbf{y}-\hat{\textbf{y}}_{\lambda}||_{2}^{2}$
by a function of $k_{\lambda}$, the effective degrees of freedom. However,
just like with other information criteria, GCV has been shown to lead to
overselection [32].

###  2.3 Post-selection Inference

Post-selection inference techniques carry out further variable selection after
a tuning parameter value has been selected [33, 34, 35, 36, 37]. The Debiased
Lasso estimator [8] is a linear adjustment to the Lasso estimate
$\frac{1}{n}\boldsymbol{\Theta}\textbf{X}^{T}(\textbf{y}-\textbf{X}\hat{\boldsymbol{\beta}}_{\lambda})$,
where $\boldsymbol{\Theta}$ is an estimate of
$(\textbf{X}^{T}\textbf{X})^{-1}$. This method focuses on estimation bias
rather than direct variable selection, so the use of an additive bias
correction is coherent; estimates for zero effect coefficients are small and
hence their bias is small. The correction leads to an approximate Normal
distribution of $\hat{\boldsymbol{\beta}}_{\lambda}$ so one may perform
hypothesis testing and construct confidence intervals.

The Covariance Test [7] takes advantage of the LARS algorithm, which
constructs the Lasso solution path by adding variables one at a time [38]. It
is distinctive in that it assesses model fit using covariance rather than
squared error loss. The test requires estimation of $\sigma^{2}$ and its
extension to non-Lasso penalties is not well studied, but it provides
precedence for the use of correlation in the evaluation of model fit for
variable selection.

In general, when the tuning parameter selection event can be written as
$\\{\textbf{A}\textbf{y}\leq\textbf{b}\\}$ for some matrix A and vector b,
there exists a general scheme for post-selection inference that gives exact
confidence intervals and p-values for Gaussian errors. Choose
$\boldsymbol{\eta}$ such that inference about
$\boldsymbol{\eta}^{T}E[\textbf{y}]$ is of interest. Using the polyhedral
lemma for Gaussian errors, [36] and [7] represent this event in terms of
$\boldsymbol{\eta}^{T}\textbf{y}$ to perform conditional inference. This
allows for inference upon multiple $\lambda$ or a fixed $\lambda$. When used
for successive steps of LARS, it is known as the Spacing Test [39] and is a
non-asymptotic version of the Covariance Test.

##  3 Methodology

###  3.1 Why Correlation Over Squared Prediction Error?

Distinguish the magnitude of $\boldsymbol{\beta}^{*}$, denoted
$\alpha^{*}=||\boldsymbol{\beta}^{*}||_{2}$, from its direction,
$\boldsymbol{\xi}^{*}=\boldsymbol{\beta}^{*}/\alpha^{*}$. Then
$\boldsymbol{\xi}^{*}$ retains all information about $\mathcal{M}^{*}$. The
same summaries may be computed from an estimate, $\hat{\boldsymbol{\beta}}$,
denoted by $\tilde{\alpha}$ and $\tilde{\boldsymbol{\xi}}$, having support
$\mathcal{M}$. Comparing $\mathcal{M}$ to $\mathcal{M}^{*}$ is equivalent to
comparing the supports of $\tilde{\boldsymbol{\xi}}$ and
$\boldsymbol{\xi}^{*}$. A tuning parameter selection strategy based on squared
prediction error, however, concerns both magnitude and direction. Let
$\\{\textbf{y},\textbf{X}\\}$ denote a holdout sample where y and X have been
centered. The squared prediction error for
$y_{i}=\textbf{x}_{i}^{T}\boldsymbol{\beta}^{*}+\epsilon_{i}$ is

$\sum_{i}(y_{i}-\textbf{x}_{i}^{T}\hat{\boldsymbol{\beta}})^{2}=\sum_{i}(\epsilon_{i}+\textbf{x}_{i}^{T}(\alpha^{*}\boldsymbol{\xi}^{*}-\tilde{\alpha}\tilde{\boldsymbol{\xi}}))^{2}\
.\ $ (3)

In the ideal situation where $\tilde{\boldsymbol{\xi}}=\boldsymbol{\xi}^{*}$,
(3) will be inflated when $\tilde{\alpha}\neq\alpha^{*}$. Indeed, for
penalized estimators with large $\lambda$, typically
$\tilde{\alpha}<\alpha^{*}$. Therefore, it is possible for an estimate having
$\tilde{\boldsymbol{\xi}}=\boldsymbol{\xi}^{*}$ to have a large APE and so
would be unlikely to be chosen by an APE-based tuning parameter selection
strategy.

Consider now the correlation between y and the predictions
$\hat{\textbf{y}}=\textbf{X}\hat{\boldsymbol{\beta}}$. After some
simplification, we get the expression

$\displaystyle\text{Corr}(\textbf{y},\hat{\textbf{y}})=\frac{(\textbf{X}\boldsymbol{\xi}^{*}+\boldsymbol{\epsilon}^{*})^{T}\textbf{X}\tilde{\boldsymbol{\xi}}}{||\textbf{X}\boldsymbol{\xi}^{*}+\boldsymbol{\epsilon}^{*}||_{2}\
||\textbf{X}\tilde{\boldsymbol{\xi}}||_{2}}$ (4)

where $\boldsymbol{\epsilon}^{*}=\boldsymbol{\epsilon}/\alpha^{*}$ is a scaled
error vector that does not depend on $\hat{\boldsymbol{\beta}}$. The
$\tilde{\alpha}$ has no influence over this summary so this measure better
compares the $\boldsymbol{\xi}^{*}$ and $\tilde{\boldsymbol{\xi}}$, and hence
better assesses support recovery than squared prediction error.

Our proposed tuning parameter selection strategy, called AR2 CV, employs
$K$-fold CV with folds $\\{\textbf{y}_{k},\textbf{X}_{k}\\}$ but replaces APE
with

$\displaystyle\text{AR2}=\frac{1}{K}\sum_{k=1}^{K}[1-\text{Corr}(\textbf{y}_{k},\hat{\textbf{y}}_{k})^{2}]\
.\ $ (5)

The optimal $\lambda$ may be chosen as the one that minimizes AR2, but we have
found significant improvements in support recovery under an analogous one-
standard-error rule. Applying AR2 CV with a 1SE rule to the toy example in
Section 1, the optimal $\lambda$ is that given in Figure 1(d). The AR2 CV
estimator was then able to compromise between support recovery and prediction
error, while the APE-based CV prioritized prediction error. The coefficient
estimates for the $j\in\mathcal{M}^{*}$ under the APE CV model exhibit less
bias than those of the AR2 CV model. A potential drawback then of AR2 CV is
that its indifference towards $\alpha^{*}$ may lead to a
$\hat{\boldsymbol{\beta}}_{\lambda}$ that exhibits more shrinkage than is
desired. A potential remedy is to follow selection of $\mathcal{M}_{\lambda}$
with unpenalized estimation for only predictors in $\mathcal{M}_{\lambda}$.
For example, one could perform OLS on only the $j\in\mathcal{M}_{\lambda}$ to
form the estimator $\hat{\boldsymbol{\beta}}^{\mathcal{M}_{\lambda}}_{OLS}$
where $\hat{\beta}^{\mathcal{M}_{\lambda}}_{OLS,j}=\hat{\beta}_{OLS,j}$ when
$j\in\mathcal{M}_{\lambda}$ and zero otherwise. We next discuss an alternative
strategy that is related to AR2 CV that adjusts the shrinkage of
$\hat{\boldsymbol{\beta}}_{\lambda}$.

###  3.2 The $\alpha$-Modification

Consider now the training data’s y and their predictions, $\hat{\textbf{y}}$.
If $\hat{\textbf{y}}\neq\textbf{0}$, calculate the least-squares estimate
$\hat{\alpha}=\text{arg
min}_{\alpha}||\textbf{y}-\alpha\hat{\textbf{y}}||_{2}^{2}=(\hat{\textbf{y}}^{T}\hat{\textbf{y}})^{-1}\hat{\textbf{y}}^{T}\textbf{y}$.
Note this $\hat{\alpha}$ likely differs from
$\tilde{\alpha}=||\hat{\boldsymbol{\beta}}||_{2}$. By definition, the modified
predictions $\hat{\alpha}\hat{\textbf{y}}$ will be as close or closer to y as
$\hat{\textbf{y}}$. The modified prediction is also equivalent to prediction
under the adjusted penalized estimate, $\hat{\alpha}\hat{\boldsymbol{\beta}}$.
Calculating these $\alpha$-modified coefficient estimates and predictions is
described in Algorithm 1.

Algorithm 1 $\alpha$-Modification for the Linear Model

1:Given: Suppose y and X are centered such that an intercept term is
unnecessary. Let $\hat{\boldsymbol{\beta}}$ be the vector of coefficient
estimates.

2:Calculate
$\hat{\alpha}=(\hat{\textbf{y}}^{T}\hat{\textbf{y}})^{-1}\hat{\textbf{y}}^{T}\textbf{y}=(\hat{\boldsymbol{\beta}}^{T}\textbf{X}^{T}\textbf{X}\hat{\boldsymbol{\beta}})^{-1}\hat{\boldsymbol{\beta}}^{T}\textbf{X}^{T}\textbf{y}$.

3:Calculate $\alpha$-modified estimates,
$\hat{\alpha}\hat{\boldsymbol{\beta}}$, and $\alpha$-modified predictions,
$\hat{\alpha}\hat{\textbf{y}}=\hat{\alpha}\textbf{X}\hat{\boldsymbol{\beta}}$.

For penalized estimators, the $\alpha$-modified estimate can be viewed as

$\operatorname*{arg\,min}_{\alpha,\boldsymbol{\xi}}\frac{1}{2n}||\textbf{y}-\alpha\textbf{X}\boldsymbol{\xi}||_{2}^{2}+\lambda
P(\boldsymbol{\xi})$ (6)

by first fixing $\alpha=1$ to get
$\hat{\boldsymbol{\xi}}=\hat{\boldsymbol{\beta}}_{\lambda}$ and then
minimizing the function for $\alpha$ given $\hat{\boldsymbol{\xi}}$. This is a
slight abuse of notation because $\hat{\boldsymbol{\xi}}$ is not required to
be a unit vector like $\boldsymbol{\xi}^{*}$, but it does help to show how the
$\alpha$-modified estimate is related to separating $\boldsymbol{\beta}^{*}$
into its magnitude and direction. As $\lambda\to 0$,
$\hat{\alpha}_{\lambda}\to 1$ because the objective function focuses most of
its attention on the loss function. Thus the impact of the
$\alpha$-Modification will be more pronounced for larger values of $\lambda$,
and ideally the $\alpha$-modified estimate will correct the bias of
$\hat{\boldsymbol{\beta}}_{\lambda}$ due to shrinkage.

The $\alpha$-Modification is similar other existing modifications to penalized
estimators. First, one can view the modification as reversing the process of
calculating the Nonnegative Garrote estimator, which starts with OLS estimates
of $\boldsymbol{\beta}^{*}$ and then performs penalization. [40] also
recommended a multiplicative adjustment to the Elastic Net estimator, although
the adjustment only involves one of the tuning parameters. For penalties that
satisfy $P(\alpha\boldsymbol{\xi})=\alpha P(\boldsymbol{\xi})$ for $\alpha>0$,
we may rewrite the penalty in (6) as
$\lambda\alpha^{-1}P(\alpha\boldsymbol{\xi})$ which resembles the Relaxed
Lasso penalty, so long as $\alpha^{-1}\in(0,1]$. Finally, the Debiased Lasso
tries to reduce bias through an additive adjustment, but this will cause some
or all $\hat{\beta}_{\lambda,j}=0$ to become nonzero, while a multiplicative
adjustment will not change the support of
$\hat{\boldsymbol{\beta}}_{\lambda}$.

Penalized estimates are typically shrunk towards zero so the
$\alpha$-Modification will correct this type of bias only if
$\hat{\alpha}_{\lambda}\geq 1$. This property is guaranteed for common
penalties:

######  Theorem 1.

Suppose
$P_{\lambda}(\boldsymbol{\beta})=\sum_{\ell=1}^{L}\lambda_{\ell}g_{\ell}(\boldsymbol{\beta})$
where $g_{\ell}(\boldsymbol{\beta})$ is convex and minimized at
$\textbf{0}_{p}$. Then $\hat{\alpha}_{\lambda}\geq 1$ when
$\hat{\boldsymbol{\beta}}_{\lambda}\neq\textbf{0}$.

Amplifying penalized estimates does not necessarily decrease bias. To evaluate
the $\alpha$-Modification as a bias-reduction tool, we have the following
result.

######  Lemma 1.

If there exists a $\lambda$ where
$P(\hat{\boldsymbol{\xi}}_{\lambda}=\boldsymbol{\xi}^{*})=1$, then
$E(\hat{\alpha}_{\lambda}\hat{\boldsymbol{\beta}}_{\lambda})=\boldsymbol{\beta}^{*}$.

Lemma 1 conditions on an event that may have probability 0. The following
lemma considers a broader condition, whereby the penalized estimate recovers
the direction of $\hat{\boldsymbol{\beta}}^{\mathcal{M}_{\lambda}}_{OLS}$,
defined at the end of Section 3.1.

######  Lemma 2.

If
$\hat{\boldsymbol{\beta}}_{\lambda}\varpropto\hat{\boldsymbol{\beta}}^{\mathcal{M}_{\lambda}}_{OLS}$
then
$\hat{\alpha}_{\lambda}\hat{\boldsymbol{\beta}}_{\lambda}=\hat{\boldsymbol{\beta}}^{\mathcal{M}_{\lambda}}_{OLS}$.

There are multiple examples that satisfy the condition of Lemma 2. The OLS
estimator itself qualifies as a scaled OLS estimator, where
$\hat{\alpha}_{\lambda}=1$. Ridge estimates when
$\textbf{X}^{T}\textbf{X}=n\textbf{I}_{p}$ also take this form, having
$\hat{\boldsymbol{\beta}}_{\lambda}=\frac{1}{1+\lambda}\hat{\boldsymbol{\beta}}_{OLS}$.
Lemma 2 also applies whenever $\hat{\boldsymbol{\beta}}_{\lambda}$ contains
exactly one non-zero entry. Finally, note the Relaxed Lasso always includes
$\hat{\boldsymbol{\beta}}_{\lambda,\phi}=\hat{\boldsymbol{\beta}}^{\mathcal{M}_{\lambda}}_{OLS}$
among its solutions by setting $\phi=0$. Lemma 2 shows that this can sometimes
occur for the $\alpha$-modified estimates as well.

The $\alpha$-Modification serves to improve predictions under a given
$\hat{\boldsymbol{\beta}}_{\lambda}$ through a positive, multiplicative
adjustment and so its ability to improve estimation depends on the properties
of $\hat{\boldsymbol{\beta}}_{\lambda}$. While this paper is mainly concerned
with tuning parameter selection for support recovery under finite sample
sizes, properties of $\hat{\boldsymbol{\beta}}_{\lambda}$ are easier to study
as $n\to\infty$ and the same is true for $\alpha$-modified estimators.
Concerning support recovery consistency, since the support of
$\hat{\alpha}_{\lambda}\hat{\boldsymbol{\beta}}_{\lambda}$ equals the support
of $\hat{\boldsymbol{\beta}}_{\lambda}$, the $\alpha$-modified estimator is
support recovery consistent if and only if
$\hat{\boldsymbol{\beta}}_{\lambda}$ is support recovery consistent. Next, the
following theorem establishes estimation consistency of $\alpha$-modified
estimators.

######  Theorem 2.

Fix $p$ and suppose there exists a positive definite matrix C where
$\frac{1}{n}\textbf{X}_{n}^{T}\textbf{X}_{n}=\frac{1}{n}\textbf{C}_{n}\rightarrow\textbf{C}$
as $n\rightarrow\infty$. For a $P_{\lambda}(\cdot)$, if there exists a
$\lambda_{n}$ where $\hat{\boldsymbol{\beta}}_{\lambda_{n}}$ converges in
probability to $c\boldsymbol{\xi}^{*}$ for some $c>0$, then
$\hat{\alpha}_{\lambda_{n}}\hat{\boldsymbol{\beta}}_{\lambda_{n}}$ converges
in probability to $\boldsymbol{\beta}^{*}$.

The case of $c=\alpha^{*}$ in Theorem 2 says that if
$\hat{\boldsymbol{\beta}}_{\lambda}$ converges in probability to
$\boldsymbol{\beta}^{*}$ then so will its corresponding $\alpha$-modified
estimator. For example, [13] determined certain conditions for which the Lasso
exhibits this property. However, the rate of convergence for the
$\alpha$-modified estimators may improve due to the relaxation of finding a
$\lambda_{n}$ where $\hat{\boldsymbol{\beta}}_{\lambda_{n}}$ converges in
probability to any $c\boldsymbol{\xi}^{*}$. Theorem 2 also suggests there may
be opportunities for other penalized estimators that are not estimation
consistent to have an $\alpha$-modified estimator that is estimation
consistent.

###  3.3 $\alpha$-Modified Cross Validation

In addition to AR2 CV, we propose $\alpha$-modified CV based on average
squared prediction error under the $\alpha$-modified estimates:

$\displaystyle\text{Mod
APE}=\frac{1}{K}\sum_{k=1}^{K}||\textbf{y}_{k}-\hat{\alpha}_{k}\hat{\textbf{y}}_{k}||_{2}^{2}\
,\ $ (7)

where $\hat{\alpha}_{k}$ is calculated from the training data. Returning to
the toy example from Section 1, the $\lambda$ chosen through $\alpha$-modified
CV with a 1SE rule returned a model with
$\mathcal{M}_{\lambda}=\mathcal{M}^{*}$. Its APE was 0.3821, which is
marginally larger than that of AR2 CV. This demonstrates the possibility that
the two CV strategies may point to different optimal $\lambda$.

Another benefit of $\alpha$-modified CV over traditional CV surprisingly
derives from the potential drawbacks of the modification. It is reasonable to
say that $\hat{\boldsymbol{\beta}}_{\lambda}$ recovers the direction of
$\boldsymbol{\xi}^{*}$ when
$\hat{\boldsymbol{\beta}}_{\lambda}/\lVert\hat{\boldsymbol{\beta}}_{\lambda}\rVert_{2}=\boldsymbol{\xi}^{*}$.
Because the modification unshrinks the $\hat{\boldsymbol{\beta}}_{\lambda}$
without changing its direction, the modification may exacerbate an estimate of
poor quality, causing the Mod APE to exceed APE based on
$\hat{\boldsymbol{\beta}}_{\lambda}$. It is unlikely then for such a $\lambda$
to be selected as the optimal value. Similarly, when
$\hat{\boldsymbol{\beta}}_{\lambda}$ recovers $\boldsymbol{\xi}^{*}$ but has
small magnitude, Mod APE will decrease significantly over the traditional APE.
Theorem 3 gives a theoretical justification for the latter situation.

######  Theorem 3.

Suppose that $\hat{\boldsymbol{\beta}}_{\lambda}$ recovers the direction of
$\boldsymbol{\beta}^{*}$ and assume all $\epsilon_{i}$ are independent with
constant variance $\sigma^{2}$. Then the expected value of the
$\alpha$-modified APE is less than the expected value of the traditional APE
when:

$\frac{{\alpha^{*}}^{2}}{E[(\lVert\hat{\boldsymbol{\beta}}_{\lambda}\rVert_{2}-\alpha^{*})^{2}]}\leq\frac{\boldsymbol{\beta}^{*T}\textbf{X}^{T}\textbf{X}\boldsymbol{\beta}^{*}}{\sigma^{2}}\
.\ $ (8)

Mod APE is then expected to be smaller than the traditional APE as long as the
signal-to-noise ratio, represented by
$\boldsymbol{\beta}^{*T}\textbf{X}^{T}\textbf{X}\boldsymbol{\beta}^{*}/\sigma^{2}$,
is sufficiently large.

##  4 The $\alpha$-Modified Lasso

The methods in Section 3 generalize to many types of penalties, but to better
understand their properties we must focus on a specific penalty. Due to its
popularity, we explore the properties for the Lasso penalty and finite $n$.
First, note that the $\alpha$-modified Lasso may be thought of as a reverse
Non-negative Garotte in that it starts with shrunken estimates and uses a
least squares modification to readjust and “un-shrink” them. The use of
correlation to assess model quality is also consistent with the premise of the
Covariance Test. The $\alpha$-modified Lasso is most similar to the Relaxed
Lasso. The $\alpha$-Modification penalty is
$\lambda\alpha^{-1}||\alpha\boldsymbol{\xi}||_{1}$ and the Relaxed Lasso
penalty is $\lambda\phi||\boldsymbol{\beta}||_{1}$ for $\phi\in(0,1]$. Theorem
1 establishes $\alpha^{-1}\in(0,1]$ but we calculate the minimum $\alpha$
directly while the Relaxed Lasso treats $\phi$ as a tuning parameter. Our
approach reduces computation, but the resulting estimators are less flexible
than the Relaxed Lasso. Specifically, the $\alpha$-modified estimate cannot
change the direction of $\hat{\boldsymbol{\beta}}_{\lambda}$.

Figure 2 illustrates the distinction between the $\alpha$-modified and Relaxed
Lasso estimator for an orthogonal X with $n=100$, $p=50$ and $p^{*}=2$ where
$(\beta_{1}^{*},\beta_{2}^{*})=(-6,6)$. The $\alpha$-modified Lasso and
Relaxed Lasso solutions were generated for a set of 100 $\lambda$ and, for the
Relaxed Lasso, 20 $\phi$ values. The solution paths for
$\hat{\beta}_{\lambda,1}$ and $\hat{\beta}_{\lambda,2}$ across all values of
$\lambda$ are shown for both the traditional and $\alpha$-modified Lasso. The
Relaxed Lasso estimates for $\phi=0.5$ are also plotted. Figure 2 also
includes a plot of the $\hat{\alpha}_{\lambda}$ for the full data. For the
$\lambda$ exhibiting perfect support recovery, we see that
$\hat{\alpha}_{\lambda}>1$, thereby demonstrating the value of the
modification.

Figure 2: The leftmost plot shows solution path for the Lasso (dashed line),
$\alpha$-modified Lasso (solid line), and the Relaxed Lasso at $\phi=0.5$
(dashed and dotted line) for $\beta_{1}^{*}$ and $\beta_{2}^{*}$. The grey
shaded area indicates the values of $\lambda$ for which support recovery has
occurred. The horizontal dotted lines show the true values of $\beta_{1}^{*}$
and $\beta_{2}^{*}$. The plot on the right shows the $\hat{\alpha}_{\lambda}$
values.

We now derive properties of the $\alpha$-modified estimator for orthogonal X.
The Lasso estimator for such X is
$\hat{\beta}_{\lambda,j}=\textrm{sign}(\hat{\beta}_{OLS,j})(|\hat{\beta}_{OLS,j}|-\lambda)_{+},$
where $\hat{\beta}_{OLS,j}$ is the OLS estimate of the $\beta^{*}_{j}$ and
$(z)_{+}=\textrm{max}(0,z)$. If the Lasso estimate for a $j^{*}$ is non-zero,
the closed form expression for
$\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}$ may be derived.

######  Lemma 3.

When X is orthogonal, the $\alpha$-modified Lasso estimator has elements:

$\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}=w_{1}\hat{\beta}_{OLS,j*}+(1-w_{1})\hat{\beta}_{\lambda,j^{*}}+w_{2}\hat{\beta}_{\lambda,j^{*}}$
(9)

where $w_{1}=\frac{d_{j*}^{2}}{\sum_{j=1}^{p}d_{j}^{2}}$,
$w_{2}=\frac{\lambda\sum_{j\neq j*}d_{j}}{\sum_{j=1}^{p}d_{j}^{2}}$, and
$d_{j}=(|\hat{\beta}_{OLS,j}|-\lambda)_{+}$.

If $d_{j^{*}}=0$, then
$\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}=\hat{\beta}_{\lambda,j^{*}}=0$.
If $\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j}=0$ for all $j\neq j^{*}$ and
$d_{j^{*}}>0$ then
$\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}=\hat{\beta}_{OLS,j}$ which
is consistent with Lemma 2. When $|\mathcal{M}_{\lambda}|>1$, (9) involves a
convex combination of the OLS and Lasso estimates, as well as an additive
term, $w_{2}\hat{\beta}_{j*,\lambda}$, that may overcorrect the
$\alpha$-modified Lasso beyond the OLS estimate. This behavior is illustrated
for a simple example in Figure 3. In that example, for $\lambda<10$ there are
potential values of $\hat{\beta}_{OLS,j^{*}}$ where the $\alpha$-modified
Lasso exceeds $\hat{\beta}_{OLS,j^{*}}$. To better understand this behavior,
Theorem 4 provides an upper bound for
$|\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}|$.

######  Theorem 4.

Suppose X is orthogonal and consider a given predictor, $j^{*}$, and $\lambda$
where at least one $j\neq j^{*}$ has $|\hat{\beta}_{OLS,j}|>\lambda$. Let
$\textbf{x}_{j^{*}}$ denote column $j^{*}$ of X. Then
$\hat{\beta}_{OLS,j*}=\beta^{*}_{j^{*}}+n^{-1}\textbf{x}_{j}^{T}\boldsymbol{\epsilon}$
where $n^{-1}\textbf{x}_{j}^{T}\boldsymbol{\epsilon}$ is fixed implies

$|\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}|\leq\lambda\times\max\left(1,\frac{\sqrt{u^{2}v+v^{2}}-v}{2v}\right)$
(10)

where $u=\sum_{j\neq j^{*}}d_{j}$ and $v=\sum_{j\neq j^{*}}d_{j}^{2}$.
Moreover,
$\lim_{|\beta_{j*}^{*}|\rightarrow\infty}|\hat{\alpha}_{\lambda}\hat{\beta}_{j*,\lambda}-\hat{\beta}_{OLS,j*}|\rightarrow
0$ .

Figure 3: $\alpha$-modified estimates of $\alpha\beta_{1}$ as
$\hat{\beta}_{OLS,1}$ changes, all other parameters being fixed.
$\hat{\beta}_{OLS,j}=[-8,5,-3,1]$ for $j=[2,...,5]$. The dotted line gives the
Lasso estimates and the solid blue line gives the $\alpha$-modified estimates.

The result of Theorem 4 is demonstrated in Figure 3 for different values of
$\lambda$. As $|\beta_{1}^{*}|$ increases in magnitude, thereby increasing
$\hat{\beta}_{OLS,1}$, the $\alpha$-modified Lasso estimate moves away from
the traditional Lasso estimate and towards the $\hat{\beta}_{OLS,1}$. This
happens instantaneously for $\lambda=10$ and $|\hat{\beta}_{OLS,1}|>10$
because $\hat{\beta}_{OLS,1}$ is the only estimate that exceeds $\lambda$. For
$\lambda=6$ and $4$, when $(\sqrt{u^{2}v+v^{2}}-v)/2v>1$ there exists some
$\beta^{*}_{1}$ in which the $\alpha$-modified Lasso estimate overshoots
$\hat{\beta}_{OLS,1}$. This occurs as $u^{2}/v$ increases, meaning multiple
$d_{j}$ are nonzero and are close to $0$.

To generalize Theorem 4 to an arbitrary X we condition on the event that the
$\lambda$ recovers the sign vector of $\boldsymbol{\beta}^{*}$ and consider
the oracle OLS estimator
$\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}^{*}}=(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})^{-1}\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{y}$
where $\textbf{X}_{\mathcal{M}^{*}}$ is the subset of columns of X
corresponding to $\mathcal{M}^{*}$. Then
$\hat{\boldsymbol{\beta}}_{OLS,j^{*}}^{\mathcal{M}^{*}}=\beta^{*}_{j^{*}}+(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})_{j^{*}}^{-1}\textbf{X}_{\mathcal{M}^{*}}^{T}\boldsymbol{\epsilon}$
where
$(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})_{j^{*}}^{-1}$
denotes the $j$-th row of
$(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})^{-1}$.
Finally, let
$\tilde{s}_{j^{*}}=(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})_{j^{*}}^{-1}\textbf{s}$
where s is the $|\mathcal{M}^{*}|\times 1$ vector of signs for the
$\beta_{j}^{*}$ in $\mathcal{M}^{*}$.

######  Theorem 5.

Suppose the Lasso estimate recovers the correct sign vector of
$\boldsymbol{\beta}^{*}$. Let
$G_{j^{*}}=\lambda(s_{j^{*}}-n\tilde{s}_{j^{*}})$. Then for a fixed
$\boldsymbol{\epsilon}$,
$\lim_{\lvert\beta^{*}_{j^{*}}\rvert\rightarrow\infty}\lvert\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}\rvert=|G_{j^{*}}|$.
Moreover $|G_{j^{*}}|<|\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}|$
if and only if

$\displaystyle\left|1-\frac{s_{j^{*}}}{n\tilde{s}_{j^{*}}}\right|<1\ .\ $ (11)

Note for an orthogonal design $n\tilde{s}_{j^{*}}=s_{j^{*}}$, making
$G_{j^{*}}=0$. Theorem 5 shows that the absolute difference between the
$\alpha$-modified Lasso estimate and the OLS estimate generally does not
approach 0 as $|\beta_{j^{*}}^{*}|\to\infty$. Rather, it approaches a constant
that is always smaller than
$\lvert\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}\rvert$ so long as
$\tilde{s}_{j^{*}}$ is not close to 0. This indicates the $\alpha$-modified
Lasso estimate generally improves the bias of $\hat{\beta}_{\lambda,j^{*}}$
for large $\beta^{*}_{j^{*}}$ when the sign is recovered.

##  5 Numerical Results

###  5.1 $\alpha$-Modified CV Under Poor Bias Adjustment

Theorems 4 and 5 show that the $\alpha$-Modification cannot guarantee bias
reduction for all $\lambda$. However, a poor bias adjustment may be detected
by CV. To illustrate, we performed a simulation study for the X used in Figure
1 and considered two $\boldsymbol{\beta}^{*}$ vectors, displayed in Figure 4.
For each $\boldsymbol{\beta}^{*}$, we generated 500 $\boldsymbol{\epsilon}\sim
N(\textbf{0},\textbf{I})$ and averaged the three 10-fold CV metrics (APE, AR2,
and Mod APE) for a range of $\lambda$. The results are presented in Figure 4.
The grey shaded area indicates perfect support recovery, where values of
$\lambda$ to the left of the grey area overselect predictors (false positives)
and values of $\lambda$ to the right of the grey area underselect predictors
(false negatives).

As shown in Figure 4(a), the Mod APE was less than or equal to APE for all
$\lambda$, and especially so for larger $\lambda$ that recover
$\mathcal{M}^{*}$. The optimal $\lambda$’s according to the minimum AR2 and
Mod APE resulted in sparser models than the minimum APE. With a 1SE rule,
traditional CV recovered the support in only 24.4% of the replications,
whereas $\alpha$-modified and AR2 CV did so in 91.6% and 93.6% of the
replications, respectively.

Figure 4(b) corresponds to $\boldsymbol{\beta}^{*}$ with larger and equal
magnitude coefficients. Generally, the Mod APE was equal to or smaller than
regular APE except for $\log(\lambda)\approx 4$, which had only one nonzero
coefficient estimate. Mod APE highlights the poor bias adjustment and so would
not recommend choosing this $\lambda$. The support recovery percentages were
19.8% for APE CV, 98.2% for $\alpha$-modified CV, and 98.8% for AR2 CV.
Surprisingly, the support recovery percentage decreased for APE CV despite
increasing magnitude of $\boldsymbol{\beta}^{*}$.

(a) $\boldsymbol{\beta}^{*}=(4.3,2.9,-5.8,-3.3,-6.1,0,...,0)^{T}$.

(b) $\boldsymbol{\beta}^{*}=(50,50,50,50,50,0,...,0)^{T}$.

Figure 4: Average 10-fold Cross Validation results across 500 replications of
linear data. X was taken from the example in Section 1. Average values of
$\lambda$ where the correct submodel was selected are represented by the grey
box, the dashed line gives the APE, the solid line gives $\alpha$-modified
APE, and the dotted line gives Average $R^{2}$. Standard errors from these
simulations were too small to be depicted.

###  5.2 Support Recovery Simulation Study

To evaluate the new methods for the Lasso, a broader simulation study
consisting of 100 replications of data from model (1) was conducted. Following
[10], the rows of X were drawn from a multivariate Normal distribution with
mean 0 and covariance $\boldsymbol{\Sigma}_{X}$. Nonzero elements of
$\boldsymbol{\beta}^{*}$ were drawn from a Gamma distribution with shape 10
and scale 0.25 with negative and positive signs randomly assigned with equal
probability, updating the coefficient vector with every replication. For each
X and $\boldsymbol{\beta}^{*}$, two independent error vectors were generated
from $N(0,\sigma^{2})$ distributions, with $\sigma$ determined by a fixed
signal-to-noise ratio
$\text{SNR}=\boldsymbol{\beta}^{*T}\boldsymbol{\Sigma}_{X}\boldsymbol{\beta}^{*}/\sigma^{2}$.
We considered $n\in\\{100,500,1000\\}$, $p\in\\{50,100,200,400,800\\}$,
$SNR\in\\{1.25,5\\}$ and $p^{*}=\\{5,6,\dots,50\\}$.

Each simulated data set was analyzed using 10-fold CV with and without a 1SE
rule under traditional APE (CV Min, CV 1SE), AR2 (AR2 Min, AR2 1SE), Mod APE
(Mod Min, Mod 1SE), and the Relaxed Lasso. For all methods, we considered 250
$\lambda$ values equally spaced on the exponential scale from $e^{-20}$ to
$e^{10}$. For the Relaxed Lasso, we also considered 100 $\phi$ values equally
spaced from $e^{-10}$ to 1. Selection under the Releaxed Lasso was done using
minimum APE as well as a 1SE rule based on [1], wherein the optimal
combination of $\lambda$ and $\phi$ is the model with the smallest
$||\hat{\boldsymbol{\beta}}_{\lambda,\phi}||_{1}$ among all models with an APE
within one standard error of the minimum. Relaxed Lasso estimates were found
using a coordinate descent algorithm, capable of admitting more than $n-1$
covariates into models, unlike LARS which was used by [10]. Support recovery
was evaluated using the Hamming Distance (HD) between the $0/1$ support
vectors of $\boldsymbol{\beta}^{*}$ and $\hat{\boldsymbol{\beta}}_{\lambda}$,
which is the sum of the number of false negatives and false positives.
Additional metrics including the false discovery rate (FDR), average number of
false positives and negatives, and the average prediction bias, where
prediction bias is
$||\textbf{X}\boldsymbol{\beta}^{*}-\textbf{X}\hat{\boldsymbol{\beta}}_{\lambda}||_{2}$
or
$||\textbf{X}\boldsymbol{\beta}^{*}-\textbf{X}\hat{\alpha}_{\lambda}\hat{\boldsymbol{\beta}}_{\lambda}||_{2}$
for the $\alpha$-Modification, can be found in the Supplementary Materials.

We first consider the case of $\boldsymbol{\Sigma}_{X}=\boldsymbol{I}_{p}$.
Tables 1, 2, and 3 give the average HD between $\boldsymbol{\beta}^{*}$ and
$\hat{\boldsymbol{\beta}}_{\lambda}$ for the traditional APE method with a 1SE
Rule, AR2 with a 1SE rule, and the Relaxed Lasso with a minimum APE approach.
AR2 and Mod APE perform fairly similarly; the latter results may be found in
the supplementary materials. In general, AR2 CV has better variable selection
than APE CV and, for small $n$ and large $p^{*}$, the Relaxed Lasso.

Table 1: Average Hamming Distance for 100 replications of CV 1SE. Standard errors given in parentheses. | $p$ |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
$n$ |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.19$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.59$
100 |  |  3.1 (0.3) |  4.3 (0.4) |  6 (0.6) |  7.8 (1.1) |  9.1 (1.2) |  |  2.2 (0.2) |  4 (0.4) |  7.5 (1.2) |  9.3 (1) |  10.7 (1.2)
500 |  |  0.6 (0.1) |  1 (0.2) |  1.3 (0.2) |  1.9 (0.5) |  2.9 (0.5) |  |  0.9 (0.2) |  0.9 (0.1) |  1.7 (0.3) |  1.7 (0.3) |  1.5 (0.3)
1000 |  |  0.2 (0) |  0.4 (0.1) |  0.4 (0.1) |  0.6 (0.1) |  0.9 (0.2) |  |  0.1 (0) |  0.4 (0.1) |  0.5 (0.1) |  0.7 (0.1) |  0.9 (0.2)
$n$ |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.6$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.3$
100 |  |  31.7 (1.3) |  43.5 (0.6) |  50.1 (0.5) |  51.1 (0.4) |  53.2 (1.1) |  |  5 (0.4) |  32 (0.6) |  47 (0.7) |  53.8 (1.1) |  54.5 (1.1)
500 |  |  1.6 (0.2) |  18.2 (0.5) |  33.3 (0.8) |  48.6 (1.3) |  57.4 (1.8) |  |  0 (0) |  16.8 (0.5) |  35.3 (0.9) |  58.4 (1.6) |  81.2 (2)
1000 |  |  0.4 (0.1) |  14.5 (0.4) |  28.5 (0.9) |  43.5 (1.4) |  57.6 (1.6) |  |  0 (0) |  14.2 (0.5) |  28 (0.8) |  44.8 (1.3) |  64.2 (1.8)
Table 2: Average Hamming Distance for 100 replications of AR2 CV 1SE. Standard errors given in parentheses. | $p$ |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
$n$ |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.19$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.59$
100 |  |  2.1 (0.3) |  2.8 (0.3) |  3.3 (0.3) |  4.9 (0.7) |  5 (0.5) |  |  0.6 (0.1) |  0.6 (0.1) |  1.9 (0.8) |  1.8 (0.4) |  2.6 (0.4)
500 |  |  0.1 (0) |  0.2 (0) |  0.1 (0) |  0.2 (0) |  0.4 (0.1) |  |  0 (0) |  0 (0) |  0 (0) |  0 (0) |  0 (0)
1000 |  |  0 (0) |  0 (0) |  0 (0) |  0 (0) |  0 (0) |  |  0 (0) |  0 (0) |  0 (0) |  0 (0) |  0 (0)
$n$ |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.6$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.3$
100 |  |  23.6 (1.3) |  40.4 (0.5) |  51.4 (0.8) |  55.1 (1.2) |  54.8 (1.2) |  |  4.2 (0.4) |  31.9 (0.5) |  47.9 (0.8) |  54.6 (1.3) |  53.5 (0.9)
500 |  |  1.8 (0.2) |  17.1 (0.5) |  28.7 (0.8) |  42.3 (1.3) |  51.9 (2.2) |  |  0.2 (0) |  9.1 (0.4) |  17.8 (0.7) |  30 (1) |  46.6 (1.3)
1000 |  |  0.7 (0.1) |  9.1 (0.4) |  15.4 (0.6) |  23.1 (0.9) |  32.5 (1.1) |  |  0 (0) |  3.9 (0.3) |  6.3 (0.4) |  11.1 (0.6) |  16.9 (0.8)

Table 3: Average Hamming Distance for 100 replications of Relaxed Lasso. Standard errors given in parentheses. | $p$ |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
$n$ |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.19$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.59$
100 |  |  3.7 (0.4) |  5.4 (0.8) |  7.4 (1.1) |  18.4 (3.4) |  28.1 (4.2) |  |  1.1 (0.3) |  1.8 (0.5) |  4.5 (1.7) |  6.6 (2.3) |  7.9 (2.6)
500 |  |  0.8 (0.3) |  1.2 (0.3) |  0.5 (0.2) |  0.7 (0.4) |  1.5 (0.6) |  |  1.2 (0.3) |  0.8 (0.2) |  0.8 (0.4) |  0.3 (0.1) |  0.1 (0)
1000 |  |  0.6 (0.2) |  0.5 (0.2) |  0.7 (0.3) |  0.4 (0.1) |  1.9 (1.5) |  |  0.6 (0.2) |  0.8 (0.3) |  0.4 (0.2) |  0.4 (0.2) |  0.4 (0.2)
$n$ |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.6$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.3$
100 |  |  19.5 (1.3) |  39.9 (0.5) |  57.6 (1.7) |  70.8 (3.1) |  67.9 (3.2) |  |  2.1 (0.3) |  34.1 (0.5) |  75.9 (2.5) |  87.6 (3.3) |  93.6 (3.8)
500 |  |  0.9 (0.1) |  22.9 (0.9) |  39.6 (1.8) |  66.2 (3.4) |  74.5 (4.2) |  |  0.1 (0) |  7.2 (0.5) |  14.8 (1.2) |  25.4 (2.3) |  36.8 (2.7)
1000 |  |  0.2 (0) |  12.3 (0.9) |  17.5 (1.4) |  27.9 (2.7) |  32.7 (2.9) |  |  0 (0) |  2.4 (0.3) |  3.1 (0.3) |  4.9 (0.5) |  5.5 (0.3)

Results from increasing values of $p^{*}$ were considered for
$n=\\{100,500,1000\\}$ and $p=100$ in Figure 5. Additional methods and results
may be found in the Supplementary Materials. For $n=100$ all methods perform
comparably, except for the Relaxed Lasso with a minimum APE rule, which tends
to have a higher number of false positives than the other methods, leading to
larger average HD. As $n$ increases, Mod APE and AR2 CV appear to perform
comparably to the Relaxed Lasso with a 1SE Rule, and have a consistently
smaller average HD than traditional CV with a 1SE Rule. The new methods of CV
consistently perform similarly to Relaxed 1SE and out performs the CV 1SE in
terms of support recovery.

Figure 5: Average Hamming Distance between $\boldsymbol{\beta}^{*}$ and
$\hat{\boldsymbol{\beta}}_{\lambda}$ from 100 replications of simulated data
with $n=\\{100,500,1000\\}$ and $p=100$ with independent predictors for both
SNR $=1.25$ and SNR $=5$. Thin dotted lines represent the mean $\pm$ one
standard error.

We next considered correlated predictors where $\boldsymbol{\Sigma}_{X}$
satisfied $\Sigma_{ii}=1$ and $\Sigma_{ij}=0.75$ whenever $i\neq j$. Figure 6
gives the average HD for 100 replications with $p=100$ and
$n=\\{100,500,1000\\}$, but further results can be found in the Supplementary
Materials. Our two proposed CV methods are highly competitive with the Relaxed
1SE, whereas Relaxed Min has a tendency to overselect as $p^{*}$ increases,
particularly for $n=100$. Similarly to the results from independent
predictors, it is clear that as $n$ increases, the average Hamming distance
for the Relaxed Lasso and the two new methods of CV decreases for all
considered $p^{*}$. Relaxed Lasso with a minimum APE rule struggles when $n$
is small, even for a high signal-to-noise ratio, primarily due to
overselection. Once again the new methods of CV generally have a smaller
average HD than CV with a 1SE Rule and are similar in support recovery to the
Relaxed Lasso with a 1SE Rule.

Figure 6: Average Hamming Distance between $\boldsymbol{\beta}^{*}$ and
$\hat{\boldsymbol{\beta}}_{\lambda}$ from 100 replications of simulated data
with $n=\\{100,500,1000\\}$ and $p=100$ for correlated predictors. Thin dotted
lines represent the mean $\pm$ one standard error.

Unlike the Relaxed Lasso, the $\alpha$-modification is straightforward to
apply to other penalties. The $\alpha$-modified Lasso was compared with the
non-convex penalties SCAD and MCP, both traditionally and with an
$\alpha$-modification. As expected, the $\alpha$-modification had a minimal
effect on the predictive models. MCP and SCAD estimates were prone to
underselection whereas the $\alpha$-modified Lasso estimates were more prone
to overselection. Details may be found in the Supplementary Materials.

##  6 Optimal EMG Placement for a Robotic Prosthesis Controller

For the EMG application introduced in Section 1, we want to identify as few
EMG sensors as needed to reliably predict hand movement. The data were
collected from an able-bodied subject and consist of concurrent measurements
of the subject’s finger position and 16 EMG signals as predictors. A full
description of the data, some of its challenges, and SAFE can be found in [1].
Six data sets were analyzed, corresponding to three consistent finger movement
patterns (FC1, FC2, FC3) and three random patterns (FR1, FR2, FR3). As data
were collected from an able-bodied subject, it was known that three of the 16
sensors, denoted $X_{5},X_{7},X_{12}$, targeted muscles known to fully explain
finger movement. Sensors $X_{5}$ and $X_{7}$ collected information from the
same muscle, however, and so only one of the pair is necessary to predict
finger position. An ideal model would thus include $X_{12}$ and either $X_{5}$
or $X_{7}$, but recovery of all three sensors is also acceptable.

Due to known biomechanical features of hand movement, [1] use finger velocity
as the response and treat the recent past EMG signals as functional
covariates. The model is

$y_{i}=\sum_{j=1}^{16}\int_{-\delta}^{0}X_{ij}(t)\gamma_{j}(t,z_{i})dt+\epsilon_{i}\
,\ $ (12)

where $y_{i}$ is the velocity, $X_{ij}(t)$ represents past EMG signals,
$t\in[-\delta,0]$ is the recent past time, and $z_{i}$ is the recent finger
position. Although this model is not the linear model introduced in (1), its
approximation via basis expansion allows it to be treated similarly to the
linear model. Following [41], [1] proposed a penalized estimation procedure
where the penalty accounted for both sparsity and smoothness of the
$\gamma_{j}(\cdot,\cdot)$:

$P_{\lambda}(\gamma_{j})=\lambda(f_{j}||\gamma_{j}||_{2}^{2}+g_{j}\lambda_{t}||\gamma_{j,t}^{\prime\prime}||_{2}^{2}+h_{j}\lambda_{z}||\gamma_{j,z}^{\prime\prime}||_{2}^{2})^{1/2}$
(13)

where $||\gamma_{j}||_{2}^{2}=\int\int\gamma_{j}(t,z_{i})^{2}dtdz$ and
$\gamma_{j,t}^{\prime\prime}=\partial^{2}\gamma_{j}(t,z_{i})/\partial t^{2}$.
There are three tuning parameters, $\lambda,\lambda_{t},$ and $\lambda_{z}$,
and adaptive weights $f_{k}$, $g_{k}$, and $h_{k}$. To facilitate estimation,
the $\gamma_{j}(\cdot,\cdot)$ were written using a tensor product basis
expansion, leading to a Group Lasso-type penalty; more details may be found in
[1] and the supplementary materials.

To perform variable selection, [1] proposed Sequential Adaptive Functional
Estimation (SAFE) that performs penalized estimation in stages. The first
stage set $f_{j}=g_{j}=h_{j}=1$ and chose optimal tuning parameters based on
an APE 1SE rule following 10-fold CV. Let $\mathcal{M}_{\lambda,1}$ denote the
support of this estimator. Adaptive weights were updated based on the
estimates of the nonzero effects and penalized estimation was performed again
using these weights and only those $j\in\mathcal{M}_{\lambda,1}$. The process
was repeated for up to 5 stages. While effective in identifying the correct
submodel, the analysis can be very time consuming. We modified their method
based on AR2 CV and modified APE for this application in hopes to reduce the
number of stages required to perform variable selection.

Table 4 gives the variable selection results for the three CV methods based on
the initial stage; results from subsequent stages of SAFE can be found in the
supplementary materials. AR2 CV and Mod APE generally give smaller models than
traditional APE CV, with one exception: the Average $R^{2}$ method has a large
model size for FC3. On average, however, both new methods have fewer false
positives and smaller model sizes. Mod APE gives very similar results to APE,
suggesting that the $\alpha$-modified CV approach requires further study for
the Group Lasso. Although AR2 CV is not perfect in this application, in
general it reduces model size and decreases false positives at no additional
computational cost.

Table 4: Variable selection results for EMG finger movements without adaptive weighting. FP indicates the total number of false positives in the model, and Size is the total number of EMG signals contained in the model. |  | FC1 | FC2 | FC3 | FR1 | FR2 | FR3 | Mean
---|---|---|---|---|---|---|---|---
APE | FP | 2 | 1 | 2 | 2 | 1 | 1 | 1.5
Size | 4 | 3 | 4 | 5 | 4 | 4 | 4
AR2 | FP | 0 | 1 | 3 | 0 | 1 | 1 | 1
Size | 2 | 3 | 6 | 2 | 4 | 3 | 3.33
Mod | FP | 1 | 1 | 2 | 2 | 1 | 1 | 1.33
Size | 3 | 3 | 4 | 5 | 4 | 4 | 3.83

##  7 Discussion

In this paper, we proposed AR2 CV to choose tuning parameters to balance
support recovery and prediction performance. This led to the
$\alpha$-Modification, a multiplicative adjustment to predictions from
penalized estimates which can also be used for $\alpha$-modified CV. The
$\alpha$-modification is simple and efficient, making it an attractive option
when less flexible approaches are unavailable. A simulation study on the
capabilities of AR2 and $\alpha$-modified CV found that their variable
selection results were highly competitive with—or, in some cases, better
than—the Relaxed Lasso. In order to ensure fair comparison, we introduced a
1SE Rule for the Relaxed Lasso. Finally, we applied the approaches to a
functional data problem in a demonstration of their flexibility.

The $\alpha$-modification and the tuning parameter selection methods proposed
here inspire several research questions. First, further theoretical analysis
of the methods is of interest. Because the two new methods of tuning parameter
selection are CV-based approaches, the theoretical properties of CV are
central. Theoretical justifications for the use of CV for penalized estimators
are relatively new and still evolving. [42] may be extended to show that
$\alpha$-modified and AR2 CV lead to estimates that are low bias and
appropriately sparse. A second area of future research is the theoretical
properties of $\alpha$-modified estimates themselves. In Theorem 2, it was
shown that any consistent estimator will still be consistent with the
$\alpha$-Modification. We posit that the rates of convergence for
$\alpha$-modified estimators are faster than unmodified, but proof of this
conjecture is the work of future research. Similarly, many of the results from
this paper assume finite sample sizes. Further study is needed to determine
more of the asymptotic properties of $\alpha$-modified estimators and to adapt
the specific results given for the Lasso penalty to other penalties.

There are also a few extensions and adaptations of the $\alpha$-Modification
that may prove fruitful. We are currently expanding the $\alpha$-modification
to Generalized Linear Models (GLMs). This extension will require an iterative
algorithm to find estimates of $\alpha$ because closed form solutions do not
exist and an accommodation for the inclusion of an intercept term will be
necessary. Additionally, as noted in some of the theoretical results in this
paper, the $\alpha$-modification does not always reduce bias. We are
interested in exploring a further penalty on $\alpha$ itself to ensure bias
reduction. Finally, the calculation of $\hat{\alpha}_{\lambda}$ described here
uses in-sample predictions and observations. As overspecification is a
particular concern, the question of whether out-of-sample data can be used to
find estimates of $\alpha$ is another subject of further research.

## Acknowledgements

This work was partially supported by the National Science Foundation, grant
IOS–2039226.

## Supplementary Materials

The supplementary material in the following document includes three sections.
Section A gives more details of the proofs for the theorems and lemmas
presented in the paper. Section B provides tables and figures that more fully
explore the simulation studies discussed in the Numerical Results. This
section also evaluates results from a simulation study of the non-convex SCAD
and MC+ penalties. Finally, Section C describes the approximations used to
achieve the estimates from the EMG data application and includes results from
multiple stages of adaptive estimation.

## A. Detailed Proofs for Theoretical Results

Proof of Theorem 1. Suppose there is at least one element of
$\hat{\boldsymbol{\beta}}_{\lambda}$ not equal to 0. Let
$\lambda\textbf{g}(\boldsymbol{\beta})=\sum_{\ell=1}^{L}\lambda_{\ell}g_{\ell}(\boldsymbol{\beta})$
and $\nabla\textbf{g}^{*}$ be the subgradient vector of
$\textbf{g}(\boldsymbol{\beta})$. Then

$-\frac{1}{n}\textbf{X}^{T}(\textbf{y}-\textbf{X}\hat{\boldsymbol{\beta}}_{\lambda})+\nabla\textbf{g}^{*}=0$
(14)

due to the KKT conditions. Further, we can say that there exists some
generalized inverse of $\textbf{X}^{T}\textbf{X}$,
$(\textbf{X}^{T}\textbf{X})^{-}$, such that
$\hat{\boldsymbol{\beta}}_{\lambda}=(\textbf{X}^{T}\textbf{X})^{-}(\textbf{X}^{T}\textbf{y}-n\nabla\textbf{g}^{*})$.
Therefore:

$\hat{\textbf{y}}_{\lambda}^{T}\textbf{y}=\hat{\textbf{y}}_{\lambda}^{T}\hat{\textbf{y}}_{\lambda}+n\lambda\nabla\textbf{g}^{*T}(\textbf{X}^{T}\textbf{X})^{-}\textbf{X}^{T}\textbf{y}-n^{2}\nabla\textbf{g}^{*T}(\textbf{X}^{T}\textbf{X})^{-}\nabla\textbf{g}^{*}\
.\ $ (15)

Given this definition, $\hat{\alpha}_{\lambda}$ may be written:

$\displaystyle\hat{\alpha}_{\lambda}$
$\displaystyle=\frac{\hat{\textbf{y}}_{\lambda}^{T}\hat{\textbf{y}}_{\lambda}+n\nabla\textbf{g}^{*T}(\textbf{X}^{T}\textbf{X})^{-}(\textbf{X}^{T}\textbf{y}-n\nabla\textbf{g}^{*})}{\hat{\textbf{y}}_{\lambda}^{T}\hat{\textbf{y}}_{\lambda}}$
(16)
$\displaystyle=\frac{\hat{\textbf{y}}_{\lambda}^{T}\hat{\textbf{y}}_{\lambda}+n\nabla\textbf{g}^{*T}\hat{\boldsymbol{\beta}}_{\lambda}}{\hat{\textbf{y}}_{\lambda}^{T}\hat{\textbf{y}}_{\lambda}}$
(17)
$\displaystyle=1+\frac{n\nabla\textbf{g}^{*T}\hat{\boldsymbol{\beta}}_{\lambda}}{\hat{\textbf{y}}_{\lambda}^{T}\hat{\textbf{y}}_{\lambda}}.$
(18)

Because the denominator of the second term is greater than 0, as is $n$, this
expression is greater than or equal to 1 whenever
$\nabla\textbf{g}^{*T}\hat{\boldsymbol{\beta}}_{\lambda}$ is greater than or
equal to zero. The vector $\nabla\textbf{g}^{*}$ the subgradient of a convex
function at $\hat{\boldsymbol{\beta}}_{\lambda}$, therefore for any
$\boldsymbol{\beta}$:

$\displaystyle g(\boldsymbol{\beta})$ $\displaystyle\geq
g(\hat{\boldsymbol{\beta}}_{\lambda})+\nabla\textbf{g}^{*T}(\boldsymbol{\beta}-\hat{\boldsymbol{\beta}}_{\lambda})$
(19) $\displaystyle\nabla\textbf{g}^{*T}\hat{\boldsymbol{\beta}}_{\lambda}$
$\displaystyle\geq
g(\hat{\boldsymbol{\beta}}_{\lambda})-g(\boldsymbol{\beta})+\nabla\textbf{g}^{*T}\boldsymbol{\beta}.$
(20)

Because this is true for all $\boldsymbol{\beta}$ vectors, it must be the case
when $\boldsymbol{\beta}=\textbf{0}$. Hence, we have:

$\displaystyle\nabla\textbf{g}^{*T}\hat{\boldsymbol{\beta}}_{\lambda}$
$\displaystyle\geq g(\hat{\boldsymbol{\beta}}_{\lambda})-g(\textbf{0})\geq 0.$
(21)

Thus $\hat{\alpha}_{\lambda}\geq 1.$ ∎

Proof of Lemma 1. When
$\lVert\hat{\boldsymbol{\beta}}_{\lambda}\rVert_{2}^{-1}\hat{\boldsymbol{\beta}}_{\lambda}=\boldsymbol{\xi}^{*}$,
$\hat{\boldsymbol{\beta}}_{\lambda}=\lVert\hat{\boldsymbol{\beta}}_{\lambda}\rVert_{2}\boldsymbol{\xi}^{*}.$
Write
$\textbf{y}=\alpha^{*}\textbf{X}\boldsymbol{\xi}^{*}+\boldsymbol{\epsilon}$,
to define:

$\hat{\alpha}_{\lambda}=\frac{\alpha^{*}\boldsymbol{\xi}^{*T}\textbf{X}^{T}\textbf{X}\boldsymbol{\xi}^{*}+\boldsymbol{\xi}^{*}\textbf{X}^{T}\boldsymbol{\epsilon}}{\lVert\hat{\boldsymbol{\beta}}_{\lambda}\rVert_{2}\boldsymbol{\xi}^{*T}\textbf{X}^{T}\textbf{X}\boldsymbol{\xi}^{*}}\
.\ $ (22)

Therefore:

$\displaystyle\hat{\alpha}_{\lambda}\hat{\boldsymbol{\beta}}_{\lambda}$
$\displaystyle=\hat{\alpha}_{\lambda}\lVert\hat{\boldsymbol{\beta}}_{\lambda}\rVert_{2}\boldsymbol{\xi}^{*}$
(23)
$\displaystyle=\frac{1}{\lVert\hat{\boldsymbol{\beta}}_{\lambda}\rVert_{2}}\Big{(}\alpha^{*}+\frac{\boldsymbol{\xi}^{*T}\textbf{X}^{T}\boldsymbol{\epsilon}}{\boldsymbol{\xi}^{*T}\textbf{X}^{T}\textbf{X}\boldsymbol{\xi}^{*}}\Big{)}\lVert\hat{\boldsymbol{\beta}}_{\lambda}\rVert_{2}\boldsymbol{\xi}^{*}$
(24)
$\displaystyle=\alpha^{*}\boldsymbol{\xi}^{*}+\Big{(}\frac{\boldsymbol{\xi}^{*T}\textbf{X}^{T}\boldsymbol{\epsilon}}{\boldsymbol{\xi}^{*T}\textbf{X}^{T}\textbf{X}\boldsymbol{\xi}^{*}}\Big{)}\boldsymbol{\xi}^{*}\
.\ $ (25)

Thus
$E[\hat{\alpha}_{\lambda}\hat{\boldsymbol{\beta}}_{\lambda}]=\alpha^{*}\boldsymbol{\xi}^{*}=\boldsymbol{\beta}^{*}$
and the estimator is unbiased. ∎

Proof of Lemma 2. Fix $\lambda$ and let
$\hat{\boldsymbol{\beta}}_{\lambda}=a\hat{\boldsymbol{\beta}}^{\mathcal{M}_{\lambda}}_{OLS}$
where $\hat{\boldsymbol{\beta}}^{\mathcal{M}_{\lambda}}_{OLS}$ is the OLS
estimate for submodel $\mathcal{M}_{\lambda}$. Let
$\textbf{X}_{\mathcal{M}_{\lambda}}$ be the submatrix of X containing only the
columns indexed by $\mathcal{M}_{\lambda}$. Then
$\hat{\boldsymbol{\beta}}^{\mathcal{M}_{\lambda}}_{OLS}=(\textbf{X}_{\mathcal{M}_{\lambda}}^{T}\textbf{X}_{\mathcal{M}_{\lambda}})^{-1}\textbf{X}_{\mathcal{M}_{\lambda}}^{T}\textbf{y}$.
Now:

$\hat{\alpha}_{\lambda}=\frac{a\textbf{y}^{T}\textbf{X}_{\mathcal{M}_{\lambda}}(\textbf{X}_{\mathcal{M}_{\lambda}}^{T}\textbf{X}_{\mathcal{M}_{\lambda}})^{-1}\textbf{X}_{\mathcal{M}_{\lambda}}^{T}\textbf{y}}{a^{2}\textbf{y}^{T}\textbf{X}_{\mathcal{M}_{\lambda}}(\textbf{X}_{\mathcal{M}_{\lambda}}^{T}\textbf{X}_{\mathcal{M}_{\lambda}})^{-1}\textbf{X}_{\mathcal{M}_{\lambda}}^{T}\textbf{X}_{\mathcal{M}_{\lambda}}(\textbf{X}_{\mathcal{M}_{\lambda}}^{T}\textbf{X}_{\mathcal{M}_{\lambda}})^{-1}\textbf{X}_{\mathcal{M}_{\lambda}}^{T}\textbf{y}}=\frac{1}{a}\
.\ $ (26)

Therefore:

$\hat{\alpha}_{\lambda}\hat{\boldsymbol{\beta}}_{\lambda}=\frac{1}{a}\times
a\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}_{\lambda}}=\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}_{\lambda}}\
.\ $ (27)

∎

Proof of Theorem 2. Consider $\hat{\alpha}_{\lambda_{n}}:$

$\hat{\alpha}_{\lambda_{n}}=\frac{\frac{1}{n}\hat{\boldsymbol{\beta}}_{\lambda_{n}}^{T}\textbf{X}_{n}^{T}\textbf{y}}{\frac{1}{n}\hat{\boldsymbol{\beta}}_{\lambda_{n}}^{T}\textbf{X}_{n}^{T}\textbf{X}_{n}\hat{\boldsymbol{\beta}}_{\lambda_{n}}}\
.\ $ (28)

By Continuous Mapping Theorem, the denominator of this expression converges in
probability to $c^{2}\boldsymbol{\xi}^{*T}\textbf{C}\boldsymbol{\xi}^{*}$. The
numerator is equal to:

$\frac{1}{n}\hat{\boldsymbol{\beta}}_{\lambda_{n}}^{T}\textbf{X}_{n}^{T}\textbf{X}_{n}\boldsymbol{\beta}+\hat{\boldsymbol{\beta}}_{\lambda_{n}}^{T}\textbf{X}_{n}^{T}\boldsymbol{\epsilon}\rightarrow_{p}\alpha^{*}c\boldsymbol{\xi}^{*T}\textbf{C}\boldsymbol{\xi}^{*}\
.\ $ (29)

Then $\hat{\alpha}_{\lambda_{n}}\rightarrow_{p}\frac{\alpha^{*}}{c}$ and, once
again by Continuous Mapping Theorem,
$\hat{\alpha}_{\lambda_{n}}\hat{\boldsymbol{\beta}}_{\lambda_{n}}\rightarrow\frac{\alpha^{*}}{c}c\boldsymbol{\xi}^{*}=\alpha^{*}\boldsymbol{\xi}^{*}=\boldsymbol{\beta}^{*}$.
∎

Proof of Theorem 3. Average Prediction Error for $K$-fold CV is calculated as:

$\frac{1}{K}\sum_{k}\frac{1}{n_{k}}\sum_{i}(y_{k_{i}}-\textbf{x}_{k_{i}}^{T}\hat{\boldsymbol{\beta}}_{\lambda})^{2}.\
$ (30)

To show that the expected value of $\alpha$-Modified APE is less than that of
regular APE, it is sufficient to show that $E[ModPE]<E[PE]$ where
$PE=\sum_{i}(y_{k_{i}}-\textbf{x}_{k_{i}}^{T}\hat{\boldsymbol{\beta}}_{\lambda})^{2}$
and
$ModPE=\sum_{i}(y_{k_{i}}-\hat{\alpha}_{\lambda}\textbf{x}_{k_{i}}^{T}\hat{\boldsymbol{\beta}}_{\lambda})^{2}$.
When the direction of $\boldsymbol{\beta}^{*}$ has been captured by
$\hat{\boldsymbol{\beta}}_{\lambda}$, we can write the estimate as
$\lVert\hat{\boldsymbol{\beta}}_{\lambda}\rVert_{2}\boldsymbol{\xi}^{*}.$
Expand
$y_{k_{i}}=\alpha^{*}\textbf{x}_{k_{i}}^{T}\boldsymbol{\xi}^{*}+\epsilon_{i}$
to find:

$PE=\sum_{i}\big{(}\epsilon_{i}-\textbf{x}_{k_{i}}^{T}\boldsymbol{\xi}^{*}(\lVert\hat{\boldsymbol{\beta}}_{\lambda}\rVert_{2}-\alpha^{*})\big{)}^{2}\
.\ $ (31)

Similarly,

$ModPE=\sum_{i}\Bigg{(}\epsilon_{i}-\textbf{x}_{k_{i}}^{T}\boldsymbol{\xi}^{*}\Big{(}\frac{\boldsymbol{\xi}^{*T}\textbf{X}^{T}\boldsymbol{\epsilon}}{\boldsymbol{\xi}^{*T}\textbf{X}^{T}\textbf{X}\boldsymbol{\xi}^{*}}\Big{)}\Bigg{)}^{2}\
.\ $ (32)

Let $\boldsymbol{\Sigma}$ be the covariance matrix for
$\boldsymbol{\epsilon}$. Therefore, $\boldsymbol{\Sigma}=\sigma^{2}\textbf{I}$
and

$\displaystyle E[PE]$
$\displaystyle=n_{k}\sigma^{2}+\sum_{i}(\textbf{x}_{k_{i}}^{T}\boldsymbol{\xi}^{*})^{2}E[(\lVert\hat{\boldsymbol{\beta}}_{\lambda}\rVert_{2}-\alpha^{*})^{2}]$
(33) $\displaystyle E[ModPE]$
$\displaystyle=E\Bigg{[}\sum_{i}\Bigg{(}\epsilon_{i}-\textbf{x}_{k_{i}}^{T}\boldsymbol{\xi}^{*}\Big{(}\frac{\boldsymbol{\xi}^{*T}\textbf{X}^{T}\boldsymbol{\epsilon}}{\boldsymbol{\xi}^{*T}\textbf{X}^{T}\textbf{X}\boldsymbol{\xi}^{*}}\Big{)}\Bigg{)}^{2}\Bigg{]}$
(34)
$\displaystyle=n_{k}\sigma^{2}+\sum_{i}\frac{(\textbf{x}_{k_{i}}^{T}\boldsymbol{\xi}^{*})^{2}E[\boldsymbol{\epsilon}^{T}\textbf{X}\boldsymbol{\xi}^{*}\boldsymbol{\xi}^{*T}\textbf{X}^{T}\boldsymbol{\epsilon}]}{(\boldsymbol{\xi}^{*T}\textbf{X}^{T}\textbf{X}\boldsymbol{\xi}^{*})^{2}}$
(35)
$\displaystyle=n_{k}\sigma^{2}+\sum_{i}\frac{(\textbf{x}_{k_{i}}^{T}\boldsymbol{\xi}^{*})^{2}(tr(\boldsymbol{\xi}^{*T}\textbf{X}^{T}\Sigma\textbf{X}\boldsymbol{\xi}^{*})+E[\boldsymbol{\epsilon}^{T}]\textbf{X}\boldsymbol{\xi}^{*}\boldsymbol{\xi}^{*T}\textbf{X}^{T}E[\boldsymbol{\epsilon}])}{(\boldsymbol{\xi}^{*T}\textbf{X}^{T}\textbf{X}\boldsymbol{\xi}^{*})^{2}}$
(36)
$\displaystyle=n_{k}\sigma^{2}+\sum_{i}\frac{(\textbf{x}_{k_{i}}^{T}\boldsymbol{\xi}^{*})^{2}\sigma^{2}(\boldsymbol{\xi}^{*T}\textbf{X}^{T}\textbf{X}\boldsymbol{\xi}^{*})}{(\boldsymbol{\xi}^{*T}\textbf{X}^{T}\textbf{X}\boldsymbol{\xi}^{*})^{2}}$
(37)
$\displaystyle=n_{k}\sigma^{2}+\frac{\sigma^{2}}{\boldsymbol{\xi}^{*T}\textbf{X}^{T}\textbf{X}\boldsymbol{\xi}^{*}}\sum_{i}(\textbf{x}_{k_{i}}^{T}\boldsymbol{\xi}^{*})^{2}$
(38)

Therefore $E[PE]\geq E[ModPE]$ whenever

$\displaystyle
E[(\lVert\hat{\boldsymbol{\beta}}_{\lambda}\rVert_{2}-\alpha^{*})^{2}]\geq\frac{\sigma^{2}}{\boldsymbol{\xi}^{*T}\textbf{X}^{T}\textbf{X}\boldsymbol{\xi}^{*}}$
(39) $\displaystyle\Leftrightarrow$
$\displaystyle\frac{\alpha^{*2}}{E[(\lVert\hat{\boldsymbol{\beta}}_{\lambda}\rVert_{2}-\alpha^{*})^{2}]}\leq\frac{\boldsymbol{\beta}^{*T}\textbf{X}^{T}\textbf{X}\boldsymbol{\beta}^{*}}{\sigma^{2}}$
(40)

because $\boldsymbol{\beta}^{*}=\alpha^{*}\boldsymbol{\xi}^{*}$. Therefore we
have the result. ∎

Proof of Lemma 3. When X is orthonormal,

$\hat{\alpha}_{\lambda}=\frac{\hat{\boldsymbol{\beta}}_{\lambda}^{T}\hat{\boldsymbol{\beta}}_{OLS}}{\hat{\boldsymbol{\beta}}_{\lambda}^{T}\hat{\boldsymbol{\beta}}_{\lambda}}=\frac{\sum_{j=1}^{p}|\hat{\beta}_{OLS,j}|(|\hat{\beta}_{OLS,j}|-\lambda)_{+}}{\sum_{j=1}^{p}(|\hat{\beta}_{OLS,j}|-\lambda)^{2}_{+}}\
.\ $ (41)

This gives the following expression:

$\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}=\Bigg{(}\frac{\sum_{j=1}^{p}|\hat{\beta}_{OLS,j}|(|\hat{\beta}_{OLS,j}|-\lambda)_{+}}{\sum_{j=1}^{p}(|\hat{\beta}_{OLS,j}|-\lambda)^{2}_{+}}\Bigg{)}\times\textrm{sign}(\hat{\beta}_{OLS,j^{*}})\times(|\hat{\beta}_{OLS,j^{*}}|-\lambda)_{+}\
.\ $ (42)

Letting $d_{j}=(|\hat{\beta}_{OLS,j}|-\lambda)_{+}$ and
$s_{j*}=\textrm{sign}(\hat{\beta}_{OLS,j^{*}})$, the expression becomes:

$\displaystyle\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}$
$\displaystyle=s_{j*}|\hat{\beta}_{OLS,j*}|\frac{d_{j*}^{2}}{\sum_{j=1}^{p}d_{j}^{2}}+s_{j*}d_{j*}\frac{\sum_{j\neq
j*}|\hat{\beta}_{OLS,j}|d_{j}}{\sum_{j=1}^{p}d_{j}^{2}}$ (43)
$\displaystyle=w_{1}\hat{\beta}_{OLS,j*}+\hat{\beta}_{\lambda,j^{*}}\frac{\sum_{j\neq
j*}(d_{j}+\lambda)d_{j}}{\sum_{j=1}^{p}d_{j}^{2}}\ ,\ $ (44)

which uses $s_{j*}|\hat{\beta}_{OLS,j*}|=\hat{\beta}_{OLS,j*}$,
$w_{1}=d_{j*}^{2}/\sum_{j}d_{j}^{2}$,
$\hat{\beta}_{\lambda,j^{*}}=s_{j*}d_{j*}$, and
$|\hat{\beta}_{OLS,j}|=(|\hat{\beta}_{OLS,j}|-\lambda)_{+}+\lambda$ whenever
$d_{j}>0$. Therefore:

$\displaystyle\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}$
$\displaystyle=w_{1}\hat{\beta}_{OLS,j*}+\hat{\beta}_{\lambda,j^{*}}\left(\frac{\sum_{j\neq
j*}d_{j}^{2}}{\sum_{j=1}^{p}d_{j}^{2}}+\lambda\frac{\sum_{j\neq
j*}d_{j}}{\sum_{j=1}^{p}d_{j}^{2}}\right)$ (45)
$\displaystyle=w_{1}\hat{\beta}_{OLS,j*}+(1-w_{1})\hat{\beta}_{\lambda,j^{*}}+w_{2}\hat{\beta}_{\lambda,j^{*}}\
,\ $ (46)

where $w_{2}=\lambda\frac{\sum_{j\neq j*}d_{j}}{\sum_{j=1}^{p}d_{j}^{2}}$. ∎

Proof of Theorem 4. For $|\hat{\beta}_{OLS,j^{*}}|\leq\lambda$, we have
$\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}=0$ so
$|\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}|=|\hat{\beta}_{OLS,j^{*}}|\leq\lambda$
so the theorem is satisfied for this range of values. Next we consider
$\hat{\beta}_{OLS,j^{*}}>\lambda$. By Lemma 3,

$\displaystyle\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}$
$\displaystyle=(w_{1}-1)\hat{\beta}_{OLS,j^{*}}+(1-w_{1}+w_{2})\hat{\beta}_{\lambda,j^{*}}$
(47) $\displaystyle=w_{2}\hat{\beta}_{OLS,j^{*}}+(1-w_{1}+w_{2})(-\lambda)$
(48) $\displaystyle=\frac{\lambda(\sum_{j\neq
j^{*}}d_{j})\hat{\beta}_{OLS,j^{*}}-\lambda(\sum_{j\neq
j^{*}}d_{j}^{2}+\lambda\sum_{j\neq
j^{*}}d_{j})}{(\hat{\beta}_{OLS,j^{*}}-\lambda)^{2}+\sum_{j\neq
j^{*}}d_{j}^{2}}$ (49)

Let $u=\sum_{j\neq j^{*}}d_{j}$, $v=\sum_{j\neq j^{*}}d_{j}^{2}$, and
$x=\hat{\beta}_{OLS,j^{*}}=\beta^{*}_{j^{*}}+n^{-1}\textbf{x}_{j}^{T}\boldsymbol{\epsilon}$.
This gives a function, $f(x)$, that is differentiable for $x\geq\lambda$:

$\displaystyle f(x)$ $\displaystyle=\frac{\lambda ux-\lambda(v+\lambda
u)}{(x-\lambda)^{2}+v}$ (50) $\displaystyle\frac{\partial f(x)}{\partial x}$
$\displaystyle=\frac{u\lambda}{(x-\lambda)^{2}+v}-\frac{2(x-\lambda)(u\lambda
x-\lambda(u\lambda+v))}{((x-\lambda)^{2}+v)^{2}}\ .\ $ (51)

Treating $n^{-1}\textbf{x}_{j}^{T}\boldsymbol{\epsilon}$ as a constant,
changes in $x$ are due to changes in $\beta^{*}_{j^{*}}$ alone. The derivative
$\frac{\partial f(x)}{\partial x}$ is equal to zero whenever $x=x^{*}$, where
$x^{*}$ is defined:

$\displaystyle x^{*}=\frac{u\lambda+v\pm\sqrt{u^{2}v+v^{2}}}{u}\ .\ $ (52)

Note that $x^{*}<\lambda$ when the negative part of the square root is taken,
therefore the root is restricted to
$x^{*}=(u\lambda+v+\sqrt{u^{2}v+v^{2}})/u.$ The second derivative of $f(x)$
is:

$\displaystyle\frac{\partial^{2}f(x)}{\partial
x^{2}}=\frac{2\lambda[-3v(x-\lambda)(u+x-\lambda)+u(x-\lambda)^{3}+v^{2}]}{(v+(x-\lambda)^{2})^{3}}\
.\ $ (53)

When $x=x^{*}$, the second derivative equals:

$\displaystyle-\frac{u^{4}\lambda(u^{2}+v)(\sqrt{u^{2}v+v^{2}}+v)}{2v^{2}(\sqrt{u^{2}v+v^{2}}+u^{2}+v)^{3}}<0$
(54)

meaning that $x^{*}$ gives the maximum of $f(x)$ when $x>\lambda$. For
$\lambda\leq x\leq x^{*}$, $f(x)$ is an increasing function bounded below by
$-\lambda$ and bounded above by

$\displaystyle\frac{\lambda}{2}\Big{(}\sqrt{\frac{u^{2}}{v}+1}-1\Big{)}\ .\ $
(55)

For $x>x^{*}$, $f(x)$ is decreasing so we consider $\lim_{x\to\infty}f(x)$,
which is easily shown to equal $0>-\lambda$. Therefore,

$|f(x)|\leq\lambda\times\max\left(1,\
\frac{1}{2}\Big{(}\sqrt{\frac{u^{2}}{v}+1}-1\Big{)}\right)\ ,\ $ (56)

and $\lim_{x\to\infty}|f(x)|=0$. For $\hat{\beta}_{OLS,j^{*}}<-\lambda$, note
that
$\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}=-(\hat{\alpha}_{\lambda}|\hat{\beta}_{\lambda,j^{*}}|-|\hat{\beta}_{OLS,j^{*}}|)$
and so analogous arguments to the case of $\hat{\beta}_{OLS,j^{*}}$ also
apply. This completes the proof. ∎

Proof of Theorem 5. If $\hat{\boldsymbol{\beta}}_{\lambda}$ recovers the true
sign vector, s, then its nonzero estimates are
$\hat{\boldsymbol{\beta}}_{\lambda}^{\mathcal{M}^{*}}=\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}^{*}}-n\lambda(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})^{-1}\textbf{s}$
and

$\displaystyle\hat{\alpha}_{\lambda}\hat{\boldsymbol{\beta}}_{\lambda}$
$\displaystyle=\left(\frac{\hat{\textbf{y}}_{\lambda}^{T}\textbf{y}}{\hat{\textbf{y}}_{\lambda}^{T}\hat{\textbf{y}}_{\lambda}}\right)\hat{\boldsymbol{\beta}}_{\lambda}.$
(57)

Now, we can find definitions of these components in terms of
$\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}^{*}}$:

$\displaystyle\hat{\textbf{y}}_{\lambda}^{T}\textbf{y}$
$\displaystyle=\hat{\boldsymbol{\beta}}_{\lambda}^{\mathcal{M}^{*}\,T}\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{y}$
(58)
$\displaystyle=\left(\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}^{*}\,T}-n\lambda\textbf{s}^{T}(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})^{-1}\right)\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{y}$
(59)
$\displaystyle=\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}^{*}\,T}\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{y}-n\lambda\textbf{s}^{T}(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})^{-1}\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{y}$
(60)
$\displaystyle=\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}^{*}\,T}\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}}(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})^{-1}\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{y}-n\lambda\textbf{s}^{T}\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}^{*}}$
(61)
$\displaystyle=\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}^{*}\,T}\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}}\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}^{*}}-n\lambda\textbf{s}^{T}\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}^{*}}$
(62) $\displaystyle\hat{\textbf{y}}_{\lambda}^{T}\hat{\textbf{y}}_{\lambda}$
$\displaystyle=\hat{\boldsymbol{\beta}}_{\lambda}^{\mathcal{M}^{*}\,T}\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}}\hat{\boldsymbol{\beta}}_{\lambda}^{\mathcal{M}^{*}}$
(63)
$\displaystyle=\left(\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}^{*}\,T}-n\lambda\textbf{s}^{T}(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})^{-1}\right)\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}}\left(\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}^{*}}-n\lambda(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})^{-1}\textbf{s}\right)$
(64)
$\displaystyle=\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}^{*}\,T}\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}}\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}^{*}}-2n\lambda\textbf{s}^{T}\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}^{*}}+n^{2}\lambda^{2}\textbf{s}^{T}(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})^{-1}\textbf{s}\
.\ $ (65)

Then both $\hat{\textbf{y}}_{\lambda}^{T}\textbf{y}$ and
$\hat{\textbf{y}}_{\lambda}^{T}\hat{\textbf{y}}_{\lambda}$ are quadratic
polynomials with respect to $\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}$ with a
quadratic coefficients of $n$ due to the scaling of the columns of
$\boldsymbol{X}$. We also have

$\displaystyle\hat{\textbf{y}}_{\lambda}^{T}\textbf{y}-\hat{\textbf{y}}_{\lambda}^{T}\hat{\textbf{y}}_{\lambda}$
$\displaystyle=n\lambda\textbf{s}^{T}\hat{\boldsymbol{\beta}}_{OLS}^{\mathcal{M}^{*}}-n^{2}\lambda^{2}\textbf{s}^{T}(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})^{-1}\textbf{s}$
(66) $\displaystyle=n\lambda
s_{j^{*}}\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}+n\lambda\sum_{j\neq
j^{*}}s_{j}\hat{\beta}_{OLS,j}^{\mathcal{M}^{*}}-n^{2}\lambda^{2}\textbf{s}^{T}(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})^{-1}\textbf{s}\
,\ $ (67)

and
$(\hat{\textbf{y}}_{\lambda}^{T}\textbf{y}-\hat{\textbf{y}}_{\lambda}^{T}\hat{\textbf{y}}_{\lambda})\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}$
is a quadratic polynomial with respect to
$\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}$ with quadratic coefficient
$n\lambda s_{j^{*}}$. Let
$\tilde{s}_{j^{*}}=[(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})^{-1}]_{j^{*}}\textbf{s}$
where
$[(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})^{-1}]_{j^{*}}$
denotes the $j^{*}$-th row of
$(\textbf{X}_{\mathcal{M}^{*}}^{T}\textbf{X}_{\mathcal{M}^{*}})^{-1}$. Then

$\displaystyle\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}$
$\displaystyle=\hat{\alpha}_{\lambda}\big{(}\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}-n\lambda\tilde{s}_{j^{*}}\big{)}-\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}$
(68)
$\displaystyle=\frac{\hat{\textbf{y}}_{\lambda}^{T}\textbf{y}(\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}-n\lambda\tilde{s}_{j^{*}}}{\hat{\textbf{y}}_{\lambda}^{T}\hat{\textbf{y}}_{\lambda}}-\frac{\hat{\textbf{y}}_{\lambda}^{T}\hat{\textbf{y}}_{\lambda}\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}}{\hat{\textbf{y}}_{\lambda}^{T}\hat{\textbf{y}}_{\lambda}}$
(69)
$\displaystyle=\frac{(\hat{\textbf{y}}_{\lambda}^{T}\textbf{y}-\textbf{y}_{\lambda}^{T}\hat{\textbf{y}}_{\lambda})\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}-n\lambda\tilde{s}_{j^{*}}\hat{\textbf{y}}_{\lambda}^{T}\textbf{y}}{\hat{\textbf{y}}_{\lambda}^{T}\hat{\textbf{y}}_{\lambda}}\
,\ $ (70)

and
$\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}$
is a ratio of quadratic polynomials with respect to
$\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}$ where the numerator and
denominator quadratic coefficients are $n\lambda(s_{j*}-n\tilde{s}_{j^{*}})$
and $n$, respectively. It follows that
$\lim_{|\beta_{j^{*}}^{*}|\to\infty}\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}=\lambda(s_{j*}-n\tilde{s}_{j^{*}})=G_{j^{*}}$
and so

$\displaystyle\lim_{|\beta_{j^{*}}^{*}|\to\infty}|\hat{\alpha}_{\lambda}\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}|=|G_{j^{*}}|\
.\ $ (71)

Clearly
$-n\lambda\tilde{s}_{j^{*}}=\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}$
so $|G_{j^{*}}|=|\lambda
s_{j^{*}}+\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}|$.
If $\tilde{s}_{j^{*}}=0$ then
$\hat{\beta}_{\lambda,j^{*}}=\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}$ so
$|G_{j^{*}}|=\lambda>0=|\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}|$.
If $\tilde{s}_{j^{*}}\neq 0$ then

$\displaystyle\lambda=\frac{-(\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}})}{n\tilde{s}_{j^{*}}}$
(72)

which makes

$\displaystyle|G_{j^{*}}|=\left|\left(1-\frac{s_{j^{*}}}{n\tilde{s}_{j^{*}}}\right)(\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}})\right|=\left|1-\frac{s_{j^{*}}}{n\tilde{s}_{j^{*}}}\right|\times|\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}|$
(73)

Hence
$|G_{j^{*}}|<|\hat{\beta}_{\lambda,j^{*}}-\hat{\beta}_{OLS,j^{*}}^{\mathcal{M}^{*}}|$
if and only if

$\displaystyle\left|1-\frac{s_{j^{*}}}{n\tilde{s}_{j^{*}}}\right|<1\ .\ $ (74)

∎

## B. Further Numerical Results

### B.1. Simulation Study for Independent Predictors

In this section we give additional results from the simulation studies
conducted in the main document. Tables 5 through 24 give the average model
size, average False Discovery Rates, average false negatives, and average
prediction bias for the four methods considered in this part of the study: APE
CV, AR2 CV, Mod CV, and the Relaxed Lasso for an additional $p^{*}$,
$p^{*}=15$. Mod CV was left out of the main document because of its similarity
to AR2 CV. We can see from these results that all False negatives are close to
zero. For smaller $p^{*}$, AR2 and Mod CV generally produce lower False
Discovery Rates than APE CV. Average Prediction Bias, however generally
appears lower for the two new approaches than APE CV, but higher than the
Relaxed Lasso on average.

Table 5: Average Hamming Distance for 100 replications of CV 1SE. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.19$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.59$
100 |  |  3.1 (0.3) |  4.3 (0.4) |  6 (0.6) |  7.8 (1.1) |  9.1 (1.2) |  |  2.2 (0.2) |  4 (0.4) |  7.5 (1.2) |  9.3 (1) |  10.7 (1.2)
500 |  |  0.6 (0.1) |  1 (0.2) |  1.3 (0.2) |  1.9 (0.5) |  2.9 (0.5) |  |  0.9 (0.2) |  0.9 (0.1) |  1.7 (0.3) |  1.7 (0.3) |  1.5 (0.3)
1000 |  |  0.2 (0) |  0.4 (0.1) |  0.4 (0.1) |  0.6 (0.1) |  0.9 (0.2) |  |  0.1 (0) |  0.4 (0.1) |  0.5 (0.1) |  0.7 (0.1) |  0.9 (0.2)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.06$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.53$
100 |  |  10.1 (0.3) |  14.1 (0.6) |  16.7 (0.8) |  19.6 (1.4) |  21.4 (1.6) |  |  9.3 (0.4) |  15.7 (0.8) |  22.5 (1.1) |  25.5 (1.4) |  30.9 (2)
500 |  |  4.1 (0.3) |  7.5 (0.4) |  9.7 (0.6) |  11.4 (0.6) |  15.7 (1) |  |  3.9 (0.2) |  6.1 (0.4) |  10.4 (0.6) |  13.9 (0.8) |  16.6 (1)
1000 |  |  2.8 (0.2) |  4.1 (0.3) |  6.5 (0.5) |  8.4 (0.6) |  9.4 (0.6) |  |  2.6 (0.2) |  3.8 (0.3) |  6.4 (0.5) |  7.8 (0.6) |  10.4 (0.7)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.6$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.3$
100 |  |  31.7 (1.3) |  43.5 (0.6) |  50.1 (0.5) |  51.1 (0.4) |  53.2 (1.1) |  |  5 (0.4) |  32 (0.6) |  47 (0.7) |  53.8 (1.1) |  54.5 (1.1)
500 |  |  1.6 (0.2) |  18.2 (0.5) |  33.3 (0.8) |  48.6 (1.3) |  57.4 (1.8) |  |  0 (0) |  16.8 (0.5) |  35.3 (0.9) |  58.4 (1.6) |  81.2 (2)
1000 |  |  0.4 (0.1) |  14.5 (0.4) |  28.5 (0.9) |  43.5 (1.4) |  57.6 (1.6) |  |  0 (0) |  14.2 (0.5) |  28 (0.8) |  44.8 (1.3) |  64.2 (1.8)
Table 6: Average Hamming Distance for 100 replications of AR2 CV 1SE. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.19$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.59$
100 |  |  2.1 (0.3) |  2.8 (0.3) |  3.3 (0.3) |  4.9 (0.7) |  5 (0.5) |  |  0.6 (0.1) |  0.6 (0.1) |  1.9 (0.8) |  1.8 (0.4) |  2.6 (0.4)
500 |  |  0.1 (0) |  0.2 (0) |  0.1 (0) |  0.2 (0) |  0.4 (0.1) |  |  0 (0) |  0 (0) |  0 (0) |  0 (0) |  0 (0)
1000 |  |  0 (0) |  0 (0) |  0 (0) |  0 (0) |  0 (0) |  |  0 (0) |  0 (0) |  0 (0) |  0 (0) |  0 (0)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.06$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.53$
100 |  |  10.1 (0.3) |  14.9 (0.7) |  18.1 (1.1) |  20.8 (1.5) |  22.9 (1.6) |  |  6.4 (0.4) |  10.2 (0.6) |  15.9 (1) |  17.7 (1) |  21.8 (1.4)
500 |  |  1.8 (0.2) |  2.8 (0.3) |  3.2 (0.2) |  4.2 (0.3) |  5.3 (0.4) |  |  0.3 (0.1) |  0.3 (0.1) |  0.6 (0.1) |  1.4 (0.2) |  1.5 (0.2)
1000 |  |  0.4 (0.1) |  0.6 (0.1) |  0.9 (0.2) |  1.2 (0.1) |  1.3 (0.1) |  |  0 (0) |  0 (0) |  0 (0) |  0.1 (0) |  0.1 (0)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.6$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.3$
100 |  |  23.6 (1.3) |  40.4 (0.5) |  51.4 (0.8) |  55.1 (1.2) |  54.8 (1.2) |  |  4.2 (0.4) |  31.9 (0.5) |  47.9 (0.8) |  54.6 (1.3) |  53.5 (0.9)
500 |  |  1.8 (0.2) |  17.1 (0.5) |  28.7 (0.8) |  42.3 (1.3) |  51.9 (2.2) |  |  0.2 (0) |  9.1 (0.4) |  17.8 (0.7) |  30 (1) |  46.6 (1.3)
1000 |  |  0.7 (0.1) |  9.1 (0.4) |  15.4 (0.6) |  23.1 (0.9) |  32.5 (1.1) |  |  0 (0) |  3.9 (0.3) |  6.3 (0.4) |  11.1 (0.6) |  16.9 (0.8)
Table 7: Average Hamming Distance for 100 replications of $\alpha$-modified CV 1SE. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.19$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.59$
100 |  |  2.3 (0.3) |  2.7 (0.3) |  3.7 (0.4) |  5.9 (1) |  6.8 (1.3) |  |  0.5 (0.1) |  0.9 (0.1) |  2.7 (0.9) |  2.7 (0.6) |  3.1 (0.5)
500 |  |  0 (0) |  0.3 (0.1) |  0.1 (0) |  0.3 (0.2) |  0.5 (0.3) |  |  0 (0) |  0 (0) |  0 (0) |  0 (0) |  0 (0)
1000 |  |  0 (0) |  0 (0) |  0 (0) |  0 (0) |  0 (0) |  |  0 (0) |  0 (0) |  0 (0) |  0 (0) |  0 (0)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.06$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.53$
100 |  |  9.9 (0.3) |  14 (0.6) |  16.8 (0.9) |  19.1 (1.4) |  21.8 (1.8) |  |  7.2 (0.4) |  11.6 (0.6) |  18 (1.1) |  21.1 (1.4) |  27.9 (2)
500 |  |  1.7 (0.2) |  2.8 (0.3) |  3.3 (0.2) |  4.2 (0.3) |  5 (0.3) |  |  0.6 (0.1) |  0.5 (0.1) |  0.9 (0.1) |  2.1 (0.3) |  2 (0.3)
1000 |  |  0.4 (0.1) |  0.6 (0.1) |  0.9 (0.1) |  1.3 (0.1) |  1.4 (0.1) |  |  0.1 (0) |  0.1 (0) |  0.1 (0) |  0.1 (0) |  0.2 (0.1)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.6$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.3$
100 |  |  35.7 (1.5) |  45.2 (0.6) |  51.5 (0.5) |  51.5 (0.7) |  54.3 (1.4) |  |  4.9 (0.5) |  32.7 (0.6) |  48.7 (0.8) |  55.5 (1.2) |  55.6 (1.4)
500 |  |  1.7 (0.2) |  16.4 (0.4) |  26.5 (0.6) |  36.8 (1) |  43.9 (1.5) |  |  0.1 (0) |  9.9 (0.4) |  20.3 (0.7) |  34.1 (1.1) |  51.5 (1.4)
1000 |  |  0.7 (0.1) |  8.6 (0.4) |  14.5 (0.5) |  21.6 (0.7) |  27.9 (1) |  |  0 (0) |  4.6 (0.3) |  7.8 (0.4) |  13.6 (0.6) |  20 (1)
Table 8: Average Hamming Distance for 100 replications of Relaxed Lasso. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.19$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.59$
100 |  |  3.7 (0.4) |  5.4 (0.8) |  7.4 (1.1) |  18.4 (3.4) |  28.1 (4.2) |  |  1.1 (0.3) |  1.8 (0.5) |  4.5 (1.7) |  6.6 (2.3) |  7.9 (2.6)
500 |  |  0.8 (0.3) |  1.2 (0.3) |  0.5 (0.2) |  0.7 (0.4) |  1.5 (0.6) |  |  1.2 (0.3) |  0.8 (0.2) |  0.8 (0.4) |  0.3 (0.1) |  0.1 (0)
1000 |  |  0.6 (0.2) |  0.5 (0.2) |  0.7 (0.3) |  0.4 (0.1) |  1.9 (1.5) |  |  0.6 (0.2) |  0.8 (0.3) |  0.4 (0.2) |  0.4 (0.2) |  0.4 (0.2)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.06$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.53$
100 |  |  12.3 (0.6) |  18.8 (1.1) |  32.2 (3) |  45.3 (4.3) |  55.9 (4.9) |  |  9.1 (0.7) |  14.2 (1) |  34.2 (3.4) |  81.1 (4.8) |  82.3 (4.7)
500 |  |  2.9 (0.5) |  3.8 (0.6) |  4.4 (0.5) |  5.1 (0.8) |  5 (0.4) |  |  0.6 (0.1) |  0.7 (0.2) |  0.4 (0.1) |  0.8 (0.2) |  0.8 (0.1)
1000 |  |  1 (0.3) |  0.8 (0.1) |  1.8 (0.8) |  1.2 (0.2) |  0.9 (0.1) |  |  0.3 (0.1) |  0.3 (0.1) |  0.1 (0) |  0.2 (0.1) |  0.1 (0)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.6$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.3$
100 |  |  19.5 (1.3) |  39.9 (0.5) |  57.6 (1.7) |  70.8 (3.1) |  67.9 (3.2) |  |  2.1 (0.3) |  34.1 (0.5) |  75.9 (2.5) |  87.6 (3.3) |  93.6 (3.8)
500 |  |  0.9 (0.1) |  22.9 (0.9) |  39.6 (1.8) |  66.2 (3.4) |  74.5 (4.2) |  |  0.1 (0) |  7.2 (0.5) |  14.8 (1.2) |  25.4 (2.3) |  36.8 (2.7)
1000 |  |  0.2 (0) |  12.3 (0.9) |  17.5 (1.4) |  27.9 (2.7) |  32.7 (2.9) |  |  0 (0) |  2.4 (0.3) |  3.1 (0.3) |  4.9 (0.5) |  5.5 (0.3)
Table 9: Average number of selected variables 100 replications of CV 1SE. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.23$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.61$
50 |  |  6.4 (0.6) |  10.6 (1.2) |  9.7 (1.3) |  11.5 (1.4) |  11.2 (1.6) |  |  11 (0.8) |  12.7 (0.8) |  13.9 (0.9) |  17.9 (1.2) |  18.6 (1.5)
100 |  |  6.8 (0.3) |  8.7 (0.7) |  9.9 (1) |  10.3 (1.3) |  12 (1.5) |  |  7.8 (0.4) |  9.8 (0.6) |  11.6 (0.7) |  14.5 (1.1) |  15.7 (1.4)
200 |  |  6.5 (0.2) |  6.8 (0.2) |  8 (0.4) |  9 (0.5) |  9.8 (0.6) |  |  6.6 (0.2) |  7.6 (0.3) |  8.6 (0.4) |  10.3 (0.9) |  11.3 (0.9)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.04$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.52$
50 |  |  8.9 (1) |  8 (1.2) |  8.5 (1.3) |  5.6 (1.2) |  6.6 (1.3) |  |  18.9 (1) |  20.6 (1.5) |  15.1 (1.4) |  13.7 (1.6) |  13.1 (1.6)
100 |  |  15.9 (0.6) |  15.2 (1.1) |  13.5 (1.3) |  11 (1.6) |  11.5 (1.9) |  |  22.8 (0.4) |  28.8 (0.9) |  33.7 (1.4) |  35 (1.7) |  29.9 (1.8)
200 |  |  19.4 (0.4) |  21.7 (0.6) |  22.8 (0.9) |  23.5 (1.1) |  21.4 (1.2) |  |  21.1 (0.3) |  25.3 (0.5) |  31.9 (0.9) |  38 (1.6) |  43.5 (1.4)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.52$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.26$
50 |  |  7.8 (1.1) |  9 (1.3) |  6.1 (1.3) |  5.8 (1.3) |  5.8 (1.2) |  |  22.2 (1.4) |  14.8 (1.5) |  10.7 (1.5) |  10 (1.5) |  8.4 (1.5)
100 |  |  19.2 (1.2) |  11.6 (1.6) |  8.6 (1.6) |  6.8 (1.6) |  7 (1.7) |  |  43.6 (0.5) |  44.3 (1.9) |  28.4 (2.4) |  20 (2.5) |  12.6 (2.2)
200 |  |  39.1 (0.6) |  37.8 (1.4) |  33.7 (2.2) |  20.6 (2.2) |  15 (2.6) |  |  49.2 (0.1) |  68.8 (0.6) |  82.8 (1.4) |  80.9 (2.4) |  58.1 (3)
Table 10: Average number of selected variables 100 replications of AR2 CV 1SE. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.23$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.61$
50 |  |  6.7 (0.9) |  7.3 (1) |  6 (0.9) |  6 (0.8) |  5.4 (0.9) |  |  7.4 (0.7) |  7.9 (0.7) |  6.8 (0.6) |  7.5 (0.6) |  7.8 (0.8)
100 |  |  4.8 (0.2) |  5.2 (0.4) |  5.6 (0.7) |  6.2 (1.1) |  6.4 (0.8) |  |  5.3 (0.1) |  5.8 (0.2) |  6.3 (0.4) |  6.6 (0.3) |  6.7 (0.4)
200 |  |  4.9 (0.1) |  5.1 (0.2) |  5.2 (0.2) |  5.3 (0.3) |  5.2 (0.3) |  |  5.1 (0.1) |  5.2 (0.1) |  5.2 (0.1) |  5.2 (0.1) |  5.2 (0.1)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.04$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.52$
50 |  |  10.8 (1.3) |  10.3 (1.2) |  9.1 (1.2) |  5.9 (0.8) |  9.2 (1.2) |  |  16.3 (1) |  13.5 (1.2) |  10.8 (1.2) |  8.4 (1.1) |  9.2 (1.2)
100 |  |  15.6 (0.7) |  15 (1.4) |  15.1 (1.8) |  9.8 (1.3) |  12.8 (1.7) |  |  18.3 (0.4) |  22.9 (0.7) |  24.1 (1.1) |  21.9 (1.2) |  22.9 (1.6)
200 |  |  16.8 (0.5) |  17 (0.6) |  17.9 (0.8) |  17.3 (1) |  15.9 (1) |  |  16.7 (0.2) |  17.6 (0.3) |  20.8 (0.5) |  22.9 (0.7) |  25.9 (1.1)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.52$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.26$
50 |  |  14.4 (1.3) |  10 (1.3) |  8.3 (1.2) |  6.6 (1) |  7.2 (1) |  |  22.5 (1.4) |  11.7 (1.3) |  9.9 (1.2) |  7.3 (0.9) |  6.3 (0.9)
100 |  |  26.4 (1.3) |  23 (2.3) |  16.4 (2.2) |  14.6 (2.1) |  13.3 (1.9) |  |  44 (0.6) |  46.6 (1.7) |  36.1 (2.3) |  24.2 (2.4) |  15.6 (2)
200 |  |  42.4 (0.4) |  48.3 (1.5) |  48.8 (2.6) |  43.8 (3.6) |  42 (4.7) |  |  49 (0.1) |  64.8 (0.7) |  76.5 (1.5) |  73.1 (2.7) |  55.7 (3.1)
Table 11: Average number of selected variables 100 replications of $\alpha$-modified CV 1SE. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.23$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.61$
50 |  |  5.2 (0.7) |  8.6 (1.2) |  8.5 (1.3) |  9.6 (1.4) |  9.8 (1.5) |  |  8.3 (0.7) |  9.7 (0.7) |  9.3 (0.8) |  12 (1.1) |  12.2 (1.3)
100 |  |  4.8 (0.2) |  5.2 (0.3) |  6.3 (0.8) |  6.5 (1.2) |  7.8 (1.3) |  |  5.5 (0.1) |  6.3 (0.3) |  6.7 (0.4) |  7.6 (0.4) |  8 (0.5)
200 |  |  5 (0.1) |  5.2 (0.2) |  5 (0.1) |  5.5 (0.3) |  5.2 (0.2) |  |  5.1 (0) |  5.3 (0.1) |  5.2 (0.1) |  5.6 (0.3) |  5.8 (0.3)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.04$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.52$
50 |  |  8 (1.1) |  7.2 (1.3) |  9 (1.4) |  5.3 (1.2) |  6.6 (1.4) |  |  17.7 (1.1) |  18.8 (1.5) |  14.2 (1.5) |  13.5 (1.7) |  13.7 (1.7)
100 |  |  12.4 (0.7) |  11.2 (1) |  9.5 (1.3) |  9.5 (2) |  10.3 (2.1) |  |  19.6 (0.4) |  24.1 (0.7) |  27.3 (1.3) |  28.3 (1.8) |  25.8 (1.7)
200 |  |  16 (0.5) |  15.9 (0.5) |  14.9 (0.7) |  15.2 (1) |  12.5 (1) |  |  17.1 (0.2) |  18.9 (0.4) |  22.1 (0.6) |  24.6 (0.8) |  28.4 (1.1)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.52$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.26$
50 |  |  7.2 (1.1) |  9 (1.4) |  5.8 (1.3) |  6.4 (1.4) |  6.5 (1.3) |  |  21.3 (1.5) |  15 (1.7) |  10.2 (1.5) |  10.1 (1.6) |  8.3 (1.6)
100 |  |  16.1 (1.5) |  7.7 (1.5) |  7.5 (2) |  6.8 (1.8) |  8.9 (2.2) |  |  43.3 (0.6) |  43.1 (2.1) |  29.8 (2.7) |  22.9 (2.9) |  13.6 (2.5)
200 |  |  40 (0.7) |  30 (1.9) |  24.3 (2.6) |  12.8 (2.4) |  11.3 (3.3) |  |  49.1 (0.1) |  65.6 (0.7) |  77.3 (1.6) |  75 (2.7) |  52.3 (3.7)
Table 12: Average number of selected variables 100 replications of Relaxed Lasso. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.23$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.61$
50 |  |  9.1 (0.8) |  22.4 (2.2) |  30.4 (2.7) |  31.8 (2.7) |  36.5 (2.9) |  |  12.2 (1.1) |  19.5 (2) |  27.5 (2.6) |  39.9 (2.7) |  43 (2.7)
100 |  |  7.8 (0.6) |  10.5 (1.3) |  11.5 (1.6) |  18.6 (3.2) |  30.2 (4.4) |  |  6 (0.2) |  7.2 (0.6) |  8.2 (1) |  13.7 (2.8) |  13.4 (2.7)
200 |  |  6.5 (0.3) |  6.5 (0.4) |  7.2 (0.9) |  8.3 (1.1) |  8.7 (1.7) |  |  5.6 (0.2) |  5.7 (0.2) |  6.8 (1) |  5.4 (0.2) |  5.6 (0.2)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.04$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.52$
50 |  |  15.2 (1.3) |  26.4 (2.4) |  31 (2.7) |  26.6 (2.7) |  30.8 (2.9) |  |  23.8 (1.1) |  47.1 (2.2) |  40 (2.6) |  44.8 (2.6) |  39.4 (2.7)
100 |  |  21.1 (0.8) |  23.6 (1.6) |  40.1 (3.9) |  35.2 (4.3) |  46.2 (5) |  |  21.6 (0.7) |  25.9 (1.1) |  49.8 (4.2) |  78.6 (5.3) |  91.5 (5.1)
200 |  |  21.2 (0.7) |  25.3 (1.2) |  29.9 (1.8) |  30 (2.6) |  34.1 (4.2) |  |  17.1 (0.4) |  18.5 (0.7) |  21.8 (1.3) |  23.4 (1.5) |  25.8 (2.4)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.52$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.26$
50 |  |  16.1 (1.3) |  27.4 (2.5) |  20 (2.4) |  25.8 (2.7) |  26.8 (2.8) |  |  27.9 (1.4) |  35.7 (2.5) |  43.1 (2.6) |  36.4 (2.7) |  28.1 (2.7)
100 |  |  31 (1.1) |  26.2 (2.1) |  37.5 (4.4) |  41.6 (4.9) |  38.8 (4.8) |  |  46.7 (0.5) |  57.4 (1.8) |  81.5 (4.5) |  77.6 (5.6) |  61 (5.7)
200 |  |  45.1 (0.4) |  56.8 (1.5) |  60.5 (2.7) |  61.7 (5.3) |  62 (7.8) |  |  49.4 (0.1) |  71 (1.1) |  92 (2.3) |  159.9 (8.5) |  205.7 (9)
Table 13: Average false discovery rate from 100 replications of CV 1SE. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.23$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.61$
50 |  |  0.3 (0) |  0.4 (0) |  0.4 (0) |  0.5 (0) |  0.4 (0) |  |  0.4 (0) |  0.5 (0) |  0.6 (0) |  0.6 (0) |  0.6 (0)
100 |  |  0.3 (0) |  0.4 (0) |  0.4 (0) |  0.4 (0) |  0.4 (0) |  |  0.3 (0) |  0.4 (0) |  0.5 (0) |  0.5 (0) |  0.5 (0)
200 |  |  0.2 (0) |  0.2 (0) |  0.3 (0) |  0.3 (0) |  0.4 (0) |  |  0.2 (0) |  0.3 (0) |  0.3 (0) |  0.4 (0) |  0.4 (0)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.04$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.52$
50 |  |  0.2 (0) |  0.3 (0) |  0.3 (0) |  0.2 (0) |  0.3 (0) |  |  0.4 (0) |  0.4 (0) |  0.4 (0) |  0.4 (0) |  0.4 (0)
100 |  |  0.3 (0) |  0.3 (0) |  0.4 (0) |  0.3 (0) |  0.3 (0) |  |  0.3 (0) |  0.5 (0) |  0.6 (0) |  0.6 (0) |  0.6 (0)
200 |  |  0.3 (0) |  0.4 (0) |  0.4 (0) |  0.5 (0) |  0.5 (0) |  |  0.3 (0) |  0.4 (0) |  0.5 (0) |  0.6 (0) |  0.6 (0)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.52$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.26$
50 |  |  0 (0) |  0.2 (0) |  0.2 (0) |  0.2 (0) |  0.2 (0) |  |  0 (0) |  0.2 (0) |  0.2 (0) |  0.3 (0) |  0.3 (0)
100 |  |  0 (0) |  0.1 (0) |  0.2 (0) |  0.2 (0) |  0.2 (0) |  |  0 (0) |  0.3 (0) |  0.3 (0) |  0.3 (0) |  0.2 (0)
200 |  |  0 (0) |  0.2 (0) |  0.3 (0) |  0.3 (0) |  0.3 (0) |  |  0 (0) |  0.3 (0) |  0.5 (0) |  0.5 (0) |  0.5 (0)
Table 14: Average false discovery rate for 100 replications of AR2 CV 1SE. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.23$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.61$
50 |  |  0.3 (0) |  0.3 (0) |  0.4 (0) |  0.4 (0) |  0.4 (0) |  |  0.2 (0) |  0.3 (0) |  0.3 (0) |  0.3 (0) |  0.4 (0)
100 |  |  0.1 (0) |  0.1 (0) |  0.2 (0) |  0.2 (0) |  0.2 (0) |  |  0.1 (0) |  0.1 (0) |  0.1 (0) |  0.2 (0) |  0.2 (0)
200 |  |  0 (0) |  0.1 (0) |  0.1 (0) |  0.1 (0) |  0.1 (0) |  |  0 (0) |  0 (0) |  0 (0) |  0 (0) |  0 (0)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.04$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.52$
50 |  |  0.3 (0) |  0.4 (0) |  0.5 (0) |  0.6 (0) |  0.7 (0) |  |  0.3 (0) |  0.3 (0) |  0.4 (0) |  0.5 (0) |  0.5 (0)
100 |  |  0.3 (0) |  0.3 (0) |  0.4 (0) |  0.3 (0) |  0.4 (0) |  |  0.2 (0) |  0.4 (0) |  0.4 (0) |  0.5 (0) |  0.5 (0)
200 |  |  0.2 (0) |  0.2 (0) |  0.3 (0) |  0.3 (0) |  0.4 (0) |  |  0.1 (0) |  0.2 (0) |  0.3 (0) |  0.3 (0) |  0.4 (0)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.52$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.26$
50 |  |  0 (0) |  0.2 (0) |  0.5 (0) |  0.6 (0) |  0.7 (0) |  |  0 (0) |  0.2 (0) |  0.4 (0) |  0.5 (0) |  0.7 (0)
100 |  |  0 (0) |  0.2 (0) |  0.4 (0) |  0.5 (0) |  0.6 (0) |  |  0 (0) |  0.3 (0) |  0.4 (0) |  0.4 (0) |  0.5 (0)
200 |  |  0 (0) |  0.3 (0) |  0.4 (0) |  0.5 (0) |  0.5 (0) |  |  0 (0) |  0.3 (0) |  0.4 (0) |  0.5 (0) |  0.5 (0)
Table 15: Average false discovery rate of 100 replications for $\alpha$-modified CV 1SE. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.23$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.61$
50 |  |  0.2 (0) |  0.3 (0) |  0.3 (0) |  0.4 (0) |  0.3 (0) |  |  0.3 (0) |  0.4 (0) |  0.4 (0) |  0.4 (0) |  0.4 (0)
100 |  |  0.1 (0) |  0.1 (0) |  0.2 (0) |  0.2 (0) |  0.2 (0) |  |  0.1 (0) |  0.1 (0) |  0.2 (0) |  0.2 (0) |  0.3 (0)
200 |  |  0 (0) |  0.1 (0) |  0.1 (0) |  0.1 (0) |  0.1 (0) |  |  0 (0) |  0.1 (0) |  0 (0) |  0.1 (0) |  0.1 (0)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.04$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.52$
50 |  |  0.2 (0) |  0.2 (0) |  0.3 (0) |  0.2 (0) |  0.2 (0) |  |  0.3 (0) |  0.4 (0) |  0.4 (0) |  0.4 (0) |  0.4 (0)
100 |  |  0.2 (0) |  0.2 (0) |  0.2 (0) |  0.2 (0) |  0.2 (0) |  |  0.2 (0) |  0.4 (0) |  0.5 (0) |  0.5 (0) |  0.5 (0)
200 |  |  0.2 (0) |  0.2 (0) |  0.3 (0) |  0.3 (0) |  0.3 (0) |  |  0.1 (0) |  0.2 (0) |  0.3 (0) |  0.4 (0) |  0.4 (0)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.52$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.26$
50 |  |  0 (0) |  0.2 (0) |  0.1 (0) |  0.2 (0) |  0.2 (0) |  |  0 (0) |  0.2 (0) |  0.2 (0) |  0.3 (0) |  0.3 (0)
100 |  |  0 (0) |  0.1 (0) |  0.1 (0) |  0.1 (0) |  0.2 (0) |  |  0 (0) |  0.3 (0) |  0.3 (0) |  0.3 (0) |  0.2 (0)
200 |  |  0 (0) |  0.2 (0) |  0.2 (0) |  0.2 (0) |  0.1 (0) |  |  0 (0) |  0.3 (0) |  0.4 (0) |  0.5 (0) |  0.5 (0)
Table 16: Average false discovery rate for 100 replications of Relaxed Lasso. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.23$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.61$
50 |  |  0.4 (0) |  0.6 (0) |  0.6 (0) |  0.7 (0) |  0.7 (0) |  |  0.4 (0) |  0.5 (0) |  0.5 (0) |  0.7 (0) |  0.7 (0)
100 |  |  0.3 (0) |  0.3 (0) |  0.3 (0) |  0.4 (0) |  0.5 (0) |  |  0.1 (0) |  0.1 (0) |  0.2 (0) |  0.2 (0) |  0.2 (0)
200 |  |  0.1 (0) |  0.1 (0) |  0.1 (0) |  0.2 (0) |  0.2 (0) |  |  0.1 (0) |  0.1 (0) |  0.1 (0) |  0 (0) |  0 (0)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.04$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.52$
50 |  |  0.4 (0) |  0.5 (0) |  0.6 (0) |  0.7 (0) |  0.6 (0) |  |  0.4 (0) |  0.7 (0) |  0.7 (0) |  0.7 (0) |  0.7 (0)
100 |  |  0.4 (0) |  0.5 (0) |  0.6 (0) |  0.5 (0) |  0.6 (0) |  |  0.3 (0) |  0.4 (0) |  0.6 (0) |  0.7 (0) |  0.8 (0)
200 |  |  0.3 (0) |  0.4 (0) |  0.5 (0) |  0.5 (0) |  0.5 (0) |  |  0.1 (0) |  0.2 (0) |  0.2 (0) |  0.3 (0) |  0.3 (0)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.52$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.26$
50 |  |  0 (0) |  0.3 (0) |  0.4 (0) |  0.5 (0) |  0.6 (0) |  |  0 (0) |  0.3 (0) |  0.5 (0) |  0.6 (0) |  0.6 (0)
100 |  |  0 (0) |  0.2 (0) |  0.4 (0) |  0.5 (0) |  0.5 (0) |  |  0 (0) |  0.3 (0) |  0.5 (0) |  0.6 (0) |  0.6 (0)
200 |  |  0 (0) |  0.3 (0) |  0.5 (0) |  0.5 (0) |  0.5 (0) |  |  0 (0) |  0.3 (0) |  0.5 (0) |  0.7 (0) |  0.8 (0)
Table 17: Average number of false negatives from 100 replications of CV 1SE. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.23$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.61$
50 |  |  1.9 (0.2) |  2.2 (0.2) |  2.9 (0.2) |  2.9 (0.2) |  3.5 (0.2) |  |  0.2 (0.1) |  0.3 (0.1) |  0.6 (0.1) |  0.6 (0.1) |  1.1 (0.1)
100 |  |  0.6 (0.1) |  0.6 (0.1) |  0.9 (0.1) |  1.2 (0.1) |  1.4 (0.1) |  |  0 (0) |  0 (0) |  0 (0) |  0.1 (0) |  0.1 (0)
200 |  |  0.2 (0) |  0.2 (0) |  0.2 (0) |  0.3 (0) |  0.2 (0) |  |  0 (0) |  0 (0) |  0 (0) |  0 (0) |  0 (0)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.04$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.52$
50 |  |  10.1 (0.4) |  12.2 (0.3) |  12.8 (0.3) |  14 (0.2) |  14.3 (0.1) |  |  4.2 (0.4) |  6.5 (0.5) |  9.7 (0.4) |  11.7 (0.3) |  12.7 (0.2)
100 |  |  4.5 (0.3) |  6.9 (0.4) |  9.1 (0.4) |  11.4 (0.3) |  12.3 (0.3) |  |  0.4 (0.1) |  0.8 (0.1) |  1.6 (0.2) |  3.2 (0.2) |  5.5 (0.3)
200 |  |  1 (0.1) |  2 (0.2) |  3 (0.2) |  4.2 (0.2) |  5.7 (0.3) |  |  0 (0) |  0.1 (0) |  0.2 (0) |  0.3 (0.1) |  0.4 (0.1)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.52$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.26$
50 |  |  42.2 (1.1) |  44.4 (0.8) |  47.9 (0.4) |  48.7 (0.3) |  49.2 (0.2) |  |  27.8 (1.4) |  40.4 (0.9) |  45.3 (0.6) |  47.3 (0.4) |  48.6 (0.3)
100 |  |  30.8 (1.2) |  41.5 (1.1) |  45.8 (0.7) |  47.6 (0.5) |  48.5 (0.3) |  |  6.3 (0.5) |  19.3 (1.1) |  34.5 (1.1) |  42 (0.9) |  46.3 (0.6)
200 |  |  10.9 (0.6) |  21.6 (0.9) |  29.8 (1.1) |  40 (0.9) |  44.4 (0.7) |  |  0.8 (0.1) |  2.1 (0.2) |  5.8 (0.3) |  13.6 (0.6) |  26.3 (0.8)
Table 18: Average number of false negatives for 100 replications of AR2 CV 1SE. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.23$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.61$
50 |  |  2.1 (0.1) |  2.4 (0.1) |  2.9 (0.1) |  3 (0.1) |  3.6 (0.1) |  |  0.6 (0.1) |  0.8 (0.1) |  1.4 (0.1) |  1.4 (0.1) |  1.9 (0.1)
100 |  |  1.1 (0.1) |  1.2 (0.1) |  1.6 (0.1) |  1.8 (0.1) |  1.9 (0.1) |  |  0.1 (0) |  0.1 (0) |  0.1 (0) |  0.2 (0.1) |  0.3 (0.1)
200 |  |  0.4 (0.1) |  0.4 (0.1) |  0.5 (0.1) |  0.6 (0.1) |  0.7 (0.1) |  |  0 (0) |  0 (0) |  0 (0) |  0.1 (0) |  0 (0)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.04$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.52$
50 |  |  9.5 (0.4) |  11.1 (0.3) |  12.3 (0.3) |  13.6 (0.1) |  13.7 (0.1) |  |  5.2 (0.4) |  8.4 (0.5) |  10.5 (0.3) |  12.4 (0.2) |  13.1 (0.2)
100 |  |  4.9 (0.3) |  7.2 (0.4) |  9.1 (0.4) |  11.3 (0.3) |  11.6 (0.3) |  |  1 (0.1) |  1.4 (0.2) |  2.7 (0.2) |  4.8 (0.3) |  6.5 (0.4)
200 |  |  1.9 (0.2) |  2.9 (0.2) |  4 (0.2) |  5.2 (0.3) |  6.8 (0.3) |  |  0.2 (0) |  0.4 (0.1) |  0.4 (0.1) |  0.7 (0.1) |  0.9 (0.1)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.52$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.26$
50 |  |  35.6 (1.3) |  43.7 (0.7) |  46.8 (0.4) |  48.2 (0.3) |  49 (0.1) |  |  27.5 (1.4) |  41.8 (0.8) |  45.2 (0.5) |  47.5 (0.3) |  48.7 (0.2)
100 |  |  23.6 (1.3) |  34.7 (1.3) |  42.2 (0.9) |  45.1 (0.6) |  46.7 (0.4) |  |  6 (0.6) |  18.1 (1) |  31 (1) |  40.1 (0.8) |  45.1 (0.5)
200 |  |  7.7 (0.4) |  16.1 (0.8) |  24.2 (0.9) |  32.9 (1) |  38.4 (0.8) |  |  1 (0.1) |  3 (0.2) |  7 (0.4) |  15.4 (0.7) |  26.9 (0.8)
Table 19: Average number of false negatives of 100 replications for $\alpha$-modified CV 1SE. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.23$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.61$
50 |  |  2.5 (0.2) |  2.6 (0.2) |  3.2 (0.2) |  3.1 (0.2) |  3.6 (0.1) |  |  0.4 (0.1) |  0.5 (0.1) |  1 (0.1) |  1 (0.1) |  1.5 (0.2)
100 |  |  1.1 (0.1) |  1.1 (0.1) |  1.4 (0.1) |  1.8 (0.1) |  1.9 (0.1) |  |  0 (0) |  0.1 (0) |  0.1 (0) |  0.1 (0) |  0.2 (0)
200 |  |  0.3 (0.1) |  0.4 (0.1) |  0.5 (0.1) |  0.6 (0.1) |  0.7 (0.1) |  |  0 (0) |  0 (0) |  0 (0) |  0 (0) |  0 (0)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.04$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.52$
50 |  |  10.9 (0.5) |  12.6 (0.3) |  12.8 (0.3) |  14 (0.2) |  14.3 (0.1) |  |  4.7 (0.4) |  7.2 (0.5) |  10 (0.4) |  11.8 (0.3) |  12.6 (0.3)
100 |  |  6.3 (0.4) |  8.5 (0.4) |  10.7 (0.4) |  12.4 (0.3) |  12.8 (0.3) |  |  0.7 (0.1) |  1.3 (0.1) |  2.4 (0.2) |  4.2 (0.3) |  6.1 (0.4)
200 |  |  2.1 (0.2) |  3.2 (0.2) |  5 (0.3) |  6.1 (0.4) |  7.8 (0.4) |  |  0.2 (0) |  0.3 (0.1) |  0.4 (0.1) |  0.6 (0.1) |  0.7 (0.1)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.52$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.26$
50 |  |  42.8 (1.1) |  44.6 (0.8) |  48.1 (0.4) |  48.5 (0.3) |  49.1 (0.2) |  |  28.7 (1.5) |  40.4 (1) |  45.5 (0.7) |  47.3 (0.4) |  48.7 (0.2)
100 |  |  33.9 (1.5) |  44.4 (1) |  47 (0.7) |  47.9 (0.5) |  48.4 (0.4) |  |  6.7 (0.6) |  20.3 (1.3) |  34.6 (1.2) |  41.6 (1) |  46.4 (0.6)
200 |  |  9.9 (0.7) |  27.3 (1.4) |  35.9 (1.3) |  44.1 (0.9) |  46.8 (0.7) |  |  0.9 (0.1) |  2.7 (0.2) |  7 (0.4) |  15.2 (0.7) |  29 (1.1)
Table 20: Average number of false negatives for 100 replications of Relaxed Lasso. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.23$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.61$
50 |  |  1.4 (0.1) |  1.4 (0.1) |  2 (0.1) |  2 (0.1) |  2.5 (0.1) |  |  0.2 (0) |  0.3 (0.1) |  0.6 (0.1) |  0.4 (0.1) |  0.7 (0.1)
100 |  |  0.6 (0.1) |  0.7 (0.1) |  0.8 (0.1) |  1.1 (0.1) |  1.2 (0.1) |  |  0 (0) |  0.1 (0) |  0.1 (0) |  0.1 (0) |  0.2 (0)
200 |  |  0.2 (0) |  0.1 (0) |  0.3 (0.1) |  0.4 (0.1) |  0.6 (0.1) |  |  0 (0) |  0 (0) |  0 (0) |  0 (0) |  0 (0)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.04$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.52$
50 |  |  7.8 (0.4) |  8.4 (0.4) |  10 (0.4) |  12 (0.3) |  12.8 (0.2) |  |  2.8 (0.3) |  3.2 (0.3) |  6.5 (0.4) |  8.3 (0.3) |  10.6 (0.3)
100 |  |  3.2 (0.2) |  5 (0.3) |  5.7 (0.3) |  8.5 (0.4) |  9.5 (0.3) |  |  0.6 (0.1) |  1.2 (0.1) |  1.8 (0.2) |  2.4 (0.2) |  3.6 (0.3)
200 |  |  0.9 (0.1) |  1.9 (0.2) |  2.7 (0.2) |  4.1 (0.3) |  5.3 (0.3) |  |  0.2 (0) |  0.4 (0.1) |  0.6 (0.1) |  0.9 (0.1) |  1.1 (0.1)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.52$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.26$
50 |  |  33.9 (1.3) |  34.6 (1.3) |  43.2 (0.8) |  44.9 (0.5) |  47.2 (0.3) |  |  22.1 (1.4) |  29.5 (1.3) |  34.9 (0.9) |  41.4 (0.6) |  46.1 (0.4)
100 |  |  19 (1.1) |  32.9 (1.2) |  35.9 (1.3) |  40.1 (1) |  43.9 (0.6) |  |  3.3 (0.5) |  13.4 (0.9) |  19.8 (1.1) |  30.7 (1.1) |  38.7 (0.9)
200 |  |  4.9 (0.4) |  12.6 (0.8) |  20.7 (0.9) |  29.8 (1) |  37 (1) |  |  0.7 (0.1) |  2.2 (0.3) |  5 (0.4) |  8.8 (0.6) |  15.2 (0.8)
Table 21: Average prediction bias from 100 replications of CV 1SE. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.23$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.61$
50 |  |  28.8 (0.9) |  28.5 (0.8) |  31.4 (0.8) |  32.4 (0.7) |  34.4 (0.8) |  |  14.2 (0.4) |  15.3 (0.5) |  16.5 (0.5) |  18.1 (0.6) |  21.2 (0.7)
100 |  |  28.2 (0.8) |  29.6 (0.9) |  32.1 (0.9) |  35.8 (0.9) |  38.3 (0.9) |  |  14.6 (0.4) |  15.1 (0.4) |  16 (0.4) |  17.8 (0.4) |  19.6 (0.5)
200 |  |  29.9 (0.8) |  32.2 (0.9) |  33.3 (0.9) |  36.4 (0.8) |  37.7 (0.8) |  |  15.1 (0.3) |  16.2 (0.4) |  16.4 (0.4) |  18.2 (0.4) |  19.1 (0.4)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.04$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.52$
50 |  |  56.4 (1.4) |  61.4 (1.1) |  61.7 (1.1) |  65.2 (1) |  66.5 (1) |  |  33.6 (1.1) |  39.1 (1.4) |  46.8 (1.5) |  51.4 (1.7) |  54.2 (1.7)
100 |  |  60 (1.2) |  69.4 (1.4) |  77.5 (1.3) |  86.4 (1.5) |  88.3 (1.5) |  |  30.3 (0.6) |  34.1 (0.6) |  40.3 (0.9) |  46.7 (1.2) |  55.8 (1.5)
200 |  |  63.9 (1.2) |  71.9 (1.3) |  79.9 (1.2) |  87.3 (1.5) |  97.1 (1.8) |  |  32 (0.5) |  35.3 (0.6) |  39.4 (0.6) |  43.8 (0.7) |  47.4 (0.7)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.52$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.26$
50 |  |  113.4 (2) |  113.2 (2.1) |  122.6 (1.8) |  123.8 (2) |  120.7 (1.9) |  |  75.3 (2.8) |  91.5 (3) |  105 (3.1) |  110.5 (3.4) |  111.4 (3)
100 |  |  133.6 (2.8) |  158.6 (2.8) |  167.7 (2.5) |  174.8 (2.2) |  176.8 (2.3) |  |  65.1 (1) |  89.4 (2.6) |  123.2 (3.4) |  144.8 (3.9) |  161.9 (3.5)
200 |  |  133.9 (2.1) |  169.7 (2.6) |  192.2 (3.2) |  221.6 (3.3) |  235.3 (3) |  |  67.3 (0.8) |  81.7 (0.9) |  97.1 (1.1) |  119.4 (2) |  156.2 (3.5)
Table 22: Average prediction bias for 100 replications of AR2 CV 1SE. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.23$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.61$
50 |  |  27.4 (0.8) |  28.6 (0.6) |  29.5 (0.6) |  31.4 (0.7) |  33 (0.6) |  |  15.3 (0.6) |  15.4 (0.5) |  18.3 (0.6) |  19.4 (0.7) |  21 (0.7)
100 |  |  27.5 (0.8) |  28.3 (0.9) |  29.5 (0.9) |  32.9 (0.8) |  34.9 (0.9) |  |  13.7 (0.5) |  14.8 (0.4) |  14.3 (0.4) |  15.9 (0.5) |  16.8 (0.6)
200 |  |  27.6 (0.9) |  28.8 (0.8) |  28.9 (0.8) |  30.9 (0.9) |  32.1 (0.9) |  |  14 (0.5) |  15 (0.4) |  14.2 (0.4) |  15.9 (0.4) |  15.8 (0.5)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.04$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.52$
50 |  |  57.5 (1.1) |  60.6 (0.9) |  61.6 (0.9) |  63.7 (0.9) |  66.1 (0.8) |  |  35.7 (1) |  43.1 (1.3) |  46.2 (1.2) |  50.7 (1.2) |  52.3 (1.2)
100 |  |  61.7 (1.1) |  71.1 (1.2) |  78.4 (1.2) |  83.3 (1.1) |  85.1 (1.1) |  |  32.2 (0.7) |  34.6 (0.6) |  41.5 (1) |  47.8 (1.2) |  53.8 (1.5)
200 |  |  62.2 (1.3) |  70.6 (1.4) |  78.3 (1.2) |  85.1 (1.3) |  93.6 (1.5) |  |  32 (0.6) |  34.1 (0.6) |  37.1 (0.6) |  40.2 (0.7) |  43.9 (0.8)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.52$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.26$
50 |  |  112.3 (1.7) |  117.1 (1.7) |  122.1 (1.4) |  121.6 (1.4) |  123.3 (1.5) |  |  73 (2) |  90.9 (2.4) |  97.8 (2.3) |  103.7 (2.5) |  103.5 (2.1)
100 |  |  131.1 (2.1) |  151.9 (1.9) |  162.1 (1.7) |  168.8 (1.7) |  170.5 (1.8) |  |  67 (1.1) |  86.1 (1.7) |  105.8 (2.5) |  126.2 (3.1) |  141.9 (3.2)
200 |  |  134.5 (1.8) |  167.6 (1.7) |  188.1 (2.1) |  210.1 (1.9) |  223 (2) |  |  70.2 (1) |  84.9 (1) |  99.3 (1) |  119 (1.7) |  142.1 (3)
Table 23: Average number of prediction bias of 100 replications for $\alpha$-modified CV 1SE. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.23$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.61$
50 |  |  28.5 (0.9) |  29.8 (0.8) |  32.1 (0.7) |  32.8 (0.8) |  34.7 (0.7) |  |  13.1 (0.4) |  14.1 (0.5) |  16.2 (0.5) |  17.4 (0.5) |  19.6 (0.7)
100 |  |  26.7 (0.9) |  26.9 (0.9) |  28.9 (0.9) |  33.3 (0.9) |  35.2 (0.9) |  |  12.4 (0.4) |  13.3 (0.4) |  12.9 (0.4) |  14.6 (0.4) |  15.5 (0.5)
200 |  |  26.6 (0.8) |  27.9 (0.8) |  28.5 (0.8) |  30.3 (0.9) |  31.6 (0.9) |  |  13.1 (0.4) |  13.7 (0.4) |  13.1 (0.4) |  14.4 (0.4) |  14.2 (0.4)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.04$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.52$
50 |  |  61 (1.3) |  65.3 (1) |  63.8 (1) |  66.4 (1) |  68.7 (0.9) |  |  34.5 (1) |  40.6 (1.4) |  45.5 (1.5) |  50.4 (1.6) |  51.9 (1.6)
100 |  |  65 (1.4) |  73.9 (1.4) |  83.1 (1.4) |  89.7 (1.3) |  90.8 (1.4) |  |  30.5 (0.6) |  33.6 (0.6) |  39.9 (0.9) |  45.6 (1.2) |  51.7 (1.4)
200 |  |  63.6 (1.4) |  72.2 (1.5) |  82 (1.5) |  88.5 (1.5) |  98.7 (1.9) |  |  30.3 (0.5) |  32.3 (0.6) |  35.8 (0.6) |  38.9 (0.6) |  42.2 (0.7)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.52$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.26$
50 |  |  120.2 (1.6) |  120.5 (1.7) |  126.1 (1.4) |  126.7 (1.6) |  125.2 (1.4) |  |  78.2 (2.8) |  91.1 (3) |  105.3 (3.1) |  108 (3.4) |  110.9 (2.9)
100 |  |  148.7 (2.8) |  169.6 (2.2) |  176.3 (2) |  179.6 (1.8) |  180.7 (1.8) |  |  67.6 (1.1) |  92.3 (2.7) |  119.2 (3.6) |  139.1 (4.2) |  157.9 (3.9)
200 |  |  139.4 (2.1) |  192 (3.5) |  215.8 (3.4) |  239.4 (2.8) |  247.6 (2.4) |  |  68.5 (0.9) |  84.2 (1) |  99 (1.1) |  117.9 (1.7) |  151.5 (4.3)
Table 24: Average prediction bias for 100 replications of Relaxed Lasso. Standard errors given in parentheses. | p |  50 |  100 |  200 |  400 |  800 |  |  50 |  100 |  200 |  400 |  800
---|---|---|---|---|---|---|---|---|---|---|---|---
n |  | $p^{*}=5;SNR=1.25;\bar{\sigma}=5.23$ |  | $p^{*}=5;SNR=5;\bar{\sigma}=2.61$
50 |  |  24.5 (0.7) |  27.8 (0.8) |  31.8 (0.8) |  32.4 (0.8) |  34.1 (0.7) |  |  12.3 (0.5) |  13.4 (0.5) |  15.6 (0.5) |  17 (0.5) |  17.8 (0.5)
100 |  |  21.4 (0.7) |  23.2 (0.9) |  24.4 (0.8) |  28.9 (1.1) |  34.5 (1.1) |  |  8.5 (0.3) |  9.8 (0.4) |  9.8 (0.5) |  11.6 (0.6) |  12.3 (0.6)
200 |  |  18.2 (0.8) |  17.4 (0.8) |  18.9 (0.8) |  22.7 (1) |  23.7 (1) |  |  7.7 (0.3) |  8.3 (0.4) |  7.8 (0.4) |  8.1 (0.4) |  7.9 (0.4)
n |  | $p^{*}=15;SNR=1.25;\bar{\sigma}=9.04$ |  | $p^{*}=15;SNR=5;\bar{\sigma}=4.52$
50 |  |  50.5 (1) |  58.1 (1.1) |  59.6 (1) |  61.8 (1) |  63.7 (1) |  |  28.3 (0.5) |  32.7 (0.5) |  34.8 (0.9) |  36.9 (1) |  39.9 (1.2)
100 |  |  52 (0.9) |  61.1 (1) |  68.7 (1.3) |  75.4 (1.3) |  80.8 (1.4) |  |  25.9 (0.6) |  31.6 (0.5) |  38 (0.8) |  42.9 (0.7) |  47 (0.8)
200 |  |  50.8 (1) |  60.9 (1) |  68.2 (1.1) |  75.7 (1.2) |  83.2 (1.5) |  |  23.2 (0.5) |  25.7 (0.7) |  29.6 (0.7) |  33.1 (0.7) |  36.6 (0.8)
n |  | $p^{*}=50;SNR=1.25;\bar{\sigma}=16.52$ |  | $p^{*}=50;SNR=5;\bar{\sigma}=8.26$
50 |  |  102.7 (1.9) |  109.4 (1.5) |  114.8 (1.4) |  117.3 (1.7) |  119.6 (1.5) |  |  63.9 (1.6) |  71.4 (2.1) |  73.5 (2.4) |  80.5 (3.1) |  86 (2.8)
100 |  |  113.4 (1.7) |  135.8 (2.5) |  150.8 (2.2) |  158.2 (2.3) |  161.4 (2.3) |  |  60.2 (0.8) |  75.3 (1.3) |  89 (1.9) |  101.9 (2.8) |  116.9 (3.8)
200 |  |  117.2 (1.4) |  144.7 (1.7) |  163.3 (2) |  186.8 (2.6) |  206.3 (3.2) |  |  61.4 (0.7) |  77 (0.8) |  90 (0.7) |  108.1 (1.1) |  119.9 (1.6)

We also evaluated the results for more limited data settings—fixing $n$ and
$p$, but varying $p^{*}$ and SNR across more methods of tuning parameter
selection, including AIC, BIC, ERIC with $\nu=0.5$, and GCV. For AIC and BIC,
fixed $\hat{\sigma}^{2}$ was calculated as the MSE of the model chosen by CV
1SE whereas ERIC was calculated assuming unknown variance for the likelihood.

Figures 7 and 8 give the average false negatives and average prediction bias
for the simulated data provided in the main document’s Figure (5). For larger
sample sizes, the number of false negatives is close to zero for most $p^{*}$,
but for $n=100$, as noted in the main document, average false negatives
increase with $p^{*}.$

Figure 7: Average false negatives from 100 replications of simulated data with
$n=100,n=500,n=1000$ and $p=100$ with independent predictors for both SNR
$=1.25$ and SNR $=5$. Thin dotted lines represent the mean $\pm$ one standard
error. Figure 8: Average prediction bias from 100 replications of simulated
data with $n=100,n=500,n=1000$ and $p=100$ with independent predictors for
both SNR $=1.25$ and SNR $=5$. Thin dotted lines represent the mean $\pm$ one
standard error.

Table 25 gives the average false negatives from the setting considered in the
main document: $n=p=100$. Figure 9 gives the average false positives and
prediction bias for data with $n=100$ and $p=800$. Table 10 gives average
false positives and prediction bias for $n=1000$ and $p=100$. Tables 26, 27,
and 28 give the false discovery rates, prediction bias, and false negatives
for this last setting.

Table 25: Average false negatives for 100 replications of simulated data with $n=100$. Standard errors given in parentheses. | $p=100$ | $p=800$
---|---|---
| SNR = 1.25 | SNR = 5 | SNR = 1.25 | SNR = 5
|  $p^{*}=5$ |  $p^{*}=33$ |  $p^{*}=5$ |  $p^{*}=33$ |  $p^{*}=5$ |  $p^{*}=33$ |  $p^{*}=5$ |  $p^{*}=33$
CV Min |  0.22 (0.05) |  15.57 (0.67) |  0 (0) |  4.05 (0.28) |  0.69 (0.09) |  28.05 (0.49) |  0.02 (0.01) |  22.24 (0.62)
CV 1SE |  0.71 (0.09) |  25.67 (0.74) |  0.01 (0.01) |  7.71 (0.46) |  1.45 (0.15) |  31.54 (0.3) |  0.04 (0.02) |  29.41 (0.46)
Mod Min |  0.48 (0.07) |  20.78 (0.97) |  0.01 (0.01) |  4.75 (0.39) |  1.11 (0.12) |  28.87 (0.55) |  0.03 (0.02) |  24.36 (0.75)
Mod 1SE |  1.4 (0.13) |  28.42 (0.7) |  0.09 (0.03) |  8.74 (0.57) |  1.96 (0.16) |  31.42 (0.36) |  0.14 (0.04) |  29.52 (0.5)
AR2 Min |  0.48 (0.08) |  11.91 (0.82) |  0.01 (0.01) |  4.26 (0.4) |  1.02 (0.11) |  26.34 (0.48) |  0.05 (0.03) |  22.83 (0.6)
AR2 1SE |  1.33 (0.12) |  21.41 (0.82) |  0.11 (0.03) |  8.78 (0.56) |  1.93 (0.14) |  30.15 (0.31) |  0.17 (0.04) |  28.42 (0.43)
R2 Cut |  0.08 (0.03) |  4.47 (0.17) |  0 (0) |  3.01 (0.16) |  0.37 (0.05) |  23.04 (0.24) |  0.02 (0.01) |  19.21 (0.27)
AIC |  0.29 (0.05) |  20.28 (0.76) |  0 (0) |  4.19 (0.35) |  0.94 (0.1) |  29.86 (0.34) |  0.02 (0.01) |  26.44 (0.48)
BIC |  0.73 (0.1) |  28.39 (0.65) |  0.01 (0.01) |  7.71 (0.56) |  1.54 (0.14) |  31.84 (0.27) |  0.06 (0.02) |  30.34 (0.43)
ERIC |  1.96 (0.16) |  32.96 (0.03) |  0.07 (0.03) |  31.91 (0.55) |  0.33 (0.05) |  20.14 (0.27) |  0.02 (0.01) |  15.35 (0.26)
GCV |  0.14 (0.03) |  12.65 (0.68) |  0 (0) |  3.67 (0.27) |  0.33 (0.05) |  20.14 (0.27) |  0.02 (0.01) |  15.35 (0.26)
Relaxed Min |  0.68 (0.09) |  19.14 (0.81) |  0.09 (0.03) |  5.64 (0.48) |  1.24 (0.12) |  28.2 (0.48) |  0.18 (0.05) |  21.11 (0.69)
Relaxed 1SE |  1.27 (0.12) |  25.67 (0.76) |  0.13 (0.04) |  9.49 (0.62) |  1.91 (0.14) |  31.71 (0.3) |  0.24 (0.05) |  26.48 (0.67)
Figure 9: Average false positives and prediction error plots from 100 replications of simulated data with $n=100$ and $p=800$ for both SNR $=0.5$ and SNR $=2$. Thin dotted lines represent the mean $\pm$ one standard error. Figure 10: Average false positives and prediction error plots from 100 replications of simulated data with $n=1000$ and $p=100$ for both SNR $=0.5$ and SNR $=2$. Thin dotted lines represent the mean $\pm$ one standard error. Table 26: Average false discovery rate for 100 replications of simulated data with $n=1000$. Standard errors given in parentheses. | SNR = 1.25 | SNR = 5
---|---|---
| $p^{*}=5$ | $p^{*}=33$ | $p^{*}=5$ | $p^{*}=33$
CV Min | 0.68 (0.01) | 0.49 (0) | 0.69 (0.01) | 0.49 (0.01)
CV 1SE | 0.06 (0.01) | 0.24 (0.01) | 0.06 (0.01) | 0.24 (0.01)
Mod Min | 0.37 (0.03) | 0.36 (0.01) | 0.26 (0.03) | 0.32 (0.01)
Mod 1SE | 0 (0) | 0.1 (0.01) | 0 (0) | 0.05 (0)
AR2 Min | 0.38 (0.03) | 0.38 (0.01) | 0.27 (0.03) | 0.32 (0.01)
AR2 1SE | 0 (0) | 0.1 (0.01) | 0 (0) | 0.03 (0)
R2 Cut | 0 (0) | 0.09 (0.01) | 0 (0) | 0.05 (0.01)
AIC | 0.68 (0.01) | 0.5 (0.01) | 0.69 (0.01) | 0.49 (0.01)
BIC | 0.2 (0.01) | 0.22 (0.01) | 0.2 (0.02) | 0.22 (0.01)
ERIC | 0.07 (0.01) | 0.11 (0.01) | 0.08 (0.01) | 0.12 (0.01)
GCV | 0.7 (0.01) | 0.5 (0.01) | 0.73 (0.01) | 0.5 (0.01)
Relaxed Min | 0.04 (0.01) | 0.1 (0.01) | 0.06 (0.02) | 0.02 (0)
Relaxed 1SE | 0 (0) | 0.06 (0.01) | 0 (0) | 0.01 (0)
Table 27: Average prediction bias for 100 replications of simulated data with $n=1000$. Standard errors given in parentheses. | SNR = 1.25 | SNR = 5
---|---|---
| $p^{*}=5$ | $p^{*}=33$ | $p^{*}=5$ | $p^{*}=33$
CV Min | 23.87 (0.67) | 111.79 (1.23) | 12.67 (0.34) | 56.79 (0.64)
CV 1SE | 37.85 (0.96) | 138.02 (1.5) | 20.31 (0.56) | 70.56 (0.77)