# The Beta-Mixture Shrinkage Prior for Sparse Covariances with Posterior
Minimax Rates111The first and second authors contributed equally to this work.

Kyoungjae Lee  Department of Statistics, Inha University  Seongil Jo
Department of Statistics, Inha University  Jaeyong Lee  Department of
Statistics, Seoul National University

###### Abstract

Statistical inference for sparse covariance matrices is crucial to reveal
dependence structure of large multivariate data sets, but lacks scalable and
theoretically supported Bayesian methods. In this paper, we propose beta-
mixture shrinkage prior, computationally more efficient than the spike and
slab prior, for sparse covariance matrices and establish its minimax
optimality in high-dimensional settings. The proposed prior consists of beta-
mixture shrinkage and gamma priors for off-diagonal and diagonal entries,
respectively. To ensure positive definiteness of the resulting covariance
matrix, we further restrict the support of the prior to a subspace of positive
definite matrices. We obtain the posterior convergence rate of the induced
posterior under the Frobenius norm and establish a minimax lower bound for
sparse covariance matrices. The class of sparse covariance matrices for the
minimax lower bound considered in this paper is controlled by the number of
nonzero off-diagonal elements and has more intuitive appeal than those
appeared in the literature. The obtained posterior convergence rate coincides
with the minimax lower bound unless the true covariance matrix is extremely
sparse. In the simulation study, we show that the proposed method is
computationally more efficient than competitors, while achieving comparable
performance. Advantages of the shrinkage prior are demonstrated based on two
real data sets.

##  1 Introduction

Suppose $X_{1},\ldots,X_{n}$ are independent $p$-dimensional random vectors
from $N_{p}(0,\Sigma)$, the $p$-dimensional normal distribution with mean
$0\in\mathbb{R}^{p}$ and covariance $\Sigma\in\mathbb{R}^{p\times p}$. The
covariance matrix of a random vector is a fundamental parameter that expresses
the marginal dependence structure of $X$. It is a basis for many multivariate
statistical methods such as principal component analysis, factor analysis,
discriminant analysis, and linear regression, to name just a few. In this
paper, we consider the Bayesian inference of covariance matrices when the
dimension of observations, $p$, tends to infinity as the sample size, $n$,
gets larger. We assume that most of the off-diagonal entries of a covariance
matrix are zero, i.e., only few pairs of variables have significant marginal
dependences. We propose the beta-mixture shrinkage prior for sparse covariance
matrix. The proposed methodology is computationally fast and attains the
minimax posterior convergence rate under the Frobenius norm when the true
covariance is not extremely sparse.

There are vast and rich frequentist literature on high-dimensional sparse
covariance estimation. Various thresholding estimators (Bickel and Levina,
2008; Rothman et al., 2009; Cai and Liu, 2011; Cai and Zhou, 2012) and lasso-
type procedures (Bien and Tibshirani, 2011) have been proposed for
simultaneously learning marginal dependence structures and estimating
covariance matrices. Among them, Cai and Liu (2011) proposed an adaptive
thresholding estimator and proved that it achieves the minimax convergence
rate for sparse covariance matrices by showing that the obtained rate
coincides with the minimax lower bound obtained in Cai and Zhou (2012).

For Bayesian inference of sparse covariance, the $G$-inverse Wishart prior
(Silva and Ghahramani, 2009) is often used. The normalizing constant of the
$G$-inverse Wishart prior is analytically intractable and needs the Monte
Carlo method for its evaluation, which makes the posterior computation
infeasible even when $p$ is moderately large. Khare and Rajaratnam (2011)
introduced a broad class of priors including $G$-inverse Wishart prior as a
special case. They provided a blocked Gibbs sampler to obtain samples from the
resulting posterior, but the priors were only applicable to decomposable
covariance graph models. Wang (2015) proposed stochastic search structure
learning (SSSL). He placed the spike-and-slab prior for the off-diagonal
elements of $\Sigma$ and put a modified version of the product of Bernoulli
prior on the sparsity structure of the covariance. The modification of the
product of Bernoulli priors allows one to avoid the intense normalizing
constant computation, but the natural interpretation of the Bernoulli prior is
lost.

In addition to the computational difficulties of the posterior of the sparse
covariance, the Bayesian literature lacks the asymptotic properties of the
posteriors. Lee and Lee (2018) showed that the inverse Wishart prior achieves
the minimax posterior convergence rate for a unstructured covariance matrix
under the spectral norm. The posterior convergence rate under the Frobenius
norm was also derived. However, they focused only on unstructured covariance
matrices and used the inverse Wishart prior, which is not suitable for sparse
covariance matrices. Neither Khare and Rajaratnam (2011) nor Wang (2015)
established the asymptotic properties of the posteriors for sparse
covariances. Up to our knowledge, asymptotic properties of the posteriors
induced by the priors for sparse covariance matrices have not been
investigated yet.

In this paper, to fill the gap in the literature, we develop a scalable
Bayesian inference for sparse covariance matrices supported by theoretical
properties of posteriors. We propose a continuous shrinkage prior for the
sparse covariance matrices. Especially, the beta-mixture prior and the gamma
prior are used for off-diagonal and diagonal entries of covariance matrices,
respectively. To ensure the positive definiteness of the resulting covariance
matrix, we further restrict the prior on a class of positive definite
matrices. A blocked Gibbs sampler is derived to obtain posterior samples.

The advantage of the proposed method are as follows. First, this is the first
Bayesian method for sparse covariance matrices with optimal minimax rate
unless the true covariance is extremely sparse. We show the posterior
convergence rate of the proposed prior under the Frobenius norm (Theorem 3.1).
We also derive a lower bound of the minimax rate for sparse covariance
matrices (Theorem 3.2) with restriction only on the total number of nonzero
off-diagonal entries, which differentiates the obtained result from the
results in the literature assuming column-wise sparsity and has more intuitive
appeal. These results show that the obtained posterior convergence rate is the
minimax rate except extremely sparse cases. Second, the proposed method is
computationally efficient. We compare computational efficiency of the
shrinkage prior and the SSSL (Wang, 2015), and find that the proposed
shrinkage prior has almost twice as many effective sample size as the SSSL.
This implies that the posterior sampling of the shrinkage prior exhibits
faster mixing than that of the SSSL.

The paper is organized as follows. In Section 2, we describe the model, prior
and the posterior computation. In Section 3, we present the theoretical
results including the asymptotic minimaxity. The numerical studies and real
data analysis are given in Section 4. Concluding remarks are given in Section
5.

##  2 Beta-Mixture Shrinkage Prior

###  2.1 Notation

Let $a_{n}$ and $b_{n},n=1,2,\ldots$ be sequences of positive real numbers. We
denote $a_{n}=O(b_{n})$, or equivalently $a_{n}\lesssim b_{n}$, if
$a_{n}/b_{n}\leq C$ for some constant $C>0$. We denote $a_{n}\asymp b_{n}$ if
$a_{n}=O(b_{n})$ and $b_{n}=O(a_{n})$. Furthermore, we denote
$a_{n}=o(b_{n})$, or equivalently $a_{n}\ll b_{n}$, if
$a_{n}/b_{n}\longrightarrow 0$ as $n\to\infty$. Let $\mathcal{C}_{p}$ be the
set of all $p\times p$ positive definite matrices. Let $A=(a_{ij})$ be a
$p\times p$ matrix. We denote the minimum and maximum eigenvalues of $A$ by
$\lambda_{\min}(A)$ and $\lambda_{\max}(A)$, respectively. The Frobenius norm
of $A$ is defined by
$\|A\|_{F}=(\sum_{i=1}^{p}\sum_{j=1}^{p}a_{ij}^{2})^{1/2}$.

###  2.2 Prior for sparse covariances

Suppose we observe $n$ independent samples ${\bf X}_{n}=(X_{1},\ldots,X_{n})$
from the $p$-dimensional normal distribution:

$X_{i}\mid\Sigma\stackrel{{\scriptstyle
iid}}{{\sim}}N_{p}\left(0,\Sigma\right),\quad i=1,\ldots,n,$ (1)

where $\Sigma\in\mathcal{C}_{p}$. We assume that the covariance matrix
$\Sigma$ is $\ell_{0}$-sparse, i.e., most of off-diagonal entries of $\Sigma$
are zero. For Bayesian inference on $\Sigma$, we need to impose a prior
distribution on a set of covariance matrices. We first define a prior for
$p\times p$ symmetric matrices and restrict it to the space of positive
definite matrices. Let

$\displaystyle\pi^{u}(\sigma_{jk}\mid\rho_{jk})$ $\displaystyle=$
$\displaystyle N\Big{(}\sigma_{jk}\mid
0,\,\frac{\rho_{jk}}{1-\rho_{jk}}\tau_{1}^{2}\Big{)},$ (2)
$\displaystyle\pi^{u}(\rho_{jk})$ $\displaystyle=$ $\displaystyle
Beta(\rho_{jk}\mid a,b),\quad 1\leq j<k\leq p,$ (3)
$\displaystyle\pi^{u}(\sigma_{jj})$ $\displaystyle=$ $\displaystyle
Gamma(\sigma_{jj}\mid c,d),\quad j=1,\ldots,p,$ (4)

for some positive constants $\tau_{1},a,b,c,d$, where $Beta(a,b)$ is the beta
distribution with parameters $a,b>0$ and $gamma(c,d)$ is the beta distribution
with shape parameter $c$ and rate parameter $d$. The prior on symmetric matrix
with positive diagonal elements is defined as

$\displaystyle\pi^{u}(\Sigma)$ $\displaystyle=$ $\displaystyle\prod_{1\leq
j<k\leq
p}\pi^{u}(\sigma_{jk}\mid\rho_{jk})\pi^{u}(\rho_{jk})I(\sigma_{jk}=\sigma_{kj})\prod_{j=1}^{p}\pi^{u}(\sigma_{jj}),$
(5)

where “u” stands for the unconstrained prior. Note that the marginal prior on
$\sigma_{jk}$ is the half-Cauchy prior if we take $a=b=1/2$, which is one of
the most popular shrinkage priors.

Other possible choices for the shrinkage prior of $\sigma_{jk}$ are the
horseshoe prior (Carvalho et al., 2010), the lasso prior (Park and Casella,
2008), the hyperlasso prior (Griffin and Brown, 2011, 2017) and the
generalized double pareto (GDP) prior (Armagan et al., 2013). In this paper,
we will focus on the half-Cauchy prior for the off-diagonal elements and
$Gamma(1,\frac{\lambda}{2}),\lambda>0,$ and derive its theoretical properties.

Now, we propose the shrinkage prior for sparse covariance matrices by
restricting $\pi^{u}(\Sigma)$ to the subspace of positive definite matrices:

$\displaystyle{\pi}(\Sigma)$ $\displaystyle=$
$\displaystyle\frac{\pi^{u}(\Sigma)I(\Sigma\in\mathcal{U}(\tau))}{\pi^{u}(\Sigma\in\mathcal{U}(\tau))},$
(6)

where

$\displaystyle\mathcal{U}(\tau)$ $\displaystyle=$
$\displaystyle\Big{\\{}\Sigma\in\mathcal{C}_{p}:\tau^{-1}\leq\lambda_{\min}(\Sigma)\leq\lambda_{\max}(\Sigma)\leq\tau\Big{\\}}$

for some constant $\tau>1$. In this paper, we consider $\tau$ as a fixed
constant to obtain desired asymptotic properties of posteriors. However, in
practice, one can use $\tau=\infty$, which results in
$\mathcal{U}(\tau)=\mathcal{C}_{p}$. Conditions on the hyperparameters will be
specified in Section 3, while practical suggestions will be given in Section
4.

###  2.3 Comparison to the SSSL

The shrinkage prior (6) proposed in this paper and the SSSL proposed by Wang
(2015) use the gamma and exponential priors for the diagonal entries,
$\sigma_{ii},i=1,2,\ldots,p$, of the covariance, respectively. For the off-
diagonal elements, $\sigma_{jk},1\leq j\neq k\leq p$, Wang (2015) used the
continuous spike and slab prior,

$\pi^{u,W}(\sigma_{jk})=(1-\pi)N(\sigma_{jk}\mid 0,\nu_{0}^{2})+\pi
N(\sigma_{jk}\mid 0,\nu_{1}^{2})$

for some constants $0<\nu_{0}<\nu_{1}$ and $\pi\in(0,1)$, while we use the
continuous beta-mixture shrinkage prior (2) and (3).

In the spike and slab prior, the prior inclusion probability, $\pi\in(0,1)$,
reflects the prior belief whether $\sigma_{jk}$ will be zero or not. Similarly
to the beta-mixture shrinkage prior (6), Wang (2015) proposed the prior,
$\pi^{W}(\sigma_{jk})$, by restricting $\pi^{u,W}(\sigma_{jk})$ to the space
of positive definite matrices. Note that due to the unknown normalizing
constant caused by the positive definiteness constraint, $\pi\in(0,1)$ is no
longer the prior inclusion probability of the resulting prior
$\pi^{W}(\sigma_{jk})$.

The main advantages of the beta-mixture prior over the SSSL are the
theoretical guarantee and computational efficiency. The proposed prior (6)
achieves the minimax posterior convergence rate for sparse covariances under
the Frobenius norm, which will be rigorously stated in Section 3. On the other
hand, asymptotic properties of posteriors based on the SSSL have not been
investigated yet. Furthermore, based on the simulation studies in Section 4,
we found that the finite sample performance of the proposed prior is
comparable to that of the SSSL while achieving almost twice as many effective
sample size.

###  2.4 Blocked Gibbs sampler

We now provide a posterior sampling algorithm for our prior described in (6).
The algorithm is based on the blocked Gibbs sampler proposed by Wang (2015).
To describe the algorithm, as in Proposition 2 of Wang (2015), we consider the
following partition of $\Sigma$, $\bm{S}={\bf X}_{n}^{T}{\bf X}_{n}$ and
$\bm{V}=(v_{jk}^{2}),~{}v_{jk}^{2}=v_{kj}^{2}=\rho_{jk}\tau_{1}^{2}/(1-\rho_{jk})$
for $j<k$ and $v_{jk}^{2}=0$ for $j=k$:

$\Sigma=\left(\begin{array}[]{cc}\Sigma_{11}&\bm{\sigma}_{12}\\\
\bm{\sigma}_{12}^{T}&\sigma_{22}\end{array}\right),\quad\bm{S}=\left(\begin{array}[]{cc}\bm{S}_{11}&\bm{s}_{12}\\\
\bm{s}_{12}^{T}&s_{22}\end{array}\right),\quad\bm{V}=\left(\begin{array}[]{cc}\bm{V}_{11}&\bm{v}_{12}\\\
\bm{v}_{12}^{T}&0\end{array}\right),$ (7)

where $\Sigma_{11},\bm{S}_{11},\bm{V}_{11}\in\mathcal{C}_{p-1}$,
$\bm{\sigma}_{12},\bm{s}_{12},\bm{v}_{12}\in\mathbb{R}^{(p-1)\times 1}$ and
$\sigma_{22},s_{22}>0$, and the change of variables:

$\left(\bm{\sigma}_{12},\sigma_{22}\right)\rightarrow\left(\bm{u}=\bm{\sigma}_{12},v=\sigma_{22}-\bm{\sigma}_{12}^{T}\Sigma_{11}^{-1}\bm{\sigma}_{12}\right).$
(8)

The posterior samples then are generated by iterating the following steps (for
details, see the Appendix D):

  * •

For $\bm{u}$,

$\displaystyle\bm{u}\mid{\tt others}$ $\displaystyle\sim$ $\displaystyle\
N_{p-1}\left[\left\\{\bm{B}+\mbox{diag}(\bm{v}_{12}^{-1})\right\\}^{-1}\bm{w},\left\\{\bm{B}+\mbox{diag}(\bm{v}_{12}^{-1})\right\\}^{-1}\right],$

where
$\bm{B}=\Sigma_{11}^{-1}\bm{S}_{11}\Sigma_{11}^{-1}v^{-1}+\lambda\Sigma_{11}^{-1}$
and $\bm{w}=\Sigma_{11}^{-1}\bm{s}_{12}v^{-1}$.

  * •

For $v$,

$\displaystyle v\mid{\tt others}$ $\displaystyle\sim$ $\displaystyle
GIG\left(1-n/2,\,\,\lambda,\,\,\bm{u}^{T}\Sigma_{11}^{-1}\bm{S}_{11}\Sigma_{11}^{-1}\bm{u}-2s_{12}^{T}\Sigma_{11}^{-1}\bm{u}+s_{22}\right),$

where $GIG(q,a,b)$ is the generalized inverse Gaussian distribution with the
probability density function $f(x)\propto x^{q-1}e^{-(ax+b/x)/2}I(x>0)$.

  * •

For $\rho_{jk}=1-1/(1+\phi_{jk})$,

$\displaystyle\psi_{jk}\mid{\tt others}$ $\displaystyle\sim$ $\displaystyle
Gamma\left(a+b,\phi_{jk}+1\right),$ $\displaystyle\phi_{jk}\mid{\tt others}$
$\displaystyle\sim$ $\displaystyle
GIG\left(a-1/2,2\psi_{jk},\sigma_{jk}^{2}/\tau_{1}^{2}\right),$

where $Gamma(a,b)$ is the Gamma distribution the shape parameter $a$ and the
rate parameter $b$.

##  3 Posterior convergence rate

In this section, we show that the beta-mixture shrinkage prior achieves the
minimax rate under the Frobenius norm when $p=O(s_{0})$ where $s_{0}$ is an
upper bound for nonzero off-diagonal elements of the covariance matrix. Let
$\Sigma_{0}$ be the true covariance matrix. For a given integer
$0<s_{0}<p(p-1)$ and a real number $\tau_{0}>1$, we define the parameter space

$\displaystyle\mathcal{U}(s_{0},\tau_{0})$ $\displaystyle=$
$\displaystyle\Big{\\{}\Sigma\in\mathcal{C}_{p}:|s(\Sigma)|\leq
s_{0},\,\,\tau_{0}^{-1}\leq\lambda_{\min}(\Sigma)\leq\lambda_{\max}(\Sigma)\leq\tau_{0}\Big{\\}},$
(9)

where $|s(\Sigma)|$ is the number of nonzero off-diagonal entries in $\Sigma$.
To attain the desired asymptotic properties of posteriors, we introduce the
following conditions.

  1. (A1)

$\Sigma_{0}\in\mathcal{U}(s_{0},\tau_{0})$ for some integer $0<s_{0}<p(p-1)$
and constant $\tau_{0}>1$.

  2. (A2)

$p\asymp n^{\beta}$ for some $0<\beta<1$.

  3. (A3)

The hyperparameters satisfy $\tau\geq\max(3,\tau_{0})$, $\tau=O(1)$,
$\lambda=O(1)$, $a=b=1/2$ and $\tau_{1}^{2}\asymp 1/(np^{4})$.

Condition (A1) implies that the true covariance matrix is sparse and has
eigenvalues bounded above as well as away from zero. The integer $s_{0}$
controls the sparsity of the true covariance matrix. The bounded eigenvalue
condition has been commonly used in high-dimensional matrix estimation
literature including Banerjee and Ghosal (2015), Gao and Zhou (2015) and Lee
et al. (2019). In this paper, the lower bound for the minimum eigenvalue is
mainly used in Lemma 5.5 to convert $\|\Sigma_{0}^{-1}-\Sigma^{-1}\|_{F}$ to
$\|\Sigma_{0}-\Sigma\|_{F}$, while the upper bound for the maximum eigenvalue
is required to ensure $\Sigma_{0}\in\mathcal{U}(\tau)$.

Condition (A2) says that the number of variables $p$ grows to infinity as
$n\to\infty$, but at a slower rate than $n$. In the literature, Lam and Fan
(2009) used a similar condition to obtain the convergence rates of penalty
estimators for sparse covariance matrices, and Liu and Martin (2019) used the
same condition to obtain the posterior convergence rate for sparse precision
matrices. This condition is inevitable to obtain the posterior convergence
rate under the Frobenius norm if one uses the traditional techniques in Ghosal
et al. (2000) which we use in this paper. In the seminal work of Ghosal et al.
(2000), they provided a sufficient condition for proving the posterior
convergence rate for densities under the Hellinger metric, which is equivalent
to the Frobenius norm for covariance matrices under the bounded eigenvalue
condition (A1). When the posterior is intractable, this is the standard way to
find the posterior convergence rate. One of necessary conditions in this
result is that the posterior convergence rate should converge to zero as
$n\to\infty$. Since the diagonal elements of the covariance are all nonzero,
this condition requires the number of diagonal elements $p=o(n)$. This can be
also seen from the minimax lower bound result, Theorem 3.2. Thus, if one use
the techniques in Ghosal et al. (2000), condition (A2) is required to prove
Theorem 3.1.

Condition (A3) gives a sufficient condition for hyperparameters to obtain the
desired theoretical property of posteriors. The choice $a=b=1/2$ implies that
we use the half-Cauchy prior for the off-diagonal entries. Note that
$\tau_{1}^{2}$ is the global shrinkage parameter in (2), thus condition (A3)
means that the global shrinkage parameter should be sufficiently small. This
corresponds to assume a sufficiently small inclusion probability in spike and
slab priors. See Lee et al. (2019) and Martin et al. (2017).

For the asymptotic minimax rate of the shrinkage prior, we first show an upper
bound of the minimax rate: Theorem 3.1 shows the posterior convergence rate of
the proposed prior under the Frobenius norm.

###### Theorem 3.1

Under model (1) and prior (6), assume conditions (A1)–(A3) hold. If
$(p+s_{0})\log p=o(n)$, as $n\to\infty$

$\displaystyle\pi\Big{\\{}\|\Sigma-\Sigma_{0}\|_{F}^{2}\geq
M\frac{(p+s_{0})\log p}{n}\mid{\bf X}_{n}\Big{\\}}$
$\displaystyle\longrightarrow$ $\displaystyle 0~{}\text{in
$\mathbb{P}_{0}$-probability}$

for some large constant $M>0$.

The condition $(p+s_{0})\log p=o(n)$ relates $p$ and $n$ to $s_{0}$, the
number of nonzero off-diagonal elements of the true covariance. Banerjee and
Ghosal (2015) and Liu and Martin (2019) used the same condition to obtain the
posterior convergence rate for sparse precision matrices.

The next theorem shows a minimax lower bound for covariance matrices, which
coincides with the posterior convergence rate in Theorem 3.1 when
$p=O(s_{0})$. In general, $p=O(s_{0})$ holds unless $\Sigma_{0}$ is extremely
sparse and it holds when the true covariance matrix has an autoregressive
structure with order $1$, $AR(1)$.

###### Theorem 3.2

For given positive integer $s_{0}$ and real number $\tau_{0}>1$, assume model
(1) with $\Sigma_{0}\in\mathcal{U}(s_{0},\tau_{0})$. If $s_{0}^{2}(\log
p)^{3}=O(p^{2}n)$ and $s_{0}^{2}=O(p^{3-\epsilon})$ for some small constant
$\epsilon>0$,

$\displaystyle\inf_{\hat{\Sigma}}\sup_{\Sigma_{0}\in\mathcal{U}(s_{0},\tau_{0})}{\mathbb{E}}_{0}\|\hat{\Sigma}-\Sigma_{0}\|_{F}^{2}$
$\displaystyle\gtrsim$ $\displaystyle\frac{s_{0}\log
p}{n}\,I(s_{0}>3p)+\frac{p}{n}.$

Cai and Zhou (2012) proved that a modified thresholding estimator attains the
minimax rate for sparse covariance matrices under the class of Bregman
divergences. They assumed sparsity for each column of the covariance matrix,
which means that each column of $\Sigma_{0}$ has nonzero entries less than
$s_{0}^{\prime}$. On the other hand, we assume that the nonzero entries of the
$\Sigma_{0}$ is less than $s_{0}$. Thus, our sparsity assumption on
$\Sigma_{0}$ is much weaker than that of Cai and Zhou (2012). Up to our
knowledge, this is the first minimax lower bound result for sparse covariance
matrices with restriction only on the total number of nonzero off-diagonal
entries. To establish the minimax rate, they assumed that
$(s_{0}^{\prime})^{2}(\log p)^{3}=O(n)$, which is roughly equivalent to
$s_{0}^{2}(\log p)^{3}=O(p^{2}n)$ in our notation. It is easy to see that the
minimax rate in Cai and Zhou (2012) coincides with the rate of the lower bound
in Theorem 3.2. Hence, Theorems 3.1 and 3.2 imply that, even though we
consider a larger parameter space than Cai and Zhou (2012), the minimax rate
is still unchanged and the proposed prior attains it.

##  4 Simulation Study

###  4.1 Synthetic data

To assess the performance of our sparse covariance estimator, we carry out a
simulation study using synthetic datasets generated from Gaussian
distributions with zero means and the following two covariance structures
$\Sigma=(\sigma_{jk})$.

  * C1.

Sparse covariance that mimics daily currency exchange rate return structure
(Wang, 2015):

$\left(\begin{array}[]{rrrrrrrrrrrrr}0.239&0.117&&&&&&0.031&&&&&\\\
0.117&1.554&&&&&&&&&&&\\\ &&0.362&0.002&&&&&&&&\\\
&&0.002&0.199&0.094&&&&&&&\\\ &&&0.094&0.349&&&&&&&-0.036\\\
&&&&&0.295&-0.229&0.002&&&&\\\ &&&&&-0.229&0.715&&&&&\\\
0.031&&&&&0.002&&0.164&0.112&-0.028&-0.008&\\\
&&&&&&&0.112&0.518&-0.193&-0.090&\\\ &&&&&&&-0.028&-0.193&0.379&0.167&\\\
&&&&&&&-0.008&-0.090&0.167&0.159&\\\
&&&&-0.036&&&&&&&0.207\end{array}\right);$

  * C2.

Random structure: $\sigma_{jj}\sim Gamma(1,1),~{}j=1,2,\ldots,p$. Sparse off-
diagonal positions ($20\%$) are randomly selected. Nonzero off-diagonal
elements are generated from $Unif(0,\mu),~{}j\neq k$.

In case C1, following Wang (2015) we take $p=12$ and $n=250$, and in case C2,
we consider $p=50$ and $n=50,100$ and choose the range of parameter $\mu$ as
$\\{0.02,0.1,0.5,1\\}$ following Castillo and Roquain (2020).

We compute the root mean squared error (rmse),
$\|\hat{\Sigma}-\Sigma\|_{F}/p$, and the maximum norm (mnorm),
$\mbox{max}_{jk}|\hat{\sigma}_{jk}-\sigma_{jk}|$, and report the average rmse
and mnorm, and their standard errors over 50 replications. As competing
methods, we consider the sample covariance (SampCov) and the SSSL estimator
(Wang, 2015). We generate 5000 posterior samples after 5000 burn-in for the
proposed shrinkage model and the SSSL.

Tables 1 and 2 show the rmse and mnorm values for simulation models with two
covariance structures, C1 and C2, respectively. Figure 1 renders the effective
sample sizes (ESS), equivalent sample sizes when the independent sampling is
done, of the posterior sampling for the shrinkage prior and the SSSL in case
C1. From Tables 1 and 2, we can see that the proposed shrinkage and the SSSL
estimators are better than the sample covariance in all cases, and that the
proposed shrinkage estimator performs better or at least comparable to the
SSSL estimator, while the posterior sampling algorithm of the shrinkage prior
is more efficient than that of the SSSL in terms of ESS (Figure 1).
Additionally, the posterior sampling of the shrinkage prior takes about 171
seconds per 1,000 ESS, but that of the SSSL takes 789 seconds with iMac Pro
with 3 GHz 10-Core Intel Xeon processor. Finally, Table 2 shows that the
continuous shrinkage prior produces more accurate estimates when signals are
small, while the spike and slab prior can capture large signals more
efficiently.

Table 1: rmse and mnorm under the covariance structure C1. | Proposed | SSSL | SampCov
---|---|---|---
rmse | 0.020 (0.003) | 0.020 (0.003) | 0.028 (0.004)
mnorm | 0.114 (0.051) | 0.120 (0.050) | 0.120 (0.054)
Table 2: rmse and mnorm under the covariance structure C2. |  | Measure | Proposed | SSSL | SampCov
---|---|---|---|---|---
$n=100$ | $\mu=0.02$ | rmse | 0.034 (0.008) | 0.038 (0.007) | 0.109 (0.007)
$(p=50)$ |  | mnorm | 0.950 (0.407) | 0.994 (0.405) | 1.137 (0.409)
| $\mu=0.1$ | rmse | 0.063 (0.004) | 0.065 (0.005) | 0.127 (0.007)
|  | mnorm | 0.985 (0.409) | 0.998 (0.418) | 1.176 (0.414)
| $\mu=0.5$ | rmse | 0.287 (0.002) | 0.281 (0.004) | 0.271 (0.014)
|  | mnorm | 1.371(0.560) | 1.470 (0.530) | 1.609 (0.560)
| $\mu=1$ | rmse | 0.561 (0.003) | 0.544 (0.006) | 0.467 (0.023)
|  | mnorm | 1.789 (0.397) | 1.800 (0.370) | 2.117 (0.403)
$n=50$ | $\mu=0.02$ | rmse | 0.043 (0.009) | 0.049 (0.009) | 0.153 (0.010)
$(p=50)$ |  | mnorm | 1.238 (0.460) | 1.265 (0.472) | 1.526 (0.474)
| $\mu=0.1$ | rmse | 0.071 (0.006) | 0.075 (0.007) | 0.179 (0.010)
|  | mnorm | 1.263 (0.487) | 1.306 (0.490) | 1.583 (0.509)
| $\mu=0.5$ | rmse | 0.292 (0.003) | 0.285 (0.003) | 0.382 (0.019)
|  | mnorm | 1.564 (0.397) | 1.580 (0.361) | 2.056 (0.461)
| $\mu=1$ | rmse | 0.582 (0.003) | 0.558 (0.006) | 0.663 (0.034)
|  | mnorm | 2.278 (0.519) | 2.203 (0.534) | 2.991 (0.530)
Figure 1: Effective sample size of the posterior samples of the shrinkage
prior and the SSSL in case C1.

###  4.2 Real data application

In this section, we consider two datasets to assess the performance of the
shrinkage prior for linear discriminant analysis (LDA) classification
(Anderson, 2003). The first is a colon cancer data, described in Alon et al.
(1999), Fisher and Sun (2011) and Touloumis (2015). The data set can be
obtained from http://genomics-pubs.princeton.edu/oncology/affydata. It
contains expression level measurements of 2000 genes on 40 normal and 22 colon
tumor tissues.

As the second example, we consider the leukemia data (Golub et al., 1999; Zhu
and Hastie, 2004; Guo et al., 2007) which consists of 7128 gene expression
measurements on 72 leukemia patients, 47 “ALL” and 25 “AML”. This data set is
available at http://web.stanford.edu/~hastie.

Following Rothman et al. (2008), we select $p$ most significant genes with the
two-sample $t$-statistic for $p=50$ (colon data) and $p=71$ (leukemia data)
and then apply LDA to the datasets for classifying each observation in each
data set into two groups. The LDA rule for an observation $X$ is given as

$\delta_{j}(X)=\operatorname*{argmax}_{j}\left\\{X^{\top}\hat{\Sigma}^{-1}\hat{\bm{\mu}}_{j}-\frac{1}{2}\hat{\bm{\mu}}_{j}^{\top}\hat{\Sigma}^{-1}\hat{\bm{\mu}}_{j}+\log\hat{\omega}_{j}\right\\},~{}~{}j=1,2,$

where $\hat{\omega}_{j}$ is the proportion of class $j$ and
$\hat{\bm{\mu}}_{j}$ is the sample mean for class $j$.

Table 3 shows leave-one-out cross validation (LOOCV) error rates for
misclassified observations. From the result, we can see that our estimator
outperforms two competitors, SampCov and SSSL for both data sets.

Table 3: Classification error for colon and leukemia data. Dataset | Proposed | SSSL | SampCov
---|---|---|---
Colon | 0.097 | 0.113 | 0.333
Leukemia | 0.014 | 0.028 | 0.486

##  5 Discussion

In this paper, we propose a theoretically supported shrinkage prior for sparse
covariance matrices. We prove that the proposed shrinkage prior achieves the
minimax posterior convergence rate under the Frobenius norm in high-
dimensional settings. The shrinkage prior performs better than or comparable
to the SSSL method in the simulation studies, while is computationally more
efficient than the SSSL. In our simulation study, the proposed shrinkage prior
is 4 times faster than the SSSL in terms of the computation time per ESS. Two
real data examples, colon and leukemia data, show the benefit of the LDA
classification based on the proposed Bayesian method.

## Acknowledgements

This work was supported by the National Research Foundation of Korea (NRF)
grant funded by the Korea government(MSIT) (No. 2020R1A4A1018207). Seongil Jo
was supported by Basic Science Research Program through the National Research
Foundation of Korea (NRF) funded by the Ministry of Education
(NRF-2017R1D1A3B03035235).

## Appendix A: Proof of Theorem 3.1

Let

$\displaystyle K(f_{\Sigma_{0}},f_{\Sigma})$ $\displaystyle:=$
$\displaystyle\int
f_{\Sigma_{0}}(x)\log\frac{f_{\Sigma_{0}}(x)}{f_{\Sigma}(x)}dx,$
$\displaystyle V(f_{\Sigma_{0}},f_{\Sigma})$ $\displaystyle:=$
$\displaystyle\int
f_{\Sigma_{0}}(x)\left(\log\frac{f_{\Sigma_{0}}(x)}{f_{\Sigma}(x)}\right)^{2}dx,$

where $f_{\Sigma}$ is the probability density function of $N_{p}(0,\Sigma)$
based on $n$ random samples $X_{1},\ldots,X_{n}$. For a given $\epsilon>0$,
let

$\displaystyle B_{\epsilon}$ $\displaystyle:=$
$\displaystyle\Big{\\{}f_{\Sigma}:\,\,\Sigma\in\mathcal{C}_{p},\,\,K(f_{\Sigma_{0}},f_{\Sigma})<\epsilon^{2},\,\,V(f_{\Sigma_{0}},f_{\Sigma})<\epsilon^{2}\Big{\\}}.$

To prove Theorem 3.1, we apply Theorem 2.1 in Ghosal et al. (2000).

######  Lemma 5.1 (A version of Theorem 2.1 in Ghosal et al. (2000))

Let $d$ be the Hellinger metric, and let
$\mathcal{P}=\\{f_{\Sigma}:\Sigma\in\mathcal{C}_{p}\\}$. Consider a sieve
$\mathcal{P}_{n}\subset\mathcal{P}$ and a sequence $\epsilon_{n}$ with
$\epsilon_{n}\to 0$ and $n\epsilon_{n}^{2}\to\infty$. If, for some constants
$C_{1}$ and $C_{2}>0$,

$\displaystyle\log D(\epsilon_{n},\mathcal{P}_{n},d)$ $\displaystyle\leq$
$\displaystyle C_{1}n\epsilon_{n}^{2}$ (10)
$\displaystyle\pi(\mathcal{P}_{n}^{c})$ $\displaystyle\leq$
$\displaystyle\exp\big{\\{}-(C_{2}+4)n\epsilon_{n}^{2}\big{\\}},$ (11)
$\displaystyle\pi(B_{\epsilon_{n}})$ $\displaystyle\geq$
$\displaystyle\exp(-C_{2}n\epsilon_{n}^{2}),$ (12)

then for sufficiently large $M>0$, we have

$\displaystyle\pi\big{(}d(f_{\Sigma_{0}},f_{\Sigma})>M\epsilon_{n}\mid{\bf
X}_{n}\big{)}$ $\displaystyle\longrightarrow$ $\displaystyle 0$

as $n\to\infty$ in $\mathbb{P}_{\Sigma_{0}}$-probability, where
$D(\epsilon,\mathcal{P}_{n},d)$ is the $\epsilon$-packing number of
$\mathcal{P}_{n}$ with respect to the distance $d$.

Based on the above lemma, it suffices to show that conditions (10)-(12) hold
under the assumptions in Theorem 3.1, using

$\displaystyle\epsilon_{n}$ $\displaystyle:=$
$\displaystyle\Big{(}\frac{(p+s_{0})\log p}{n}\Big{)}^{1/2}.$

For the rest, $\pi$ denotes the proposed shrinkage prior (6). With a slightly
abuse of notation, $\pi$ also denotes the prior for $f_{\Sigma}$ induced by
the shrinkage prior (6).

### A1. The upper bound of the packing number

We define

$\displaystyle\mathcal{P}_{n}$ $\displaystyle=$
$\displaystyle\Big{\\{}f_{\Sigma}:|s(\Sigma,\delta_{n})|\leq
s_{n},\,\,\tau^{-1}\leq\lambda_{\min}(\Sigma)\leq\lambda_{\max}(\Sigma)\leq\tau,\,\,\|\Sigma\|_{\max}\leq
L_{n}\Big{\\}},$ $\displaystyle\mathcal{U}(\delta_{n},s_{n},L_{n},\tau)$
$\displaystyle=$
$\displaystyle\Big{\\{}\Sigma\in\mathcal{C}_{p}:|s(\Sigma,\delta_{n})|\leq
s_{n},\,\,\tau^{-1}\leq\lambda_{\min}(\Sigma)\leq\lambda_{\max}(\Sigma)\leq\tau,\,\,\|\Sigma\|_{\max}\leq
L_{n}\Big{\\}}$

for some positive constants $\delta_{n},s_{n},L_{n}$ and $\tau$, where
$\|\Sigma\|_{\max}=\max_{ij}|\sigma_{ij}|$ for $\Sigma=(\sigma_{ij})$. Here
$\tau$ is a fixed large constant such that $\tau_{0}<\tau$. $\delta_{n},s_{n}$
and $L_{n}$ are specified in the following theorem. It gives the upper bound
of the packing number in (10).

######  Theorem 5.2 (The upper bound of the packing number)

If $\tau^{4}\leq p$, $p\asymp n^{\beta}$ for some $0<\beta<1$,
$s_{n}=c_{1}n\epsilon_{n}^{2}/\log p$, $L_{n}=c_{2}n\epsilon_{n}^{2}$ and
$\delta_{n}=\epsilon_{n}/\tau^{3}$ for some constants $c_{1}>1$ and $c_{2}>0$,
we have

$\displaystyle\log D(\epsilon_{n},\mathcal{P}_{n},d)$ $\displaystyle\leq$
$\displaystyle(12+1/\beta)c_{1}n\epsilon_{n}^{2}.$

### A2. The upper bound of the prior mass on $\mathcal{P}_{n}^{c}$

###### Lemma 5.3

If $a=b=1/2$, $\tau_{1}^{2}\asymp 1/(np^{4}\tau^{2})$ and $\tau>3$, we have

$\displaystyle\pi^{u}(\Sigma\in\mathcal{U}(\tau))>\Big{\\{}\frac{\lambda\tau}{8}\exp\big{(}-\frac{\lambda\tau}{4}-\frac{C}{\sqrt{n}}\big{)}\Big{\\}}^{p}$

for some constant $C>0$.

The following theorem gives the upper bound of the prior mass on
$\mathcal{P}_{n}^{c}$ in (11).

######  Theorem 5.4 (The upper bound of the prior mass)

If $\delta_{n}=\epsilon_{n}/\tau^{3}$, $3<\tau\leq(\log p)/\lambda$,
$a=b=1/2$, $\tau_{1}^{2}\asymp 1/(np^{4}\tau^{2})$,
$\tau^{4}\ll(p+s_{0})^{2}\log p$, we have

$\displaystyle\pi(\mathcal{P}_{n}^{c})$ $\displaystyle\leq$
$\displaystyle\exp\big{\\{}-(c_{1}-1)n\epsilon_{n}^{2}/3\big{\\}}.$

### A3. The lower bound for $\pi(B_{\epsilon_{n}})$

###### Lemma 5.5

If $\Sigma_{0}\in\mathcal{U}(s_{0},\tau_{0})$ and
$\Sigma\in\mathcal{U}(\tau)$, we have

  1. (i)

$K(f_{\Sigma_{0}},f_{\Sigma})\leq\tau^{4}\tau_{0}^{2}\|\Sigma-\Sigma_{0}\|_{F}^{2}$;

  2. (ii)

$V(f_{\Sigma_{0}},f_{\Sigma})\leq\frac{3}{2}\tau^{4}\tau_{0}^{2}\|\Sigma-\Sigma_{0}\|_{F}^{2}$.

###### Lemma 5.6

If $a=b=1/2$ and $\tau_{1}^{2}\asymp 1/(np^{4}\tau^{2})$, then

$\displaystyle\pi_{ij}^{u}(x)$ $\displaystyle\geq$
$\displaystyle\sqrt{\frac{1}{2\pi^{3}}}\frac{\tau_{1}}{x^{2}},$

for any $x>1$, where $\pi_{ij}^{u}(\sigma_{ij})$ is the unconstrained marginal
prior density of $\sigma_{ij}$.

The following theorem gives the lower bound for $\pi(B_{\epsilon_{n}})$ in
(12).

######  Theorem 5.7 (The lower bound for $\pi(B_{\epsilon_{n}})$)

If $\Sigma_{0}\in\mathcal{U}(s_{0},\tau_{0})$ with $\tau_{0}<\tau$, $p\asymp
n^{\beta}$ for some $0<\beta<1$, $\tau^{4}\leq p$, $\tau^{2}\tau_{0}^{2}\leq
s_{0}\log p$,
$n\geq\max\big{\\{}1/\tau_{0}^{4},s_{0}/(1-\tau_{0}/\tau)^{2}\big{\\}}\log
p/\tau^{4}$, $p^{-1}<\lambda<\log p/\tau_{0}$, $a=b=1/2$ and
$\tau_{1}^{2}\asymp 1/(np^{4}\tau^{2})$, we have

$\displaystyle\pi(B_{\epsilon_{n}})$ $\displaystyle\geq$
$\displaystyle\exp\Big{\\{}-\big{(}5+\frac{1}{\beta}\big{)}n\epsilon_{n}^{2}\Big{\\}}.$

  * Proof of Theorem 3.1

The proof follows from Theorems 5.2, 5.4, 5.7 and Lemma 5.1, with
$c_{1}=28+3/\beta$. Note that $\lambda>p^{-1}$ and
$\tau^{4}\leq\min\\{p,s_{0}\log p,(\log p)^{4}/\lambda^{4}\\}$ hold for all
sufficiently large $n$ because we assume $\tau=O(1)$ and $\lambda=O(1)$.
$\blacksquare$

## Appendix B: Proof of auxiliary results

###### Lemma 5.8

For any $p\times p$ matrices $A$ and $B$, we have

$\displaystyle\|AB\|_{F}$ $\displaystyle\leq$ $\displaystyle\|A\|\,\|B\|_{F}.$

  * Proof

Let $b_{j}$ be the $j$th column of $B$. Then,

$\displaystyle\|AB\|_{F}^{2}$ $\displaystyle=$
$\displaystyle\|(Ab_{1},\ldots,Ab_{p})\|_{F}^{2}$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{p}\|Ab_{j}\|_{2}^{2}$ $\displaystyle\leq$
$\displaystyle\|A\|^{2}\sum_{j=1}^{p}\|b_{j}\|_{2}^{2}$ $\displaystyle=$
$\displaystyle\|A\|^{2}\,\|B\|_{F}^{2}.\quad\blacksquare$

  * Proof of Theorem 5.2

Note that the $\epsilon_{n}$-packing number
$D(\epsilon_{n},\mathcal{P}_{n},d)$ is the maximal number of points in
$\mathcal{P}_{n}$ such that the distance between each pair is greater than or
equal to $\epsilon_{n}$. By Lemma A.1 in Banerjee and Ghosal (2015),

$\displaystyle d(f_{\Sigma_{1}},f_{\Sigma_{2}})$ $\displaystyle\leq$
$\displaystyle C\tau\|\Omega_{1}-\Omega_{2}\|_{F}$

for some constant $C>0$ and any
$\Sigma_{1},\Sigma_{2}\in\mathcal{U}(\delta_{n},s_{n},L_{n},\tau)$, where
$\Omega_{i}=\Sigma_{i}^{-1}$ for $i=1,2$. Further note that for any
$\Sigma_{1},\Sigma_{2}\in\mathcal{U}(\delta_{n},s_{n},L_{n},\tau)$,

$\displaystyle\|\Omega_{1}-\Omega_{2}\|_{F}$ $\displaystyle\leq$
$\displaystyle\|\Omega_{1}\|\,\|\Omega_{2}\|\,\|\Sigma_{1}-\Sigma_{2}\|_{F}$
$\displaystyle\leq$ $\displaystyle\tau^{2}\|\Sigma_{1}-\Sigma_{2}\|_{F},$

which gives

$\displaystyle d(f_{\Sigma_{1}},f_{\Sigma_{2}})$ $\displaystyle\leq$
$\displaystyle C\tau^{3}\|\Sigma_{1}-\Sigma_{2}\|_{F}.$

By the definition of the $\epsilon_{n}$-packing number it implies that

$\displaystyle\log D(\epsilon_{n},\mathcal{P}_{n},d)$ $\displaystyle\leq$
$\displaystyle\log
D(\epsilon_{n}/(C\tau^{3}),\mathcal{U}(\delta_{n},s_{n},L_{n},\tau),\|\cdot\|_{F})$
$\displaystyle\leq$
$\displaystyle\log\Big{\\{}\Big{(}\frac{CL_{n}\tau^{3}}{\epsilon_{n}}\Big{)}^{p}\sum_{j=1}^{s_{n}}\Big{(}\frac{2CL_{n}\tau^{3}}{\epsilon_{n}}\Big{)}^{j}\binom{\binom{p}{2}}{j}\Big{\\}}$
$\displaystyle\leq$ $\displaystyle
p\log\Big{(}\frac{CL_{n}\tau^{3}}{\epsilon_{n}}\Big{)}+\log\Big{\\{}\sum_{j=1}^{s_{n}}\Big{(}\frac{2CL_{n}\tau^{3}}{\epsilon_{n}}\Big{)}^{j}\Big{(}\frac{p^{2}}{2}\Big{)}^{j}\Big{\\}}$
$\displaystyle\leq$ $\displaystyle
p\log\Big{(}\frac{CL_{n}\tau^{3}}{\epsilon_{n}}\Big{)}+s_{n}\log\Big{(}\frac{2CL_{n}\tau^{3}p^{2}}{\epsilon_{n}}\Big{)}$
$\displaystyle\leq$
$\displaystyle(p+s_{n})\log(2CL_{n})+(p+s_{n})\log\tau^{3}+(p+s_{n})\log(1/\epsilon_{n})+2s_{n}\log
p$ $\displaystyle\leq$ $\displaystyle 3s_{n}\log p+\frac{3}{4}(p+s_{n})\log
p+\frac{1}{2}(p+s_{n})\log\frac{n}{(p+s_{0})\log p}+2s_{n}\log p$
$\displaystyle\leq$ $\displaystyle 3s_{n}\log p+\frac{3}{4}(p+s_{n})\log
p+\frac{1}{2\beta}(p+s_{n})\log p+2s_{n}\log p$ $\displaystyle\leq$
$\displaystyle\\{6+1/(2\beta)\\}(s_{n}/c_{1}+s_{n})\log
p\,\,=\,\,(12+1/\beta)c_{1}n\epsilon_{n}^{2}$

for all sufficiently large $n$, because $c_{1}>1$, $\tau^{4}\leq p$, $p\asymp
n^{\beta}$, $\delta_{n}=\epsilon_{n}/\tau^{3}$,
$s_{n}=c_{1}n\epsilon_{n}^{2}/\log p$ and $L_{n}=c_{2}n\epsilon_{n}^{2}$.
$\blacksquare$

  * Proof of Lemma 5.3

By Gershgorin circle theorem, every eigenvalue of $\Sigma$ lies within as
least one of $[\sigma_{jj}-\sum_{k\neq j}|\sigma_{kj}|,\sigma_{jj}+\sum_{k\neq
j}|\sigma_{kj}|]$ for $j=1,\ldots,p$ (Brualdi and Mellendorf, 1994). Thus, it
suffices to show

$\displaystyle\pi^{u}(\Sigma\in\mathcal{U}(\tau))$ $\displaystyle\geq$
$\displaystyle\pi^{u}\Big{(}\min_{j}\big{(}\sigma_{jj}-\sum_{k\neq
j}|\sigma_{kj}|\big{)}>0,\,\,\tau^{-1}\leq\lambda_{\min}(\Sigma)\leq\lambda_{\max}(\Sigma)\leq\tau\Big{)}.$

On the event $\min_{j}\big{(}\sigma_{jj}-\sum_{k\neq
j}|\sigma_{kj}|\big{)}>0$, we have

$\displaystyle\lambda_{\max}(\Sigma)$ $\displaystyle\leq$
$\displaystyle\|\Sigma\|_{1}$ $\displaystyle=$
$\displaystyle\max_{j}\big{(}\sigma_{jj}+\sum_{k\neq j}|\sigma_{kj}|\big{)}$
$\displaystyle\leq$ $\displaystyle\max_{j}2\sigma_{jj}$

and

$\displaystyle\lambda_{\min}(\Sigma)$ $\displaystyle\geq$
$\displaystyle\min_{j}\big{(}\sigma_{jj}-\sum_{k\neq j}|\sigma_{kj}|\big{)}.$

Therefore,

$\displaystyle\pi^{u}(\Sigma\in\mathcal{U}(\tau))$ $\displaystyle\geq$
$\displaystyle\pi^{u}\Big{(}\tau^{-1}\leq\min_{j}\big{(}\sigma_{jj}-\sum_{k\neq
j}|\sigma_{kj}|\big{)}\leq 2\max_{j}\sigma_{jj}\leq\tau\Big{)}$
$\displaystyle=$
$\displaystyle\pi^{u}\Big{(}\tau^{-1}\leq\min_{j}\big{(}\sigma_{jj}-\sum_{k\neq
j}|\sigma_{kj}|\big{)}\leq
2\max_{j}\sigma_{jj}\leq\tau\,\,\big{|}\,\,\max_{k\neq j}|\sigma_{kj}|<(\tau
p)^{-1}\Big{)}\pi^{u}(\max_{k\neq j}|\sigma_{kj}|<(\tau p)^{-1})$
$\displaystyle\geq$
$\displaystyle\pi^{u}\Big{(}\tau^{-1}\leq\min_{j}\big{(}\sigma_{jj}-\tau^{-1}\big{)}\leq
2\max_{j}\sigma_{jj}\leq\tau\,\,\big{|}\,\,\max_{k\neq j}|\sigma_{kj}|<(\tau
p)^{-1}\Big{)}\pi^{u}(\max_{k\neq j}|\sigma_{kj}|<(\tau p)^{-1})$
$\displaystyle=$
$\displaystyle\pi^{u}\Big{(}\tau^{-1}\leq\min_{j}\big{(}\sigma_{jj}-\tau^{-1}\big{)}\leq
2\max_{j}\sigma_{jj}\leq\tau\Big{)}\pi^{u}(\max_{k\neq j}|\sigma_{kj}|<(\tau
p)^{-1}).$

Note that

$\displaystyle\pi^{u}\Big{(}\tau^{-1}\leq\min_{j}\big{(}\sigma_{jj}-\tau^{-1}\big{)}\leq
2\max_{j}\sigma_{jj}\leq\tau\Big{)}$ $\displaystyle\geq$
$\displaystyle\pi^{u}\Big{(}2\tau^{-1}\leq\sigma_{jj}\leq\tau/2,\,\,\forall
j\Big{)}$ $\displaystyle=$
$\displaystyle\prod_{j=1}^{p}\pi^{u}\Big{(}2\tau^{-1}\leq\sigma_{jj}\leq\tau/2\Big{)}$
$\displaystyle\geq$
$\displaystyle\Big{\\{}\big{(}\frac{\tau}{2}-2\tau^{-1}\big{)}\frac{\lambda}{2}\exp\big{(}-\frac{\lambda\tau}{4}\big{)}\Big{\\}}^{p}$
$\displaystyle\geq$
$\displaystyle\Big{\\{}\frac{\lambda\tau}{8}\exp\big{(}-\frac{\lambda\tau}{4}\big{)}\Big{\\}}^{p}$

because $\tau>3$. Furthermore, by Theorem 1 in Carvalho et al. (2010) and the
change of variables,

$\displaystyle\pi^{u}(\sigma_{kj})$ $\displaystyle\leq$
$\displaystyle\frac{1}{\tau_{1}\sqrt{2\pi^{3}}}\log\Big{(}1+\frac{2\tau_{1}^{2}}{\sigma_{kj}^{2}}\Big{)}$
(13)

for any $\sigma_{kj}\neq 0$, which implies

$\displaystyle\pi^{u}(|\sigma_{kj}|\geq(\tau p)^{-1})$ $\displaystyle\leq$
$\displaystyle\frac{1}{\tau_{1}}\sqrt{\frac{2}{\pi^{3}}}\int_{(\tau
p)^{-1}}^{\infty}\log\Big{(}1+\frac{2\tau_{1}^{2}}{x^{2}}\Big{)}dx$
$\displaystyle\leq$ $\displaystyle\sqrt{\frac{2}{\pi^{3}}}\int_{(\tau
p)^{-1}}^{\infty}\frac{2\tau_{1}}{x^{2}}dx$ $\displaystyle=$
$\displaystyle\frac{2\sqrt{2}}{\sqrt{\pi^{3}}}\tau_{1}\tau p.$

Thus, we have

$\displaystyle\pi^{u}(\max_{k\neq j}|\sigma_{kj}|<(\tau p)^{-1})$
$\displaystyle=$ $\displaystyle\prod_{k\neq
j}\Big{\\{}1-\pi^{u}(|\sigma_{kj}|\geq(\tau p)^{-1})\Big{\\}}$
$\displaystyle\geq$
$\displaystyle\Big{(}1-\frac{2\sqrt{2}}{\sqrt{\pi^{3}}}\tau_{1}\tau
p\Big{)}^{p^{2}}$ $\displaystyle\geq$
$\displaystyle\exp\Big{(}-\frac{4\sqrt{2}}{\sqrt{\pi^{3}}}\tau_{1}\tau
p^{3}\Big{)}$ $\displaystyle=$
$\displaystyle\exp\Big{(}-C\frac{p}{\sqrt{n}}\Big{)}$

for some constant $C>0$, because $\tau_{1}^{2}\asymp 1/(np^{4}\tau^{2})$.
$\blacksquare$

  * Proof of Theorem 5.4

Note that

$\displaystyle\pi(\mathcal{P}_{n}^{c})$ $\displaystyle\leq$
$\displaystyle\pi(|s(\Sigma,\delta_{n})|>s_{n})+\pi(\|\Sigma\|_{\max}>L_{n}).$

First, we focus on the upper bound for $\pi(|s(\Sigma,\delta_{n})|>s_{n})$.
For any $1\leq k\neq j\leq p$, by applying inequality (13), we have

$\displaystyle\nu_{n}\,\,\equiv\,\,\pi^{u}(|\sigma_{kj}|>\delta_{n})$
$\displaystyle\leq$
$\displaystyle\frac{2\sqrt{2}}{\sqrt{\pi^{3}}}\tau_{1}\delta_{n}^{-1}$
$\displaystyle\leq$ $\displaystyle\frac{C}{\sqrt{n}\tau
p^{2}}\frac{\tau^{3}\sqrt{n}}{\sqrt{(p+s_{0})\log p}}$ $\displaystyle=$
$\displaystyle\frac{C\tau^{2}}{\sqrt{p^{4}(p+s_{0})\log p}}$

for some constant $C>0$, because $\tau_{1}^{2}\asymp 1/(np^{4}\tau^{2})$.
Thus,

$\displaystyle\pi^{u}(|s(\Sigma,\delta_{n})|>s_{n})$ $\displaystyle=$
$\displaystyle\mathbb{P}\Big{(}B\Big{(}\binom{p}{2},\nu_{n}\Big{)}>s_{n}\Big{)}$
$\displaystyle\leq$ $\displaystyle
1-\Phi\Big{(}\sqrt{2\binom{p}{2}H\Big{(}\nu_{n},s_{n}/\binom{p}{2}\Big{)}}\Big{)}$

by Lemma A.3 of Song and Liang (2018), provided $s_{n}>\binom{p}{2}\nu_{n}$,
where $\Phi$ is the cdf of $N(0,1)$ and

$\displaystyle H(\nu,k/n)$ $\displaystyle=$
$\displaystyle(k/n)\log\\{k/(n\nu)\\}+(1-k/n)\log\\{(1-k/n)/(1-\nu)\\}.$

Note that the condition $s_{n}>\binom{p}{2}\nu_{n}$ is met because we assume
that $\tau^{4}\ll(p+s_{0})^{2}\log p$.

Note that

$\displaystyle
1-\Phi\Big{(}\sqrt{2\binom{p}{2}H\Big{(}\nu_{n},s_{n}/\binom{p}{2}\Big{)}}\Big{)}$
$\displaystyle\leq$
$\displaystyle\frac{\exp\big{[}-\binom{p}{2}H\\{\nu_{n},s_{n}/\binom{p}{2}\\}\big{]}}{\sqrt{2\pi}\sqrt{2\binom{p}{2}H\\{\nu_{n},s_{n}/\binom{p}{2}\\}}},$

where

$\displaystyle\binom{p}{2}H\Big{(}\nu_{n},s_{n}/\binom{p}{2}\Big{)}$
$\displaystyle=$ $\displaystyle
s_{n}\log\Big{(}\frac{s_{n}}{\binom{p}{2}\nu_{n}}\Big{)}+\Big{\\{}\binom{p}{2}-s_{n}\Big{\\}}\log\Big{(}\frac{\binom{p}{2}-s_{n}}{\binom{p}{2}-\binom{p}{2}\nu_{n}}\Big{)}.$

We have

$\displaystyle s_{n}\log\Big{(}\frac{s_{n}}{\binom{p}{2}\nu_{n}}\Big{)}$
$\displaystyle\geq$ $\displaystyle
s_{n}\log\Big{(}c_{1}C\sqrt{\frac{(p+s_{0})^{3}\log p}{\tau^{4}}}\Big{)}$
$\displaystyle\geq$ $\displaystyle s_{n}\log\Big{(}\sqrt{p+s_{0}}\Big{)}$
$\displaystyle\geq$ $\displaystyle s_{n}(\log
p)/2\,\,=\,\,c_{1}n\epsilon_{n}^{2}/2$

for some constant $C>0$ because $\tau^{4}\ll(p+s_{0})^{2}\log p$ and
$c_{1}>1$, and

$\displaystyle\Big{\\{}\binom{p}{2}-s_{n}\Big{\\}}\log\Big{(}\frac{\binom{p}{2}-s_{n}}{\binom{p}{2}-\binom{p}{2}\nu_{n}}\Big{)}$
$\displaystyle=$
$\displaystyle\Big{\\{}\binom{p}{2}-s_{n}\Big{\\}}\log\Big{(}1-\frac{s_{n}-\binom{p}{2}\nu_{n}}{\binom{p}{2}(1-\nu_{n})}\Big{)}$
$\displaystyle\geq$
$\displaystyle-\frac{1}{2}\Big{\\{}\binom{p}{2}-s_{n}\Big{\\}}\frac{s_{n}-\binom{p}{2}\nu_{n}}{\binom{p}{2}(1-\nu_{n})}$
$\displaystyle\geq$
$\displaystyle-\frac{1}{2}\Big{\\{}1-s_{n}/\binom{p}{2}\Big{\\}}\frac{s_{n}}{1-\nu_{n}}$
$\displaystyle\gtrsim$ $\displaystyle-
s_{n}\Big{\\{}1-c_{1}(p+s_{0})/p^{2}\Big{\\}}\,\,\gtrsim\,\,-\frac{c_{1}n\epsilon_{n}^{2}}{\log
p}$

for all sufficiently large $n$. Therefore, due to Lemma 5.3 and
$\lambda\tau\leq\log p$,

$\displaystyle\pi(|s(\Sigma,\delta_{n})|>s_{n})$ $\displaystyle\leq$
$\displaystyle\pi^{u}(|s(\Sigma,\delta_{n})|>s_{n})/\pi^{u}(\Sigma\in\mathcal{U}(\tau))$
$\displaystyle\leq$
$\displaystyle\exp\big{(}-c_{1}n\epsilon_{n}^{2}/3\big{)}/\pi^{u}(\Sigma\in\mathcal{U}(\tau))$
$\displaystyle\leq$
$\displaystyle\Big{\\{}\frac{8}{\lambda\tau}\exp\big{(}\frac{\lambda\tau}{4}+\frac{C}{\sqrt{n}}\big{)}\Big{\\}}^{p}\exp\big{(}-c_{1}n\epsilon_{n}^{2}/3\big{)}$
$\displaystyle\leq$ $\displaystyle\exp\big{(}-c_{1}n\epsilon_{n}^{2}/3+p\log
p/3\big{)}$ $\displaystyle\leq$
$\displaystyle\exp\big{\\{}-(c_{1}-1)n\epsilon_{n}^{2}/3\big{\\}}$

for some constant $C>0$ and all sufficiently large $n$.

Now we focus on the upper bound for the second term
$\pi(\|\Sigma\|_{\max}>L_{n})$. Since
$\pi(\Sigma:\lambda_{\max}(\Sigma)\leq\tau)=1$ and
$\|\Sigma\|_{\max}\leq\lambda_{\max}(\Sigma)$, it means that
$\pi(\|\Sigma\|_{\max}>L_{n})=0$ for all sufficiently large $n$, because
$L_{n}\to\infty$ as $n\to\infty$. It completes the proof. $\blacksquare$

  * Proof of Lemma 5.5

In the proof, we follow the calculation of Banerjee and Ghosal (2015). Let
$d_{i}$’s be the eigenvalues of
${\Sigma_{0}}^{\frac{1}{2}}\Sigma^{-\frac{1}{2}}{\Sigma_{0}}^{\frac{1}{2}}$.
By Lemma A.1 in Banerjee and Ghosal (2015), we obtain

$\displaystyle\sum_{i=1}^{p}(1-d_{i})^{2}$ $\displaystyle\leq$
$\displaystyle\tau^{2}\|{\Sigma_{0}}^{-1}-\Sigma^{-1}\|_{F}^{2}.$

Also,

$\displaystyle\|{\Sigma_{0}}^{-1}-\Sigma^{-1}\|_{F}$ $\displaystyle\leq$
$\displaystyle\|\Sigma^{-1}\|\,\|{\Sigma_{0}}^{-1}\|\,\|\Sigma-\Sigma_{0}\|_{F}$
$\displaystyle\leq$ $\displaystyle\tau\tau_{0}\|\Sigma-\Sigma_{0}\|_{F}.$

Following the calculation of Banerjee and Ghosal (2015), we obtain

$\displaystyle K(f_{\Sigma_{0}},f_{\Sigma})$ $\displaystyle=$
$\displaystyle-\frac{1}{2}\sum_{i=1}^{p}\log
d_{i}-\frac{1}{2}\sum_{i=1}^{p}(1-d_{i})$ $\displaystyle\leq$
$\displaystyle\sum_{i=1}^{p}(1-d_{i})^{2}$ $\displaystyle\leq$
$\displaystyle\tau^{2}\|{\Sigma_{0}}^{-1}-\Sigma^{-1}\|_{F}^{2}$
$\displaystyle\leq$
$\displaystyle\tau^{4}\tau_{0}^{2}\|\Sigma-\Sigma_{0}\|_{F}^{2}.$

Similarly, we obtain

$\displaystyle V(f_{\Sigma_{0}},f_{\Sigma})$ $\displaystyle=$
$\displaystyle\frac{1}{2}\sum_{i=1}^{p}(1-d_{i})^{2}+K(f_{\Sigma_{0}},f_{\Sigma})^{2}$
$\displaystyle\leq$ $\displaystyle\frac{3}{2}\sum_{i=1}^{p}(1-d_{i})^{2}$
$\displaystyle\leq$
$\displaystyle\frac{3}{2}\tau^{2}\|{\Sigma_{0}}^{-1}-\Sigma^{-1}\|_{F}^{2}$
$\displaystyle\leq$
$\displaystyle\frac{3}{2}\tau^{4}\tau_{0}^{2}\|\Sigma-\Sigma_{0}\|_{F}^{2}.$

This completes the proof. $\blacksquare$

  * Proof of Lemma 5.6

Because we have $a=b=1/2$,

$\displaystyle\sigma_{ij}/\tau_{1}\mid\rho_{ij}$ $\displaystyle\sim$
$\displaystyle N\Big{(}0,\frac{\rho_{ij}}{1-\rho_{ij}}\Big{)},$
$\displaystyle\rho_{ij}$ $\displaystyle\sim$ $\displaystyle Beta(a,b)$

is equivalent to

$\displaystyle\sigma_{ij}/\tau_{1}\mid\lambda_{ij}$ $\displaystyle\sim$
$\displaystyle N\Big{(}0,\lambda_{ij}^{2}\Big{)},$ $\displaystyle\lambda_{ij}$
$\displaystyle\sim$ $\displaystyle C^{+}(0,1),$

where $C^{+}(0,s)$ denotes the standard half-Cauchy distribution on positive
real with a scale parameter $s$. Then, by Theorem 1 in Carvalho et al. (2010)
and the change of variables,

$\displaystyle\pi_{ij}^{u}(x)$ $\displaystyle\geq$
$\displaystyle\frac{1}{2\tau_{1}}\sqrt{\frac{1}{2\pi^{3}}}\log\Big{(}1+\frac{4\tau_{1}^{2}}{x^{2}}\Big{)}$
$\displaystyle\geq$
$\displaystyle\frac{1}{4\tau_{1}}\sqrt{\frac{1}{2\pi^{3}}}\frac{4\tau_{1}^{2}}{x^{2}}$
$\displaystyle\geq$
$\displaystyle\sqrt{\frac{1}{2\pi^{3}}}\frac{\tau_{1}}{x^{2}}$

for any $x>1$, because $\tau_{1}\asymp 1/(np^{4}\tau^{2})$. $\blacksquare$

  * Proof of Theorem 5.7

By Lemma 5.5, it suffices to show that

$\displaystyle\pi\Big{(}\|\Sigma-\Sigma_{0}\|_{F}^{2}\leq\frac{2}{3\tau^{4}\tau_{0}^{2}}\epsilon_{n}^{2}\Big{)}$
$\displaystyle\geq$ $\displaystyle\exp\big{(}-Cn\epsilon_{n}^{2}\big{)}.$

Note that

$\displaystyle\pi\Big{(}\|\Sigma-\Sigma_{0}\|_{F}^{2}\leq\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{(p+s_{0})\log
p}{n}\Big{)}$ $\displaystyle\geq$ $\displaystyle\pi\Big{(}\sum_{i\neq
j}(\sigma_{ij}-\sigma_{ij}^{*})^{2}\leq\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{s_{0}\log
p}{n},\,\,\sum_{j=1}^{p}(\sigma_{jj}-\sigma_{jj}^{*})^{2}\leq\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{p\log
p}{n}\Big{)}$ $\displaystyle\geq$ $\displaystyle\pi\Big{(}\max_{i\neq
j}(\sigma_{ij}-\sigma_{ij}^{*})^{2}\leq\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{s_{0}\log
p}{p(p-1)n},\,\,\max_{1\leq j\leq
p}(\sigma_{jj}-\sigma_{jj}^{*})^{2}\leq\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{\log
p}{n}\Big{)}$ $\displaystyle\equiv$ $\displaystyle\pi(A_{n,\Sigma_{0}}),$

where $\Sigma_{0}=(\sigma_{ij}^{*})$. By Weyl’s theorem, if $\Sigma\in
A_{n,\Sigma_{0}}$,

$\displaystyle\lambda_{\min}(\Sigma)$ $\displaystyle\geq$
$\displaystyle\lambda_{\min}(\Sigma_{0})-\|\Sigma-\Sigma_{0}\|$
$\displaystyle\geq$
$\displaystyle\lambda_{\min}(\Sigma_{0})-\|\Sigma-\Sigma_{0}\|_{1}$
$\displaystyle\geq$
$\displaystyle\tau_{0}^{-1}-p\sqrt{\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{s_{0}\log
p}{p(p-1)n}}-\sqrt{\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{\log p}{n}}$
$\displaystyle\geq$ $\displaystyle\tau^{-1}$

and

$\displaystyle\lambda_{\max}(\Sigma)$ $\displaystyle\leq$
$\displaystyle\lambda_{\max}(\Sigma_{0})+\|\Sigma-\Sigma_{0}\|$
$\displaystyle\leq$ $\displaystyle\tau_{0}+\|\Sigma-\Sigma_{0}\|_{1}$
$\displaystyle\leq$
$\displaystyle\tau_{0}+p\sqrt{\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{s_{0}\log
p}{p(p-1)n}}+\sqrt{\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{\log p}{n}}$
$\displaystyle\leq$ $\displaystyle\tau$

for all sufficiently large $n$, because
$\Sigma_{0}\in\mathcal{U}(s_{0},\tau_{0})$, $\tau_{0}<\tau$ and
$\tau^{4}(1-\tau_{0}/\tau)^{2}n\geq s_{0}\log p$. Thus, if $\Sigma\in
A_{n,\Sigma_{0}}$, then we have $\Sigma\in\mathcal{U}(\tau)$. Because

$\displaystyle{\pi}(\Sigma)$ $\displaystyle=$
$\displaystyle\frac{\pi^{u}(\Sigma)I(\Sigma\in\mathcal{U}(\tau))}{\pi^{u}(\Sigma\in\mathcal{U}(\tau))},$

we have

$\displaystyle\pi(A_{n,\Sigma_{0}})$ $\displaystyle\geq$
$\displaystyle\pi^{u}(A_{n,\Sigma_{0}}).$

Note that

$\displaystyle\pi^{u}(A_{n,\Sigma_{0}})$ $\displaystyle=$
$\displaystyle\pi^{u}\Big{(}\max_{i\neq
j}(\sigma_{ij}-\sigma_{ij}^{*})^{2}\leq\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{s_{0}\log
p}{p(p-1)n}\Big{)}\times\pi^{u}\Big{(}\max_{1\leq j\leq
p}(\sigma_{jj}-\sigma_{jj}^{*})^{2}\leq\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{\log
p}{n}\Big{)}$ $\displaystyle=$
$\displaystyle\prod_{i<j}\pi^{u}\Big{(}(\sigma_{ij}-\sigma_{ij}^{*})^{2}\leq\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{s_{0}\log
p}{p(p-1)n}\Big{)}\times\prod_{j=1}^{p}\pi^{u}\Big{(}(\sigma_{jj}-\sigma_{jj}^{*})^{2}\leq\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{\log
p}{n}\Big{)}$

and

$\displaystyle\prod_{j=1}^{p}\pi^{u}\Big{(}(\sigma_{jj}-\sigma_{jj}^{*})^{2}\leq\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{\log
p}{n}\Big{)}$ $\displaystyle\geq$
$\displaystyle\prod_{j=1}^{p}2\sqrt{\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{\log
p}{n}}\,\,\frac{\lambda}{2}\exp\Big{\\{}-\frac{\lambda}{2}\Big{(}\sigma_{jj}^{*}+\sqrt{\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{\log
p}{n}}\Big{)}\Big{\\}}$ $\displaystyle\geq$
$\displaystyle\left[\lambda\sqrt{\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{\log
p}{n}}\exp\Big{\\{}-\frac{\lambda}{2}\Big{(}\tau_{0}+\sqrt{\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{\log
p}{n}}\Big{)}\Big{\\}}\right]^{p}$ $\displaystyle\geq$
$\displaystyle\exp\Big{\\{}-p\lambda\tau_{0}-p\log\Big{(}\frac{\sqrt{3\tau^{4}\tau_{0}^{2}n}}{\lambda\sqrt{2\log
p}}\Big{)}\Big{\\}}$ $\displaystyle\geq$ $\displaystyle\exp\Big{\\{}-p\log
p-p\log\Big{(}\frac{\tau^{3}p^{1/(2\beta)}}{\lambda}\Big{)}\Big{\\}}$
$\displaystyle\geq$ $\displaystyle\exp\Big{\\{}-p\log
p-\big{(}2+\frac{1}{2\beta}\big{)}p\log p\Big{\\}}$ $\displaystyle=$
$\displaystyle\exp\Big{\\{}-\big{(}3+\frac{1}{2\beta}\big{)}p\log p\Big{\\}}$

for all sufficiently large $n$, because $\log p/(\tau^{4}\tau_{0}^{4})\leq n$,
$\tau^{4}\leq p$ and $p^{-1}<\lambda<\log p/\tau_{0}$. Furthermore,

$\displaystyle\prod_{i<j}\pi^{u}\Big{(}(\sigma_{ij}-\sigma_{ij}^{*})^{2}\leq\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{s_{0}\log
p}{p(p-1)n}\Big{)}$ (14) $\displaystyle\geq$ $\displaystyle\prod_{(i,j)\in
s(\Sigma_{0})}\pi^{u}\Big{(}(\sigma_{ij}-\sigma_{ij}^{*})^{2}\leq\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{s_{0}\log
p}{p(p-1)n}\Big{)}\prod_{(i,j)\notin
s(\Sigma_{0}),i<j}\pi^{u}\Big{(}\sigma_{ij}^{2}\leq\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{s_{0}\log
p}{p(p-1)n}\Big{)}.\,\,\,\,$

The second term in (14) is bound below by

$\displaystyle\prod_{(i,j)\notin
s(\Sigma_{0}),i<j}\left\\{1-\pi^{u}\Big{(}\sigma_{ij}^{2}>\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{s_{0}\log
p}{p(p-1)n}\Big{)}\right\\}$ $\displaystyle\geq$
$\displaystyle\left\\{1-2\tau_{1}\sqrt{\frac{2}{\pi^{3}}}\sqrt{\frac{3\tau^{4}\tau_{0}^{2}}{2}\frac{p(p-1)n}{s_{0}\log
p}}\right\\}^{p^{2}}$ $\displaystyle\geq$
$\displaystyle\exp\Big{(}-4\sqrt{\frac{3}{\pi^{3}}}\tau_{1}\tau^{2}\tau_{0}p^{3}\sqrt{\frac{n}{s_{0}\log
p}}\Big{)}$ $\displaystyle\geq$
$\displaystyle\exp\Big{(}-C\tau\tau_{0}p\sqrt{\frac{1}{s_{0}\log p}}\Big{)}$
$\displaystyle\geq$ $\displaystyle\exp(-Cp)$

for some constant $C>0$, because $\tau^{2}\tau_{0}^{2}\leq s_{0}\log p$ and
$\tau_{1}^{2}\asymp 1/(np^{4}\tau^{2})$. Let
$\pi_{ij}^{u}(\sigma_{ij})=\int_{0}^{1}\pi^{u}(\sigma_{ij}\mid\rho_{ij})\pi^{u}(\rho_{ij})d\rho_{ij}$
be the unconstrained marginal prior density of $\sigma_{ij}$. The first term
in (14) is bounded below by

$\displaystyle\prod_{(i,j)\in
s(\Sigma_{0})}\pi_{ij}^{u}\Big{(}\sigma_{ij}^{*}+\sqrt{\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{s_{0}\log
p}{p(p-1)n}}\,\,\Big{)}\,\,2\sqrt{\frac{2}{3\tau^{4}\tau_{0}^{2}}\frac{s_{0}\log
p}{p(p-1)n}}$ $\displaystyle\geq$
$\displaystyle\left\\{2\pi_{ij}^{u}(2\tau_{0})\sqrt{\frac{2s_{0}\log
p}{3\tau^{4}\tau_{0}^{2}p(p-1)n}}\right\\}^{s_{0}}$ $\displaystyle\geq$
$\displaystyle\exp\Big{\\{}s_{0}\log(\pi_{ij}^{u}(2\tau_{0}))-\frac{1}{2}s_{0}\log\Big{(}\frac{3\tau^{4}\tau_{0}^{2}p^{2}n}{2s_{0}\log
p}\Big{)}\Big{\\}}$ $\displaystyle\geq$
$\displaystyle\exp\Big{\\{}s_{0}\log(\pi_{ij}^{u}(2\tau_{0}))-\frac{1}{2}s_{0}\log\Big{(}\frac{3\tau^{2}p^{2}n}{2}\Big{)}\Big{\\}}$
$\displaystyle\geq$
$\displaystyle\exp\Big{\\{}-\frac{1}{2}s_{0}\log\big{(}\tau_{0}^{4}\tau^{2}np^{4}\big{)}-\frac{1}{2}s_{0}\log\Big{(}\frac{3\tau^{2}p^{2}n}{2}\Big{)}\Big{\\}}$
$\displaystyle\geq$
$\displaystyle\exp\Big{\\{}-\frac{1}{2}s_{0}\log\Big{(}\frac{3\tau_{0}^{4}\tau^{4}p^{6}n^{2}}{2}\Big{)}\Big{\\}}$
$\displaystyle\geq$
$\displaystyle\exp\Big{\\{}-\frac{1}{2}s_{0}\log\Big{(}Cp^{8+2/\beta}\Big{)}\Big{\\}}$
$\displaystyle\geq$
$\displaystyle\exp\Big{\\{}-\big{(}5+\frac{1}{\beta}\big{)}s_{0}\log
p\Big{\\}}$

for some constant $C>0$, because $p\asymp n^{\beta}$, $\log
p/(\tau^{4}\tau_{0}^{4})\leq n$,
$\pi_{ij}^{u}(2\tau_{0})\geq\tau_{1}/(4\sqrt{2\pi^{3}}\tau_{0}^{2})\asymp
1/(\tau_{0}^{2}\tau\sqrt{n}p^{2})$ and $\tau^{4}\leq p$, by Lemma 5.6.
Therefore, we have

$\displaystyle\pi^{u}(A_{n,\Sigma_{0}})$ $\displaystyle\geq$
$\displaystyle\exp\Big{\\{}-\big{(}3+\frac{1}{2\beta}\big{)}p\log
p-Cp-\big{(}5+\frac{1}{\beta}\big{)}s_{0}\log p\Big{\\}}$ $\displaystyle\geq$
$\displaystyle\exp\Big{\\{}-\big{(}5+\frac{1}{\beta}\big{)}(p+s_{0})\log
p\Big{\\}}\,\,=\,\,\exp\Big{\\{}-\big{(}5+\frac{1}{\beta}\big{)}n\epsilon_{n}^{2}\Big{\\}}.\quad\blacksquare$

## Appendix C: Proof of Theorem 3.2

  * Proof

We will first show that

$\displaystyle\inf_{\hat{\Sigma}}\sup_{\Sigma_{0}\in
B_{1}}{\mathbb{E}}_{0}\|\hat{\Sigma}-\Sigma_{0}\|_{F}^{2}$
$\displaystyle\gtrsim$ $\displaystyle\frac{s_{0}\log p}{n}$ (15)

for some $B_{1}\subset\mathcal{U}(s_{0},\tau_{0})$ when $s_{0}>3p$, and show
that

$\displaystyle\inf_{\hat{\Sigma}}\sup_{\Sigma_{0}\in
B_{2}}{\mathbb{E}}_{0}\|\hat{\Sigma}-\Sigma_{0}\|_{F}^{2}$
$\displaystyle\gtrsim$ $\displaystyle\frac{p}{n}$ (16)

for some $B_{2}\subset\mathcal{U}(s_{0},\tau_{0})$ when $s_{0}\leq 3p$.

#### (i) Proof of (15)

Let $r=\lfloor p/2\rfloor$ and $\epsilon_{np}=\nu\sqrt{\log p/n}$ with
$\nu=\sqrt{\epsilon/4}$. For any $u\in\mathbb{R}^{p}$, let $A_{m}(u)$ be a
$p\times p$ symmetric matrix whose the $m$th row and column are equal to $u$
and the rest of entries are zero. Define a parameter space

$\displaystyle B_{1}$ $\displaystyle:=$
$\displaystyle\Big{\\{}\Sigma(\theta):\Sigma(\theta)=I_{p}+\epsilon_{np}\sum_{m=1}^{r}\gamma_{m}A_{m}(\lambda_{m}),\,\,\theta=(\gamma,\lambda)\in\Theta\Big{\\}},$

where $\gamma=(\gamma_{1},\ldots,\gamma_{r})\in\Gamma=\\{0,1\\}^{r}$,
$\lambda=(\lambda_{1},\ldots,\lambda_{r})^{T}\in\Lambda\subset\mathbb{R}^{r\times
p}$ and $\Theta=\Gamma\times\Lambda$. Here, we let

$\displaystyle\Lambda$ $\displaystyle:=$
$\displaystyle\big{\\{}\lambda=(\lambda_{1},\ldots,\lambda_{r})^{T}:\lambda_{m}=(\lambda_{mi})\in\\{0,1\\}^{p},\,\,\|\lambda_{m}\|_{0}=k,\,\,\sum_{i=1}^{p-r}\lambda_{mi}=0$
$\displaystyle\quad\quad\quad\quad\quad\text{ for any }m=1,\ldots,r,\text{ and
satisfies }\max_{1\leq i\leq p}\sum_{m=1}^{r}\lambda_{mi}\leq 2k\big{\\}},$

$k=\lceil c_{np}/2\rceil-1$ and $c_{np}=\lceil s_{0}/p\rceil$.

We will first show that $B_{1}\subset\mathcal{U}(s_{0},\tau_{0})$. Note that
$\|\Sigma(\theta)\|\leq\|\Sigma(\theta)\|_{1}\leq 1+2k\epsilon_{np}\leq
1+c_{np}\nu\sqrt{\log p/n}\leq\tau_{0}$ for any $\tau_{0}>1$ and sufficiently
large $n$ due to our assumption, $s_{0}^{2}(\log p)^{3}=O(p^{2}n)$ . Also note
that $2k\epsilon_{np}\leq c_{np}\nu\sqrt{\log p/n}\leq(1+s_{0}/p)\nu\sqrt{\log
p/n}\leq 1-\tau_{0}^{-1}$ for any $\tau_{0}>1$ and sufficiently large $n$,
which implies that $\Sigma(\theta)-\tau_{0}^{-1}I_{p}$ is diagonally dominant.
Thus, we have $\lambda_{\min}(\Sigma(\theta))\geq\tau_{0}^{-1}$. Because
$|s(\Sigma(\theta))|\leq 2kp\leq s_{0}$, it holds that
$B_{1}\subset\mathcal{U}(s_{0},\tau_{0})$.

For given $\theta\in\Theta$ and $a\in\\{0,1\\}$, let $\mathbb{P}_{\theta}$ and
$\bar{\mathbb{P}}_{i,a}$ be the joint distribution of random samples
$X_{1},\ldots,X_{n}$ from $N_{p}(0,\Sigma(\theta))$ and
$\\{2^{r-1}|\Lambda|\\}^{-1}\sum_{\theta\in\Theta_{i,a}}\mathbb{P}_{\theta}$,
respectively, where $\Theta_{i,a}=\\{\theta\in\Theta:\gamma_{i}(\theta)=a\\}$.
For any two probability measures $\mathbb{P}$ and $\mathbb{Q}$, let
$\|\mathbb{P}\wedge\mathbb{Q}\|=\int(p\wedge q)d\mu$, where $p$ and $q$ are
probability densities corresponding to $\mathbb{P}$ and $\mathbb{Q}$,
respectively, with respect to a common dominating measure $\mu$. By applying
Lemma 3 of Cai and Zhou (2012) with $s=2$, we have

$\displaystyle\inf_{\hat{\Sigma}}\max_{\theta\in\Theta}2^{2}{\mathbb{E}}_{\theta}\|\hat{\Sigma}-\Sigma(\theta)\|_{F}^{2}$
$\displaystyle\geq$ $\displaystyle\alpha\,\frac{r}{2}\,\min_{1\leq i\leq
r}\|\bar{\mathbb{P}}_{i,0}\wedge\bar{\mathbb{P}}_{i,1}\|,$

where ${\mathbb{E}}_{\theta}$ denotes the expectation with respect to
$X_{1},\ldots,X_{n}\overset{iid}{\sim}N_{p}(0,\Sigma(\theta))$ and

$\alpha=\min_{(\theta,\theta^{\prime}):H(\gamma(\theta),\gamma(\theta^{\prime}))\geq
1}\|\Sigma(\theta)-\Sigma(\theta^{\prime})\|_{F}^{2}/H(\gamma(\theta),\gamma(\theta^{\prime})).$

Here, $H(x,y)=\sum_{j=1}^{r}|x_{j}-y_{j}|$ for any $x,y\in\\{0,1\\}^{r}$. By
the definition of $\Sigma(\theta)$,

$\displaystyle\|\Sigma(\theta)-\Sigma(\theta^{\prime})\|_{F}^{2}$
$\displaystyle\geq$ $\displaystyle
2k\epsilon_{np}^{2}H(\gamma(\theta),\gamma(\theta^{\prime}))$

for any $\theta,\theta^{\prime}\in\Theta$, which implies

$\displaystyle\alpha r$ $\displaystyle\geq$ $\displaystyle
2k\epsilon_{np}^{2}\,r\,\,\geq\,\,\nu^{2}\Big{(}\frac{1}{2}-\frac{p}{s_{0}}\Big{)}\frac{s_{0}\log
p}{n}\,\,\asymp\,\,\frac{s_{0}\log p}{n}$

due to $s_{0}>3p$. Therefore, we complete the proof if we show that

$\displaystyle\min_{1\leq i\leq
r}\|\bar{\mathbb{P}}_{i,0}\wedge\bar{\mathbb{P}}_{i,1}\|$ $\displaystyle\geq$
$\displaystyle c_{1}$ (17)

for some constant $c_{1}>0$. Note that, without loss of generality, it
suffices to show that
$\|\bar{\mathbb{P}}_{1,0}\wedge\bar{\mathbb{P}}_{1,1}\|\geq c_{1}$.

Let $\Lambda_{1}=\\{\lambda_{1}(\theta)\in\mathbb{R}^{p}:\theta\in\Theta\\}$
and
$\Lambda_{-1}=\\{\lambda_{-1}(\theta)\equiv(\lambda_{2}(\theta),\ldots,\lambda_{r}(\theta))^{T}\in\mathbb{R}^{(r-1)\times
p}:\theta\in\Theta\\}$. For any $a\in\\{0,1\\}$, $b\in\\{0,1\\}^{r-1}$ and
$c\in\Lambda_{-1}$, we define

$\displaystyle\bar{\mathbb{P}}_{(1,a,b,c)}$ $\displaystyle=$
$\displaystyle\frac{1}{|\Theta_{(1,a,b,c)}|}\sum_{\theta\in\Theta_{(1,a,b,c)}}\mathbb{P}_{\theta},$
$\displaystyle\Theta_{(1,a,b,c)}$ $\displaystyle=$
$\displaystyle\big{\\{}\theta\in\Theta:\gamma_{1}(\theta)=a,\gamma_{-1}(\theta)=b,\lambda_{-1}(\theta)=c\big{\\}},$

where
$\gamma_{-1}(\theta)\equiv(\gamma_{2}(\theta),\ldots,\gamma_{r}(\theta))\in\\{0,1\\}^{r-1}$
. Let ${\mathbb{E}}_{(\gamma_{-1},\lambda_{-1})}f(\gamma_{-1},\lambda_{-1})$
be the expectation of $f(\gamma_{-1},\lambda_{-1})$ over
$\Theta_{-1}=\\{0,1\\}^{r-1}\times\Lambda_{-1}$, i.e.,

$\displaystyle{\mathbb{E}}_{(\gamma_{-1},\lambda_{-1})}f(\gamma_{-1},\lambda_{-1})$
$\displaystyle=$
$\displaystyle\frac{1}{2^{r-1}|\Lambda|}\sum_{(b,c)\in\Theta_{-1}}|\Theta_{(1,a,b,c)}|f(b,c),$

where the probability distribution of $(\gamma_{-1},\lambda_{-1})$ is induced
by the uniform distribution over $\Theta$. To show (17), it suffices to prove
that there exists $0<c_{2}<1$ such that

$\displaystyle{\mathbb{E}}_{(\gamma_{-1},\lambda_{-1})}\Big{\\{}\int\Big{(}\frac{d\bar{\mathbb{P}}_{(1,1,\gamma_{-1},\lambda_{-1})}}{d\bar{\mathbb{P}}_{(1,0,\gamma_{-1},\lambda_{-1})}}\Big{)}^{2}d\bar{\mathbb{P}}_{(1,0,\gamma_{-1},\lambda_{-1})}-1\Big{\\}}$
$\displaystyle\leq$ $\displaystyle c_{2}^{2},$ (18)

by Lemma 8 (ii) of Cai and Zhou (2012).

Since $\bar{\mathbb{P}}_{(1,0,\gamma_{-1},\lambda_{-1})}$ assumes that
$\gamma_{1}=0$, this is the distribution function of the $p$-dimensional
normal distribution with a zero mean vector and a covariance matrix

$\displaystyle\Sigma_{0}$ $\displaystyle=$
$\displaystyle\begin{pmatrix}1&0_{1\times(p-1)}\\\ 0_{(p-1)\times
1}&S_{(p-1)\times(p-1)}\end{pmatrix},$

where $S_{(p-1)\times(p-1)}=(s_{ij})\in\mathbb{R}^{(p-1)\times(p-1)}$ is a
symmetric matrix uniquely determined by $(\gamma_{-1},\lambda_{-1})$:

$\displaystyle s_{ij}=\begin{cases}1,\quad i=j,\\\
\epsilon_{np},\quad\gamma_{i+1}=\lambda_{i+1}(j+1)=1,\\\ 0,\quad\text{
otherwise}.\end{cases}$

Let

$\displaystyle\Lambda_{1}(c)$ $\displaystyle=$
$\displaystyle\big{\\{}a\in\mathbb{R}^{p}:\lambda_{1}(\theta)=a,\lambda_{-1}(\theta)=c\,\,\text{
for some }\theta\in\Theta\big{\\}}$

be the set of all possible values of the first row, $\lambda_{1}(\theta)$,
given the rest of the rows, $\lambda_{-1}(\theta)=c$. For a given
$\lambda_{-1}\equiv\lambda_{-1}(\theta)=(\lambda_{2}(\theta),\ldots,\lambda_{r}(\theta))^{T}\in\mathbb{R}^{(r-1)\times
p}$, denote $n_{\lambda_{-1}}$ be the number of columns of $\lambda_{-1}$
whose sums are equal to $2k$, and let $p_{\lambda_{-1}}=r-n_{\lambda_{-1}}$.
Then, by the definition of $\Theta$, $p_{\lambda_{-1}}$ is the number of
entries in $\lambda_{1}$ which can be either $0$ or $1$. Note that
$|\Lambda_{1}(\lambda_{-1})|=\binom{p_{\lambda_{-1}}}{k}$ and
$p_{\lambda_{-1}}\geq p/4-1$ for any $\lambda_{-1}$. Then,
$\bar{\mathbb{P}}_{(1,1,\gamma_{-1},\lambda_{-1})}$ is an average of
$\binom{p_{\lambda_{-1}}}{k}$ normal distributions with covariance matrices of
the form:

$\displaystyle\begin{pmatrix}1&r^{T}\\\ r&S_{(p-1)\times(p-1)}\end{pmatrix},$
(19)

where $\|r\|_{0}=k$ and nonzero entries of $r$ are equal to $\epsilon_{np}$.
For a given $(\gamma_{-1},\lambda_{-1})$, let $\Sigma_{1}$ and $\Sigma_{2}$ be
covariance matrices of the form (19) with the first row
$\lambda_{1}\in\Lambda_{1}(\lambda_{-1})$ and
$\lambda_{1}^{\prime}\in\Lambda_{1}(\lambda_{-1})$, respectively. Then, by the
similar arguments in page 2411 of Cai and Zhou (2012),

$\displaystyle{\mathbb{E}}_{(\gamma_{-1},\lambda_{-1})}\Big{\\{}\int\Big{(}\frac{d\bar{\mathbb{P}}_{(1,1,\gamma_{-1},\lambda_{-1})}}{d\bar{\mathbb{P}}_{(1,0,\gamma_{-1},\lambda_{-1})}}\Big{)}^{2}d\bar{\mathbb{P}}_{(1,0,\gamma_{-1},\lambda_{-1})}-1\Big{\\}}$
(20) $\displaystyle=$
$\displaystyle{\mathbb{E}}_{(\gamma_{-1},\lambda_{-1})}\left[{\mathbb{E}}_{(\lambda_{1},\lambda_{1}^{\prime})\mid\lambda_{-1}}\Big{\\{}\exp\big{(}\frac{n}{2}R_{\lambda_{1},\lambda_{1}^{\prime}}^{\gamma_{-1},\lambda_{-1}}\big{)}-1\Big{\\}}\right]$
$\displaystyle=$
$\displaystyle{\mathbb{E}}_{(\lambda_{1},\lambda_{1}^{\prime})}\left[{\mathbb{E}}_{(\gamma_{-1},\lambda_{-1})\mid(\lambda_{1},\lambda_{1}^{\prime})}\Big{\\{}\exp\big{(}\frac{n}{2}R_{\lambda_{1},\lambda_{1}^{\prime}}^{\gamma_{-1},\lambda_{-1}}\big{)}-1\Big{\\}}\right],$

where
$\lambda_{1},\lambda_{1}^{\prime}\mid\lambda_{-1}\overset{iid}{\sim}Unif\\{\Lambda_{1}(\lambda_{-1})\\}$,
$(\gamma_{-1},\lambda_{-1})\mid(\lambda_{1},\lambda_{1}^{\prime})\sim
Unif\\{\Theta_{-1}(\lambda_{1},\lambda_{1}^{\prime})\\}$,

$\displaystyle\Theta_{-1}(a_{1},a_{2})$ $\displaystyle=$
$\displaystyle\\{0,1\\}^{r-1}\times\big{\\{}c\in\Lambda_{-1}:\exists\theta_{i}\in\Theta,i=1,2\text{
such that }\lambda_{1}(\theta_{i})=a_{i},\lambda_{-1}(\theta_{i})=c\big{\\}}$

and

$\displaystyle
R_{\lambda_{1},\lambda_{1}^{\prime}}^{\gamma_{-1},\lambda_{-1}}=-\log\det\\{I_{p}-\Sigma_{0}^{-2}(\Sigma_{0}-\Sigma_{1})(\Sigma_{0}-\Sigma_{2})\\}.$
(21)

By Lemma 5.9, (20) is bounded above by

$\displaystyle{\mathbb{E}}_{J}\left[\exp\big{\\{}-n\log(1-J\epsilon_{np}^{2})\big{\\}}{\mathbb{E}}_{(\lambda_{1},\lambda_{1}^{\prime})\mid
J}\Big{\\{}{\mathbb{E}}_{(\gamma_{-1},\lambda_{-1})\mid(\lambda_{1},\lambda_{1}^{\prime})}\exp\big{(}\frac{n}{2}R_{1,\lambda_{1},\lambda_{1}^{\prime}}^{\gamma_{-1},\lambda_{-1}}\big{)}\Big{\\}}-1\right]$
(22) $\displaystyle\leq$
$\displaystyle{\mathbb{E}}_{J}\Big{[}\exp\Big{\\{}-n\log(1-J\epsilon_{np}^{2})\Big{\\}}\,\frac{3}{2}-1\Big{]},$

where $J$ is the number of overlapping nonzero entries between the first rows
of $\Sigma_{1}$ and $\Sigma_{2}$, i.e.,
$J=\lambda_{1}^{T}\lambda_{1}^{\prime}$. Note that for any $0\leq j\leq k$,

$\displaystyle{\mathbb{E}}_{J}\big{\\{}I(J=j)\mid\lambda_{-1}\big{\\}}$
$\displaystyle=$
$\displaystyle\frac{\binom{k}{j}\binom{p_{\lambda_{-1}}-k}{k-j}}{\binom{p_{\lambda_{-1}}}{k}}$
$\displaystyle=$
$\displaystyle\Big{\\{}\frac{k!}{(k-j)!}\Big{\\}}^{2}\frac{\\{(p_{\lambda_{-1}}-k)!\\}^{2}}{p_{\lambda_{-1}}!(p_{\lambda_{-1}}-2k+j)!}\frac{1}{j!}$
$\displaystyle\leq$
$\displaystyle\Big{(}\frac{k^{2}}{p_{\lambda_{-1}}-k}\Big{)}^{j},$

because
$\lambda_{1},\lambda_{1}^{\prime}\mid\lambda_{-1}\overset{iid}{\sim}Unif\\{\Lambda_{1}(\lambda_{-1})\\}$.
Then, we have

$\displaystyle{\mathbb{E}}_{J}I(J=j)$ $\displaystyle=$
$\displaystyle{\mathbb{E}}_{\lambda_{-1}}\Big{[}{\mathbb{E}}_{J}\big{\\{}I(J=j)\mid\lambda_{-1}\big{\\}}\Big{]}$
$\displaystyle\leq$
$\displaystyle{\mathbb{E}}_{\lambda_{-1}}\Big{\\{}\Big{(}\frac{k^{2}}{p_{\lambda_{-1}}-k}\Big{)}^{j}\Big{\\}}$
$\displaystyle\leq$ $\displaystyle\Big{(}\frac{k^{2}}{p/4-1-k}\Big{)}^{j}$

because $p_{\lambda_{-1}}\geq p/4-1$ for any $\lambda_{-1}$. Thus, (22) is
bounded above by

$\displaystyle\sum_{j=0}^{k}\Big{(}\frac{k^{2}}{p/4-1-k}\Big{)}^{j}\Big{[}\exp\big{\\{}-n\log(1-j\epsilon_{np}^{2})\big{\\}}\frac{3}{2}-1\Big{]}$
$\displaystyle=$
$\displaystyle\frac{1}{2}+\sum_{j=1}^{k}\Big{(}\frac{k^{2}}{p/4-1-k}\Big{)}^{j}\Big{[}\exp\big{\\{}-n\log(1-j\epsilon_{np}^{2})\big{\\}}\frac{3}{2}-1\Big{]}$
$\displaystyle\leq$
$\displaystyle\frac{1}{2}+\frac{3}{2}\sum_{j=1}^{k}\Big{(}\frac{k^{2}}{p/4-1-k}\Big{)}^{j}p^{2\nu^{2}j}$
$\displaystyle\leq$
$\displaystyle\frac{1}{2}+\frac{3C}{2}\sum_{j=1}^{k}p^{-\epsilon
j}p^{(\epsilon/2)j}$ $\displaystyle\leq$ $\displaystyle c_{2}^{2}$

for some constant $C>0$ and all sufficiently large $p$, by setting
$c_{2}^{2}=3/4<1$, where the second inequality follows from
$s_{0}^{2}=O(p^{3-\epsilon})$ and $\nu=\sqrt{\epsilon/4}$. This implies (18),
which completes the proof of (15).

#### (ii) Proof of (16)

Define a parameter space

$\displaystyle B_{2}$ $\displaystyle:=$
$\displaystyle\Big{\\{}\Sigma(\theta):\Sigma(\theta)=I_{p}+\frac{\nu}{\sqrt{n}}diag(\theta),\quad\theta\in\Theta=\\{0,1\\}^{p}\Big{\\}}$

for some small constant $\nu>0$. Then, it is easy to see that
$B_{2}\subset\mathcal{U}(s_{0},\tau_{0})$ for any $\tau_{0}>1$ and any
sufficiently large $n$. By the Assouad lemma in Cai and Zhou (2012), we have

$\displaystyle\inf_{\hat{\Sigma}}\max_{\Sigma(\theta)\in
B_{2}}2^{2}{\mathbb{E}}_{\theta}\|\hat{\Sigma}-\Sigma(\theta)\|_{F}^{2}$
$\displaystyle\geq$ $\displaystyle\min_{H(\theta,\theta^{\prime})\geq
1}\frac{\|\Sigma(\theta)-\Sigma(\theta^{\prime})\|_{F}^{2}}{H(\theta,\theta^{\prime})}\,\frac{p}{2}\,\min_{H(\theta,\theta^{\prime})=1}\|\mathbb{P}_{\theta}\wedge\mathbb{P}_{\theta^{\prime}}\|.$

For any two probability measures $\mathbb{P}$ and $\mathbb{Q}$, let
$\|\mathbb{P}-\mathbb{Q}\|_{1}=\int|p-q|d\mu$, where $p$ and $q$ ar
probability densities corresponding to $\mathbb{P}$ and $\mathbb{Q}$,
respectively, with respect to a common dominating measure $\mu$. Since
$\|\Sigma(\theta)-\Sigma(\theta^{\prime})\|_{F}^{2}=H(\theta,\theta^{\prime})\nu^{2}/n$
and
$\|\mathbb{P}_{\theta}\wedge\mathbb{P}_{\theta^{\prime}}\|=1-\|\mathbb{P}_{\theta}-\mathbb{P}_{\theta^{\prime}}\|_{1}/2$,
it suffices to prove that

$\displaystyle\|\mathbb{P}_{\theta}-\mathbb{P}_{\theta^{\prime}}\|_{1}$
$\displaystyle\leq$ $\displaystyle\frac{1}{2}$ (23)

for any $\theta,\theta^{\prime}\in\Theta$ such that
$H(\theta,\theta^{\prime})=1$.

By inequality (C.11) in Lee and Lee (2018), we have

$\displaystyle\|\mathbb{P}_{\theta}-\mathbb{P}_{\theta^{\prime}}\|_{1}$
$\displaystyle\leq$ $\displaystyle
n\,\Big{[}tr\big{\\{}\Sigma(\theta^{\prime})\Sigma(\theta)^{-1}\big{\\}}-\log\det\big{\\{}\Sigma(\theta^{\prime})\Sigma(\theta)^{-1}\big{\\}}-p\Big{]}$
$\displaystyle\equiv$ $\displaystyle
n\,\Big{[}tr\big{\\{}\Sigma(\theta)^{-1/2}D_{1}\Sigma(\theta)^{-1/2}\big{\\}}-\log\det\big{\\{}\Sigma(\theta)^{-1/2}D_{1}\Sigma(\theta)^{-1/2}+I_{p}\big{\\}}\Big{]},$

where $D_{1}=\Sigma(\theta^{\prime})-\Sigma(\theta)$. Then, by Lemma C.2 of
Lee and Lee (2018),

$\displaystyle\|\mathbb{P}_{\theta}-\mathbb{P}_{\theta^{\prime}}\|_{1}$
$\displaystyle\leq$ $\displaystyle n\,R$ $\displaystyle\leq$ $\displaystyle
n\,c\|D_{1}\Sigma(\theta)^{-1}\|_{F}^{2}\,\,\leq\,\,c\nu^{2}$

for some constant $c>0$ and any $\theta,\theta^{\prime}\in\Theta$ such that
$H(\theta,\theta^{\prime})=1$. Therefore, by taking $\nu^{2}=1/(2c)$, it shows
that (23) holds. $\blacksquare$

###### Lemma 5.9

If $s_{0}^{2}(\log p)^{3}=O(p^{2}n)$ and $s_{0}^{2}=O(p^{3-\epsilon})$ for
some small constant $\epsilon>0$, then

$\displaystyle
R_{\lambda_{1},\lambda_{1}^{\prime}}^{\gamma_{-1},\lambda_{-1}}$
$\displaystyle=$
$\displaystyle-2\log(1-J\epsilon_{np}^{2})+R_{1,\lambda_{1},\lambda_{1}^{\prime}}^{\gamma_{-1},\lambda_{-1}},$
(24)

where $R_{\lambda_{1},\lambda_{1}^{\prime}}^{\gamma_{-1},\lambda_{-1}}$ is
defined in (21), and
$R_{1,\lambda_{1},\lambda_{1}^{\prime}}^{\gamma_{-1},\lambda_{-1}}$ satisfies

$\displaystyle{\mathbb{E}}_{(\lambda_{1},\lambda_{1}^{\prime})\mid
J}\left[{\mathbb{E}}_{(\gamma_{-1},\lambda_{-1})\mid(\lambda_{1},\lambda_{1}^{\prime})}\Big{\\{}\exp\big{(}\frac{n}{2}R_{1,\lambda_{1},\lambda_{1}^{\prime}}^{\gamma_{-1},\lambda_{-1}}\big{)}\Big{\\}}\right]$
$\displaystyle\leq$ $\displaystyle\frac{3}{2}$ (25)

uniformly over all $J$.

  * Proof of Lemma 5.9

Note that Lemma 5.9 corresponds to Lemma 11 of Cai and Zhou (2012). Equation
(24) follows from equation (60) of Cai and Zhou (2012). However, to obtain
(25), Cai and Zhou (2012) assumed $p\geq n^{\beta}$ for some $\beta>1$ and
$nk\epsilon_{np}^{3}$ is sufficiently small for all large $n$. We will show
that one can still prove that (25) holds under the conditions in Lemma 5.9.

Let $A_{*}=(I_{p}-\Sigma_{0})(\Sigma_{0}-\Sigma_{1})(\Sigma_{0}-\Sigma_{2})$,
then by the same arguments used in pages 3-5 of Supplementary of Cai and Zhou
(2012), one can show that

$\displaystyle{\mathbb{E}}_{(\lambda_{1},\lambda_{1}^{\prime})\mid
J}\left[{\mathbb{E}}_{(\gamma_{-1},\lambda_{-1})\mid(\lambda_{1},\lambda_{1}^{\prime})}\Big{\\{}\exp\big{(}\frac{n}{2}R_{1,\lambda_{1},\lambda_{1}^{\prime}}^{\gamma_{-1},\lambda_{-1}}\big{)}\Big{\\}}\right]$
$\displaystyle\leq$
$\displaystyle{\mathbb{E}}_{(\lambda_{1},\lambda_{1}^{\prime})\mid
J}\left[{\mathbb{E}}_{(\gamma_{-1},\lambda_{-1})\mid(\lambda_{1},\lambda_{1}^{\prime})}\Big{\\{}\exp\big{(}Cn\,\max\\{\|A_{*}\|_{1},\|A_{*}\|_{\infty}\\}\big{)}\Big{\\}}\right]$

for some constant $C>0$, and

$\displaystyle{\mathbb{E}}_{(\lambda_{1},\lambda_{1}^{\prime})\mid
J}\left({\mathbb{E}}_{(\gamma_{-1},\lambda_{-1})\mid(\lambda_{1},\lambda_{1}^{\prime})}\Big{[}I\Big{\\{}\max(\|A_{*}\|_{1},\|A_{*}\|_{\infty})\geq
2t\,k\epsilon_{np}^{3}\Big{\\}}\Big{]}\right)$ $\displaystyle\leq$
$\displaystyle 2p\Big{(}\frac{k^{2}}{p/8-1-k}\Big{)}^{t-1}$

for every $t>2$. Thus,

$\displaystyle{\mathbb{E}}_{(\lambda_{1},\lambda_{1}^{\prime})\mid
J}\left[{\mathbb{E}}_{(\gamma_{-1},\lambda_{-1})\mid(\lambda_{1},\lambda_{1}^{\prime})}\Big{\\{}\exp\big{(}Cn\,\max\\{\|A_{*}\|_{1},\|A_{*}\|_{\infty}\\}\big{)}\Big{\\}}\right]$
(26) $\displaystyle\leq$ $\displaystyle
a+\int_{x>a}{\mathbb{E}}_{(\lambda_{1},\lambda_{1}^{\prime})\mid
J}\left({\mathbb{E}}_{(\gamma_{-1},\lambda_{-1})\mid(\lambda_{1},\lambda_{1}^{\prime})}\Big{[}I\Big{\\{}\exp\big{(}Cn\,\max\\{\|A_{*}\|_{1},\|A_{*}\|_{\infty}\\}\big{)}>x\Big{\\}}\Big{]}\right)dx$
$\displaystyle\leq$
$\displaystyle\exp\Big{(}\frac{1+2\epsilon}{\epsilon}\,2Cnk\epsilon_{np}^{3}\Big{)}+\int_{t>(1+2\epsilon)/\epsilon}2Cnk\epsilon_{np}^{3}\exp\Big{(}2Ctnk\epsilon_{np}^{3}\Big{)}\,2p\Big{(}\frac{k^{2}}{p/8-1-k}\Big{)}^{t-1}dt$
$\displaystyle\leq$
$\displaystyle\exp\Big{(}\frac{1+2\epsilon}{\epsilon}\,2Cnk\epsilon_{np}^{3}\Big{)}$
$\displaystyle+$
$\displaystyle\int_{t>(1+2\epsilon)/\epsilon}\exp\Big{\\{}\log(2p)-(t-1)\log\frac{p/8-1-k}{k^{2}}+2C(t+1)nk\epsilon_{np}^{3}\Big{\\}}dt,$
(27)

where the second inequality follows by choosing
$a=\exp\\{2Cnk\epsilon_{np}^{3}(1+2\epsilon)/\epsilon\\}$. Since $k=\lceil
c_{np}/2\rceil-1$, $c_{np}=\lceil s_{0}/p\rceil$, $\epsilon_{np}=\nu\sqrt{\log
p/n}$ with $\nu=\sqrt{\epsilon/4}$ and we assume that $s_{0}^{2}(\log
p)^{3}=O(p^{2}n)$, term (26) is less than $3/2$ for any sufficiently small
$\epsilon>0$. Thus, we complete the proof if we show that term (27) is of
order $o(1)$. Note that

$\displaystyle(t-1)\log\frac{p/8-1-k}{k^{2}}$ $\displaystyle\geq$
$\displaystyle\Big{(}1+\frac{1}{\epsilon}\Big{)}\log\frac{p/8-1-k}{k^{2}}$
$\displaystyle\geq$
$\displaystyle\Big{(}1+\frac{1}{\epsilon}\Big{)}\log\frac{p^{3}/8-p^{2}-ps_{0}}{s_{0}^{2}}+C^{\prime}$
$\displaystyle=$
$\displaystyle\Big{(}1+\frac{1}{\epsilon}\Big{)}\log\Big{\\{}\frac{p^{3}}{s_{0}^{2}}\Big{(}\frac{1}{8}-\frac{1}{p}-\frac{s_{0}}{p^{2}}\Big{)}\Big{\\}}+C^{\prime}$
$\displaystyle\geq$
$\displaystyle\Big{(}1+\frac{1}{\epsilon}\Big{)}\log(p^{\epsilon})+C^{\prime\prime}$
$\displaystyle=$ $\displaystyle(1+\epsilon)\log p+C^{\prime\prime},$

for any $t>(1+2\epsilon)/\epsilon$ and some constnats $C^{\prime}>0$ and
$C^{\prime\prime}>0$. The third inequality follows from the assumption
$s_{0}^{2}=O(p^{3-\epsilon})$. Therefore, it implies that (27) is of order
$o(1)$, which gives the desired result. $\blacksquare$

## Appendix D: Full Conditionals

The joint posterior distribution of $\bm{\Sigma}$ and $\bm{\rho}=(\rho_{jk})$
with shrinkage priors (2), (3), and (4) is proportional to

$|\bm{\Sigma}|^{-n/2}\exp\left\\{-\frac{1}{2}tr(S\bm{\Sigma}^{-1})\right\\}\prod_{j<k}\left[\exp\left\\{-\frac{\sigma_{jk}}{2\tau_{1}^{2}}\left(\frac{1-\rho_{jk}}{\rho_{jk}}\right)\right\\}\rho_{jk}^{a-1}(1-\rho_{jk})^{b-1}\right]\prod_{j=1}^{p}\exp\left\\{-\frac{\lambda}{2}\sigma_{jj}\right\\},$

and under partitions (7) and the transformation (8), the joint conditional
posterior of $\bm{u}$ and $v$ given $\bm{\rho}$ (Wang, 2015) is

$\displaystyle\pi\left(\bm{u},v\mid\bm{\rho},\bm{X}_{n}\right)\propto\exp\Bigg{\\{}-\frac{1}{2}\big{(}n\log(v)+\bm{u}^{\top}\bm{\Sigma}_{11}^{-1}\bm{S}_{11}\bm{\Sigma}_{11}^{-1}\bm{u}v^{-1}-2\bm{s}_{12}^{\top}\bm{\Sigma}_{11}^{-1}\bm{u}v^{-1}+s_{22}v^{-1}$
$\displaystyle+\bm{u}^{\top}\bm{D}^{-1}\bm{u}+\lambda\bm{u}^{\top}\bm{\Sigma}_{11}^{-1}\bm{u}+\lambda
v\big{)}\Bigg{\\}},$

where $\bm{D}=\mbox{diag}(\bm{v}_{12})$.

This gives the full conditional posteriors of $\bm{u}$ and $v$ as follows
(Wang, 2015):

$\displaystyle\pi(\bm{u}\mid v,\bm{\rho},\bm{X}_{n})$ $\displaystyle=$
$\displaystyle
N_{p-1}\left[\left\\{\bm{B}+\bm{D}^{-1}\right\\}^{-1}\bm{w},\left\\{\bm{B}+\bm{D}^{-1}\right\\}^{-1}\right],$
$\displaystyle\pi(v\mid\bm{u},\bm{\rho},\bm{X}_{n})$ $\displaystyle=$
$\displaystyle
GIG\left(1-n/2,\,\,\lambda,\,\,\bm{u}^{T}\Sigma_{11}^{-1}\bm{S}_{11}\Sigma_{11}^{-1}\bm{u}-2s_{12}^{T}\Sigma_{11}^{-1}\bm{u}+s_{22}\right),$

where
$\bm{B}=\bm{\Sigma}_{11}^{-1}\bm{S}_{11}\bm{\Sigma}_{11}^{-1}v^{-1}+\lambda\bm{\Sigma}_{11}^{-1}$
and $\bm{w}=\bm{\Sigma}_{11}^{-1}\bm{s}_{12}v^{-1}$.

Finally, to dervie the full conditional of $\bm{\rho}$, we consider a
reparametrization of $\rho_{jk}$ as

$\phi_{jk}=\frac{\rho_{jk}}{1-\rho_{jk}},$

then the shrinkage prior can be represented as follows (Armagan et al., 2011):

$\sigma_{jk}\mid\phi_{jk}\sim
N(0,\phi_{jk}\tau_{1}^{2}),\,\,\,\phi^{1/2}_{jk}\sim C^{+}(0,1),$

where $C^{+}(0,1)$ denotes a half-Cauchy distribution on $(0,\infty)$. The
full conditional distribution of $\phi_{jk}$ with an additional parameter
$\psi_{jk}$ (Carvalho et al., 2010) is given as

$\displaystyle\pi(\psi_{jk}\mid\phi_{jk},\bm{\Sigma},\bm{X}_{n})$
$\displaystyle=$ $\displaystyle Gamma(a+b,\phi_{jk}+1),$
$\displaystyle\pi(\phi_{jk}\mid\psi_{jk},\bm{\Sigma},\bm{X}_{n})$
$\displaystyle=$ $\displaystyle
GIG(a-1/2,2\psi_{jk},\sigma_{jk}^{2}/\tau_{1}^{2}).$

## References

  * Alon et al. (1999) Alon, U., Barkai, N., Notterman, D.A., Gish, K., Ybarra, S., Mack, D., Levine, A.J., 1999.  Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays.  Proc. Natl. Acad. Sci. 96, 6745–6750.
  * Anderson (2003) Anderson, T.W., 2003.  An introduction to multivariate statistical analysis.  John Wiley & Sons, Inc.
  * Armagan et al. (2011) Armagan, A., Clyde, M., Dunson, D.B., 2011.  Generalized beta mixtures of gaussians, in: Shawe-Taylor, J., Zemel, R.S., Bartlett, P.L., Pereira, F., Weinberger, K.Q. (Eds.), Advances in Neural Information Processing Systems 24. Curran Associates, Inc., pp. 523–531.
  * Armagan et al. (2013) Armagan, A., Dunson, D.B., Lee, J., 2013.  Generalized double Pareto shrinkage.  Stat. Sin. 23, 119–143.
  * Banerjee and Ghosal (2015) Banerjee, S., Ghosal, S., 2015\.  Bayesian structure learning in graphical models.  Journal of Multivariate Analysis 136, 147–162.
  * Bickel and Levina (2008) Bickel, P.J., Levina, E., 2008\.  Covariance regularization by thresholding.  The Annals of Statistics 36, 2577–2604.
  * Bien and Tibshirani (2011) Bien, J., Tibshirani, R.J., 2011\.  Sparse estimation of a covariance matrix.  Biometrika 98, 807–820.
  * Brualdi and Mellendorf (1994) Brualdi, R.A., Mellendorf, S., 1994\.  Regions in the complex plane containing the eigenvalues of a matrix.  The American mathematical monthly 101, 975–985.
  * Cai and Liu (2011) Cai, T., Liu, W., 2011.  Adaptive thresholding for sparse covariance matrix estimation.  Journal of the American Statistical Association 106, 672–684.
  * Cai and Zhou (2012) Cai, T.T., Zhou, H.H., 2012\.  Optimal rates of convergence for sparse covariance matrix estimation.  The Annals of Statistics 40, 2389–2420.
  * Carvalho et al. (2010) Carvalho, C.M., Polson, N.G., Scott, J.G., 2010.  The horseshoe estimator for sparse signals.  Biometrika 97, 465–480.
  * Castillo and Roquain (2020) Castillo, I., Roquain, É., 2020\.  On spike and slab empirical Bayes multiple testing.  The Annals of Statistics 48, 2548–2574.
  * Fisher and Sun (2011) Fisher, T.J., Sun, X., 2011\.  Improved Stein-type shrinkage estimators for the high-dimensional multivariate normal covariance matrix.  Comput. Statist. Data. Anal. 55, 1909–1918.
  * Gao and Zhou (2015) Gao, C., Zhou, H.H., 2015.  Rate-optimal posterior contraction for sparse pca.  The Annals of Statistics 43, 785–818.
  * Ghosal et al. (2000) Ghosal, S., Ghosh, J.K., Van Der Vaart, A.W., 2000.  Convergence rates of posterior distributions.  Annals of Statistics 28, 500–531.
  * Golub et al. (1999) Golub, T., Slonim, D., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov, J., Coller, H., Loh, M., Downing, J., Caligiuri, M., 1999\.  Molecular classification of cancer: class discovery and class prediction by gene expression monitoring.  Science 286, 531–536.
  * Griffin and Brown (2011) Griffin, J.E., Brown, P.J., 2011\.  Bayesian hyper-lassos with non-convex penalization.  Aust. N. Z. J. Stat. 53, 423–442.
  * Griffin and Brown (2017) Griffin, J.E., Brown, P.J., 2017\.  Hierarchical shrinkage priors for regression models.  Bayesian Anal. 12, 135–159.
  * Guo et al. (2007) Guo, Y., Hastie, T., Tibshirani, R., 2007.  Regularized linear discriminant analysis and its application in microarrays.  Biostatistics 8, 86–100.
  * Khare and Rajaratnam (2011) Khare, K., Rajaratnam, B., 2011\.  Wishart distributions for decomposable covariance graph models.  The Annals of Statistics 39, 514–555.
  * Lam and Fan (2009) Lam, C., Fan, J., 2009.  Sparsistency and rates of convergence in large covariance matrix estimation.  The Annals of Statistics 37, 4254–4278.
  * Lee and Lee (2018) Lee, K., Lee, J., 2018.  Optimal bayesian minimax rates for unconstrained large covariance matrices.  Bayesian Analysis 13, 1211–1229.
  * Lee et al. (2019) Lee, K., Lee, J., Lin, L., 2019.  Minimax posterior convergence rates and model selection consistency in high-dimensional dag models based on sparse cholesky factors.  The Annals of Statistics 47, 3413–3437.
  * Liu and Martin (2019) Liu, C., Martin, R., 2019.  An empirical $g$-wishart prior for sparse high-dimensional gaussian graphical models.  arXiv preprint arXiv:1912.03807 .
  * Martin et al. (2017) Martin, R., Mess, R., Walker, S.G., 2017.  Empirical bayes posterior concentration in sparse high-dimensional linear models.  Bernoulli 23, 1822–1847.
  * Park and Casella (2008) Park, T., Casella, G., 2008\.  The Bayesian lasso.  J. Amer. Statist. Assoc. 103, 681–686.
  * Rothman et al. (2008) Rothman, A.J., Bickel, P.J., Levina, E., Zhu, J., 2008\.  Sparse permutation invariant covariance estimation.  Electron. J. Stat. 2, 494–515.
  * Rothman et al. (2009) Rothman, A.J., Levina, E., Zhu, J., 2009.  Generalized thresholding of large covariance matrices.  Journal of the American Statistical Association 104, 177–186.
  * Silva and Ghahramani (2009) Silva, R., Ghahramani, Z., 2009\.  The hidden life of latent variables: Bayesian learning with mixed graph models.  Journal of Machine Learning Research 10, 1187–1238.
  * Song and Liang (2018) Song, Q., Liang, F., 2018.  Nearly optimal bayesian shrinkage for high dimensional regression.  arXiv preprint arXiv:1712.08964 .
  * Touloumis (2015) Touloumis, A., 2015.  Nonparametric Stein-type shrinkage covariance matrix estimators in high-dimensional settings.  Comput. Statist. Data. Anal. 83, 251–261.
  * Wang (2015) Wang, H., 2015.  Scaling it up: stochastic search structure learning in graphical models.  Bayesian Anal. 10, 351–377.
  * Zhu and Hastie (2004) Zhu, J., Hastie, T., 2004.  Classification of gene microarrays by penalized logistic regression.  Biostatistics 5, 427–443.