Bayesian Structure Learning in Graphical Models using Shrinkage priors
Sayantan Banerjee
math.ST stat.ME stat.TH

	
Channel Extrapolation for FDD Massive MIMO: Procedure and Experimental Results
    
    Thomas Choi1, François Rottenberg1, Jorge Gomez-Ponce1, Akshay Ramesh1, Peng Luo1
    Jianzhong Zhang2, and Andreas F. Molisch1
    1University of Southern California, Los Angeles, CA, USA
    2Samsung Research America, Richardson, TX, USA

     ^⋆  Laboratoire d'Informatique et des Systèmes, Université de Toulon, Aix Marseille Univ, CNRS UMR 7020, LIS, Marseille, France

boizot@univ-tln.fr; 

^†  Department of Mathematics, Lehigh University, Bethlehem, PA, USA. 

lus219@lehigh.edu; 

 

This research has been partially supported by the ANR SRGI (reference ANR-15-CE40-0018). 


This is a post-peer-review, pre-copyedit version of an article published in Journal of Mathematical Imaging and Vision (JIMIV). The final authenticated version is available online at: <http://dx.doi.org/10.1007/s10851-020-00990-5>

   
      
   

 
 

====================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





§ INTRODUCTION

We consider the problem of learning the structure of an undirected graphical model corresponding to a p-dimensional Gaussian random variable based on an iid sample of size n, where p can be much larger than n. A Gaussian graphical model captures the conditional independence structure of the underlying random variable, with absence of an edge signifying that the corresponding components of the random variable are conditionally independent given the rest. Thus the sparsity structure of the graphical model is exactly given by the sparsity structure of the precision matrix (inverse covariance matrix) of the random variable. 

Standard statistical procedures like the maximum likelihood estimator performs poorly or even does not exist in cases where the dimension p is large. Regularized estiamtors or penalty based estimators have been proposed in this regard to tackle the high-dimensional situation under assumptions of sparsity. Bayesian techniques in this direction include putting sparse or spike and slab based priors on individual elements of the precision matrix. 

In this work, we focus on learning the structure of a Gaussian graphical model through estimation of the precision matrix using continuous shrinkage priors. In the next section, we present the model assumptions along with specifying the prior distributions followed by evaluating the posterior distribution for the various parameters along with a sampling scheme for the same. We also establish some theoretical guarantees of our method by deriving the posterior convergence rates of the distribution of the precision matrix. 	

	


§ MODEL ASSUMPTIONS AND PRIOR DISTRIBUTION

Consider multivariate Gaussian data X_1,…, X_n iid∼ N_p(0, Σ), where Σ is a p-dimensional positive definite matrix. Let Ω = Σ^-1 denote the corresponding inverse covariance matrix or the precision matrix. Here we consider a high-dimensional situation such that p ≫ n. Suppose the true precision matrix is sparse, that is, they belong to the following class of positive definite matrices:

    𝒰(s_p) = {Ω∈ℳ_p^+: #(ω_ij≠ 0) ≤ s_p, i<j = 1,…,p },

ℳ_p^+ being the cone of positive definite matrices of dimension p.
	
	
We propose the following prior distribution on the elements of Ω = ((ω_ij)).
	
	
    ω_ii   ∼    1,  i = 1,…,p 
    ω_ij   ind∼    N(0, ψ_jϕ_j^2τ^2),  i < j = 1,…,p,
    ψ_ij   ind∼    Exp(1/2),  i < j = 1,…,p,
    ϕ   ∼    Dir(a,…,a), 
    τ   ∼    Gamma(ν a, 1/2), ν = p  2.

	
The above prior distribution is motivated by the Dirichlet-Laplace shrinkage priors introduced by <cit.> for the sparse Gaussian mean problem. The above prior is a global-local shrinkage prior in the sense that the parameter τ^2 induces a global shrinkage while ψ_j,ϕ_j offering deviations in shrinkage locally for individual parameters.
		


§ POSTERIOR DISTRIBUTION AND SAMPLING SCHEME

In this section, we provide the posterior distribution of the precision matrix Ω and devise a sampling scheme for the parameters. The conditional posterior density of Ω is given by

    p(Ω| X, ψ, ϕ, τ) ∝{det(Ω)}^n/2exp{-1/2tr(SΩ)}∏_i < jexp{-ω_ij^2/2ψ_ijϕ_ij^2τ^2}.

	
We partition the precision matrix as

    Ω = [ Ω_-p,-p  ω_-p,p; ω_-p,p'    ω_pp ], 
    S = [ S_-p,-p  s_-p,p; s_-p,p'    s_pp ].
	
Also define Λ = ((λ_ij)), where λ_ij = ψ_ijϕ_ij^2. Then partition Λ as

    Λ = [ Λ_-p,-p  λ_-p,p; λ_-p,p'    λ_pp ].

Then, we have,

    p(ω_-p,p, ω_pp|Ω_-p,-p,X,Λ,τ)    ∝    (ω_pp - ω_-p,p'Ω_-p,-p^-1ω_-p,p)^n/2
          ×exp{-s_-p,p'ω_-p,p - s_ppω_pp/2 - ω_-p,p'(Λ^*τ^2)^-1ω_-p,p/2 },

where Λ^* = diag(λ_-p,p).
Let θ = ω_-p,p, η = ω_pp - ω_-p,p'Ω_-p,-p^-1ω_-p,p.
Then,

    p(θ, η|Ω_-p,-p, X,Λ, τ)    ∝   η^n/2exp[-1/2{s_ppη + θ's_ppΩ_-p,-p^-1θ + θ'(Λ^*τ^2)^-1θ + 2s_-p,p'θ}] 
       ∼    Gamma(n/2 + 1, s_pp/2)N(-As_-p,p,A),

where A = {s_ppΩ_-p,-p^-1 + (Λ^*τ^2)^-1}^-1.

So simulation of θ and η can be done easily. For the rest of the parameters, we follow the same Gibbs sampler as proposed by B<cit.>, that is, 
Simulate 
    ψ̃_-i,i|ϕ,τ, ω∼ iG(ϕ_-i,iτ/|ω_-i,i|,1),

and then let ψ_-i,i = 1/ψ̃_-i,i.

Simulate

    T_ij∼ giG(a - 1,1,2 |ω_ij|),

and then set ϕ_ij = T_ij/∑_i<jT_ij.

Finally, simulate 

    τ|ϕ,ω∼ giG(ν a - ν, 1, 2∑(ω_ij/ϕ_ij)),

where ν = p  2.

In the above, iG denotes the inverse Gaussian distribution and giG denotes the generalised inverse Gaussian distribtion.




§ POSTERIOR CONVERGENCE RATE

In this section, we establish some theoretical guarantees of our proposed method. In particular, we show that under certain sparsity assumptions, the posterior distribution of Ω converges to the true precision matrix. We also derive the posterior convergence rates.





 §.§ Estimating prior concentration

	
	Following <cit.>, we have,
	
    P(|ω_ij| < δ) ≥ 1 - Clog(1/δ)/Γ(a),

	for some constant C > 0.
	Let us consider the set
	
    ℬ(p_Ω_0,ϵ_n) = {p_Ω: K(p_Ω_0,p_Ω) ≤ϵ_n^2,  V(p_Ω_0,p_Ω) ≤ϵ_n^2 }.

	Following <cit.>, under assumptions on the eigenvalues of precision matrices being bounded away from 0 and infinity, we have,
	
    ℬ(p_Ω_0,ϵ_n) ⊃{p_Ω: Ω - Ω_0_∞≤ cϵ_n/p }.

	
	Now, we have, for the choice of a = p  2^-1,
	
    Π(Ω - Ω_0 _∞≤ cϵ_n/p) ≳ (cϵ_n/p)^p+s_p(1 - C_1log(p/cϵ_n)/p^2)^p_0.

	Matching with the prior concentration rate gives,
	
    (p + s_p)(log p + logϵ_n^-1) + p_0 log(1 - C_1(log p + logϵ_n^-1)/p^2) ≍ nϵ_n^2.

Here we need to check the rate ϵ_n, which comes out to be n^-1/2(p+s_p)^1/2(log n)^1/2.
	
	

 §.§ Choosing the sieve

	The Dirichlet-Laplace prior is a shrinkage prior and does not set the value of any off-diagonal element of the precision matrix to be exactly zero. In this situation, we consider the sieve 𝒫_n to be the space of all densities p_Ω such that |supp_δ(Ω)|, where 
	
    supp_δ(Ω)  = {(i,j): |ω_ij| > δ, i < j = 1,…,p }
 satisfies
	
    |supp_δ(Ω)| < r < 1/2p  2,

	for suitably chosen threshold δ,
	and each entry of Ω is at most L in absolute value.
	
	Now, from Theorem 3.2 in <cit.>, we have, for s_p ≳log(p) and choice of a = 1/p^2, and for δ_p = s_p/p^2,
	
    lim_n →∞E_Ω_0P(|supp_δ_p(Ω)| > Ms_p |X) = 0,

	for some constant M > 0.
	The above result will take care of a part (the size of the support mentioned above) of controlling the probability of the complement of the chosen sieve. For the other part (maximum absolute value of the elements), we can show that,
	
    Π(|ω_ij| > L) ≤M'/Γ(a){C' - log(1 - e^-2L)},

	where M' and C' are constants independent of L. It follows that the rate obtained using the prior concentration matches the one obtained using the above metric entropy calculations.
	
	The metric entropy using the sieve can be verified in similar lines with <cit.>, so as to get the posterior convergence rate as ϵ_n = n^-1/2(p+s_p)^1/2(log n)^1/2. 
	
	apalike
	
	
	
