title: parallelize each iteration in MCMC
body: <p>Suppose that I am using MCMC to sample from some distribution whose full conditionals can be written as:
  $$f(\beta|\alpha_1,\ldots,\alpha_{K}),$$
  $$g_s(\alpha_{s}|\beta) \mbox{ for }s=1,\ldots,K,$$
  and the <strong>computation of sampling is dominated by $g_s(\alpha_{s}|\beta)$'s</strong>.</p>
  
  <p>Apparently we can first sample each $g_s(\alpha_{s}|\beta)$ separately (in parallel) within each iteration and collect $\alpha_{s}$'s, and then sample $f(\beta|\alpha_1,\ldots,\alpha_{K})$. We can do this in each iteration of the MCMC.</p>
  
  <p>My understanding is that there is a lot of communication (collecting $\alpha_{s}$'s and then sampling $f(\beta|\alpha_1,\ldots,\alpha_{K})$) going on. Hence the parallelization needs not be very efficient. </p>
  
  <p>I was wondering to how much extent do I need to worry about the communication in-efficiency in this scenario. Is there a better way to improve the sampling efficiency? Any suggestions would be really welcome.</p>