text: user6498: WGCNA carefully takes advantage of server parallelization in several functions, and I can switch this on with:
  enableWGCNAThreads(nThreads=28)
  
  That speeds up some functions significantly, but the two bottleneck functions in the pipeline (in my experience) are part of the "Projective k-means" operation, primarily "..k-means clustering.." and "..merging smaller clusters..".
  Unfortunately, parallelization does not work for those two functions.
  Am I missing something?  How can I parallelize these functions?  Otherwise I'm wasting HPC resources and cluster time.
  For reference, I'm working on a 28-processor HPC node with 96GB RAM, and I'm trying to run analysis on a 15,000-square correlation matrix.
  
  user851: The bad news is that, indeed, projectiveKMeans is not parallelized and I am not sure how much of it is (easily) parallelizable. The good news is that with 15k features (genes) and 96GB RAM you don't need to run preclustering at all. Just analyze the whole data set in one block.
  If you for some reason insist on splitting the data into blocks, make sure the BLAS R uses on the cluster is a fast BLAS, ideally one that can parallelize matrix operations. A lot of time in merging of small clusters is spent calculating singular value decompositions, which becomes much faster using a good BLAS. HPC clusters will often have Intel KML available, or the admins could install OpenBLAS.
  
original_text: 6498: <p>WGCNA carefully takes advantage of server parallelization in several functions, and I can switch this on with:</p>
  
  <pre><code>enableWGCNAThreads(nThreads=28)
  </code></pre>
  
  <p>That speeds up some functions significantly, but the two bottleneck functions in the pipeline (in my experience) are part of the "Projective k-means" operation, primarily <code>"..k-means clustering.."</code> and <code>"..merging smaller clusters.."</code>.</p>
  
  <p>Unfortunately, parallelization does not work for those two functions.</p>
  
  <p>Am I missing something?  How can I parallelize these functions?  Otherwise I'm wasting HPC resources and cluster time.</p>
  
  <p>For reference, I'm working on a 28-processor HPC node with 96GB RAM, and I'm trying to run analysis on a 15,000-square correlation matrix.</p>
   851: <p>The bad news is that, indeed, projectiveKMeans is not parallelized and I am not sure how much of it is (easily) parallelizable. The good news is that with 15k features (genes) and 96GB RAM you don't need to run preclustering at all. Just analyze the whole data set in one block.</p>
  
  <p>If you for some reason insist on splitting the data into blocks, make sure the BLAS R uses on the cluster is a fast BLAS, ideally one that can parallelize matrix operations. A lot of time in merging of small clusters is spent calculating singular value decompositions, which becomes much faster using a good BLAS. HPC clusters will often have Intel KML available, or the admins could install OpenBLAS.</p>
  