text: user9698: In single-cell RNA sequencing analysis, I am wondering how many percents explained variance by the first 50 principal components of PCA analysis is suitable for downstream KNN and umap?
  If there are lots of samples (>40) and cells (>100,000) to analyze together, there are a lot of highly_variable_genes (>4000) can be found
  import scanpy as sc
  sc.pp.highly_variable_genes(adata, flavor='cell_ranger', n_top_genes=None)
  
  The explained variance by the first 50 principal components of PCA analysis sc.pp.pca(adata, n_comps=50, use_highly_variable=True, svd_solver='arpack') is very low (<12%), increase the number of principal components does not help a lot.
  I assume that the low explained variance of PCs will make KNN and umap difficult to separate the cells well. Am I correct?
  sc.pp.neighbors(adata, n_neighbors=15)
  sc.tl.umap(adata, min_dist=0.3, spread=2.0)
  
  I can reduce the number of highly_variable_genes to get more explained variance by the first 50 principal components, but KNN and umap still can not separate the cells well.
  I checked the first 5 PCs when I reduce the number of highly_variable_genes, and it doesn't look very high:
  >>> np.cumsum(adata.uns['pca']['variance_ratio'])
  
  array([0.08419679, 0.12635055, 0.13610176, 0.14490107, 0.15326007])
  
  Any suggestions would be greatly appreciated.
  
  user3848: Edit To answer the question, you're looking for 80% variance .... from the comments the actual command is np.cumsum which changes things a lot.
  Previous response ...
  Firstly it's still good work. It wouldn't be sensible to proceed with KNN if 12% explained 50 PCs, now there are 15% in 5 PCs.
  You wrote,
  
  0.08419679, 0.12635055, 0.13610176, 0.14490107, 0.15326007
  
  PC1 ~ 0.085, PC2 ~ 0.068
  I think that you've cross checked the analysis and that is a good sign of good analysis.
  The only easy possibility is to completely tighten down on all "variable genes" parameter. The total variance for the analysis think 50% total variance rather than an idealistic 80%.
  The emergency measures ...
  
  Simply use say 10 samples
  Reducing the total number of genes, e.g. removing more variable genes looks good, anything else
  Check the input internally to make sure there is no formatting bug
  Check via Seraut
  
  The thing I would cross check is reducing the sample size a lot ... is there a stray sample(s) here that is throwing everything out? That would be thing I'd really look into. What happens to the total variance when say 20 samples, or 10 samples are used?
  Just to finalise, are there any natural population divisions in the sample that could be used? If this results in a huge jump in variance that would be an answer. The only thing that is unusual is that there are a lot of samples for this type of study (from my perspective).
  
original_text: 9698: <p>In single-cell RNA sequencing analysis, I am wondering how many percents explained variance by the first 50 principal components of PCA analysis is suitable for downstream <code>KNN</code> and <code>umap</code>?</p>
  <p>If there are lots of samples (&gt;40) and cells (&gt;100,000) to analyze together, there are a lot of <code>highly_variable_genes</code> (&gt;4000) can be found</p>
  <pre><code>import scanpy as sc
  sc.pp.highly_variable_genes(adata, flavor='cell_ranger', n_top_genes=None)
  </code></pre>
  <p>The explained variance by the first 50 principal components of PCA analysis <code>sc.pp.pca(adata, n_comps=50, use_highly_variable=True, svd_solver='arpack')</code> is very low (&lt;12%), increase the number of principal components does not help a lot.
  I assume that the low explained variance of PCs will make <code>KNN</code> and <code>umap</code> difficult to separate the cells well. Am I correct?</p>
  <pre><code>sc.pp.neighbors(adata, n_neighbors=15)
  sc.tl.umap(adata, min_dist=0.3, spread=2.0)
  </code></pre>
  <p>I can reduce the number of <code>highly_variable_genes</code> to get more explained variance by the first 50 principal components, but <code>KNN</code> and <code>umap</code> still can not separate the cells well.</p>
  <p>I checked the first 5 PCs when I reduce the number of highly_variable_genes, and it doesn't look very high:</p>
  <pre><code>&gt;&gt;&gt; np.cumsum(adata.uns['pca']['variance_ratio'])
  
  array([0.08419679, 0.12635055, 0.13610176, 0.14490107, 0.15326007])
  </code></pre>
  <p>Any suggestions would be greatly appreciated.</p>
   3848: <p><strong>Edit</strong> To answer the question, you're looking for 80% variance .... from the comments the actual command is <code>np.cumsum</code> which changes things a lot.</p>
  <p>Previous response ...
  Firstly it's still good work. It wouldn't be sensible to proceed with <code>KNN</code> if 12% explained 50 PCs, now there are 15% in 5 PCs.</p>
  <p>You wrote,</p>
  <blockquote>
  <p>0.08419679, 0.12635055, 0.13610176, 0.14490107, 0.15326007</p>
  </blockquote>
  <p>PC1 ~ 0.085, PC2 ~ 0.068</p>
  <p>I think that you've cross checked the analysis and that is a good sign of good analysis.</p>
  <p>The only easy possibility is to completely tighten down on all &quot;variable genes&quot; parameter. The total variance for the analysis think 50% total variance rather than an idealistic 80%.</p>
  <p>The emergency measures ...</p>
  <ul>
  <li>Simply use say 10 samples</li>
  <li>Reducing the total number of genes, e.g. removing more variable genes looks good, anything else</li>
  <li>Check the input internally to make sure there is no formatting bug</li>
  <li>Check via Seraut</li>
  </ul>
  <p>The thing I would cross check is reducing the sample size a lot ... is there a stray sample(s) here that is throwing everything out? That would be thing I'd really look into. What happens to the total variance when say 20 samples, or 10 samples are used?</p>
  <p>Just to finalise, are there any natural population divisions in the sample that could be used? If this results in a huge jump in variance that would be an answer. The only thing that is unusual is that there are a lot of samples for this type of study (from my perspective).</p>
  