<p>Have a look at the PCA implementation in dask-ML, <a href="https://ml.dask.org/modules/generated/dask_ml.decomposition.PCA.html" rel="nofollow noreferrer">https://ml.dask.org/modules/generated/dask_ml.decomposition.PCA.html</a>,
this might already work for your case, as it uses the tsqr algorithm (<a href="https://arxiv.org/abs/1301.1071" rel="nofollow noreferrer">https://arxiv.org/abs/1301.1071</a>)</p>
How to use incremental PCA on dask dataframe?
<p>I am using a dask dataframe which can not be loaded directly into the memory because of the size of it. I want to perform dimentionality reduction of top of using incremental PCA.</p>

<p>My dataframe is sparse in nature, so the question is can I perform it and if yes then how to do so.</p>

<pre><code>image_features_df.head(3)
   feat1   feat2   feat3  ...  feat25087    feat25088   fid   selling_price
0   0.0    0.0     0.0    ...   0.0          0.0         2       269.00
4   0.3    0.1     0.0    ...   0.0          0.8         26      1720.00    
6   0.8    0.0     0.0    ...   0.0          0.1         50      18145.25
</code></pre>

<p>The above is a view of my dataframe. I want the output to have 95% cumulative varience. How to do so?</p>

<p>My dataframe has 100,000 rows and 25088 columns so please tell a solution which is memory efficient.</p>