<p>This isn't a python problem, per se. It seems to be more of a machine learning/data validation/data segmentation question, IMO.  That being said, you are correct in thinking that you must parallelize your work, but it matters in what ways you do it.  There are things like 8-bit quantization and model parallelism in your model that you may look into to help you get at what you're after: training a model on large datasets, in a timely manner, without sacrificing data quality or fidelity.  </p>

<p>Here is a blog post about quantization: <a href="https://petewarden.com/2016/05/03/how-to-quantize-neural-networks-with-tensorflow/" rel="nofollow noreferrer">https://petewarden.com/2016/05/03/how-to-quantize-neural-networks-with-tensorflow/</a></p>

<p>Here is a blog post about model parallelism and 8-bit quantization from Tim Dettmers' Blog: <a href="http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/" rel="nofollow noreferrer">http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/</a></p>

<p>and the associated paper: <a href="https://arxiv.org/pdf/1511.04561.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1511.04561.pdf</a></p>

<p>Though you will want to keep in mind that, depending on FP operations on your GPU, you may not see substantial benefit from this route: <a href="https://blog.inten.to/hardware-for-deep-learning-part-3-gpu-8906c1644664" rel="nofollow noreferrer">https://blog.inten.to/hardware-for-deep-learning-part-3-gpu-8906c1644664</a></p>

<p>HTH and YMMV.</p>

<p>Also, you may want to look into data folding, but can't remember the details nor the paper I read at this point in time.  I'll land this here to remember once I do though.  </p>
Gaussian Mixture Models: Difference between Spark MLlib and scikit-learn
<p>I'm trying to use Gaussian Mixture models on a sample of a dataset.
I used both<code>MLlib</code> (with <code>pyspark</code>) and <code>scikit-learn</code> and get very different results, the <code>scikit-learn</code> one looking more realistic. </p>

<pre><code>from pyspark.mllib.clustering import GaussianMixture as SparkGaussianMixture
from sklearn.mixture import GaussianMixture
from pyspark.mllib.linalg import Vectors
</code></pre>

<p><strong>Scikit-learn</strong>:</p>

<pre><code>local = pd.DataFrame([ x.asDict() for x in df.sample(0.0001).collect() ])
model1 = GaussianMixture(n_components=3)
model1.fit([ [x] for x in local['field'].tolist() ])

model1.means_
array([[7.56123598e+00],
   [1.32517410e+07],
   [3.96762639e+04]])

model1.covariances_
array([[[6.65177423e+00]],
   [[1.00000000e-06]],
   [[8.38380897e+10]]])
</code></pre>

<p><strong>MLLib</strong>:</p>

<pre><code>model2 = SparkGaussianMixture.train(
    sc.createDataFrame(local).rdd.map(lambda x: Vectors.dense(x.field)),
    k=3,
    convergenceTol=1e-4,
    maxIterations=100
)

model2.gaussians
[MultivariateGaussian(mu=DenseVector([28736.5113]), sigma=DenseMatrix(1, 1, [1094083795.0001], 0)),
 MultivariateGaussian(mu=DenseVector([7839059.9208]), sigma=DenseMatrix(1, 1, [38775218707109.83], 0)),
 MultivariateGaussian(mu=DenseVector([43.8723]), sigma=DenseMatrix(1, 1, [608204.4711], 0))]
</code></pre>

<p>However, I'm interested in running the entire dataset through the model which I'm afraid would require parallelizing (and hence use MLlib) to get results in finite time. Am I doing anything wrong / missing something?</p>

<p><strong>Data</strong>:</p>

<p>The complete data has an extremely long tail and looks like:
<a href="https://i.stack.imgur.com/RxS3T.png" rel="noreferrer"><img src="https://i.stack.imgur.com/RxS3T.png" alt="enter image description here"></a></p>

<p>whereas the data has a clearly normal dist ceneterd somewhere closer to one clustered by <code>scikit-learn</code>:</p>

<p><a href="https://i.stack.imgur.com/1ZAb1.png" rel="noreferrer"><img src="https://i.stack.imgur.com/1ZAb1.png" alt="enter image description here"></a></p>

<p>I am using Spark 2.3.0 (AWS EMR).</p>

<p>Edit: Initialization params:</p>

<pre><code>local = pd.DataFrame([ x.asDict() for x in df.sample(0.0001).collect() ])
model1 = GaussianMixture(n_components=3, init_params='random')
model1.fit([ [x] for x in local['field'].tolist() ])

model1.means_
array([[2.17611913e+04],
   [8.03184505e+06],
   [7.56871801e+00]])

model1.covariances_
rray([[[1.01835902e+09]],
   [[3.98552130e+13]],
   [[6.95161493e+00]]])
</code></pre>