title: How to use functions from sklearn into pyspark
body: <p>I have a training set with 201,917 rows, 3 features and 1 target. My aim is to calculate the strength of the relationship of the individual features with the target. My choice of method for this is <code>sklearn.feature_selection.mutual_info_regression</code> because it works for continuous variables and can detect non-linear relationships better than the counterpart <code>sklearn.feature_selection.f_regression</code>. This is the line I tried to run -</p>
  <pre><code>feature_selection.mutual_info_regression(trainPD[['feature_1']],trainPD['target'])
  </code></pre>
  <p>Now the problem is if I run <code>sklearn.feature_selection.mutual_info_regression</code> in Colab, the system crashes. Hence my idea was to shift to pyspark. But <code>pyspark.ml</code> does not have support for <code>sklearn.feature_selection.mutual_info_regression</code>. So what are my options to use <code>sklearn.feature_selection.mutual_info_regression</code> in <code>pyspark</code>?</p>
  <p>I am not sure if <code>pandas_udf</code> will help because here it is not the traditional <code>pd.Series -&gt; pd.Series</code> conversion where <code>pyspark</code> parallelization works.</p>