title: is there any way to split Spark Dataset in given logic
body: <p>i am looking for Spark Dataset split application which is similar to bellow mentioned logic. </p>
  
  <pre><code>&gt;&gt;&gt; import pandas as pd
  &gt;&gt;&gt; import numpy as np
  
  &gt;&gt;&gt; df1 = pd.DataFrame(np.random.randn(10, 4), columns=['a', 'b', 'c', 'd'])
  &gt;&gt;&gt; df1
            a         b         c         d
  0 -0.398502 -1.083682  0.850632 -1.443868
  1 -2.124333  1.093590 -0.338162 -1.414035
  2  0.753560  0.600687 -0.998277 -2.094359
  3 -0.635962 -0.291226  0.428961  1.158153
  4 -0.900506 -0.545226 -0.448178 -0.567717
  5  0.112911  0.351649  0.788940  2.071541
  6 -0.358625  0.500367  1.009819 -1.139761
  7  1.003608  0.246925  0.225138 -0.586727
  8  0.355274 -0.540685  1.482472  0.364989
  9  3.089610 -1.415088 -0.072107 -0.203137
  &gt;&gt;&gt;
  
  &gt;&gt;&gt; mask = df1.applymap(lambda x: x &lt;-0.7)
  &gt;&gt;&gt;
  &gt;&gt;&gt; mask
         a      b      c      d
  0  False   True  False   True
  1   True  False  False   True
  2  False  False   True   True
  3  False  False  False  False
  4   True  False  False  False
  5  False  False  False  False
  6  False  False  False   True
  7  False  False  False  False
  8  False  False  False  False
  9  False   True  False  False
  &gt;&gt;&gt; mask.any(axis=1)
  0     True
  1     True
  2     True
  3    False
  4     True
  5    False
  6     True
  7    False
  8    False
  9     True
  dtype: bool
  &gt;&gt;&gt; df1 = df1[-mask.any(axis=1)]
  &gt;&gt;&gt; df1
            a         b         c         d
  3 -0.635962 -0.291226  0.428961  1.158153
  5  0.112911  0.351649  0.788940  2.071541
  7  1.003608  0.246925  0.225138 -0.586727
  8  0.355274 -0.540685  1.482472  0.364989
  &gt;&gt;&gt;
  </code></pre>
  
  <p>In spark i gone though <code>df.filter</code> but its trying pick only matched ,  but in my case i need to filter(remove) data into 3 -4 level. only one level i shown above.  which is kind of filtering. </p>