title: PySpark - Error using randomSplit on Dataframe
body: <p>I have created a DataFrame with my data to run some Machine Learning experiments. I'm trying to split it into training and test sets by using the randomSplit() function, but it is giving me some exceptions which I can't figure out the reason. My code is similar to this:</p>
  
  <pre><code>Features = ['A', 'B', 'C', 'D', 'E', 'aVec', 'bVec', 'cVec', 'dVec']
  
  vec = VectorAssembler(inputCols = Features, outputCol = 'features')
  df = vec.transform(df)
  df = df.select("features", "Target")
  
  (train, test) = df.randomSplit([0.8, 0.2])
  
  print(df.count())
  
  print(train.count())
  print(test.count())
  </code></pre>
  
  <p>The letters inside 'Features' represent numeric features, and the *Vec elements represent OneHotEncoding vectors (created using OneHotEncoding() function of pyspark).</p>
  
  <p>When Spark reaches print(train.count()) it launches the following exception:</p>
  
  <pre><code>Py4JJavaError: An error occurred while calling o2274.count.
  : org.apache.spark.SparkException: Job aborted due to stage failure: 
  Task 5 in stage 1521.0 failed 1 times, most recent failure: Lost task 
  5.0 in stage 1521.0 (TID 122477, localhost, executor driver): 
  java.lang.IllegalAccessError: tried to access field 
  org.apache.spark.sql.execution.BufferedRowIterator.partitionIndex from 
  class 
  </code></pre>
  
  <p>The print on df works well, so I'm thinking that randomSplit is corrupting my data somehow. </p>
  
  <p>I did a small test, and if I remove any of the OneHotEncoding Vectors it starts to work for some reason. (I removed 'aVec', for example, and it worked). The problem does not seem to be about a specific column because I could remove any of them (if I ran my code with Features = ['aVec', 'bVec', 'cVec'] or Features = ['bVec', 'cVec', 'dVec'] it will work, but not with Features = ['aVec', 'bVec', 'cVec', 'dVec']).</p>
  
  <p>Is there a reason for this error I'm getting? </p>