title: KeyError when training a model with pyspark.ml on AWS EMR with data from s3 bucket
body: <p>I am training a machine learning model with pyspark.ml on .json data from an s3 bucket on AWS EMR in a JupyterLab notebook. The bucket is not mine, but I think access works fine because data preprocessing, feature engineering etc. works fine. But when I call the <code>cv.fit(training_data)</code> function, the training process runs until it almost finishes (indicated by the status bar), but then throws an error:</p>
  
  <pre><code>Exception in thread cell_monitor-64:
  Traceback (most recent call last):
    File "/opt/conda/lib/python3.7/threading.py", line 917, in _bootstrap_inner
      self.run()
    File "/opt/conda/lib/python3.7/threading.py", line 865, in run
      self._target(*self._args, **self._kwargs)
    File "/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py", line 178, in cell_monitor
      job_binned_stages[job_id][stage_id] = all_stages[stage_id]
  KeyError: 6571
  </code></pre>
  
  <p>I could not find any information on this error yet. What is going on? </p>
  
  <p>This is my pipeline:</p>
  
  <pre><code>train, test = clean_df.randomSplit([0.8, 0.2], seed=42)
  
  va1 = VectorAssembler(inputCols="vars", outputCol="vars")
  
  scaler = StandardScaler(inputCol="to_scale", outputCol="scaled_features")
  
  va2 = VectorAssembler(inputCols=["more_vars","scaled_features"], outputCol="features")
  
  gbt = GBTClassifier()   
  
  pipeline = Pipeline(stages=[va1, scaler,va2,gbt])
  
  paramGrid = ParamGridBuilder()\
      .addGrid(gbt.maxDepth, [2, 5])\
      .addGrid(gbt.maxIter, [10, 100])\
      .build() 
  
  crossval = CrossValidator(estimator=pipeline,
                            estimatorParamMaps=paramGrid,
                            evaluator=MulticlassClassificationEvaluator(metricName='f1'),
                            numFolds=3)
  
  cvModel = crossval.fit(train)
  </code></pre>
  
  <p>Second, I have a hunch that I might by resolved in Python 3.8; can I install Python 3.8 on EMR? </p>