title: How to integrate spark.ml pipeline fitting and hyperparameter optimisation in AWS Sagemaker?
body: <p>Here is a high-level picture of what I am trying to achieve: I want to train a LightGBM model with spark as a compute <a href="https://microsoft.github.io/SynapseML/docs/features/lightgbm/about/" rel="nofollow noreferrer">backend</a>, all in SageMaker using their Training Job api.
  To clarify:</p>
  <ol>
  <li>I have to use LightGBM in general, there is no option here.</li>
  <li>The reason I need to use spark compute backend is because the training with the current dataset does not fit in memory anymore.</li>
  <li>I want to use SageMaker Training job setting so I could use SM Hyperparameter optimisation job to find the best hyperparameters for LightGBM. While LightGBM spark interface itself does offer some hyperparameter tuning capabilities, it does not offer Bayesian HP tuning.</li>
  </ol>
  <p>Now, I know the general approach to running custom training in SM: build a container in a certain way, and then just pull it from ECR and kick-off a training job/hyperparameter tuning job through <code>sagemaker.Estimator</code> <a href="https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html" rel="nofollow noreferrer">API</a>. Now, in this case SM would handle resource provisioning for you, would create an instance and so on. What I am confused about is that essentially, to use spark compute backend, I would need to have an EMR cluster running, so the SDK would have to handle that as well. However, I do not see how this is possible with the API above.</p>
  <p>Now, there is also that thing called <a href="https://sagemaker-pyspark.readthedocs.io/en/latest/" rel="nofollow noreferrer">Sagemaker Pyspark SDK</a>. However, the provided <code>SageMakerEstimator</code> API from that package does not support on-the-fly cluster configuration either.</p>
  <p>Does anyone know a way how to run a Sagemaker training job that would use an EMR cluster so that later the same job could be used for hyperparameter tuning activities?</p>
  <p>One way I see is to run an EMR cluster in the background, and then just create a regular SM estimator job that would connect to the EMR cluster and do the training, essentially running a spark driver program in SM Estimator job.</p>
  <p>Has anyone done anything similar in the past?</p>
  <p>Thanks</p>