title: How to use joblib parallelbackend for fitting LightGBM?
body: <p>I am developing a model with 2M rows but the pandas dataframe is throwing memory error while reading the file. I am using python 3.6
  I used dask dataframe to read the file and tried to use scikit learn using joblib to train the model. Code worked fine but with more waiting time, high memory and cpu usage impacting other running jobs. Code below for the reference.</p>
  
  <pre><code>import dask
  import dask.dataframe as dd
  from dask.distributed import Client
  from dask_ml.model_selection import train_test_split
  import joblib
  from sklearn.model_selection import GridSearchCV
  from sklearn.metrics import roc_curve, auc
  import lightgbm as lgb
  client = Client(processes=False)
  #read data and split
  train_data= dd.read_parquet('s3://bucket/my-parquet-data')
  X_train, X_test, y_train, y_test = train_test_split(train_data.loc[:,X],
                                                      train_data.loc[:,y],
                                                      test_size=0.20,
                                                      random_state=42)
  estimator= lgb.LGBMClassifier(boosting_type='gbdt',n_jobs=-1, silent=True, importance_type='gain'
  objective='binary','is_unbalance': True)
  
  param_grid = {
  'learning_rate': [0.7, 0.1, 0.15],
  'n_estimators': [150, 200, 250],
  'max_depth': [3,4,5]
  }
  gbm = GridSearchCV(estimator, param_grid, cv=3)
  with joblib.parallel_backend('dask'):
  # Will this use each chunk to train data. So parallel processing and boosting will work fine here? Referense on [joblib][1]
       gbm.fit(X_train, y_train,
       eval_set=[(X_test, y_test)],
       eval_metric='logloss',
       early_stopping_rounds=20)
  #AUC and Curve
  #Any suggestions to avoid compute and still able to get auc and plot the curve?
  y_test=test_y.compute().ravel()
  prd_test = gbm.predict_proba(X)[:,1].compute() 
  #for curve;
  fpr, tpr, threshold = roc_curve(y_test, prd_test )
  auc_ = auc(fpr, tpr)
  </code></pre>
  
  <p>Could anyone please help me with: </p>
  
  <p>1) Is parallel-backend right way to train boosting tree? Am I missing any theory here such as if parallel processing and boosting go right here?. Will this take care of all the partitions at the backend i.e. read all to fit the model.</p>
  
  <p>2)Optimize the code to run fast and consume less memory. Any parameters to the estimator. I have used n_job=-1 for parallel run</p>
  
  <p>3) How to calculate ROC/AUC  without using compute (put arrays in-memory)? </p>
  
  <p>4) Is there any way to plot AUC without using compute and put arrays in-memory.</p>
  
  <p>Do I need the <code>.persist()</code> anywhere in between to speed up the process? I am using 3 workers with 2 cores each
  Thank you for your help.</p>