title: Training Error is Lower than Testing error in a Random Forest Model
body: <p>I have been working on machine learning model and I am confused about which model to choose or if there is any other technique I should try. I am working on Random Forest to predict the propensity to convert with a higly imbalanced data set. The class balance for the target variable is given below.</p>
  
  <pre><code>   label   count                                                                
  0    0.0  1,021,095
  1    1.0    4459
  </code></pre>
  
  <p>The two model I trained was using UpSampling and then using Undersampling. Below are the codes I am using for Upsampling and Undersampling</p>
  
  <pre><code>train_initial, test = new_data.randomSplit([0.7, 0.3], seed = 2018)
  train_initial.groupby('label').count().toPandas()
  test.groupby('label').count().toPandas()
  
  #Sampling Techniques --- Should be done one of these
  #Upsampling ----
  df_class_0 = train_initial[train_initial['label'] == 0]
  df_class_1 = train_initial[train_initial['label'] == 1]
  df_class_1_over = df_class_1.sample(True, 100.0, seed=99)
  train_up = df_class_0.union(df_class_1_over)
  train_up.groupby('label').count().toPandas()
  
  #Down Sampling
  stratified_train = train_initial.sampleBy('label', fractions={0: 3091./714840, 1: 1.0}).cache()
  stratified_train.groupby('label').count().toPandas()
  </code></pre>
  
  <p>Below is how I am training my model</p>
  
  <pre><code>labelIndexer = StringIndexer(inputCol='label',
                               outputCol='indexedLabel').fit(new_data)
  
  
  featureIndexer = VectorIndexer(inputCol='features',
                                 outputCol='indexedFeatures',
                                 maxCategories=2).fit(new_data)
  
  from pyspark.ml.classification import RandomForestClassifier
  rf_model = RandomForestClassifier(labelCol="indexedLabel", featuresCol="indexedFeatures")
  
  labelConverter = IndexToString(inputCol="prediction", outputCol="predictedLabel",
                                 labels=labelIndexer.labels)
  
  # Chain indexers and tree in a Pipeline
  pipeline = Pipeline(stages=[labelIndexer, featureIndexer, rf_model, labelConverter])
  
  # Search through random forest maxDepth parameter for best model
  paramGrid = ParamGridBuilder() \
      .addGrid(rf_model.numTrees, [ 200, 400,600,800,1000]) \
      .addGrid(rf_model.impurity,['entropy','gini']) \
      .addGrid(rf_model.maxDepth,[2,3,4,5]) \
      .build()
  
  
  # Set up 5-fold cross validation
  crossval = CrossValidator(estimator=pipeline,
                            estimatorParamMaps=paramGrid,
                            evaluator=BinaryClassificationEvaluator(),
                            numFolds=5)    
  
  train_model = crossval.fit(train_up/stratified_train)
  </code></pre>
  
  <p>Below are the results from both the methods</p>
  
  <pre><code>#UpSampling - Training                                 
  Train Error = 0.184633
  precision: 0.8565508112679312
  recall: 0.6597217024736883                                            
  auroc: 0.9062348758176568
  f1 : 0.7453609484359377
  
  #Upsampling - Test                                
  Test Error = 0.0781619                             
  precision: 0.054455645977569946
  recall: 0.6503868471953579
  auroc: 0.8982212236597943
  f1 : 0.10049688048716704
  
  #UnderSampling - Training                               
  Train Error = 0.179293           
  precision: 0.8468290542023261
  recall: 0.781807131280389
  f1 : 0.8130201200884863                                          
  auroc: 0.9129391668636556
  
  #UnderSamping - Test                               
  Test Error = 0.147874
  precision: 0.034453223699706645
  recall: 0.778046421663443
  f1 : 0.06598453935901905
  auroc: 0.8989720777537427
  </code></pre>
  
  <p>Referring to various articles on StackOverflow I understand that if the test error is lower than the train error there is likely to be error in implementation. However I am not quite sure where am I going error in order to train my models. Also, which sampling is better to use in the case of such an highly imbalanced class. If I do undersampling I am worried if there would be a loss of information. </p>
  
  <p>I was hoping if someone could please help me out with this model and help me to clear my doubts.</p>
  
  <p>Thanks a lot in advance !!</p>