title: low training score in MLPClassifier (and other classifiers) of scikit
body: <p>(Update: <a href="https://stackoverflow.com/a/45152761/1361529">Posted</a> final findings as a separate answer)</p>
  
  <p>I am getting started with trying to understand how to use scikit models for training. I've experimented with well known datasets like iris, MNIST etc - they are all well structured data, ready to be used. This is the first time I am trying to build a model out of raw data on my own and the results are less than desirable.</p>
  
  <p>The data I chose to use is <a href="https://data.ny.gov/Transportation/Motor-Vehicle-Crashes-Vehicle-Information-Three-Ye/xe9x-a24f" rel="nofollow noreferrer">NHSTA's crash data</a> for the last 3 years. </p>
  
  <p>Here is a snapshot of the data, to give you an idea of the fields without having to download the data.</p>
  
  <p><a href="https://i.stack.imgur.com/LEDcw.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/LEDcw.png" alt="Snapshot of crash data columns"></a></p>
  
  <p>My first experiment is simple - try and build a model that given "License State Code" and "Age", try and predict the gender (M or F). </p>
  
  <pre><code>import pandas as pd
  import matplotlib.pyplot as plt
  import numpy as np
  from sklearn.cluster import KMeans
  from sklearn.decomposition import PCA
  from sklearn.preprocessing import LabelEncoder
  from sklearn.preprocessing import StandardScaler
  from sklearn.model_selection import train_test_split
  from sklearn.ensemble import RandomForestClassifier
  from sklearn import metrics
  from sklearn.neural_network import MLPClassifier
  from sklearn.model_selection import learning_curve
  from sklearn.model_selection import ShuffleSplit
  import tensorflow.contrib.learn as skflow
  from tensorflow.contrib.learn.python.learn.estimators import run_config
  from sklearn.svm import SVC
  import pickle, seaborn
  
  
  def plot_learning_curve(estimator, title, X, y, ylim=None, cv=None,
                          n_jobs=1, train_sizes=np.linspace(.1, 1.0, 5)):
      #http://scikit-learn.org/stable/auto_examples/model_selection/plot_learning_curve.html
      plt.figure()
      plt.title(title)
      if ylim is not None:
          plt.ylim(*ylim)
      plt.xlabel("Training examples")
      plt.ylabel("Score")
      train_sizes, train_scores, test_scores = learning_curve(
          estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)
      train_scores_mean = np.mean(train_scores, axis=1)
      train_scores_std = np.std(train_scores, axis=1)
      test_scores_mean = np.mean(test_scores, axis=1)
      test_scores_std = np.std(test_scores, axis=1)
      plt.grid()
  
      plt.fill_between(train_sizes, train_scores_mean - train_scores_std,
                       train_scores_mean + train_scores_std, alpha=0.1,
                       color="r")
      plt.fill_between(train_sizes, test_scores_mean - test_scores_std,
                       test_scores_mean + test_scores_std, alpha=0.1, color="g")
      plt.plot(train_sizes, train_scores_mean, 'o-', color="r",
               label="Training score")
      plt.plot(train_sizes, test_scores_mean, 'o-', color="g",
               label="Cross-validation score")
  
      plt.legend(loc="best")
      plt.show()
  
  #MAIN
  
  crashes = pd.read_csv("crashes.csv", nrows=100000)
  
  # drop useless cols
  
  crashes.drop(["Year","Case Individual ID", "Case Vehicle ID", "Transported    
  By", "Injury Location", "Role Type"],axis=1, inplace=True)
  
  crashes = crashes [pd.notnull(crashes['Age'])]
  crashes = crashes[crashes.Age &gt;= 10 ] # There are ages &lt; 10 - likely junk data. I don't think they drive
  
  
  # lets drop rows that are empty
  crashes = crashes [pd.notnull(crashes['License State Code'])]
  crashes = crashes [pd.notnull(crashes['Injury Severity'])]
  crashes = crashes [pd.notnull(crashes['Safety Equipment'])]
  crashes = crashes [pd.notnull(crashes['Sex'])]
  
  # converts text fields to numerical values
  le = LabelEncoder()
  crashes = crashes[crashes.columns[:]].apply(le.fit_transform)
  crashes = crashes._get_numeric_data()
  
  # lets plot a heat map to show correlation
  corr = crashes.corr()
  ax = seaborn.heatmap (corr, xticklabels=corr.columns.values,    
  yticklabels=corr.columns.values, annot=True)
  plt.setp( ax.xaxis.get_majorticklabels(), rotation=45 )
  plt.setp( ax.yaxis.get_majorticklabels(), rotation=-45 )
  plt.show()
  
  crashes_train, crashes_test = train_test_split(crashes, test_size = 0.2)
  Y_train = crashes_train['Sex']
  X_train =  crashes_train[[ 'Age',  'License State Code']]
  Y_test = crashes_test['Sex']
  X_test =  crashes_test[[ 'Age', 'License State Code']]
  
  
  names_train  = crashes_train.columns.values
  
  print "train size ",len (X_train)
  print "test size",len (X_test)
  #
  # cls = RandomForestClassifier(verbose = True)
  #
  cls = MLPClassifier(hidden_layer_sizes=(10,10,10), max_iter=500, alpha=1e-4,
                    solver='sgd', verbose=10, tol=1e-4, random_state=1,
                    learning_rate_init=0.01)
  
  #cls = tf.contrib.learn.DNNClassifier(feature_columns=feats, 
  #                                               hidden_units=[50, 50, 50], 
  #                                              n_classes=3)
  
  #
  #
  
  #cls = SVC(verbose = True)
  
  print "Fitting..."
  cls.fit(X_train, Y_train)
  
  plot_learning_curve(cls,"Crash Learning", X_train, Y_train)
  
  
  print("Training set score: %f" % cls.score(X_train, Y_train))
  print("Test set score: %f" % cls.score(X_test, Y_test))
  </code></pre>
  
  <p>I've tried multiple models (from RandomForest, to SVC to MLP etc) - they all come up with a training score of approx 0.56 and a loss of 0.6x</p>
  
  <p>Finally, here is the plot that is being generated for MLP in the current config:
  <a href="https://i.stack.imgur.com/MgCfi.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/MgCfi.png" alt="enter image description here"></a></p>
  
  <p>Here is the plot when I switch to RandomForest. <a href="https://i.stack.imgur.com/fsxZw.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/fsxZw.png" alt="enter image description here"></a></p>
  
  <p>It looks like the score <em>decreases</em> in RandomForest, but overall it ends similar to MLP. What am I doing wrong and how do I improve this approach? Thanks</p>
  
  <p><strong>Edit</strong>: Based on both answers below, I did a heat map of the correlation between all columns (after dropping the obviously useless ones) - thats pretty bad, but is this the right approach? I could do a PCA as well, but if the basic inter-field correlation is so poor, does it indicate the dataset is largely useless to mine for predictions?</p>
  
  <p><a href="https://i.stack.imgur.com/mCLJ7.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mCLJ7.png" alt="heatmap vis &lt;code&gt;df.corr&lt;/code&gt;"></a></p>