 <p>This is common phenomena occurring if the class frequencies are unbalanced, e.g. nearly all samples belong to one class. For examples if 80% of your samples belong to class "No", then classifier will often tend to predict "No" because such a trivial prediction reaches the highest overall accuracy on your train set. </p> <p>In general, when evaluating the performance of a binary classifier, you should not only look at the overall accuracy. You have to consider other metrics such as the ROC Curve, class accuracies, f1 scores and so on.</p> <p>In your case you can use sklearns <a href="http://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html" rel="nofollow">classification report</a> to get a better feeling what your classifier is actually learning:</p> <pre><code>from sklearn.metrics import classification_report print(classification_report(y_test, model_1.predict(X_test))) print(classification_report(y_test, model_2.predict(X_test))) </code></pre> <p>It will print the precision, recall and accuracy for every class. </p> <p>There are three options on how to reach a better classification accuracy on your class "Yes"</p> <ul> <li>use sample weights, you can increase the importance of the samples of the "Yes" class thus forcing the classifier to predict "Yes" more often</li> <li>downsample the "No" class in the original X to reach more balanced class frequencies </li> <li>upsample the "Yes" class in the original X to reach more balanced class frequencies</li> </ul>