I am trying to classify (37 possible classes) a dataset that has 9 features and 900,000 occurrences. I have tried a couple different algorithms but the one that gave the lowest logarithmic loss was the naive bayes with 5.4.

The accuracy of this same naive bayes was 7% so I knew there was room for improvement.

After adding a new feature and that went through 2 different feature selection (chi2, and elasticnet) I was able to get the accuracy up to 23% but log loss score increased to 6.2?

How was such a large increase in accuracy accompanies by a decrease in the models precision?

  • $\begingroup$ Do you actually mean that the model was accurate only 7% of the time and then improved to being accurate only 23% of the time? Is this a binary or multiclass problem? $\endgroup$ – jld Jun 23 '15 at 21:44
  • $\begingroup$ @Chaconne yes that is what I mean and it is a multiclass problem. $\endgroup$ – grasshopper Jun 23 '15 at 21:57

The two metrics measure slightly different things.

Naive Bayes tries to learn how to approximate $P(\textrm{class } | \textrm{ data})$ from your training data. To classify new data, the algorithm takes each example and computes $P(\textrm{class}=c\ | \textrm{ data})$ for each possible class $c$. We label each test example with the $c$ that maximizes that probability.

The accuracy metric only concerns itself with final classification output. For each example in the test set, you get a 1 if the largest $c$ is the correct class, and a zero otherwise. Average these values together and you'll get an accuracy measurement. In other words, accuracy is 0/1 loss, scaled into a percentage. Log-loss, however, considers the probability $P(\textrm{class}=c\ | \textrm{ data})$ directly. It's essentially the sum of the negative log-likelihoods of the true classes given your model. Kaggle defines it slightly differently, but in either case, the actual probabilities matter, not just which class's probability is the largest.

Here's an example. Suppose your first model is stupid but noncommittal. For 9 examples, it estimates $P(\textrm{true class})=0.45$ and $P(\textrm{other}) =0.55$, while it does the opposite for one example. This yields an accuracy of 10% and a log-loss of $\frac{1}{10}\big(9\log(0.45) + 1\log(0.55)\big) = 0.78$.

The second model is better but also more decisive, even when it's wrong. Suppose it calculates $P(\textrm{true class})=0.9$ for two examples, and $P(\textrm{true class})=0.1$ for the remaining eight. This classifier has an accuracy of 20%, but a log-loss of 1.8, which mimics your situation exactly.


Your Answer

