title: Multiclass classification problem with loose (not quite accurate) labeling. How to approach?
body: <p>I'm looking for alt. ways to to build multiclass classification problem described below. The problem with using regular classification algorithms like Decision Tree or Logistic Regression is that they consider one value only to be True Positive, while in my data some classification are loosely done, and by that I mean instead of one single right label, 2 or more could be kindof &quot;right&quot;, some of them just a bit more accurate than the others, but only one is always chosen. Here is example to demonstrate what I meant:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Feature</th>
  <th>Label</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1. &quot;My credentials do not work&quot;</td>
  <td>&quot;Can't login&quot;</td>
  </tr>
  <tr>
  <td>2. &quot;I do not get email with confirmation code&quot;</td>
  <td>&quot;Two Factor Verification Issue&quot;</td>
  </tr>
  <tr>
  <td>3. &quot;Do not get a receipt&quot;</td>
  <td>&quot;Billing issue&quot;</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>Feature #2 is classified as &quot;Two Factor Verification Issue&quot; in the training set, but could be classified as &quot;Can't login&quot;, which is Ok too. The classification was done by human in the past, and it was up to human how to label the feature, and someone did really classified &quot;2-factor verification&quot; issues as just &quot;can't login&quot;. The dataset is huge and re-labeling is not possible. Due to such labeling <strong>ambiguity</strong> training score comes out pretty lame.</p>
  <p>I think about ways to better measure quality of a training: What if algorithm offers 2-3 options of classification outcome, and if at least one of them matches defined label, we count it as a successful guess. However a gap in my knowledge of machine learning algorithms do not let me figure out how to do that. Any suggestions? Thanks</p>