<p>If you have some unlabeled training data, you could add a <em>dustbin class</em> that contains all your unlabeled data. In your example this class would have the interpretation "not one of the colors green, blue or red". This approach is described in detail in <a href="http://arxiv.org/abs/1511.03719" rel="nofollow">http://arxiv.org/abs/1511.03719</a></p>
scikit classifier with unknown prediction
<p>I'm planning to use Python Scikit to do some text classification, and was planning to use using TfidfVectorizer and MultinomialNB.</p>

<p>but I realized that MultinomialNB will always predict my sample into an existing (known) category. </p>

<p>for example, if I have:</p>

<pre><code>category A: trained with sample "this is green"
category B: trained with sample "this is blue"
category C: trained with sample "this is red"
</code></pre>

<p>and I try to predict: <code>"this is yellow"</code></p>

<p>it will give me <code>category A</code> (or any other, because the probablity is the same for all categories in this case).</p>

<p>my question is: is there a classifier that would give me "unknown" (or none, or false, or error) for the test case above?</p>

<p>I would like to know when my test case could not be predicted with the given training set.</p>

<p>I think I could check if <code>my_classifier.predict_proba(X_test))</code> returns an array with all equal or close values (in this example case: <code>[[ 0.33333333  0.33333333  0.33333333]]</code>). </p>

<p>actually, I would have to check if the values are close to their defaults, because the probabilities might not be the same for each category :)</p>

<p>so... any better approach or... is there a classifier with some confidence threshold I could use?</p>