title: How can a machine learning model handle unseen data and unseen label?
body: <p>I am trying to solve a text classification problem. I have a limited number of labels that capture the category of my text data. If the incoming text data doesn't fit any label, it is tagged as 'Other'. In the below example, I built a text classifier to classify text data as 'breakfast' or 'italian'. In the test scenario, I included couple of text data that do not fit into the labels that I used for training. This is the challenge that I'm facing. Ideally, I want the model to say - 'Other' for 'i like hiking' and 'everyone should understand maths'. How can I do this? </p>
  
  <pre><code>import numpy as np
  from sklearn.pipeline import Pipeline
  from sklearn.feature_extraction.text import CountVectorizer
  from sklearn.naive_bayes import MultinomialNB
  from sklearn.feature_extraction.text import TfidfTransformer
  
  X_train = np.array(["coffee is my favorite drink",
                      "i like to have tea in the morning",
                      "i like to eat italian food for dinner",
                      "i had pasta at this restaurant and it was amazing",
                      "pizza at this restaurant is the best in nyc",
                      "people like italian food these days",
                      "i like to have bagels for breakfast",
                      "olive oil is commonly used in italian cooking",
                      "sometimes simple bread and butter works for breakfast",
                      "i liked spaghetti pasta at this italian restaurant"])
  
  y_train_text = ["breakfast","breakfast","italian","italian","italian",
                  "italian","breakfast","italian","breakfast","italian"]
  
  X_test = np.array(['this is an amazing italian place. i can go there every day',
                     'i like this place. i get great coffee and tea in the morning',
                     'bagels are great here',
                     'i like hiking',
                     'everyone should understand maths'])
  
  classifier = Pipeline([
      ('vectorizer', CountVectorizer()),
      ('tfidf', TfidfTransformer()),
      ('clf', MultinomialNB())])
  
  classifier.fit(X_train, y_train_text)
  predicted = classifier.predict(X_test)
  proba = classifier.predict_proba(X_test)
  print(predicted)
  print(proba)
  
  ['italian' 'breakfast' 'breakfast' 'italian' 'italian']
  [[0.25099411 0.74900589]
   [0.52943091 0.47056909]
   [0.52669142 0.47330858]
   [0.42787443 0.57212557]
   [0.4        0.6       ]]
  </code></pre>
  
  <p>I consider the 'Other' category as noise and I cannot model this category. </p>