<p>Before going into the solution I would first comment on the proposed solution of the questions. The first solution would work better compared to the second. This is because It is very hard to interpret the (probability )values of the neural network output. Closeness of the values might be caused by similarity of the classes involving(in this case a dog might look like a cat). Sometimes you may end up getting unseen classes being assigned to one of the classes with high probability.</p>
<p>Most of supervised classification machine learning algorithms are designed to map an input to one of some fixed number of classes. This type of classification is called <strong>closed world classification</strong>.<br />
E.g.</p>
<ul>
<li><strong>MNIST</strong> - handwritten digit classification</li>
<li><strong>Cat - Dog</strong> classification</li>
</ul>
<p>When classification involves some unlabeled/unknown classes, the approach is called Open-world classification. There are various papers published[<a href="https://www.cs.uic.edu/%7Eliub/publications/emnlp17-camera-ready.pdf" rel="nofollow noreferrer">1</a>, <a href="https://www.kdd.org/kdd2016/papers/files/rpp0426-feiA.pdf" rel="nofollow noreferrer">2</a>, <a href="https://arxiv.org/pdf/1801.05609.pdf" rel="nofollow noreferrer">3</a>].</p>
<p>I will explain my solution using the solution proposed by <a href="https://arxiv.org/pdf/1801.05609.pdf" rel="nofollow noreferrer">3</a>.
There are two options to apply the Open world classification(Here on I will refer to OWC) to the problem in question.</p>
<ol>
<li>Classifying all new classes as single class</li>
<li>Classifying all new classes as single class, then further grouping similar samples into single class and different samples into different classes.</li>
</ol>
<h3>1. Classifying all new classes as single class</h3>
<p>Although there could be many types of model that could fit to this type of classification(One of could be the first solution proposed by the question.) I would discusses model of  <a href="https://arxiv.org/pdf/1801.05609.pdf" rel="nofollow noreferrer">3</a>. Here the network first decides to classify or to reject the input. Ideally if the sample is from seen classes then the network will classify into one of seen classes. Other wise the network rejects. The authors of <a href="https://arxiv.org/pdf/1801.05609.pdf" rel="nofollow noreferrer">3</a> called this network Open classification network(OCN). Keras implementation of OCN could be(I've simplified the network to just focus on output of the model.</p>
<pre><code>inputs = keras.layers.Input(shape=(28, 28,1))
x = keras.layers.Conv2D(64, 3, activation=&quot;relu&quot;)(inputs)
x = keras.layers.Flatten()(x)

embedding = keras.layers.Dense(256, activation=&quot;linear&quot;, name=&quot;embedding_layer&quot;)(x)
reject_output = keras.layers.Dense(1, activaton=&quot;sigmoid&quot;, name=&quot;reject_layer&quot;)(embedding)

classification_output = keras.layers.Dense(num_of_classes, activaton=&quot;softmax&quot;, name=&quot;reject_layer&quot;)(embedding)

ocn_model = keras.models.Model(inputs=inputs, outputs=[reject_output, classification_output)
</code></pre>
<p>The model is trained in a way that jointly optimizes both <code>reject_output</code> and <code>classification_output</code> losses.</p>
<h3>2. Classifying all new classes as single class, then further grouping similar</h3>
<p>The authors of <a href="https://arxiv.org/pdf/1801.05609.pdf" rel="nofollow noreferrer">3</a> used another network to find similarity between samples. They called the network Pairwise Classification Network(PCN). PCN classifies whether two inputs are from the same classes or different classes. We can use the <code>embedding</code> of the first solution and use pairwise similarity metrics to create PCN network. In PCN the weights are shared for both inputs. This could be implemented using keras</p>
<pre><code>embedding_model = keras.layers.Sequential([
    keras.layers.Conv2D(64, 3, activation=&quot;relu&quot;, input_shape=(28, 28,1))
    keras.layers.Flatten(),
    embedding = keras.layers.Dense(256, activation=&quot;linear&quot;, name=&quot;embedding_layer&quot;)
])

input1 = keras.layers.Input(shape=(28, 28, 1))
input2 = keras.layers.Input(shape=(28, 28, 1))

embedding1 = embedding_model(input1)
embedding2 = embedding_model(input2)

merged = keras.layers.Concatenate()([embedding1, embedding2])
output = keras.layers.Dense(1, activation=&quot;sigmoid&quot;)(merged)

pcn_model = keras.models.Model(inputs=[input1, input2], outputs=output)
</code></pre>
<p>PCN model will be trained to reduce the distance from the same and increase the distance between different classes.</p>
<p>After the PCN network is trained auto-encoder is trained to learn useful representations from the unseen classes. Then Clustering algorithm is used to group(cluster) unseen classes by using PCN model as distance function.</p>
Classes never seen before on Deep Learning Models
<p>I have a basic question. Supposedly I am training an image classifier for cats and dogs. But I need an extra functionality. If an image does not belong to any of the category, how do I get to know it. Some of the options I was thinking of were:</p>

<ol>
<li>Instead of 2 neurons I add a 3rd Neuron to the last layer. And get my training labels y as a one hot encoding of 3 labels, 3rd for being not in either of cat or dog class. I will use some random examples for my 3rd class.</li>
<li>I will use only 2 neurons and using some probability threshold I will use it to tell which class should my image belong. </li>
</ol>

<p>However I do not think any of the methods is viable.</p>

<p>Can anyone suggest I a good technique to classify images which do not belong to my training category?</p>