<p>In the context of autoencoders the input and output of the model is the same. So, if the input values are in the range [0,1] then it is acceptable to use <code>sigmoid</code> as the activation function of last layer. Otherwise, you need to use an appropriate activation function for the last layer (e.g. <code>linear</code> which is the default one).</p>

<p>As for the loss function, it comes back to the values of input data again. If the input data are <s>only</s> between zeros and ones <s>(and not the values between them)</s>, then <code>binary_crossentropy</code> is acceptable as the loss function. Otherwise, you need to use other loss functions such as <code>'mse'</code> (i.e. mean squared error) or <code>'mae'</code> (i.e. mean absolute error). Note that in the case of input values in range <code>[0,1]</code> you can use <code>binary_crossentropy</code>, as it is usually used (e.g. <a href="https://blog.keras.io/building-autoencoders-in-keras.html" rel="noreferrer">Keras autoencoder tutorial</a> and <a href="https://arxiv.org/abs/1708.08487" rel="noreferrer">this paper</a>). However, don't expect that the loss value becomes zero since <code>binary_crossentropy</code> does not return zero when both prediction and label are not either zero or one (no matter they are equal or not). <a href="https://www.youtube.com/watch?v=xTU79Zs4XKY" rel="noreferrer">Here</a> is a video from <a href="http://www.dmi.usherb.ca/~larocheh/index_en.html" rel="noreferrer">Hugo Larochelle</a> where he explains the loss functions used in autoencoders (the part about using <code>binary_crossentropy</code> with inputs in range [0,1] starts at <a href="https://youtu.be/xTU79Zs4XKY?t=330" rel="noreferrer">5:30</a>)</p>

<p>Concretely, in your example, you are using the MNIST dataset. So by default the values of MNIST are integers in the range [0, 255]. Usually you need to normalize them first:</p>

<pre><code>trainX = trainX.astype('float32')
trainX /= 255.
</code></pre>

<p>Now the values would be in range [0,1]. So <code>sigmoid</code> can be used as the activation function and either of <code>binary_crossentropy</code> or <code>mse</code> as the loss function.</p>

<hr>

<p><strong>Why <code>binary_crossentropy</code> can be used even when the true label values (i.e. ground-truth) are in the range [0,1]?</strong></p>

<p>Note that we are trying to minimize the loss function in training. So if the loss function we have used reaches its minimum value (which may not be necessarily equal to zero) when prediction is equal to true label, then it is an acceptable choice. Let's verify this is the case for binray cross-entropy which is defined as follows:</p>

<pre><code>bce_loss = -y*log(p) - (1-y)*log(1-p)
</code></pre>

<p>where <code>y</code> is the true label and <code>p</code> is the predicted value. Let's consider <code>y</code> as fixed and see what value of <code>p</code> minimizes this function: we need to take the derivative with respect to <code>p</code> (I have assumed the <code>log</code> is the natural logarithm function for simplicity of calculations):</p>

<pre><code>bce_loss_derivative = -y*(1/p) - (1-y)*(-1/(1-p)) = 0 =&gt;
                      -y/p + (1-y)/(1-p) = 0 =&gt;
                      -y*(1-p) + (1-y)*p = 0 =&gt;
                      -y + y*p + p - y*p = 0 =&gt;
                       p - y = 0 =&gt; y = p
</code></pre>

<p>As you can see binary cross-entropy have the minimum value when <code>y=p</code>, i.e. when the true label is equal to predicted label and this is exactly what we are looking for. </p>
How does binary cross entropy loss work on autoencoders?
<p>I wrote a vanilla autoencoder using only <code>Dense</code> layer. 
Below is my code:</p>



<pre class="lang-py prettyprint-override"><code>iLayer = Input ((784,))
layer1 = Dense(128, activation='relu' ) (iLayer)
layer2 = Dense(64, activation='relu') (layer1)
layer3 = Dense(28, activation ='relu') (layer2)
layer4 = Dense(64, activation='relu') (layer3)
layer5 = Dense(128, activation='relu' ) (layer4)
layer6 = Dense(784, activation='softmax' ) (layer5)
model = Model (iLayer, layer6)
model.compile(loss='binary_crossentropy', optimizer='adam')

(trainX, trainY), (testX, testY) =  mnist.load_data()
print ("shape of the trainX", trainX.shape)
trainX = trainX.reshape(trainX.shape[0], trainX.shape[1]* trainX.shape[2])
print ("shape of the trainX", trainX.shape)
model.fit (trainX, trainX, epochs=5, batch_size=100)
</code></pre>

<h2>Questions:</h2>

<p>1) <code>softmax</code> provides probability distribution. Understood. This means, I would have a vector of 784 values with probability between 0 and 1. For example [ 0.02, 0.03..... upto 784 items], summing all 784 elements provides 1. </p>

<p>2) I don't understand how the binary crossentropy works with these values. Binary cross entropy is for two values of output, right?</p>