<h2>Actually, Softmax functions are already used deep within neural networks, in certain cases, when dealing with differentiable memory and with attention mechanisms!</h2>

<p>Softmax layers can be used within neural networks such as in <a href="https://arxiv.org/pdf/1410.5401v2.pdf" rel="noreferrer">Neural Turing Machines (NTM)</a> and an improvement of those which are <a href="http://www.nature.com/articles/nature20101.epdf?author_access_token=ImTXBI8aWbYxYQ51Plys8NRgN0jAjWel9jnR3ZoTv0MggmpDmwljGswxVdeocYSurJ3hxupzWuRNeGvvXnoO8o4jTJcnAyhGuZzXJ1GEaD-Z7E6X_a9R-xqJ9TfJWBqz" rel="noreferrer">Differentiable Neural Computer (DNC)</a>. </p>

<p>To summarize, those architectures are <a href="https://github.com/guillaume-chevalier/LSTM-Human-Activity-Recognition" rel="noreferrer">RNNs/LSTMs</a> which have been modified to contain a differentiable (neural) memory matrix which is possible to write and access through time steps. </p>

<p>Quickly explained, the softmax function here enables a normalization of a fetch of the memory and other similar quirks for content-based addressing of the memory. About that, I really liked <a href="http://distill.pub/2016/augmented-rnns/" rel="noreferrer">this article</a> which illustrates the operations in an NTM and other recent RNN architectures with interactive figures. </p>

<p>Moreover, Softmax is used in attention mechanisms for, say, machine translation, such as in <a href="https://arxiv.org/pdf/1409.0473.pdf" rel="noreferrer">this paper</a>. There, the Softmax enables a normalization of the places to where attention is distributed in order to "softly" retain the maximal place to pay attention to: that is, to also pay a little bit of attention to elsewhere in a soft manner. However, this could be considered like to be a mini-neural network that deals with attention, within the big one, as explained in the paper. Therefore, it could be debated whether or not Softmax is used only at the end of neural networks.</p>

<p>Hope it helps!</p>

<p>Edit - More recently, it's even possible to see Neural Machine Translation (NMT) models where only attention (with softmax) is used, without any RNN nor CNN: <a href="http://nlp.seas.harvard.edu/2018/04/03/attention.html" rel="noreferrer">http://nlp.seas.harvard.edu/2018/04/03/attention.html</a></p>
Why use softmax only in the output layer and not in hidden layers?
<p>Most examples of neural networks for classification tasks I've seen use the a softmax layer as output activation function. Normally, the other hidden units use a sigmoid, tanh, or ReLu function as activation function. Using the softmax function here would - as far as I know - work out mathematically too.</p>

<ul>
<li>What are the theoretical justifications for not using the softmax function as hidden layer activation functions?</li>
<li>Are there any publications about this, something to quote?</li>
</ul>