<blockquote>
  <p>The choice of the softmax function seems <strong>somehow arbitrary</strong> as there are many other possible normalizing functions. It is thus unclear why the log-softmax loss would perform better than other loss alternatives.</p>
</blockquote>

<p>From "<strong>An Exploration of Softmax Alternatives Belonging to the Spherical Loss Family</strong>" <a href="https://arxiv.org/abs/1511.05042" rel="noreferrer">https://arxiv.org/abs/1511.05042</a></p>

<p>The authors explored some other functions among which are Taylor expansion of <code>exp</code> and so called spherical softmax and found out that sometimes they might perform better than usual <code>softmax</code>.</p>
Why use softmax as opposed to standard normalization?
<p>In the output layer of a neural network, it is typical to use the softmax function to approximate a probability distribution:</p>

<p><img src="https://i.stack.imgur.com/r1MZm.png" alt="enter image description here"></p>

<p>This is expensive to compute because of the exponents. Why not simply perform a Z transform so that all outputs are positive, and then normalise just by dividing all outputs by the sum of all outputs?</p>