Geometrically, softmax is constant along diagonals: this is the dimension that is eliminated, and corresponds to the softmax output being independent of a translation in the input scores (a choice of 0 score). One can normalize input scores by assuming that the sum is zero (subtract the average: formula_63 where formula_64), and then the softmax takes the hyperplane of points that sum to zero, formula_65, to the open simplex of positive values that sum to 1formula_66, analogously to how the exponent takes 0 to 1, formula_67 and is positive.