transformers;attention;softmax;grokking;sudden loss drop;local optimum;softmax temperature;phase transition;analysis paper
Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems