Carnegie Mellon University; Microsoft Research NYC; University of Pennsylvania
Bingbin Liu, Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, Cyril Zhang
Transformers;language models;hallucinations;long-range dependencies;generalization;extrapolation;out-of-distribution
Exposing Attention Glitches with Flip-Flop Language Modeling