Department of Biostatistics and Computational Biology, University of Rochester Medical Center, Rochester, NY 14642; Depts. of Biostatistics and Computational Biology, Neuroscience, University of Rochester Medical Center, Depts. of Brain and Cognitive Sciences, Biomedical Engineering, University of Rochester, Rochester, NY 14642
David Skrill, Samuel Norman-Haignere
language modeling;temporal integration;transformers;timescales;model interpretation
Large language models transition from integrating across position-yoked, exponential windows to structure-yoked, power-law windows