Title: How and Why We Detect Distribution Shift: Critical Analysis of Methods and Benchmarks. Abstract: Detecting test-time distribution shift has emerged as a key capability for safely deployed machine learning models, with the question being tackled under various guises in recent years. In this paper, we aim to provide a consolidated view of the two largest sub-fields within the community: open-set recognition (OSR) and out-of-distribution detection (OOD). In particular, we aim to provide rigorous empirical analysis of different methods across settings and provide actionable takeaways for practitioners and researchers. Concretely, we make the following contributions:
(i) For the first time, we perform rigorous cross-evaluation between state-of-the-art methods in the OOD and OSR settings and identify a strong correlation between the performances of methods for them;
(ii) We propose a new, large-scale benchmark setting which we suggest better disentangles the problem tackled by OOD and OSR;
(iii) We thoroughly examine SOTA methods for OOD and OSR on our large-scale benchmark; 
and (iv) Finally, we find that the best performing method on previous benchmarks struggles on our large-scale benchmark, while magnitude-aware scoring rules consistently show promise.