Deep Neural Networks Tend To Extrapolate Predictably
Conventional wisdom suggests that neural network predictions tend to be
unpredictable and overconfident when faced with out-of-distribution (OOD)
inputs. Our work reassesses this assumption for neural networks with
high-dimensional inputs. Rather than extrapolating in arbitrary ways, we
observe that neural network predictions often tend towards a constant value as
input data becomes increasingly OOD. Moreover, we find that this value often
closely approximates the optimal constant solution (OCS), i.e., the prediction
that minimizes the average loss over the training data without observing the
input. We present results showing this phenomenon across 8 datasets with
different distributional shifts (including CIFAR10-C and ImageNet-R, S),
different loss functions (cross entropy, MSE, and Gaussian NLL), and different
architectures (CNNs and transformers). Furthermore, we present an explanation
for this behavior, which we first validate empirically and then study
theoretically in a simplified setting involving deep homogeneous networks with
ReLU activations. Finally, we show how one can leverage our insights in
practice to enable risk-sensitive decision-making in the presence of OOD
inputs.
Katie Kang, Amrith Setlur, Claire Tomlin, Sergey Levine
# Deep Neural Networks Tend To Extrapolate Predictably

###### Abstract

Conventional wisdom suggests that neural network predictions tend to be unpredictable and overconfident when faced with out-of-distribution (OOD) inputs. Our work reassesses this assumption for neural networks with high-dimensional inputs. Rather than extrapolating in arbitrary ways, we observe that neural network predictions often tend towards a constant value as input data becomes increasingly OOD. Moreover, we find that this value often closely approximates the optimal constant solution (OCS), i.e., the prediction that minimizes the average loss over the training data without observing the input. We present results showing this phenomenon across 8 datasets with different distributional shifts (including CIFAR10-C and ImageNet-R, S), different loss functions (cross entropy, MSE, and Gaussian NLL), and different architectures (CNNs and transformers). Furthermore, we present an explanation for this behavior, which we first validate empirically and then study theoretically in a simplified setting involving deep homogeneous networks with ReLU activations. Finally, we show how one can leverage our insights in practice to enable risk-sensitive decision-making in the presence of OOD inputs.

+
Footnote †: Code: [https://github.com/katiekang1998/cautious_extrapolation](https://github.com/katiekang1998/cautious_extrapolation)

+
Footnote †: Code: [https://github.com/katiekang1998/cautious_extrapolation](https://github.com/katiekang1998/cautious_extrapolation)

## 1 Introduction

The prevailing belief in machine learning posits that deep neural networks behave erratically when presented with out-of-distribution (OOD) inputs, often yielding predictions that are not only incorrect, but incorrect with _high confidence_[19, 37]. However, there is some evidence which seemingly contradicts this conventional wisdom - for example, Hendrycks and Gimpel [24] show that the softmax probabilities outputted by neural network classifiers actually tend to be _less confident_ on OOD inputs, making them surprisingly effective OOD detectors. In our work, we find that this softmax behavior may be reflective of a more general pattern in the way neural networks extrapolate: as inputs diverge further from the training distribution, a neural network's predictions often converge towards a _fixed_ constant value. Moreover, this constant value often approximates the best prediction the network can produce without observing any inputs, which we refer to as the optimal constant solution (OCS). We call this the "reversion to the OCS" hypothesis:

_Neural networks predictions on high-dimensional OOD inputs tend to revert towards the optimal constant solution_.

In classification, the OCS corresponds to the marginal distribution of the training labels, typically a high-entropy distribution. Therefore, the our hypothesis posits that classifier outputs should become higher-entropy as the input distribution becomes more OOD, which is consistent with the findings in Hendrycks and Gimpel [24]. Beyond classification, to the best of our knowledge, we are the first to present and provide evidence for the "reversion to the OCS" hypothesis in its full generality. Our experiments show that the amount of distributional shift correlates strongly with the distance between model outputs and the OCS across 8 datasets, including both vision and NLP domains, 3 loss functions, and for both CNNS and transformers.
Having made this observation, we set out to understand why neural networks have a tendency to behave this way. Our empirical analysis reveals that the feature representations corresponding to OOD inputs tend to have smaller norms than those of in-distribution inputs, leading to less signal being propagated from the input. As a result, neural network outputs from OOD inputs tend to be dominated by the input-independent parts of the network (e.g., bias vectors at each layer), which we observe to often map closely to the OCS. We also theoretically analyze the extrapolation behavior of deep homogeneous networks with ReLU activations, and derived evidence which supports this mechanism in the simplified setting.

Lastly, we leverage our observations to propose a simple strategy to enable risk-sensitive decision-making in the face of OOD inputs. The OCS can be viewed as a "backup default output" to which the neural network reverts when it encounters novel inputs. If we design the loss function such that the OCS aligns with the desirable cautious behavior as dictated by the decision-making problem, then the neural network model will automatically produce cautious decisions when its inputs are OOD. We describe a way to enable this alignment, and empirically demonstrate that this simple strategy can yield surprisingly good results in OOD selective classification.

In summary, our key contributions are as follows. First, we present the observation that neural networks often exhibit a predictable pattern of extrapolation towards the OCS, and empirically illustrate this phenomenon for 8 datasets with different distribution shifts, 3 loss functions, and both CNNs and transformers. Second, we provide both empirical and theoretical analyses to better understand the mechanisms that lead to this phenomenon. Finally, we make use of these insights to propose a simple strategy for enabling cautious decision-making in face of OOD inputs. Although we do not yet have a complete characterization of precisely when, and to what extent, we can rely on "reversion to the OCS" to occur, we hope our observations will prompt further investigation into this phenomenon.

## 2 Related Work

A large body of prior works have studied various properties of neural network extrapolation. One line of work focuses on the failure modes of neural networks when presented with OOD inputs, such as poor generalization and overconfidence [50, 18, 42, 5, 30]. Other works have noted that neural networks are ineffective in capturing epistemic uncertainty in their predictions [39, 32, 35, 19, 13], and that a number of techniques can manipulate neural networks to produce incorrect predictions with high confidence [48, 37, 40, 22]. However, Hendrycks et al. [25] observed that neural networks assign lower maximum softmax probabilities to OOD than to in-distribution point, meaning neural networks

Figure 1: A summary of our observations. On in-distribution samples (top), neural network outputs tend to vary significantly based on input labels. In contrast, on OOD samples (bottom), we observe that model predictions tend to not only be more similar to one another, but also gravitate towards the optimal constant solution (OCS). We also observe that OOD inputs tend to map to representations with smaller magnitudes, leading to predictions largely dominated by the (constant) network biases, which may shed light on why neural networks have this tendency.
may actually exhibit less confidence on OOD inputs. Our work supports this observation, while further generalizing it to arbitrary loss functions. Other lines of research have explored the influence of architectural decisions on generalization [55, 56, 7, 53], the relationship between in-distribution and OOD performance [34, 2, 3], and the behavior of neural network representations under OOD conditions [52, 28, 41]. While our work also analyzes representations in the context of extrapolation, our focus is on understanding the mechanism behind "reversion to the OCS", which differs from the aforementioned works.

Our work also explores risk-sensitive decision-making using selective classification as a testbed. Selective classification is a well-studied problem, and various methods have been proposed to enhance selective classification performance [15, 12, 6, 38, 8, 54]. In contrast, our aim is not to develop the best possible selective classification approach, but rather to providing insights into the effective utilization of neural network predictions in OOD decision-making.

## 3 Reversion to the Optimal Constant Solution

In this work, we will focus on the widely studied covariate shift setting [17, 47]. Formally, let the training data \(\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{N}\) be generated by sampling \(x_{i}\sim P_{\text{train}}(x)\) and \(y_{i}\sim P(y|x_{i})\). At test time, we query the model with inputs generated from \(P_{\text{OOD}}(x)\neq P_{\text{train}}(x)\), whose ground truth labels are generated from the same conditional distribution, \(P(y|x)\), as that in training. We will denote a neural network model as \(f_{\theta}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{m}\), where \(d\) and \(m\) are the dimensionalities of the input and output, and \(\theta\in\Theta\) represents the network weights. We will focus on settings where \(d\) is high-dimensional. The neural network weights are optimized by minimizing a loss function \(\mathcal{L}\) using gradient descent, \(\hat{\theta}=\arg\min_{\theta\in\Theta}\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}(f _{\theta}(x_{i}),y_{i})\).

Figure 2: Neural network predictions from training with cross entropy and Gaussian NLL on MNIST (top 3 rows) and CIFAR10 (bottom 3 rows). The models were trained with 0 rotation/noise, and evaluated on increasingly OOD inputs consisting of the digit 6 for MNIST, and of automobiles for CIFAR10. The blue plots represent the average model prediction over the evaluation dataset. The orange plots show the OCS associated with each model. We can see that as the distribution shift increases (going left to right), the network predictions tend towards the OCS (rightmost column).
### Main Hypothesis

In our experiments, we observed that as inputs become more OOD, neural network predictions tend to revert towards a constant prediction. This means that, assuming there is little label shift, model predictions will tend to be more similar to one another for OOD inputs than for the training distribution. Furthermore, we find that this constant prediction is often similar to the optimal constant solution (OCS), which minimizes the training loss if the network is constrained to ignore the input. The OCS can be interpreted as being the maximally cautious prediction, producing the class marginal in the case of the cross-entropy loss, and a high-variance Gaussian in the case of the Gaussian NLL. More precisely, we define the OCS as

\[f^{*}_{\text{constant}}=\operatorname*{arg\,min}_{f\in\mathbb{R}^{m}}\frac{1 }{N}\sum_{1\leq i\leq N}\mathcal{L}(f,y_{i}).\]

Based on our observations, we hypothesize that **as the likelihood of samples from \(P_{\text{OOD}}(x)\) under \(P_{\text{train}}(x)\) decreases, \(f_{\theta}(x)\) for \(x\sim P_{\text{OOD}}(x)\) tends to approach \(f^{*}_{\text{constant}}\).**

As an illustrative example, we trained models using either cross-entropy or (continuous-valued) Gaussian negative log-likelihood (NLL) on the MNIST and CIFAR10 datasets. The blue plots in Fig. 2 show the models' predictions as its inputs become increasingly OOD, and the orange plots visualize the OCS associated with each model. We can see that even though we trained on _different_ datasets and evaluated on _different_ kinds of distribution shifts, the neural network predictions exhibit the _same_ pattern of extrapolation: as the distribution shift increases, the network predictions move closer to the OCS. Note that while the behavior of the cross-entropy models can likewise be explained by the network simply producing lower magnitude outputs, the Gaussian NLL models' predicted variance actually increases with distribution shift, which contradicts this alternative explanation.

### Experiments

We will now provide empirical evidence for the "reversion to the OCS" hypothesis. Our experiments aim to answer the question: **As the test-time inputs become more OOD, do neural network predictions move closer to the optimal constant solution?**

Experimental setup.We trained our models on 8 different datasets, and evaluated them on both natural and synthetic distribution shifts. See Table 1 for a summary, and Appendix B.1 for a more detailed description of each dataset. Models with image inputs use ResNet [21] or VGG [45] style architectures, and models with text inputs use DistilBERT [43], a distilled version of BERT [10].

We focus on three tasks, each using a different loss functions: classification with cross entropy (CE), selective classification with mean squared error (MSE), and regression with Gaussian NLL. Datasets with discrete labels are used for classification and selective classification, and datasets with continuous labels are used regression. The cross entropy models are trained to predict the likelihood that the input belongs to each class, as is typical in classification. The MSE models are trained to predict rewards for a selective classification task. More specifically, the models output a value for each class as well as an abstain option, where the value represents the reward of selecting that option given the input. The

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline
**Dataset** & **Label Type** & **Input Modality** & **Distribution Shift Type** \\ \hline CIFAR10 [31] / CIFAR10-C [23] & Discrete & Image & Synthetic \\ \hline ImageNet [9] / ImageNet-R(condition) [26] & Discrete & Image & Natural \\ \hline ImageNet [9] / ImageNet-Sketch [51] & Discrete & Image & Natural \\ \hline DomainBed OfficeHome [18] & Discrete & Image & Natural \\ \hline Skin-elsonPieks [20] & Continuous & Image & Natural \\ \hline UTKFace [57] & Continuous & Image & Synthetic \\ \hline BREEDS living-17 [44] & Discrete & Image & Natural \\ \hline BREEDS non-living-26 [44] & Discrete & Image & Natural \\ \hline WILDS Amazon [30] & Discrete & Text & Natural \\ \hline \end{tabular}
\end{table}
Table 1: Summary of the datasets that we train/evaluate on in our experiments.
ground truth reward is +1 for the correct class, -4 for the incorrect classes, and +0 for abstaining. We train these models by minimizing the MSE loss between the predicted and ground truth rewards. We will later use these models for decision-making in Section 5. The Gaussian NLL models predict a mean and a standard deviation, parameterizing a Gaussian distribution. They are trained to minimize the negative log likelihood of the labels under its predicted distributions.

Evaluation protocol.To answer our question, we need to quantify (1) the dissimilarity between the training data and the evaluation data, and (2) the proximity of network predictions to the OCS. To estimate the former, we trained a low-capacity model to discriminate between the training and evaluation datasets and measured the average predicted likelihood that the evaluation dataset is generated from the evaluation distribution, which we refer to as the OOD score. This score is 0.5 for indistinguishable train and evaluation data, and 1 for a perfect discriminator. To estimate the distance between the model's prediction and the OCS, we compute the KL divergence between the model's predicted distribution and the distribution parameterized by the OCS, \(\frac{1}{N}\sum_{i=1}^{N}D_{\text{KL}}(P_{\theta}(y|x_{i})||P_{f^{*}_{\text{ constant}}}(y))\), for models trained with cross-entropy and Gaussian NLL. For MSE models, the distance is measured using the mean squared error, \(\frac{1}{N}\sum_{i=1}^{N}||f_{\theta}(x_{i})-f^{*}_{\text{constant}}||^{2}\). See Appendix B.3 for more details on our evaluation protocol, and Appendix B.4 for the closed form solution for the OCS for each loss.

Results.In Fig. 3, we plot the OOD score (x-axis) against the distance between the network predictions and the OCS (y-axis) for both the training and OOD datasets. Our results indicate a clear trend: as the OOD score of the evaluation dataset increases, neural network predictions move closer to the OCS. Moreover, our results show that this trend holds relatively consistently across different loss functions, input modalities, network architectures, and types of distribution shifts. We also found instances where this phenomenon did not hold, such as adversarial inputs, which we discuss in greater detail in Appendix A. However, the overall prevalence of "reversion to the OCS" across different settings suggests that it may capture a general pattern in the way neural networks extrapolate.

Figure 3: Evaluating the distance between network predictions and the OCS as the input distribution becomes more OOD. Each point represents a different evaluation dataset, with the red star representing the (holdout) training distribution, and circles representing OOD datasets. The vertical line associated with each point represents the standard deviation over 5 training runs. As the OOD score of the evaluation dataset increases, there is a clear trend of the neural network predictions approaching the OCS.
## 4 Why do OOD Predictions Revert to the OCS?

In this section, we aim to provide insights into why neural networks have a tendency to revert to the OCS. We will begin with an intuitive explanation, and provide empirical and theoretical evidence in Sections 4.1 and 4.2. In our analysis, we observe that weight matrices and network representations associated with training inputs often occupy low-dimensional subspaces with high overlap. However, when the network encounters OOD inputs, we observe that their associated representations tend to have less overlap with the weight matrices compared to those from the training distribution, particularly in the later layers. As a result, OOD representations tend to diminish in magnitude as they pass through the layers of the network, causing the network's output to be primarily influenced by the accumulation of model constants (e.g. bias terms). Furthermore, both empirically and theoretically, we find that this accumulation of model constants tend to closely approximate the OCS. We posit that reversion to the OCS for OOD inputs occurs due to the combination of these two factors: that accumulated model constants in a trained network tend towards the OCS, and that OOD points yield smaller-magnitude representations in the network that become dominated by model constants.

### Empirical Analysis

We will now provide empirical evidence for the mechanism we describe above using deep neural network models trained on MNIST and CIFAR10. MNIST models use a small 4 layer network, and CIFAR10 models use a ResNet20 [21]. To more precisely describe the quantities we will be illustrating, let us rewrite the neural network as \(f(x)=g_{i+1}(\sigma(W_{i}\phi_{i}(x)+b_{i}))\), where \(\phi_{i}(x)\) is an intermediate representation at layer \(i\), \(W_{i}\) and \(b_{i}\) are the corresponding weight matrix and bias, \(\sigma\) is a nonlinearity, and \(g_{i+1}\) denotes the remaining layers of the network. Because we use a different network architecture for each domain, we will use variables to denote different intermediate layers of the network, and defer details about the specific choice of layers to Appendix C.

First, we will show that \(W_{i}\phi_{i}(x)\) tends to diminish for OOD inputs. The first column of plots in Fig. 4 show \(\mathbb{E}_{x\sim P_{\text{OOD}}(x)}[||W_{i}\phi_{i}(x)||^{2}]/\mathbb{E}_{x \sim P_{\text{train}}(x)}[||W_{i}\phi_{i}(x)||^{2}]\) for \(P_{\text{OOD}}\) with different level of rotation or noise. The x-axis represents different layers in the network, with the leftmost being the input and the rightmost being the output. We can see that in the later layers of the network, \(||W_{i}\phi_{i}(x)||^{2}\) consistently became smaller as inputs became more OOD (greater rotation/noise). Furthermore, the diminishing effect becomes more pronounced as the representations pass through more layers.

Figure 4: Analysis of the interaction between representations and weights as distribution shift increases. Plots in first column visualize the norm of network features for different levels of distribution shift at different layers of the network. In later layer of the network, the norm of features tends to decrease as distribution shift increases. Plots in second column show the proportion of network features which lie within the span of the following linear layer. This tends to decrease as distributional shift increases. Error bars represent the standard deviation taken over the test distribution. Plots in the third and fourth column show the accumulation of model constants as compared to the OCS for a cross entropy and a MSE model; the two closely mirror one another.
Next, we will present evidence that this decrease in representation magnitude occurs because \(\phi_{j}(x)\) for \(x\sim P_{\text{train}}(x)\) tend to lie more within the low-dimensional subspace spanned by the rows of \(W_{j}\) than \(\phi_{j}(x)\) for \(x\sim P_{\text{OOD}}(x)\). Let \(V_{\text{top}}\) denote the top (right) singular vectors of \(W_{j}\). The middle plots of Fig. 4 show the ratio of the representation's norm at layer \(j\) that is captured by projecting the representation onto \(V_{\text{top}}\) i.e., \(||\phi_{j}(x)^{\top}V_{\text{top}}V_{\text{top}}^{\top}||^{2}/||\phi_{j}(x)|| ^{2}\), as distribution shift increases. We can see that as the inputs become more OOD, the ratio goes down, suggesting that the representations lie increasingly outside the subspace spanned by the weight matrix.

Finally, we will provide evidence for the part of the mechanism which accounts for the optimality of the OCS. Previously, we have established that OOD representations tend to diminish in magnitude in later layers of the network. This begs the question, what would the output of the network be if the input representation at an intermediary layer had a magnitude of 0? We call this the accumulation of model constants, i.e. \(g_{k+1}(\sigma(b_{k}))\). In the third and fourth columns of Fig. 4, we visualize the accumulation of model constants at one of the final layers \(k\) of the networks for both a cross entropy and a MSE model (details in Sec. 3.2), along with the OCS for each model. We can see that the accumulation of model constants closely approximates the OCS in each case.

### Theoretical Analysis

We will now explicate our empirical findings more formally by analyzing solutions of gradient flow (gradient descent with infinitesimally small step size) on deep homogeneous neural networks with ReLU activations. We adopt this setting due to its theoretical convenience in reasoning about solutions at convergence of gradient descent [33, 14, 27], and its relative similarity to deep neural networks used in practice [36, 11].

**Setup:** We consider a class of homogeneous neural networks \(\mathcal{F}:=\{f(W;x):W\in\mathcal{W}\}\), with \(L\) layers and ReLU activation, taking the functional form \(f(W;x)=W_{L}\sigma(W_{L-1}\ldots\sigma(W_{2}\sigma(W_{1}x))\ldots)\), where \(W_{i}\in\mathbb{R}^{m\times m},\forall i\in\{2,\ldots,L-1\}\), \(W_{1}\in\mathbb{R}^{m\times 1}\) and \(W_{L}\in\mathbb{R}^{1\times m}\). Our focus is on a binary classification problem where we consider two joint distributions \(P_{\text{train}},P_{\text{OOD}}\) over inputs and labels: \(\mathcal{X}\times\mathcal{Y}\), where inputs are from \(\mathcal{X}:=\{x\in\mathbb{R}^{d}:\|x\|_{2}\leq 1\}\), and labels are in \(\mathcal{Y}:=\{-1,+1\}\). We consider gradient descent with a small learning rate on the objective: \(L(W;\mathcal{D}):=\sum_{(x,y)\in\mathcal{D}}\ell(f(W;x),y)\) where \(\ell(f(W;x),y)\mapsto\exp\left(-yf(W;x)\right)\) is the exponential loss and \(\mathcal{D}\) is an IID sampled dataset of size \(N\) from \(P_{\text{train}}\). For more details on the setup, background on homogeneous networks, and full proofs for all results in this section, please see Appendix D.

We will begin by providing a lower bound on the expected magnitude of intermediate layer features corresponding to inputs from the training distribution:

**Proposition 4.1** (\(P_{\text{train}}\) **observes high norm features)**: _When \(f(\hat{W};x)\) fits \(\mathcal{D}\), i.e., \(y_{i}f(\hat{W};x_{i})\geq\gamma\), \(\forall i\in[N]\), then w.h.p \(1-\delta\) over \(\mathcal{D}\), layer \(j\) representations \(f_{j}(\hat{W};x)\) satisfy \(\mathbb{E}_{P_{\text{train}}}[\|f_{j}(\hat{W};x)\|_{2}]\geq(1/C_{0})(\gamma- \tilde{\mathcal{O}}(\sqrt{\log(1/\delta)/N}+C_{1}\log m/N\gamma))\), if \(\exists\) constants \(C_{0},C_{1}\) s.t. \(\|\hat{W}_{j}\|_{2}\leq C_{0}^{1/L}\), \(C_{1}\geq C_{0}^{3L/2}\)._

Here, we can see that if the trained network perfectly fits the training data (\(y_{i}f(\hat{W};x_{i})\)\(\geq\)\(\gamma\), \(\forall i\in[N]\)), and the training data size \(N\) is sufficiently large, then the expected \(\ell_{2}\) norm of layer \(j\) activations \(f_{j}(\hat{W};x)\) on \(P_{\text{train}}\) is large and scales at least linearly with \(\gamma\).

Next, we will analyze the size of network outputs corresponding to points which lie outside of the training distribution. Our analysis builds on prior results for gradient flow on deep homogeneous nets with ReLU activations which show that the gradient flow is biased towards the solution (KKT point) of a specific optimization problem: minimizing the weight norms while achieving sufficiently large margin on each training point [49, 1, 33]. Based on this, it is easy to show that that the solution for this constrained optimization problem is given by a neural network with low rank matrices in each layer for sufficiently deep and wide networks. Furthermore, the low rank nature of these solutions is exacerbated by increasing depth and width, where the network approaches an almost rank one solution for each layer. If test samples deviate from this low rank space of weights in any layer, the dot products of the weights and features will collapse in the subsequent layer, and its affect rolls over to the final layer,
which will output features with very small magnitude. Using this insight, we present an upper bound on the magnitude of the final layer features corresponding to OOD inputs:

**Theorem 4.1** (Feature norms can drop easily on \(P_{\textsc{OOD}}\)): _If \(\exists\) a shallow network \(f^{\prime}(W;x)\) with \(L^{\prime}\) layers and \(m^{\prime}\) neurons satisfying conditions in Proposition 4.1 (\(\gamma{=}1\)), then optimizing the training objective with gradient flow over a class of deeper and wider homogeneous network \(\mathcal{F}\) with \(L>L^{\prime},m>m^{\prime}\) would converge directionally to a solution \(f(\hat{W};x)\), for which the following is true: \(\exists\) a set of rank \(1\) projection matrices \(\{A_{i}\}_{i=1}^{L}\), such that if representations for any layer \(j\) satisfy \(\mathbb{E}_{P_{\textsc{OOD}}}\|A_{j}f_{j}(\hat{W};x)\|_{2}\leq\epsilon\), then \(\exists C_{2}\) for which \(\mathbb{E}_{P_{\textsc{OOD}}}[\|f(\hat{W};x)\|]\lesssim C_{0}(\epsilon+C_{2}^{ -1/L}\sqrt{L+1/L})\)._

This theorem tells us that for any layer \(j\), there exists only a narrow rank one space \(A_{j}\) in which OOD representations may lie, in order for their corresponding final layer outputs to remain significant in norm. Because neural networks are not optimized on OOD inputs, we hypothesize that the features corresponding to OOD inputs tend to lie outside this narrow space, leading to a collapse in last layer magnitudes for OOD inputs in deep networks. Indeed, this result is consistent with our empirical findings in the first and second columns of Fig. 4.1, where we observed that OOD features tend to align less with weight matrices, resulting in a drop in OOD feature norms.

To study the accumulation of model constants, we now analyze a slightly modified class of functions \(\tilde{\mathcal{F}}=\{f(W;\cdot)+b:b\in\mathbb{R},f(W;\cdot)\in\mathcal{F}\}\), which consists of deep homogeneous networks with a bias term in the final layer. In Proposition 4.2, we show that there exists a set of margin points (analogous to support vectors in the linear setting) which solely determines the model's bias \(\hat{b}\).

**Proposition 4.2** (Analyzing network bias): _If gradient flow on \(\tilde{\mathcal{F}}\) converges directionally to \(\hat{W},\hat{b}\), then \(\hat{b}\propto\sum_{k}y_{k}\) for margin points \(\{(x_{k},y_{k}):y_{k}\cdot f(\hat{W};x_{k})=\arg\min_{j\in[N]}y_{j}\cdot f( \hat{W};x_{j})\}\)._

If the label marginal of these margin points mimics that of the overall training distribution, then the learnt bias will approximate the OCS for the exponential loss. This result is consistent with our empirical findings in the third and fourth columns on Fig. 4.1, where we found the accumulation of bias terms tends to approximate the OCS.

## 5 Risk-Sensitive Decision-Making

Lastly, we will explore an application of our observations to decision-making problems. In many decision-making scenarios, certain actions offer a high potential for reward when the agent chooses them correctly, but also higher penalties when chosen incorrectly, while other more cautious actions consistently provide a moderate level of reward. When utilizing a learned model for decision-making, it is desirable for the agent to select the high-risk high-reward actions when the model is likely to be accurate, while opting for more cautious actions when the model is prone to errors, such as when the inputs are OOD. It turns out, if we leverage "reversion to the OCS" appropriately, such risk-sensitive behavior can emerge automatically. If the OCS of the agent's learned model corresponds to cautious actions, then "reversion to the OCS" posits that the agent will take increasingly cautious actions as its inputs become more OOD. However, not all decision-making algorithms leverage "reversion to the OCS" by default. **Depending on the choice of loss function (and consequently the OCS), different algorithms which have similar in-distribution performance can have different OOD behavior.** In the following sections, we will use selective classification as an example of a decision-making problem to more concretely illustrate this idea.

### Example Application: Selective Classification

In selective classification, the agent can choose to classify the input or abstain from making a decision. As an example, we will consider a selective classification task using CIFAR10, where the agent receives
a reward of +1 for correctly selecting a class, a reward of -4 for an incorrect classification, and a reward of 0 for choosing to abstain.

Let us consider one approach that leverages "reversion to the OCS" and one that does not, and discuss their respective OOD behavior. An example of the former involves learning a model to predict the reward associated with taking each action, and selecting the action with the highest predicted reward. This reward model, \(f_{\theta}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{|\mathcal{A}|}\), takes as input an image, and outputs a vector the size of the action space. We train \(f_{\theta}\) using a dataset of images, actions and rewards, \(\mathcal{D}=\{(x_{i},a_{i},r_{i})\}_{i=1}^{N}\), by minimizing the MSE loss, \(\frac{1}{N}\sum_{1\leq i\leq N}(f_{\theta}(x_{i})_{a_{i}}-r_{i})^{2}\), and select actions using the policy \(\pi(x)=\arg\max_{a\in\mathcal{A}}f_{\theta}(x)_{a}\). The OCS of \(f_{\theta}\) is the average reward for each action over the training points, i.e., \((f_{\text{constant}}^{*})_{a}=\frac{\sum_{1\leq i\leq N}r_{i}\cdot[a_{i}=a]}{ \sum_{1\leq j\leq N}\overline{[a_{j}=a]}}\). In our example, the OCS is -3.5 for selecting each class and 0 for abstaining, so the policy corresponding to the OCS will choose to abstain. Thus, according to "reversion to the OCS", this agent should choose to abstain more and more frequently as its input becomes more OOD. We illustrate this behavior in Figure 5. In the first row, we depict the average predictions of a reward model when presented with test images of a specific class with increasing levels of noise (visualized in Figure 1). In the second row, we plot a histogram of the agent's selected actions for each input distribution. We can see that as the inputs become more OOD, the model's predictions converged towards the OCS, and consequently, the agent automatically transitioned from making high-risk, high-reward decisions of class prediction to the more cautious decision of abstaining.

One example of an approach which does not leverage "reversion to the OCS" is standard classification via cross entropy. The classification model takes as input an image and directly predicts a probability distribution over whether each action is the optimal action. In this case, the optimal action given an input is always its ground truth class. Because the OCS for cross entropy is the marginal distribution of labels in the training data, and the optimal action is never to abstain, the OCS for this approach corresponds to a policy that _never_ chooses to abstain. In this case, "reversion to the OCS" posits that the agent will continue to make high-risk high-reward decisions even as its inputs become more OOD. As a result, while this approach can yield high rewards on the training distribution, it is likely to yield very low rewards on OOD inputs, where the model's predictions are likely to be incorrect.

### Experiments

We will now more thoroughly compare the behavior of a reward prediction agent with a standard classification agent for selective classification on a variety of different datasets. Our experiments aim to answer the questions: **How does the performance of a decision-making approach which leverages "reversion to the OCS" compare to that of an approach which does not?**

**Experimental Setup.** Using the same problem setting as the previous section, we consider a selective classification task in which the agent receives a reward of +1 for selecting the correct class, -4 for

Figure 5: Selective classification via reward prediction on CIFAR10. We evaluate on holdout datasets consisting of automobiles (class 1) with increasing levels of noise. X-axis represents the agent’s actions, where classes are indexed by numbers and abstain is represented by ”A”. We plot the average reward predicted by the model for each class (top), and the distribution of actions selected by the policy (bottom). The rightmost plots represent the OCS (top), and the actions selected by an OCS policy (bottom). As distribution shift increased, the model predictions approached the OCS, and the policy automatically selected the abstain action more frequently.
selecting an incorrect class, and +0 for abstaining from classifying. We experiment with 4 datasets: CIFAR10, DomainBed OfficeHome, BREEDS living-17 and non-living-26. We compare the performance of the reward prediction and standard classification approaches described in the previous section, as well as a third oracle approach that is optimally risk-sensitive, thereby providing an upper bound on the agent's achievable reward. To obtain the oracle policy, we train a classifier on the training dataset to predict the likelihood of each class, and then calibrate the predictions with temperature scaling on the OOD evaluation dataset. We then use the reward function to calculate the theoretically optimal threshold on the classifier's maximum predicted likelihood, below which the abstaining option is selected. Note that the oracle policy has access to the reward function and the OOD evaluation dataset for calibration, which the other two approaches do not have access to.

**Results.** In Fig. 6, we plot the frequency with which the abstain action is selected for each approach. As distribution shift increased, both the reward prediction and oracle approaches selected the abstaining action more frequently, whereas the standard classification approach never selected this option. This discrepancy arises because the OCS of the reward prediction approach aligns with the abstain action, whereas the OCS of standard classification does not. In Fig. 7, we plot the average reward received by each approach. Although the performance of all three approaches are relatively similar on the training distribution, the reward prediction policy increasingly outperformed the classification policy as distribution shift increased. Furthermore, the gaps between the rewards yielded by the reward prediction and classification policies are substantial compared to the gaps between the reward prediction and the oracle policies, suggesting that the former difference in performance is nontrivial. Note that the goal of our experiments was not to demonstrate that our approach is the best possible method for selective classification (in fact, our method is likely not better than SOTA approaches), but rather to highlight how the OCS associated with an agent's learned model can influence its OOD decision-making behavior. To this end, this result shows that appropriately leveraging "reversion to the OCS" can substantial improve an agent's performance on OOD inputs.

Figure 6: Ratio of abstain action to total actions; error bars represent standard deviation over 5 random seeds; (t) denotes the training distribution. While the oracle and reward prediction approaches selected the abstain action more frequently as inputs became more OOD, the standard classification approach almost never selected abstain.

Figure 7: Reward obtained by each approach. While all three approaches performed similarly on the training distribution, reward prediction increasingly outperformed standard classification as inputs became more OOD.
Conclusion

We presented the observation that neural network predictions for OOD inputs tend to converge towards a specific constant, which often corresponds to the optimal input-independent prediction based on the model's loss function. We proposed a mechanism to explain this phenomenon and a simple strategy that leverages this phenomenon to enable risk-sensitive decision-making. Finally, we demonstrated the prevalence of this phenomenon and the effectiveness of our decision-making strategy across diverse datasets and different types of distributional shifts.

Our understanding of this phenomenon is not complete. Further research is needed to to discern the properties of an OOD distribution which govern when, and to what extent, we can rely on "reversion to the OCS" to occur. Another exciting direction would be to extend our investigation on the effect of the OCS on decision-making to more complex multistep problems, and study the OOD behavior of common algorithms such as imitation learning, Q-learning, and policy gradient.

As neural network models become more broadly deployed to make decisions in the "wild", we believe it is increasingly essential to ensure neural networks behave safely and robustly in the presence of OOD inputs. While our understanding of "reversion to the OCS" is still rudimentary, we believe it offers a new perspective on how we may predict and even potentially steer the behavior of neural networks on OOD inputs. We hope our observations will prompt further investigations on how we should prepare models to tackle the diversity of in-the-wild inputs they must inevitably encounter.

## 7 Acknowledgements

This work was supported by DARPA ANSR, DARPA Assured Autonomy, and the Office of Naval Research under N00014-21-1-2838. Katie Kang was supported by the NSF GRFP. We would like to thank Dibya Ghosh, Ilya Kostribov, Karl Pertsch, Aviral Kumar, Eric Wallace, Kevin Black, Ben Eysenbach, Michael Janner, Young Geng, Colin Li, Manan Tomar, and Simon Zhai for insightful feedback and discussions.
