# Aligning Model Properties via Conformal Risk Control

William Overman Email<EMAIL_ADDRESS>Graduate School of Business,
Stanford University  Jacqueline Jil Vallon Email<EMAIL_ADDRESS>Management Science and Engineering, Stanford University  Mohsen Bayati Email:
<EMAIL_ADDRESS>Graduate School of Business, Stanford University

###### Abstract

AI model alignment is crucial due to inadvertent biases in training data and
the underspecified pipeline in modern machine learning, where numerous models
with excellent test set metrics can be produced, yet they may not meet end-
user requirements. Recent advances demonstrate that post-training model
alignment via human feedback can address some of these challenges. However,
these methods are often confined to settings (such as generative AI) where
humans can interpret model outputs and provide feedback. In traditional non-
generative settings, where model outputs are numerical values or classes,
detecting misalignment through single-sample outputs is highly challenging.

In this paper we consider an alternative strategy. We propose interpreting
model alignment through property testing, defining an aligned model $f$ as one
belonging to a subset ${\mathcal{P}}$ of functions that exhibit specific
desired behaviors. We focus on post-processing a pre-trained model $f$ to
better align with ${\mathcal{P}}$ using conformal risk control. Specifically,
we develop a general procedure for converting queries for a given property
${\mathcal{P}}$ to a collection of loss functions suitable for use in a
conformal risk control algorithm. We prove a probabilistic guarantee that the
resulting conformal interval around $f$ contains a function approximately
satisfying ${\mathcal{P}}$.

Moreover, given the remarkable capabilities of modern AI models with large
parameters and extensive training data, one might argue that alignment
problems will naturally resolve, making alignment techniques unnecessary.
However, we show that increasing the size of the training data or the number
of parameters in a random feature model does not eliminate the need for
alignment techniques when the pre-training data contains biased labels. We
exhibit applications of our alignment methodology on a collection of
supervised learning datasets for (shape-constrained) properties such as
monotonicity and concavity. The general procedure is flexible and can be
applied to a wide range of desired properties.

##  1 Introduction

The emergence of large foundation models has heightened the focus on the
problem of alignment. Aligned models are artificial intelligences designed to
pursue goals that align with human values, principles, and intentions (Leike
et al., 2018; Ouyang et al., 2022; Hendrycks et al., 2023; Ngo et al., 2024).
Although the alignment problem is predominantly examined in the context of
potential artificial general intelligence (AGI), large language models (LLMs),
and reinforcement learning (RL) agents, it also has roots in the modern
machine learning pipeline (D’Amour et al., 2022). Motivated by this, we
introduce a broader notion of alignment in this paper, extending beyond the
aforementioned generative models to include even tabular regression models.

As an example, consider a regression task where ${\mathcal{P}}$ represents
models that are monotonically decreasing in a given feature. Constraining a
prediction model during training to maintain monotonicity in this feature can
be viewed as a form of alignment. However, ensuring that the predictions of a
pre-trained model $f$ are monotonically decreasing in this feature can be more
complex. This complexity arises particularly in non-generative settings where
the user cannot update a pre-trained $f$ or obtain any outputs other than
point predictions $f(X)$ for a given input $X$.

In this work, we propose an approach to this problem of aligning a pre-trained
model $f$ that is inspired by property testing (Ron, 2008; Goldreich, 2017)
and utilizes conformal risk control (Angelopoulos et al., 2024). We propose a
definition of functions that either satisfy a property ${\mathcal{P}}$ or are
significantly far from satisfying ${\mathcal{P}}$, drawn from the field of
property testing (Ron, 2008; Goldreich, 2017). Property testing focuses on
designing testing algorithms for determining membership to ${\mathcal{P}}$
that are more efficient than the corresponding learning algorithms for
${\mathcal{P}}$ (Ron, 2008). From the perspective of modern deep learning
models, we can imagine a user who wishes to determine whether a pre-trained
model $f$ belongs to ${\mathcal{P}}$ but is unable to train a model of $f$’s
size.

Property testing algorithms generally perform local queries to determine, with
high probability, whether a function has a given global property or is far
from having that property. In this work, we consider mapping such queries for
a given property of interest ${\mathcal{P}}$ to a corresponding set of loss
functions. These loss functions can then be used in a conformal risk control
procedure (Angelopoulos et al., 2024) to obtain a notion of alignment for
${\mathcal{P}}$. In particular, we prove that running this procedure results
in a conformal interval around $f$ that contains a function close to
${\mathcal{P}}$.

We demonstrate our methodology on real-world datasets for the properties of
monotonicity and concavity. To obtain asymmetric conformal intervals around
$f$ we provide a straight-forward generalization of the results of
Angelopoulos et al. (2024) to multi-dimensional
$\boldsymbol{\lambda}\in\mathbb{R}_{\geq 0}^{k}$. We focus the bulk of our
experiments on monotonicity for the sake of exposition and because
monotonicity constraints have been shown to be useful in promoting crucial
aspects of alignment to human values, such as fairness and adherence to social
norms (Wang and Gupta, 2020).

Given the outstanding capabilities of modern AI models with substantially
large numbers of parameters and training data, one may argue that the
alignment problem may naturally disappear as such advances continue (Kaplan et
al., 2020), rendering alignment techniques such as those presented in this
paper unnecessary. However, another contribution of this paper is to refute
this argument in a stylized setting, building on recent advances in the theory
of linearized neural networks (Mei and Montanari, ; Misiakiewicz and
Montanari, 2023). Specifically, we show that increasing the size of the
training data or the number of parameters in a random feature model cannot
help it satisfy a property ${\mathcal{P}}$, if the pre-training data has
biased labels. Our simulations show that the result holds even if only a small
fraction of the training labels are impacted by the bias.

Summarizing our main contributions, we: (1) introduce an alignment perspective
based on property testing, (2) use conformal risk control to post-process
predictions of pre-trained models for better alignment, (3) generalize
conformal risk control to multi-dimensional $\boldsymbol{\lambda}$, and (4)
demonstrate that increasing training data and parameters in a random feature
model does not eliminate the need for alignment. We discuss related work in
Section 6, particularly our connections to Yadkori et al. (2024), who use
conformal risk control to address LLM hallucinations (Ji et al., 2023).

##  2 Preliminaries

In this section we provide condensed overviews of property testing and
conformal prediction and risk control. The main definitions and concepts we
will use from property testing are $\varepsilon$-farness and proximity
oblivious testers (POTs). We also discuss one-sided testers, but primarily to
motivate these former two definitions; our proposed alignment procedure will
mainly rely on the definition of proximity oblivious testers. We also discuss
conformal prediction, mainly focusing on conformal risk control, as this will
be the main tool used to achieve alignment in our setting.

###  2.1 Properties and property testing

Our perspective on alignment in this work is motivated by the field of
property testing (Ron, 2008; Goldreich, 2017). The seminal work in this field
by Blum et al. (1993) studied linearity testing of Boolean functions
$f:\\{0,1\\}^{n}\to\\{0,1\\}$, and since then, there has been significant work
on algorithms that are sublinear in $n$ for querying properties
${\mathcal{P}}$ such as low-degree polynomials (Kaufman and Ron, 2006;
Bhattacharyya et al., 2009) and $k$-juntas (Blais, 2009). In this spirit,
property testing algorithms generally make local queries to a small part of a
large object, such as a function or graph. From these sublinear number of
queries, the algorithm attempts to determine whether the object has the
property at a global level or is significantly far from having the property.

######  Definition 1 ($\varepsilon$-farness).

For a function $f:{\mathcal{X}}\to{\mathcal{Y}}$, a distribution
${\mathcal{D}}$ over ${\mathcal{X}}$, $\varepsilon>0$, and a property
${\mathcal{P}}$, we say $f$ is $\varepsilon$-far from ${\mathcal{P}}$ with
respect to ${\mathcal{D}}$ if
$\delta_{{\mathcal{P}},{\mathcal{D}}}(f)>\varepsilon$, where

$\delta_{{\mathcal{P}},{\mathcal{D}}}(f)\overset{\text{def}}{=}\min_{g\in{\mathcal{P}}}\delta_{{\mathcal{D}}}(f,g)\quad\text{and}\quad\delta_{{\mathcal{D}}}(f,g)\overset{\text{def}}{=}\Pr_{x\sim{\mathcal{D}}}[f(X)\neq
g(X)].$

That is, any function $g\in{\mathcal{P}}$ agrees with $f$ on inputs from
${\mathcal{D}}$ with probability at most $1-\varepsilon$ (Goldreich, 2017).
While property testing often assumes ${\mathcal{D}}$ to be the uniform
distribution, we will keep ${\mathcal{D}}$ general, referred to as the
distribution-free setting (Halevy and Kushilevitz, 2007), as we are concerned
with queries over arbitrary calibration data distributions.

Following the definition introduced by Halevy and Kushilevitz (2007), a
distribution-free one-sided error tester for a property ${\mathcal{P}}$ is a
probabilistic oracle machine $\mathsf{M}$, which is provided with a distance
parameter $\varepsilon>0$, oracle access to an arbitrary function
$f:\mathcal{X}\to\mathcal{Y}$, and oracle access to samples from a fixed but
unknown distribution ${\mathcal{D}}$ over $\mathcal{X}$, and satisfies the
following two conditions:

  1. 1.

If $f$ satisfies $\mathcal{P}$, then
$\Pr\\{\mathsf{M}^{f,{\mathcal{D}}}=\text{Accept}\\}=1$.

  2. 2.

If $f$ is $\varepsilon$-far from $\mathcal{P}$ with respect to
${\mathcal{D}}$, then
$\Pr\\{\mathsf{M}^{f,{\mathcal{D}}}=\text{Accept}\\}\leq\frac{1}{3}$.

####  2.1.1 Proximity oblivious testers

Given the definition of testers above, it is natural to expect that any
reasonable tester must take in $\varepsilon$ as input. Many testing
algorithms, however, rely on $\varepsilon$ only for the sake of determining
the appropriate number of times some basic test should be invoked (Goldreich
and Ron, 2008). The basic test itself does not use $\varepsilon$, so such
basic tests are referred to as proximity oblivious (Goldreich and Ron, 2008).

Following Goldreich and Ron (2008), a proximity-oblivious tester (POT) for
${\mathcal{P}}$ is a probabilistic oracle machine $\mathsf{T}$ that satisfies
the following two conditions:

  1. 1.

If $f$ satisfies ${\mathcal{P}}$, then
$\Pr\\{\mathsf{T}^{f,D}=\text{Accept}\\}=1$.

  2. 2.

There exists a monotone function $\rho:(0,1]\to(0,1]$ such that if
$f\not\in{\mathcal{P}}$, then
$\Pr\\{\mathsf{T}^{f,D}=\text{Reject}\\}\geq\rho(\delta_{{\mathcal{P}},{\mathcal{D}}}(f))$.

The function $\rho$ represents the detection probability of the tester
$\mathsf{T}$. Then we can obtain a one-sided error tester with parameter
$\varepsilon$ by simply making $\Theta(1/\rho(\varepsilon))$ calls to the POT
$\mathsf{T}$ and accepting if and only if all the calls return Accept
(Goldreich and Ron, 2008).

For example, consider the BLR test (Blum et al., 1993) over the boolean
hypercube ${\mathcal{X}}=\\{0,1\\}^{n}$, where the underlying distribution
${\mathcal{D}}$ is uniform. This is a one-sided error tester that uses
$\varepsilon$ only to determine the number of calls it makes to a proximity
oblivious tester. The POT $\mathsf{T}^{f,{\mathcal{D}}}$ is the machine that
samples $X_{1},X_{2}\sim{\mathcal{D}}$ and then checks whether
$f(X_{1})+f(X_{2})=f(X_{1}+X_{2})$ or not. Clearly if $f$ is indeed linear,
then $\Pr\\{\mathsf{T}^{f,D}=\text{Accept}\\}=1$, while if $f$ is sufficiently
far from ${\mathcal{P}}$, then there will be some nonzero probability of
rejection by this tester (Goldreich, 2017).

###  2.2 Conformal prediction and conformal risk control

Our main tool for achieving alignment from this property perspective is built
on conformal prediction and conformal risk control (Vovk et al., 2005; Bates
et al., 2021; Angelopoulos et al., 2024). Conformal prediction post-processes
the outputs of any model $f$ to create prediction intervals $C(\cdot)$ that
ensure certain statistical coverage guarantees. Using a calibration dataset
$\\{(X_{i},Y_{i})\\}_{i=1}^{n}$ consisting of ground truth input-output pairs,
conformal prediction constructs intervals around the predictions of $f$ such
that $\Pr[Y_{n+1}\notin C(X_{n+1})]\leq\alpha$ for a user-specified error rate
$\alpha$ on a test point $(X_{n+1},Y_{n+1})$.

This guarantee is notably distribution-free and holds for any function $f$.
The probability is over the randomness in all $n+1$ points; both the
calibration set and the test point. The construction of $C(\cdot)$ depends on
both the model $f$ and the draw of the calibration data.

The conformal risk control framework extends conformal prediction to notions
of error beyond miscoverage (Angelopoulos et al., 2024). Given an exchangeable
collection of non-increasing, random loss functions
$L_{i}:\Lambda\to(-\infty,B]$, $i=1,\ldots,n+1$, conformal risk control uses
the first $n$ loss functions and calibration data
$\\{(X_{i},Y_{i})\\}_{i=1}^{n}$ to determine $\hat{\lambda}$ such that

$\mathbb{E}[L_{n+1}(\hat{\lambda})]\leq\alpha.$

Consider loss functions of the form
$L_{i}(\lambda)=\ell(C_{\lambda}(X_{i}),Y_{i})$, where $C_{\lambda}(X_{i})$ is
a set of outputs constructed by $f$ and the calibration data. Larger values of
$\lambda$ generate more conservative prediction sets $C_{\lambda}(\cdot)$. Let
the risk on the calibration data for a given $\lambda$ be
$\hat{R}_{n}(\lambda)=\frac{1}{n}\sum_{i=1}^{n}L_{i}(\lambda)$. For a user-
specified risk rate $\alpha$, we let

$\hat{\lambda}=\inf\left\\{\lambda:\frac{n}{n+1}\hat{R}_{n}(\lambda)+\frac{B}{n+1}\leq\alpha\right\\}.$

This choice of $\hat{\lambda}$ guarantees the desired risk control
$\mathbb{E}[L_{n+1}(\hat{\lambda})]\leq\alpha$ (Angelopoulos et al., 2024).

##  3 Conformal property alignment

Our main methodology is to use conformal risk control to create prediction
intervals that align with specific properties $\mathcal{P}$. Our approach
allows us to post-process the outputs of a pre-trained model $f$ to ensure
that within the resulting conformal band, with a given probability, there
exists predictions that adhere to desired properties such as monotonicity. We
first present a generalization of conformal risk control (Angelopoulos et al.,
2024) that exends to multi-dimensional $\boldsymbol{\lambda}$.

###  3.1 Multi-lambda conformal risk control

We generalize the conformal risk control algorithm to a $k$-dimensional vector
of lambda values
$\boldsymbol{\lambda}=(\lambda_{1},\lambda_{2},\ldots,\lambda_{k})$, where
larger $\boldsymbol{\lambda}\in\mathbb{R}^{k}$ yield more conservative
outputs. The construction of $C_{\boldsymbol{\lambda}}(X_{i})$ depends on the
application and allows for flexibility in the use of $f(X_{i})$ and
$\boldsymbol{\lambda}$.

######  Definition 2 ($C_{\boldsymbol{\lambda}}(X)$ and
$C_{\boldsymbol{\lambda}}(f)$).

For each point $X_{i}\in\mathcal{X}$, define $C_{\boldsymbol{\lambda}}(X_{i})$
as the conformal set of predictions for $X_{i}$ constructed from $f$ and
$\boldsymbol{\lambda}$. Then, $C_{\boldsymbol{\lambda}}(f)$ is the conformal
interval around $f$ that includes $C_{\boldsymbol{\lambda}}(X_{i})$ for every
$X_{i}\in\mathcal{X}$.

Following the original scalar $\lambda$ setting (Angelopoulos et al., 2024),
we assess $C_{\boldsymbol{\lambda}}$ using non-increasing random loss
functions $L_{i}=\ell(C_{\boldsymbol{\lambda}}(X_{i}),Y_{i})\in(-\infty,B]$
for $B<\infty$. In particular, we consider an exchangeable collection of non-
increasing random functions
$L_{i}:\boldsymbol{\Lambda}\to(-\infty,B],i=1,...,n+1$, where
$\boldsymbol{\Lambda}\subset\mathbb{R}_{\geq 0}^{k}$ is a bounded subset of
$\mathbb{R}_{\geq 0}^{k}$, with bound $\lambda^{\max}_{j}$ in each dimension
$j\in[k]$.

As in Angelopoulos et al. (2024), we use the first $n$ functions to determine
$\boldsymbol{\hat{\lambda}}$ so that the risk on the $(n+1)$-th function is
controlled, specifically so that
${\mathbb{E}}[L_{n+1}(\boldsymbol{\hat{\lambda}})]\leq\alpha$.

We apply a similar algorithm. Given $\alpha\in(\infty,B)$ and letting
$\hat{R}_{n}(\boldsymbol{\lambda})=\frac{L_{1}(\boldsymbol{\lambda})+\cdots+L_{n}(\boldsymbol{\lambda})}{n}$,
define

$\boldsymbol{\Lambda}_{\min}=\text{Min}\left\\{\boldsymbol{\lambda}:\frac{n}{n+1}\hat{R}_{n}(\boldsymbol{\lambda})+\frac{B}{n+1}\leq\alpha\right\\}$

to be the set of minimal elements (Boyd and Vandenberghe, 2004) of
$\boldsymbol{\Lambda}$ that satisfy the condition
$\frac{n}{n+1}\hat{R}_{n}(\boldsymbol{\lambda})+\frac{B}{n+1}\leq\alpha$. Let
$g:\boldsymbol{\Lambda}\to\mathbb{R}$ be a strictly increasing function with
respect to the product order on $\mathbb{R}^{k}$. Thus each
$L_{i}(\boldsymbol{\lambda})$ is non-increasing with respect to the level sets
defined by $g(\boldsymbol{\lambda})$. Then select
$\boldsymbol{\hat{\lambda}}\in\boldsymbol{\Lambda}_{\min}$ to be a minimizer
of $g$ over $\boldsymbol{\Lambda}_{\min}$.

We then deploy the resulting conformal band $C_{\boldsymbol{\hat{\lambda}}}$
on the test point $X_{n+1}$. For this choice of $\boldsymbol{\hat{\lambda}}$,
we have a risk control guarantee that mimics the result of Angelopoulos et al.
(2024), specifically:

######  Proposition 1.

Assume that $L_{i}(\boldsymbol{\lambda})$ is non-increasing with respect to
the partial ordering of $\boldsymbol{\Lambda}$ inherited from the product
order on $\mathbb{R}^{k}$. Additionally let $g(\boldsymbol{\lambda})$ be a
strictly increasing function $g:\boldsymbol{\Lambda}\to\mathbb{R}$. Also
assume $L_{i}$ is right-continuous in each dimension,
$L_{i}(\boldsymbol{\lambda}^{\max})\leq\alpha$, and
$\sup_{\boldsymbol{\lambda}}L_{i}(\boldsymbol{\lambda})\leq B<\infty$ almost
surely. Then

${\mathbb{E}}[L_{n+1}(\boldsymbol{\hat{\lambda}})]\leq\alpha.$

The proof is similar to the proof of the guarantee for the conformal risk
control algorithm in Angelopoulos et al. (2024) and is deferred to Appendix A.

To provide intuition on $g(\boldsymbol{\lambda})$, we note that for our
primary use case we will take
$g(\boldsymbol{\lambda})=\sum_{i=1}^{k}\lambda_{i}$. Clearly this function is
strictly increasing in $\boldsymbol{\lambda}$ and intuitively it is reasonable
to consider loss functions $L_{i}$ that are non-increasing as the sum of the
components of $\boldsymbol{\lambda}$ increases.

###  3.2 Conformal property alignment from proximity oblivious testers

Now we show how to construct a conformal risk control problem from proximity
oblivious testers for a given property ${\mathcal{P}}$. Let
$\mathsf{T}(f,X,Y)$ be the random output of $\mathsf{T}^{f,{\mathcal{D}}}$
evaluated at $(X,Y)\sim D$. Note there can be randomness in the machine
besides sampling $(X,Y)\sim{\mathcal{D}}$, and each individual loss function
will be defined with respect to fixing this randomness:

######  Definition 3 ($C_{\boldsymbol{\lambda}}(\mathsf{T})$).

We set $L_{i}=\ell(C_{\boldsymbol{\lambda}}(X_{i}),Y_{i})=0$ if after fixing
the remaining randomness in $\mathsf{T}$, there exists $\tilde{f}$ such that
$\mathsf{T}(\tilde{f},X_{i},Y_{i})=\text{Accept}$ where $\tilde{f}$ is the
function resulting from modifying $f$ to take some value $\tilde{Y}\in
C_{\boldsymbol{\lambda}}(X_{i})$ at $X_{i}$. Otherwise,
$L_{i}=\ell(C_{\boldsymbol{\lambda}}(X_{i}),Y_{i})=1$. Define the set of loss
functions obtained from this mapping to be
$C_{\boldsymbol{\lambda}}(\mathsf{T})$.

Intuitively, the POT may fail for the original value of the function
$f(X_{i})$ at $X_{i}$, but if we can adjust the value of $f$ to take some
value $\tilde{Y}$ that falls within $C_{\boldsymbol{\lambda}}(X_{i})$ such
that the function now satisfies the query at point $X_{i}$, then we take the
loss to be zero. We use conformal risk control to find a
$\boldsymbol{\lambda}$ such that with a sufficiently high probability this
loss will be zero at a random data point. The goal is not to obtain a specific
new function, but rather a band around $f$ for which we can establish
probabilistic guarantees of predictions within the band satisfying
${\mathcal{P}}$ as determined by $\mathsf{T}$.

##### Linearity example.

Consider the BLR test mentioned in Section 2.1.1. We use a 2 dimensional
$\boldsymbol{\lambda}=(\lambda^{-},\lambda^{+})$ to generate
$C_{\boldsymbol{\lambda}}(X_{i})=[f(X_{i})-\lambda^{-},f(X_{i})+\lambda^{+}]$.
Now the POT $\mathsf{T}$ for linearity samples
$(X_{1},Y_{1})\sim{\mathcal{D}}$ and then uses the rest of its randomness to
sample $X_{2}\sim\text{Unif}({\mathcal{X}})$. We have
$L_{1}=\ell(C_{\boldsymbol{\lambda}}(X_{1}),Y_{1})=0$ if there exists an
output $\tilde{f}(X_{1})\in[f(X_{1})-\lambda^{-},f(X_{1})+\lambda^{+}]$ such
that $\tilde{f}(X_{1})=f(X_{1}+X_{2})-f(X_{2})$, and
$L_{1}=\ell(C_{\boldsymbol{\lambda}}(X_{1},Y_{1})=1$ otherwise. Intuitively,
we set the loss to 0 if there is some output
$\tilde{Y}\in[f(X_{1})-\lambda^{-},f(X_{1})+\lambda^{+}]$ such that $f$ taking
that value would satisfy linearity with respect to $X_{2}$.

Note that in this case the tester and loss function does not depend on the
$Y_{i}$. This is because the property of $f$ being monotone does not depend on
the $Y_{i}$ and here ${\mathcal{D}}$ is only used to obtain samples of the
$X_{i}$. This is not the case in general, however, and properties can be
defined with respect to the whole sample $(X_{i},Y_{i})\sim{\mathcal{D}}$. For
example, we could consider the property ${\mathcal{P}}$ that $f$ does not
over-predict, that is, for $(X,Y)\sim{\mathcal{D}}$ we have $f(X)\leq Y$. Now
we state our main theorem.

######  Theorem 1.

If we run conformal risk control on a calibration dataset
$\\{(X_{i},Y_{i})\\}_{i=1}^{n}$ from a distribution ${\mathcal{D}}$ using
parameter $\alpha$ and loss functions $L=C_{\boldsymbol{\lambda}}(\mathsf{T})$
mapped from a proximity oblivious tester $\mathsf{T}$ for property
${\mathcal{P}}$, then with probability at least $1-\alpha$ there exists a
function $\tilde{f}\in C_{\boldsymbol{\hat{\lambda}}}(f)$ such that
$\tilde{f}$ is at most $\alpha$-far from ${\mathcal{P}}$ with respect to
${\mathcal{D}}$.

###### Proof.

Conformal risk control guarantees us that
${\mathbb{E}}[L_{n+1}(C_{\boldsymbol{\hat{\lambda}}}(X_{n+1}),Y_{n+1})]\leq\alpha$.
Note that the randomness in this expectation is over the randomness in all of
the $n+1$ points. We have

$\displaystyle\alpha$
$\displaystyle\geq{\mathbb{E}}[L_{n+1}(C_{\boldsymbol{\hat{\lambda}}}(X_{n+1}),Y_{n+1})]$
$\displaystyle\alpha$
$\displaystyle\geq\Pr_{(X_{1},Y_{1}),...,(X_{n+1},Y_{n+1})\sim{\mathcal{D}}}[\forall\tilde{f}\in
C_{\boldsymbol{\hat{\lambda}}}(f),\mathsf{T}(\tilde{f},X_{n+1},Y_{n+1})=\text{Reject}]$
$\displaystyle 1-\alpha$
$\displaystyle\leq\Pr_{(X_{1},Y_{1}),...,(X_{n+1},Y_{n+1})\sim{\mathcal{D}}}[\exists\tilde{f}\in
C_{\boldsymbol{\hat{\lambda}}}(f),\mathsf{T}(\tilde{f},X_{n+1},Y_{n+1})=\text{Accept}]$
$\displaystyle 1-\alpha$
$\displaystyle\leq\Pr_{(X_{n+1},Y_{n+1})\sim{\mathcal{D}}}[\exists\tilde{f}\in
C_{\boldsymbol{\hat{\lambda}}}(f),\mathsf{T}(\tilde{f},(X_{n+1},Y_{n+1}))=\text{Accept}|\boldsymbol{\hat{\lambda}}]\Pr_{(X_{1},Y_{1}),...,(X_{n},Y_{n})\sim{\mathcal{D}}}[\boldsymbol{\hat{\lambda}}]$

So with probability at least $1-\alpha$ over the random draw of
$(X_{1},Y_{1}),...,(X_{n},Y_{n})$ we have that $\boldsymbol{\hat{\lambda}}$ is
such that with probability at least $1-\alpha$ over sample
$(X_{n+1},Y_{n+1})\sim{\mathcal{D}}$ we can find a point $\tilde{Y}_{n+1}\in
C_{\boldsymbol{\hat{\lambda}}}(X_{n+1})$ such that the POT $\mathsf{T}$
accepts. For the points $(X,Y)$ in the support of ${\mathcal{D}}$ for which we
can find such a $\tilde{Y}\in C_{\boldsymbol{\lambda}}(X)$, set
$\tilde{f}(X)=\tilde{Y}$. Then define $\tilde{g}(X)$ to agree with
$\tilde{f}(X)$ at these points and take a value such that $\mathsf{T}$ accepts
at other points in ${\mathcal{X}}\times{\mathcal{Y}}$. Thus
$\tilde{g}\in{\mathcal{P}}$. Then we have that
$\Pr_{(X,Y)\sim{\mathcal{D}}}[\tilde{f}(X)\not=\tilde{g}(X)]\leq\alpha$. So
$\min_{g\in{\mathcal{P}}}\Pr_{(X,Y)\sim D}[\tilde{f}(X)\not=g(X)]\leq\alpha$,
hence $\tilde{f}$ is at most $\alpha$ far from ${\mathcal{P}}$ with respect to
${\mathcal{D}}$. ∎

##  4 Examples

###  4.1 Monotonicity

Monotonic behavior is important in various applications. We focus on
monotonicity in a single feature, where we expect that $f(X)$ should have
monotonically increasing or decreasing behavior with respect to a certain
feature $x^{k}$ when other features $x^{-k}$ are held fixed. While there is a
long-standing literature on using monotonic constraints for regularization
(Brunk et al., 1973; Sill and Abu-Mostafa, 1996; You et al., 2017;
Bonakdarpour et al., 2018) and on integrating such monotonic shape constraints
into prediction models (Groeneboom and Jongbloed, 2014; Cano et al., 2018;
Runje and Shankaranarayana, 2023), our aim is not to view monotonicity as a
possible means to improve test accuracy, but rather as a user-desired property
for safe or fair deployment of a given model. For example, Wang and Gupta
(2020) highlight the importance of monotonicity in models for criminal
sentencing, wages, and medical triage.

Consider a user given a pre-trained model $f$ that was not trained with
monotonic constraints. The user, however, wishes for the sake of safe or fair
deployment to make predictions in a way that is as monotonic as possible. In
particular, let ${\mathcal{P}}$ be the property that $f$ is monotonically
decreasing in dimension $k$. To apply our methodology we consider the
following proximity oblivious tester $\mathsf{T}$ in Algorithm 1.

Algorithm 1 POT $\mathsf{T}$ for monotonicitally decreasing in a fixed
dimension $k$

1: Sample $(X_{1},Y_{1})\sim\mathcal{D}$. Let $X_{1}=(x_{1},x^{-k})$

2: Sample $x_{2}$ uniformly from the range of dimension $k$. Set
$X_{2}=(x_{2},x^{-k})$

3: if $x_{1}<x_{2}$ and $f(X_{1})<f(X_{2})$ then

4: return Reject

5: else if $x_{2}<x_{1}$ and $f(X_{2})<f(X_{1})$ then

6: return Reject

7: end if

8: return Accept

Clearly this tester will always return Accept when $f\in{\mathcal{P}}$, and if
$\delta_{{\mathcal{P}},{\mathcal{D}}}(f)>0$, then there will be some nonzero
probability that $\mathsf{T}$ returns Reject. We apply our conformal property
alignment from POT procedure to obtain loss functions
$L_{i}=\ell(C_{\boldsymbol{\lambda}}(X_{i}),Y_{i})$. On our calibration set
$\\{(X_{i},Y_{i})\\}_{i=1}^{n}$ we obtain a loss function $L_{i}$ for each $i$
that is obtained from fixing the randomness in $\mathsf{T}$ to obtain a
$x_{i2}$ for each point $i$. Note that after fixing this randomness, since the
calibration data point $(X_{i},Y_{i})$ is also fixed, each loss function
$L_{i}$ becomes a deterministic function of $\boldsymbol{\lambda}$.

Now according to our construction of the loss functions in Section 3.2, we
have that $L_{i}=\ell(C_{\boldsymbol{\lambda}}(X_{i1}),Y_{i})=0$ if there
exists $\tilde{f}$ such that, with the fixed randomness of sampling $X_{i2}$,
we have $\mathsf{T}(\tilde{f},X_{i1})=\text{Accept}$, where $\tilde{f}$ is the
function resulting from modifying $f$ to take some value
$\tilde{Y}\in[f(X_{i1})-\lambda^{-},f(X_{i1})+\lambda^{+}]$. Otherwise,
$L_{i}=\ell(C_{\boldsymbol{\lambda}}(X_{i1}),Y_{i})=1$. Intuitively, we have
zero loss if we can find some point in the conformal band around around
$f(X_{i1})$ such that monotonicity (decreasing) is preserved with respect to
$X_{i2}$ in the sense that monotonicity POT $\mathsf{T}$ returns Accept.

##### Setup.

We align for monotonicity on various UCI ML repository datasets (Dua and
Graff, 2023) with a 70-15-15 train-calibrate-test split, averaged over 10
random splits. We use XGBoost regression models (Chen and Guestrin, 2016). For
each dataset, we select a feature for which we desire the model to be
monotonic, not with the intention of improving test-set accuracy, but from the
perspective of a user who desires this property.

We train two models per dataset: one unconstrained, trained on the training
set, and another constrained to be monotonic, trained on both the training and
calibration sets. The conformal risk control procedure is applied to the
unconstrained model using the calibration data. The constrained model can be
considered best possible from the user’s perspective, using all available pre-
test data and satisfying the monotonicity property ${\mathcal{P}}$ during
training.

To compare performance with respect to the training metric of accuracy, we
convert conformal intervals into point predictions by taking $k$-quantiles of
the constrained feature, linearly interpolating between adding $\lambda^{+}$
at the lowest quantile to subtracting $\lambda^{-}$ at the highest quantile
for monotonically decreasing, or vice versa for monotonically increasing.

##### Results.

Table 1 presents results on the test set for the Combined Cycle Power Plant
dataset (Tfekci and Kaya, 2014). In practice, Exhaust-vacuum is known to
negatively influence turbine efficiency (Tfekci and Kaya, 2014). The conformal
procedure outperforms the constrained model in terms of MSE for $\alpha=0.1$
and $\alpha=0.05$, but worsens for $\alpha=0.01$. The risk metric closely
matches the theoretical guarantee from conformal risk control and achieves
optimal performance of 0 for the constrained model. Additional datasets and
results are detailed in the appendix.

Table 1: Power Plant, $n=9568$. Monotonically decreasing on Exhaust Vacuum. $\boldsymbol{\lambda}^{\max}=(10,10)$, $g(\boldsymbol{\lambda})=\lambda^{+}+\lambda^{-}$. $\alpha$ | $\boldsymbol{\lambda}$ | Metric | Unconstrained | Conformal | Constrained
---|---|---|---|---|---
0.1 | $\lambda^{+}=0.96$ | MSE | 9.99 | 10.35 | 15.31
$\lambda^{-}=1.18$ | Risk | 0.17 | 0.111 | 0
0.05 | $\lambda^{+}=1.94$ | MSE | 9.99 | 11.37 | 15.31
$\lambda^{-}=2.68$ | Risk | 0.17 | 0.058 | 0
0.01 | $\lambda^{+}=4.76$ | MSE | 9.99 | 15.56 | 15.31
$\lambda^{-}=4.88$ | Risk | 0.17 | 0.012 | 0
Figure 1: Univariate partial dependence plot of unconstrained model. Risk
control band for $\alpha=0.05$. Dashed line exemplifying Theorem 1
demonstrating existence of monotonically decreasing function falling within
the conformal band on $0.97>1-\alpha$ fraction of the domain.

###  4.2 Concavity

Concavity and convexity are crucial behaviors in many applications. In this
context, we focus on concavity in a single feature. A common example where
users might expect concave behavior is in recommendation or preference
prediction models. According to economic theory, the utility function with
respect to the quantity of an item is often quasi-concave, reflecting the
principle of diminishing marginal utility (Mas-Colell et al., 1995). Jenkins
et al. (2021) propose a novel loss function to account for this expected
concavity, which aligns the model with the concavity property ${\mathcal{P}}$
during training. Here we again consider aligning a pre-trained model, not
trained to satisfy ${\mathcal{P}}$, using a proximity oblivious tester
$\mathsf{T}$ for ${\mathcal{P}}$ as described in Algorithm 2.

Algorithm 2 POT $\mathsf{T}$ for concavity in a fixed dimension $k$

1: Sample $(X,Y)\sim\mathcal{D}$. Let $X=(x,x^{-k})$

2: Let $L$ and $R$ be the subset of points in dimension $k$ less than $x$ and
greater than $x$, respectively.

3: Sample $x_{l}$ uniformly from $L$ and $x_{r}$ uniformly from $R$.

4: Set $X_{l}=(x_{l},x^{-k}),X_{r}=(x_{r},x^{-k})$

5: if $f(X)<\frac{f(X_{l})+f(X_{r})}{2}$ then

6: return Reject

7: end if

8: return Accept

In this case we use a scalar $\lambda$ and have that that
$L=\ell(C_{\lambda}(X),Y)=0$ if $f(X)+\lambda\geq\frac{f(X_{l})+f(X_{r})}{2}$,
and $L=\ell(C_{\lambda}(X),Y)=1$ otherwise. We demonstrate running conformal
risk control with this loss function on a real-world dataset in the appendix.

##  5 A stylized examination of alignment persistence in AI models

Consider data generated as:

$y=g(X)+h(X)+\varepsilon\,,$

where $\varepsilon$ is mean-zero noise with variance $\tau^{2}$ independent of
$X$. Here, $h(X)$ is biased noise we want to ignore, aiming to learn only
$g(X)$. Consider the case in which experts expect data to follow
$g(X)+\textit{unbiased noise}$, but biased noise $h(X)$ can obscure this.

One potential reason for the presence of biased noise in data could be due to
a measurement error of the outcome that is correlated with select features,
leading to an incorrectly calculated outcome. A biased measurement error could
occur if there is incomplete data and the presence of the incomplete data is
correlated with select features in an unexpected, systematic way. Our goal is
to understand how this bias affects model behavior when trying to learn $g(X)$
alone.

Given $n$ i.i.d. samples $\\{(X_{i},Y_{i})\\}_{i=1}^{n}$ from the above model,
we denote this dataset by ${\mathcal{D}}_{n}$. We use a random feature model:

$f_{{\mathsf{RF}}}(X;{\mathbf{a}},\\{{\mathbf{w}}_{j}\\}_{j\in[N]})=\frac{1}{\sqrt{N}}\sum_{j\in[N]}a_{j}\sigma(\langle
X,{\mathbf{w}}_{j}\rangle)\,,$

where ${\mathbf{a}}\in{\mathbb{R}}^{N}$ are learned weights, and
$\\{{\mathbf{w}}_{j}\\}_{j\in[N]}$ are fixed random weights. The squared loss
is minimized by ridge regression:

$\hat{{\mathbf{a}}}_{\lambda}=\arg\min_{{\mathbf{a}}\in{\mathbb{R}}^{N}}\sum_{i\in[n]}\left(Y_{i}-f_{{\mathsf{RF}}}(X_{i})\right)^{2}+\lambda\|{\mathbf{a}}\|_{2}^{2}\,.$

Users expect a model to exhibit a property ${\mathcal{P}}$, satisfied by
$g(X)$ but not necessarily by $g(X)+h(X)$. We can constrain training to ensure
${\mathcal{P}}$. Let
$C_{{\mathcal{P}}}=\\{{\mathbf{a}}~{}|~{}{\mathbf{a}}\in{\mathbb{R}}^{N}~{}\text{and}~{}f_{{\mathsf{RF}}}(X;{\mathbf{a}})~{}\text{satisfies}~{}{\mathcal{P}}\\}\,,$
yielding a constrained model:
$\hat{{\mathbf{a}}}_{\lambda,{\mathcal{P}}}=\arg\min_{{\mathbf{a}}\in
C_{{\mathcal{P}}}}\sum_{i\in[n]}\left(Y_{i}-f_{{\mathsf{RF}}}(X_{i})\right)^{2}+\lambda\|{\mathbf{a}}\|_{2}^{2}$.

Assuming $g$ and $h$ are polynomials with $\deg_{g}<\deg_{h}$, and given
specific conditions on data size and model parameters, we consider two
settings: (i) Classic: $d^{\deg_{g}+\delta}<N<d^{\deg_{h}-\delta}$, and (ii)
Underspecified: $N>d^{\deg_{h}+\delta}$ for a small $\delta>0$.

In Appendix C, we utilize results from Misiakiewicz and Montanari (2023) to
derive insights into the impact of model complexity and data size on adherence
to ${\mathcal{P}}$. In particular, we show that under certain assumptions,
including small noise bias and robustness of property ${\mathcal{P}}$, the
constrained and unconstrained models have zero distance in the classic
setting:
$\hat{{\mathbf{a}}}_{\lambda,{\mathcal{P}}}=\hat{{\mathbf{a}}}_{\lambda}$.
However, in the underspecified setting, the constrained and unconstrained
models will differ, resulting in a non-zero distance:
$\hat{{\mathbf{a}}}_{\lambda,{\mathcal{P}}}\neq\hat{{\mathbf{a}}}_{\lambda}$.
This result implies that in the presence of noise bias, the overparameterized
models (i.e., underspecified setting) fail to satisfy the property
${\mathcal{P}}$, and this cannot be remedied as the data size increases.

##  6 Related work

Our paper draws from a broad range of areas, hence we refer the reader to
textbooks and surveys in alignment (Everitt et al., 2018; Hendrycks et al.,
2022; Ji et al., 2024; Hendrycks, 2024), conformal prediction (Angelopoulos
and Bates, 2022), property testing (Ron, 2008; Goldreich, 2017), and
linearized neural networks (Misiakiewicz and Montanari, 2023).

RLHF (Christiano et al., 2017) has been notably effective in aligning LLMs
with human values and intentions, as demonstrated by (Ouyang et al., 2022).
Our work considers attempts at alignment that generalzies to models without
human-interpretable outputs, which has connections to the scalable oversight
problem (Irving et al., 2018; Christiano et al., 2018; Wu et al., 2021). Goal
misgeneralization (Langosco et al., 2022; Shah et al., 2022) has potential
connections to the underspecified pipeline (D’Amour et al., 2022) considered
in this paper in the sense that models with equivalent performance according
to the training metric may differ in some other user-desired property during
deployment. One of the main methods of assurance (Batarseh et al., 2021),
which is concerned with assessing the alignment of pre-trained AI systems, is
safety evaluations (Perez et al., 2022; Shevlane et al., 2023) meant to assess
risk during deployment, which also has connections to our approach.

##### Connections to Yadkori et al. (2024).

The work of Yadkori et al. (2024) closely aligns with ours in both methodology
and theme, utilizing conformal risk control to reduce LLM hallucinations (Ji
et al., 2023). To demonstrate our framework’s flexibility and generality, we
apply it to their context. They examine a potentially random LLM
$f:{\mathcal{X}}\to\mathcal{Y}$ and assume access to a confidence score
function $g:{\mathcal{X}}\to\mathbb{R}$. They use an abstention function
$a:\Lambda\times{\mathcal{X}}\to\\{0,1\\}$ that decides whether the model
should abstain from answering where $a_{\lambda}(X)=1$ if $g(X)<\lambda$ and
$a_{\lambda}(X)=0$ if $g(X)\geq\lambda$ . The match function
$m:{\mathcal{X}}\times\mathcal{Y}\times\mathcal{Y}\to\\{0,1\\}$ is used to
determine if a response $Y^{\prime}$ is semantically equivalent to $Y$,
indicating hallucination if not.

Yadkori et al. (2024) use conformal risk control to find an optimal
$\hat{\lambda}$ such that the pair $(a_{\hat{\lambda}},f)$ hallucinates at a
rate less than $\alpha$. The loss function they use is
$\ell:{\mathcal{X}}\times\mathcal{Y}\times\Lambda\to\mathbb{R}$ such that
$\ell$ punishes failure to abstain when the response does not match the label,
that is

$\displaystyle\ell(X,Y;\lambda)=(1-a_{\lambda}(X))(1-m(X;f(X),Y)).$ (1)

In this case we do not change the outputs of the function $f$ itself, instead
we use conformal risk control to change the outputs of $a$. Consider the
property ${\mathcal{P}}$ to be that $(a,f)$ does not hallucinate. A tester for
${\mathcal{P}}$ is then simply to query $(a,f)$ on some input $X$ and Reject
if $f$ hallucinates or Accept otherwise. Now when $g(X)<\lambda$, we have
$C_{\lambda}(X)=\\{f(X),\text{Abstain}\\}$ meaning that $(a,f)$ can avoid
hallucinating, so the tester will Accept.

Yadkori et al. (2024) use a separate conformal procedures for setting
$\lambda$ and a parameter $\beta$ with which they define
$m(X;Y^{\prime},Y)=\mathbb{I}(s(X;Y^{\prime},Y)\geq\beta)$, where the
similarity function
$s:{\mathcal{X}}\times{\mathcal{Y}}\times{\mathcal{Y}}\to\mathbb{R}$ is
assumed to be given. We can use our multi-lambda conformal risk control with
$\boldsymbol{\lambda}=(\lambda_{a},\lambda_{m})$ to set these parameters
simultaneously using a single calibration set $\\{(X_{i},Y_{i})\\}_{i=1}^{n}$
of ground truth query-response pairs sampled from the true data distribution
${\mathcal{D}}$. Let

$\displaystyle\ell(X,Y;\boldsymbol{\lambda})$
$\displaystyle=(1-a_{\lambda_{a}}(X))(1-\mathbb{I}(s(X;f(X),Y)\geq
1-\lambda_{m}))$

Then we can take $g(\boldsymbol{\lambda})=c_{1}\lambda_{a}+c_{2}\lambda_{m}$
for $c_{1},c_{2}\geq 0$ and choose $\boldsymbol{\lambda}$ by our multi-lambda
conformal risk control preocedure. As required, this loss function is non-
increasing with respect to $\boldsymbol{\Lambda}$ and
$g(\boldsymbol{\lambda})$.

##  7 Discussion

We introduce a method to align pre-trained models with desired user properties
using conformal risk control. By post-processing outputs using property
dependent loss functions, we provide probabilistic guarantees that conformal
intervals contain functions close to the desired set. This allows for
alignment without retraining, effective in both generative and non-generative
contexts. Future work should extend these techniques to more properties,
explore sample complexity and adaptive querying, and potentially apply them to
policy functions in MPD settings for RL agent safety guarantees.

## References

  * Angelopoulos and Bates [2022] Anastasios N. Angelopoulos and Stephen Bates.  A gentle introduction to conformal prediction and distribution-free uncertainty quantification, 2022.
  * Angelopoulos et al. [2024] Anastasios Nikolas Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster.  Conformal risk control.  In _The Twelfth International Conference on Learning Representations_ , 2024.
  * Batarseh et al. [2021] Feras A. Batarseh, Laura Freeman, and Chih-Hao Huang.  A survey on artificial intelligence assurance.  _Journal of Big Data_ , 8(1), April 2021.  ISSN 2196-1115.  doi: 10.1186/s40537-021-00445-7.  URL http://dx.doi.org/10.1186/s40537-021-00445-7.
  * Bates et al. [2021] Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan.  Distribution-free, risk-controlling prediction sets, 2021.
  * Bhattacharyya et al. [2009] Arnab Bhattacharyya, Swastik Kopparty, Grant Robert Schoenebeck, Madhu Sudan, and David Zuckerman.  Optimal testing of reed-muller codes.  _2010 IEEE 51st Annual Symposium on Foundations of Computer Science_ , pages 488–497, 2009.  URL https://api.semanticscholar.org/CorpusID:25924.
  * Blais [2009] Eric Blais.  Testing juntas nearly optimally.  pages 151–158, 05 2009.  doi: 10.1145/1536414.1536437.
  * Blum et al. [1993] Manuel Blum, Michael Luby, and Ronitt Rubinfeld.  Self-testing/correcting with applications to numerical problems.  _Journal of Computer and System Sciences_ , 47(3):549–595, 1993.  ISSN 0022-0000.  doi: https://doi.org/10.1016/0022-0000(93)90044-W.  URL https://www.sciencedirect.com/science/article/pii/002200009390044W.
  * Bonakdarpour et al. [2018] Matt Bonakdarpour, Sabyasachi Chatterjee, Rina Foygel Barber, and John Lafferty.  Prediction rule reshaping, 2018.
  * Boyd and Vandenberghe [2004] Stephen Boyd and Lieven Vandenberghe.  _Convex optimization_.  Cambridge university press, 2004.
  * Brunk et al. [1973] Hugh D. Brunk, Richard E. Barlow, David J. Bartholomew, and Joan M. Bremner.  Statistical inference under order restrictions : the theory and application of isotonic regression.  _International Statistical Review_ , 41:395, 1973.  URL https://api.semanticscholar.org/CorpusID:120349543.
  * Cano et al. [2018] José-Ramón Cano, Pedro Antonio Gutiérrez, Bartosz Krawczyk, Michał Woźniak, and Salvador García.  Monotonic classification: an overview on algorithms, performance measures and data sets, 2018.
  * Chen and Guestrin [2016] Tianqi Chen and Carlos Guestrin.  Xgboost: A scalable tree boosting system.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16. ACM, August 2016.  doi: 10.1145/2939672.2939785.  URL http://dx.doi.org/10.1145/2939672.2939785.
  * Christiano et al. [2018] Paul Christiano, Buck Shlegeris, and Dario Amodei.  Supervising strong learners by amplifying weak experts, 2018.
  * Christiano et al. [2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf.
  * D’Amour et al. [2022] Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, Farhad Hormozdiari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yian Ma, Cory McLean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymyrov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D. Sculley.  Underspecification presents challenges for credibility in modern machine learning.  _Journal of Machine Learning Research_ , 23(1), jan 2022.  ISSN 1532-4435.
  * Dua and Graff [2023] Dheeru Dua and Casey Graff.  Uci machine learning repository, 2023.  URL http://archive.ics.uci.edu/ml.
  * Everitt et al. [2018] Tom Everitt, Gary Lea, and Marcus Hutter.  Agi safety literature review, 2018.
  * Goldreich [2017] Oded Goldreich.  _Introduction to Property Testing_.  Cambridge University Press, 2017.
  * Goldreich and Ron [2008] Oded Goldreich and Dana Ron.  On proximity-oblivious testing.  _Electronic Colloquium on Computational Complexity (ECCC)_ , 15, 01 2008.  doi: 10.1137/100789646.
  * Groeneboom and Jongbloed [2014] Piet Groeneboom and Geurt Jongbloed.  _Nonparametric Estimation under Shape Constraints: Estimators, Algorithms and Asymptotics_.  Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2014.  ISBN 978-0-521-86401-5.  doi: 10.1017/CBO9781139020893.
  * Halevy and Kushilevitz [2007] Shirley Halevy and Eyal Kushilevitz.  Distribution-free property-testing.  _SIAM J. Comput._ , 37:1107–1138, 01 2007.  doi: 10.1007/978-3-540-45198-3_26.
  * Hendrycks [2024] Dan Hendrycks.  _AI Safety, Ethics, and Society_.  Center for AI Safety, 2024.  URL https://www.aisafetybook.com/.
  * Hendrycks et al. [2022] Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt.  Unsolved problems in ml safety, 2022.
  * Hendrycks et al. [2023] Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt.  Aligning ai with shared human values, 2023.
  * Irving et al. [2018] Geoffrey Irving, Paul Christiano, and Dario Amodei.  Ai safety via debate, 2018.
  * Jenkins et al. [2021] Porter Jenkins, Ahmad Farag, J. Stockton Jenkins, Huaxiu Yao, Suhang Wang, and Zhenhui Li.  Neural utility functions.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(9):7917–7925, May 2021.  doi: 10.1609/aaai.v35i9.16966.  URL https://ojs.aaai.org/index.php/AAAI/article/view/16966.
  * Ji et al. [2024] Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan O’Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo, and Wen Gao.  Ai alignment: A comprehensive survey, 2024.
  * Ji et al. [2023] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38, March 2023.  ISSN 1557-7341.  doi: 10.1145/3571730.  URL http://dx.doi.org/10.1145/3571730.
  * Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models, 2020.
  * Kaufman and Ron [2006] Tali Kaufman and Dana Ron.  Testing polynomials over general fields.  _SIAM J. Comput._ , 36:779–802, 10 2006.  doi: 10.1137/S0097539704445615.
  * Langosco et al. [2022] Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger.  Goal misgeneralization in deep reinforcement learning.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 12004–12019. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/langosco22a.html.
  * Leike et al. [2018] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg.  Scalable agent alignment via reward modeling: a research direction, 2018.
  * Mas-Colell et al. [1995] Andreu Mas-Colell, Michael D. Whinston, and Jerry R. Green.  _Microeconomic Theory_.  Oxford University Press, New York, 1995.
  * [34] Song Mei and Andrea Montanari.  The generalization error of random features regression: Precise asymptotics and the double descent curve.  75(4):667–766.  ISSN 1097-0312.  doi: 10.1002/cpa.22008.
  * Misiakiewicz and Montanari [2023] Theodor Misiakiewicz and Andrea Montanari.  Six Lectures on Linearized Neural Networks.  _arXiv e-prints_ , art. arXiv:2308.13431, August 2023.  doi: 10.48550/arXiv.2308.13431.
  * Ngo et al. [2024] Richard Ngo, Lawrence Chan, and Sören Mindermann.  The alignment problem from a deep learning perspective, 2024.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems_ , volume 35, pages 27730–27744. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf.
  * Perez et al. [2022] Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan.  Discovering language model behaviors with model-written evaluations, 2022.
  * Ron [2008] Dana Ron.  Property testing: A learning theory perspective.  _Found. Trends Mach. Learn._ , 1(3):307–402, 2008.  doi: 10.1561/2200000004.  URL https://doi.org/10.1561/2200000004.
  * Runje and Shankaranarayana [2023] Davor Runje and Sharath M. Shankaranarayana.  Constrained monotonic neural networks, 2023.
  * Shah et al. [2022] Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton.  Goal misgeneralization: Why correct specifications aren’t enough for correct goals, 2022.
  * Shevlane et al. [2023] Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, Will Hawkins, Been Kim, Iason Gabriel, Vijay Bolina, Jack Clark, Yoshua Bengio, Paul Christiano, and Allan Dafoe.  Model evaluation for extreme risks, 2023.
  * Sill and Abu-Mostafa [1996] Joseph Sill and Yaser Abu-Mostafa.  Monotonicity hints.  In M.C. Mozer, M. Jordan, and T. Petsche, editors, _Advances in Neural Information Processing Systems_ , volume 9. MIT Press, 1996.  URL https://proceedings.neurips.cc/paper_files/paper/1996/file/c850371fda6892fbfd1c5a5b457e5777-Paper.pdf.
  * Tfekci and Kaya [2014] Pnar Tfekci and Heysem Kaya.  Combined Cycle Power Plant.  UCI Machine Learning Repository, 2014.  DOI: https://doi.org/10.24432/C5002N.
  * Vovk et al. [2005] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer.  _Algorithmic Learning in a Random World, Second Edition_.  January 2005.  doi: 10.1007/978-3-031-06649-8.  Springer-Verlag New York, Inc. 2005.
  * Wang and Gupta [2020] Serena Wang and Maya Gupta.  Deontological ethics by monotonicity shape constraints, 2020.
  * Wu et al. [2021] Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano.  Recursively summarizing books with human feedback, 2021.
  * Yadkori et al. [2024] Yasin Abbasi Yadkori, Ilja Kuzborskij, David Stutz, András György, Adam Fisch, Arnaud Doucet, Iuliya Beloshapka, Wei-Hung Weng, Yao-Yuan Yang, Csaba Szepesvári, Ali Taylan Cemgil, and Nenad Tomasev.  Mitigating llm hallucinations via conformal abstention, 2024.
  * You et al. [2017] Seungil You, David Ding, Kevin Canini, Jan Pfeifer, and Maya Gupta.  Deep lattice networks and partial monotonic functions, 2017.

##  Appendix A Proof of multi-lambda conformal risk control

Proof of Proposition 1. Our proof closely mimics the original proof of boudned
risk found in Angelopoulos et al. [2024].

###### Proof.

Let

$\hat{R}_{n+1}(\boldsymbol{\lambda})=\frac{(L_{1}(\boldsymbol{\lambda})+\ldots+L_{n+1}(\boldsymbol{\lambda}))}{(n+1)}\quad\text{and}$
$\displaystyle\boldsymbol{\Lambda}_{\min}$
$\displaystyle=\text{Min}\left\\{\boldsymbol{\lambda}\in\boldsymbol{\Lambda}:\hat{R}_{n+1}(\boldsymbol{\lambda})\leq\alpha\right\\}$
$\displaystyle\boldsymbol{\hat{\lambda}}^{\prime}$
$\displaystyle\in\operatorname*{arg\,min}_{\boldsymbol{\lambda}\in\boldsymbol{\Lambda}_{\min}}g(\boldsymbol{\lambda})$

The fact that
$\inf_{\boldsymbol{\lambda}}L_{i}(\boldsymbol{\lambda})=L_{i}(\boldsymbol{\lambda}^{\max})\leq\alpha$
and the fact that $g$ is striclty increasing implies
$\boldsymbol{\hat{\lambda}}^{\prime}$ is well-defined almost surely. Since
$L_{n+1}(\boldsymbol{\lambda})\leq B$, we have

$\hat{R}_{n+1}(\boldsymbol{\lambda})=\frac{n}{n+1}\hat{R}_{n}(\boldsymbol{\lambda})+\frac{L_{n+1}(\boldsymbol{\lambda})}{n+1}\leq\frac{n}{n+1}\hat{R}_{n}(\boldsymbol{\lambda})+\frac{B}{n+1}.$

So if $\boldsymbol{\lambda}$ is such that

$\frac{n}{n+1}\hat{R}_{n}(\boldsymbol{\lambda})+\frac{B}{n+1}\leq\alpha,$

Then we know $\hat{R}_{n+1}(\boldsymbol{\lambda})\leq\alpha$. So when there
exists $\boldsymbol{\lambda}\in\boldsymbol{\Lambda}$ such that the the above
inequality holds, this implies that either
$\boldsymbol{\hat{\lambda}}^{\prime}\leq\boldsymbol{\hat{\lambda}}$ or
$\boldsymbol{\hat{\lambda}}\in\boldsymbol{\Lambda}_{\min}$, which in either
case implies $g(\boldsymbol{\hat{\lambda}}^{\prime})\leq
g(\boldsymbol{\hat{\lambda}})$. And if
$\frac{n}{n+1}\hat{R}_{n}(\boldsymbol{\lambda})+\frac{B}{n+1}>\alpha$ for all
$\boldsymbol{\lambda}\in\boldsymbol{\Lambda}$, then we must have
$\boldsymbol{\hat{\lambda}}=\boldsymbol{\lambda}^{\max}\geq\boldsymbol{\hat{\lambda}}^{\prime}$.
So we have $g(\boldsymbol{\hat{\lambda}}^{\prime})\leq
g(\boldsymbol{\hat{\lambda}})$ almost surely. And since
$L_{i}(\boldsymbol{\lambda})$ is non-increasing with respect to
$g(\boldsymbol{\lambda})$ we have

$\mathbb{E}\left[L_{n+1}(\boldsymbol{\hat{\lambda}})\right]\leq\mathbb{E}\left[L_{n+1}(\boldsymbol{\hat{\lambda}}^{\prime})\right].$

Let $E=\\{L_{1},\ldots,L_{n+1}\\}$, where $L_{n+1}(\boldsymbol{\lambda})\mid
E\sim\text{Uniform}(E)$ by exchangeability, and
$\boldsymbol{\hat{\lambda}}^{\prime}$ is constant conditional on $E$. Then
with the right-continuity of $L_{i}$ in each dimension we have

$\mathbb{E}\left[L_{n+1}(\boldsymbol{\hat{\lambda}}^{\prime})\mid
E\right]=\frac{1}{n+1}\sum_{i=1}^{n+1}L_{i}(\boldsymbol{\hat{\lambda}}^{\prime}_{i})\leq\alpha.$

Then applying the law of total expectation and our inequality from above we
have

$\displaystyle\mathbb{E}\left[L_{n+1}(\boldsymbol{\hat{\lambda}})\right]\leq\alpha,$

completing the proof. ∎

##  Appendix B Additional Examples

###  B.1 Additional monotonicity examples

We present more examples of the conformal alignment procedure for monotonicity
from Section 4.1. We keep everything the same in terms of 70-15-15 train-
calibrate-test split, averaging over 10 runs, and using XGBoost regression
models. All experiments were run on an Apple M1 MacBook Pro. We train one
constrained model on the train and calibration sets and one unconstrained
model on the training set. We use default parameters without hyperparameter
tuning. We use the same POT $\mathsf{T}$ to obtain loss functions for applying
conformal risk control on the unconstrained model using the calibration set.

Table 2: Abalone, $n=4177$. Monotonically increasing on Shell_weight. $\boldsymbol{\lambda}^{\max}=(5,5)$. $g(\boldsymbol{\lambda})=\lambda^{+}+\lambda^{-}$. $\alpha$ | $\boldsymbol{\lambda}$ | Metric | Unconstrained | Conformal | Constrained
---|---|---|---|---|---
0.1 | $\lambda^{+}=0.07$ | MSE | 5.49 | 5.49 | 5.38
$\lambda^{-}=0.00$ | Risk | 0.096 | 0.093 | 0
0.05 | $\lambda^{+}=0.29$ | MSE | 5.49 | 5.60 | 5.38
$\lambda^{-}=0.47$ | Risk | 0.096 | 0.062 | 0
0.01 | $\lambda^{+}=0.63$ | MSE | 5.49 | 6.83 | 5.38
$\lambda^{-}=1.95$ | Risk | 0.096 | 0.018 | 0
Table 3: Concrete, $n=1030$. Monotonically increasing on Cement. $\boldsymbol{\lambda}^{\max}=(2,2)$. $g(\boldsymbol{\lambda})=\lambda^{+}+\lambda^{-}$. $\alpha$ | $\boldsymbol{\lambda}$ | Metric | Unconstrained | Conformal | Constrained
---|---|---|---|---|---
0.1 | $\lambda^{+}=0.004$ | MSE | 24.4 | 24.4 | 17.55
$\lambda^{-}=0.008$ | Risk | 0.081 | 0.081 | 0
0.05 | $\lambda^{+}=0.092$ | MSE | 24.4 | 24.4 | 17.55
$\lambda^{-}=0.052$ | Risk | 0.081 | 0.075 | 0
0.01 | $\lambda^{+}=0.710$ | MSE | 24.4 | 24.6 | 17.55
$\lambda^{-}=0.739$ | Risk | 0.081 | 0.043 | 0

###  B.2 Concavity results

We report the results of applying the conformal alignment procedure for the
property of concacity using Algorithm 2 as our POT. Implementation details are
the same as for the monotonicity examples in Section 4.1 besides this change
of POT and our procedure for choosing a specific point in order to measure
performance on the accuracy metric, which here we take to be MAE. In this case
our choice of the point prediction uses a scaling adjustment factor based on
the position relative to the feature value at the maximum prediction, scaling
from adding 0 to the predictions at the minimum feature value to adding
$\lambda$ to the prediction at the feature value of the maximum prediction,
then adding 0 again at the maximum feature value. We re-emphasize, though,
that the actual point prediction is not our main concern but rather the risk,
which we can see matches the anticipated values well.

Table 4: Seoul Bike Sharing, $n=8760$. Concave on temperature. $\lambda^{\max}=100$. $\alpha$ | $\boldsymbol{\lambda}$ | Metric | Unconstrained | Conformal
---|---|---|---|---
0.1 | $\lambda=28.8$ | MAE | 115 | 117
Risk | 0.31 | 0.090
0.05 | $\lambda=45.0$ | MAE | 114 | 119
Risk | 0.31 | 0.049
0.01 | $\lambda=84.2$ | MAE | 115 | 129
Risk | 0.31 | 0.011

##  Appendix C Details for a stylized examination of alignment persistence in
AI models

We begin by describing the stylized model setting that can help us better
understand the impact of overparameterization and data quality on a model’s
ability to satisfy a property.

##### Notation.

We consider feature vectors $X$ to be samples from the uniform distribution on
the sphere of radius $\sqrt{d}$ in ${\mathbb{R}}^{d}$, that is denoted by
${\mathbb{S}}^{d-1}(\sqrt{d})$. For a function $f$, defined on a domain of the
random variable $X$, we define its $L^{2}$ norm by
$\|f\|_{L^{2}}^{2}={\mathbb{E}}_{X}[f(X)^{2}]$ where the expectation is with
respect to the randomness in $X$. The notation $o_{n,\mathbb{P}}(1)$ for a
positive integer means a quantity that goes to zero as $n$ goes to infinity.

###  C.1 Noisy data setting

##### Data model.

Consider a model where the data is generated according to the following
equation,

$y=g(X)+h(X)+\varepsilon\,,$

where $\varepsilon$ has mean $0$ and variance $\tau^{2}$ and is independent of
$X$. Assume that $h(X)$ represents ‘bad’ data that is undesired and we do not
want to learn. That is, we only want to learn $g(X)$, but there is a presence
of biased noise, $h(X)$, in the data generating model.

Relating this to an alignment setting, domain experts expect the data to be
generated according to $g(X)+\textit{unbiased noise}$, such that $g(X)$
exhibits some desired property ${\mathcal{P}}$/ However, the data is generated
according to $g(X)+\textit{biased noise}$, potentially obscuring the desired
behavior. One potential reason for the presence of biased noise in data could
be due to a measurement error of the outcome that is correlated with select
features, leading to an incorrectly calculated outcome. A biased measurement
error could occur if there is incomplete data and the presence of the
incomplete data is correlated with select features in an unexpected,
systematic way (e.g., it could hypothetically be correlated to the type of
hospital or location at which a patient is treated).

The goal of these analyses is then to examine the impact of the noise bias
term on a model’s ability to describe a behavior that is upheld by $g(X)$, but
not necessarily $g(X)+h(X)$, across model complexity. For example, $g(X)$
might be monotonically increasing in the first coordinate of ${\mathbf{x}}$,
but the summation of $g(X)+h(X)$ is not always. We will then try to draw
conclusions about the distance between two kinds of models: one that is
trained regularly without additional constraints, and another where we add a
constraint to the training that forces the model to replicate the behavior of
$g(X)$, despite $h(X)$. In the stylized model, when referring to the
‘distance’ between two models, we consider it to be zero if their learned
parameters are the same, and non-zero if their learned parameters are
different, when trained on the same data.

##### Learning method.

We will assume we have access to a data set of $n$ iid samples
$\\{(X_{i},Y_{i})\\}_{i=1}^{n}$ obtained from the above data generating model,
i.e., $Y_{i}=g(X_{i})+h(X_{i})+\varepsilon_{i}$. We denote this data set by
$D_{n}$.

The function class we use to fit the data is the random feature model, which
can be thought of as a two-layer neural network, with $N$ hidden nodes, with
the weights of the first layer set randomly, while the second layer weights
are learned. This means the function class can be written as:

$f_{{\mathsf{RF}}}(X;{\mathbf{a}},\\{{\mathbf{w}}_{j}\\}_{j\in[N]})=\frac{1}{\sqrt{N}}\sum_{j\in[N]}a_{j}\sigma(\langle
X,{\mathbf{w}}_{j}\rangle)\,,$ (2)

where ${\mathbf{a}}\in{\mathbb{R}}^{N}$ denotes the second layer weights and
$\sigma(\cdot):{\mathbb{R}}\to{\mathbb{R}}$ is a non-linear activation
function. The random weights $\\{{\mathbf{w}}_{j}\\}_{j\in[N]}$ are such that
each ${\mathbf{w}}_{j}$ is an iid sample from a fixed distribution. For
brevity, we drop the reference to ${\mathbf{a}}$ and ${\mathbf{w}}_{j}$’s and
use $f_{{\mathsf{RF}}}(X)$ or $f_{{\mathsf{RF}}}(X;{\mathbf{a}})$.

For the stylized model analysis, we consider a random feature model, because
this function class has been studied in recent theory of deep learning models,
and has shown some of the specific properties of modern overparameterized deep
learning models such as concepts of benign overfitting and double descent Mei
and Montanari , Misiakiewicz and Montanari [2023]. We can thereby use this
recent literature as a theoretical foundation to explain the potential impact
of model complexity on model’s ability to satisfy a property in large train
set sizes.

Interestingly, the squared loss function for this problem is convex and we can
minimize it by ridge regression. Mathematically, this means for any
regularization parameter, $\lambda$, we will search for a vector of real
numbers $\hat{{\mathbf{a}}}_{\lambda}\in{\mathbb{R}}^{N}$ such that,

$\hat{{\mathbf{a}}}_{\lambda}=\arg\min_{{\mathbf{a}}\in{\mathbb{R}}^{N}}\sum_{i\in[n]}\left(Y_{i}-f_{{\mathsf{RF}}}(X_{i})\right)^{2}+\lambda\|{\mathbf{a}}\|_{2}^{2}\,.$
(3)

###  C.2 RF theory

Assume a certain property, denoted by ${\mathcal{P}}$ such that the function
$g(X)$ satisfies ${\mathcal{P}}$. However, assume that $h(X)$ is such that
$g(X)+h(X)$ does not satisfy ${\mathcal{P}}$.

Assume there is a way of constraining the training of $f_{{\mathsf{RF}}}(X)$
to satisfy the ${\mathcal{P}}$. More formally, assume
$C_{{\mathcal{P}}}\subset{\mathbb{R}}^{N}$ is the set of all coefficients
${\mathbf{a}}$ such that $f_{{\mathsf{RF}}}(X;{\mathbf{a}})$ satisfies
${\mathcal{P}}$, i.e.,

$C_{{\mathcal{P}}}=\\{{\mathbf{a}}~{}|~{}{\mathbf{a}}\in{\mathbb{R}}^{N}~{}\text{and}~{}f_{{\mathsf{RF}}}(X;{\mathbf{a}})~{}\text{satisfies}~{}{\mathcal{P}}\\}\,.$

Then we can consider the following constrained version of the model:

$\hat{{\mathbf{a}}}_{\lambda,{\mathcal{P}}}=\arg\min_{{\mathbf{a}}\in
C_{{\mathcal{P}}}}\sum_{i\in[n]}\left(Y_{i}-f_{{\mathsf{RF}}}(X_{i})\right)^{2}+\lambda\|{\mathbf{a}}\|_{2}^{2}\,.$
(4)

Now assume that the functions $g$ and $h$ are polynomials such that
$\deg_{g}<\deg_{h}$. Further assume that $n>d^{\deg_{h}+\delta}$, where
$\delta$ is some small positive constant (e.g., $\delta=0.1$) and that $d$ is
sufficiently large. Also assume that the activation function $\sigma$ upholds
the following genericity conditions (as stated and assumed in Theorem 3 in
Section 4.2 of Misiakiewicz and Montanari [2023]): 1) $|\sigma(x)|\leq
c_{0}\exp(c_{1}|x|)$ for constants $c_{0},c_{1}>0$; 2) for any $k\geq 0$,
$\sigma$ has a non-zero Hermite coefficient
$\mu_{k}(\sigma):=\mathbb{E}_{G}\\{\sigma(G)\text{He}_{k}(G)\\}\neq 0$ (where
$G\sim{\mathcal{N}}(0,1)$ and $\text{He}_{k}$ is the $k$-th Hermite
polynomial, with the standard normalization
$\langle\text{He}_{j},\text{He}_{k}\rangle_{L^{2}({\mathcal{N}}(0,1))}=k!\mathbbm{1}_{\\{j=k\\}}$).

To analyze the impact of model complexity on a model’s adherence to
${\mathcal{P}}$, we consider two settings for the stylized model:

  * •

Classic: $d^{\deg_{g}+\delta}<N<d^{\deg_{h}-\delta}$.

  * •

Underspecified: $N>d^{\deg_{h}+\delta}$.

To formalize our results, we will in part rely on Theorem 3 in Section 4.2 of
Misiakiewicz and Montanari [2023]. This theorem illustrates the roles that the
number of hidden nodes, $N$, and the number of data points, $n$, play in
determining the test error that can be achieved by a random feature model when
approximating a function. Given the assumptions made in their work, in the
setting where $N\ll n$, the number of hidden nodes, $N$, will limit the test
error, while if $n\ll N$, the number of data points, $n$, will limit the test
error Misiakiewicz and Montanari [2023]. Building on this intuition, we first
state an informal implication of this theorem Misiakiewicz and Montanari
[2023] applied to our stylized model: under the same assumptions made in their
work, for a sufficiently large $d$, in the classic setting, there exists a
range of regularization parameters $[0,\lambda^{*}]$ such that for all
$\lambda$ in that range, $f_{{\mathsf{RF}}}(X;\hat{{\mathbf{a}}}_{\lambda})$
will learn a polynomial of at least $\deg_{g}$, but strictly less than
$\deg_{h}$. But in the underspecified setting,
$f_{{\mathsf{RF}}}(X;\hat{{\mathbf{a}}}_{\lambda})$ learns both parts $g(X)$
and $h(X)$, for all $\lambda$ in a different range of regularization
parameters $[0,\lambda^{*}]$, meaning it learns the noise bias. This means, in
a large data domain, the behavior of the classic and underspecified setting is
different.

In order to formalize this intuition for the underspecified setting, we need
to state an assumption on the degree of violation of property ${\mathcal{B}}$
around the function $g+h$.

######  Assumption 1 (Property Unsatisfied in a Neighborhood).

There exists a positive constant $R$ such that, for all
${\mathbf{a}}^{\prime}$ with
$\|(g+h)-f_{{\mathsf{RF}}}(\cdot;{\mathbf{a}}^{\prime})\|_{L^{2}}\leq R$, the
function $f_{{\mathsf{RF}}}(X,{\mathbf{a}}^{\prime})$ does not satisfy the
property ${\mathcal{P}}$.

######  Theorem 2 (Non-vanishing Distance in Underspecified Setting).

Assume $N>d^{\deg_{h}+\delta}$, $n>d^{\deg_{h}+\delta}$, $\max(N/n,n/N)\geq
d^{\delta}$ for a constant $\delta>0$, Assumption 1 holds, $\sigma$ upholds
the genericity conditions stated above, and $d$ is sufficiently large. Then,
in the underspecified setting, there exists a constant $\lambda^{*}$, such
that for any $\lambda\in[0,\lambda^{*}]$ the following holds:

$\hat{{\mathbf{a}}}_{\lambda,{\mathcal{P}}}\neq\hat{{\mathbf{a}}}_{\lambda}\,.$
(5)

###### Proof.

Using Theorem 3 of Misiakiewicz and Montanari [2023], we have that

$\displaystyle\|(g+h)-f_{{\mathsf{RF}}}(\cdot;\hat{{\mathbf{a}}}_{\lambda})\|_{L^{2}}^{2}$
$\displaystyle=o_{d,\mathbb{P}}(1)\cdot(\|g+h\|_{L^{2+\eta}}^{2}+\tau^{2}).$

for all $\eta>0$. With $d$ sufficiently large, the square root of the quantity
on the right hand side will be less than $R$, which combined with Assumption
1, means $f_{{\mathsf{RF}}}(\cdot;\hat{{\mathbf{a}}}_{\lambda})$ does not
satisfy ${\mathcal{P}}$, and therefore, $\hat{{\mathbf{a}}}_{\lambda}\notin
C_{{\mathcal{P}}}$. ∎

Under two additional assumptions, we can further state a result showing that
in the classic setting, for a given range of regularization parameters, we
will be able to learn the property ${\mathcal{P}}$. At a high level, these
assumptions are that 1) the overall impact of $h$ is small, for example, when
it only impacts a small portion of the data; and 2) the property
${\mathcal{P}}$ is preserved in a small neighborhood of $g$ among the function
class learned in the classic setting. Formalizing the assumptions and stating
the result:

######  Assumption 2 (Small Noise Bias).

Assume there is a small constant $\epsilon_{h}$ such that

$\|h\|_{L^{2}}\leq\epsilon_{h}\,.$

######  Assumption 3 (Property Robustness in Function Class).

For any fixed $N$, there exists a positive constant $R_{N}$ such that, for all
${\mathbf{a}}^{\prime}$ with
$\|g-f_{{\mathsf{RF}}}(\cdot;{\mathbf{a}}^{\prime})\|_{L^{2}}\leq R_{N}$, the
function $f_{{\mathsf{RF}}}(X,{\mathbf{a}}^{\prime})$ satisfies the property
${\mathcal{P}}$.

######  Theorem 3 (Zero Distance in Classic Setting).

Assume $d^{\deg_{g}+\delta}<N<d^{\deg_{h}-\delta}$, $n>d^{\deg_{h}+\delta}$,
$n/N\geq d^{\delta}$ for a constant $\delta>0$, Assumptions 2-3 hold, $\sigma$
upholds the genericity conditions, $d$ is sufficiently large, and
$R_{N}>2\epsilon_{h}$. Then, in the classic setting, there exists a constant
$\lambda^{*}$, such that for any $\lambda\in[0,\lambda^{*}]$, the following
holds:

$\hat{{\mathbf{a}}}_{\lambda,{\mathcal{P}}}=\hat{{\mathbf{a}}}_{\lambda}\,.$
(6)

###### Proof.

Using the triangle inequality and Theorem 3 of Misiakiewicz and Montanari
[2023] again, we have that

$\displaystyle\|f_{{\mathsf{RF}}}(\cdot;\hat{{\mathbf{a}}}_{\lambda})-g\|_{L^{2}}^{2}$
$\displaystyle\leq
2(\|f_{{\mathsf{RF}}}(\cdot;\hat{{\mathbf{a}}}_{\lambda})-g-h\|_{L^{2}}^{2}+\|h\|_{L^{2}}^{2})$
$\displaystyle\leq 4\|h\|_{L^{2}}^{2}$ $\displaystyle\leq
4\epsilon_{h}^{2}\,.$

Taking the square root, the right hand side of the last term is less than
$R_{N}$ which combined with Assumption 3 means
$f_{{\mathsf{RF}}}(\cdot;\hat{{\mathbf{a}}}_{\lambda})$ satisfies property
${\mathcal{P}}$. ∎

######  Remark 1.

Note that the constant $R_{N}$ will converge to zero as $N$ grows, meaning the
assumption $R_{N}>2\epsilon_{h}$ will break down when $N$ is large. For
example, in the underspecified setting, this assumption breaks down and hence
the distance does not vanish.

###  C.3 Synthetic validation under a relaxed assumption on the noise bias,
and conformal correction

In this secton, we first numerically validate the above theoretical results,
in a setting that noise bias only impacts a small fraction of the pre-training
data. Next, we apply our proposed procedure of alignment through conformal
risk control. We take the desired property ${\mathcal{P}}$ to be monotonicity
and use the same procedure as on the real-world datasets in Section 4.1.

We consider $g(X)=\langle X,\beta\rangle$ with $\beta$ a constant in
$\mathbb{R}^{d}$ with $\beta_{1}>0$ and $\|\beta\|_{2}=1$. Then let
$h(X)=\min(M,\alpha Z_{p}x_{1}^{4})$ where $Z_{p}$ is a Bernoulli random
variable with a success probability $p$, that is sampled independently for
each data point, and $\alpha$ and $M$ are positive constants. This function is
aimed to model a biased noise that is capped by magnitude $M$ and only applies
to a fraction $p$ of the population. This is to mimic characteristics of a
real dataset where only a portion of the data might have the bias (i.e., ‘bad’
data). This is therefore capturing a more realistic case of the stylized model
setting, where the measurement error may only occur for a small subset of the
data.

Using a large train set size, we fit the data generated according to the
outcome model above using the random feature model with $\lambda$ tuned via
cross-validation, assuming the ReLU activation function, i.e.,
$\sigma(x)=\max(0,x)$, and ${\mathbf{w}}_{j}$ is iid ${\mathcal{N}}(0,1)$. We
then assess how model complexity affects whether the predicted function is
monotone in $x_{1}$ by looking at the univariate partial dependence plot of
the predicted function versus the (counterfactual) unbiased outcome, $g(X)$,
for the first coordinate $x_{1}$. We vary model complexity by $N$, the number
of hidden nodes in the model.

Applying this synthetic simulation for specific parameters, Figure 2 shows
that when training a relatively simple model ($N=5$) on the generated data,
the model describes a monotonically increasing relationship. From this we
conclude that if we had constrained the model to uphold monotonicity in
$x_{1}$, it would not have changed the final model’s parameters. Hence, the
distance between the constrained and unconstrained model would be zero and we
require no conformal correction. In contrast, as seen in Figure 3, a more
complex model ($N=5000)$ predicts a non-monotone relationship, and therefore,
if we had constrained the model to uphold monotonicity in $x_{1}$, it would
have changed the final model’s parameters, such that the distance between the
constrained and unconstrained model would be non-zero and we do require a
conformal correction to post-process and align this model.

The synthetic simulation seems to validate that with a seemingly large
training dataset, increasing model complexity, on the one hand, can enable
high performance, but can also make ML models more sensitive to small noise
bias in the data, which can subsequently lead to model behavior inconsistent
with requirements by the end-user.

Figure 2: Random Feature model (N=5). Figure 3: Random Feature model (N=5000).


Easwarakumar, 2017, ‘View-Invariant Gait Recognition Through Genetic Template
Segmentation,’ IEEE Signal Processing Letters. Volume 24, Issue 8, pp
1188–1192. Impact Factor: 2.528.

  3. 3.

Ebenezer R.H.P. Isaac, Susan Elias, Srinivasan Rajagopalan, and K.S.
Easwarakumar, ‘Gait Verification System through Multiperson Signature Matching
for Unobtrusive Biometric Authentication,’ revised version submitted to
Journal of Signal Processing Systems, Springer. Impact Factor: 0.893.

  4. 4.

Ebenezer R.H.P. Isaac, Susan Elias, Srinivasan Rajagopalan, and K.S.
Easwarakumar, ‘Multiview Gait-Based Gender Classification through Pose-Based
Voting,’ revised version submitted to Pattern Recognition Letters, Elsevier.
Impact Factor: 1.995

  5. 5.

Ebenezer R.H.P. Isaac, Susan Elias, Srinivasan Rajagopalan, and K.S.
Easwarakumar, ‘Template-Based Gait Authentication through Bayesian
Thresholding,’ submitted to IEEE/CAA Journal of Automatica Sinica.