# Robust model-based estimation for binary outcomes in genomics studies

Suyoung Park1, Alexander E. Lipka2, Daniel J. Eck1
1\. Department of Statistics, University of Illinois Urbana-Champaign
2\. Department of Crop Sciences, University of Illinois Urbana-Champaign

(October, 2021)

###### Abstract

In quantitative genetics, statistical modeling techniques are used to
facilitate advances in the understanding of which genes underlie agronomically
important traits and have enabled the use of genome-wide markers to accelerate
genetic gain. The logistic regression model is a statistically optimal
approach for quantitative genetics analysis of binary traits. To encourage
more widespread use of the logistic model in such analyses, efforts need to be
made to address separation, which occurs whenever a specific combination of
predictors can perfectly predict the value of a binary trait. Data separation
is especially prevalent in applications where the number of predictors is near
the sample size. In this study we motivate a logistic model that is robust to
separation, and we develop a novel prediction procedure for this robust model
that is appropriate when separation exists. We show that this robust model
offers superior inferences and comparable predictions to existing approaches
while remaining true to the logistic model. This is an improvement to
previously existing approaches which treats separation as a modeling
shortcoming and not an antagonistic data configuration. Previous approaches,
therefore, change the modeling paradigm to consider separation that, before
our robust model exists, is problematic to logistic models. Our comparisons
are conducted on several didactic examples and a genomics study on the kernel
color in maize. The ensuing analyses reaffirm the billed superior inferences
and comparable predictive performance of our robust model. Therefore, our
approach provides scientists with an appropriate statistical modeling
framework for analyses involving agronomically important binary traits.

Keywords: Logistic regression; Complete separation; Quantitative genetics

##  1 Introduction

The application of quantitative genetics approaches to crops has facilitated
advances in the basic understanding of which genes underlie agronomically
important traits, and has enabled the use of genome-wide markers to accelerate
genetic gain. For example, the use of statistical models in genome-wide
association studies has provided insight into the role of pleiotropy in the
genetic architecture of leaf and inflorescence-related traits in maize [Rice
et al., 2020]. Similarly, multi-kernel genomic prediction (GP) models that
include environmental covariate information have made it possible to
accurately predict genomic estimated breeding values (GEBVs) for grain yield
in wheat in specific environments [Jarquín et al., 2014]. Although less
commonplace than quantitative traits, agronomically important binary traits do
occur. For example the color of kernels in maize is often dichotomized into a
binary trait (white versus yellow; [Romay et al., 2013]), and breeding for
kernel color is a critical step for increasing bioavailablity of provitamin A
carotenoids in maize grain [Chandler et al., 2013; Harjes et al., 2008]. Thus,
the application of quantitative genetic analysis to binary traits has great
potential to have a meaningful impact on future agronomic efforts. However, a
major setback is that some of the most widely-used quantitative genetics
approaches in agronomy do not account for the dichotomous configuration of a
binary trait. Consequently, direct application of state-of-the-art
quantitative genetic approaches to study binary traits could result in
negative statistical ramifications, including inadequate control of inflation
of test statistics due to subpopulation structure (as shown in, e.g.
[Shenstone et al., 2018]).

From a statistical perspective, the most logical model to apply to the
analysis of binary traits is the logistic regression model [Agresti, 2013], in
which a binary outcome variable depends on a set of explanatory variables.
However, a major setback of the logistic regression model is that it breaks
down whenever there exists a linear combination of the explanatory variables
in which a binary outcome of ”0” or ”1” is guaranteed to be observed. Whenever
this phenomenon, known as separation, is present in data, the logistic model
coefficient estimates are not finite (or unstable), and it becomes impossible
to obtain a reasonable estimate of the sizes of marker effects. Given that
statistical software typically fails to diagnose this issue [Eck and Geyer,
2021] and that the number of markers included in a logistic regression model
is likely to be large and can exceed the sample size (especially in genomic
prediction applications), it is critical that the issue of separation in
quantitative genetic data is carefully assessed and the potential of remedial
statistical approaches are considered.

The easiest way to deal with separation, when it is detected, is to remove
markers with uninterpretable effect sizes from the model. However, this naïve
approach often leads us to get rid of the markers with the strongest
associations with the binary trait [Zorn, 2005]. To eliminate the need to take
such drastic measures, modifications to logistic regression have been
proposed. For example Heinze and Schemper [2002] used the Firth’s penalized
maximum likelihood estimation to reduce the bias of maximum likelihood
estimator, and thus obtained reasonable, finite estimates of marker effects.
Kosmidis and Firth [2009] generalized this method for the nonlinear
exponential family, thus making Firth’s penalized likelihood available to
traits following any distribution of the exponential family. Additionally,
many have proposed a Bayesian framework to handle the estimation problems that
arise from separation [Heinze and Schemper, 2002; Dunson et al., 2006; Genkin
et al., 2007; Gelman et al., 2008]. In fact, Heinze and Schemper’s method can
be interpreted from a Bayesian perspective as the application of Jeffrey’s
invariant prior. Further examples of approaches used to deal with separation
include Dunson et al. [2006], which used the mixture prior distributions for
the logistic model with large number of markers, Genkin et al. [2007]
considered the Laplace prior distribution, and Gelman et al. [2008] suggested
the Cauchy distribution with center 0 and scale 2.5 as the default choice.
This latter approach showed faster and comparable predictive performance in
terms of cross-validation predictive error than other methods.

Both bias reduction and Bayesian approaches handled the separation issue by
switching the modeling paradigm to accommodate problematic data rather than
solving the issue within the original model. Geyer [2009] developed
methodology for directly finding maximum likelihood estimates (MLE) when
problematic separation exists and the traditional MLE calculations do not
converge. This method required a massive computation cost which makes it time
consuming to apply in practice. Eck and Geyer [2021] proposed new, faster and
scalable methodology to find the MLE in the completion when MLE does not
exist. The methods of Geyer [2009] and Eck and Geyer [2021] work by first
detecting separation when it exists, and then providing one-sided confidence
intervals for problematic data points that exist in the separated space and
usual two-sided confidence intervals for non-problematic data points. Eck and
Geyer’s method is implemented in the R package glmdr, software which detects
and remedies separation in logistic regression.

In this study we developed a prediction framework compatible with Eck and
Geyer [2021]’s method. This prediction method did not previously exist. We
begin by motivating the logistic regression model for quantitative genetics
analysis of binary, that is, Bernoulli-distributed quantitative traits. We
then describe the problem of separation in the data, and compare different
techniques (Bayesian, penalized likelihood, and maximum likelihood estimation)
for handling separation on several datasets, as well as a quantitative
genetics dataset from maize [Romay et al., 2013]. Next we assess performance
of these techniques on their inferential and predictive ability. Because the
MLE asymptotically achieves the Cramér-Rao lower bound, we expected the MLE
technique in Eck and Geyer [2021] to yield the tightest inferences among all
techniques under consideration. This finding is confirmed in all datasets that
we considered. We then motivate our novel prediction procedure within the
methodological context of Eck and Geyer [2021]. We expected that our developed
method and the other considered methods to exhibit even predictive performance
with a computational edge towards the Bayesian techniques that we considered.
While the method of Eck and Geyer [2021] is far more computationally
convenient than that of Geyer [2009] it is still rather involved when adapted
for prediction. Ultimately we want to develop methodology in Eck and Geyer
[2021] that is applicable for genomic prediction when there may be far more
predictors than cases. The prediction procedures developed here are an
important step in that direction.

##  2 Materials and Methods

###  2.1 Logistic Regression

The logistic regression is a special case of the generalized linear model
which the outcome variable follows a Bernoulli distribution (i.e.,
$y\in\\{0,1\\}$) [Nelder and Wedderburn, 1972]. By convention, we encode 1 as
a “success” and 0 as a “failure.” In logistic regression the conditional
success probability at a particular $x$ is modeled as

$\text{Pr}(Y=1|X=x)=\frac{\exp{(x^{T}\beta)}}{1+\exp{(x^{T}\beta)}}=\pi(x),$
(1)

where $\beta$ is an unknown canonical parameter vector (coefficient vector),
$X$ and $Y$ are the covariate and outcome random variables, and $x$ is an
observed value.

From the linear regression’s point of view, this logistic regression is
equivalent to:

$g(\pi(x))=\log\left(\frac{\pi(x)}{1-\pi(x)}\right)=x^{T}\beta$ (2)

where $g(x)=\log(\frac{x}{1-x})$ is a logit link (log-odds ratio).

Therefore, as in classical ordinary least squares (OLS) regression, we can
estimate model parameters using maximum likelihood estimation. Statistical
inferences about model parameters can be obtained from estimates of the Fisher
information. Unlike in OLS regression, estimates for $\hat{\beta}$ are not
given in closed form. The log-likelihood function for the logistic regression
model is

$\log
L(\beta|Y)=\sum_{i=1}^{n}y_{i}\log{(\pi(x_{i}))}+(1-y_{i})\log{(1-\pi(x_{i}))},$
(3)

one then obtains $\hat{\beta}$ by solving the score function equation

$\frac{\partial\log{L(\beta|Y)}}{\partial\beta}=\sum_{i=1}^{N}(y_{i}-\log{(\pi(x_{i}))})x_{i}^{T}=\sum_{i=1}^{N}[y_{i}+\log{(1+\exp{(-x_{i}^{T}\beta)})}]=0.$
(4)

Conventional softwares finds $\hat{\beta}$ through Fisher-scoring or
iteratively reweighted least squares algorithms [Agresti, 2013, Chapter 4]. We
then obtain inferences using an estimate of the Fisher information matrix
evaluated at the MLE solution $\hat{\beta}$

$\widehat{\text{Var}(\beta)}=[I(\hat{\beta})]^{-1}=\left(-E\left[\frac{\partial^{2}\log{L(\beta|Y)}}{\partial\beta_{i}\partial\beta_{j}}\right]\right)^{-1}\Big{|}_{\beta=\hat{\beta}}.$
(5)

Conventional software provides (5).

###  2.2 Complete Separation

Traditional maximum likelihood estimation for logistic regression does not
work well when there is complete or quasi-complete separation in the data, a
problem that is widespread in applications [Geyer, 2009]. Agresti [2013]
defined complete separation when there exists a vector $b$ such that

$\begin{split}x_{i}^{T}b&>0\text{ whenever }y_{i}=1,\\\ x_{i}^{T}b&<0\text{
whenever }y_{i}=0.\end{split}$ (6)

That is, complete separation occurs when the one or more covariates can
perfectly predict the outcome variable [Albert and Anderson, 1984]. For
example, as shown in the Figure 1, consider the following case that when $z$
is less than 50, all corresponding $y$ are $0$ and when $z$ is greater than
50, all corresponding $y$ are $1$. Suppose we are interested in a simple
logistic regression model $x_{i}^{T}=[1,z_{i}]$. Then this data is completely
separated with $b=[-50,1]^{T}$. Moreover, we have $\hat{\pi}(x)=0$ for $z<50$
and $\hat{\pi}(x)=1$ for $z>50$.

Figure 1: Example of complete separation from Section 6.5.1 of Agresti [2013].
The conventional MLE of a logistic model does not exist.

The parameter estimates $\hat{\beta}$ are “at infinity” when separation exists
in the data. The iteration based estimation algorithms under the hood of
conventional software packages provide a sequence of estimates that goes to
infinity, and the log likelihood becomes flat when evaluated along this
sequence. The left panel of Figure 2 shows the log likelihood of logistic
model for this example with different working estimate from glm function in R.
We can see that each iteration, the norm of $\beta$ becomes larger and
asymptote of the log likelihood value goes to infinity. The right panel of
Figure 2 is the zoomed part of the left panel of Figure 2 where the log of
norm of working estimates is between 4.5 and 5. It displays the log likelihood
value still approaches near zero although the left panel of Figure 2 looks
flat in the same region.

Standard statistical inferences are not appropriate when separation exists.
The standard errors of predicted probabilities of success are very small,
which leads to extremely narrow confidence intervals for each observation.
Unfortunately, none of common statistical software such as R, SAS and Python
can handle the separation issue properly and uninformed users sometimes uses
the wrong model without knowing it [R Core Team, 2020; SAS Institute Inc.,
2003; Van Rossum and Drake Jr, 1995]. The glmdr software package [Geyer et
al., 2021] is designed to provide users with a description of the separation
problem, and provide statistical inferences when it occurs. This package did
not have a prediction routine before this work.

Figure 2: Left panel: Log likelihood values of logistic model at different
working estimates. Blue dot represents the log likelihood value at each
iteration. Right panel: Zoom in view of a log likelihood values of logistic
model where log of norm of working estimates lie between 4.5 and 5.

Quasi-complete separation is another case of separation that there are both a
success and a failure on the hyperplane that separates the successes from the
failures [Lesaffre and Albert, 1989]. For instance, we can consider additional
two points that $z=50$ with $y=1$ and $y=0$ to the previous complete
separation example. That is, we have $y_{i}=0$ for $z\leq 50$ and $y_{i}=1$
for $z\geq 50$. In this case, the maximized log likelihood is always negative
and we experience same phenomenon as the complete separation case.

###  2.3 Mean-value Parameters

The mean-value parameter is the conditional expectation of the response
variable given a covariate value $\text{E}(Y|X=x)$. In logistic regression the
mean-value parameter is modeled as (1) where
$\text{E}(Y|X=x)=\text{Pr}(Y=1|X=x)$. The mean-value parameter is often the
most important parameter for inference, we want to know the success
probability at a particular $x$. An example would be to model the probability
of the kernel color being white or yellow given the marker genotypes encoded
in the covariate vector $x$. Unlike in linear regression, this parameter is
not easily interpreted from $\beta$. Thus inferences about $\beta$ does not
provide useful inferences for $\text{Pr}(Y=1|X=x)$. Furthermore, the natural
constraints on a conditional probability corresponding to a binary outcome
variable require an alteration to the linear model. We expand on these points
in the next paragraph.

In linear regression we can easily obtain $\pi(x)=\text{E}(Y|X=x)$ from
$\beta$ since $\text{E}(Y|X=x)=x^{T}\beta$. Plugging in $\hat{\beta}$ produces
the MLE for this expectation
$\hat{\pi}(x)=\widehat{\text{E}}(Y|X=x)=x^{T}\hat{\beta}$ with $x$ fixed. On
the other hand, in the logistic model,
$\pi(x)=\text{E}(Y|X=x)=\text{Pr}(Y=1|X=x)$ where
$\log(\frac{\pi(x)}{1-\pi(x)})=x^{T}\beta$. Thus, $\beta$ does not offer an
easy interpretation about changes in the expected outcome as the covariates
change, and it is therefore less useful as a parameter for understanding how
$\pi(x)$ changes with $x$. The mean-value parametrization is the primary
parameter of interest in both regression contexts, but in linear regression
the mean-value parameter and $\beta$ are interchangeable.

Another benefit of the mean-value parameterization over $\beta$ in the
logistic regression model is when complete separation exists. When complete
separation exists $\beta$ is estimated to be at infinity while $\pi(x)$ is
estimated to be 0 or 1. We discuss complete separation and methods which
address it in the next Section.

###  2.4 One-Sided Confidence Interval

Standard statistical inferences cannot be performed in the presence of
separation. In this setting, statistical inferences will be made for estimated
mean-value parameters. Separation necessitates the use of one-sided confidence
intervals for estimated mean-value parameters since $\beta$ is estimated to be
at infinity. This is because when separation exists $\pi(x)$ is estimated to
be 0 or 1, the boundary of allowable values, and two-sided intervals would
therefore include values that are lower than 0 or greater than 1. The original
concept for the one-sided intervals that we use in this paper can be found in
Section 3.16 of Geyer [2009] with implementation details can be found in
Section 4.3 of Eck and Geyer [2021]. These one-sided intervals are
specifically tailored to handle separation and they are what form our robust
logistic regression model.

For completeness we briefly explain how we construct these one-sided
confidence interval for mean-value parameters. One endpoint of the one-sided
interval is constrained to be the observed outcome variable (i.e., lower bound
if $y_{i}=0$ and upper bound if $y_{i}=1$), and the other endpoint is obtained
by solving the optimization problem:

$\begin{split}\text{minimize }&\quad-\theta_{k}\\\ \text{subject to
}&\sum_{i\in
I}[y_{i}\log{(\pi(x_{i}))}+(1-y_{i})\log{(1-\pi(x_{i})})]-\log{(\alpha)\geq
0},\end{split}$ (7)

where $\theta_{k}=x_{k}^{T}\beta$ for any $k\in I$, I is a index of
problematic points that cause the separation, $\pi$ is a mean-value parameter,
and $\alpha$ is a significance level. For example, Figure 3 shows the one-
sided confidence interval for the complete separation example we discussed in
Section 2.2. We can see the confidence interval increases as $z$ increases
until $z=40$ then it starts to decrease as $z$ increases from $z=60$. Also, we
have a widest interval where $z=40$ and $z=60$ with the length of intervals,
$1-\alpha$. It means our uncertainty on estimation keep increases from $z=10$
to $z=40$ and we have the highest uncertainty near the separation occurs. Then
it diminishes as it furthers away from the boundary of the separation. In
glmdr, inference function provides this confidence intervals using the
sequential quadratic programming (SQP) to solve the constrained nonlinear
problem (7).

Figure 3: One-sided 95% confidence interval for the example of complete
separation from Section 2.2. Solid dot represents the observed value and bar
shows the interval. $\hat{\pi}(x)$ is the estimated probability of a success
given $z$ where $x=(1,\;z)^{T}$.

###  2.5 Prediction

Model based prediction is different when data separation is present. Standard
techniques fail in the presence of data separation. In the absence of
separation we can compute the predicted value for new data point from the
logistic model using
$\hat{\pi}(x_{\text{new}})=(1+\exp{(-x_{\text{new}}^{T}\hat{\beta})})^{-1}.$
However, when complete separation is present, this approach breaks down since
$\hat{\beta}$ is at infinity in this setting, and this implies that
$\hat{\pi}(x_{\text{new}})$ is either 0 or 1. Standard estimates of
variability suffer from a similar problem. Another difficulty is due to
uncertainty in the separation itself. Data separation occurs with probability
tending to zero as the sample size increases with the number of predictors
fixed. Data separation is therefore a sampling issue, not a modeling issue. We
propose a new method for prediction that addresses the practical and
conceptual difficulties of prediction when separation exists.

This method is as follows:

  1. 1.

Pick a point $x_{\text{new}}$ for which a prediction is desired and there
exists separation at $x_{\text{new}}$. A prediction at $x_{\text{new}}$ is
either 0 or 1 using traditional methods.

  2. 2.

We then combine this point with the observed data. We will make a model-based
estimate at $x_{\text{new}}$ by fitting separate logistic regression models,
one outcome label $y_{\text{new}}=0$ and the other with $y_{\text{new}}=1$.
Fitting two separate models in this way is intended to address the uncertainty
in the data separation.

  3. 3.

We then compute the estimated probability of a success for new data points,
$\hat{\pi}(x_{\text{new,0}})$ and $\hat{\pi}(x_{\text{new,1}})$. Note that one
of $\hat{\pi}(x_{\text{new,0}})$ or $\hat{\pi}(x_{\text{new,1}})$ will be 0 or
1 and the other will not be, this is because $y_{\text{new}}=0$ or
$y_{\text{new}}=1$ decreases the uncertainty in the separation by adding a
pseudo-outcome in its favor, while the other pseudo-outcome alleviates the
separation at $x_{\text{new}}$.

  4. 4.

We now combine $\hat{\pi}(x_{\text{new,0}})$ and $\hat{\pi}(x_{\text{new,1}})$
to form a prediction using model averaging. Our model averaging procedure
judges the fit of each model based on weights similar to the the Akaike
weights in Burnham and Anderson [2002]. These weights are

$w_{j}=\frac{{\exp(-\frac{IC_{j}}{2})}}{\exp(-\frac{IC_{0}}{2})+\exp(-\frac{IC_{1}}{2})},$

where $IC_{j}$ is the information criteria of model $j$.

  5. 5.

We then can calculate the model averaged estimate,
$\hat{\pi}^{*}(x_{\text{new}})=\sum_{j=0}^{1}w_{j}\hat{\pi}(x_{\text{new,j}})$.
We used Akaike information criteria corrected (AICc) as $IC_{j}$. The primary
reason for its use is that AICc does not have an overfit problem when the
sample size is small [Sugiura, 1978]. The presence of data separation is an
indication that one is not close to asymptopia, the sample size is small in
this sense.

  6. 6.

We then label $y^{*}_{\text{new}}$ as $1$ if
$\hat{\pi}^{*}(x_{\text{new}})\geq C^{*}$ and $0$ if
$\hat{\pi}^{*}(x_{\text{new}})<C^{*}$ where $C^{*}$ is the optimal cut-off
that maximizes the overall accuracy. The main motivation of using optimal cut-
off is that threshold of $0.5$ produces unreliable and poor model accuracy
when the outcome variable is highly unbalanced [Freeman and Moisen, 2008].

These six steps are outlined in Figure 4. Detailed implementation of this
prediction method and examples are given in the Supplementary Materials.

Figure 4: Flowchart for the prediction method.

We also calculate the prediction intervals for model averaged estimate,
$\hat{\pi}^{*}(x_{\text{new}})$. We construct the Wilson intervals [1927] for
predicted probabilities. Wilson intervals show better coverage probability
although $\hat{\pi}^{*}(x_{\text{new}})$ is near $0$ and $1$ boundaries in
comparison to the standard binomial confidence interval because Wilson
intervals are asymmetric [Brown et al., 2001].

###  2.6 Model Performance

####  2.6.1 Competing models

We compare the performance of direct maximum likelihood estimation fit with
glmdr with Bayesian generalized linear regression models fit with bayesglm,
bias reduction generalized linear regression models fit with brglm, and the
classical linear model fit with lm. Briefly, these different models are
summarized as:

  1. 1.

glmdr: glmdr stands for “generalized linear models done right” [Geyer et al.,
2021]. It solves the separation issue within the original model settings.
glmdr provides the inference by estimating the probability of success based on
MLE in Barndorff-Nielsen completion [1978], and calculating one-sided
confidence intervals. glmdr makes a prediction based on the model averaged
estimate.

  2. 2.

bayesglm: bayesglm is the generalized linear model with student-t prior
distribution [Gelman et al., 2008]. It scales the original data then uses
Cauchy distribution as a prior distribution on the coefficients to handle the
separation issue.

  3. 3.

brglm: brglm denotes “bias reduction in generalized linear model” [Kosmidis
and Firth, 2009]. It modifies the score function with powers of the Jeffreys
prior as penalty to produce the finite coefficients (penalized MLE) to deal
with the separation issue.

  4. 4.

lm: lm stands for “linear model” that provides the estimates based on ordinary
least squares method. It is one of the most common statistical models for
regression problems.

####  2.6.2 Performance metrics

We assess model performance using several metrics, these are:

  1. 1.

In-sample estimation accuracy in which we count the number of correctly
classified observations in the training set divided by total number of
observations in the training set.

  2. 2.

Out-of-sample prediction accuracy which is assessed via leave-one-out cross
validation (LOOCV). Since the predicted value of linear model does not have to
fall into $[0,1]$ range, we assign $1$ for any predicted values greater than
$1$ and $0$ for negative values.

  3. 3.

The average length of each confidence and prediction interval.

  4. 4.

The computational cost in seconds of each prediction method. This cost is
calculated using the proc.time function in R.

###  2.7 Data

We provide inference and prediction results for the maize data, a dataset
corresponding to an endometrial cancer study, and several didactic examples.
These didactic examples are motivated in Agresti [2013] and Geyer [2009], they
are designed to be understandable for audiences new to separation in the data.
A description of these data sets follows:

Complete separation: This example comes from Agresti [2013] and is discussed
in Section 2.2. In this example, there is a binary outcome variable,
$y\in\\{0,1\\}$ and one covariate variable, $z$, with 8 data points.
Specifically, $y_{i}=1$ at $z=10,20,30,40,$ and $y_{i}=0$ at $z=60,70,80,90$.
Since $y$ could be completely separable by $z$, we observed the complete
separation in this example.

Quasi-complete separation: This example is an extension of the complete
separation example in Agresti [2013] with two points added, $y_{i}=1$ and
$y_{i}=0$ at $z=50$. This is an example of quasi-complete separation.

Quadratic logistic regression model: This example comes from Section 2.2 of
Geyer [2009]. There is one binary outcome variable $y\in\\{0,1\\}$ and one
covariate variable $z$ which takes integer values from 1 to 30. The outcome
variable was $y_{i}=1$ when $12<z_{i}<24$ and $y_{i}=0$ otherwise. A quadratic
logistic model is considered in this example and complete separation is
observed.

Endometrial Cancer Study: This example comes from Heinze and Schemper [2002].
The main purpose of this study was to describe histology of cases (HG) in
terms of three risk factors: neovasculation (NV), endometrium height (EH) and
pulsatility index of arteria uterina (PI). The outcome variable had 30
patients classified grading 0-II for histology (HG = 1) and 49 patients for
grading III-IV (HG = 0). There were 13 patients who had neovasculization (NV =
1) and absent for 66 patients (NV = 0). Pulsatility index (PI) ranges from 0
to 49 with mean of 17.38 and median of 16.00, and endometirum height (EH)
ranges from 0.27 to 3.61 with mean of 1.662 and median of 1.640. Quasi-
complete separation was observed in this example, this separation is
determined by NV.

Maize data: This example comes from Romay et al. [2013], and it consists of
2,815 maize lines. The binary outcome variable is the kernel color, where 1
indicated non-white kernel color and 0 indicated white kernel color. We fitted
a logistic regression model with kernel color as the outcome variable and
covariate variables consisting of subpopulations and 24 DNA markers
surrounding the psy1 gene [Palaisa et al., 2004; Chandler et al., 2013; Ford,
2000]. The psy1 gene has been shown to be each marker had a value from 0 to 1,
where 0 and 1 indicated homozygous genotypes and a value in (0,1) indicated
that missing genotypic data were imputed. In the final data set, 309 lines had
a white kernel and 1,238 had non-white kernel color. These maize lines were
subdivided into six subpopulations, namely 115 non-stiff stalk, 54 popcorn,
120 stiff stalk, 116 sweet corn, 159 tropical, and 983 unclassified. In this
example, there was no separation issues when we used a single marker for
covariate. However, we had a separation issue for saturated model with 24 DNA
markers and subpopulations.

###  2.8 Software used

We implemented our methodology in R package glmdr. We used R version 3.6.1 and
the required R packages for glmdr is nloptr version 1.2.2.2. To compare its
performance, we considered arm version 1.11-1, brglm2 0.7.0, and stats version
3.6.1. To determine the optimal cut-off for the logistic regression, we used
PresenceAbsence version 1.1.9. For visualization, data wrangling and
experiments, we used ggplot2 version 3.3.3, gridExtra version 2.3, latex2exp
version 0.4.0, foreach version 1.4.7, doParallel version 1.0.15, and tidyverse
version 1.2.1. Further details are included in the technical reports. glmdr is
available on https://github.com/DEck13/complete_separation.

##  3 Results

###  3.1 Inference

Inferences are for mean-value parameters $\pi(x)$ for observed values of $x$.
We report the in-sample accuracy for all observations and confidence intervals
for problematic observations that raise the (quasi) complete separation issue
to compare each method. For confidence intervals, we compute the average
length of one-sided confidence interval for glmdr and average length of Wilson
intervals for bayesglm, brglm, and linear models. In Table 1, we can see all
methods show the equivalent in-sample accuracy for the complete separation and
quasi-complete separation examples. Meanwhile, the logistic models, glmdr,
bayesglm, and brglm, display the higher in-sample accuracy for quadratic,
endometrial, and maize examples in comparison to the linear model. Within
these examples, glmdr has the highest in-sample accuracy in maize example than
other two logistic models. For confidence intervals, glmdr demonstrates the
smallest length in all examples. Especially, in quadratic and endometrial
examples, its lengths of confidence intervals are significantly smaller than
other methods. Two logistic models, bayesglm and brglm generally shows smaller
lengths of confidence intervals but they are not highly different from that of
linear model in all examples. This result suggests that linear model perform
worse than logistic model remedies, and glmdr, which solves the complete
separation within the MLE framework, produces the most accurate inferences.

Table 1: Model performances for all examples. Estimation accuracy and length
of confidence intervals are displayed. glmdr denotes Generalized Linear Model
Done Right [Geyer et al., 2021], bayesglm denotes Generalized Linear Model
with Student-t prior distribution [Gelman et al., 2008], brglm denotes Bias
Reduction in Generalized Linear Models [Kosmidis and Firth, 2009], and linear
denotes the multiple linear model using ordinary least squares.

|  | Complete Separation | Quasi Separation | Quadratic | Endometrial | Maize
---|---|---|---|---|---|---
accuracy | glmdr | 100 % | 90 % | 100 % | 88.61 % | 87.14 %
| bayesglm | 100 % | 90 % | 100 % | 88.61 % | 87.07 %
| brglm | 100 % | 90 % | 100 % | 88.61 % | 87.01 %
| linear | 100 % | 90 % | 90 % | 86.08 % | 86.81 %
length | glmdr | 0.550 | 0.308 | 0.199 | 0.194 | 0.563
| bayesglm | 0.828 | 0.827 | 0.823 | 0.804 | 0.814
| brglm | 0.835 | 0.831 | 0.811 | 0.808 | 0.826
| linear | 0.829 | 0.829 | 0.859 | 0.806 | 0.838

###  3.2 Prediction

To compare the performance of prediction, we compare out-of-sample accuracy,
prediction intervals, and computational cost. In Table 2, we can see all
methods show comparable accuracy for the complete separation and quasi-
complete separation examples. glmdr shows the highest out-of-sample accuracy
in endometrial example where other three methods perform the same. In the
quadratic example, brglm performs the best followed by other two logistic
models and linear model, but linear model is better than the logistic models
in maize example although their differences are not large. This result is
surprising because the linear model is generally not recommended for binary
classification, yet it shows a better performance than the logistic models.
For prediction intervals, overall there is no significant difference between
each method. We notice that glmdr has the smallest lengths of prediction
intervals in three examples. However, bayesglm produces the smallest interval
for the quasi-complete separation, and the linear model produces the smallest
interval for the endometrial example.

Table 2: Prediction results. length of prediction intervals, and computational
cost for all examples. glmdr denotes Generalized Linear Model Done Right
[Geyer et al., 2021], bayesglm denotes Generalized Linear Model with Student-t
prior distribution [Gelman et al., 2008], brglm denotes Bias Reduction in
Generalized Linear Models [Kosmidis and Firth, 2009], and linear denotes the
multiple linear model using ordinary least squares.

|  | Complete Separation | Quasi Separation | Quadratic | Endometrial | Maize
---|---|---|---|---|---|---
accuracy | glmdr | 100 % | 80 % | 93.33 % | 87.34 % | 86.23 %
| bayesglm | 100 % | 80 % | 93.33 % | 86.08 % | 86.36 %
| brglm | 100 % | 80 % | 100 % | 86.08 % | 86.36 %
| linear | 100 % | 80 % | 90 % | 86.08 % | 86.55 %
length | glmdr | 0.822 | 0.859 | 0.807 | 0.848 | 0.837
| bayesglm | 0.839 | 0.845 | 0.828 | 0.843 | 0.837
| brglm | 0.843 | 0.847 | 0.813 | 0.844 | 0.837
| linear | 0.833 | 0.844 | 0.861 | 0.851 | 0.839
cost | glmdr | 0.13 secs | 0.27 secs | 0.31 secs | 1.06 secs | 3.70 mins
| bayesglm | 0.11 secs | 0.12 secs | 0.35 secs | 0.31 secs | 45.35 secs
| brglm | 0.19 secs | 0.19 secs | 0.44 secs | 0.49 secs | 3.80 mins
| linear | 0.07 secs | 0.06 secs | 0.09 secs | 0.14 secs | 4.63 secs

We present the computational cost of each method in Table 2. In all examples,
linear model is much faster than logistic model remedies. Although there is no
significant difference in complete separation, quasi-complete separation,
quadratic, and endometrial examples, computational cost of glmdr increases in
maize example. This increase is due to the execution time for glmdr to solve
its internal optimization problem when the data point to be predicted yields
separation. Similarly, brglm is as slow as glmdr because it needs to handle
the optimization problem to find the penalized MLE for each iteration.
However, bayesglm does not suffer this issue because it does not carry the
computation for the optimization problem in their method.

Considering all aspects, all of four methods demonstrate comparable out-of-
sample accuracy and length of prediction intervals. However, there are several
notable differences. glmdr provides the smallest lengths of prediction
intervals except in the quasi-complete separation example. It also shows
better performance in the endometrial example. But, it may not be scalable to
the large datasets due to relatively high computational cost. bayesglm
performs well on all examples with the lowest computational cost, which
indicates the bayesglm is suitable for prediction on large data. brglm
achieves the highest out-of-sample accuracy in the quadratic example, but
brglm may not be suitable for big datasets because of relatively expensive
computational cost. Meanwhile, the linear model performs well in spite of
binary outcome. It shows comparable out-of-sample accuracy with small
prediction intervals and the lowest computational cost.

##  4 Discussion

In this work, we investigate the potential of glmdr for inference and
prediction through a comparison of its performance relative to several
competing approaches. We found that glmdr yielded statistically optimal
inferences and provided predictions that were comparable to other approaches.
Overall these results are promising, and suggest that future investigation
into applying glmdr to quantitative genetics analyses is warranted.

###  4.1 Statistical benefits and implications of using glmdr

The logistic model is one of the most common statistical models for analysis
of binary traits. Although the classic linear model is an attractive option to
use because of its popularity, binary outcome variables violate linear
regression modeling assumptions such as homoscedasticity and linearity (i.e.
Gauss-Markov assumptions) as well as normality. Therefore, even though results
from Section 3.1 and 3.2 show that the performance of linear model is
comparable to the logistic models, we cannot fully utilize asymptotic
properties of linear model and make a proper inference such as significance
tests for coefficients. Furthermore, linear regression modeling can yield
nonsensical estimates or predictions for a conditional success probability. In
some settings confidence intervals can be strictly outside of the $[0,1]$
interval. Such a setting is equivalent to stating that uncertainty exists
(i.e., the confidence can take on a range of values) while simultaneously
stating that there is no uncertainty (i.e., all values of the confidence
interval outside of $[0,1]$ are forced to take on values of exactly $0$ or $1$
due to the basic definition of a probability).

Within the binary outcome modeling framework we found that glmdr to be the
most preferable technique for accounting for data separation. This is based on
its overall performance across inference and prediction. A strength of glmdr
is that it provides the sharpest inferences and the most natural remedy for
data separation within the logistic regression modeling framework. Meanwhile,
other methods solve the separation problem by switching the modeling paradigm.
For example, bayesglm adopts a Bayesian approach which scales the data first
and then places Cauchy distribution as a prior distribution on the
coefficients, and brglm modifies the score function to produce finite
coefficients. As a result, not only are both models’ results in inference not
the best, but it is also hard to see their outputs as a natural remedy for
data separation. Our novel prediction method for glmdr is comparable with
bayesglm and brglm in terms of LOOCV prediction error. However, it may be
expensive with a large number of observations and a large number of
predictors.

###  4.2 Implications of these results for quantitative genetics

One of the most notable results was that the linear regression model performed
comparably to the other models for both inference and prediction. Thus, it
might be tempting to conclude that a linear regression model is sufficient for
the purposes of inference and prediction when applied to the quantitative
genetics analysis of binary traits. However, the use of a linear regression
model on a binary trait is fundamentally flawed for many reasons [Agresti,
2013]. As previously alluded to, the most apparent is that it is possible to
obtain predicted values of $\pi(x)=\text{Pr}(Y=1|X=x)>1$ or $\pi(x)<0$. Such a
result contradicts the basic definition of a probability, and thus renders the
ensuing statistical analysis useless.

We have clearly demonstrated the usefulness of glmdr for inferences, in that
the benefits suggested by statistical theory translates into optimal precision
of estimates (Table 1). However, one important shift that will need to occur
for quantitative genetics analyses like genome-wide association studies (GWAS)
is that inferences will need to be made in terms of one-sided confidence
intervals for $\pi(x)$. The use of such confidence intervals will need to
replace the traditional practice of using coefficient estimates in the model
to quantify additive, dominance, and epistatic genetic signals. In the
presence of complete and quasi-complete separation, slope estimates will be
‘at infinity’ and will therefore not be of practical use. However, one can
still perform likelihood ratio tests on nested models under such circumstances
to infer the significance of genetic signals included in the model as
explanatory variables. This nested model hypothesis testing strategy can work
even when separation exists in the null model [Geyer, 2009, Section 3.15].
Therefore, the presence of additive, dominant, and epistatic genetic effects
can still be inferred without looking at coefficient estimates.

To make glmdr amenable for use in GWAS, future research should investigate the
incorporation of random effects so that it can be used in diversity panels
where subpopulation structure and differential familial relatedness are
present. Essentially, this would lead to a model analogous to the unified
mixed linear model [Yu et al., 2006], which will account for false positive
marker-trait in the presence of these two sources of variability. The glmdr
method can potentially be used to make predictions when data separation exists
within the fixed effects of a mixed effects model. The main challenge is in
the development of a fast likelihood maximizing routine which can compute the
optimizations in displayed equation 6 in Section 4.3 of Eck and Geyer [2021].
The main difficulty is that the data separation problem appears as the
integrand in the integration of the random effects terms. This difficulty is
substantially mitigated in the mixed effects model when data separation
appears in a binary predictor since this integrand is monotonic in $\beta$.

One of the most exciting developments in this study was the development of a
novel method that uses glmdr to make predictions. We used this method to show
that glmdr is at least as competitive as other approaches for prediction. To
expand upon the findings of this study and assess the potential of glmdr for
genomic prediction, we will need to to explore the issues of separation for
$p\gg n$ situations, as such a situation will be ubiquitous for genomic
prediction in the foreseeable future. The next logical step for investigating
glmdr in the $p\gg n$ setting would be to use a penalized likelihood based
approach to search for a logistic regression model with $k<n<p$ relevant terms
where separation is still present. There are two ways forward within this
avenue. One is to use a model selection criteria that penalizes model
complexity (AIC, BIC, AICc, [Konishi and Kitagawa, 2008]). The biggest
challenge in this front is the development of a fast searching algorithm that
preserves the “correct” $k$ variables. The other way is to use penalties on
the size of $\beta$, such as LASSO [Tibshirani, 1996] and ridge regression
[Hoerl and Kennard, 1970], to screen for slope estimates that are ’at
infinity’.

###  4.3 Conclusions

The ever-increasing amount of available high-throughput genotypic and
phenotypic crop data is making it possible to both study and utilize genotype-
to-phenotype relationships. To ensure that important biological conclusions
are based on statistically rigorous analyses, it is critical that the most
appropriate methods are used. Because we demonstrated that one of the most
appropriate methods, maximum likelihood estimation for handling separation in
binary trait data [Geyer, 2009; Eck and Geyer, 2021] is capable of adequate
inference and prediction, we conclude that glmdr [Geyer et al., 2021] is
appropriate for quantitative genetics analyses. However, further research
should be conducted to further refine glmdr to handle the most challenging
issues typically observed in genotype-to-phenotype analyses.

## Acknowledgements

We are grateful to Ioannis Kosmidis for feedback on the brglm method and
helpful comments, and David Firth for helpful comments.

## References

  * Agresti [2013] A. Agresti.  _Categorical data analysis_.  Wiley series in probability and statistics. Wiley, 3rd ed edition, 2013\.  ISBN 9780470463635.
  * Albert and Anderson [1984] A. Albert and J. A. Anderson.  On the existence of maximum likelihood estimates in logistic regression models.  _Biometrika_ , 71(1):1–10, 04 1984.  ISSN 0006-3444.  doi: 10.1093/biomet/71.1.1.  URL https://doi.org/10.1093/biomet/71.1.1.
  * Barndorff-Nielsen [1978] O. Barndorff-Nielsen.  _Information and exponential families: in statistical theory_.  J. Wiley & Sons, 1978.
  * Brown et al. [2001] L. D. Brown, T. T. Cai, and A. DasGuptal.  Interval estimation for a binomial proportion.  _Statistical Science_ , 16(2):101 – 133, 2001\.  doi: 10.1214/ss/1009213286.  URL https://doi.org/10.1214/ss/1009213286.
  * Burnham and Anderson [2002] K. P. Burnham and D. R. Anderson.  _Model selection and multimodel inference - 2nd ed.: a practical information-theoretic approach_.  Springer-verlag new york Inc., 2002.
  * Chandler et al. [2013] K. Chandler, A. E. Lipka, B. F. Owens, H. Li, E. S. Buckler, T. Rocheford, and M. A. Gore.  Genetic analysis of visually scored orange kernel color in maize.  _Crop Science_ , 53(1):189–200, 2013.
  * Dunson et al. [2006] D. B. Dunson, A. H. Herring, and S. A. M. Engel.  Bayesian selection and clustering of polymorphisms in functionally-related genes.  _J. AM. STATIST. ASSOC_ , 2006.
  * Eck and Geyer [2021] D. J. Eck and C. J. Geyer.  Computationally efficient likelihood inference in exponential families when the maximum likelihood estimator does not exist.  _Electronic Journal of Statistics_ , 15(1), 2021.  doi: 10.1214/21-ejs1815.
  * Ford [2000] R. H. Ford.  Inheritance of kernel color in corn: explanations & investigations.  _The American Biology Teacher_ , 62(3):181–188, 2000.
  * Freeman and Moisen [2008] E. A. Freeman and G. G. Moisen.  A comparison of the performance of threshold criteria for binary classification in terms of predicted prevalence and kappa.  _Ecological Modelling_ , 217(1):48–58, 2008.  ISSN 0304-3800.  doi: https://doi.org/10.1016/j.ecolmodel.2008.05.015.  URL https://www.sciencedirect.com/science/article/pii/S0304380008002275.
  * Gelman et al. [2008] A. Gelman, A. Jakulin, M. G. Pittau, and Y.-S. Su.  A weakly informative default prior distributionfor logistic and other regression models.  _The Annals of Applied Statistics_ , 2(4):1360 – 1383, 2008.  doi: 10.1214/08-AOAS191.  URL https://doi.org/10.1214/08-AOAS191.
  * Genkin et al. [2007] A. Genkin, D. D. Lewis, and D. Madigan.  Large-scale bayesian logistic regression for text categorization.  _Technometrics_ , 49(3):291–304, 2007.  doi: 10.1198/004017007000000245.  URL https://doi.org/10.1198/004017007000000245.
  * Geyer [2009] C. J. Geyer.  Likelihood inference in exponential families and directions of recession.  _Electronic Journal of Statistics_ , 3:259–289, 2009.  doi: 10.1214/08-ejs349.
  * Geyer et al. [2021] C. J. Geyer, D. J. Eck, and S. Park.  _glmdr: Exponential Family Generalized Linear Models Done Right_ , 2021.  URL https://github.com/DEck13/complete_separation.  R package version 0.3.
  * Harjes et al. [2008] C. E. Harjes, T. R. Rocheford, L. Bai, T. P. Brutnell, C. B. Kandianis, S. G. Sowinski, A. E. Stapleton, R. Vallabhaneni, M. Williams, E. T. Wurtzel, et al.  Natural genetic variation in lycopene epsilon cyclase tapped for maize biofortification.  _Science_ , 319(5861):330–333, 2008.
  * Heinze and Schemper [2002] G. Heinze and M. Schemper.  A solution to the problem of separation in logistic regression.  _Statistics in Medicine_ , 21(16):2409–2419, 2002.  doi: 10.1002/sim.1047.
  * Hoerl and Kennard [1970] A. E. Hoerl and R. W. Kennard.  Ridge regression: Biased estimation for nonorthogonal problems.  _Technometrics_ , 12(1):55–67, 1970.
  * Jarquín et al. [2014] D. Jarquín, J. Crossa, X. Lacaze, P. Du Cheyron, J. Daucourt, J. Lorgeou, F. Piraux, L. Guerreiro, P. Pérez, M. Calus, et al.  A reaction norm model for genomic selection using high-dimensional genomic and environmental data.  _Theoretical and applied genetics_ , 127(3):595–607, 2014.
  * Konishi and Kitagawa [2008] S. Konishi and G. Kitagawa.  _Information criteria and statistical modeling_.  Springer Science & Business Media, 2008.
  * Kosmidis and Firth [2009] I. Kosmidis and D. Firth.  Bias reduction in exponential family nonlinear models.  _Biometrika_ , 96(4):793–804, 2009.  doi: 10.1093/biomet/asp055.
  * Lesaffre and Albert [1989] E. Lesaffre and A. Albert.  Partial separation in logistic discrimination.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 51(1):109–116, 1989.  doi: 10.1111/j.2517-6161.1989.tb01752.x.
  * Nelder and Wedderburn [1972] J. A. Nelder and R. W. M. Wedderburn.  Generalized linear models.  _Journal of the Royal Statistical Society. Series A (General)_ , 135(3):370–384, 1972.  ISSN 00359238.  URL http://www.jstor.org/stable/2344614.
  * Palaisa et al. [2004] K. Palaisa, M. Morgante, S. Tingey, and A. Rafalski.  Long-range patterns of diversity and linkage disequilibrium surrounding the maize y1 gene are indicative of an asymmetric selective sweep.  _Proceedings of the National Academy of Sciences_ , 101(26):9885–9890, 2004.
  * R Core Team [2020] R Core Team.  _R: A Language and Environment for Statistical Computing_.  R Foundation for Statistical Computing, Vienna, Austria, 2020.  URL https://www.R-project.org/.
  * Rice et al. [2020] B. R. Rice, S. B. Fernandes, and A. E. Lipka.  Multi-trait genome-wide association studies reveal loci associated with maize inflorescence and leaf architecture.  _Plant and Cell Physiology_ , 61(8):1427–1437, 2020.
  * Romay et al. [2013] M. C. Romay, M. J. Millard, J. C. Glaubitz, J. A. Peiffer, K. L. Swarts, T. M. Casstevens, R. J. Elshire, C. B. Acharya, S. E. Mitchell, S. A. Flint-Garcia, M. D. McMullen, J. B. Holland, E. S. Buckler, and C. A. Gardner.  Comprehensive genotyping of the USA national maize inbred seed bank.  _Genome Biology_ , 14(6):R55, 2013.  ISSN 1474-760X.  doi: 10.1186/gb-2013-14-6-r55.  URL https://doi.org/10.1186/gb-2013-14-6-r55.
  * SAS Institute Inc. [2003] SAS Institute Inc.  _SAS/STAT Software, Version 9.1_.  Cary, NC, 2003.  URL http://www.sas.com/.
  * Shenstone et al. [2018] E. Shenstone, J. Cooper, B. Rice, M. Bohn, T. M. Jamann, and A. E. Lipka.  An assessment of the performance of the logistic mixed model for analyzing binary traits in maize and sorghum diversity panels.  _PloS one_ , 13(11):e0207752, 2018.
  * Sugiura [1978] N. Sugiura.  Further analysts of the data by akaike’ s information criterion and the finite corrections.  _Communications in Statistics - Theory and Methods_ , 7(1):13–26, 1978.  doi: 10.1080/03610927808827599.
  * Tibshirani [1996] R. Tibshirani.  Regression shrinkage and selection via the lasso.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 58(1):267–288, 1996.
  * Van Rossum and Drake Jr [1995] G. Van Rossum and F. L. Drake Jr.  _Python tutorial_.  Centrum voor Wiskunde en Informatica Amsterdam, The Netherlands, 1995\.
  * Wilson [1927] E. B. Wilson.  Probable inference, the law of succession, and statistical inference.  _Journal of the American Statistical Association_ , 22(158):209–212, 1927.  doi: 10.1080/01621459.1927.10502953.
  * Yu et al. [2006] J. Yu, G. Pressoir, W. H. Briggs, I. V. Bi, M. Yamasaki, J. F. Doebley, M. D. McMullen, B. S. Gaut, D. M. Nielsen, J. B. Holland, et al.  A unified mixed-model method for association mapping that accounts for multiple levels of relatedness.  _Nature genetics_ , 38(2):203–208, 2006.
  * Zorn [2005] C. Zorn.  A solution to separation in binary response models.  _Political Analysis_ , 13(2):157–170, 2005.  doi: 10.1093/pan/mpi009.


[affiliation=*1,3]JakubHoscilowicz [affiliation=*1]AdamWiacek
[affiliation=1,2]JanChojnacki [affiliation=1]AdamCieslak
[affiliation=1]LeszekMichon []VitaliiUrbanevych [affiliation=3]ArturJanicki

# NL-ITI: Optimizing Probing and Intervention for Improvement of ITI Method

###### Abstract

Large Language Models (LLM) are prone to returning false information. It
constitutes one of major challenges in the AI field. In our work, we explore
paradigm introduced by Inference-Time-Intervention (ITI). In first stage, it
identifies attention heads, which contain the highest amount of desired type
of knowledge (e.g., truthful). Afterwards, during inference, LLM activations
are shifted for chosen subset of attention heads. We further improved the ITI
framework by introducing a nonlinear probing and multi-token intervention -
Non-Linear ITI (NL-ITI). NL-ITI is tested on diverse multiple-choice
benchmarks, including TruthfulQA, on which we report around 14% MC1 metric
improvement with respect to the baseline ITI results. NL-ITI achieves also
encouraging results on other testsets - on Business Ethics subdomain of MMLU,
around 18% MC1 improvement over baseline LLaMA2-7B. Additionally, NL-ITI
performs better while being less invasive in the behavior of LLM at the same
time (as measured by Kullback-Leibler divergence). Code is available at
https://github.com/Samsung/NL-ITI.

##  1 Introduction

Large Language Models (LLMs) are a major achievement in the domain of
artificial intelligence, particularly within natural language processing
(NLP). Their capabilities span a wide array of applications, from generating
human-like texts to understanding and processing complex language structures.
However, the probabilistic nature of these models often gives rise to certain
challenges, including the phenomena of hallucinations [1, 2] and the
generation of toxic or counterfactual content [3]. These issues underscore the
motivation behind the development of robust benchmarks and methodologies aimed
at evaluating and enhancing the safety, fairness, and truthfulness of LLM
responses.

LLM models trained on extensive datasets, inadvertently absorb and repeat
cultural, gender-based, racial, or ideological biases in their training
dataset. The unintentional propagation of such biases by LLMs in their output
not only raises ethical concerns but also underscores the potential for these
technologies to reinforce existing societal disparities and prejudices [4, 5].
Addressing bias in LLMs is essential for developing ethical AI technologies.
It requires a comprehensive strategy that includes diversifying training
datasets, developing algorithms to detect and neutralize bias, and
implementing robust testing protocols for biased outputs. Some works already
present us with a solutions how to assess and mitigate model's bias [6, 7, 8,
9, 10].

Representation editing approaches like ITI [7] modify the model's internal
representations in order to influence its behavior. ITI paradigm involves a
two-step process, where attention heads are first evaluated for their probing
accuracy, and then Mass Mean Shift vectors are applied to a subset of top-
performing heads during inference. These vectors, calculated as the mean
difference between activations for true and false answers, are pre-computed
and attached to the model, enabling a truthfulness improvement with negligible
computational overhead. In this subfield, authors of [11] introduced recently
the Truth Forest (TrFr) that employs multi-dimensional orthogonal probes and
achieves better performance than ITI.

Benchmarks such as TruthfulQA [10] have been introduced to evaluate model
truthfulness across various domains, including health, law, finance, and
politics. These benchmarks challenge models with questions designed to elicit
imitative falsehoods, thereby testing the model's ability to maintain
truthfulness across topics. Similarly, datasets like BBQ [9] and ToxiGen [8]
assess LLM fairness and ability to handle nuanced manifestations of hate
speech. Moreover, recent works have sparked an interest in LLM personality
categorization and psychometrics. Contributions in this area include adjusting
LLM personality traits [12], using the Myers-Briggs Type Indicator (MBTI) for
evaluation [13], and simulating diverse personalities in models [14].

Our main contribution is a notable enhancement of the Inference Time
Intervention method [7], that leads to significantly higher generalization
capabilities. The improvement encompasses two distinct aspects: firstly, we
increased capacity of probing model which results in a more effective
identification of attention heads that contain desired type of knowledge
(e.g., truthful). Secondly, we expanded token context of intervention vector
that is used to steer attention heads. This enhancement is derived from the
hypothesis that truthful knowledge is not solely concentrated in the vector
corresponding to the final token, but is distributed across a broader context.

##  2 Method

As reported in [7], LLMs seem to preserve an internal representation of
abstract concepts such as truth and honesty, even though often they do not
generate factual responses. Simple prompt stimulation may not be enough to
access the full potential stored in the internal representations. In [7],
authors use labeled data (i.e. Q+A pairs from split of TruthfulQA) to train a
linear probing model, which identifies attention heads storing the truthful
representations. For each such head, the truthful direction is calculated.
During the inference, attention head activations are shifted in the truthful
direction.

ITI encompasses two phases. Initially, a linear probing model is trained on
representations returned by the attention head for a given probing trainset.
The assumption is that the higher the accuracy of the probing model, the
higher the amount of desired knowledge (e.g., truthful). Mathematically, the
probing operation $p_{\theta}$ might be described as:

$p_{\theta}\left(x_{l}^{h}\right)=\textrm{sigmoid}\left(\left\langle\theta,x_{l}^{h}\right\rangle\right),$
(1)

where $\theta$ is a set of trainable parameters, and $x^{h}_{l}$ is an
activation of token $x$ at head $h$ and layer $l$. In the process of probing
training, $N$ question-answers pairs are concatenated and the activations that
correspond to the last token $\left\\{(x^{h}_{l},y)_{i}\right\\}^{N}_{i=1}$
are collected (where $y$ is a binary label that indicates true of false
answer).

Subsequently, Mass Mean Shift vectors representing the average difference
between activations for truthful versus untruthful responses, are applied to a
select group of top-performing attention heads during inference. These
vectors, computed once and affixed to the model during inference, introduce
negligible additional computation time. The intervention can be described as:

$x_{l+1}=x_{l}+\sum^{H}_{h=1}Q^{h}_{l}\left(\textrm{Att}^{h}_{l}\left(P^{h}_{l}x_{l}\right)+\alpha\sigma^{h}_{l}\theta^{h}_{l}\right),$
(2)

where $x_{l}\in\mathbb{R}^{DH}$ is $l$-th token high-dimensional embedding,
$P^{h}_{l}\in\mathbb{R}^{D\times DH}$ is a mapping operator from token
embedding to $D$-dimensional attention head space, and $\textrm{Att}^{h}_{l}$
operator connects information from other tokens and gives us activations seen
in Equation 1: $x^{h}_{l}=\textrm{Att}^{h}_{l}\left(P^{h}_{l}x_{l}\right)$.
Finally, the last term in Equation 2 is the intervention term. It can be
understood as follows: when calculating the next-token-prediction $x_{l+1}$
the residual stream (previous $x_{l}$ token and the weighted sum of
activations) is modified by adding the biasing direction $\theta^{h}_{l}$
multiplied by its standard deviation (with respect to all of the Q+A pairs)
and the intervention strength $\alpha$. The biasing directions are appended
only if the probing accuracy is satisfactory for given attention head. To
control this aspect, a second hyperparameter was introduced: the number of top
heads included in the intervention procedure $K$.

To calculate the biasing direction for each head and layer, one takes an
average of activations of the last tokens produced with the Q+A dataset:

$\theta^{h}_{l}=\frac{1}{N}\sum^{N}_{i=1}\left(-1\right)^{\sigma}\left(x^{h}_{l}\right)_{i},\quad
h\in\textrm{Top Heads}.$ (3)

The parameter $\sigma$ takes values from $\\{0,1\\}$ for respectively true or
false Q+A pair. This is equivalent of taking the difference between the
truthful and untruthful vector representations.

###  2.1 Proposed improvement

The probing procedure is necessary to identify attention heads, which contain
the highest amount of given concept. Probes need to be trained on a labeled
dataset reflecting the concept (throughout this work we used split of
TruthfulQA [10]). ITI [7] used probing based on a logistic regression
(Equation 1), which may not fully capture the complexity of the concept
representation in the activation space. Consequently, we used simple, non-
linear Multi-Layer Perceptron (MLP) for this task, changing Equation 1 to:

$p_{\theta}(x^{h}_{l})=\textrm{MLP}\left(\left\langle\theta,x_{l}^{h}\right\rangle\right).$
(4)

We kept the sigmoid activation at the output and used ReLU for other layers.
Our assumption was that increasing capacity of probing model leads to more
appropriate choice of the top heads. As noted by [15, 16, 17], the higher
capacity of probing model, the better estimation of lower bound of amount of
given information (e.g., truthful) in attention head. The top heads are then
used in the ITI procedure described in Equation 2.

Moreover, instead of using only the vector corresponding to the last token for
the training of probing model, we took the average of vectors that corresponds
to last tokens. Consequently, more information encoded in multiple tokens is
provided as input to the MLP. This modification can be mathematically
expressed as a change in which one collects the training dataset:

$\left\\{\left(x^{h}_{l},y\right)_{i}\right\\}^{N}_{i=1}\xrightarrow{\textrm{multi-
token}}\left\\{\left(\left\langle
x^{h}_{l}\right\rangle_{\tau},y\right)_{i}\right\\}^{N}_{i=1},$ (5)

where instead of taking the vector $x^{h}_{l}$ returned by an attention head
that corresponds to the last token from LLM input, we take an average of the
optimal number $\tau$ last tokens. This optimum is found experimentally and
described in detail in Section 3.

Similarly, extending the number of tokens used in intervention is also
relevant. In particular, during the inference, to each attention head, we add
biasing vector that corresponds to the mean of the last $\rho$ tokens in a Q+A
pair. To calculate the biasing direction vector, this is averaged over all Q+A
pairs as in Equation 3:

$\theta^{h}_{l}=\frac{1}{N}\sum^{N}_{i=1}\left(-1\right)^{\sigma}\left(\left\langle
x^{h}_{l}\right\rangle_{\rho}\right)_{i},\quad h\in\textrm{Top Heads}.$ (6)

As before, the exact optimal number of tokens $\rho$, over which the
activations are averaged, needs to be found empirically. From now on, we will
refer to those optimizations as Non-Linear-Inference Time Intervention (NL-
ITI).

##  3 Experiments

Figure 1:  How MC1 correlates with KL divergence. KL estimates strength of
change of LLM behavior due to representation editing - the lower, the better.
The results were collected for ITI and NL-ITI using different hyperparameter
sets ($\alpha$, $K$) for TruthfulQA and OpenBookQA datasets. On each
benchmark, baseline LLaMA-2-7B performance is shown.

###  3.1 Evaluation

In [7] the authors report MC1 and MC2 scores introduced in TruthfulQA [10]
benchmark. We also focus on MC1 and MC2, as they are reliable and
reproducible. MC scores do not depend on the underlying judging model
reasoning capabilities (e.g., GPT-4) - instead "deterministic" labeled
multiple choice QA data is used for the calculation of LLM performance.
Moreover in [7], Cross Entropy (CE) and Kullback-Leibler divergence (KL) are
calculated to see how much the modified LLM diverges from the original
LLaMA-2-7B [18] tokens distribution. For both metrics, lower values correspond
to less change in the model's behavior. Large CE & KL values may suggest that
the generalization capabilities of the LLM are impaired (intervention skews
LLM too much). However, it is difficult to estimate at which values these
metrics point to the generalization collapse. For example, in Table 3.5 NL-ITI
outperforms ITI on ARC [19], MMLU [20], and OpenBook [21] benchmarks despite
slightly higher CE.

Additionally, on Figure 1 we provide two plots with function MC1 of KL on two
datasets: TruthfulQA and OpenBookQA. Each point on the plot corresponds to a
different hyperparameter set ($\alpha$, $K$). KL is calculated with opendomain
OpenWebText [22] corpus. It is visible that the trade-off between
representation editing performance and KL occurs. NL-ITI creates a peak in
$\textrm{KL}\sim 0.5$, the correlation is also non-linear around this region.
We also report that NL-ITI gives higher MC1 than ITI for every value of KL.

###  3.2 Evaluation Metrics Details

MC scores metrics were introduced in [10] and are widely used in the field of
representation editing. Though, as far as we know, there is no detailed
description of how those metrics are calculated. Consequently, we briefly
describe construction of the MC1 and MC2 scores, as they are defined in
publicly available code of TruthfulQA dataset [10] and ITI [7].

For a given Question $Q_{i}$ in the dataset we create 4-5 prompts:
$\\{Q_{i}+A_{1},Q_{i}+A_{2},Q_{i}+A_{3},Q_{i}+A_{4}\\}$, where only one is the
correct answer. Then we use this pair as an input to the transformer-based
[23] LLM, which outputs a list of (log-) distributions of transition
probabilities between adjacent tokens $\log P(T_{k}$$\,\to\,$$T_{k+1})$. The
shape of such output is (the length of the tokenized prompt, the size of the
token dictionary). This gives us insight into how the model would predict
subsequent tokens if we ask it the question $Q_{i}$. In particular, we can
check what are the probabilities of getting one the configurations of interest
$A_{j}$ \- we just need to find the successive id of the answer tokens in the
(log-)distribution $\log P(T_{k}$$\,\to\,$$T_{k+1})^{i,j}$ coming out of the
language model. The question now is, which is the most likely answer? We can
compare the probabilities of occurrence of a sequence of tokens corresponding
to each question. Assuming that the draw of successive tokens is independent,
if there were simply $P(T_{k}$$\,\to\,$$T_{k+1})^{i,j}$ probabilities coming
out of the language model, we would have to multiply them sequentially. As
$\log P(T_{k}$$\,\to\,$$T_{k+1})^{i,j}$ probabilities come out, we have to add
them. These sums are colloquially referred to as true score for factually true
statement $Q_{i}+A_{j}$ and conversely false score for a false statement.
Mathematically the true/false score corresponding to $Q_{i}+A_{j}$ pair can be
written:

$\displaystyle S_{t/f}^{i,j}=\sum_{k}\log P(T_{k}\,\to\,T_{k+1})^{i,j}_{t/f},$
(7)

where the token configuration $T_{k}$ corresponds respectively to true/false
$Q_{i}+A_{j}$ pair.

With this knowledge we can define MC1 score is the accuracy over all questions
$Q_{i}$. For question $Q_{i}$, label 1 is given if $S^{i,j}_{t}>S^{i,j}_{f}$,
for each answer $A_{j}$, otherwise the label is 0.

MC2 score is applicable for question sets with multiple true answers. For each
question $Q_{i}$, it can be written using probabilities of true/false
$Q_{i}+A_{j}$ pair occurring: $P_{t/f}^{i,j}=\exp{(S_{t/f}^{i,j})}$ giving:

$\displaystyle\textrm{MC2}^{i}=\frac{\sum_{j}P_{t}^{i,j}}{\sum_{j}P_{t}^{i,j}+\sum_{j}P_{f}^{i,j}}.$
(8)

We will report on the averaged MC2 scores over all questions $Q_{i}$:
$\textrm{MC2}=\langle\textrm{MC2}^{i}\rangle$.

###  3.3 Datasets

We used TruthfulQA for probing, intervention and evaluation, ensuring that
data used for probing and intervention do not overlap with test set. Similarly
to ITI, we splitted TruthfulQA into two folds, fold $A$ consists of 2915 QA
pairs, fold B of 2967 QA pairs. Firstly training was performed on fold $A$ and
test on fold $B$, then the folds were flipped. The reported MC results are the
average from both folds.

To test generalization beyond TruthfulQA (as presented in subsection 3.5),
probing and intervention were performed on a full TruthfulQA dataset, while
testing was performed on test split of the OpenBookQA and ARC-C. In case of
tests on MMLU, we took the validation split, because its test split is large
(14k questions) and it results in large computational overhead during
hyperparameter optimization.

###  3.4 Results

Throughout this work, all experiments were performed on LLaMA-2-7B [18] model.
This open-source LLM has been used extensively in previous works on AI safety.
Particularly, it let us to directly compare NL-ITI to related methods [7, 11].
All ITI [7] results in this work were calculated on our code to ensure
reliable comparison.

Combining three optimizations described in Section 2.1 leads to surprising
performance improvements over the standard ITI. As can be seen in Table 1, the
MC1 score has improved by $50\text{\,}\mathrm{\char 37\relax}$ relative to the
baseline LLaMA-2-7B result. The effect of the intervention is
$13\text{\,}\mathrm{\char 37\relax}$ relative higher than the ITI.

Table 1: Comparison between baseline LLaMA-2-chat-7B, ITI and NL-ITI on TruthfulQA. Compared values have all been achieved with few-shot-prompting. |  | MC1 [%]
---
| MC2 [%]
---
| CE
---
| KL
---
| LLaMA-2-7B
---
33.54 | 50.34 | 2.53 | 0.00
| ITI
---
36.35 | 54.72 | 2.65 | 0.40
| TrFr
---
39.30 | - | 2.59 | 0.22
| NL-ITI (ours)
---
50.19 | 67.73 | 2.85 | 0.43
|  w/o optimized probe
---
42.96 | 61.48 | 2.66 | 0.25
|  w/o multi-token
---
40.75 | 59.83 | 3.33 | 1.40

In the Figure 2, we depicted accuracy of non-linear probing for each attention
head (compared to baseline linear probing). Interestingly, the effect of
applying non-linear probing to the attention heads was the most profound in
the first 6 layers of LLM. Significant amount of truthful knowledge is present
also in the first layers of LLM, but simple, linear probing models were not
able to decode it efficiently. Figure 2 shows that the truthful knowledge is
much more diffused along the attention heads than liner probing suggested.
The rows in Figure 3 correspond to the number of tokens used for training of
probing models, specifically to $\tau$ defined in Equation 5. Columns present
the influence of increasing the token number used in the intervention - $\rho$
from Equation 6. The reported impact of using an increased number of tokens is
particularly strong in probing. However, having the increased number of tokens
both in probing and intervention produces a joint effect and yields the
optimum 50.19 MC1 at $(\rho,\tau)=(6,4)$.
Our results suggest that significant amount of information about concept
(truthfulness) might be contained not only in vector corresponding to the last
token of LLM's input QA pair, but also in preceding vectors. For reference, in
the ITI approach, only the activations from the last token were used for
probing and intervention.

Figure 2: Probing accuracy for each attention head of the LLM on TruthfulQA
dataset for linear probing (ITI) – bottom, and non-linear probing (NL-ITI) –
top. Accuracy results are `smoothed' between neighboring attention heads
(lower standard deviation). Figure 3: Heat map of MC1 evaluation scores of
TruthfulQA dataset for different combinations of number of tokens used during
probing and intervention. The best performing model corresponds to
$(\rho,\tau)=(6,4)$.

###  3.5 Generalization of NL-ITI beyond TruthfulQA

Apart from tests on TruthfulQA dataset, the generalization power of NL-ITI can
be further measured through the evaluation on LLM reasoning benchmarks that
are related to concept of Truthfulness. In this scenario, we use TruthfulQA
for probing and intervention, but evaluation is performed on ARC, MMLU and
OpenBookQA datasets.

We intentionally omitted TriviaQA [24] and NaturalQuestions [25] benchmarks.
Together with the ITI [7], a modified version of those datasets111Additional
Datasets in https://github.com/likenneth/honest_llama was provided with false
answers generated by asking GPT-4 for 'most plausible sounding but false'
answer. We find that the automatic answer generation results often in low-
quality data. To be precise, we find that in the case of TriviaQA, around 35%
of questions contain 'false answers' which also appear in 'true answers'. For
example, to question Q: where did the allies go after north africa the dataset
provides answers T: Italy and supposedly adversarial, false answer F: the
Italian Campaign, Italy. In summary, answers generated by GPT-4, might distort
the evaluation results. For these reasons, we decided to evaluate NL-ITI on
public benchmarks (MMLU, TriviaQA, OpenBook), that we did not modify and that
does not require additional artificial data generation.

In summary, Table 3.5 shows that NL-ITI generalizes better to out-of-
distribution benchmarks. Lowest overall improvement can be observed on MMLU
dataset. Though if we look at particular subdomains of this dataset that are
related to TruthfulQA (e.g., Business Ethics), significant increase of
performance is visible. Interestingly, NL-ITI notably increases capabilities
of LLM in elementary mathematics subdomain that is not the main focus of
TruthfulQA dataset.

Table 2: Comparison of generalization of ITI and NL-ITI on out-of-
distributions benchmarks

|  | Dataset
---
| MC1 [%]
---
| MC2 [%]
---
| CE
---
| KL
---
LLaMA-2-7B | ARC | 41.20 | 40.69 | 2.53 | 0.00
ITI | ARC | 40.34 | 38.78 | 2.54 | 0.12
| NL-ITI
---
ARC | 44.27 | 43.20 | 2.82 | 0.40
LLaMA-2-7B | MMLU | 38.48 | 38.66 | 2.53 | 0.00
ITI | MMLU | 38.55 | 38.27 | 2.58 | 0.04
| NL-ITI
---
MMLU | 40.31 | 39.82 | 2.60 | 0.10
LLaMA-2-7B | OBQA | 27.31 | 26.36 | 2.53 | 0.00
ITI | OBQA | 30.52 | 28.26 | 2.82 | 0.40
| NL-ITI
---
OBQA | 33.94 | 32.65 | 2.86 | 0.32

Table 3: MC1 accuracy of ITI and NL-ITI on subdomains of MMLU dataset (Business Ethics, Elementary Mathematics, High School Government and Politics). | Ethics | Math | Politics
---|---|---|---
LLaMA-2-7B | 45.45 | 46.34 | 42.86
ITI | 54.55 | 51.22 | 33.33
NL-ITI | 63.64 | 60.98 | 57.14

##  4 Conclusions

In our work, we propose enhancements that significantly improve the
performance of the ITI method. NL-ITI outperforms ITI on 4 major benchmarks,
including TruthfulQA, on which we report around 14% MC1 metric improvement (up
to 50.19). Additionally, NL-ITI achieves significant improvement (+10 p.p.)
over recently released TrFR method.

NL-ITI outperforms ITI also on the out-of-distribution benchmarks (MMLU,
OpenBook, ARC) - highest improvement can be observed on ARC (+4 p.p.).
Moreover, for given level of intervention invasiveness (as measured by KL),
NL-ITI achieves better MC accuracy (as observed in Figure 1). In summary, NL-
ITI can generalize better at the lower cost (less invasive in the behavior of
LLM).

Our optimizations results from reasoning presented in the works that analyzed
probing usability [15, 16, 17]. Authors noticed that increasing the capacity
of the probing model is crucial for better estimation of the amount of desired
type of knowledge (e.g., truthful) in attention heads. Except for increasing
probing capacity, we also experimented with inputting more vectors
(information) to probe the model. Positive results of those experiments led us
to hypothesis that inputting more vectors to LLM intervention might result in
further performance improvement of ITI.

Editing LLM's internal representations is a promising direction for ensuring
safe, truthful, and more human-centric AI. In our opinion, valuable future
research would be to check the usability of NL-ITI in other important
scenarios. For example, whether it can efficiently steer LLMs towards
desirable ``personality" traits. Apart from that, using NL-ITI in synergy with
other representation engineering methods (e.g., TrFR) and ``fine-tuning"
methods (e.g., DPO, RLHF), might result in further and significant improvement
of performance.

## References

  * [1] P. Manakul, A. Liusie, and M. J. F. Gales, ``SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models,'' _arXiv Preprint, arXiv:2303.08896_ , 2023.
  * [2] J. Li, X. Cheng, W. X. Zhao, J.-Y. Nie, and J.-R. Wen, ``HaluEval: A large-scale hallucination evaluation benchmark for large language models,'' _arXiv Preprint, arXiv:2305.11747_ , 2023.
  * [3] O. Shaikh, H. Zhang, W. Held, M. Bernstein, and D. Yang, ``On second thought, let's not think step by step! bias and toxicity in zero-shot reasoning,'' _arXiv Preprint, arXiv:2212.08061_ , 2023.
  * [4] A. Taubenfeld, Y. Dover, R. Reichart, and A. Goldstein, ``Systematic biases in LLM simulations of debates,'' _arXiv Preprint, arXiv:2402.04049_ , 2024.
  * [5] K.-C. Yeh, J.-A. Chi, D.-C. Lian, and S.-K. Hsieh, ``Evaluating interfaced LLM bias,'' in _Proceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023)_ , 2023, pp. 292–299.
  * [6] A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, M. Fredrikson, J. Z. Kolter, and D. Hendrycks, ``Representation engineering: A top-down approach to ai transparency,'' 2023.
  * [7] K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg, ``Inference-time intervention: Eliciting truthful answers from a language model,'' 2023.
  * [8] T. Hartvigsen, S. Gabriel, H. Palangi, M. Sap, D. Ray, and E. Kamar, ``Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection,'' 2022.
  * [9] A. Parrish, A. Chen, N. Nangia, V. Padmakumar, J. Phang, J. Thompson, P. M. Htut, and S. Bowman, ``BBQ: A hand-built bias benchmark for question answering,'' in _Findings of the Association for Computational Linguistics: ACL 2022_ , S. Muresan, P. Nakov, and A. Villavicencio, Eds. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 2086–2105. [Online]. Available: https://aclanthology.org/2022.findings-acl.165
  * [10] S. Lin, J. Hilton, and O. Evans, ``Truthfulqa: Measuring how models mimic human falsehoods,'' 2022.
  * [11] Z. Chen, X. Sun, X. Jiao, F. Lian, Z. Kang, D. Wang, and C.-Z. Xu, ``Truth forest: Toward multi-scale truthfulness in large language models through intervention without tuning,'' _arXiv Preprint, arXiv:2312.17484_ , 2024.
  * [12] S. Mao, N. Zhang, X. Wang, M. Wang, Y. Yao, Y. Jiang, P. Xie, F. Huang, and H. Chen, ``Editing personality for LLMs,'' _arXiv Preprint, arXiv:2310.02168_ , 2023.
  * [13] K. Pan and Y. Zeng, ``Do LLMs possess a personality? Making the MBTI test an amazing evaluation for large language models,'' _arXiv Preprint, arXiv:2307.16180_ , 2023.
  * [14] J. tse Huang, W. Wang, M. H. Lam, E. J. Li, W. Jiao, and M. R. Lyu, ``Revisiting the reliability of psychological scales on large language models,'' _arXiv Preprint, arXiv:2305.19926_ , 2023.
  * [15] T. Pimentel, J. Valvoda, R. H. Maudslay, R. Zmigrod, A. Williams, and R. Cotterell, ``Information-theoretic probing for linguistic structure,'' _arXiv Preprint, arXiv:2004.03061_ , 2020.
  * [16] J. C. White, T. Pimentel, N. Saphra, and R. Cotterell, ``A non-linear structural probe,'' _arXiv Preprint, arXiv:2105.10185_ , 2021.
  * [17] J. Hościłowicz, M. Sowański, P. Czubowski, and A. Janicki, ``Can we use probing to better understand fine-tuning and knowledge distillation of the BERT nlu?'' in _Proceedings of the 15th International Conference on Agents and Artificial Intelligence (ICAART), Volume 3, Lisbon, Portugal, February 22-24, 2023_ , A. P. Rocha, L. Steels, and H. J. van den Herik, Eds. SCITEPRESS, 2023, pp. 625–632. [Online]. Available: https://doi.org/10.5220/0011724900003393
  * [18] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, ``Llama: Open and efficient foundation language models,'' 2023.
  * [19] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, ``Think you have solved question answering? try arc, the ai2 reasoning challenge,'' _arXiv:1803.05457v1_ , 2018.
  * [20] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, ``Measuring massive multitask language understanding,'' _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2021.
  * [21] T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, ``Can a suit of armor conduct electricity? a new dataset for open book question answering,'' in _EMNLP_ , 2018.
  * [22] A. Gokaslan, V. Cohen, E. Pavlick, and S. Tellex, ``Openwebtext corpus,'' http://Skylion007.github.io/OpenWebTextCorpus, 2019.
  * [23] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, ``Attention is all you need,'' 2023.
  * [24] M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer, ``Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension,'' 2017.
  * [25] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov, ``Natural questions: a benchmark for question answering research,'' _Transactions of the Association of Computational Linguistics_ , 2019.