#
Tight Bounds on the Chromatic Edge Stability Index of Graphs

Saieed<EMAIL_ADDRESS>Department of Mathematical Sciences,
Sharif University of Technology, Tehran, Iran John
<EMAIL_ADDRESS>Mathematical Institute, University of
Oxford, UK  Mehrbod<EMAIL_ADDRESS>Department of
Mathematical Sciences, Sharif University of Technology, Tehran, Iran Nasim
<EMAIL_ADDRESS>Department of Mathematical Sciences, Sharif
University of Technology, Tehran, Iran Helia
<EMAIL_ADDRESS>Department of Mathematical Sciences, Sharif
University of Technology, Tehran, Iran

###### Abstract

The chromatic edge stability index $\mathrm{es}_{\chi^{\prime}}(G)$ of a graph
$G$ is the minimum number of edges whose removal results in a graph with
smaller chromatic index. We give best-possible upper bounds on
$\mathrm{es}_{\chi^{\prime}}(G)$ in terms of the number of vertices of degree
$\Delta(G)$ (if $G$ is Class 2), and the numbers of vertices of degree
$\Delta(G)$ and ${\Delta(G)-1}$ (if $G$ is Class 1). If $G$ is bipartite we
give an exact expression for $\mathrm{es}_{\chi^{\prime}}(G)$ involving the
maximum size of a matching in the subgraph induced by vertices of degree
$\Delta(G)$. Finally, we consider whether a minimum mitigating set, that is a
set of size $\mathrm{es}_{\chi^{\prime}}(G)$ whose removal reduces the
chromatic index, has the property that every edge meets a vertex of degree at
least $\Delta(G)-1$; we prove that this is true for some minimum mitigating
set of $G$, but not necessarily for every minimum mitigating set of $G$.

2010 Mathematics Subject Classification: 05C15, 05C70
Key words: Edge coloring, chromatic index, chromatic edge stability index

##  1 Introduction

Throughout this paper all graphs have at least one edge and are simple, that
is, with no loops or multiple edges. Let $G$ be a graph with vertex set
${V}(G)$ and edge set ${E}(G)$. The degree of a vertex $v$ is denoted by
$d_{G}(v)$. The core of $G$, denoted by $G_{\Delta}$, is the subgraph of $G$
induced by the vertices of degree $\Delta(G)$, where $\Delta(G)$ is the
maximum degree of $G$. We denote the number of vertices of degree $i$ by
$t_{i}$. The graph $G$ is called acyclic if $G$ does not contain any cycles.
If $S\subseteq V(G)$, then the subgraph of $G$ induced by $S$ is denoted by
$G[S]$. Similarly, if $S^{\prime}\subseteq E(G)$, then the subgraph of $G$
induced by $S^{\prime}$ is denoted by $G[S^{\prime}]$.

A proper edge coloring (resp. vertex coloring) assigns colors to the edges
(resp. vertices) of a graph so that no two incident edges (resp. adjacent
vertices) have the same color. The chromatic index of $G$, $\chi^{\prime}(G)$
is the minimum number of colors required for a proper edge coloring of $G$.
The chromatic number of a graph $G$, $\chi(G)$ is similarly defined as the
minimum number of colors required for a proper vertex coloring of $G$. A
matching $M$ in $G$ is a set of pairwise non-incident edges. The matching
number $\alpha^{\prime}(G)$ of a graph $G$ is the size of a maximum matching.

The chromatic edge stability number $\mathrm{es}_{\chi}(G)$ of a graph $G$ is
the minimum number of edges whose removal results in a graph $H$ with
$\chi(H)=\chi(G)-1$. This concept was first introduced in the 1980’s by Staton
[7], and some inequalities and sharp bounds on $\mathrm{es}_{\chi}$ were
proved in [3]. Similarly, the chromatic vertex stability number $vs_{\chi}(G)$
of a graph $G$, defined as the minimum number of vertices whose removal
results in a graph $H$ with $\chi(H)=\chi(G)-1$, was first studied in [2].

A mitigating set is a set of edges $F$ in $G$ such that
$\chi^{\prime}(G\setminus F)<\chi^{\prime}(G)$.

In this paper, we focus on the chromatic edge stability index of $G$ denoted
by $\mathrm{es}_{\chi^{\prime}}(G)$, defined as the minimum number of edges
whose removal results in a graph $H$ with
${\chi^{\prime}(H)=\chi^{\prime}(G)-1}$. This concept was first studied in
[1].

As an example, consider the Petersen graph, $P$, in which
$\chi^{\prime}(P)=4$, see [8, p. 7]. As was proved in [8, p. 16], removing one
vertex of $P$ does not reduce the chromatic index. Hence, the removal of one
edge cannot reduce the chromatic index. However, one can reduce the chromatic
index by removing two edges, as illustrated in Figure 1. Hence,
$\mathrm{es}_{\chi^{\prime}}(P)=2$.

$a$$b$$a$$c$$b$$a$$a$$b$$c$$c$$a$$b$$c$ Figure 1: A $3$-edge-colorable graph
obtained by removing two edges from $P$.

We now recall Vizing’s Theorem [9].

######  Theorem 1.

[9] For a graph $G$, ${\Delta(G)\leq\chi^{\prime}(G)\leq\Delta(G)+1}$.

A graph $G$ is Class $1$ if ${\chi^{\prime}(G)=\Delta(G)}$ and it is Class $2$
if ${\chi^{\prime}(G)=\Delta(G)+1}$. The following theorem was first stated by
Fournier [5].

######  Theorem 2.

[5] Let $G$ be a graph. If $G_{\Delta}$ is acyclic, then $G$ is Class 1.

######  Theorem 3.

[4] Let $G$ be a connected graph with ${|G_{\Delta}|=3}$. Then $G$ is Class 2
if and only if for some integer $n$, $G$ is obtained from $K_{2n+1}$ by
removing $n-1$ independent edges.

##  2 Main Results

We now state our first theorem regarding the chromatic edge stability index.

######  Theorem 4.

Let $G$ be a graph of Class $2$. Then,
${\mathrm{es}_{\chi^{\prime}}(G)\leqslant{\frac{t_{\Delta}-1}{2}}}$.
Furthermore, if $G$ is connected and $t_{\Delta}$ is odd, then equality holds
if and only if $G$ is obtained from $K_{2n+1}$ by removing $s$ independent
edges for some integers $n$ and $s$, ${0\leq s\leq n-1}$.

###### Proof.

Note that it is sufficient to remove a set of edges $F$ such that
$|F|\leq\frac{t_{\Delta}-1}{2}$ and ${G\setminus F}$ is Class 1, since then we
have ${\chi^{\prime}(G\setminus F)=\Delta(G\setminus
F)\leq\Delta(G)<\chi^{\prime}(G)}$.

By Theorem 2, since $G$ is Class 2 it contains a cycle of vertices of degree
$\Delta(G)$; removing an edge from that cycle reduces the number of vertices
of degree $\Delta(G)$ by $2$. Continue in this manner until either the
remaining graph is Class 1 or ${\lfloor\frac{t_{\Delta}-1}{2}}\rfloor$ edges
have been removed. In the latter case the number of vertices of degree
$\Delta(G)$ remaining is at most $2$, so there is no cycle of such vertices
and Theorem 2 implies the remaining graph is Class 1. Thus
${\mathrm{es}_{\chi^{\prime}}(G)\leqslant\frac{t_{\Delta}-1}{2}}$.

Now, we prove the second part of the theorem. Let $A$ be the set of vertices
of maximum degree. First, assume that
${\mathrm{es}_{\chi^{\prime}}(G)=\frac{t_{\Delta}-1}{2}}$. Consider a maximum
matching $M$ in $G_{\Delta}$ and call the set of saturated vertices $V_{M}$.
Remove the edges of $M$ from $G$ and call the new graph $G^{\prime}$. Since
$t_{\Delta}$ is odd, $A\neq V_{M}$. Note that
$G^{\prime}_{\Delta}=G[A\setminus V_{M}]$ has no edges and by Theorem 2
$G^{\prime}$ is Class 1. Hence,
${\mathrm{es}_{\chi^{\prime}}(G)}\leqslant\alpha^{\prime}(G_{\Delta})$. Notice
that we have ${\alpha^{\prime}(G_{\Delta})\leq\frac{t_{\Delta}-1}{2}}$. By
combining these two inequalities and our initial assumption, we obtain
${\alpha^{\prime}(G_{\Delta})=\frac{t_{\Delta}-1}{2}}$. Hence, ${|A\setminus
V_{M}|=1}$. Suppose that ${A\setminus V_{M}=\\{v\\}}$. We claim that $v$ is
adjacent to all vertices of $V_{M}$. By contradiction, assume ${uv\notin
E(G_{\Delta})}$ for some ${u\in V_{M}}$. Suppose $uw\in M$. Remove all edges
in $M$ except $uw$ from $G$ and call the new graph $G^{\prime\prime}$. Note
that ${V(G^{\prime\prime}_{\Delta})=\\{u,v,w\\}}$ and
$G^{\prime\prime}_{\Delta}$ is a forest so by Theorem 2, $G^{\prime\prime}$ is
Class 1. Hence,
${\chi^{\prime}(G^{\prime\prime})=\Delta(G^{\prime\prime})=\Delta(G)<\chi^{\prime}(G)}$
and $G^{\prime\prime}$ is obtained by removing
$\mathrm{es}_{\chi^{\prime}}(G)-1$ edges from $G$, a contradiction. Similarly,
for every vertex ${u\in V_{M}}$, if $uw\in M$, by considering the matching
$M\setminus\\{uw\\}\cup\\{uv\\}$, it is proved that $u$ is adjacent to all
vertices in $A$. Therefore, ${G_{\Delta}\cong K_{t_{\Delta}}}$. Now, remove
$\alpha^{\prime}(G_{\Delta})-1$ edges of $M$ from $G$ and call the remaining
graph $H$. Since ${\mathrm{es}_{\chi^{\prime}}(G)-1}$ edges are removed from
$G$, ${\chi^{\prime}(H)=\chi^{\prime}(G)}$. Note that ${\Delta(H)=\Delta(G)}$.
Hence, we have ${\chi^{\prime}(H)=\chi^{\prime}(G)=\Delta(G)+1=\Delta(H)+1}$
and $H$ is Class 2. Since $H_{\Delta}\cong K_{3}$ and $H$ is connected, by
Theorem 3, $H$ is obtained from $K_{2n+1}$ by removing $n-1$ independent
edges, for some integer $n$. It is straightforward to see that $G$ is obtained
from $K_{2n+1}$ by removing $s$ independent edges, where
${s=(n-1)-(\alpha^{\prime}(G_{\Delta})-1)=n-\alpha^{\prime}(G_{\Delta})=n-\mathrm{es}_{\chi^{\prime}}(G)\leq
n-1}$.

Now, assume that $G$ is obtained from $K_{2n+1}$ by removing a matching $M$,
where $|M|=s$ and ${0\leq s\leq n-1}$. It is not hard to see that $G$ is Class
2. Note that ${\frac{t_{\Delta}-1}{2}=\frac{2n+1-2s-1}{2}=n-s}$. Therefore, by
the first part of the theorem, we have ${\mathrm{es}_{\chi^{\prime}}(G)\leq
n-s}$. Now, we prove that ${\mathrm{es}_{\chi^{\prime}}(G)\geq n-s}$. If
$\chi^{\prime}(G)$ is reduced by removing less than $n-s$ edges, then the
maximum number of edges of $G$ is:

$2n^{2}+(n-s)-1<{2n+1\choose 2}-s=|E(G)|,$

a contradiction. Hence,
${\mathrm{es}_{\chi^{\prime}}(G)=n-s=\frac{t_{\Delta}-1}{2}}$ and the proof is
complete. ∎

######  Remark 1.

For every value of $t_{\Delta}\geq 3$, the upper bound
${\lfloor\frac{t_{\Delta}-1}{2}\rfloor}$ can be attained for
$\mathrm{es}_{\chi^{\prime}}(G)$. If $t_{\Delta}$ is odd, this follows from
the equality case of Theorem 4. If ${t_{\Delta}=2k}$ is even, construct $G$ as
follows: start from $K_{2k+1}$ and remove an edge $xy$, then add a new vertex
$z$ and the edge $xz$. Note that ${\Delta(G)=2k}$ and there are $2k$ vertices
of degree $2k$ (all except $y$ and $z$). In a proper edge-coloring of a
subgraph of $G$, there are at most $k+1$ edges of the color used on $xz$, and
at most $k$ edges of any other color. Thus any ${H\subset G}$ with
${\chi^{\prime}(H)=2k}$ satisfies ${|E(H)|\leq 2k^{2}+1=|E(G)|-(k-1)}$, and so
${\mathrm{es}_{\chi^{\prime}}(G)\geqslant k-1}$.

######  Remark 2.

For every function $f(t_{\Delta})$, we can find a connected graph $G$ such
that ${\mathrm{es}_{\chi^{\prime}}(G)>f(t_{\Delta})}$. Consider the Petersen
graph, remove one of its edges, and call the remaining graph $Q$, as shown in
Figure 2.

Figure 2: The graph $Q\cong P\setminus\\{e\\}$.

The graph $G$ illustrated in Figure 3 is a chain of copies of $Q$ joined to a
copy of $K_{1,3}$. Note that $\chi^{\prime}(G)=4$, $t_{\Delta}=1$ and
$\Delta(G)=4$. As was previously stated, one must remove at least one more
edge from each copy of $Q$ in order to reduce the chromatic index. Therefore,
one can arbitrarily increase $\mathrm{es}_{\chi^{\prime}}(G)$ by adding more
copies of $Q$ to the chain.

$\hskip 15.6491ptQ\hskip 15.6491pt$$\hskip 15.6491ptQ\hskip 15.6491pt$$\hskip
15.6491ptQ\hskip 15.6491pt$… Figure 3: A graph $G$ with $t_{\Delta}=1$ and
$\mathrm{es}_{\chi^{\prime}}(G)$ arbitrarily large.

In the next result, we provide an upper bound for the chromatic edge stability
index in terms of $t_{\Delta-1}$ and $t_{\Delta}$.

######  Theorem 5.

For every graph $G$, $\mathrm{es}_{\chi^{\prime}}(G)\leq
t_{\Delta}+\frac{t_{\Delta-1}}{2}$. Furthermore, equality holds if and only if
$t_{\Delta-1}=0$ and $G_{\Delta}$ has no edges.

###### Proof.

We prove the following slightly stronger statement by induction. Let $H$ be a
graph with maximum degree at most $d$. Let $t$ be the number of vertices of
degree $d$, and let $s$ be the number of vertices of degree $d-1$ that have a
neighbor of degree at least $d-1$. Then we may remove at most $k$ edges from
$H$ to leave a graph of chromatic index at most $d-1$, where $k=t$ if $s=0$
and the vertices of degree $d$ form an independent set, and
$k=t+\lfloor\frac{s-1}{2}\rfloor$ otherwise.

We prove this by induction on $t$. If $t=0$ and $s=0$, the statement follows
immediately from Theorem 1. If $t=0$ and $s\geq 1$ then $H$ has maximum degree
$d-1$ and it suffices to remove $\lfloor\frac{s-1}{2}\rfloor$ edges to leave a
graph of Class 1. We do this as in the proof of Theorem 4: remove edges
between vertices of degree $d-1$ until at most one such edge remains, leaving
a Class 1 graph by Theorem 2. All such edges lie within the set $A$ of
vertices of degree $d-1$ meeting another vertex of degree $d-1$, initially
$|A|=s$, and each edge removed reduces $|A|$ by $2$, so after at most
$\lfloor\frac{s-1}{2}\rfloor$ edges are removed we have $|A|\leq 2$ and so
$H[A]$ has at most one edge, as required.

Now suppose $t>0$. If there is an edge $uv$ between two vertices of degree
$d$, remove that edge. If not, let $w$ be an arbitrary vertex of degree $d$.
If $w$ has a neighbor $x$ of degree $d-1$, remove the edge $wx$. Otherwise,
choose an arbitrary neighbor $y$ and remove the edge $wy$. In each case we
have removed one edge to leave a graph $H^{\prime}$; let $t^{\prime}$ be the
number of vertices of $H^{\prime}$ having degree $d$, and $s^{\prime}$ be the
number of vertices of degree $d-1$ that are adjacent to some other vertex of
degree at least $d-1$. We now apply the induction hypothesis to $H^{\prime}$.

In the first case, we have $t^{\prime}=t-2$ and $s^{\prime}\leq s+2$. We may
therefore remove at most $t-2+\lfloor\frac{s+1}{2}\rfloor$ edges from
$H^{\prime}$ to leave a graph $H^{\prime\prime}$ with
$\chi^{\prime}(H^{\prime\prime})\leq d-1$; together with the edge $uv$, we
have removed at most $t+\lfloor\frac{s-1}{2}\rfloor$ edges from $H$ to obtain
$H^{\prime\prime}$.

In the second case, we have $t^{\prime}=t-1$ and $s^{\prime}\leq s$, since the
set of vertices of degree $d-1$ with a neighbor of degree at least $d-1$ may
gain $w$, but must lose $x$. We may therefore remove at most $k^{\prime}$
edges from $H^{\prime}$ to leave a graph $H^{\prime\prime}$ with
$\chi^{\prime}(H^{\prime\prime})\leq d-1$, where $k^{\prime}=t^{\prime}=t-1$
if $s^{\prime}=0$ and
$k^{\prime}=t^{\prime}+\lfloor\frac{s^{\prime}-1}{2}\rfloor$ if
$s^{\prime}\geq 1$. It follows that $k^{\prime}\leq k-1$ and so we have
removed at most $k$ edges from $H$ to obtain $H^{\prime\prime}$.

In the final case, we have $t^{\prime}=t-1$ and $s^{\prime}=s$, since
$d_{H^{\prime}}(w)=d-1$ but $w$ has no neighbor of degree at least $d-1$, so
is not counted in $s^{\prime}$. Thus we may remove at most $k^{\prime}=k-1$
edges from $H^{\prime}$ to leave a graph $H^{\prime\prime}$ with
$\chi^{\prime}(H^{\prime\prime})\leq d-1$, as required.

This completes the proof of the stronger statement. The required upper bounds
follow by taking $H=G$ and $d=\Delta$, since $\chi^{\prime}(G)\geq\Delta$.
Note that we obtain the stronger bound of
$t_{\Delta}+\frac{t_{\Delta-1}-1}{2}$ unless $t_{\Delta-1}=s=0$ and
$G_{\Delta}$ has no edges.

It remains to show that equality holds if $t_{\Delta-1}=0$ and $G_{\Delta}$
has no edges. In this case, by Theorem 2, we have $\chi^{\prime}(G)=\Delta$.
In order to reduce the chromatic index to $\Delta-1$, we must remove at least
one edge meeting every vertex of degree $\Delta$. Since no edge meets more
than one of these vertices, we have $\mathrm{es}_{\chi^{\prime}}(G)\geq
t_{\Delta}$, as required. ∎

######  Remark 3.

Theorem 5 gives $\mathrm{es}_{\chi^{\prime}}(G)\leq
t_{\Delta}+\lfloor\frac{t_{\Delta-1}-1}{2}\rfloor$ whenever $t_{\Delta-1}\geq
1$. In fact this is sharp for any values of $t=t_{\Delta}\geq 1$ and
$s=t_{\Delta-1}\geq 1$. To see this (for $s\geq 3$), let $H_{1}$ be a Class 2
graph having $s$ vertices of maximum degree, such that
$\mathrm{es}_{\chi^{\prime}}(H_{1})={\lfloor\frac{s-1}{2}\rfloor}$ (see Remark
1). Let $H_{2}$ be a graph with $t$ vertices of degree
$\Delta(H_{2})=\Delta(H_{1})+1$, forming an independent set, and none of
degree $\Delta(H_{1})$ (for example, a disjoint union of stars). Then the
disjoint union $G=H_{1}\cup H_{2}$ has $\chi^{\prime}(G)=\Delta(G)$,
$t_{\Delta}=t$, $t_{\Delta-1}=s$, and at least $t$ edges from $H_{2}$ and
$\lfloor\frac{s-1}{2}\rfloor$ edges from $H_{2}$ must be removed to reduce its
chromatic index. If $s\leq 2$ then the bound reduces to $t_{\Delta}$, so the
graph $H_{1}$ should be omitted (and $\Delta(H_{2})$ may be chosen
arbitrarily).

Now, we provide a formula for the chromatic edge stability index of bipartite
graphs in terms of $t_{\Delta}$ and the matching number of $G_{\Delta}$.

######  Theorem 6.

For every bipartite graph $G$,
$\mathrm{es}_{\chi^{\prime}}(G)=t_{\Delta}-\alpha^{\prime}(G_{\Delta})$.

###### Proof.

First, we prove that ${\mathrm{es}_{\chi^{\prime}}(G)\geq
t_{\Delta}-\alpha^{\prime}(G_{\Delta})}$. Assume that $S^{\prime}$ is the
smallest mitigating set of $G$. Note that since $G$ is bipartite, it is Class
1 [6]. Hence, in order to reduce $\chi^{\prime}(G)$, it is necessary and
sufficient that there are no vertices of degree $\Delta(G)$ left in the
remaining graph. We now state two claims:
Claim 1. $G[S^{\prime}]$ is acyclic. By contradiction, assume that
$G[S^{\prime}]$ contains a cycle $C$. Consider an edge $e=uv$ in $C$ and let
$G^{\prime}$ be the graph obtained by removing the edges of the path $C-e$
from $G$. Note that $d_{G^{\prime}}(u)$ and $d_{G^{\prime}}(v)$ do not exceed
$\Delta(G)-1$. Hence, by removing $S^{\prime}-e$ from $G$, $\chi^{\prime}(G)$
is reduced, a contradiction.
Claim 2. $G[S^{\prime}]$ contains no path of length 3. Similarly to the
previous claim, if $G[S^{\prime}]$ contains a path of length 3, by removing
the middle edge of such a path from $S^{\prime}$, we get a subset of the edges
of size $|S^{\prime}|-1$ whose removal reduces $\chi^{\prime}(G)$, a
contradiction.

By the claims above, one can conclude that each component of $G[S^{\prime}]$
is the graph $K_{1,r}\ \text{for some}\ r\geq 1$.

Assume that the components of $G[S^{\prime}]$ are ${S_{1},\ldots,S_{k}}$. If
$S_{i}$ is a component with at least two edges and $l$ is a leaf of $S_{i}$,
then ${d_{G}(l)=\Delta(G)}$, because otherwise by removing the edge incident
with $l$ from $S^{\prime}$, we get a subset of edges of size $|S^{\prime}|-1$
whose removal reduces $\chi^{\prime}(G)$, a contradiction. By a similar
argument, if $S_{i}$ is a single edge, at least one of its endpoints has
degree $\Delta(G)$ in $G$. Now, assume that in exactly $r$ components of
$G[S^{\prime}]$, all vertices have degree $\Delta(G)$ in $G$. Hence, if we
take one edge of each of these components, we get a matching of size $r$ in
$G_{\Delta}$. Therefore, ${\alpha^{\prime}(G_{\Delta})\geq r}$. Note that
$G[S^{\prime}]$ contains all vertices of degree $\Delta(G)$ in $G$. Since
$t_{\Delta}=\sum_{i=1}^{k}{|E(S_{i})|}+r$, we have the following:

$\mathrm{es}_{\chi^{\prime}}(G)=|S^{\prime}|=\sum_{i=1}^{k}{|E(S_{i})|}=t_{\Delta}-r\geq
t_{\Delta}-\alpha^{\prime}(G_{\Delta}).$

Now, we prove that ${\mathrm{es}_{\chi^{\prime}}(G)\leq
t_{\Delta}-\alpha^{\prime}(G_{\Delta})}$. Consider a maximum matching $M$ in
$G_{\Delta}$. There are $t_{\Delta}-2|M|$ vertices of degree $\Delta(G)$ in
$G$ which are not saturated by $M$. Let $G^{\prime}$ be the graph obtained by
removing the edges of $M$ and one incident edge of each of these vertices from
$G$. Notice that ${\Delta(G^{\prime})\leq\Delta(G)-1}$ and since $G^{\prime}$
is Class 1,
${\chi^{\prime}(G^{\prime})=\Delta(G^{\prime})<\Delta(G)=\chi^{\prime}(G)}$.
Hence, we get the following:

$\mathrm{es}_{\chi^{\prime}}(G)\leq|M|+t_{\Delta}-2|M|=t_{\Delta}-\alpha^{\prime}(G_{\Delta}).$

The proof is complete. ∎

##  3 Structure of minimum mitigating sets

The mitigating sets constructed in the previous section all have the property
that they only contain edges meeting vertices of degree $\Delta(G)$ or
$\Delta(G)-1$. It is natural to ask whether minimum-size mitigating sets (i.e.
sets of $\mathrm{es}_{\chi^{\prime}}(G)$ edges whose removal reduces the
chromatic index) necessarily have this property. We will show that the answer
is “yes” in the sense that every graph has a minimum-size mitigating set with
this property, but “no” in the sense that some graphs also have minimum-size
mitigating sets that do not satisfy this property.

######  Theorem 7.

Let $G$ be any graph. Then there exists a mitigating set $S$ for $G$ of size
$\mathrm{es}_{\chi^{\prime}}(G)$ such that if $e=uv\in S$ then

$\max(d(u),d(v))\geq\Delta(G)-1.$ (1)

###### Proof.

Let $T$ be any mitigating set of $G$ of size $\mathrm{es}_{\chi^{\prime}}(G)$,
and suppose that $T$ contains an edge $e=uv$ which does not satisfy (1). We
will adapt the proof of Vizing’s theorem to show that there is some edge
$uv^{\prime}$ with $d(v^{\prime})\geq\Delta(G)-1$, such that
$T\cup\\{uv^{\prime}\\}\setminus\\{uv\\}$ is also a mitigating set for $G$. By
a sequence of such replacements, we eventually obtain a set $S$ with the
required properties.

Write $G^{\prime}=G\setminus T$. Then, by minimality of $T$,
$\chi^{\prime}(G^{\prime})=\chi^{\prime}(G)-1$ but
$\chi^{\prime}(G^{\prime}\cup\\{uv\\})=\chi^{\prime}(G)$. Write $k$ for
$\chi^{\prime}(G^{\prime})$; note that $k\geq\Delta(G)-1$. Fix a $k$-edge-
coloring of $G^{\prime}$. For any two colors $c_{1}$ and $c_{2}$, we say that
a _$c_{1}c_{2}$ -path_ is a path whose edges alternate colors $c_{1}$ and
$c_{2}$.

We will construct a sequence $v=v_{0},v_{1},\ldots,v_{r}$ of neighbors of $u$
such that $d_{G}(v_{r})\geq d_{G^{\prime}}(v_{r})\geq k$. Since
$d_{G^{\prime}}(u)<d_{G}(u)<k$, there is a color $c$ which is not used at $u$,
and similarly there is a colour $c_{0}$ which is not used at $v_{0}$. However,
$c_{0}$ must be used at $u$, and $c$ must be used at $v_{0}$, since otherwise
we could color $uv_{0}$ with one of these colors to obtain a $k$-edge-coloring
of $G^{\prime}\cup\\{uv\\}$. Also, there must be a $cc_{0}$-path from $u$ to
$v_{0}$, since otherwise we could swap colours in the $cc_{0}$ path meeting
$u$ and then color $uv_{0}$ with $c_{0}$ to obtain a $k$-edge-coloring of
$G^{\prime}\cup\\{uv\\}$. Choose the neighbor $v_{1}$ such that $uv_{1}$ has
color $c_{0}$.

Suppose we have constructed a sequence $v_{0},\ldots,v_{i}$ of distinct
neighbors of $u$, and a sequence of colors $c_{0},\ldots,c_{i-1}$ such that
for each $j<i$ the color $c$ is used at $v_{j}$, the color $c_{j}$ is not used
at $v_{j}$, there is a $cc_{j}$-path from $u$ to $v_{j}$, and $uv_{j+1}$ has
color $c_{j}$.

Suppose also that $d_{G^{\prime}}(v_{i})<k$. Then there is some color $c_{i}$
not used at $v_{i}$. The color $c_{i}$ must be used at $u$, and $c$ must be
used at $v_{i}$, since otherwise we could recolor $uv_{j}$ with one of these
colors, and give $uv_{j}$ color $c_{j}$ for each $j<i$ to obtain a $k$-edge-
coloring of $G^{\prime}\cup\\{uv\\}$. Also, there must be a $cc_{i}$-path from
$u$ to $v_{i}$, since otherwise we could first swap colors in the
$cc_{i}$-path meeting $u$ and then give $uv_{j}$ color $c_{j}$ for each $j\leq
i$ to obtain a $k$-edge-coloring of $G^{\prime}\cup\\{uv\\}$. In particular it
follows that $c_{i}\neq c_{j}$ for each $j<i$, since there cannot be
$cc_{i}$-paths from $u$ to both $v_{j}$ and $v_{i}$. Choose the neighbor
$v_{i+1}$ such that $uv_{i+1}$ has color $c_{i}$; this ensures $v_{i+1}$ is
distinct from $v_{0},\ldots,v_{i}$.

Thus the sequence can be extended until we have $d_{G^{\prime}}(v_{r})\geq
k\geq\Delta-1$. Now we may remove the edge $uv_{r}$ and give $uv_{j}$ color
$c_{j}$ for each $j<r$, to obtain a $k$-edge-coloring of
$G^{\prime}\cup\\{uv\\}\setminus\\{uv_{r}\\}$. Thus
$T\cup\\{uv_{r}\\}\setminus\\{uv\\}$ is also a mitigating set of $G$, but has
fewer edges violating (1). By choosing $S$ to minimise the number of edges
violating (1), we must have (1) holding for all edges of $S$, as required. ∎

To complete this section, we show the following.

######  Proposition 8.

There exists a graph $G$ having a mitigating set $S$ of size
$\mathrm{es}_{\chi^{\prime}}(G)$ and an edge $uv\in S$ which does not satisfy
(1).

###### Proof.

Construct a nine-vertex graph $G^{\prime}$ as follows. Let
$V(G^{\prime})=A\cup B\cup C\cup\\{u,v\\}$, with $|A|=|C|=2$ and $|B|=3$. Add
all edges between $u$ and $A$, between $A$ and $B$, between $B$ and $C$, and
between $C$ and $v$. Now define a graph $G$ by adding two edges to
$G^{\prime}$: the edges $uv$ and $xy$, where $x$ and $y$ may be any two
vertices in $A\cup B\cup C$ that are not adjacent in $G^{\prime}$. (There are
several possible choices for the second edge, giving three non-isomorphic
possibilities for $G$.)

$u$$v$$1$$2$$2$$3$$4$$1$$4$$3$$3$$4$$4$$1$$2$$3$$2$$1$$x$$y$$A$$B$$C$ Figure
4: The graphs $G^{\prime}$ (solid edges) and one choice of $G$ (solid and
dashed edges), with a $4$-edge-coloring of $G$.

Figure 4 shows a $4$-edge-colouring for $G^{\prime}$. Since $\Delta(G)=5$ we
clearly have $\chi^{\prime}(G)\geq 5$, and indeed the colouring shown may be
extended to a $5$-edge-colouring of $G$ by using a fifth color on the two
extra edges.

The set $S=\\{uv,xy\\}$ thus shows that $\mathrm{es}_{\chi^{\prime}}(G)\leq
2$, and since $uv$ does not satisfy (1), it is sufficient to show that $S$ is
a minimum mitigating set, i.e. $\mathrm{es}_{\chi^{\prime}}(G)=2$.

Suppose that this is not the case, so that
$\chi^{\prime}(G\setminus\\{e\\})=4$ for some $e\in E(G)$. In order for this
to happen, certainly we need $\Delta(G\setminus\\{e\\})\leq 4$, and so we must
have $e=xy$. Therefore $G\setminus\\{e\\}=G^{\prime}\cup\\{uv\\}$ would have
to be $4$-edge-colorable. Take any $4$-edge-coloring of $G^{\prime}$ with
colors $c_{1},c_{2},c_{3},c_{4}$; let $c_{1},c_{2}$ be the colors used at $u$.
Since the vertices in $A$ have degree $4$, each color is used the same number
of times on edges meeting $A$. Consequently $c_{1}$ and $c_{2}$ are used once
on edges between $A$ and $B$, and $c_{3}$ and $c_{4}$ are used twice on edges
between $A$ and $B$. Similarly, since each color must be used the same number
of times on edges meeting $B$, $c_{1}$ and $c_{2}$ are used twice on edges
between $B$ and $C$, and $c_{3}$ and $c_{4}$ are used once on edges between
$B$ and $C$. Finally, since each color must be used the same number of times
on edges meeting $C$, the colors used at $v$ must be $c_{3}$ and $c_{4}$. Thus
there is no way to extend any $4$-edge-coloring of $G^{\prime}$ to a $4$-edge-
coloring of $G^{\prime}\cup\\{uv\\}$, which completes the proof. ∎

######  Remark 4.

The same proof works with $|A|=|C|=k$ and $|B|=2k-1$ to give an infinite
sequence of graphs with this property.

## Acknowledgements

The research of the first author was supported by grant number G981202 from
Sharif University of Technology. The second author was supported by the
European Research Council under the European Union’s Horizon 2020 research and
innovation programme (grant agreement no. 883810).

## References

  * [1] S. Akbari, A. Beikmohammadi, B. Brešar, T. Gologranc, M. M. Habibollahi, N. Movarraei, On the chromatic edge stability index of graphs. European J. Combin., to appear.
  * [2] S. Akbari, A. Beikmohammadi, S. Klavžar, N. Movarraei, On the chromatic vertex stability number of graphs, European J. Combin., 102 (2022), 103504, 7 pp.
  * [3] S. Akbari, S. Klavžar, N. Movarraei, M. Nahvi, Nordhaus-Gaddum and other bounds for the chromatic edge-stability number, European J. Combin., 84 (2020), 103042, 8 pp.
  * [4] A.G. Chetwynd and A.J.W. Hilton, Regular graphs of high degree are 1-factorizable, Proc. London Math. Soc., 50 (1985), 193–206.
  * [5] J.-C. Fournier. Coloration des aretes dun graphe. Cahiers du CERO (Bruxelles), 15 (1973), 311–314.
  * [6] D. Kőnig, Gráok és alkalmazásuk a determinánsok és a halmazok elméletére, Matematikai és Természettudományi Értesitő, 34 (1916), 104–119.
  * [7] W. Staton, Edge deletions and the chromatic number, Ars Combin. 10 (1980), 103–106.
  * [8] M. Stiebitz, D. Scheide, B. Toft, L. M. Favrholdt, Graph edge coloring: Vizing’s Theorem and Goldberg’s Conjecture, New Jersey, Wiley (2012).
  * [9] V. G. Vizing, On an estimate of the chromatic class of a $p$-graph, Diskret. Analiz., 3 (1964), 25–30.


# An overview of methods for receiver operating characteristic analysis, with
an application to SARS-CoV-2 vaccine-induced humoral responses in solid organ
transplant recipients

Nathaniel P. Dowd<EMAIL_ADDRESS>
Department of Biostatistics
Vanderbilt University
Nashville, TN 37203, USA Bryan Blette<EMAIL_ADDRESS>
Department of Biostatitics
Vanderbilt University Medical Center
Nashville, TN 37203, USA James D. Chappell<EMAIL_ADDRESS>
Division of Pediatric Infectious Diseases
Vanderbilt University Medical Center
Nashville, TN 37232, USA Natasha B. Halasa<EMAIL_ADDRESS>
Division of Pediatric Infectious Diseases
Vanderbilt University Medical Center
Nashville, TN 37232, USA Andrew J. Spieker<EMAIL_ADDRESS>
Department of Biostatitics
Vanderbilt University Medical Center
Nashville, TN 37203, USA

###### Abstract

Receiver operating characteristic (ROC) analysis is a tool to evaluate the
capacity of a numeric measure to distinguish between groups, often employed in
the evaluation of diagnostic tests. Overall classification ability is
sometimes crudely summarized by a single numeric measure such as the area
under the empirical ROC curve. However, it may also be of interest to estimate
the full ROC curve while leveraging assumptions regarding the nature of the
data (parametric) or about the ROC curve directly (semiparametric). Although
there has been recent interest in methods to conduct comparisons by way of
stochastic ordering, nuances surrounding ROC geometry and estimation are not
widely known in the broader scientific and statistical community. The
overarching goals of this manuscript are to (1) provide an overview of
existing frameworks for ROC curve estimation with examples, (2) offer
intuition for and considerations regarding methodological trade-offs, and (3)
supply sample R code to guide implementation. We utilize simulations to
demonstrate the bias-variance trade-off across various methods. As an
illustrative example, we analyze data from a recent cohort study in order to
compare responses to SARS-CoV-2 vaccination between solid organ transplant
recipients and healthy controls.

Keywords: Nonparametric, Parametric, Receiver Operating Characteristic,
Semiparametric, SARS-CoV-2, Software, Tutorial

##  1 Introduction

Receiver operating characteristic (ROC) analysis is a tool to summarize the
extent to which a numeric measure accurately distinguishes between groups. ROC
methodology has historical roots in electrical engineering as a method to
detect enemy objects in battlefields, but became heavily associated with
assessment of diagnostic utility shortly after its development (Green and
Swets, 1966). In clinical practice, it is often of interest to determine
whether a diagnostic test based on some numeric measure has the ability to
discriminate between two states of health. In particular, it is often a goal
to develop classification measures that help reduce the invasiveness and/or
costs associated with diagnostic tests while maintaining accuracy of
classification. For example, a “gold standard” test in the diagnosis of
prostate cancer as determined by biopsy is invasive and impractical as a
routine clinical screening measure; the prostate-specific antigen is a
biomarker that has been used for prostate cancer detection (Barry, 2001).
Numerous examples of ROC analysis have been published, and its use is still
widespread in the modern era (Leblhuber et al., 1988; Spieker et al., 2013;
Johnson et al., 2016; Xu et al., 2024; Zhu et al., 2024; Zhou et al., 2024).
Summary measures associated with ROC analysis have also long been used to
compare classification ability across machine learning methods (Hanley and
McNeil, 1983).

More recently, researchers have more broadly recognized stochastic ordering as
a scientifically relevant measure in its own right to compare groups—including
in contexts outside of diagnostic tests (Acion et al., 2006; Zhang et al.,
2019; Spieker et al., 2021). Proper procedures for model selection,
implementation, inference, and interpretation are less widely known in the
statistical and scientific community as compared to, for example, methods to
compare means across population strata (Freedman, 2009).

Parametric, nonparametric, and semiparametric approaches to ROC estimation
have been characterized in the literature, largely separately (Metz, 1986;
Thompson and Zucchini, 1989; Kim and Zeng, 1998; Pepe, 2000, 2003). A bias-
variance trade-off is to be expected when comparing estimators across these
classes (with more structure generally offering greater efficiency under
correct assumptions, but introducing bias under assumption violations), though
the nature of that trade-off in the space of ROC methods is not as well
explored. Further, inference regarding the ROC curve is deceptively simple due
to the fact that both the x\- and y-coordinates of the overall curve are
estimated. Subtleties and nuances regarding the fundamental geometry of ROC-
based methods, as well as considerations surrounding estimation, merit more
explicit characterization and further illustration. The fundamental goal of
this manuscript is to address this need.

Throughout this manuscript, we anchor our illustrations using two foundational
examples for ease of characterizing various trade-offs: exponential methods
and normal methods. The remainder of this manuscript is organized as follows.
In Section 2, we provide definitions and an overview of the ROC curve and
considerations regarding its geometry. In Section 3, we discuss and provide
sample R code for estimation procedures associated with the ROC curve,
including related summary measures. In Section 4, we describe and present the
results of various simulation studies to highlight the trade-offs between
approaches. In Section 5, we apply several methods to a recent study seeking
to characterize immune responses to SARS-CoV-2 vaccination in solid organ
transplant (SOT) recipients. Finally, we conclude with a discussion of our
findings in Section 6.

##  2 The ROC curve and its geometry

###  2.1 Notation

Throughout this manuscript, we let $i=1,\dots,n_{0}$ index independently
sampled observations from a reference distribution with outcomes
$Y_{01},\dots,Y_{0n_{0}}$, each having common cumulative distribution function
(CDF) $F_{0}(t)=\text{P}(Y_{0}\leq t)$, survivor function
$S_{0}(t)=1-F_{0}(t)$, and quantile function $F_{0}^{-1}(p)$. Analogously, we
let $j=1,\dots n_{1}$ index independently sampled observations from a
comparator distribution with outcomes $Y_{11},\dots,Y_{1n_{1}}$, each having
common CDF $F_{1}(t)$, survivor function $S_{1}(t)$, and quantile function
$F_{1}^{-1}(p)$. Note that in cases where $F_{0}$ and $F_{1}$ are not
invertible (e.g., associated with discrete-valued outcomes),
$F_{0}^{-1}(\cdot)$ and $F_{1}^{-1}(\cdot)$ denote generalized inverses (for
instance, $F_{0}^{-1}(p)=\inf\\{t:F_{0}(t)>p\\}$). For ease of notation, we
let $n=n_{0}+n_{1}$ denote the combined sample size, and we will often refer
generically to the measure of interest as $Y$.

The utility of the measure $Y$ as a classifier is most easily described first
at a particular cut-off point, $c\in\text{supp}(Y)$. The true positive rate
(TPR) marks the proportion of observations in the comparator group correctly
classified at that cut-off; the false positive rate (FPR) marks the proportion
of observations in the reference group incorrectly classified. The graph of an
ROC curve summarizes these two complementary classifying features of the
measure $Y$ across all possible cut-off points, and is defined as a plot of
the FPR (i.e., one minus the specificity) at a particular cut-off point on the
x-axis and the TPR (i.e., sensitivity) at that cut-off on the y-axis. The ROC
curve is a monotone, surjective function
$\text{ROC}:[0,1]\longrightarrow[0,1]$, so that $p$, the FPR, denotes the
input and $\text{ROC}(p)$, the TPR, denotes the output (Pepe, 2000; Fawcett,
2006).

###  2.2 Conventions and symmetries

To characterize the ROC curve as a function of observable data, we must
specify whether higher or lower values of $Y$ signify a less desirable
outcome. In many settings, scientific knowledge will provide a researcher with
this information a priori: for example, higher hemoglobin A1c (HbA1c) is
generally known to be less desirable in patients with type 2 diabetes mellitus
(Cavero-Redondo et al., 2017), and lower CD4+ T cell count is known to be less
desirable in patients with human immunodeficiency virus (Doitsh et al., 2014).
When lower values of $Y$ are considered less desirable, the ROC function is
best defined as $\text{ROC}(p)=F_{1}(F_{0}^{-1}(p))$, for $0\leq p\leq 1$. On
the other hand, when higher values of $Y$ are considered less desirable, the
ROC function is instead best defined as $\text{ROC}(p)=S_{1}(S_{0}^{-1}(p))$.
In either case, the ROC curve can be thought of as a sensible composition of
functions: the inner function determines the cut-off value, $c_{p}$, such that
$100\times(1-p)\%$ of those in the reference group are correctly classified,
and the outer function determines the relative frequency of those in the
comparator group correctly classified at said cut-off. This is illustrated in
Figure 1.

Figure 1: Illustration of the ROC curve under the convention that lower values
are less desirable. In Panel (A), the reference group has a higher mean value
of $Y$ and substantially greater variability; a cut-off point of $c=5.5$
results in a FPR of approximately 61% (shaded pink) and a TPR of approximately
99% (shaded blue). Panel (B) depicts the full ROC curve, which can be thought
of as sliding the cut-off point across all possible values of $c$ (the
specific point corresponding to $c=5.5$ is marked on the ROC curve).
Increasing values of $c$ correspond to sliding “up and to the right” on the
ROC curve, while decreasing values of $c$ correspond to sliding “down and to
the left.”

In addition to setting a directional convention, we must also decide which
group serves as the reference group (0) and which is the comparator (1). This
will often have a natural synergy with the convention of direction described
above (for example, healthy controls will tend to have lower HbA1c as compared
to patients with type 2 diabetes). However, in a comparative study of two
active treatments (e.g., the effect of proton vs. photon therapy for cancer
treatment), context will not always allow the researcher to so easily intuit
which group should serve as a reference in advance (Baumann et al., 2020).
Further, it may not even always be possible to characterize a higher value of
$Y$ as “better” or “worse” (e.g., a comparison of hormone levels between males
and non-males), such that two independent researchers analyzing the same data
could reasonably choose opposite conventions (Thakur and Paramanik, 2009). To
maintain the typical convention of an ROC curve that, loosely speaking—and
with further elaboration in Section 2.3—lies mostly on the upper-left side of
the unit square $[0,1]\times[0,1]$, a change in convention of directionality
necessitates a simultaneous reversal of groups. This translates to an
important symmetry in the resulting ROC curve, described as follows: a change
in directional convention corresponds to the mapping $(x,y)\mapsto(1-x,1-y)$,
and a reversal of groups corresponds to the mapping $(x,y)\mapsto(y,x)$, so
that if $(p,\text{ROC}(p))$ denotes the graph of the original ROC curve, the
graph of the updated ROC curve is denoted by $(1-\text{ROC}(p),1-p)$. This
phenomenon is illustrated in Figure 2.

Figure 2: Illustration of an ROC convention change. An ROC curve (shown in
solid blue) is depicted so that it, loosely speaking, “lies mostly on the
upper-left side.” Importantly, consideration of stochastic ordering can be of
scientific interest even when there is no natural way to define “better” or
“worse.” Following the symmetries associated with a change in directionality
and a reversal of groups, the ROC curve under a change in convention (dashed
green) can be realized as a reflection of the original curve about the line
$y=1-x$.

In line with our ultimate motivating example (Section 5), we will follow the
convention that higher values are more desirable (largely without loss of
generality, with a notable exception serving as the topic of discussion in
Section 3.4).

###  2.3 Area under the curve

The overall quality of $Y$ as a classifier can be summarized by the area under
the ROC curve (AUC), given by:

$\displaystyle\text{AUC}=\int_{0}^{1}\text{ROC}(p)dp=\int_{0}^{1}F_{1}(F_{0}^{-1}(p))dp=\text{P}(Y_{0}>Y_{1}).$

The value of the AUC is bounded below by 0 and above by 1; a change of
directional convention alone or a reversal of groups alone would each update
the AUC via the relationship $\text{AUC}^{\prime}=1-\text{AUC}$, which is to
say that the AUC is preserved by the convention change illustrated in Figure
2; typically, a convention is chosen such that the AUC is no smaller than
$0.5$. An ROC curve for two well-separated distributions will pass through the
point (0, 1), with an AUC of 1.0 (Fawcett, 2006). An ROC curve for two
identical distributions will lie along the line $y=x$, with an AUC of 0.5.
Importantly, equality of $F_{0}(t)$ and $F_{1}(t)$ for all values of $t$
implies an AUC of 0.5, but not the other way around (i.e., it is possible to
achieve an AUC of 0.5 when comparing two distributions that are different; an
example will be provided in Section 2.5). This distinction is important, as
there are two fundamental tests that may be of interest when evaluating the
predictive capacity of $Y$: a test of a weak null hypothesis,
$H_{0}:\text{AUC}=0.5$, and a test of a strong null hypothesis,
$H_{0}:\text{ROC}(p)=p$ for all $0\leq p\leq 1$ (i.e., the ROC curve lies
along the line $y=x$). Of note, and as previously pointed out (Hart, 2001), a
test of the weak null hypothesis is sometimes mischaracterized as a test of a
difference in medians: it can be shown that two groups can have the same
median value of $Y$ when the weak null is false, and that two groups can have
different median values of $Y$ when the weak null is true. Although not our
emphasis, we offer methodological guidance on both characterizing and testing
the weak and the strong null hypotheses.

###  2.4 ROC curves emerging from exponential data

Suppose that $Y_{0}\sim\text{Exponential}(\lambda_{0})$, with CDF given by
$F_{0}(t)=1-\exp(-\lambda_{0}t)$ and quantile function given by
$F_{0}^{-1}(p)=-\lambda_{0}^{-1}\log(1-p)$; analogously, assume that
$Y_{1}\sim\text{Exponential}(\lambda_{1})$. It is readily shown in this
setting that the family of ROC curves indexed by this parameterization is
given by $\text{ROC}(p)=1-(1-p)^{\alpha}$, where
$\alpha=\lambda_{1}/\lambda_{0}\geq 1$. We choose to refer to this family of
ROC curves as the biexponential ROC family in order to follow the previously
established terminology used for normal data (Section 2.5). This family is
illustrated in Figure 3(A) for different choices of $\alpha$. It is
straightforward to show that the AUC in this example can be expressed in
closed form as $\text{AUC}=1-(\alpha+1)^{-1}$, $\alpha\geq 1$, and that both
the weak and strong null hypotheses are represented by $H_{0}:\alpha=1$. A
convention change as described in Section 2.2 and illustrated in Figure 2
updates the form of the ROC curve as $\text{ROC}(p)=p^{\alpha^{\prime}}$,
$0<\alpha^{\prime}<1$, which, quite notably, indexes a different family of ROC
curves than the family indexed by the original convention. Together, these
facts signify a certain restrictiveness in the form of the single-parameter
biexponential ROC family.

###  2.5 ROC curves emerging from normal data

Suppose now that $Y_{0}\sim\mathcal{N}(\mu_{0},\sigma_{0}^{2})$, with CDF
given by $F_{0}(t)=\Phi((t-\mu_{0})/\sigma_{0})$ and quantile function given
by $F_{0}^{-1}(p)=\mu_{0}+\sigma_{0}\Phi^{-1}(p)$; analogously, assume that
$Y_{1}\sim\mathcal{N}(\mu_{1},\sigma_{1}^{2})$. It is straightforward to show
that the family of ROC curves indexed by this parameterization is given by
$\text{ROC}(p)=\Phi(\delta\sigma_{1}^{-1}+\sigma_{0}\sigma_{1}^{-1}\Phi^{-1}(p))$,
where $\delta=\mu_{0}-\mu_{1}$. This family is often referred to as the
binormal ROC family (Hanley, 1988), and is illustrated in Figures 3(B) and
3(C) for different choices of $\delta$ and $\sigma_{0}/\sigma_{1}$. For
reasons that will become more apparent when we discuss estimation, the
binormal family is often expressed as
$\text{ROC}(p)=\Phi(\beta_{0}+\beta_{1}\Phi^{-1}(p))$, with
$\beta_{0}=\delta\sigma_{1}^{-1}$ and $\beta_{1}=\sigma_{0}\sigma_{1}^{-1}$.
The binormal ROC curve offers certain flexibility not offered by the
biexponential family. First, note that although no closed-form expression
exists for the AUC of a binormal ROC curve, the weak null is represented by
$H_{0}:\beta_{0}=0$ (that is, $\mu_{0}=\mu_{1}$), and the strong null is
represented by $H_{0}:\beta_{0}=0\text{ and }\beta_{1}=1$ (that is,
$\mu_{0}=\mu_{1}$ and $\sigma_{0}=\sigma_{1}$). Secondly, note that a
convention change corresponds to a re-parameterization that remains in the
same binormal family under the original convention, with
$\beta_{0}\mapsto\beta_{0}/\beta_{1}$ and $\beta_{1}\mapsto-1/\beta_{1}$. The
latter fact, in particular, speaks to the broader range of shapes that can be
taken by the binormal family under a single convention.

Figure 3: Illustration of families of ROC curves. Panel (A) illustrates a
family of biexponential ROC curves; note that for $\alpha>1$, the ROC curve
always adheres to the same class of asymmetry about $y=-x$ under a chosen
convention. Panel (B) illustrates a family of binormal ROC curves indexed by
normal data with equal variances (i.e., $\beta_{1}=1$); Panel (C) illustrates
a family of binormal ROC curves indexed by normal data with unequal variances
(in particular, $\beta_{1}=\sigma_{0}/\sigma_{1}=0.5$; note that choosing
$\beta_{1}=-2$ would result in a family of curves reflected about the line
$y=-x$ for specific values of $\delta$; not depicted).

###  2.6 More on the biexponential and binormal curves

We have now derived two forms for the ROC curve: one that emerges from
exponentially distributed data (Section 2.4) and one that emerges from
normally distributed data (Section 2.5). Importantly, ROC curves taking either
of these forms can emerge from other combinations of distributions. That is to
say, for example, that exponentially distributed data imply a biexponential
form for the ROC curve, but not the reverse. Consider the following examples.
Suppose first that $Y_{0}$ follows a normal distribution with quantile
function $F_{0}^{-1}(p)=\mu_{0}+\sigma_{0}\Phi^{-1}(p)$. It is straightforward
to verify that a biexponential ROC curve is achieved by assigning the
following CDF to $Y_{1}$:
$F_{1}(t)=1-(1-\Phi((t-\mu_{0})/\sigma_{0}))^{\alpha}$. As another example,
suppose that $Y_{0}$ follows an exponential distribution with quantile
function $F_{0}^{-1}(p)=-\lambda_{0}^{-1}\log(1-p)$. It is again
straightforward to verify that a binormal ROC curve is achieved by assigning
the following CDF to $Y_{1}$:
$F_{1}(t)=\Phi(\beta_{0}+\beta_{1}\Phi^{-1}(1-\exp(-\lambda_{0}t)))$. The CDFs
that induce a desired structure on the form of the ROC curve do not go by
known names, and are unlikely to offer practical relevance except to
illustrate the lack of a one-to-one relationship between parametric
assumptions on the data and the form of the ROC family. Nevertheless, the idea
that multiple parametric structures on the data can produce the same family of
ROC curves is crucial to understand why direct modeling of the ROC curve
without parametric assumptions on the underlying data can be conceptualized as
a semiparametric method. These examples offer a heuristic but nevertheless
constructive argument that approaches to directly model the ROC curve do not
implicitly impose a single parametric structure on the data (which, if they
did, would render the direct modeling approaches trivial).

##  3 Estimation of the ROC curve

In this section, we outline various methods for estimation of the ROC curve.
For ease of exposition, we assume for the majority of our examples that we are
interested in obtaining point and interval estimates for the sensitivity,
$\text{ROC}(p)$, at given values of the FPR, $p$. In Section 3.4, we briefly
discuss alternative ways of characterizing uncertainty.

###  3.1 Nonparametric estimation

Let $\mathbb{F}_{0}(t)=n_{0}^{-1}\sum_{i=1}^{n_{0}}I(Y_{0i}\leq t)$ denote the
empirical CDF for the reference group, and
$\mathbb{F}_{0}^{-1}(p)=\inf\\{t:\mathbb{F}_{0}(t)>p\\}$; define
$\mathbb{F}_{1}(t)$ and $\mathbb{F}_{1}^{-1}(p)$ analogously. The empirical
ROC curve can be estimated nonparametrically as
$\widehat{\text{ROC}}(p)=\mathbb{F}_{1}(\mathbb{F}_{0}^{-1}(p))$. In a finite
sample, the empirical ROC curve takes the shape of a staircase. Given a set of
data, the empirical ROC curve can be generated using the R code presented in
Code Block 1.

⬇

1## Code Block 1: Nonparametric Estimation of the ROC curve

2

3## values : string of values for the measure of interest

4## group : string of indicators for group membership

5compute_roc <- function(values, group, plot = "TRUE") {

6 sorted_values <- values[order(values)]

7 sorted_group <- group[order(values)]

8 cumsum_pos <- cumsum(sorted_group == 1)

9 cumsum_neg <- cumsum(sorted_group == 0)

10 n_pos <- sum(group == 1)

11 n_neg <- sum(group == 0)

12 TPR <- cumsum_pos/n_pos

13 FPR <- cumsum_neg/n_neg

14 TPR <- c(0, TPR, 1)

15 FPR <- c(0, FPR, 1)

16 if (plot == "TRUE") {

17 plot(FPR, TPR, type = "l", xlim = c(0,1), ylim = c(0,1),

18 frame.plot = FALSE, lwd = 2, ylab = "Sensitivity",

19 xlab = expression(paste("1 ", "\u2013", " Specificity")),

20 main = "Empirical ROC Curve")

21 }

22 return(list(FPR = FPR, TPR = TPR))

23}

24## output : ordered pairs to be plotted on ROC curve

Constructing point-wise confidence bands is deceptively simple. We are
oriented to think in terms of conditioning on the values marked on the x-axis
and characterizing uncertainty in corresponding values on the y-axis (e.g.,
forming point-wise confidence bands in linear regression). However, the values
marked on both the x\- and y-axes are the result of estimation at fixed values
of a cut-off point. The temptation may be, since the point-wise confidence
bands involve vertical slices, to form a confidence interval for a one-sample
proportion (e.g., a Wilson interval) for $\text{P}(Y_{1}\leq t)$—where $t$ is
the estimated quantile of of the reference group associated with specificity
$p$. However, this effectively treats the reference distribution as if fixed
and known in advance, and fails to account for duality in sampling variation.
Instead, a point-wise confidence band can be formed by way of the
nonparametric bootstrap. If $B$ denotes the number of bootstrap replicates
(sufficiently large), the procedure is as follows in the case where $n_{0}$
and $n_{1}$ are fixed features of the study design:

for $b$ in 1 to $B$:

  1. 1.

Obtain a full-size resample (with replacement) of data from the reference
group, $Y_{01}^{b},\dots,Y_{0n_{0}}^{b}$. Let $\mathbb{F}_{0b}^{-1}(p)$ denote
the empirical quantile function of the bootstrapped observations.

  2. 2.

Obtain a full-size resample (with replacement) of data from the comparator
group $Y_{11}^{b},\dots,Y_{1n_{1}}^{b}$. Let $\mathbb{F}_{1b}(t)$ denote the
empirical CDF of the bootstrapped observations.

  3. 3.

Let $\widehat{\text{ROC}}_{b}(p)=\mathbb{F}_{1b}(\mathbb{F}_{0b}^{-1}(p))$
denote the value of the empirical ROC curve at the value of $p$.

end

A 95% confidence band for $\text{ROC}(p)$ can be based, for instance, on the
2.5${}^{\text{th}}$ and 97.5${}^{\text{th}}$ percentiles of
$\\{\widehat{\text{ROC}}_{b}(p)\\}_{b=1}^{B}$. When $n_{0}$ and $n_{1}$ are
not fixed characteristics of the study design, a full (i.e., unconditional)
bootstrap can be conducted in place of the above procedure (Davison and
Hinkley, 1997).

The estimated AUC associated with the empirical ROC curve can be computed
using its connection to the Mann-Whitney U-statistic (Hanley and McNeil,
1982). In particular, if $R$ denotes the sum of the ranks (i.e., of the pooled
data of size $n$, and accounting for ties via averaging ranks) in either
group. The estimated empirical AUC is given as follows:

$\displaystyle\widehat{\text{AUC}}=\frac{1+|2(1+(n_{0}+1)/(2n_{1})-R/(n_{0}n_{1}))-1|}{2}.$

This formula is expressed to automatically follow the convention
$0.5\leq\text{AUC}\leq 1$. An R function is provided in Code Block 2 to
compute the empirically estimated AUC.

⬇

1## Code Block 2: Calculation of the Empirical AUC

2

3## values : string of values for the measure of interest

4## group : string of indicators for group membership

5compute_auc <- function(values, group) {

6 n0 <- length(y0)

7 n1 <- length(y1)

8 R <- sum(rank(values, ties.method = "average")[group == 0])

9 auc <- (1 + abs(2*(1 + (n0 + 1)/(2*n1) - R/(n0*n1)) - 1))/2

10 return(auc)

11}

12## output : numeric value of AUC

A test of the weak null hypothesis, $H_{0}:\text{AUC}=0.5$, can be conducted
using the Mann-Whitney test (e.g., wilcox.test() in R). The strong null
hypothesis, $H_{0}:\text{ROC}(p)=p$, can be tested nonparametrically using,
for example, the Kolmogorov-Smirnov test (e.g., ks.test() in R; Conover,
1971).

###  3.2 Parametric estimation

Suppose now that we are willing to posit a parametric form for the data, so
that $Y_{0}$ has quantile function $F_{0}^{-1}(p;\boldsymbol{\theta}_{0})$ and
$Y_{1}$ has CDF $F_{1}(t;\boldsymbol{\theta}_{1})$. If
$\widehat{\boldsymbol{\theta}}_{0}$ and $\widehat{\boldsymbol{\theta}}_{1}$
denote, say, maximum likelihood estimates (MLEs) of $\boldsymbol{\theta}_{0}$
and $\boldsymbol{\theta}_{1}$ (respectively), then the ROC curve can be
consistently estimated as
$\widehat{\text{ROC}}(p)=F_{1}(F_{0}^{-1}(p;\widehat{\boldsymbol{\theta}}_{0});\widehat{\boldsymbol{\theta}}_{1})$
by the invariance property of MLEs.

####  3.2.1 Exponential data

Consider the parametric form of the biexponential ROC model described in
Section 2.4, whereby $\widehat{\lambda}_{0}=1/\overline{Y}_{0}$ and
$\widehat{\lambda}_{1}=1/\overline{Y}_{1}$ denote the MLEs of $\lambda_{0}$
and $\lambda_{1}$. To form a point-wise confidence interval for
$\text{ROC}(p)$, we note the utility of the following variance-stabilizing
transformation:

$\displaystyle\sqrt{n_{0}}(\log(\widehat{\lambda}_{0})-\log(\lambda_{0}))\overset{d}{\longrightarrow}\mathcal{N}(0,1),$

so that, by independence of the observations,
$\text{Var}(\log(\widehat{\lambda}_{1})-\log(\widehat{\lambda}_{0}))\approx
n_{0}^{-1}+n_{1}^{-1}$ is an asymptotically valid approximation that lends
itself to the following form for a point-wise confidence interval for
$\text{ROC}(p)$ under the parametric biexponential model:

$\displaystyle
1-(1-p)^{\exp\left(\left(\log(\widehat{\lambda}_{1})-\log(\widehat{\lambda}_{0})\right)\pm
z_{1-\alpha/2}\sqrt{n_{0}^{-1}+n_{1}^{-1}}\right)}.$

Notably, the point estimates and confidence intervals do not depend upon the
specific values of $\widehat{\lambda}_{1}$ and $\widehat{\lambda}_{0}$ except
through the value of
$\widehat{\alpha}=\widehat{\lambda}_{1}/\widehat{\lambda}_{0}$, which is
intuitive as the ROC curve is invariant to re-scaling the data. The
biexponential ROC curve can reasonably be thought of as a one-parameter model.
The AUC can be estimated as
$\widehat{\text{AUC}}=1-(\widehat{\alpha}+1)^{-1}$. Under the exponential
model, a test of both the weak and the strong null can be implemented using
the likelihood ratio statistic,
$\Lambda=n\log(\widehat{\lambda})-n_{0}\log(\widehat{\lambda}_{0})-n_{1}\log(\widehat{\lambda}_{1})$,
where $\widehat{\lambda}$ denotes the MLE of the rate parameter in the
combined data. The likelihood ratio test statistic has an asymptotic
$\chi_{1}^{2}$ distribution under the strong null.

####  3.2.2 Normal data

Now, consider the parametric form of the binormal model described in Section
2.5, whereby $\widehat{\mu}_{0}=\overline{Y}_{0}$,
$\widehat{\mu}_{1}=\overline{Y}_{1}$,
$\widehat{\sigma}_{0}^{2}=n_{0}^{-1}\sum_{i=1}^{n_{0}}(Y_{0i}-\widehat{\mu}_{0})^{2}$,
and
$\widehat{\sigma}_{1}^{2}=n_{1}^{-1}\sum_{i=1}^{n_{1}}(Y_{1i}-\widehat{\mu}_{1})^{2}$;
in practice, we may use the sample variances (e.g.,
$S_{0}^{2}=n_{0}\widehat{\sigma}_{0}^{2}(n_{0}-1)^{-1}$) in place of the MLEs.
Following a similar line of logic as with the exponential model reveals the
following form for a point-wise confidence interval for $\text{ROC}(p)$ under
the binormal model:

$\displaystyle\Phi\left(\frac{(\widehat{\mu}_{0}-\widehat{\mu}_{1})+S_{0}\Phi^{-1}(p)\pm
t_{1-\alpha/2,\text{df}}\sqrt{S_{0}^{2}n_{0}^{-1}+S_{1}^{2}n_{1}^{-1}}}{S_{1}}\right),$

where df denotes the Welch-Satterthwaite approximation to degrees of freedom
(Welch, 1947). Similarly to the above example, the point estimates and
confidence intervals do not depend upon the specific values of the estimates
except through the values of
$\widehat{\delta}=\widehat{\mu}_{0}-\widehat{\mu}_{1}$ and $S_{0}/S_{1}$; the
binormal ROC curve can be conceptualized as a two-parameter model. The AUC can
be estimated numerically given the MLEs using, for example, the integrate()
function in R. A test of the weak null can be based on an unequal-variance
t-test of means. On the other hand, a test of the strong null can be based on
the likelihood ratio test, with
$\Lambda=n\log(\widehat{\sigma}^{2})-n_{0}\log(\widehat{\sigma}_{0}^{2})-n_{1}\log(\widehat{\sigma}_{1}^{2})$,
where $\widehat{\sigma}^{2}$ denotes the MLE of the variance in the combined
data. The likelihood ratio test statistic has an asymptotic $\chi_{2}^{2}$
distribution under the strong null (note, however, that the MLEs for
$\sigma_{0}^{2}$ and $\sigma_{1}^{2}$ may not be replaced by the corresponding
sample variances in the test statistic for this to hold).

###  3.3 Semiparametric estimation

Let $U_{ij}=I(Y_{1j}\leq Y_{0j})$ denote the indicator of the $j^{\text{th}}$
observation from the comparator group having a value no larger than the
$i^{\text{th}}$ observation from the reference group (there are a total of
$n_{0}\times n_{1}$ observations of the form $U_{ij}$). It has been previously
shown (Pepe, 2000) that $\text{E}[U_{ij}|F_{0}(Y_{0i})=p]=\text{ROC}(p)$. This
lends itself to the use of generalized linear models (suitable for binomial
data) to estimate an ROC curve directly, bypassing the need to impose an
explicit parametric structure.

####  3.3.1 The semiparametric biexponential model

Consider modeling the ROC curve using the biexponential form described in
Section 2.4 (which is implied by, but does not itself imply, exponentially
distributed data). Letting $V_{ij}=1-U_{ij}$, the corresponding semiparametric
biexponential ROC model is given by
$\log(\text{E}[V_{ij}|F_{0}(Y_{0i})=p])=\alpha\log(1-p)$. Note the absence of
an intercept in this single-parameter family. In practice, the value of $p$ is
not known but for each $V_{ij}$ is estimated as
$\widehat{p}_{ij}=\mathbb{F}_{0}(Y_{0i})$. The associated estimating equations
for $\alpha$ take the following form:

$\displaystyle\mathbb{G}(\alpha)=\sum_{i=1}^{n_{0}}\sum_{j=1}^{n_{1}}\frac{\log(1-\widehat{p}_{ij})}{1-(1-\widehat{p}_{ij})^{\alpha}}(V_{ij}-(1-\widehat{p}_{ij})^{\alpha})=0.$

Observations with $\widehat{p}_{ij}=1$ (so that
$\log(1-\widehat{p}_{ij})=-\infty$) may be excluded for the estimating
equation to be well defined; this will occur for the pairs that involve the
maximum order statistic of the $Y_{0i}$’s, so that the estimating equation
relies on $(n_{0}-1)\times n_{1}$ of the observations $U_{ij}$. An R function
is provided in Code Block 3 to estimate and implement semiparametric
biexponential estimation of the ROC curve.

⬇

1## Code Block 3: Biexponential Semiparametric Estimation of the ROC curve

2

3## values : string of values for the measure of interest

4## group : string of indicators for group membership

5ROC.GLM.E <- function(group, values) {

6 y0 <- values[group == 0]

7 y1 <- values[group == 1]

8 n0 <- length(y0)

9 n1 <- length(y1)

10 Pij <- matrix(colMeans(outer(y0, y0, "<=")), nrow = n0, ncol = n1)

11 P <- matrix(Pij, ncol = 1)

12 Q <- cbind(log(1 - P)[P != 1])

13 V <- 1 - as.numeric(t(outer(y1, y0, "<=")))[P != 1]

14 zz <- glm.fit(Q, V, family = binomial(link = "log"))

15 zz$coefficients

16}

17## output : estimated coefficient from semiparametric biexponential ROC

This estimating equation possesses no closed-form solution, but can be solved
numerically using a Gauss-Newton algorithm (Mittelhammer et al., 2000). Due to
the correlation of the $U_{ij}$’s, however, a bootstrap approach is typically
employed in order to obtain asymptotically valid confidence intervals for
$\text{ROC}(p)$. As in Section 3.2.1, the AUC can be estimated as
$\widehat{\text{AUC}}=1-(\widehat{\alpha}+1)^{-1}$. Both the weak and strong
null can be tested using the bootstrap samples under an asymptotic normality
assumption on $\widehat{\alpha}$.

####  3.3.2 The semiparametric binormal model

Now, consider modeling the ROC curve using the binormal form (which is implied
by, but does not itself imply, normally distributed data). The corresponding
semiparametric binormal ROC model is given by
$\Phi^{-1}(\text{E}[U_{ij}|F_{0}(Y_{0i})=p])=\beta_{0}+\beta_{1}\Phi^{-1}(p)$.
The associated estimating equations for
$\boldsymbol{\beta}=(\beta_{0},\beta_{1})^{\top}$ take the following form:

$\displaystyle\mathbb{G}(\boldsymbol{\beta})=\sum_{i=1}^{n_{0}}\sum_{j=1}^{n_{1}}\boldsymbol{X}_{ij}\frac{\phi(z_{ij})}{\Phi(z_{ij})(1-\Phi(z_{ij}))}(U_{ij}-\Phi(z_{ij}))=\textbf{0},$

where $z_{ij}=\beta_{0}+\beta_{1}\Phi^{-1}(\widehat{p}_{ij})$, and
$\boldsymbol{X}_{ij}=(1,\Phi^{-1}(\widehat{p}_{ij}))^{\top}$. For reasons
analogous to those described in Section 3.3.1, observations with
$\widehat{p}_{ij}=1$ may be excluded for the estimating equations to be well
defined. These estimating equations, too, possess no closed-form solutions,
and a nonparametric bootstrap is typically employed to conduct inference. As
in Section 3.2.2, the AUC can be estimated numerically given the MLEs using,
for example, the integrate() function in R. The weak and/or strong null can be
tested by conducting the appropriate tests on $\boldsymbol{\beta}$, also
previously described. An R function is provided in Code Block 4 to estimate
implement semiparametric binormal estimation of the ROC curve.

⬇

1## Code Block 4: Binormal Semiparametric Estimation of the ROC curve

2

3## values : string of values for the measure of interest

4## group : string of indicators for group membership

5ROC.GLM.N <- function(group, values) {

6 y0 <- values[group == 0]

7 y1 <- values[group == 1]

8 n0 <- length(y0)

9 n1 <- length(y1)

10 Pij <- matrix(colMeans(outer(y0, y0, "<=")), nrow = n0, ncol = n1)

11 P <- matrix(Pij, ncol = 1)

12 Q <- cbind(1, qnorm(P)[P != 1])

13 U <- as.numeric(t(outer(y1, y0, "<=")))[P != 1]

14 zz <- glm.fit(Q, U, family = binomial(link = "probit"))

15 zz$coefficients

16}

17## output : estimated coefficient from semiparametric binormal ROC

###  3.4 Other considerations for characterizing uncertainty

We acknowledged at the beginning of Section 3 that our outlined estimation
procedures assume that it is of interest to characterize uncertainty in
estimating the TPR at specific values of the FPR. It could be of equal
interest to characterize uncertainty of the FPR at specific values of the TPR.
We do not consider this nuance in extraordinary depth because estimation of
this form can be accomplished by a convention change as described in Section
2.2 and illustrated in Figure 2, so that the fundamental mechanics of the
procedure are unchanged. We acknowledge that the setting in which there is no
clear ordering of groups, nor a natural way to define higher values of the
measure of interest as “better” or “worse” poses a unique challenge. Others
have considered intuitively combining information from bi-directional
confidence regions, including radial sweeps, though there is little
theoretical or empirical justification for these methods (Horváth et al.,
2008; Macskassy et al., 2003).

##  4 Simulation-based comparison of estimation methods

The goal of this section is to illustrate the bias-variance tradeoff
associated with various approaches in finite samples. We utilize two general
data generating mechanisms (DGMs): the first is an exponential DGM of the form
$Y_{0}\sim\text{Exponential}(\lambda_{0}=1)$ and
$Y_{1}\sim\text{Exponential}(\lambda_{1}=4)$, and the second is a normal DGM
$Y_{0}\sim\mathcal{N}(\mu_{0}=5.5,\sigma_{0}^{2}=1)$ and
$Y_{1}\sim\mathcal{N}(\mu_{1}=4.0,\sigma_{1}^{2}=1)$. We further consider two
pairs of sample sizes: $n_{0}=n_{1}=30$ (low) and $n_{0}=n_{1}=60$ (high). In
Section 4.1, we seek to confirm the inadequacy of the Wilson interval as
compared to the nonparametric bootstrap for the empirical ROC curve. In
Section 4.2, we generate data from the exponential DGM and compare the finite
sample properties of the various methods described in Section 3. In Section
4.3, we do the same under the normal DGM.

To evaluate the finite sample properties of these methods in various
scenarios, we compare the average estimated value of the ROC curve to the true
value, and determine the average width and coverage associated with 95%
confidence intervals. For each simulation scenario presented, we utilize
$M=1000$ simulation replicates and, where applicable, $B=3000$ bootstrap
replicates. All simulations were conducted in R (R Core Team, 2020).

Table 1: Results of a comparison between the Wilson interval and the quantile-based nonparametric bootstrap interval for the empirical ROC curve. $\widehat{\text{ROC}}(p)$ denotes the average estimate of the ROC curve at the specified value of $p$. $\Delta$ denotes the average confidence interval width and CP denotes coverage probability. “-W” denotes the Wilson interval and “-B” denotes the bootstrap interval. $n_{0}=n_{1}$ | $p$ | $\text{ROC}(p)$ | $\widehat{\text{ROC}}(p)$ |  $\Delta$-W | CP-W |  $\Delta$-B | CP-B
---|---|---|---|---|---|---|---
30 | 0.0027 | 0.1 | 0.31 | 0.29 | 0.31 | 0.51 | 0.30
30 | 0.0215 | 0.3 | 0.31 | 0.29 | 0.59 | 0.51 | 0.84
30 | 0.0670 | 0.5 | 0.55 | 0.32 | 0.73 | 0.59 | 0.96
30 | 0.1650 | 0.7 | 0.67 | 0.31 | 0.75 | 0.51 | 0.97
30 | 0.4140 | 0.9 | 0.90 | 0.22 | 0.94 | 0.27 | 0.96
60 | 0.0027 | 0.1 | 0.22 | 0.19 | 0.37 | 0.38 | 0.40
60 | 0.0215 | 0.3 | 0.33 | 0.22 | 0.60 | 0.45 | 0.91
60 | 0.0670 | 0.5 | 0.52 | 0.24 | 0.72 | 0.45 | 0.96
60 | 0.1650 | 0.7 | 0.69 | 0.22 | 0.78 | 0.37 | 0.97
60 | 0.4140 | 0.9 | 0.89 | 0.15 | 0.89 | 0.20 | 0.98

###  4.1 The Wilson interval vs. the bootstrap for the empirical ROC

In this simulation study, we generate data from the normal DGM in order to
compare the Wilson interval to the quantile-based nonparametric bootstrap-
based interval associated with empirical ROC curve. We consider both the low
and high sample size cases. The results of this simulation are shown in Table
1, which depicts the results at the specified values of $p$ such that
$\text{ROC}(p)=0.1,0.3,0.5,0.7$, and $0.9$; the justification for considering
values of $p$ in this range that it encompasses a wide range of ordered pairs
on the ROC curve that approach the boundary of the curve.

We observe a marked degree of finite-sample bias associated with the empirical
ROC curve for very small values of $p$. The Wilson intervals uniformly have
lower width as compared to those associated with the empirical bootstrap,
which is not surprising given that it treats the distribution of the reference
group as if fixed. With the exception of the lower values of $p$, the
empirical bootstrap appears to possess adequate coverage. On the other hand,
the Wilson interval generally does not possess proper coverage across the
board. The intuition for the general pattern of under-coverage is that the
Wilson interval fails to account for variation in estimating $p$, the
specificity associated with a particular cut-off point. We no longer consider
the Wilson interval as a competing method for the remainder of this
manuscript.

###  4.2 Comparisons under the exponential DGM

In this section, we generated data under the exponential DGM, and considered
four models: (1) a parametric binormal model, (2) a parametric exponential
model, (3) a semiparametric biexponential model, and (4) a semiparametric
binormal model. We consider both the low and high sample size cases. The
results of this simulation are depicted in Figure 4. As expected, the
parametric binormal ROC curve demonstrates a bias and a lack of coverage that
are not overcome in a larger sample. The finite sample behavior is more
favorable for the competing methods. The validity of the parametric and
semiparametric biexponential models is not surprising (Figures 4(A) and 4(D)),
nor is the apparent greater degree of efficiency associated with the
parametric method (Figures 4(B) and 4(E)). It is noteworthy that the
semiparametric binormal model exhibits so little bias given that it is not
correctly specified, though we have pointed out that it offers greater
flexibility as compared to the biexponential family. Given the additional
model complexity of the binormal model, the finding that the confidence
intervals are generally wider as compared to those of the respective
biexponential models is not unexpected (Figures 4(B) and 4(E)).

Figure 4: Results of a simulation study under the exponential DGM. Panels (A)
through (C) correspond to the lower sample size ($n_{0}=n_{1}=30$) and (D)
through (F) correspond to the higher sample size ($n_{0}=n_{1}=60$). Panels
(A) and (D) depict the average estimated value of the ROC curve across
methods, with the truth depicted as a reference. Panels (B) and (E) depict the
average confidence interval width across $p$, the FPR. Panels (C) and (F)
present coverage across methods, with a reference line of $y=0.95$.

Figure 5: Results of a simulation study under the normal DGM. Panels (A)
through (C) correspond to the lower sample size ($n_{0}=n_{1}=30$) and (D)
through (F) correspond to the higher sample size ($n_{0}=n_{1}=60$). Panels
(A) and (D) depict the average estimated value of the ROC curve across
methods, with the truth depicted as a reference. Panels (B) and (E) depict the
average confidence interval width across $p$, the FPR. Panels (C) and (F)
present coverage across methods, with a reference line of $y=0.95$.

###  4.3 Comparisons under the normal DGM

We conduct a simulation study that mirrors the setup described in Section 4.2,
but instead generating data under the normal DGM. The results of this
simulation are depicted in Figure 5. The parametric biexponential ROC curve
performs poorly with respect to bias and coverage. This can be heuristically
explained by the fact that the parametric biexponential model uses information
from the mean ratio, despite the fact that the data are parameterized by the
(scaled) mean difference and the ratio of standard deviations. Since the mean
ratio of $\mu_{1}/\mu_{0}=5.5/4.0$ is close to one, the parametric
biexponential model does not “perceive,” so to speak, a meaningful difference
in stochastic ordering under the (albeit incorrect) exponential assumption.
The finite sample behavior is more favorable for the competing methods,
although the parametric binormal model suffers a notable loss of coverage near
the boundaries. The semiparametric biexponential model is not free of bias,
although its relatively good performance is likely related to the fact that
the variances were generated to be equal in this data generating mechanism
(that is to say that the biexponential form is strictly concave, whereas
unequal variances in a normal model introduces infection points into the ROC
curve). This study is consistent with the relative efficiency associated with
the correct parametric specification of the data (Figures 5(B) and 5(C)).

##  5 Application to solid organ transplants

In this section, we apply several estimation methods to data from a study that
sought to investigate SARS-CoV-2 vaccination-induced immunogenicity.

###  5.1 Study cohort and measures

Eligible individuals enrolled in this study included SOT recipients
($n_{1}=54$) and healthy controls (HCs; $n_{0}=26$) over 55 years old who were
scheduled to receive a two-dose regimen of BNT162b2 as part of routine care at
Vanderbilt University Medical Center (Nashville, TN). Specific transplantation
groups included prior heart (11%), kidney (41%), liver (37%), lung (7%), and
kidney/liver (4%) transplant recipients. Immunoglobulin G to SARS-CoV-2 spike
extracellular domain (ECD) was evaluated by enzyme-linked immunosorbent assay.
Three SOT recients were excluded from our analysis due to serological evidence
of an ongoing or prior SARS-CoV-2 infection. Further details pertaining to
recruitment, enrollment, and prior analysis are detailed by Yanis et al.
(2022).

###  5.2 Empirical analysis

Our illustrative analysis specifically focuses on ECD response to the second
vaccination dose (henceforth denoted $\text{ECD}_{2}$). Higher values of
$\text{ECD}_{2}$ serve as a correlate of higher immunity (Oppenheimer et al.,
2023), and we anticipate a priori that SOT recipients, due to
immunocompromised status, tend to have lower $\text{ECD}_{2}$ values as
compared to HCs. Figure 6(A) presents histograms of $\text{ECD}_{2}$,
stratified by group, and Figure 6(B) presents an empirical ROC curve with a
95% confidence band as estimated by the nonparametric bootstrap procedure
described in Section 3.1. The degree of separation between the distribution of
groups is apparent from examining the histograms. The empirical ROC curve,
particularly when considering the associated confidence bands, offers
additional context regarding our certainty regarding how well-separated the
groups are in their $\text{ECD}_{2}$ response.

Figure 6: Panel (A) depicts histograms of $\text{ECD}_{2}$. Panel (B) depicts
the empirical ROC curve with a confidence band. Panels (C) through (F) depict
various ROC estimates, including point estimates (dark gray solid line) and
confidence bands (light gray); the empirical ROC curve is also depicted in
solid blue as a reference. Panel (C): parametric biexponential. Panel (D):
parametric binormal. Panel (E): semiparametric biexponential method. Panel (F)
semiparametric binormal.

###  5.3 Parametric estimation

The estimated ROC curve under the parametric biexponential model is depicted
in Figure 6(C), and the estimated curve under the parametric binormal model is
depicted in Figure 6(D). For the parametric biexponential model, we estimate
$\widehat{\alpha}=2.72$. For the binormal model, we estimate
$\widehat{\beta}_{0}=1.70$ and $\widehat{\beta}_{1}=0.52$. Of note, the
distribution of $\text{ECD}_{2}$ among SOT recipients exhibits right-skewness,
whereas the distribution among HCs more closely resembles the symmetry of a
normal distribution. That is to say that results from both parametric methods
are likely biased, which is reflected in part by the discrepancy between the
point estimates for the estimated ROC curves and the empirical curve; this
discrepancy is particularly notable for the parametric biexponential model.

###  5.4 Semiparametric estimation

We now estimate the ROC curve using the correspoinding semiparametric methods.
The results of the semiparametric biexponential model are shown in Figure 6(E)
and those of the semiparametric binormal model are shown in Figure 6(F). For
the biexponential model, we estimate $\widehat{\alpha}=4.48$, a marked
difference from the point estimate obtained using parametric methods. Recall
that $\alpha$ does not possess the interpretation of a mean ratio when the
biexponential curve is estimated in a semiparametric fashion. We see from
comparing Figure 6(E) to 6(C) that the semiparametric biexponential
model—while still notably different from the empirical curve—is better
calibrated to the empirical curve across the broader range of $p$.

For the semiparametric binormal model, we estimate $\widehat{\beta}_{0}=1.35$
and $\widehat{\beta}_{1}=0.49$. A particularly noteworthy result from the
semiparametric binormal fit is that the empirical ROC curve is almost entirely
captured within its confidence band; this is not true for any of the other
three fits. The curve associated with the parametric model possesses a similar
shape, but likely overstates certainty. This example is again consistent with
the flexibility associated with the semiparametric binormal curve, especially
as compared to the more parsimonious biexponential model.

One key comparison is also between the semiparametric biexponential model
(Figure 6(E)) and the parametric binormal model (Figure 6(D)). The confidence
bands are considerably wider under the former model, despite the fact that it
has one fewer parameter. This might be a more surprising result if had the two
models been nested, but this particular comparison is a stark example of the
fact relative efficiency is driven by more than simply the relative number of
parameters possessed by the model. One possible source of this discrepancy is
a “ceiling effect” in the variability given the boundedness of the ROC
function.

##  6 Discussion

In this manuscript, we have described the ROC curve and highlighted aspects of
its geometry that we believe may be underappreciated. We have outlined and
offered software code for a number of competing nonparametric, parametric, and
semiparametric estimation approaches. Simulation studies revealed, as
expected, a notable trade-off between efficiency and bias associated with
structure.

We acknowledge that the ROC curve may, for many, serve as a visual tool to
complement a primary analysis and that explicitly conducting inference on the
value of an ROC curve at a specified value of $p$ may not always reflect the
main scientific goal. However, the width of confidence bands offer visual
clarity to uncertainty and speak in some respects to how trustworthiness of
the findings regarding diagnostic utility or stochastic ordering. Our
simulations and application revealed non-trivial differences in both bias and
efficiency across approaches. To that end, knowledge regarding the bias and
efficiency of the approach used to generate an ROC curve is relevant even if
confidence intervals for the curve are not specifically generated in a given
researcher’s implementation.

We note that loss of coverage is a commonly encountered feature near the
boundary of the ROC curve. This may not always be of major concern in the
sense that the extreme ends of the curve signify either the setting of zero-
specificity or zero-sensitivity, reflecting cut-off points that may not be
considered viable candidates. However, further investigation may bring clarity
to the downstream consequences on coverage associated with confidence
intervals for other summary measures (say, the AUC).

Our finding that the binormal estimation techniques—parametric and
semiparametric—appear to offer a certain degree of robustness, is consistent
with previous findings (Hanley, 1988; Devlin et al., 2013; Bandos et al.,
2017). Heuristically, the space of monotone increasing functions from $[0,1]$
to $[0,1]$ likely does not need to be indexed by more than two parameters in
order to cover a wide range of real-world scenarios. Our work contributes to
the literature by underscoring the inadequacy of the single-parameter
biexponential model. The semiparametric variant of the biexponential model has
previously been compared to other methods (Devlin et al., 2013), though in the
context of estimating $\alpha$ rather than $\text{ROC}(p)$ at various values
of $p$. Note that the findings of this work do not necessarily support the
uptake of the semiparametric biexponential model in the real world.

In our application of ROC methods to vaccine-associated immunogenicity in
Section 5, there is a notable degree of heterogeneity in the comparator group
that is not captured by our illustrative analysis. For example, liver
transplant recipients undergo the lowest degree of immunosuppression as
compared to other organ transplant types (Pilch et al., 2021). Over the years,
there has been research on regression of the ROC curve, whereby one seeks to
evaluate how the nature of an ROC curve varies across population strata (Pepe,
1998; Alonzo and Pepe, 2002). The techniques used to accomplish this are a
natural extension of those presented in Section 3.3. This area has also been
explored independent of the setting of diagnostic markers (Illenberger et al.,
2022).

We acknowledge that the suite of methods compared in this manuscript are not
exhaustive. For example, Wan and Zhang (2007), Kim and Zeng (2013), and
Ghebremichael and Michael (2024) consider alternative semiparametric methods
for ROC estimation based on accelerated regression models and/or kernel
density estimation. The goal of this work was not to serve as an exhaustive
literature review of all existing ROC estimation techniques, but instead to
provide insights into ROC analysis based on a selection of estimation
approaches that are straightforward to implement in the real world. We
hypothesize that the patterns of behavior for other semiparametric approaches
would in many ways mirror those of the approaches we considered, although
further study would be needed to confirm this.

An additional trade-off that we did not consider in this work is that of
computational efficiency. The bootstrap is a computationally taxing procedure,
particularly when one needs to employ enough replicates to justify quantile-
based confidence intervals. Alonzo and Pepe consider coarsening the grid of
FPR values at which to evaluate the TPR; more variability is introduced when
failing to leverage the full distribution of the reference group (Alonzo and
Pepe, 2002). In the modern era of improving computing resources, this trade-
off may seem to wane in relevance; however, we are also in a modern era of
access to increasing larger data sets, so considerations regarding the trade-
off of computational burden should not be totally ignored.

We must offer our full acknowledgement that ROC methods are not without
limitations. Some of the most meaningful criticisms of the methodology include
that the ROC curve gives consideration (both visually and numerically via the
AUC) to the non-viable region $[0.5,1]\times[0,0.5]$ in which both the
specificity and sensitivity are seriously lacking, and that the ROC curve
offers no meaningful insights into the positive predictive value and negative
predictive value (Lobo et al., 2008). These challenges have not gone
completely unaddressed in the literature. For example, weighted AUC methods
have been proposed to more heavily consider viable regions (Li and Fine,
2010).

There are a number of important future directions for this work.
Considerations regarding study design, matching, and repeated measures remain
under-explored in this space. However, if the trends of interest in measures
of stochastic ordering continue, we will soon identify unmet needs for
continued expansions and extensions of ROC methodology to accommodate the
nature of real-world data collection.

Acknowledgments

The Vanderbilt Institute for Clinical and Translational Research (VICTR) is
funded by the National Center for Advancing Translational Sciences (NCATS)
Clinical Translational Science Award (CTSA) Program, Award Number
5UL1TR002243-03. The content is solely the responsibility of the authors and
does not necessarily represent the official views of the NIH.

## References

  * Acion et al. (2006) L. Acion, J. J. Peterson, S. Temple, and S. Arndt.  Probabilistic index: An intuitive non-parametric approach to measuring the size of treatment effects.  _Statistics in Medicine_ , 25:591–602, 2006.
  * Alonzo and Pepe (2002) T.A. Alonzo and M.S. Pepe.  Distribution-free ROC analysis using binary regression techniques.  _Biostatistics_ , 3:421–432, 2002.
  * Bandos et al. (2017) A.I. Bandos, B. Guo, and D. Gur.  Estimating the area under ROC curve when the fitted binormal curves demonstrate improper shape.  _Academic Radiology_ , 24:209–219, 2017.
  * Barry (2001) M.J. Barry.  Clinical practice. Prostate-specific-antigen testing for early diagnosis of prostate cancer.  _New England Journal of Medicine_ , 344:1373–1377, 2001\.
  * Baumann et al. (2020) B.C. Baumann, N. Mitra, J.G. Harton, Y. Xiao, A.P. Wojcieszynski, P.E. Gabriel, H. Zhong, H. Geng, A. Doucette, J. Wei, P.J. O’Dwyer, J.E. Bekelman, and J.M. Metz.  Comparative effectiveness of proton vs photon therapy as part of concurrent chemoradiotherapy for locally advanced cancer.  _JAMA Oncology_ , 6:237–246, 2020.
  * Cavero-Redondo et al. (2017) I. Cavero-Redondo, B. Peleteiro, and V. Martínez-Vizcaíno.  Glycated haemoglobin A1c as a risk factor of cardiovascular outcomes and all-cause mortality in diabetic and non-diabetic populations: A systematic review and meta-analysis.  _BMJ Open_ , 7:e015949, 2017.
  * Conover (1971) W.J. Conover.  _Practical Nonparametric Statistics_.  John Wiley & Sons, New York, 1971.
  * Davison and Hinkley (1997) A. C. Davison and D. V. Hinkley.  _Bootstrap Methods and Their Application_.  Cambridge University Press, Cambridge, 1997.
  * Devlin et al. (2013) S.M. Devlin, E.G. Thomas, and S.S. Emerson.  Robustness of approaches to ROC curve modeling under misspecification of the underlying probability model.  _Communications in Statistics – Theory and Methods_ , 44:3655–3664, 2013.
  * Doitsh et al. (2014) G. Doitsh, N.L. Galloway, X. Geng, Z. Yang, K.M. Monroe, O. Zepeda, P.W. Hunt, H. Hatano, S. Sowinski, I. Muñoz Arias, and W.C. Greene.  Cell death by pyroptosis drives CD4 T-cell depletion in HIV-1 infection.  _Nature_ , 505:509–514, 2014.
  * Fawcett (2006) T. Fawcett.  An introduction to ROC analysis.  _Pattern Recognition Letters_ , 27:861–874, 2006.
  * Freedman (2009) D.A. Freedman.  _Statistical Models: Theory and Practice_.  Cambridge University Press, New York, 2009.
  * Ghebremichael and Michael (2024) M. Ghebremichael and H. Michael.  Comparison of the binormal and lehman receiver operating characteristic curves.  _Communications in statistics - Simulation and Computation_ , 53:772–785, 2024.
  * Green and Swets (1966) D.M. Green and J.A. Swets.  _Signal Detection Theory and Psychophysics_.  John Wiley & Sons, Inc, New York, 1966.
  * Hanley (1988) J.A. Hanley.  The robustness of the “binormal”’ assumptions used in fitting ROC curves.  _Medical Decision Making_ , 8:197–203, 1988.
  * Hanley and McNeil (1982) J.A. Hanley and B.J. McNeil.  The meaning and use of the area under a receiver operating characteristic (ROC) curve.  _Radiology_ , 143:29–36, 1982.
  * Hanley and McNeil (1983) J.A. Hanley and B.J. McNeil.  A method of comparing the areas under receiver operating characteristic curves derived from the same cases.  _Radiology_ , 148:839–843, 1983.
  * Hart (2001) A. Hart.  Mann-Whitney test is not just a test of medians: Differences in spread can be important.  _BMJ_ , 323:391–393, 2001.
  * Horváth et al. (2008) T. Horváth, Z. Horváth, and W. Zhou.  Confidence bands for ROC curves.  _Journal of Statistical Planning and Inference_ , 138:1894–1904, 2008.
  * Illenberger et al. (2022) N. Illenberger, N. Mitra, and A.J. Spieker.  A regression framework for a probabilistic measure of cost-effectiveness.  _Health Economics_ , 31:1438–1451, 2022.
  * Johnson et al. (2016) M.C. Johnson, E.R. Pierson, A.J. Spieker, A.S. Nielsen, S. Posso, M. Kita, J.H. Buckner, and J.M. Goverman.  Distinct T cell signatures define subsets of patients with multiple sclerosis.  _Neurology: Neuroimmunology & Neuroinflammation_, 3:e278, 2016.
  * Kim and Zeng (1998) E. Kim and D. Zeng.  Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data.  _Statistics in Medicine_ , 17:1033–1053, 1998.
  * Kim and Zeng (2013) E. Kim and D. Zeng.  Semiparametric ROC analysis using accelerated regression models.  _Statistica Sinica_ , 23:829–851, 2013.
  * Leblhuber et al. (1988) F. Leblhuber, F. Reisecker, H. Boehm-Jurkovic, A. Witzmann, and E. Deisenhammer.  Diagnostic value of different electrophysiologic tests in cervical disk prolapse.  _Neurology_ , 38:1879–1881, 1988.
  * Li and Fine (2010) J. Li and J.P. Fine.  Weighted area under the receiver operating characteristic curve and its application to gene selection.  _Journal of the Royal Statistical Society, Series C – Applied Statistics_ , 59:673–692, 2010.
  * Lobo et al. (2008) J.M. Lobo, A. Jiménez-Valverde, and R. Real.  AUC: A misleading measure of the performance of predictive distribution models.  _Global Ecology and Biogeography_ , 17:145–151, 2008.
  * Macskassy et al. (2003) S. Macskassy, F. Provost, and M.L. Littman.  Confidence bands for ROC curves.  _NYU Working Paper No. 2451/14153_ , page Available at SSRN: https://ssrn.com/abstract=1282995, 2003.
  * Metz (1986) C.E. Metz.  ROC methodology in radiologic imaging.  _Investigative Radiology_ , 21:720–733, 1986.
  * Mittelhammer et al. (2000) R. C. Mittelhammer, D. J. Miller, and G.G. Judge.  _Econometric Foundations_.  Cambridge University Press, Cambridge, 2000.
  * Oppenheimer et al. (2023) H. Oppenheimer, S. Zismanov, A. Kuzmina, L.M. Friedman, S. Trifkovic, D. Brice, L. Chun-Yang, L. Cohen-Lavi, Y. Shemer-Avni, M. Cohen-Lahav, D. Amichay, A. Keren-Naus, O. Voloshin, G. Weber, R. Najjar-Debbiny, B. Chazan, M.A. McGargill, R. Webby, M. Chowers, L. Novack, V. Novack, R. Taube, L. Nesher, and Weinstein O.  Correlates of protection for booster doses of the SARS-CoV-2 vaccine BNT162b2.  _Nature Communications_ , 14:4575, 2023.
  * Pepe (1998) M.S. Pepe.  Three approaches to regression analysis of receiver operating characteristic curves for continuous test results.  _Biometrics_ , 54:124–135, 1998.
  * Pepe (2000) M.S. Pepe.  An interpretation for the ROC curve and inference using GLM procedures.  _Biometrics_ , 56:352–359, 2000.
  * Pepe (2003) M.S. Pepe.  _The Statistical Evaluation of Medical Tests for Classification and Prediction_.  Oxford University Press, New York, 2003.
  * Pilch et al. (2021) N.A. Pilch, L.J. Bowman, and D.J. Taber.  Immunosuppression trends in solid organ transplantation: The future of individualization, monitoring, and management.  _Pharmacotherapy_ , 41:119–131, 2021.
  * R Core Team (2020) R Core Team.  _R: A Language and Environment for Statistical Computing_.  Vienna, 2020.  URL https://www.R-project.org/.
  * Spieker et al. (2013) A.J. Spieker, P. Narayanaswami, L. Fleming, J.C. Keel, S.C Muzin, and S.B. Rutkove.  Electrical impedance myography in the diagnosis of radiculopathy.  _Muscle & Nerve_, 48:800–805, 2013.
  * Spieker et al. (2021) A.J. Spieker, N. Illenberger, J.A. Roy, and N. Mitra.  Net benefit separation and the determination curve: A probabilistic framework for cost-effectiveness estimation.  _Statistical Methods in Medical Research_ , 30:1306–1319, 2021.
  * Thakur and Paramanik (2009) M.K. Thakur and V. Paramanik.  Role of steroid hormone coregulators in health and disease.  _Hormone Research_ , 71:194–200, 2009.
  * Thompson and Zucchini (1989) M.L. Thompson and W. Zucchini.  On the statistical analysis of ROC curves.  _Statistics in Medicine_ , 8:1277–1290, 1989.
  * Wan and Zhang (2007) S. Wan and B. Zhang.  Smooth semiparametric receiver operating characteristic curves for continuous diagnostic tests.  _Statistics in Medicine_ , 26:2565–2586, 2007.
  * Welch (1947) B.L. Welch.  The generalization of “Student’s” problem when several different population variances are involved.  _Biometrika_ , 34:28–35, 1947.
  * Xu et al. (2024) J. Xu, W.B. Zhang, W.F. Deng, B.L. He, and T.Y. Qi.  Comparative analysis of the diagnostic efficiency of the four thyroid imaging reporting and data systems.  _WFUMB Ultrasound Open_ , 2:100037, 2024.
  * Yanis et al. (2022) A. Yanis, Z. Haddadin, AJ. Spieker, D. Waqfi, R. Talj, D.A. Rankin, L. Thomas, K.A. Birdwell, L. Ezzell, M. Blair, J. Eason, R. Varjabedian, C.M. Warren, C.H. Nochowicz, E.C. Olson, J.D. Simmons, S. Yoder, M. Guy, I. Thomsen, J.D. Chappell, S.A. Kalams, and N.B. Halasa.  Humoral and cellular immune responses to the SARS-CoV-2 BNT162b2 vaccine in a cohort of solid organ transplant recipients and healthy controls.  _Transplant Infectious Disease_ , 24:e13772, 2022.
  * Zhang et al. (2019) Z. Zhang, S. Ma, C. Shen, and C. Liu.  Estimating Mann–Whitney-type causal effects.  _International Statistical Review_ , 87:514–530, 2019.
  * Zhou et al. (2024) X. Zhou, Y. Wang, M. Bao, Y. Chu, R. Liu, Q. Chen, and Y. Lin.  Advanced detection of cervical cancer biomarkers using engineered filamentous phage nanofibers.  _Applied Microbiology and Biotechnology_ , 108:221, 2024\.
  * Zhu et al. (2024) T. Zhu, D. Li, J. Qiao, Q. Li, Y. Xu, B. Ge, and L. Xia.  Accuracy of subtraction fractional flow reserve with computed tomography in identifying early revascularization in patients with coronary artery disease.  _Scandinavian Cardiovascular Journal_ , 58:2373082, 2024\.