AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systems

Clara Punzi

Scuola Normale Superiore
Piazza dei Cavalieri, 7

Roberto Pellungrini

Scuola Normale Superiore
Piazza dei Cavalieri, 7

Mattia Setzu

University of Pisa
Largo Bruno Pontecorvo, 3

Fosca Giannotti

Scuola Normale Superiore
Piazza dei Cavalieri, 7

Dino Pedreschi

University of Pisa
Largo Bruno Pontecorvo, 3

Everyday we increasingly rely on machine learning models to automate and support high-stake tasks and decisions. This growing presence means that humans are now constantly interacting with machine learning-based systems, training and using models everyday. Several different techniques in computer science literature account for the human interaction with machine learning systems, but their classification is sparse and the goals varied. This survey proposes a taxonomy of Hybrid Decision Making Systems, providing both a conceptual and technical framework for understanding how current computer science literature models interaction between humans and machines.

<concept_desc>Computing methodologies Learning paradigms</concept_desc>
<concept_desc>Human-centered computing HCI theory, concepts and models</concept_desc>

[500]Computing methodologies Learning paradigms
[300]Human-centered computing HCI theory, concepts and models

§ INTRODUCTION

Advances in Artificial Intelligence (AI) systems have reached a widespread use in a variety of domains and applications.
Their effectiveness has grown exponentially over the past decades, and now AI systems are able to achieve surprising performances on tasks previously thought to be out of reach for artificial systems.
Often acting as standalone

systems with little to no human control, such models are particularly susceptible to distrust <cit.>, misuse and disuse <cit.> by human actors,

with the risk of incurring in undesired patterns of user behavior, such as algorithmic aversion <cit.> and overreliance <cit.>.

At the same time, AI models have found success in several high-stakes domains, such as medicine <cit.>, finance <cit.> and law <cit.>.
Here, they often act as decision-support systems providing human domain experts with advice, rather than fully replacing them.
These “collaborative” decision-making systems are plagued by the same trust, misuse, and disuse issues of stand-alone AI systems, on top of novel and unique issues.
Collaborative systems have to define an effective communication channel between the AI system and its human decision makers, and the capability to properly integrate the two.

Integrating coarse-grained and fine-grained control of AI systems has proven to be a non-trivial task in itself, indicating that steering AI systems to desired complex behaviors <cit.>, preventing them from learning spurious and undesired correlations <cit.>, and ultimately aligning them with human values <cit.> is far from a solved problem.

Whether AI systems are standalone or collaborative, we find
an inherent mismatch between the two parties: humans and AI systems have fundamentally different ways of understanding and representing the world. Therefore, human and AI reasoning are fundamentally different, and leveraging both their complementary strengths could greatly benefit decision-making processes.

Due to the heterogeneity of the interacting agents in these systems, some being human, others being artificial, these

systems are often referred to as Hybrid Decision Making Systems or Hybrid Systems (HS) for short.
In this survey, we are going to provide an overview of different learning paradigms for hybrid systems, their evolution over time, a taxonomy to properly categorize existing hybrid systems, and their strengths and weaknesses.

Within hybrid systems we identify two archetypal agents of different nature: a human agent, which is endowed with peculiar reasoning abilities, domain, and commonsense knowledge; and a machine agent, which is instead endowed with high computing power.
The two agents[We refer to “agents” simply as entities, i.e., humans and AI systems, within a system, rather than “entities with agency”, that is, we do not attribute intentions to agents.],
which we simply refer to as human and machine, are best suited to solve different tasks.

Unlike humans, machines often fail in tasks requiring (potentially) complex reasoning, domain knowledge <cit.>, commonsense <cit.>, contextualization, and general human touch <cit.>.
On the other hand, machines excel in computation-heavy tasks and are uniquely capable of performing such tasks at scale.
It is no surprise that human and machine agents often disagree <cit.>, and even when they do not, they may be agreeing for different, possibly conflicting, reasons <cit.>.
Further exacerbating this, traditional machine evaluation metrics are often blind to this disagreement, since they are designed to simply target the performance of the machine, rather than its internal reasoning.

Whenever an agent attempts to solve a task, they are performing a computational step, also known as computational turn <cit.>, which is a human step if the agent is human, and a machine step if the agent is a machine.
The definition of the task of interest, of the agents involved, of their computational steps, and of their joint behavior, defines a Hybrid

Generally, we use the term system to identify the whole hybrid system, and machine (sub)system and human (sub)system to refer to the machine and human component of the system, respectively.

The system is specifically designed to solve a given task on which are defined metrics of performance, e.g., accuracy or mean error.
The system

to maximize performance by carefully leveraging its human and machine subsystems.
Optimizing this pairing can yield additional benefits by its own design.

Successive observations of the behavior of the machine allow humans to best understand the abilities and limitations of a machine <cit.>,

allowing the human agent to develop
a mental model of the machine, that is, an internal representation of how they perceive the machine.
If the machine is also able to observe the human in return and gather feedback and corrections from them, then

we close the feedback loop and have a hybrid system with bidirectional communication: the human understanding the machine, the machine understanding the human.

A bidirectional system is also at the basis of several uniquely human benefits: humans who perceive to understand a machine through a good mental model are more likely to trust it and rely on it <cit.>, thus boosting future use of the machine itself; humans that perceive to be able to “steer”

a machine, are more likely to do so <cit.>, and thus ease alignment between themselves and the machine.
Machines instead, are able to better integrate uniquely human abilities to which they are not preview to, or that are particularly difficult for them to learn <cit.>.

The ultimate goal of a hybrid system is to promote and facilitate human and machine collaboration, thus improving the overall decision quality, reliability, transparency, fairness, etc..
[colback=blue!5!white,colframe=blue!60!white,title=Example of a Generic Hybrid Decision Making System]
For clarity, we will use an example to guide us during the survey.

An online forum, e.g., a social network, is faced with a large number of messages, some of which may require moderation or further consideration.
The platform may be interested in detecting hate speech or individuals at high-risk of self-harm, or harm to others.
Given the sheer number of potential messages of interests, systems managing such forums, e.g., through content moderation, are largely automated.
For the sake of simplicity, we can think of moderation as a binary task: either flag a message for moderation or not.
This deceptively simple task hides several complexities.
Content moderation relies on policy guidelines that are frequently updated and are often open to interpretation due to their inherent vagueness.
Messages rely heavily on context, which however is not always understandable from the message alone (e.g., when a message mentions an event that occurred outside of the platform).    The context is constantly changing, so messages may or may not be considered suitable for moderation at different times.
Irony, hyperbole, and metaphors further increase the complexity of such task.
AI systems have become increasingly better at tackling this task, but are still far from being reliable enough for a platform to blindly trust them, hence online forums heavily rely on hybrid systems where human moderators collaborate with said AI systems.

In this paper, we focus on hybrid decision making systems with some degree of interaction between human and machines, and as such we do not specifically address systems where humans are not active agents nor they are pure teaching agents within an environment,

such as knowledge injection systems, and reinforcement learning systems.
Given their strong focus on interaction and little to no focus on integration of the heterogeneous agents into a cohesive system, we do not focus on conversational agents, and interactive XAI algorithms.

We refer to works in the literature on each of the aforementioned topics <cit.>.

[node distance=1cm, auto,->,shorten >=2pt]

[draw=none,fill=none] (data) at (0,0) Data;
[shape=rectangle,draw=black] (machine) at (0,-1) Machine;
[shape=circle,draw=black] (human) at (0,-3) Human;
[draw=none,fill=none] (prediction) at (0,-4.5) Prediction;

[->] (machine) edge node Prediction (human);
[->] (data) edge node  (machine);
[->] (human) edge node  (prediction);

Human overseers.

[node distance=1cm, auto,->,shorten >=2pt]

[draw=none,fill=none] (data) at (0,0) Data;
[shape=rectangle,draw=black] (policy) at (0,-1) Orchestrator;
[shape=circle,draw=black] (human) at (1,-3) Human;
[shape=rectangle,draw=black] (machine) at (-1,-3) Machine;
[draw=none,fill=none] (prediction_human) at (1,-4.5) Prediction;
[draw=none,fill=none] (prediction_machine) at (-1,-4.5) Prediction;

[->] (data) edge node  (policy);
[->] (policy) edge node  (human);
[->] (policy) edge node Data (machine);
[->] (human) edge node  (prediction_human);
[->] (machine) edge node  (prediction_machine);

Learn to Abstain.

[node distance=1cm, auto,->,shorten >=2pt]

[draw=none,fill=none] (data) at (0,0) Data;
[shape=rectangle,draw=black] (machine) at (0,-1) Machine;
[shape=circle,draw=black] (human) at (0,-3) Human;
[draw=none,fill=none] (prediction) at (0,-4.5) Prediction;

[->] (data) edge node  (machine);
[->] (human) edge[bend left] node Correction (machine);
[->] (machine) edge[bend left] node Prediction (human);
[->] (human) edge node  (prediction);

Learn together.

Paradigms of hybrid systems, where human (circle) and machine (rectangle) steps alternate to form a cohesive system.
In human overseers (<ref>), the machine performs a prediction, and the human accepts it or rejects it in favor of their own.
In Learn to Abstain (<ref>), an orchestrator assigns the prediction task to either of the two, which makes the prediction on their own.
In Learn Together (<ref>), the two agents repeatedly interact by showing and correcting each other's reasoning, finally coming to a prediction.

§.§ Paradigms of Hybrid

Hybrid systems aim at synergically integrate humans and AI systems, and as such can be categorized according to the depth of such integration.
This allows us to identify three distinct families or paradigms of hybrid systems with significant differences: Human overseeing, Learn to Abstain, and Learn Together – we visually depict the paradigms in Figure <ref>.
Each of these systems furthers the control that the human can exert on the AI system, and as a consequence also the level of integration.

Unsurprisingly, they also roughly follow in chronological order, with systems becoming increasingly integrated as research progresses.

Human overseeing, the simplest of the three, implements a pipeline with no integration between human and machine, the former overseeing the predictions of the latter.
Here, the two steps are one after the other.
Since the human is aware of the results of the machine step, and is also able to overwrite it, compliance is maximal.
Learn to Abstain
instead only allows for one of the two steps to take place, either the human step or the machine step.
Here, the step of choice is given by an additional machine step that guides the choice.
Unlike machine oversight, agents perform their step independently of other agents, that is, each

is exclusively taken by one of the agents, and an orchestrator decides which agent

predicts what.
Learning Together systems instead enable a more sophisticated alternation of the two steps, which create a true loop in which human and machine steps are repeated in sequence an arbitrary number of times.

Unlike the previous paradigms, the Learning Together paradigm features a communication loop between human and machine agents.

The survey is organized as follows: we provided a general formulation of hybrid systems in Section <ref>.
We than detail the three paradigms, namely, Human overseeing, Learn to Abstain, and Learn Together, in Sections <ref>,<ref>, and <ref>, respectively.
We  provide an overview of the open problems in Section <ref>,
and finally conclude in Section <ref>.

§.§ Paper selection criteria

To precisely outline three paradigms, we have explored the scientific literature adapting the paper selection criteria to

the significance of the specific paradigm in relation to the scope of this study.

As for the Paradigm 1, we have we examined papers that have investigated the impact of human oversight on ML models through qualitative or quantitative analysis, but without completing a formal survey. On the contrary, in Paradigm 2

we conducted a comprehensive review based on the keyword "learning to reject", "learning to defer", "learning with a reject option", "selective classification", "deferral policy", "deferral function" and "defer to expert". After a preliminary screening of top conference/journal papers and highly cited papers, where we gathered 150 papers focusing mainly on endowing algorithms with abstention mechanisms, we selected 37 papers representing the wide spectrum of solutions of Learning to Reject, and 53 papers encompassing the most significant findings in Learning to Defer.

Finally, in Paradigm 3, we examined studies that suggest incorporating human input directly into the learning process of an ML model.

The chosen papers represent a promising future path for research on Hybrid Decision Making Systems.

All papers where searched using Google Scholars and DBLP computer science bibliography, selecting papers with high citation counts and/or published in top journals or conferences, such as AAAI, IEEE, ACM, NEURIPS etc..

§ GENERAL FORMULATION OF HYBRID SYSTEMS

Symbol     Definition     Description

2* Agents
$H$     $\mathcal{H}$     Human agent.

$M$     $\mathcal{M}$     Machine agent.

3*R. Variables
$X$     $\Omega \rightarrow \mathcal{X}$     Feature matrix. $\mathcal{X}$ includes the empty element $\emptyset$.

$Y_{(\cdot)}$     $\Omega \rightarrow \mathcal{Y}_{(\cdot)}$     Label vector: $\mathcal{Y}_{(\cdot)}$ $\subset$ $\mathbb{R}$ (regression), $\mathcal{Y}_{(\cdot)}\subset$ $\mathbb{N}$ (classification) provided by a given agent $(\cdot)\in\{M,H\}$.

$Y^*$     $\Omega \rightarrow \mathcal{Y}$     Ground truth label vector.

$f, f_\theta$     $\mathcal{X} \rightarrow \mathcal{Y}_{(\cdot)}$     Predictor function implemented by a machine agent ${(\cdot)}$, belongs to the family of functions $\mathcal{F}_M$. The parameter $\theta\in\Theta$ is omitted if not relevant.

$f^*$     $\mathcal{X} \rightarrow \mathcal{Y}$     Ground truth function. Belongs to the family of functions $\mathcal{F}_M$

$\mathscr{L}_{(\cdot)}$     $\mathcal{Y} \times \mathcal{Y}_{(\cdot)} \rightarrow \mathbb{R}_{>0}$     Real-valued loss function of an agent $(\cdot)\in\{M,H\}$.

$\widetilde{\mathscr{L}}_M$     $\mathcal{Y} \times \mathcal{Y}_M  \rightarrow \mathbb{R}_{>0}$     Real-valued surrogate loss function of a machine agent $M$.

$\rho_{(\cdot)}$     $\Rho_{(\cdot)} \rightarrow \{0,1\}$     Deferral or rejection policy of agent $(\cdot)\in\{M,H\}$.

4*Humans       $Z_{(\cdot)}$     $\mathcal{Z}$     Set of artifact(s) only available to the human agent $(\cdot)$.

$h_{(\cdot)}$     $\mathcal{X}\times\mathcal{Z} \rightarrow \mathcal{Y}_H$     Predictor function implemented by the human agent $(\cdot)$.

$A$     $\mathcal{A}$     Hybrid artifact, enables communication among agents.

$B$     $\mathcal{P}(\mathcal{A})$     Artifact bank, memory storing artifacts.

Table of symbols. We provide a definition for each symbol, depending on the context. We will use lowercase letters for elements of a space, e.g., elements $x$ of $\mathcal{X}$.

A Hybrid

System is composed of two types of agents: a machine agent $M$ and a human agent $H$.
Consider a probability space $(\Omega,\mathscr{F},\mathbb{P})$ and let $\mathcal{X}$, $\mathcal{Z}$, $\mathcal{Y}, \mathcal{Y}_M$, and $\mathcal{Y}_H$ be measurable spaces such that $\mathcal{X}$ represents the machine feature space, $\mathcal{Z}$ the human expertise, that can be modeled as features, decision rules, etc.,

and $\mathcal{Y}, \mathcal{Y}_M$, and $\mathcal{Y}_H$ the ground truth, machine and human label spaces for a certain task $T$, respectively. Full list of symbols can be found in Table <ref>.

Machine model.
Let $X:\Omega\rightarrow\mathcal{X}$ and $Y_M:\Omega\rightarrow\mathcal{Y}_M$ be random variables representing the input and output of a machine $M$, and let $Y^*:\Omega\rightarrow\mathcal{Y}$ be the random variable representing the true labels.
In the classical setting of supervised learning, given independent and identically distributed pairs $\{(X_i,Y_i^*)\}_{i=1}^n\overset{\text{iid}}{\sim} (X,Y^*)$ drawn from the same unknown joint distribution over $X\times Y^*$, we aim to learn a predictor $f:\mathcal{X}\rightarrow \mathcal{Y}_M$ that approximates, as accurately as possible, the unknown function $f^*:\mathcal{X}\rightarrow\mathcal{Y}$ representing the true relationship between the features and the target.
Note that $f$ is chosen from a space of hypothesis $\mathcal{F}_M$ parameterized by $\theta\in\Theta$, however, in our work we often omit the parameter $\theta$ to simplify our notation.

The ultimate goal of the learning algorithm is to find an hypothesis $f\in\mathcal{F_M}$ that

minimizes the empirical expected risk, which is defined on the training set as follows:

\begin{equation}\label{eq:empRiskM}
\widehat{\mathscr{R}}_n[f] = \frac{1}{n}\sum_{i=1}^n \mathscr{L}_M(Y_{M,i},f(X_i))
\end{equation}

The function $\mathscr{L}_M:\mathcal{Y}\times\mathcal{Y}_M\rightarrow\mathbb{R}_{>0}$ represents the machine loss, which quantifies how far the predictions of a hypothesis $f$ are from the true outcome.

In many cases, the formulation of the loss may have a level of complexity that renders direct computation infeasible due to the

curse of dimensionality or potential model mis-specification <cit.>;

equation (<ref>) is often not continuous nor differentiable, hence extremely hard to optimize and computationally intractable for many nontrivial classes of functions <cit.>. The approach typically employed to overcome this issue is to define and optimize a surrogate loss function $\widetilde{\mathscr{L}}_M:\mathcal{Y}\times\mathcal{Y}_M\rightarrow\mathbb{R}_{>0}$, that is, a function with good computational guarantees (e.g., differentiability and convexity) that can be easily optimized and whose optimal values approximate well the minimizer of the original computationally hard loss function <cit.>.
The exact formulation of a surrogate loss function is not straightforward, as it depends on the particular task at hand and the desired properties one seeks to guarantee. Refer to Appendix <ref> for an extensive discussion about the fundamental mathematical properties that characterize surrogate losses.

Human model.
Let $Z:\Omega\rightarrow\mathcal{Z}$ and $Y_H:\Omega\rightarrow\mathcal{Y}_H$ be random variables representing,
respectively, the human expertise and the predictions of

a human agent $H$ for a certain task $T$, where $H$ is modeled as a predictor $h:\mathcal{X}\times\mathcal{Z} \rightarrow \mathcal{Y}_H$. Generally, we consider $\lvert \mathcal{X} \bigcap \mathcal{Z} \rvert \geq 0$, implying that there may exist shared information between the machine and the human, or there may not.

It is worth noting that there may be a divergence between the predictions made by human agents and the ground truth labels.

Indeed, different levels of background knowledge, experience, or personal biases can lead to distinct decision-making outcomes, resulting in both correct and incorrect predictions across various domains within the input space <cit.>.
Therefore, we additionally take into account a  loss function $\mathscr{L}_H:\mathcal{Y}\times\mathcal{Y}_H\rightarrow\mathbb{R}_{>0}$ as an indicator of the quality of human predictions.
The observation that various types of errors are not only made by distinct human agents but also occur between humans and AI models provides support for the transition towards paradigms that incorporate hybrid combinations of human and machine predictions <cit.>.

§ PEERING INTO THE MACHINE: HUMAN OVERSEERS

Human oversight <cit.> is probably the simplest and most straightforward form of hybrid system.
In this first paradigm, machine and human agents are independent of each other, the former performing a task, and the latter verifying its predictions.
Informally, the human agents perform a straightforward task: given the machine computation and/or the input data, either accept or reject the computation.
More formally, a human oversight policy $\rho_H$ is a binary function that, given a machine $M$ implementing a function $f \in \mathcal{F}_M$, an overseeing human $H$ with additional expertise $Z \in \mathcal{Z}$, and some input data $x \in \mathcal{X}$, either accepts or rejects the prediction given by $M$.
In its most general formulation, $\rho_H$ is formulated as follows:

\begin{equation*}
\rho_H: \mathcal{X} \times \mathcal{Y}_M \times \mathcal{Z} \times \mathcal{F}_M \rightarrow \{\text{accept}, \text{reject}\}.
\label{eq:prdg1:def:gen}
\end{equation*}

In many cases, the human agent either does not have access or does not take into consideration the machine itself, thus the above formulation is often reduced to:

\begin{equation*}
\rho_H: \mathcal{X} \times \mathcal{Y}_M \times \mathcal{Z} \rightarrow \{\text{accept}, \text{reject}\}.
\label{eq:prdg1:def}
\end{equation*}

[colback=blue!5!white,colframe=blue!60!white,title=Example of online content moderation: Human Oversight]
In the paradigm of Human Oversight, the human moderator is presented with a message $X$, the prediction $Y_M$ given by the machine, and has to decide whether to agree ($\rho_H(X, Y_M, Z) = 1$) or not ($\rho_H(X, Y_M, Z) = 0$) with the prediction of the machine $Y_M$, e.g., whether the message is hate speech, or is indicative of an individual at risk of self-harm.

Other than simply leveraging their own expertise $Z$, overseers often leverage external factors in their decision.
In the simplest of cases, rejected patterns of misbehavior are limited to machine-specific failures in which the underlying context is of little or no impact.
In these context-independent scenarios, the overseers aim to identify machine failures induced by machine-specific causes by tracking a set of subjects of monitoring around which the overseeing policy will be centered.
Machine-specific failures

may be induced by wildly different factors in each machine,

we abstract over the underlying causes, since the goal of machine oversight is to identify, rather than diagnose, undesired behavior.
We tackle behavior correction later in Section <ref>.
Common subjects of monitoring are:

* Data shift. Data shift is generally intended as a change in the data distribution <cit.>, and may be due to a change in feature distribution, i.e., covariate shift or knowledge drift, or label distribution, i.e., prior shift;
these shifts may occur frequently in real-world scenarios, namely due to data seasonality, sampling bias, or naturally-occurring distribution changes.

outlier instances, dataset shifts are responsible for consistent and predictable failures in the model, thus they have to be accounted for.

* (Partial) Model performance. Partially stemming from data shift, model performance is another primary subject of interest.
Here, we distinguish between two classes of performance metrics: “global metrics”, where the model is evaluated wholly, and “partial metrics”, where the model is evaluated partially on a suitable subset of data.
Data shift may indeed only affect a subset of data, hence global metrics may easily deceive the overseers.

* Model uncertainty. Classical learning algorithms are trained to predict, setting confidence calibration aside.
Simply put,

operate under a closed world assumption where the only available option is to give a prediction, that is, the model has no notion of uncertainty nor of the unknown.
This projects a false high confidence, seldom tricking the algorithm user into overestimating its competency <cit.>.

* Decision complexity. Even with highly sophisticated and precise models, some decisions are inherently suitable for human rather than algorithmic reasoning <cit.>.
In this context, it is crucial to anticipate which decision is which.

Operating as self-contained agents, machines often lack the decision context wherein their predictions are evaluated and applied.
Failures in this domain are said to be context-dependent.
Context can be of primary importance, and humans are far better suited than machines in understanding it and integrating it in their decision-making process.
For a human, concerns such as fairness, legality, and explainability of the decision are strong contextual motivations that a machine does not necessarily take into proper account.
Unsurprisingly, most of them are already being encoded in several legislatures, which strongly discourage or punish discriminative or otherwise illegal <cit.>, unexplainable <cit.> decisions and behaviors.
Much of this stems from the current use of machine agents in ethically-charged contexts.
For instance,

machines are leveraged in monitoring and discouraging <cit.> illegal activities, where they often yield unfair or biased predictions <cit.>; furthermore, they are a critical component of speech regulation, an extremely dynamic use case where human scrutiny and decision autonomy are essential, yet they often regulate marginally- or fully-free <cit.> speech;
they aid hiring in public and private companies, yet they are biased <cit.>.

What's more, context is often dynamic and loosely defined <cit.>, and thus integrating it into the machine is an open challenge in and of itself.
Jointly, machine-specific and context-specific failures offer a strong motivation for machine oversight. Yet, even though the why is clear, how machine oversight is to be implemented is still an open problem.
Even worse, machine oversight poses a set of inherently human problems to face.

§.§ Monitoring pitfalls

While technical solutions to machine-specific failures have already been developed, context-dependent failures cause a plethora of additional and more complex problems. Given that humans are usually the time bottleneck when it comes to decision-making, one cannot let the overseers monitor every prediction, thus one needs to understand when to let the overseer monitor the machine.
A conventional
solution, which we explore in Section <ref>, is to let the machine itself call the human into action.

Here, we focus instead on the overseers and their inherently human fallacies, which lead to some natural pitfalls of the whole monitoring process.

Human Executors and Skeptics.
Overseers need to be aware of two possible cognitive biases: algorithmic aversion <cit.> and overreliance <cit.>.
Algorithmic aversion pushes the overseers towards excessively doubting the machine, thus introducing unnecessary monitoring in the decision-making process.
Algorithmic aversion manifest itself independently of the performance of the machine <cit.>, and more strongly when the machine fails.
In other words, every single perceived mistake of the machine compounds in increasing the rejection rate of the overseer.

Unlike algorithm aversion, algorithm overreliance
occurs when human agents under-monitor a machine system, and thus act as mere executors.
The two biases are well-documented in the literature, and regardless of the machine system they human agents interface with, they are to be accounted for.

Biased monitoring.
Automation bias is particularly strong when human agents oversee fairness- related tasks where the task directly involves other humans.
When monitoring decision on pre-held stereotypes, say on

groups, overseers either avoid monitoring in the first place <cit.> or further confirm the stereotypes <cit.>.
On similar reasoning, particularly in cases of fairness evaluation, human agents tend to under-monitor when they perceive affinity towards the monitor case at hand <cit.>, or simply when they deem their reasoning more “human” than the machine's <cit.>.
On an even more biological level, intrinsic demographic traits are also likely to play a role in the decision-making of the human agents <cit.>.

To further increase the complexity of setting up a set of overseers, it is often the case that human agents, in part for the aforementioned reasons, have a low level of agreement on the correct task solution <cit.>.

Failure to oversee and trust calibration.
Even more worrying than biased monitoring is the failure to reject obvious machine failures.
In a pilot study with legal experts, <cit.> showed that, when assisted by a faulty machine agent, domain experts incorporate into their decision-making machine recommendations based on irrelevant or random factors, with extreme cases in which the domain experts were knowingly introducing random factors themselves – a clear case of placebo effect where the mere presence of a machine prediction, regardless of its correctness, induces an almost blind trust in the human agent.

Unsurprisingly, overseers have repeatedly shown to be unable to properly assess their ability to assess the performance of a given machine <cit.>.

§.§ Enhanced monitoring: the case for Explainable AI

Overseeing a machine simply through its predictions and uncertainty provides minimal tools to a human agent, which can easily fall into one or more of the aforementioned pitfalls.
To enhance their overseeing power, humans are often accompanied by explanation algorithms, that is, algorithms able to further explain the predictions given by a machine.
Explainable AI (XAI) <cit.> is a recent field of research aiming to shed light on the prediction process of AI models by extracting human-understandable explanations.
Explanations allow an overseer to peer into the machine and get a grasp of what features a machine is relying upon <cit.> and what rule-like logic it is following <cit.> to make its predictions, what training instances have had a particular influence on the learning process <cit.>, and how one could change the input instance to achieve a different prediction <cit.>.
Explanations have shown to empower the human agent into better understanding the machine, thus improving their ability to monitor it.

§ LEARNING TO ABSTAIN

To mitigate human failures in the monitoring process of the aforementioned HS paradigm, a potential approach involves developing an enhanced machine architecture that enables the machine learning model to refrain from predicting on certain instances. In broad terms, whenever the machine prediction can't be relied upon, it may be better to pay an extra costs and defer the prediction to a human agent <cit.>. The ultimate goal of a Learning to Abstain system is thus to maximize the overall performance of the hybrid system by effectively identifying, for each single instance, the agent best suited for a given prediction.

This particular setting does not involve any formally defined interaction between human and machine agents. Instead, it can be better characterized as a policy in which a machine autonomously directs input data towards an agent, whether it be a human or an AI, based on an estimation of superior performance in each individual case. In other words, the algorithms in this paradigm produce models optimized to leverage the option to abstain w.r.t. their primary predictive task. The human interaction in this paradigm may come afterwards, e.g., operating on those instances rejected by the Learning to Abstain system.

At the most general level, the machine learning systems that fall within this framework can be categorized into Learning to Reject (L2R) <cit.> and Learning to Defer (L2D) <cit.> systems, depending on whether the model assumes a pre-specified cost for abstaining or instead is designed to work adaptively with the

human agent.
Both classes of algorithms share the same set-up, namely an

where the machine (or a system of machines) should learn not only a standard classification function but also a rejection function (also called deferral function in the L2D domain) under the optimization objective of maximizing the performance of the human-AI system as a whole. The deferral action incurs an additional cost which is offset by the payoff of an expected gain in performance resulting from querying a human expert, who can use information that the machine cannot rely on (e.g., professional experience and common sense). Although L2R and L2D have been tackled as separate and independent problems, a seminal paper by <cit.> demonstrated that L2R can be regarded as the specific case of L2D in which a fixed cost is allocated to each deferred instance. Consequently, they proposed a revised definition of L2D as an adaptive L2R approach.
Therefore, we devote most of our discussion to L2D (Section <ref>) while giving a more general overview of L2R (Section <ref>).

§.§ Learning to Reject

Learning to Reject (L2R) was first introduced by  <cit.>, and its general formulation constitutes a base for more advanced learning to defer algorithms (Section <ref>).
[colback=blue!5!white,colframe=blue!60!white,title=Example of online content moderation: Learning to Reject]

In the L2R paradigm, the algorithm is trained to consider a third alternative with respect to the original binary classification task, namely the rejection option. Specifically, the model decides to abstain from making a prediction in those cases where it is more likely to make a wrong prediction.
For example, a model could be trained to recognize those messages that contain sarcasm or humor, typical cases where a classification model for moderation is likely to make a wrong prediction, and abstain on those messages. In this way, a person responsible for monitoring the system can focus solely on the instances that are rejected by the model, rather than having to deal with all of them.

In the literature, this area of research is referred to with several names: learning to reject <cit.>, selective classification <cit.>, or machine learning with a reject option <cit.>. The general idea is to provide machine learning algorithms with the means to learn when to abstain from actually making a certain prediction. Here, the focus is on optimizing machine learning algorithms to include a reject option, thus improving the final performance by refraining from predicting on critical instances.
Therefore, humans are not an active agent in this framework, as the focus is on finding the best combination of machine prediction and machine abstention.
However, in the context of Hybrid Systems, one could envision real life scenarios where L2R models can be used to focus the attention of a human overseer on the subset of instances that are rejected by the model.
Learning to Reject is a widely studied problem with a long history of research, already touched upon in several previous surveys. Therefore, we report a similar general definition to the work of  <cit.> and  <cit.>.
The goal of L2R algorithms is to learn a model $f_{\rho_M}$ composed of two parts: a prediction function $f:\mathcal{X}\rightarrow \mathcal{Y}_M$ and a rejection policy $\rho_M:\mathcal{X} \rightarrow \{0,1\}$

The composed system can be defined as $f_{\rho_M}:\mathcal{X} \rightarrow \mathcal{Y}_M \bigcup \{\emptyset\}$ such that:

\begin{equation}
\label{eq:l2rSystem}
f_{\rho_M}(x) = \begin{cases*}
\emptyset   & if $\rho_M(x)=1$\\
f(x)       & otherwise
\end{cases*}
\end{equation}

That is, if the rejection policy $\rho_M$ rejects $x$, then no prediction is made. If instead $\rho_M$ accepts $x$, then the prediction function $f$ is applied to $x$ and the result $f(x)$ is obtained. The function $f$ is generally assumed to be a classifier.
Ideally, $\rho_M$ should be able to prevent misprediction of $f$ while conversely accepting examples for which a good prediction is more likely. The policy $\rho_M$ that actually performs the rejection operation is called rejection function or rejector.

The focus of a rejector, as mentioned, is to refuse those examples for which the classifier $f$ is expected to output an incorrect prediction. More generally, the focus of research in this field is to find the optimal balance between accuracy on accepted instances and number of rejected instances. We can generally divide the types of rejection into two macro-categories, depending on the type of data that are being rejected:

* Novelties rejection, i.e., rejections of examples that differ significantly from the data points in the training set $X$ of $f$ and therefore are likely to cause a misprediction.

* Ambiguities rejection, i.e., rejections of examples in proximity of the decision boundary of $f$.

Novelties are examples that are, in general, very different from the ones that $f$ was trained on. Novelties are usually data that may be considered out of distribution (OOD), or simply particularly unusual instances.
Ambiguities are typically examples that can either be classified as any by $f$ as their probability of being a member of one is almost equal to the probability of belonging to the others <cit.>. Depending on the kind of mistake that needs to be addressed, different methods for learning the rejector can be used.
The rejector can be based only on feature observation or on model evaluation. Therefore, $\rho_M$ may have access to only the feature space $\mathcal{X}$ or to both $\mathcal{X}$ and the output of the predictor $f$, or even to the architecture and parametrization of $f$. In essence, a rejector can either be:

* Independent, i.e., the rejection function is learned regardless of the predictor, by observing solely the feature space.

* Dependent, i.e., the rejection function is built by either querying the predictor or by relying on particular characteristics of the predictor.

§.§.§ Independent rejectors
Independent rejectors are mostly designed to perform Novelties rejections. Most of the works in this particular area frame the problem as Outlier detection <cit.> or Anomaly detection <cit.>. Another widely-used term is Open-set recognition <cit.>, where the focus is on optimizing the accuracy of a predictor both on in-distribution and out-of-distribution data. The basic idea is to recognize and handle those examples that present out-of-distribution characteristics. This is done independently from the predictor function $f$, i.e., the rejection mechanism can be applied directly to the data in an agnostic way w.r.t. the trained model $f$.

Some of the earliest work in this area leverage statistical methods to find anomalies in the data.  use the estimated posterior variance of Gaussian processes  as a testing point to understand whether to reject a data point or not <cit.>. Another similar technique was proposed by  <cit.> which relies on statistical models studied in Extreme Value Theory to understand which examples may actually be considered outside expected distributions. This technique has been employed in many different tasks, e.g. facial recognition <cit.> and network analysis <cit.>.

Recently, <cit.> proposed the Extreme Value Machine, a model for kernel-free variable bandwidth incremental learning based on Extreme Value Theory.

One limitation of

this model
is that it relies on the distances between the instances belonging to different classes in the training set, so that classes that are unreasonably distant from the majority classes in the training may be rejected in their entirety.
Some works address this limitation by relying on generalized Pareto distribution approximation to determine whether a new example is a normal or abnormal data point <cit.>.

Other approaches rely on models trained to recognize the training distribution and perform anomaly detection as a separate task. One such method was proposed by  and it relies on One-Class Support Vector Machines <cit.> specifically fitted to recognize the training distribution. Alternatively, Gaussian Mixture Models can be employed to estimate the training distribution, and thus detect novelties <cit.>. Recently <cit.> proposed a few-shot learning approach where margin-loss w.r.t. the training class is used to train a model to detect anomalies.

While independent rejectors are mainly designed to detect anomalies,  <cit.> propose a framework for generalized rejection based on jointly-trained Neural Networks. These networks are trained with a dual-penalization both on misprediction and incorrect rejection. Since the rejection network is trained independently, it can then be used with any other prediction model, and it can also perform ambiguity rejections.

§.§.§ Dependent rejectors
Dependent rejectors are much more studied than independent ones. They are tied to the outputs or properties of the predictor and as such can be learned either after the learning phase of $f$ or simultaneously.  Adopting the same naming convention that we will use for Learning to Defer models (Section <ref>) we call the former staged rejectors and the latter joint rejectors.

Staged rejectors look at the predictor $f$ output to estimate the confidence or uncertainty of the model. The works that fall in this category are usually referred to as confidence based or uncertainty based. These approaches estimate uncertainty by i) formulating a good confidence metric in and ii) selecting an appropriate threshold for determining the actual rejection <cit.>.
Formally, the rejection of an instance can be seen as:

$$\rho_M(x,f) = \mathbbm{1}_{c(x,f) > \tau}$$

where $c(x,f)$ is the confidence metric, which depends both on the instance $x$ and the prediction function $f$, and $\tau$ is the threshold hyperparameter <cit.>.
Many different metrics for confidence have been proposed in the literature. Confidence metrics can be

based on:

* Hard predictions where the exact class-output of predictor $f$,i.e., $y_M=f(x)$, is used to assess the confidence of the prediction. These works usually observe multiple predictions and consider the class-wise variance of these predictions as indication of uncertainty <cit.>.

* Soft predictions where some scoring output of $f$ is used as an estimation of $P(y_M|x)$  to determine the confidence. This approximation of confidence is more widely studied, and the array of solutions that leverage soft predictions is much varied <cit.>.

The are also other methods that, while still exploiting some output of the predictor, have slightly different mechanisms to approximate the confidence.
For example, in the work of  <cit.> the authors exploit directly the score of SVMs as a distance from the decision boundary.
In other works, the distance from the k-th nearest prototype is used as a proxy of confidence <cit.>.

The confidence threshold $\tau$ can either be global, or local: a global threshold is usually used for those predictors with an equally well calibrated confidence metric over the entire feature space <cit.>.
Multiple local thresholds $\tau_1, \ldots, \tau_n$ <cit.> are instead best suited for predictors with

variable accuracy
<cit.> or for different class-wise variance for the confidence metric.

Joint rejectors Also sometimes called integrated rejectors <cit.> are directly part of the prediction model, i.e., the rejection option is treated as an additional class to be learned by the predictor. In this setting, predictor $f$ and rejector $\rho_M$ are simultaneously learned with a single algorithm, specifically designed to learn both functions. This means that it is difficult to technically distinguish and separate the predictor from the rejector <cit.>.
Some joint-learning approached rely on the definition of a specifically formulated objective function able to penalize both incorrect predictions and rejections. Several works resort to surrogate loss functions in place of a more natural discrete loss in order to more easily solve the problem via minimization <cit.>. Other works aim at specifically leverage some evaluation metric, for example  <cit.> design a model-agnostic approach for probabilistic binary classifiers with a reject option specifically aimed at optimizing Area Under Curve. Other works directly allocate an extra class for rejection to any given predictor and assign a specific penalization cost for predicting such class <cit.>.
Finally, several joint-learning algorithms for rejection have been developed specifically for certain kinds of machine learning models, for example for Support Vector Machines <cit.> and for Neural Networks <cit.>.

§.§.§ Cost model for rejection

Approaches for L2R need to achieve a balance between predictive performance and rejection rate. In fact, classifier with a reject option can entirely trade performance for coverage of vice-versa. This means that, in theory, to achieve maximum performance, a predictor with a reject option can opt to reject all instances and therefore never actually make a misprediction. Otherwise, the predictor can opt to never reject in order to achieve maximum coverage over the data, thus completely forgoing the benefits of implementing a reject option in the first place. To avoid this, an appropriate cost model must be implemented. One foundational cost model can be found in the work of <cit.>:

\begin{equation}
\label{eq:l2rcmod}
cost(x) = \begin{cases*}
0,  & if $\rho_M(x)=0 \wedge f(x)=y^*$\\
1,  & if $\rho_M(x)=0 \wedge f(x)\neq y^*$\\
\mathcal{R} & if $\rho_M(x)=1.$
\end{cases*}
\end{equation}

Where $y^*$ is the ground truth for example $x$, and $\mathcal{R}$ is a fixed, predefined cost for rejection.
If we adopt such a cost model, we can express the the learning objective of a L2R algorithm as follows:

\begin{equation}
\label{eq:l2robj}
%\min_{f,\rho_M} \sum_{x \in X}[(1-\rho_M(x))\mathbbm{1}_{f(x)\neq y^*}+\rho_M(x)\mathcal{R}]
\min_{f,\rho_M} \sum_{x \in X}[\mathbbm{1}_{\rho_M(x)=0}\mathbbm{1}_{f(x)\neq y^*}+\mathbbm{1}_{\rho_M(x)=1}\mathcal{R}]
\end{equation}

In essence, we either pay a cost if we mispredict ($\mathbbm{1}_{f(x)\neq y^*}$) some instance $x$ that we accepted ($\rho_M(x)=0$) or we pay a cost $\mathcal{R}$ if we reject that instance ($\rho_M(x)\mathcal{R}$).

Although sensible, this cost model
has the major drawback of having to determine the rejection cost $\mathcal{R}$ beforehand. Rejection cost can be determined depending on the application domain, but this is not always an option <cit.>.

The work of  <cit.> is among the most important in this field, as it tries to tackle the aforementioned problem. In their proposal, the authors propose a method called Selection with Guaranteed Risk Control that looks at the problem of rejection cost determination from a different angle, that is looking at the coverage of the model. Coverage of a model is defined as the portion of the data that the model accepts for prediction: $\frac{1}{n}\sum_{x \in X}\mathbbm{1}_{\rho_M(x)=0}$.

Following Eq. (<ref>) the learning objective can be expressed as a minimization of risk subject to a coverage constraint that forces the model to predict at least a certain portion of instances. Formally:

\begin{equation*}
\begin{aligned}
%\min_{f,\rho_M} \quad & \frac{\sum_{x \in X}(1-\rho_M(x))\mathbbm{1}_{f(x)\neq y^*}}{\sum_{x \in X}(1-\rho_M(x))} \\ \textrm{s.t.} \quad & \frac{1}{n}\sum_{x \in X}(1-\rho_M(x)) > \mathcal{C}
\min_{f,\rho_M} & \frac{\sum_{x \in X}\mathbbm{1}_{\rho_M(x)=0}\mathbbm{1}_{f(x)\neq y^*}}{\sum_{x \in X}(1-\rho_M(x))} & \textrm{s.t.} \frac{1}{n}\sum_{x \in X}\mathbbm{1}_{\rho_M(x)=0} > \mathcal{C}
\end{aligned}
\end{equation*}

where $0<\mathcal{C}<1$ is a coverage threshold. The problem has thus shifted from having to determine cost $\mathcal{R}$ to having to select threshold $\mathcal{C}$ which is easier. It is important to note that if $\mathcal{C}=1$ the model reverts to a standard classifier with no rejection option. This coverage based formulation has been proven to be theoretically equivalent to the original cost model of Eq. (<ref>) <cit.>.

§.§.§ Strengths and limitations of Learning to Reject

In summary, the L2R paradigm allows for the development of ML models with a reject option, which is a foundational starting point for developing models able to interact with humans. Indeed, ML models equipped with the reject option can, in principle, reject exactly those instances that would yield a prediction error, and therefore call for human intervention only when strictly needed. However, the actual benefit of these techniques in a collaborative setting with humans has never been thoroughly investigated. Indeed, if human intervention is called only for those instances for which a decision is difficult, the human expert may find the same difficulties, and thus deem the model as not so useful for solving the task. Moreover, there are studies pointing to possible fairness issues when using classifiers with a reject option <cit.>.

§.§ Learning to Defer

Learning to Defer (L2D) systems embed human knowledge directly into the training process of a ML model. The goal is to equip the model with the ability to call for human intervention on those instances where the human is likely to give accurate prediction and the machine is likely to fail.
[colback=blue!5!white,colframe=blue!60!white,title=Example of online content moderation: Learning to Defer]

By comparing predictions made by human workers to the correct labels, an L2D system can be trained to determine which instances can be accurately predicted by AI and which are better handled by humans. For instance, in both cases of hate speech and self-harm content recognition, humans are expected to outperform machines on texts characterized by toxic and satirical content, mostly owing to their greater capacity for comprehending common sense and contextual information, as well as discerning implicit meanings hidden within words and phrases.

In contrast to L2D, an L2R system learns its rejector policy only from the feature set $\mathcal{X}$ (e.g., the text of the message to be flagged) and, possibly, some properties of the predictor used by the AI system. L2D instead actively considers the human expertise in the task domain.

Architecture.     2p95mmDesign according to which classifier and deferral policy are integrated.

Staged learning          Classifier and deferral policy are learnt in two subsequent steps.     <cit.>

Joint learning      Classifier and deferral policy are learnt jointly.     <cit.>

Others     Alternative solutions, e.g., iterative models.     <cit.>

Multiplicity.     2p95mmNumber of human agents in the hybrid system.

Single      One human agent.     <cit.>

Multiple (1 predicts)     One human is selected out of many.     <cit.>

Multiple ($j$ predict)    A subset of agents is selected out of many.     <cit.>

Theoretical guarantees.     2p95mmThe machine phase in which the interaction occurs.

Fisher consistency       The optimization objective has the correct target.     <cit.>

Classification-calibration      Agents are given realistic uncertainty estimates.     <cit.>

Realizable consistency     The problem is well-defined under specific choices of the classifier and deferral hypothesis function spaces.     <cit.>

Constraints.     2p95mmAdditional conditions that the hybrid system should satisfy.

Coverage      Number of instances that can be deferred.     <cit.>

Budget     Total cost to query human agents.     <cit.>

Fairness     Metrics to guarantee algorithmic fairness.      <cit.>

Others     Others, e.g., on the selection of human agents.     <cit.>

Properties of systems in the Learning to Defer paradigm.

§.§.§ General formulation

Keeping the same notation introduced in Section <ref>, we consider

Hybrid Systems
composed of a machine $M$ and a human $H$. Similarly to L2R, in L2D the machine $M$ is equipped with the possibility of abstaining from making a prediction.
In addition, an L2D model also embeds a representation of the human agent $H$, thereby taking into account their estimated performance when assessing the act of deferral. By doing so, the human expertise $\mathcal{Z}$ is taken into account.

Nevertheless, note that the predictor $h:\mathcal{X}\times\mathcal{Z} \rightarrow \mathcal{Y}_H$ modeling $H$ is fixed,

meaning that L2D algorithms have no control nor visibility on the function $h$ itself; rather, they only have access to its image, that is, the set $\{Y_{H,i}\}_{i=1}^n\coloneqq \{h(x_i,z_i) \}_{i=1}^n$ of human predictions about the training data.

In analogy to Eq. (<ref>), a L2D system can be formulated as a function $f_{\rho_M}:\mathcal{X} \rightarrow \mathcal{Y}_M \bigcup \mathcal{Y}_H$ defined from the classifier $f:\mathcal{X}\rightarrow\mathcal{Y}_M$ and deferral policy $\rho_M:\mathcal{X}\rightarrow \{0,1\}$:

\begin{equation*}
f_{\rho_M}(x) = \begin{cases*}
h(x)   & if $\rho_M(x)=1$\\
f(x)       & otherwise
\end{cases*}
\end{equation*}

The goal of L2D is to find the classifier-rejector pair $(\hat{f},\hat{\rho}_M)$ that minimizes the system loss $\mathscr{L}_{\text{defer}}$.

can be expressed as the summation of the machine loss $\mathscr{L}_M$ and the human loss $\mathscr{L}_H$:

\begin{equation}
\label{eq:L2DsystemLoss}
% \begin{split}
%     \mathscr{L}_{\text{defer}}(Y,Y_M,Y_H,\rho_M) & \coloneqq \mathbbm{1}_{\rho_M(X)=0}\cdot\mathscr{L}_M(Y,Y_M)+\mathbbm{1}_{\rho_M(X)=1}\cdot\mathscr{L}_H(Y,Y_H)
%     %+ \lambda \mathcal{R}(Y,Y_H,Y_M,\rho_M)
%     = \\
%      & =\underbrace{(1-\rho_M(X))}_{M\text{ predicts}}\underbrace{\mathscr{L}_M(Y,Y_M)}_\text{machine cost} +\underbrace{\rho_M(X)}_{\text{defer to }H}\underbrace{\mathscr{L}_H(Y,Y_H)}_\text{human cost}
%      %+ \underbrace{ \lambda \mathcal{R}(Y,Y_H,Y_M,\rho_M)}_{\text{regularization term}}
% \end{split}
\mathscr{L}_{\text{defer}}(Y^*,Y_M,Y_H,\rho_M) \coloneqq
\underbrace{\mathbbm{1}_{\rho_M(X)=0}}_{M\text{ predicts}}\underbrace{\mathscr{L}_M(Y^*,Y_M)}_\text{machine cost} +\underbrace{\mathbbm{1}_{\rho_M(X)=1}}_{\text{defer to }H}\underbrace{\mathscr{L}_H(Y^*,Y_H)}_\text{human cost}
%+ \underbrace{\lambda \mathcal{R}(Y^*,Y_M,Y_H,\rho_M)}_\text{regularization term}
\end{equation}

where $\mathbbm{1}$ denotes the indicator function.

Note that the optimization problem has $Y_M$ and $\rho_M$ as unique learnable parameters since both $Y_H$ and $Y^*$ are fixed (i.e., they belong to the training data).

In general, the individual losses $\mathscr{L}_M$ and $\mathscr{L}_H$ can take several forms to account for different "costs", such as the misprediction error as in the 0-1 loss, or the cost of querying the human agent.

Notably, when there exists a constant $\mathcal{R}> 0$ such that $\mathscr{L}_H(y^*,y_H)=\mathcal{R}$ for all $(y^*,y_H)\in (Y^*,Y_H)$, then the loss (<ref>) matches the rejection loss described in <cit.> for the L2R framework

\begin{equation}
\label{eq:l2rLoss}
\mathscr{L}_{\text{reject}}(Y^*,Y_M,\rho_M) \coloneqq \mathbbm{1}_{\rho_M(X)=0}\mathscr{L}_M(Y^*,Y_M)+\mathcal{R}\mathbbm{1}_{\rho_M(X)=1}
%+ \lambda \mathcal{R}(Y^*,Y_M,\rho_M)
\end{equation}

Note that Eq. (<ref>) and Eq. (<ref>) are equivalent under the assumption of using the 0/1 cost model for prediction/rejection.

Optimization constraints.
Depending on the specific context of use, the application of specific constraints may be necessary for hybrid systems. This objective is commonly accomplished by incorporating regularization terms into the system loss or by imposing specific bounding conditions. Examples of such constraints include:

* Coverage or triage level <cit.>: the number of instances that can be deferred.

* Fairness metrics <cit.>: for instance, the Minimax Pareto Fairness criterion <cit.> or the equalized odds metric with respect to a protected attribute.

* Budget <cit.>: total cost that can be allocated to query human agents.

Model architectures.

L2D systems typically adhere to either of two general designs, referred to as staged learning and joint learning, which vary in terms of when the classifier and rejector are learned. In the former case, the algorithmic process starts by learning the classifier and only subsequently fits the deferral policy on top of it.
On the other hand, in a joint learning setting the classifier and rejector are learnt simultaneously through the direct minimization of the system loss (<ref>).
While most of the proposals documented in the literature can be categorized as staged or joint learning models, a few exceptions also exists that do not fit in either category (Section <ref>).

Number of human agents.

L2D systems can be characterized along another dimension, that is, the size of the pool of human agents to which the decision can be deferred to.
Whenever the number of these is greater than one, term Multiple-Expert L2D (L2D-ME) is used, as opposed to to Single-Expert L2D (L2D-SE or L2D), which considers one human only. An example where L2D-ME modeling may be more suitable is in a medical setting, where a critical decision regarding a complex case could be made either by an automated classifier or by one or more doctors chosen from a team of experts with potentially diverse expertise and opinions.

As compared to L2D-SE, the formalization of L2D-ME makes the deferral function more complex in nature. Specifically, it should not only determine when to defer, but also to which human agent(s) <cit.>. Furthermore,

the deferral policy can be designed in a manner that allocates predictions to either

one single agent or to a

subset of agents from the available pool.

§.§.§ Staged learning architectures

In L2D models characterized by staged learning architectures, the classifier $f$ and deferral function $\rho_M$ are learnt separately. Specifically, algorithms in this category first fit a classifier on the training dataset, then they learn a second model that predicts the probability that the human makes a mistake on the same dataset, and finally they defer based on which has the lowest error probability instance-wise.

For instance, <cit.> developed a basic heuristic for L2D consisting of two independent models trained on the full dataset: a multiclass classifier representing the machine agent, and a binary classifier representing the correctness of the human agent. At inference time, an instance is deferred to the human if the predicted classifier error probability is higher than that of the human. In case of coverage constraints, then the samples whose difference between human and classifier error probability is higher are chosen first.
Interestingly, the authors also suggest a reduction of L2D-ME to L2D-SE by modeling the human subsystem in terms of average disagreement between human agents on each single prediction. This approach has been further developed in <cit.>.

Another common baseline for staged learning is the model proposed by <cit.>, who described a staged learning setting aimed to maximize the expected utility of the system, which is measured in terms of the accuracy of the final decision, the cost of deferring, and the individual accuracy of both the human and machine component.  Differently from other L2D models,

this method has been claimed to be user-initiated, since the action of deferral is triggered through an (over-simplified) threshold-based policy that represents the humans’ mental model of the AI.

Finally, a third relevant staged learning method known as fixed value of information approach has been proposed in <cit.>. It consists in training independently three probabilistic models describing, respectively, the distribution of the label given the input data, the human predictions given the input data, and the label given both the input data and human predictions. At inference, the deferral policy evaluates the estimated expected utility of the classifier in two scenarios: when the human is not consulted and when the human is queried, while also taking into account the distribution of human predictions and a constant cost for querying the human.

As noted by <cit.>, the staged learning approach presents some important advantages: first of all, it is suited for convenient implementation, since already known appropriate algorithms can be adopted to solve the two stages separately. Secondly, theoretical and experimental results suggest that it outperforms the joint learning approach in realistic scenarios where only a limited portion of data is labeled by the human agent. In these cases, the classifier $f$ can still be optimized over the full dataset; on the other hand, in joint learning $f$ can be learnt from the subset of human labeled data only, thus leading to a reduction int performance dependent on the proportion of unlabeled data.
However, <cit.> also pointed out that staged learning is sub-optimal with respect to joint learning and provide both theoretical and experimental results showing the existence of a performance gap between the two approaches in terms of model complexity.

§.§.§ Joint learning architectures

^^JNeural network without text

^^J  Layer

ı[evaluate==/2-ı; =; =;] in 1,...,

[node ] (N-ı) at (,) $g_1$;
[node ] (N-ı) at (,) $\dots$;
[node ] (N-ı) at (,) $g_n$;
[node ] (N-ı) at (,) ;
[node , label=[label distance=1.45cm, align=center]
AI system
] (N-ı) at (,) ;
[node ] (N-ı) at (,) ;

ȷin 1,...,[]
[connect,white,line width=1] (N-ȷ) – (N-ı);
[connect] (N-ȷ) – (N-ı);

[node 4, label=-90$<$] (N3-4) at (3, -2.0) $g_\bot$;

[shape=rectangle,draw=black,inner ysep=2mm] (N4-1) at (4, -0.0) Machine;
[shape=rectangle,draw=black,inner ysep=2mm] (machine_predict) at (5, -0.0) Prediction;

[node 5, inner ysep=2mm] (N4-2) at (4, -2.0) Human;
[shape=rectangle,draw=black, inner ysep=2mm] (human_predict) at (5, -2.0) Prediction;

[connect] (N3-4) – (N4-2);

[connect] (N4-2) – (human_predict);
[connect] (N4-1) –  node[above=2.3cm,midway,color=black]Prediction (machine_predict);

[pen colour=red, decorate, decoration = calligraphic brace, amplitude=15pt, ultra thick] (3.3,1.3) –  (3.3,-1.3) node [red,midway,xshift=-0.1,rotate=-90] argmax;

[shape=rectangle,draw=black,align=left,inner ysep=3mm,     label=[label distance=1cm, align=center]above:
Input data
e.g., social media post] (outer) at (-1, -0.5)

[shape=rectangle,draw=black,align=center, inner ysep=3mm, inner xsep=2mm] (message)

“Goddam tarantula NEXT to
my bed and a stinking bug
ON my bed?! suicidal fr no lie”;

x y
[node 1, align=center, inner ysep=2mm] (user) at (-1.9, 0.25) =5pt. .

[shape=circle,draw=white,align=center] (phantom_connect) at (0.8, -0.5) ;
[connect] (outer) – (phantom_connect);

Overview of the joint learning architecture for the Single-Expert Learning to Defer (L2D-SE) setting, illustrated in the application of flagging online contents for moderation. Adapted from <cit.>.

In L2D systems characterized by a joint learning architecture the classifier $f$ and deferral function $\rho_M$ are learnt simultaneously.

In order to implement this design, the task is shaped as a $K+1$ multiclass problem over an augmented label space $\mathcal{Y}^{\emptyset} \coloneqq \mathcal{Y}_M\cup\{\emptyset\}$, where $\mathcal{Y}_M=\{1,\ldots,K\}$.
In particular, we set $\textbf{g}=(g_1,\ldots,g_K,g_\emptyset)$ to be the set of real-valued scoring functions
$g_i:\mathcal{X}\rightarrow\mathbb{R}$, such that $g_\emptyset$ returns the human predictions $Y_H$, while the classifier and rejector are defined, respectively, as:

\begin{equation*}
f(x) = \arg\max_{i\in\mathcal{Y}_M} g_i(x)
\quad \quad \quad \quad \quad
\rho_M(x)= \begin{cases*}
1   & if $\max_{i\in\mathcal{Y}_M} g_i(x)\leq g_\emptyset (x)$\\
0       & otherwise.
\end{cases*}
\end{equation*}

In general, the optimal classifier-rejector pair is found by minimizing the system loss expressed in Eq. (<ref>).

However, the function (<ref>) is often computationally hard to optimize.

Such a problem is addressed by replacing (<ref>) with a surrogate loss $\widetilde{\mathscr{L}}_\text{defer}$ that is easy to optimize and is chosen to guarantee specific properties with respect to the original loss $\mathscr{L}_\text{defer}$ (refer to Appendix <ref> for a formal discussion of desirable properties of surrogate loss functions in the context of L2D). Hence, in case of appropriate choices of the human and machine surrogate loss functions, the joint learning HS gives theoretical guarantees for optimal performance.

Single-Expert L2D

Most of the literature on joint learning models for L2D-SE focuses on the cost-sensitive formulation of the problem over an augmented label space $\mathcal{Y}^\emptyset$ developed by  <cit.> and illustrated in Figure <ref>.
This setting considers random costs $\textbf{c}=\{c_1,\ldots,c_{K+1}\}\in\mathbb{R}_{>0}^{K+1}$ where each component $c_i$ represents the cost of predicting the label $i\in\mathcal{Y}^\emptyset$. In this section, we review and categorize the most relevant proposals according to their statistical properties.

* Fisher Consistency (FC) has been described as a minimal requirement that surrogate loss functions should satisfy to achieve reasonable performance, since it posits that, if an estimator were computed using the complete population instead of a sample, it would yield the true value of the estimated parameter <cit.>.
To the best of our knowledge, FC approximation of the 0-1 loss in the L2D setting have been implemented (up to adaptations) only by <cit.>.
In particular, the surrogate loss $\mathscr{L}_{CE}^{\alpha}$ <cit.> consist of a generalization of the cross-entropy loss with the costs corresponding to multiclass misclassification, where the cost of the $K+1$ class represents the action of deferral, and $\alpha\in\mathbb{R}_{>0}$ is a weighting parameter that modulates deferral. When $\alpha=1$, $\mathscr{L}_{CE}^{\alpha}$ has FC and can be expressed as:

\begin{equation*}
\mathscr{L}_{CE}^{1}(\textbf{g};x,y^*,y_H)\coloneqq -\log\left(\frac{\exp(g_{y^*}(x))}{\sum_{y\in\mathcal{Y^\emptyset}}\exp(g_{y}(x))} \right) - \mathbbm{1}_{y_H=y^*} \log\left(\frac{\exp(g_\emptyset(x))}{\sum_{y\in\mathcal{Y^\emptyset}}\exp(g_{y}(x))} \right)
\end{equation*}

intuitively, the first term maximizes the scoring function associated with the true label, while the second maximizes the rejection (scoring) function but only if the human’s prediction is correct.
Notably, <cit.> describes a family of cost-sensitive surrogate loss functions for the 0-1 loss that generalizes prior work and encompasses $\mathscr{L}_{CE}$ as well.

A few adaptations have been proposed to enhance the L2D algorithms based on the surrogate $\mathscr{L}_{CE}$, with the aim of better capturing specific properties. These include:

* Learning to Defer with Uncertainty (LDU), where the deferral policy accounts for the epistemic uncertainty of the model (i.e., the uncertainty resulting from limited data availability and lack of knowledge about the system of interest) <cit.>;

* The customization of the model to suit the expertise of particular human agent, which however requires the availability of supplementary data that has been annotated by that particular human <cit.>.

* Confidence calibration refers to the property of an estimator (e.g., a probabilistic classifier) to produce a predictive distribution that is consistent with the empirical frequencies observed from realized outcomes <cit.>. <cit.> proposed a surrogate loss $\mathscr{L}_{OvA}$ <cit.> that satisfies both Fisher consistency and classification-calibration. This solution consists in solving the L2D problem via a One-vs-All classification method which breaks down the original $K+1$ classes into $K+1$ binary classifier models. The resulting objective function to be optimized is thus a surrogate loss function composed of different logistic loss components, each accounting for the error on one of the $K+1$ different classes.

\begin{equation*}
\mathscr{L}_{OvA}(\textbf{g};x,y^*,y_H)\coloneqq \phi[g_{y^*}(x)]+\sum_{\substack{y\in\mathcal{Y}_M\\ y\neq y^*}}\phi[-g_y(x)] + \phi[-g_\emptyset(x)] + \mathbbm{1}_{y_H=y^*} (\phi[g_\emptyset(x)]-\phi[-g_\emptyset(x)])
\end{equation*}

where $\phi:\{0,1\}\times\mathbb{R}\rightarrow\mathbb{R}_{>0}$ is a binary surrogate loss (e.g., the logistic loss).
Experimental findings show that $\mathscr{L}_{OvA}$ results in better calibrated models w.r.t. ones trained with $\mathscr{L}_{CE}$, with competitive performance w.r.t. other L2D baselines <cit.>.

* $(\mathcal{F}_M,\Rho_M)$-Realizable consistency is a property that refines the notion of FC by addressing the optimization process over restricted hypothesis classes $\mathcal{F}_M$ and $\Rho_M$ for the predictor and deferral function. For instance, the surrogate $\mathscr{L}_{RS}$ <cit.> is differentiable, non-convex, and realizable $(\mathcal{F}_M,\mathcal{\Rho}_M)$-consistent for classes $\mathcal{F}_M$ and $\mathcal{\Rho}_M$ closed under scaling:

\begin{equation*}
\mathscr{L}_{RS}(\textbf{g};x,y^*,y_H)\coloneqq -2\log \left( \frac{\exp(g_{y^*}(x))+\mathbbm{1}_{y_H=y^*}\exp(g_\emptyset(x))}{\sum_{y\in\mathcal{Y^\emptyset}}\exp(g_{y}(x))}\right)
\end{equation*}

Other formulations of joint learning L2D-SE also exist.

This is the case of:

* the seminal paper by <cit.>, which presents a framework for addressing the L2D problem using a Mixture-of-Experts (MoE) approach, with the deferral policy acting as a gating function. The classifier and deferral function are learned together by negative log-likelihood minimization over the augmented label space $\mathcal{Y}^\emptyset$. An alternative version of the algorithm is also introduced, wherein a regularization component is added to the system loss to account for a fairness. Unfortunately, this method was proven to not have FC <cit.>.

* Preferential MoE <cit.>, a variant of <cit.> where human knowledge is encoded in the form of decision rules that should be followed as much as possible, that is, whenever they are applicable and do not decrease the system performance. The algorithm first checks the applicability of the available rules and, in case of positive response, a deferral function selects whether to rely on the human or machine prediction based on their performance.

Notably, the deferral function is chosen to be interpretable (e.g., a linear classifier or decision tree) guaranteeing transparency in the selection of the human agent, and also highlighting reasons for forgoing the human-based rules.

* In the joint value of information method <cit.>, the three probabilistic models already introduced in the fixed value of information method described in the Section <ref> are trained together through a single neural network which includes a final Platt calibration layer that guarantees the estimation of meaningful expected utilities. Experimental findings show that joint learning yields greater advantages compared to analogous staged learning method.

* The Mixed Integer Linear Program (MILP) <cit.> is a scheme to exactly minimize the mis-classification error of the HS.

It comes with generalization bounds and allows to provably and easily integrate any linear constraints on the variables. However, it suffers from two limitations: it is computationally expensive and it does not generalize to non-linear predictors.

Multiple-Expert L2D (1 out of $J$)

In this first scenario of L2D-ME, the goal of the multi-expert deferral policy $\rho^{ME}_M$ is to choose either the classifier or exactly one human agent from the set of $J$ available ones. Hence, $\rho^{ME}_M$ takes the form of $\rho^{ME}_M:\mathcal{X}\rightarrow\{0,1,\ldots,J\}$, where $\rho^{ME}_M(x)=0$ means that the classifier decides, while $\rho^{ME}_M(x)=j$ for $j\neq0$ indicates that the decision is deferred to the $j^{\text{th}}$ human agent.

<cit.> adopt a mixture of experts (MoE) approach whith the deferral policy serving

as a gating function that assigns each instance either to the predictor or one specific human agent. The joint learning of the classifier and deferral function is carried out through a surrogate loss function based on the negative log-likelihood of the system. However, subsequent work <cit.> has proven this surrogate to be not FC and proposed instead

two surrogate loss functions, namely one based on cross-entropy and one on the One-vs-All classification, which are consistent with the 0-1 loss in the L2D-ME setting and extend their single-expert analogue. The experimental findings indicate that the OvA-trained model frequently achieves superior performance compared to both the cross-entropy variant and the MoE baseline <cit.>. Additionally, it exhibits better calibration in terms of the correctness of agents' decisions.

Finally, <cit.> study the problem of L2D-ME in a bandit feedback setting (i.e., a sequential dynamic allocation problem). Specifically,

the deferral policy is specifically learned using a supervised learning model that has been previously trained on historical data that reflect human decisions and corresponding outcomes in order to maximize the complementarity of the machine and human agents. However, by doing so, it is assumed that the human agents who generated the historical data are the same individuals who will be assigned decisions at inference time.

Multiple-Expert L2D ($j$ out of $J$)

In the second scenario of L2D-ME, the multi-expert deferral policy is defined as $\rho^{ME}_M:\mathcal{X}\rightarrow\{0,1\}^{J+1}$. For each input $x\in\mathcal{X}$, the goal is to choose the committee of agents $C(x) \subseteq \{0,\ldots,J\}$, possibly including the classifier, who are likely to make the most accurate decision for $x$. Hence, the $i^{th}$ vector component of the deferral policy will be defined as $\rho^{ME}_M(x)_{(i)}=1$ for all $i\in C(x)$, and $\rho^{ME}_M(x)_{(i)}=0$ for all $i\notin C(x)$. In the event that the designated committee comprises multiple agents, the resulting outcome will be an aggregated decision.
This setting has been firstly addressed by <cit.>, who proposed a joint loss functions obtained by linearly combining the losses associated to the classifier and deferral function via context-dependent hyperparameters. The authors proved that the combined loss is convex with respect to the classifier and deferral function whenever the loss associated to the former is convex; under such assumption, it can be optimized using the projected-gradient descent algorithm. Additionally, the authors outlined a few adaptations of the L2D-ME framework to account for potential real-world constraints and requirements:

* Fair learning: this variant takes into account the possibility of performance discrepancies that may occur with respect to individuals belonging to different protected categories.

* Sparse Committee Selection: this variant enables the deferral function to exclusively choose a limited number of agents on a per-instance basis.

* Dropout: this variant aims to reduce the dependence on a single agent and achieve a more equitable distribution of workload.

* Regularized versions: additional constraints can be added to the joint framework as regularizers of the loss function. For instance, this solution can be employed in cases where specific costs associated with individual human agent consultations are provided.

Subsequent work <cit.> further developed this setting to adapt to closed deferral pipeline, wherein the human agents of the HS also provided the training labels.

This is achieved through an online framework in which

input samples are received in a continuous stream. After each prediction is made, which involves aggregating the outputs of agents in the chosen committee, the sample are utilized to retrain the classifier and deferral function.

Alternatively, <cit.> suggest to use Conformal Inference <cit.> to find ensembles of agents $C(x)$ that include the best agent with high marginal probability. The size of $C(x)$ is computed dynamically as a function of the input $x$, thereby ensuring optimal utilization of agent queries. The authors propose two test statistics for the estimation of $C(x)$: a naive score function that sums up the correctness scores of all agents who correctly predict the given instance, and a regularized statistics that employ conformal risk control <cit.> to increase the robustness to noise. The experimental findings demonstrate that the latter approach yields a nearly flawless identification of the appropriate number of agents. Moreover, the conformal approach exhibits superior performance in system accuracy compared to a fixed-size ensemble of agents.

§.§.§ Further model architectures

While most of the proposals documented in the literature can be categorized as staged or joint learning models, a few exceptions also exists. A notable example often used as a baseline in the L2D literature is the method proposed by <cit.>, namely, an iterative algorithm that optimizes the classifier and triage policy alternately. At each iteration, the optimization process is carried out for the classifier on instances where it outperforms the human agents, while the remaining data points are optimized for the triage policy. The authors show that their method converges to a local minimum. Nevertheless, subsequent experimental studies have shown that this method exhibits lower performance in comparison to other L2D algorithms <cit.>. Additionally, similar algorithms have been implemented

to address the issue of L2D for model-specific settings, namely Support Vector Machines <cit.> and Ridge Regression <cit.>.

§.§.§ L2D with limited human predictions

A significant drawback of L2D is the requirement of human predictions, alongside ground truth labels, for every instance within the training set <cit.>. Ideally, the L2D system has to be trained on human labels belonging to the same human that will then interact with the system itself. By doing so, the L2D system will learn to complement that specific human <cit.>. Due to the significant computational and human costs, it is likely that the implementation of the L2D algorithm would be impractical for most real-world scenarios.

A couple proposals have been put forward to address this problem and implement L2D-SE algorithms with only a limited amount of human predictions. <cit.> derived an active learning scheme known as Disagreement on Disagreements (DoD) which can learn a classifier-rejector pair by making a minimal number of queries to the human. The DoD algorithm functions in two steps: i) first, a standard active learning algorithm (i.e., CAL <cit.>) is executed on the hypothesis space to learn the human disagreement with the ground truth: at each round, the disagreement set of the predictors is computed, and then the human is queried on the instances in this set to learn their error boundary; ii) second, a consistent classifier-rejector pair is learnt from the pseudo-labeled data derived from the preceding stage.

Alternatively, <cit.> proposed a three-step methodology that, based on a limited set of human predictions, generates synthetic human labels that simulate their capabilities. The first step consist in training an embedding model that maps each instance into feature representations. These, along with the ground truth labels, are used in the second step to learn an expertise predictor model, which is designed to approximate the capabilities of a human agent based on their labeling behavior. To harness the potential of both labeled and unlabeled instances, this component incorporates a semi-supervised learning technique. Finally, the expertise predictor model produces synthetic human predictions for instances that have not been labeled by a human and that can thus be used as input of any downstream L2D algorithm. The results of the empirical analysis shows that a small number of human predictions per class is enough to effectively generate synthetic predictions.

§.§.§ Strength and limitations of Learning to Defer

In contrast to algorithms that operate under oversight (see Section <ref>), Learning to Abstain Hybrid Systems are trained not to predict when their performance is weak. As a result,

when using an L2R or L2D algorithm to make decisions, one can expect to receive two kinds of evidence: the machine's prediction concerning the action of deferral, and, if the AI does not abstain, the result of the prediction task.
L2D algorithms improve upon L2R by incorporating a representation of human knowledge directly in the training process.

In such a way, the deferral policy is trained to adapt to both the AI model and the human decision-maker, ideally the same that will employ the Hybrid System.

Recent empirical investigations involving human subjects yielded evidence for the additional advantages that abstaining systems bring to Hybrid Systems. <cit.> found that such algorithms improve both human task performance compared to a human or an AI working alone,

and human task satisfaction compared to a human working alone.

A different study <cit.> also investigated the effects of employing abstaining Hybrid Systems on the human

perception of AI performance and credibility. The results indicate that users are frequently influenced by the system's recommendation also on ambiguous instances, even without conscious awareness, and thus support the adoption of L2D algorithms.

However, L2D also comes with several limitations  <cit.>. Most importantly, these include: data availability issues, which are primarily due to the need of human predictions in addition to the ground truth for all instances within the training set and all human agents involved in the Hybrid System;
and fairness concerns that may stem from the introduction of bias by both human and machine agents, as well as from the abstention mechanism itself <cit.>. Although there have been suggestions to deal with such issues,

these proposals still do not offer a straightforward solutions.

In terms of the human's role in the Hybrid System, Learning to Abstain enhances only marginally the paradigm of human oversight over machines (Section <ref>), as the deferral is exclusively a machine-side operation and there is no direct human-side interaction considered in the design of the algorithms.

§ LEARNING TOGETHER: HUMANS TEACHING MACHINES, AND MACHINES TEACHING HUMANS

The next natural step in hybrid

is a two-way collaboration in which human agents are not mere executors or overseers, but can directly interact with the machine to best infuse their human decision-making abilities directly into the machine.
While Learning to Defer aims to identify which agent is best suited for a given

there is little to no effort in integrating the decision-making abilities of one agent to the other.
In such a setting, the

capability of the overall system is at best but a sum of the

abilities of its agents, that is, the system is not synergic.
Usually, in machine learning machines learn directly from data, while in a Learning Together system they aim to learn from other agents.
To reach the ultimate goal of learning from other human agents, we need to build on top of a foundational stepping stone that enables machines to learn from other machines, the Teacher-Student paradigm.

§.§ Machines Learning from Other Machines: The Teacher-Student Model

At the basis of machine-to-machine learning systems is the Teacher-Student paradigm.
In this approach, we identify two agents: a Teacher, whose goal is to train, and a Student, whose goal is to learn from the Teacher.
Among Teacher-Student models, four are of particular note:
Learning with Privilege <cit.>,

Knowledge Distillation <cit.>,
Transfer Learning <cit.>,
and Active Learning <cit.>.

Learning with Privilege.
Initially introduced by  <cit.>, Learning with Privilege is defined as a learning paradigm in which a teaching agent $f_T$, i.e., a Teacher, provides the learning agent $f_S$, i.e., the Student, with additional “privileged” information about the input data which is not present in the data itself.

The additional data is considered privileged because only available at training time.
In our formulation, this is roughly equivalent to integrating a proxy $\widetilde{Z}$ of the human knowledge $Z$ into the training data.
At learning time, a privileged estimator implements a function

\begin{equation*}
\mathcal{F}_M: \mathcal{X} \times \mathcal{Z} \rightarrow \mathcal{Y_M},
\end{equation*}

which directly integrates the human knowledge into the model.
Integration relies on a machine-specific encoding function that allows to map the input data $X \in \mathcal{X}$ and the human knowledge $Z \in \mathcal{Z}$ into the input space $\mathcal{X}$.
At inference time, the privileged information is unavailable, and the encoding function reduces to its projection $(x,\{\emptyset\})\mapsto x$,

thus allowing the machine to seamlessly operate either with or without privileged knowledge.

Knowledge distillation.
Knowledge distillation

aims to distill a given Teacher into a more suitable Student, usually to reduce the running cost of inference <cit.>.
Unlike Privileged learning, here the privileged information is exclusively derived from a machine, usually involving its internal state, which the Student tries to emulate.
Formally, at training time the Student is directly conditioned on the parameters of the Teacher, thus yielding an estimator $f_\theta$ of the family:

\begin{equation*}
\mathcal{F}_{M,S}: \mathcal{X} \times \mathcal{F}_{M,T} \rightarrow \mathcal{Y_M},
% f_{\theta \mid \theta'}(X).
\end{equation*}

where we assume without loss of generality that the Teacher and Student are members of the same hypothesis space $\mathcal{F}_M$.
The Student loss $\widetilde{\mathcal{L}_S}$ is thus augmented with an additional component $\widetilde{\mathcal{L}}_{T,S}(f_T, f_S)$ penalizing a distance between Teacher and Student:

\begin{equation*}
\widetilde{\mathcal{L}_S}(X, Y) = \frac{1}{n}\sum_{i=1}^n\mathit{\widetilde{\mathscr{L}}(Y_i,f_S(X_i))} + \lambda_S \widetilde{\mathcal{L}}_{T, S}(f_T, f_S),
\end{equation*}

for a predefined weight $\lambda_S$ balancing task performance and distance from the Teacher.
In neural models, where Knowledge Distillation is prevalent, the Teacher-Student loss $\widetilde{\mathcal{L}}_{T, S}$ is usually implemented as $d(\theta_T, \theta_S)$, e.g., Euclidean or Cosine distance, between the parametrization $\theta_T$ of the Teacher and the parametrization $\theta_S$ the Student.
In both Learning with Privilege and Knowledge Distillation settings, the final goal is to transfer information between Teacher and Student.

Transfer Learning.
In this the stpe where the Students tries to leverage the knowledge of the Teacher, to then adapt it to the specific task at hand <cit.>.

This paradigm is typical of dynamic settings in which tasks or data distributions change rapidly, yet they are all strongly related.
In this context, learning from agents which have already learned to solve related task is of great benefit.

Active Learning.
The aforementioned approaches all provide a straightforward and direct way to inject learning capabilities from the Teacher to the Student.
However, the latter remains a passive agent in this process, resulting in a unidirectional connection between the two.

Furthermore, knowledge is injected one-shot at training time, that is, these paradigms are not designed for subsequent Teacher-Student interactions.
Active learning <cit.> builds on this paradigm by empowering the Student to actively seek the most relevant information.
Like a student that poses questions to a teacher, an Active Student queries the Teacher (in our formulation, the human agent) to maximize its own performance.
Formally, the Student has at its disposal a pool of unlabeled data $X_U \in \mathbb{P}(\mathcal{X})$ that can, in principle, be labeled with the predictions of the Teacher.
Querying the Teacher is feasible but costly, thus one wishes to maximize Student performance while staying under a given budget.
To tackle this problem, we aim to learn a query policy

\begin{equation*}
\pi: \mathcal{F}_M \times \mathrm{P}(\mathcal{X}) \mapsto \mathrm{P}(\mathcal{X})
\end{equation*}

that, given the Student $f_S$ and the unlabeled data $X_U$, selects a subset $\hat{X}_U \subset X_U$ for which the Teacher then provides labels $\hat{Y}_U = \langle f_T(x_U) \rangle_{x_U \in \hat{X}_U}$.

Training the Student involves an iterative process of querying and training, and in each iteration the Student seeks to best select the subset of instances $\hat{X}_U \subset X_U$ that will maximize its performance.
$X_U$ can vary in its nature: it can be static or dynamic <cit.>, it can be updated with a handful <cit.> or a large set of novel unlabeled samples <cit.>, and the Student may even populate $X_U$ itself.
Initial formulations of Active learning <cit.> strongly resembles Learn to Reject solutions,as the goal of both is to identify regions in the input space $\mathcal{X}$ where the machine is less accurate.

Teacher-Student Model: Limitations.

The Teacher-Student paradigm is limited to one-way learning between agents, which makes it extremely flexible, as it does not have to model the interaction between agents, nor the agents themselves.
This introduces several layers of complexity, each compounding on the previous, making for novel approaches to Learning Together, which takes inspiration from the aforementioned paradigms.

§.§ Learning Together

Communication language.     2cThe communication language that allows human and machine to interact.

Hard reasoning          First-Order logic.     <cit.>

Soft reasoning          Logic-like languages with soft reasoning engines.     <cit.>

Explanations            Feature relevance, decision rules, and concepts.     <cit.>

Artifact type.     2cThe type of interaction artifact.

Intrinsic               Artifact internal to the machine, the human agent cannot directly influence the machine.     <cit.>

Extrinsic               Artifact external to the machine, the human agent can directly influence the machine.     <cit.>

Time of interaction.     2cThe machine phase in which the interaction occurs.

Training                 The human agent influences the machine directly in its training process.     <cit.>

Inference                The human agent influences the machine at inference time.     <cit.>

Learning cost.     2cThe cost incurred by the machine to integrate the feedback by the human agent.

Training                The correction prompts an additional training phase of the machine.     <cit.>

Null                    The correction does not cause any significant cost to the machine.     <cit.>

Properties of systems in the Learning Together paradigm.

Unlike the aforementioned Teacher-Student paradigm, the Learning Together paradigm aims to

create a two-way street in which human and machines can communicate effectively, one learning from the other: the machine explaining its prediction to the human, and the human explaining its prediction to the machine, improving the overall performance of the system.

In this context, improvement can take many forms.
For the machine, it can

provide more
accurate predictions, a better ability to generalize, a shallower learning curve, or higher level of transparency.
On the other hand, human agents can derive additional advantages from exerting

a tighter grip on how the machine solves the task, rather than simply solving the task

in place of the machine.

[colback=blue!5!white,colframe=blue!60!white,title=Example of online content moderation: Learning Together]
In Learning Together systems, users may interact with a feature-importance based explanation about messages.
This would allow them to adjust the importance of different terms based on their

own judgement
After receiving corrections from humans, the machine could integrate them to enhance the learning process and better flag future posts.

Generally, a machine implementing this paradigm partially integrates a human agent with expertise $Z$, thus yielding an estimator of the form:
$f_{\theta}(X, Z)$.
For the sake of simplicity, here we model a single human agent, but the formulation can be extended to multiple agents.

We characterize systems in the Learning Together paradigm according to four properties that we summarize in Table <ref>: communication language, which guides the communication between agents;interaction artifacts, which guide the feedback mechanism; time of interaction, which defines when the interaction occurs; and learning cost, which defines what cost the interaction creates.

§.§.§ Communication Languages and Interaction Artifacts

The communication language plays a critical role in facilitating successful high-level human-machine interactions.
On one hand, its level must be sufficiently low to allow the machine to understand and interface with it; on the other hand, its level should be high enough for the human agent to understand it.
Communication languages are comprised of two intertwined components: a language, which defines how the agents communicate, and a set of interaction artifacts, which comprise the atomic unit of interaction, and enable the feedback of one agent to be integrated into the other.

Artifacts. At the core of a communication language lies a set $A$ of interaction artifacts that enable effective communication between humans and machines, and enable the embedding of human component $Z$ into the machine.

That is, the human encodes

their knowledge by acting on the interaction artifacts $A \in \mathcal{A}$ presented by the machine.
Then, the machine integrates the human feedback by integrating the action of the human agent on $A$.
Typical artifacts include logic rules, relevant features, and decision rules.
The human action can also take several forms, and is inherently dependent on the type of the interaction artifact.

Regardless of their nature and formulation, artifacts are:

* understandable by both the human and the machine, since they must enable communication;

* malleable by the human agent, as they transfer their expertise $Z$ to the machine  by acting on the artifact;

* embeddable into the machine, since the human action on the artifact is to be integrated into the machine.

The last property puts a strong constraint on machines, which, unlike artifacts, tend to employ subsymbolic, e.g., neural, rather than symbolic models.
This often results in systems where the family and architecture of the machine are all but determined by the choice of artifacts, thus communication language and machine end up being tightly coupled.

Interaction artifacts come in one of two forms: intrinsic and extrinsic, yielding intrinsic and extrinsic machines, respectively.
Intrinsic artifacts are machine-oriented, thus after the human acts upon them, the integration step is hidden to the human agent, who has no direct control of how the machine integrates the action.
This often allows easier definition of the interaction artifact set at the cost of the effectiveness of the action.
The result of an interaction between an intrinsic machine parametrized by $\theta$ and a human agent acting on artifact $a$ is in an update of the model parameters $\theta$, which yields an estimator of the form

$f_{\theta \mid a}(X)$,

where $\theta$ is conditioned on the artifact $a$ acting as proxy feedback for the human agent.
Intrinsic artifacts are most often employed for systems using explanations as communication language.

Extrinsic artifacts instead, are human-oriented, thus once the human acts upon them, the integration is straightforward for the machine, which does not require additional integration steps.
Typically, such artifacts are organized in an artifact bank $B$ that the user can inspect and act upon.

With an artifact bank to attend to, the interactive machine $M$ implements an estimator of the form

$f_\theta(X, B)$.

Note that the whole artifact bank is a parameter of the machine, hence the parameters $\theta$ of the machine are independent of the artifact bank,

Extrinsic artifacts are typically more complex than intrinsic ones, and so are the communication languages built upon them.

Communication languages.
Communication languages can be broadly categorized in two families: reasoning languages, both hard and soft, and explanation languages.

Hard reasoning languages: Logic.
Turning to languages, logic natively offers languages and artifacts that satisfy all three requirements.

Logic programs are naturally readable by humans due to their transparency and similarity to human reasoning, and thus are a suitable candidate for communication language.
They are also naturally malleable but it is not straightforward to embed them in the machine due to their symbolic nature, which is in stark contrast with the subsymbolic nature of the vast majority of machines.
Learning Together systems using logic as a communication language are strongly coupled and often neuro-symbolic in nature, that is, they combine subsymbolic and symbolic components.
The subsymbolic component is typically a neural one, e.g., a neural network, which fully embodies machine reasoning.
On the other hand, the symbolic component is geared towards the human, and thus employs a logic language.
Neuro-symbolic systems aim to combine and integrate these two components.

Generally, logic and reasoning-like languages define a set of clauses or rules $\mathcal{R}$, and facts $\mathcal{T}$ that allow a
logic engine to reason and present its logical derivations to the human agent as extrinsic artifacts on which they can act.

Possible actions on logic artifacts, i.e., logic rules, facts, and compositions of the two, include addition and deletion of new/existing rules or facts.
Hard logic enjoys strong theoretical properties, which makes it suitable for highly compliant systems where providing feedback to the machine is highly likely to result in a successful integration in the machine.

Logic as a communication language is expert-driven, rather than data-driven.
This means that artifacts in these languages are often hand-crafted, domain-specific, and automating their definition

usually involves a significant effort on the human side.

An example of Question Answering machine employing a hard reasoning language. The agent maps the question to a program by using a set of primitives (right-hand side of the picture), and then executes the program on the input (left-hand side of the picture), providing the human agent with both a prediction, and a malleable program that the user can correct.

Figure <ref> provides an example extracted from the CLEVR dataset <cit.>.
Here, the task is to answer questions about the provided image, e.g., “What is the color of the cube to the right of the yellow sphere?”
The interaction artifact consists of a program solving the task by first identifying the yellow sphere, then finding objects to its right and finally filtering them by shape.
The intermediate result of this program is then used to extract the color of such object.
When presented with it, the human agent can inspect and appropriately modify the program to solve the task.
We can find another classical example of hard reasoning in <cit.>, where the machine is tasked to predict the final outcome of a Tic-Tac-Toe board by leveraging a set of logic rules, which the human can inspect and change at their leisure.

Neurosymbolic models also offer another related subfield, neural program synthesis.
Like inductive logic programming, neural program synthesis tries to solve tasks by learning interpretable programs, only in this case the program is not expressed in a logic language, but in an actual programming language.
DreamCoder <cit.> is a neurosymbolic model to solve tasks by writing computer programs in a given minimal computer language.
Notably, DreamCoder progressively grows a library of functions, small reusable snippets of computer code re-used throughout the program.
A human agent can then inspect and change functions in the library that they deem incorrect.

Soft reasoning languages.
Soft reasoning communication languages improve the flexibility of Logic languages at the cost of their strong theoretical properties by replacing logic rules and facts with logic-like rules, and the symbolic reasoning engine with a subsymbolic approximate one.
Like in hard reasoning languages, possible actions on artifacts include addition and deletion of new/existing rules or facts.
These languages are almost exclusively Natural Languages, and are themselves applied to Natural Language models.

Originating from the work by , this family of machines builds on top of two other families, knowledge injection models <cit.> and soft reasoners <cit.>, the former providing models able to integrate external possibly human-sourced knowledge in their inference, and the latter providing soft reasoning engines for Natural Language.
Due to their strong emphasis on flexibility,

soft reasoning languages are usually loosely defined in terms of a set $\mathcal{T}$ of facts and informal rules $\mathcal{R}$.

On inference, the machine presents the human agent with a subset $T \subset \mathcal{T}$ of facts, optionally accompanied by a set of derivation rules $R \subset \mathcal{R}$ it leveraged.
According to the complexity of the task, facts may be enriched with intermediate reasoning steps that the machine has derived or generated to solve the task.

[node distance=1cm, auto,->,shorten >=2pt,

[shape=rectangle,draw=black,fill=myblue,fill opacity=0.25,text opacity=1] (h1) at (-3,0) Can a magnet pick up a penny?;
[shape=rectangle,draw=black,align=center,fill=myred,fill opacity=0.25,text opacity=1] (m1) at (0,-1) Yes: i) pennies are made of metal,
ii) metals are magnetic.;
[shape=rectangle,align=center,draw=black,fill=myblue,fill opacity=0.25,text opacity=1] (h2) at (-3.0,-2.2) Not all metals are magnetic and
copper is not magnetic.;

[shape=rectangle,draw=black,fill=myblue,fill opacity=0.25,text opacity=1] (h3) at (-3,-3.2) Can a magnet pick up a penny?;
[shape=rectangle,draw=black,fill=myred,align=center,fill opacity=0.25,text opacity=1] (m2) at (0.4,-4.2) No! Pennies are made of copper,
which is not magnetic.;

[database] (db1) at (6,-0.6) Pennies are made
of metal
Metals are magnetic;
[database] (db2) at (6,-2.1) Updating...;
[database] (db3) at (6,-4.25) ...
Copper isn't magnetic;

[->] (db1.west) |- (m1.east);
[->] (h2.east) |- (db2.west);

[->] (db2.south) – (db3.north);
[->] (db3.west) |- (m2.east);

An example of a Question Answering hybrid system employing a soft reasoning language. Unlike hard reasoning languages, the human agent can interact with the machine through Natural Language.
Here, the user is first provided with a prediction and a rationale by the machine, and is able to correct the rationale, feed it back to the machine, which updates its artifact bank (denoted as “Feedback memory” in <cit.>).
On a subsequent interaction, the machine is presented with the same question, but this time it provides the correct prediction and rationale by leveraging the feedback provided by the human agent in the previous step.

Figure <ref> shows an example on a Question Answering task where the hybrid system is tasked to answer basic physics questions by the user <cit.>.
The machine is asked whether “A magnet can pick up a penny”, and generated both an answer, “Yes”, and an artifact supporting it in the form of syllogistic reasoning.
The human agent notices a flaw in the premises, corrects the artifact by stating that there are non-magnetic materials, one of which is copper.
Here, the integration step is minimal, and consists in adding the correction by the human agent to the artifacts bank.
Then, on a successive iteration, the machine is presented with the same question, and successfully answers by leveraging the claims introduced by the user, without incurring any additional training cost.

Other works operate on similar terms by providing full reasoning trees <cit.>,

improving on the artifacts by artificial generation <cit.>,
or by providing ad-hoc artifacts to correct the machine <cit.>.
The human agent can act upon them by correcting any number of facts and/or rules used in inference, yielding updated facts $\mathcal{T}'$ and $\mathcal{R}'$.

Another clear advantage of soft reasoning languages is the ability to integrate artifacts from multiple sources to aid reasoning.
Knowledge Graphs are a primary source for several reasons:

they are easily available, enjoy widespread use, can be mined from Natural Language, cover a wide range of domains, and are immediately understandable to human agents, thus providing high understandability and malleability of the artifacts.
Knowledge Graphs can also be easily turned into Natural Language through a process of verbalization in which the Knowledge Graph is encoded into a set of Natural Language claims.
Improving on Natural Language, they are usually verified by a wide pool of users, hence they make for reliable sources which require minimal to no verification.
This last property is particularly of interest because it allows the system to scale to more human agents than the ones who are directly designing the system.
To further strengthen the case for Knowledge Graphs, their integration into machines has been a well-studied problem for several years, and already presents several effective solutions <cit.>.

Knowldege Graphs also allow some basic form of reasoning which can be easily extended: they provide a strict reasoning, both commonsense <cit.> and factual <cit.>, all of which can be integrated in Natural Language.

Question: What can the red object on the ground be used for?
Answer: Firefighting.Support fact: Fire hydrants can be used to fight fires.

An example of a Question Answering hybrid system employing a soft reasoning language and leveraging Knowledge Graphs. Here, the machine consults its artifact bank to retrieve a supporting fact for its prediction, and provides it to the human agent alongside its prediction.

Figure <ref> presents an example from <cit.>.
Here, the goal is to understand what the purpose of the red object (the hydrant) in the photo is.
We ought to remark the importance of the context, since even though the machine is aware of the concept of “fire hydrant”, it is highly unlikely to have seen either a fire hydrant or a fireman in a forest.
Moreover, since the image itself does not provide any information regarding the purpose of the hydrant, one should expect that the machine could not solve the task without the additional information provided by the human agent in the Knowledge Graph.

Explanations are designed to explain the machine, which makes them highly understandable artifacts out of the box.
Common families of explanations include counterfactuals, prototypes, feature relevance, and decision rules, the last two leveraged in

Learning Together
Feature relevance provides the human agent with the estimated influence that each input feature has on the prediction of the machine: the higher the relevance, the higher the sensitivity of the machine to changes in that feature.
Decision rules, instead, provide a descriptive understanding by

articulating with logic rules the predictions of the machine.

It is worth to mention that explanations are often extracted post-hoc and in a model-agnostic fashion, i.e., after the machine has been trained and regardless of its form.
Jointly, these two characteristics minimize coupling between the interaction artifacts and the machine, thus granting more flexibility in the design of the machine.

Some more recent proposal propose conversational agents that are able to provide an interface for the human agent to best query the machine for explanations <cit.>.
On this account, explanations are major artifacts of interest, particularly for systems in which the interaction is

the focal point, such as

Interactive Machine Learning <cit.> and eXplainable Interactive Learning (XIL) <cit.> systems.
In particular, the latter allow the human agent to inspect and provide feedback to the machine

by correcting its explanation.
A XIL machine is based on a simple algorithmic kernel comprised of five steps:

* the machine performs a learning step by optimizing its parametrization $\theta$;

* the machine generates explanations $A_M$ of (a subset of) its predictions;

* the human examines $A_M$, and provides a (optionally) corrected explanation $A_H$;

* the machine performs a learning step by optimizing $\theta$ according to the corrected explanation;

* if no stopping criterion is met, the machine returns to step (1).

By iteratively querying the human, the machine parametrization is thus conditioned on the corrected explanations $A_H$, that is, a XIL machine ends up implementing an estimator of the form:

$f_{\theta \mid A_H}(X),$

where the correction $A_H$ acts as a proxy for the human knowledge $Z$.

Integration is directly dependent on the family of explanations.
By far the most widespread one is feature importance, which assigns a relevance score to each element of the input data $X$.

The interaction then consists in a possible correction of the relevance scores, with the human agent activating or deactivating each feature according to their judgement.

Integration follows either a learning approach, in which the correction is directly encoded in the machine training objective <cit.>, or a generative one, where the correction is implemented via training on additional synthetic data <cit.>.
In a learning approach, the corrections of the human agent are encoded in a correction matrix $C \in \{0, 1\}^{n \times m}$ that states what feature relevance have been corrected for each single instance.

In a generative approach, $C$ is instead used to generate synthetic data $\widetilde{X}$ to further train the machine.
Features with low relevance have randomized or copied values, while features with high relevance are kept as-is <cit.>.

A small subset of machines are designed to explicitly encode feature relevance in their architecture, thus allowing direct manipulation by human agents <cit.>.

The same two approaches are also found in two other explanation types, namely Decision Rules and Concepts.
In the former case, we have a generative approach in which the human agent is tasked with directly creating the additional synthetic data themselves <cit.>.

In the latter, we have a learning approach in which the explanation is itself a component of the architecture, thus the correction is again an added component of the machine objective <cit.>.

A third emerging approach, mainly aimed at Concept explanations, is the structured explanation approach, where explanations are provided as complex structures that the human can act upon.
<cit.> presents an application on concept hierarchies, where concepts are laid on a tree-like hierarchy such that the concept of a parent node, e.g., “Animal”, is a generalization of the concepts in its children, e.g., “Dog” and “Cat”.
The machine, based on a k-NN model, is tasked to solve two tasks: a downstream task, and a concept drift task, that is, to identify if the relationships within the structure have changed.
Once detected, the machine presents the concepts of interest to the human, who in turn corrects their structure, e.g., by removing or adding concepts, or by acting on the structure itself, i.e., removing or adding parent-child relationships between concepts.
The correction is integrated by removal/addition of appropriate instances from the training set of the machine, thus directly impacting the k-NN model.

§.§.§ Time of interaction.

Artifacts are integrated at different times in the lifetime of the machine, hence either at training or at inference time.
In the former case, the machine integrates the artifact at training time, and cannot be interacted with at inference time.
In this case, the human agent is effectively providing feedback only on training.
In the latter case, the human agent has more control on the machine, and receives and acts upon interaction artifacts at inference time.
Machines based on explanation languages, such as XIL, tend to provide training-time interaction, while more recent approaches based on hard or soft reasoning languages tend to provide inference-time interaction.

§.§.§ Cost of Learning

A critical distinction in Learning Together systems is the cost of learning, that is, the cost the system has to pay to properly integrate the actions of the human within the machine.
Systems tend to fall to the two ends of the spectrum.
In traditional approaches such as XIL, the interaction triggers a costly training step for the machine.
Here, the cost varies with respect to both the magnitude of the human agent correction and the intrinsic features of the machine.
Conversely, more recent approaches, such as extrinsic artifact-based systems, have no additional cost due to the nature of the machine itself.
Why then rely on traditional approaches if they incur in an inevitable additional cost?
Extrinsic artifacts have to be generated in the first place: the cost of populating the artifact bank is as inevitable as the training cost for traditional approaches.
As previously mentioned, when such banks already exist, e.g., in knowledge graphs, this cost can be greatly reduced.

§.§ Strengths and limitations

Unlike the Human overseers and Learn to Abstain paradigms, Learn Together systems integrate humans and machine in a fully hybrid system with bidirectional communication.
Thus, the two agents are rarely truly decoupled, their design and use being often task, domain, data and user specific.
Machines are limited by and tightly coupled with the language of choice, thus i) designing such systems requires significant ground-up effort, and ii) effective communication in one language does not appear to transfer to other languages.

This extreme heterogeneity partially hinders progress

, each system often yielding valuable insight only for future systems operating in similar tasks, domains, data type and desired user interaction.
In other words, Learn Together systems tend to improve vertically, their success being often unpredictable.

At the moment

, Learn Together systems are largely static: they are not able to switch language on demand, nor to adapt to different human agents, or defend themselves against possibly incorrect feedback provided by the human agents.
As a relatively recent development in Hybrid systems, they also tend to lack proper measures of validation tailored to them.
While classical validation measures apply to the system as a whole, as a predictive system, there appears to be little effort in properly assessing the compliance of a machine with the corrections provided by the human agent, and provide guarantees on the effects of such corrections.

§ LITERATURE GAPS

Hybrid Systems are a still novel topic, and there are several problems that are open to new research.

Following our taxonomy, we identify three categories of open challenges:

Machine-related problems. Unlike Learning Together systems, Human Oversight and Learning to Abstain systems still lack meaningful means of communication to engage with the human agent.

When presented with a prediction to oversee, or with an uncertainty estimate of such prediction, human agents are rarely also presented with suggestions on why the prediction should be accepted or rejected, or why the machine is uncertain of its prediction, let alone how to tackle the uncertainty itself.
We have highlighted some first steps in tackling this problem in Subsection <ref>, but this is far from a solved problem.

Human-related problems. Both Learning to Defer and Learning Together systems require a considerable human effort, i.e., a high volume of labels or artifacts, to be implemented.
This is a particularly taxing for the human agent, and while current solutions aim to tackle the problem with a given fixed budget, there are no clear solutions as to how to reduce such high cost.
When it comes to Learning to Defer, human-AI “collaboration” heavily depends on a global data annotation industry, wherein a vast and often invisible human labor force is engaged with tedious and taxing jobs.
Yet, the prevailing labor standards for data annotators are mostly characterized by lax regulations, low wages and few legal protections <cit.>.

Interaction-related problems. Related to the interaction itself, there is still little experimentation on understanding what language is best suited to which task, and how to adapt the language to different users within the same Hybrid System.
In particular, current hybrid systems are monolingual, and are developed with one language and one type of human agent in mind, thus lack the ability to adapt to different humans.
Underlying the development of such systems is the assumption that the human agents in the system and their capabilities and understanding are static, which is rarely the case.
As a result, hybrid systems lack flexibility and have little ability to adapt to the heterogeneity of the humans they may interact with.

When Hybrid Systems interact with several users, e.g. in Multiple-Expert Learning to Defer and in Learning Together systems, machine agents are rarely able to seamlessly handle higher multiplicity of agents, and basic concepts such as malicious agents who wish to poison the system, steer it in an undesirable direction, or simply introduce conflicts with existing artifacts are yet to be properly defined and studied.

§ CONCLUSIONS

Hybrid Systems, where humans and machines collaborate in predictive tasks, represent a new paradigm for how AI systems are designed and implemented.
Synergetically integrating these two types of heterogeneous agents allows to maximally exploit the strength of both, each also trying to overcome the weaknesses of the other.
In this survey, we presented a taxonomy of the fundamental literature for Hybrid Systems, categorizing the research in this field in three paradigms: Human Overseers, Learn to Abstain and Learn Together.
Each paradigm represents progressive steps towards greater human interaction with the AI system: while in Human Oversight the human merely verifies the final output of the system in Learn To Abstain, the human's knowledge is represented in the data and used to fit the AI agent specifically to the human expertise.
Finally, in Learn Together, the human actively corrects the reasoning AI agent.
We provided a thorough overview of the most important works for understanding each paradigm, and highlighted strengths and weaknesses of each approach.
We hope that this survey can serve as a strong foundation for future research on Hybrid Systems.

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

journaltitleLaw, Economics, and Business Fellows’ Discussion Paper Series
titleIf you give a judge a risk score: evidence from Kentucky bail decisions

journaltitleJournal of Public Administration Research and Theory
titleHuman–AI interactions in public sector decision making:“automation bias” and “selective adherence” to algorithmic advice

journaltitleAI Mag.
titlePower to the People: The Role of Humans in Interactive Machine Learning

booktitleECML/PKDD 2023
titleAnomaly Detection Via Few-Shot Learning On Normality

journaltitlearXiv preprint arXiv:2208.02814
titleConformal Risk Control

booktitle2020 International Joint Conference on Neural Networks (IJCNN)
titleGeneralized Neural Framework for Learning with Rejection

booktitlePredictive Policing and Artificial Intelligence
titleMachine learning predictive algorithms and the policing of future crimes: Governance and oversight

titleIronies of automation

booktitleAAAI Conference on Artificial Intelligence
titleIs the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork

titleBeyond Accuracy: The Role of Mental Models in Human-AI Team Performance

titleA Coverage-Based Utility Model for Identifying Unknown Unknowns

journaltitleJournal of the American Statistical Association
titleConvexity, Classification, and Risk Bounds

titleClassification with a Reject Option Using a Hinge Loss

journaltitleNeural Networks
titleDemocracy in neural nets: Voting schemes for classification

journaltitleData Min. Knowl. Discov.
titleHuman-in-the-loop handling of knowledge drift

titleFlexible Generation of Natural Language Deductions

journaltitleat - Automatisierungstechnik
titleInterpretable machine learning with reject option

booktitleInternational Workshop on Self-Organizing Maps and Learning Vector Quantization, Clustering and Data Visualization
titleProbabilistic extension and reject options for pairwise LVQ

booktitleSIGKDD 2006
titleModel compression

journaltitleProc. ACM Hum. Comput. Interact.
titleTo Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making

IEEE Computer Society

titleRejection Schemes in Multi-class Classification - Application to Handwritten Character Recognition

booktitleProc. of the 39th International Conference on Machine Learning
titleSample Efficient Learning of Predictors that Complement Humans

booktitleACL (1)
titleSay What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge

journaltitleIEEE Transactions on Information Theory
titleOn optimum recognition error and reject tradeoff

booktitleEMNLP (1)
titlePerhaps PTLMs Should Go to School - A Task to Assess Open Book and Closed Book QA

titleTransformers as Soft Reasoners over Language

journaltitle2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA)
titleProbability of default estimation, with a reject option

journaltitleMachine Learning
titleImproving generalization with active learning

journaltitleMach. Learn.
titleImproving Generalization with Active Learning

seriesSpringer Series in Statistics
titleAn introduction to statistical modeling of extreme values

booktitleInternational Conference on Algorithmic Learning Theory
titleLearning with rejection

Shaker Verlag

journaltitleSchriftenreihe des Fachbereichs Elektrotechnik und Informatik Hochschule Niederrhein
titleReject options and confidence measures for knn classifiers

Informa UK Limited

journaltitleJournal of the American Statistical Association
titleThe Well-Calibrated Bayesian

booktitleAAAI Conference on Artificial Intelligence
titleRegression Under Human Assistance

booktitleAAAI Conference on Artificial Intelligence
titleClassification Under Human Assistance

journaltitleIEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)
titleTo reject or not to reject: that is the question-an answer in case of neural classifiers

journaltitleJournal of Experimental Psychology: General
titleAlgorithm aversion: people erroneously avoid algorithms after seeing them err.

journaltitleManagement science
titleOvercoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them

titleDreamCoder: bootstrapping inductive program synthesis with wake-sleep library learning

journaltitlePersonality and Social Psychology Bulletin
titlePlaying dice with criminal sentences: The influence of irrelevant anchors on experts’ judicial decision making

journaltitleThe Conversation
titleHigh-tech surveillance amplifies police bias and overreach

titleEfficient rejection strategies for prototype-based classification

booktitleICANN '14
titleLocal rejection strategies for learning vector quantization

booktitleProceedings of ICML '19
titleOn discriminative learning of prediction uncertainty

booktitleInternational Joint Conference on Artificial Intelligence
titleHuman-AI Collaboration with Bandit Feedback

booktitleAdvances in Neural Information Processing Systems
titleSelective Classification for Deep Neural Networks

booktitleInternational Conference on Machine Learning
titleSelectiveNet: A Deep Neural Network with an Integrated Reject Option

titleCustodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media

journaltitleJournal of Behavioral and Experimental Finance
titleArtificial intelligence and machine learning in finance: Identifying foundations, themes, and research clusters from bibliometric analysis

titleDeep Learning

booktitleAdvances in Neural Information Processing Systems
titleSupport Vector Machines with a Reject Option

booktitleProc. of the 2018 World Wide Web Conference on World Wide Web
titleHuman Perceptions of Fairness in Algorithmic Decision Making: A Case Study of Criminal Risk Prediction

booktitleEquity and Access in Algorithms, Mechanisms, and Optimization
titleDimensions of Diversity in Human Perceptions of Algorithmic Fairness

journaltitleIEEE Intell. Syst.
titleFactual and Counterfactual Explanations for Black Box Decision Making

journaltitleACM Comput. Surv.
titleA Survey of Methods for Explaining Black Box Models

booktitleIUI 2022
titleBuilding Trust in Interactive Machine Learning via User Contributed Interpretable Rules

journaltitleProblems of Control and Information Theory
titleBayesian decision with rejection

journaltitleOpen Systems & Information Dynamics
titleThe Error-Reject Tradeoff

booktitleProceedings of the Thirty-First International Joint Conference on Artificial Intelligence
titleForming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts

journaltitleProceedings of the AAAI Conference on Artificial Intelligence
titleLearning to Defer with Limited Expert Predictions

booktitleProceedings of the 28th International Conference on Intelligent User Interfaces
titleHuman-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction

titleMachine Learning with a Reject Option: A survey

titlePrivacy, due process and the computational turn: The philosophy of law meets the philosophy of technology

journaltitleArtificial Intelligence Review
titleA Survey of Outlier Detection Methodologies

title(Comet-) Atomic 2020: On Symbolic and Neural Commonsense Knowledge Graphs

titleCLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

titleSelective Classification Can Magnify Disparities Across Groups

booktitleICLR 2021
titleSelective Classification Can Magnify Disparities Across Groups

journaltitleMachine Learning
titleNearest neighbors distance ratio open-set classifier

booktitleEMNLP 21
titleBeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief

journaltitleACM Trans. Knowl. Discov. Data
titleLeakage in data mining: Formulation, detection, and avoidance

journaltitleAI & society