# Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning

Oyvind Tafjord, Bhavana Dalvi Mishra, Peter Clark
Allen Institute for AI, Seattle, WA
<EMAIL_ADDRESS>
###### Abstract

Our goal is a question-answering (QA) system that can show how its answers are
implied by its own internal beliefs via a systematic chain of reasoning. Such
a capability would allow better understanding of why a model produced the
answer it did. Our approach is to recursively combine a trained backward-
chaining model, capable of generating a set of premises entailing an answer
hypothesis, with a verifier that checks that the model itself believes those
premises (and the entailment itself) through self-querying. To our knowledge,
this is the first system to generate multistep chains that are both faithful
(the answer follows from the reasoning) and truthful (the chain reflects the
system’s own internal beliefs). In evaluation using two different datasets,
users judge that a majority (70%+) of generated chains clearly show how an
answer follows from a set of facts - substantially better than a high-
performance baseline - while preserving answer accuracy. By materializing
model beliefs that systematically support an answer, new opportunities arise
for understanding the model’s system of belief, and diagnosing and correcting
its misunderstandings when an answer is wrong.

##  1 Introduction

Figure 1: Given a question, Entailer searches for an answer hypothesis that is
supported by an entailment proof. First it over-generates candidate proofs,
then it removes those that the model itself does not “believe” (i.e., confirms
via self-querying that it considers all the generated proof elements to be
true). Finally it selects the best verified proof. Multistep proofs are
generated by iteratively backward chaining on the premises (Section 3.2).

Although pretrained language models (PTLMs) have shown remarkable question-
answering (QA) performance, it is often unclear why their answers follow from
what they know. While there has been substantial work on training models to
also generate explanations for their answers Wiegreffe and Marasović (2021),
or produce them via few-shot prompting, e.g., “chains of thought” Wei et al.
(2022), those explanations may not be faithful (the answer does not
necessarily follow from them) and may not be truthful, in the sense that the
language model itself does not believe111 We here adopt a simple operational
definition of belief, namely that a model believes X if it answers ”yes” to
the question ”Is X true?”. Other definitions could also be used. the
explanation statements that it generated. Rather, our goal is to generate
answers that systematically follow from the model’s own internal beliefs,
materializing those beliefs as explicit statements that can then be inspected.
Such a capability offers new opportunities for understanding, diagnosing, and
ultimately correcting errors in a language model’s behavior.

Q: A magnet will stick to (A) a belt buckle (B) a wooden table (C) a plastic
cup (D) a paper plate A: (A) A magnet will stick to a belt buckle because A
belt buckle is sometimes magnetic. because Metal is sometimes magnetic. A belt
buckle is made of metal. A magnet will stick to magnetic metals. Q: You can
make a telescope with a (A) straw (B) Glass (C) Candle (D) mailing tube A: (D)
You can make a telescope with a mailing tube. because A telescope is made of a
tube for observing / seeing. A mailing tube is a kind of tube. Q: Quartz may
produce rainbows when light is shined (A) around the crystal’s area (B)
through any of its sides (C) in the room its in (D) in to a mirror at it A:
(B) Quartz may produce rainbows when light is shined through any of its sides.
because A rainbow is produced when light shines through a prism. because The
rainbow is made of all different colors in visible light. A prism can split
light into different colors. A quartz is a kind of prism.

Figure 2: Questions (from the OBQA dataset) and Entailer’s answers, showing
its chain of reasoning.

Our approach uses a combination of generation and verification, implemented in
a system called Entailer222Entailer data and models are available at
https://allenai.org/data/entailer. Chains are constructed by backward chaining
from candidate answers, recursively using a language model (LM) trained for a
single backward-chaining step. For each step, Entailer over-generates
candidate entailments, then filters out those that do not conform to its own
internal knowledge (“beliefs”) by self-querying, asking itself whether (a) the
generated premises (leaves of the proof step) are true, and (b) each
entailment step is valid (Figure 1). It then recursively backward-chains on
premises until the overall proof confidence cannot be further improved (or a
depth limit $d$ is reached). Finally, the candidate answer supported by the
highest-scoring chain of a reasoning is returned. As a result, the system has
materialized some of its latent knowledge from which the selected answer
follows. Most significantly, the resulting proof is thus both faithful (the
answer follows from the proof) and truthful (the proof reflects the system’s
beliefs), providing a previously unavailable window into the model’s beliefs
about the world and their implications, e.g., Figure 2.

To train the Entailer model, we use a combination of the existing
EntailmentBank dataset Dalvi et al. (2021), plus a new crowd-annotated dataset
that we construct by bootstrapping (train an initial model, generate candidate
entailment examples with it, then annotate those examples as extra training
data). The model is then frozen, and Entailer is then applied zero-shot to new
datasets, i.e., Entailer is a treated as a general-purpose, fixed model
specialized for reasoning, rather than requiring fine-tuning for new tasks.

We evaluate Entailer on two existing datasets, OBQA Mihaylov et al. (2018) and
QuaRTz Tafjord et al. (2019). We find that its reasoning-based QA accuracy is
similar to its direct QA accuracy, with the advantage that a supporting chain
of reasoning is also produced. We also perform a human evaluation, and find
that 70% of time users judge the chains to clearly show how an answer followed
from their premises, substantially higher than for explanations produced by a
comparable high-performance QA system, Macaw Tafjord and Clark (2021). Our
contributions are thus:

  * 1.

The first system to generate chains of reasoning showing how answers are
systematically implied by a model’s own internal beliefs, making relevant
model beliefs explicit. The chains are both faithful (the answer follows from
the reasoning) and truthful (the chain reflects the system’s own beliefs).

  * 2.

A new, crowdsourced dataset of multi-premise entailments, doubling the amount
of data available in EntailmentBank Dalvi et al. (2021), and including
examples of both positive and negative entailments (EntailmentBank only
includes positive examples)333The dataset is provided in the supplementary
material..

##  2 Related Work

Systematic Reasoning: Several recent systems have demonstrated the ability to
perform systematic reasoning directly over natural language (Natural Language
Inference Manning and MacCartney (2009)), namely deriving conclusions from
known facts via step-wise application of well-defined inference operations.
One approach is to retrain a black-box model end-to-end Clark et al. (2020),
but has been limited to small rulebases. An alternative approach, which we
follow here, is to have an outside loop around a model, where the model
generates individual inference steps (i.e., rules), and a controller chains
them together. SCSearch Bostrom et al. (2022), NLProofS Yang et al. (2022),
IRGR Ribeiro et al. (2022), ProofWriter (iterative version) Tafjord et al.
(2020), and Selection-Inference Creswell et al. (2022) do this in a forward-
chaining fashion, MetGen Hong et al. (2022) does this bidirectionally, while
Braid Kalyanpur et al. (2020) (like us) does this in a backward-chaining
fashion. In all these systems, the required facts were expected to be provided
explicitly to the model. In contrast, Entailer’s reasoning uses its own
internal, latent knowledge, as well as (optionally) externally provided facts.
LeapOfThought Talmor et al. (2020) demonstrated that reasoning with a
combination of implicit and explicit knowledge was possible for simple 1-step
inferences. We expand this for multi-step inference, and (unlike
LeapOfThought) have the system also explicitly articulate the implicit
knowledge it uses, and its chain of reasoning.

Recent work has shown that generating a free-form explanation (“chain of
thought”) before an answer can also improve performance on a variety of tasks
Wei et al. (2022); Cobbe et al. (2021); Nye et al. (2021). In these works,
however, the explanations are unstructured, and there are no claims of
faithfulness that the answer follows from the generation, nor that the
explanations themselves represent model beliefs.

Materializing a Model’s Internal Knowledge: Pretrained LMs contain a vast
amount of knowledge, and can be thought of as a kind of knowledge base to tap
into Petroni et al. (2019). Recent work has shown that this latent knowledge
can be materialized as explicit English sentences or a knowledge graph using
generative techniques, e.g., COMeT Bosselut et al. (2019), ParaCOMET Gabriel
et al. (2021). Our work with Entailer similarly materializes its latent
knowledge, but here in a goal-directed way, namely by producing a faithful
chain of reasoning from facts it validates (“believes”) as true to an answer.
This articulation can be seen as a kind of self-talk, where a self-generated
context can improve QA Shwartz et al. (2020). However, here our generations
are not used as context for opaque problem-solving, but are assembled into a
well-defined chain of reasoning.

Beliefs: We refer to the model’s factual opinions as “beliefs” rather than
“knowledge” because those opinions may be wrong. In general, an agent can be
said to believe p if it acts as if p was true Schwitzgebel (2019). Following
Kassner et al. (2021), we take a simple, syntactic operationalization of this,
namely the agent answers “yes” to the question “p?”, but also note that more
semantic versions could be used, e.g., the agent also answers “yes” to
paraphrases and implications of p. In general, models can sometimes be
inconsistent in their beliefs Elazar et al. (2021); Kassner and Schütze
(2020); Ribeiro et al. (2019). For our purposes here, we simply note that such
inconsistencies may occasionally exist, and that techniques for inconsistency
resolution could be applied in future to reduce these, e.g., Kassner et al.
(2021); Li et al. (2019).

Figure 3: An entailment tree is a set of multi-premise, 1-step entailments
(red boxes) showing how the hypothesis (root node, green) is entailed from the
leaf nodes (white). If all the leaf nodes are true, and all the 1-step
entailment relations are valid, then we say the tree is a valid chain of
reasoning for the hypothesis.

##  3 Approach

Like several previous systems (Section 2), Entailer treats reasoning as
Natural Language Inference (NLI). In NLI, the basic unit of knowledge is
(represented as) a sentence rather than a structure, and a proof444 We use the
word “proof” for convenience but note that the term is somewhat approximate,
as entailment “proofs” do not have the guarantees of formal, deductive proofs.
is a tree of multi-step, multi-premise entailments, e.g., Figures 2 and 3.

Within this framework, given a question, Entailer first generates candidate
answers, then tries to prove each one, selecting the answer with the highest-
scoring proof. We now describe these steps.

Angle Input $\rightarrow$ Output (example) $H\rightarrow P$ "H: A paperclip is
made of metal. P:" $\rightarrow$ "[PREMISE] A paperclip is made of steel.
[PREMISE] Steel is a metal." $H\rightarrow S_{d}$ "H: A paperclip is made of
steel. V:" $\rightarrow$ 0.995 $PH\rightarrow S_{e}$ "H: A paperclip is made
of metal. P: [PREMISE] A paperclip is made of steel. [PREMISE] Steel is a
metal. I:" $\rightarrow$ 0.998

Table 1: Examples of the three input/output angles used by Entailer. The first
generates a candidate entailment rule P$\vdash$H given H. The second and third
score whether each premise, and the entailment itself, is valid, using tokens
V/I in the input to indicate that $S_{d}$/$S_{e}$ is the desired output.
Figure 4: The entailment tree is grown recursively, the algorithm looking for
the best tree (maximizes the overall score of the root node). Each node has a
fixed, direct (“fast”) score (in red), and (for internal nodes) a proof
(“slow”) score (in blue) computed from its children. The overall node score
(highlighted) is the highest of the two. If expanding a node increases its
overall score (e.g., step 3), that increase is propagated upwards and
recursion continues. If expansions cannot improve a node’s score further
(e.g., steps 2 and 4), the expansions are pruned and that node becomes a leaf
(red bars).

###  3.1 Hypothesis Generation

Given a question, Entailer first generates candidate answers and converts
these into declarative hypotheses (e.g., “Is the sky (A) blue (B) yellow”
$\rightarrow$ { $H_{1}$ = “The sky is blue.”, $H_{2}$ = “The sky is
yellow.”).555Conversion of a QA pair to a declarative hypothesis D uses a
custom T5-11B model trained on the QA2D dataset Demszky et al. (2018). An
$N$-way multiple choice question yields $N$ hypotheses. A true/false question
yields 2 hypotheses. For open-ended questions, Entailer first collects $N$
candidate answers generated by an external source (Macaw Tafjord and Clark
(2021) using nucleus sampling Holtzman et al. (2019)) then forms $N$
hypotheses from them.

###  3.2 Generating Entailment Trees

####  3.2.1 Generating a Backward-Chaining Step

##### Models:

The core of Entailer is generating and validating a single entailment step
that entails a hypothesis. We define the following data types:

  * H:

A hypothesis (English statement) to prove.

  * P:

A set of premises {$p_{1}$,…,$p_{i}$} (sentences) that together may entail the
hypothesis H. Together, P and H form a one-deep entailment step, denoted by P
$\vdash$ H.

  * Q:

A question posed to Entailer.

  * A:

A candidate answer for consideration.

  * C:

An optional context (set of sentences) relevant to the problem. This allows
Entailer to also use external knowledge, if available, when generating a tree.

We train a model (details in Section 4) with the three input/output behaviors
below (optional inputs shown in parentheses):

  * $(QAC)H\rightarrow P$:

Given a hypothesis H, generate a set of premises P that may entail H

  * $(QAC)H\rightarrow S_{d}$:

Score whether the model believes that a hypothesis H (or premise $p_{i}$) is
true ($S_{d}>0.5$) or not, (i.e. perform yes/no QA). We call this the direct
score (range 0-1).

  * $(QAC)PH\rightarrow S_{e}$:

Score whether the model believes a candidate entailment (P $\vdash$ H) is
valid $(S_{e}>0.5$) or not, i.e., P validly entails H (range 0-1).

Examples of these three angles are in Table 1.

##### Algorithm:

To generate a single backward-chaining step we adopt an overgenerate-and-
filter approach, also found useful elsewhere Yang et al. (2022); Cobbe et al.
(2021); Li et al. (2022). First, given $H$, we use the angle $H\rightarrow P$
to generate a set of premises $P$ that may entail $H$. We then check that the
model believes all the premises $p_{i}\in P$ using the $H(=p_{i})\rightarrow
S_{d}$ angle, and that it also believes the inference step $P\vdash H$ itself
is valid (independent of whether the $p_{i}$ are true or not) using the
$PH\rightarrow S_{e}$ angle. The proof score, denoting how well the 1-step
proof supports the hypothesis, is the product of the premises’ and entailment
scores: [vskip=0mm,leftmargin=2mm]
$s_{r\textnormal{-}1deep}(H)=(\Pi_{i}s_{d}(p_{i})).s_{e}(P\vdash{}H)$ We
repeat this $k$ times using nucleus sampling to obtain a diversity of
alternative proof steps, and then select the highest-scoring one $P\vdash H$,
as illustrated in Figure 1.

####  3.2.2 Backward Chaining

This one-step backward chainer is embedded in a larger control algorithm that
grows a multi-step entailment tree, searching for the best tree. This
algorithm is illustrated in Figure 4 and described below. The full algorithm
is in Appendix A.

Each node $N$ in the tree has two associated scores, the direct (“fast”) score
$s_{d}$, denoting the model’s direct belief in $N$ (in red in Figure 4), and
(for internal nodes) the proof (“slow”) score $s_{r}$, denoting how
confidently the model can prove $N$ (in blue), computed from its children. The
overall score $s$ is the max of the two. The proof score $s_{r}$ is
recursively defined as the product of its children’s overall scores times the
entailment score: [vskip=0mm,leftmargin=2mm]
$s_{r}(N)=(\Pi_{i}s(p_{i})).s_{e}(P\vdash{}N)$ The algorithm starts with an
initial hypothesis node $H$, then iteratively expands each leaf node $N$,
looking for a proof that scores higher than the direct score $s_{d}$ of that
node. In other words, can the system prove $N$ with a more confidence than
simply “guessing” if $N$ is true? If it can, the node’s overall score $s$ (max
of the two) will increase, that increase is be propagated up the tree, and the
expansion is retained, e.g., step 3 in Figure 4. If expansions cannot improve
a node’s score further (e.g., steps 2 and 4), the expansions are pruned and
that node becomes a leaf (red bars in Figure 4).

Note that because premises are self-generated rather than externally provided,
this stopping condition has a different semantics to earlier work, e.g.,
Kalyanpur et al. (2020); Bostrom et al. (2021): Rather than stopping when
externally known facts are reached, Entailer stops when strongly believed
facts are reached, and more backward chaining can no longer improve a
hypothesis’ confidence.

This whole procedure is repeated for each candidate answer hypothesis (Section
3.1). Finally the system selects the answer corresponding to the hypothesis
with the top-scoring proof $s(H)$.

##  4 Model Training

The core of Entailer is the model for one-step inference (Section 3.2.1), with
the three functionalities listed in Table 1. As Entailer is intended to be a
general-purpose system, the model is trained one-time for these three
functionalities, and then frozen. It is then applied zero-shot to new
datasets, e.g., the evaluation in Section 5.

###  4.1 Data Sources

####  4.1.1 EntailmentBank

To train Entailer’s model, we leverage (the training partition of) the
EntailmentBank dataset Dalvi et al. (2021). EntailmentBank contains 1840
multiple-choice science questions (1313 in the training partition) along with
their correct answer, a hypothesis expressing the question and answer in
declarative form, and a multistep entailment tree expressing how the
hypothesis follows from a set of facts drawn from a corpus of science facts
(the WorldTree corpus Xie et al. (2020)). Using the notation introduced
earlier, each EntailmentBank example is of the form:
[vskip=0mm,leftmargin=2mm] $<Q,A,H_{0},\\{~{}(P_{i}\vdash H_{i})*~{}\\}>$
where ($P_{i}\vdash H_{i}$)* denotes a set of entailment steps forming a tree
(with root $H_{i}=H_{0}$), describing how the corpus facts entail the
hypothesis $H_{0}$.

####  4.1.2 Crowdsourced Data

EntailmentBank only contains positive examples of entailments. To obtain
negative examples, we first ran an earlier, positive-only-trained version of
Entailer to generate 1-step proofs of (hypotheses for) incorrect answer
options in EntailmentBank’s questions (4-way multiple choice), resulting in
(necessarily bad) proofs for the three false hypotheses. (This was done using
just the H$\rightarrow$P angle, without verification). Note that Entailer will
generate some kind of proof even for false hypotheses, e.g.,
[vskip=0mm,leftmargin=2mm]  A rabbit has six legs because: 1\. A rabbit is an
animal 2\. Animals have six legs These invalid proofs will be incorrect either
because a generated fact is false, and/or because the inference itself is not
a valid entailment. To distinguish these, we use crowdworkers to assess
whether the generated facts were true, and if the entailment itself was valid.
The 1313 questions result in 3939 proofs for false hypotheses. Dropping the
few with more than two premises (to simplify the crowdsourcing interface),
crowdworkers annotated 3673 proofs, using labels T/F/? for each premise and
T/F/? for the entailment itself. Each proof was annotated by 3 crowdworkers,
then an additional 3 workers provided additional annotations for cases with no
initial majority verdict. Dropping premises/entailments without a final
majority verdict, we end up with 7013 additional labeled premises for the
$H\rightarrow S_{d}$ angle, and 3391 additional labeled entailments for the
$PH\rightarrow S_{e}$ angle. The crowdworker interface is in Appendix B.

####  4.1.3 Optional Fields

We also augment the training data with duplicate examples but with additional,
optional input fields:

  * QA:

The input QA question/answer-option pair, as well as the hypothesis H

  * C:

A context consisting of up to 5 relevant sentences, allowing explicit external
knowledge to be provided if available.

To generate examples of C during training, we use a mixture of sentences drawn
from (a) the gold (target) entailment (i.e., the gold premises), and (b)
sentences retrieved from a corpus of similar-styled knowledge (namely all the
leaf sentences used in the EntailmentBank entailment trees), mixed in
different ratios so the model is exposed to a mixture of high and medium
relevance sentences.

In this work we do not use any context C at test time, but it is utilized in
concurrent work for providing feedback to the overall system Mishra et al.
(2022).

Further details of training are given in Appendix C1.

###  4.2 Model Details

We train a T5-11B multi-angle model, Entailer, following the multi-task setup
similar to Macaw Tafjord and Clark (2021) for the three functionalities
described in Table 1.666The scores use the token probability of generated
”true” or ”false” output text. Details of hyperparameters and other settings
are provided in Appendix C2.

##  5 Evaluation

Our goal is to generate answers supported by faithful, truthful chains of
reasoning, without significantly impacting performance. Our two corresponding
reserach questions are:

  * 1.

How does Entailer’s proof-based answer accuracy compare with the direct QA
accuracy (both zero-shot)?

  * 2.

How good are the generated entailment-based proofs? And are they better than
those produced by a purely generative model?

For the first question, we evaluate using two existing multiple-choice
datasets, namely OBQA Mihaylov et al. (2018) and QuaRTz Tafjord et al. (2019).
These datasets contain multiple-choice questions that (typically) require
multihop reasoning, rather than being simple lookup questions. We use the test
partitions777 For QuaRTz, for synchronization with other work Mishra et al.
(2022), we use a portion of the training set as our test partition here. As we
are applying Entailer zero-shot, this makes no material difference
experimentally. with sizes 500 questions (OBQA) and 557 questions (QuaRTz).

For the second question, we collect human judgements of whether the hypothesis
clearly follows from facts in the proof, and compare its proofs with
explanations generated by a high-quality baseline model, Macaw Tafjord and
Clark (2021).

Figure 5: QA accuracy of Direct QA, Entailer, and the two combined on two
datasets.

###  5.1 QA Accuracy

We evaluate Entailer’s proof-based QA accuracies under various conditions, and
compare them against its direct QA accuracy:

1\. Direct QA: Here we measure the model’s ability to directly answer the test
questions (zero shot) using the $H\rightarrow S_{d}$ angle. One can think of
this as the “fast thinking” answer. Note that this capability of the frozen
model was trained on the same data as the rest of Entailer, so is a fair
comparison.
2\. Entailer: Here we measure Entailer’s ability to answer the test questions
(again zero shot) by generating, scoring, and comparing entailment proofs for
each answer option. One can think of this as the “slow thinking” answer. We
vary:

  * •

maximum proof depth $d$ = 1 (blue in Figure 5) or 3 (green)

  * •

degree of search: (a) greedy: use the first ($k=1$) generated entailment at
each step, or (b) sampled top level: pick the best of $k=6$ entailments for
expanding the root hypothesis node. ($\approx$ six times more computationally
expensive than (a)).

Note that these proofs are are faithful explanations of why an answer was
chosen, as the selected answer by definition is always the one with the
highest-scoring proof.
3\. Entailer + Direct: Here we combine the two by selecting the overall most
confident answer of Direct and Entailer (using $c_{d}(H)$ or $c_{r}(H)$,
Appendix A). Here, the proofs are not always faithful explanations of an
answer, as the chosen answer may not be the one with the highest-scoring
proof.

###  5.2 Results

The results are shown in Figure 5, and suggest that:

Proof Scores are competitive with direct answer scores. In Entailer’s best
configuration (sampled top-level $k=6$, max depth $d=3$, last bar), its
reason-based answers have an accuracy of 75.2/75.4 for OBQA/QuaRTz
respectively, not significantly different to the Direct scores of 75.2/74.1.
This is important, as it suggests there is no significant accuracy penalty for
proof-based answers.
Sampling proofs helps: Using sampling for the top-level proofs (last 4 bars)
always outperforms greedy proof selection by a small amount.
Allowing deeper proofs does not significantly affect accuracy: Although deeper
proofs888The distribution of proofs with different depths 1, 2, and 3 was 162,
232, and 106 respectively for OBQA, and 123, 256, and 178 respectively for
QuaRTz. slightly help in the combination of Entailer+Direct, and may provide
more information to a user, the accuracy differences are not significant.

When Entailer + Direct are combined, by selecting the most confident answer
(last two columns), we lose the guarantee of faithfulness, as the selected
answer may not be the one with the highest-scoring proof. In practice, this
occurs for 16.8% of the questions (OBQA), 14.2% (QuaRTz). In addition, we find
this combination does not have significant performance gains, so has no
obvious advantage in these experiments.

Note that we are measuring zero-shot performance on our test datasets, so our
results are not comparable with leaderboard entries.999For a comparable
datapoint on OBQA, Brown et al. (2020) report zero-shot GPT3 scores of 57.6%
and few-shot 65.4% on OBQA, so Entailer’s $\sim$75% scores are a strong zero-
shot baseline for an 11B model. More recent work using rationale-augmented
prompt ensembling with much larger models Wang et al. (2022) has reached few-
shot (with rationales) scores of 91.0% (PaLM-540B) and 88.4% (GPT3-175B) on
OBQA. More importantly, though, our goal is not a state-of-the-art zero-shot
model, but rather a model that can show how answers systematically follow from
its own internal beliefs. Our results suggest this is possible with high
reliability, and, as an additional bonus, without loss of zero-shot accuracy.
Thus, rather than just answers, we now get answers supported by faithful
chains of reasoning.

###  5.3 Human Judgements

For our second question of evaluating the quality of Entailer’s proofs, we
compare against explanations generated by Macaw,101010 Note that comparing
with a baseline explanation generator trained on our data, i.e., just using
$H\rightarrow P$ angle without verification, would not be informative, as such
explanations would necessarily be worse: By definition, the verifiers remove
explanations with either (believed to be) false premises and/or bad
inferences, hence removing the verifiers will increase the frequency of false
premises and bad inferences (confirmed through sample analysis). Hence we
instead use a strong, external system as a comparative reference point. a
public, state-of-the-art QA system with explanation capability Tafjord and
Clark (2021). Examples of explanations from both systems are in Appendix D.
Six annotators compared 1-deep Entailer proofs with explanations from Macaw,
scoring each along four dimensions, and then comparing the two:

  * 1.

Does the conclusion clearly follow from the premises?

  * 2.

Are all the premises correct?

  * 3.

Are all of the premises relevant?

  * 4.

Does the explanation introduce something new and useful, i.e., does more than
just restate the conclusion in different words?

  * 5.

Which explanation do you prefer?

Answer options for questions 1-4 were yes/somewhat/no, and for question 5 were
first/similar/second. The ordering of explanations were scrambled so the
annotators did not know which explanation was which, and in fact they were
unaware that the different explanations had been generated by different
systems. We collected annotations for 100 pairs of explanations for correct
answers to 100 OBQA dev questions. The annotators were recruited from our
institute, spanning a broad range of age (20-60) and experience.

The results (Figure 6) suggest several findings:

Figure 6: Users’ assessments of Entailer’s proofs (red), compared to Macaw’s
explanations (blue), showing percent of times annotators answered “Yes”.
Entailer’s conclusions were judged to “clearly follow from the premises” in
over 70% of the proofs (first bar), substantially more than Macaw’s
explanations (34%).

1\. In absolute terms, Entailer’s conclusions were judged to clearly follow
from the premises in the large majority (over 70%) of the explanations, and
substantially more than Macaw’s explanations (34%). This potentially
contributes to system trustworthiness, where understanding how evidence
supports a conclusion is critical.
2\. $\approx$90% of Entailer’s self-verified premises were judged correct by
users. Of the remainder, virtually all were labeled “unsure” (only 1 Entailer
fact was labeled not correct), indicating that the there are few false beliefs
in proofs for correct answers. Rather, vague facts (e.g., “Claws are used for
cracking open shells”) make up the remainder.
3\. Entailer’s explanations were clearly preferred (57% to 23%, last bar) over
Macaw’s. In particular, Entailer’s arrangement of facts into a tree
discourages irrelevant information (bar #3).
Finally we note Entailer’s proofs are faithful (showing how an answer was
derived) and truthful (reflecting the system’s own beliefs), while Macaw’s
explanations are post-hoc generated text. These all suggest the value of
entailment trees as trustable explanations of a system’s behavior.

###  5.4 Analysis

####  5.4.1 Failure Analysis

If Entailer answers a question incorrectly, either a model belief (belief
error) and/or the reasoning itself (reasoning error) must necessarily be
incorrect, unless the question itself is malformed (dataset error). To better
understand these, we classified the 51/500 cases in OBQA where Entailer
selected the wrong answer while the Direct answer was correct, and found:
1\. belief errors (33%), where an incorrect belief led to a wrong answer, for
example:
In a desert… plants grow closer together because 1\. A desert…contains a low
amount of water. 2\. As the amount of water decreases, plants are forced to
grow closer together to conserve water. Note that here the reasoning is
correct but the second is premise is false.
2\. dataset errors (20%). In several cases, the question was ambiguous (e.g.,
does “animal” include humans?) or more than one answer option was valid (OBQA
is a crowdsourced dataset).
3\. reasoning errors (47%):
3a. Near-tautologies (20%): of the form “$X$ because 1\. $X^{\prime}$, 2. …”,
where $X^{\prime}$ is a near-repeat of the hypothesis. In such cases, the
proof offers little new information.
3b. Basic reasoning errors (10%):, e.g.,  During …fall, new leaves begin to
grow because 1\. During fall, old leaves begin to fall. 2\. New leaves begin
to grow. Here entailment is simply invalid.
3c. Incorrect abductive inferences (9%): of the form $A$ because
$(A\rightarrow{}B)$ and $B$. While sometimes useful, these inferences are not
sound and can produce incorrect conclusions, e.g.,
Cooking food requires a fire to be built because
1\. Cooking food requires heating the food. 2\. Fire can be used to heat
things. 3d. Quantification and exceptions (8%): where both beliefs and
reasoning seem reasonable, but the specific case does not hold, e.g.,  Seeds
are often found inside a strawberry because 1\. Seeds are often found inside
fruits. 2\. A strawberry is a kind of fruit.

Figure 7: Causes of Entailer’s failures (%) among belief (blue), reasoning
(red shades), and dataset (purple).

####  5.4.2 When do proofs do better?

At its best, Entailer decomposes an uncertain hypothesis $H$ into premises $P$
which it is very confident about. For example, Entailer is unsure whether A
suit of armor conducts electricity, but it confidently believes the generated
premises:
A suit of armor conducts electricty because 1\. A suit of armor is made of
metal 2\. Metals conduct electricity Thus an uncertain conclusion is replaced
by more certain premises, and we see Entailer performing particularly well for
such questions. However, there are questions that are largely “fact lookup”,
where a proof may be less helpful. For example, the model is already very
confident about the hypothesis Jupiter is the largest planet in the Solar
System; decomposing this into 1\. The largest planet has the greatest mass
plus 2\. Jupiter has the greatest mass in the solar system has not obviously
made answering easier. In fact, Entailer’s algorithm is specifically designed
to account for this, only growing a proof when the confidence in the premises
improves the confidence in $H$, thus tailoring its degree of reasoning to the
complexity of the problem at hand.

##  6 Towards Teachable Systems

If a system can show how its answers systematically follows from its own
beliefs, articulating those beliefs in the process, then this opens a window
into the model’s internal state for the user. As a result, new opportunities
arise for identifying and correcting the model’s misunderstandings, so the
system gradually improves over time (sometimes referred to as explanatory
interactive machine learning (XIL) Teso and Kersting (2019)). One vehicle for
doing this is to use Entailer’s (currently unused) context field C at runtime:
If the user asks a question, receives a wrong answer, and sees an incorrect
belief in the proof, they would provide the corrected belief, then re-ask the
question with the corrected belief in the context. This encourages the model
to use the corrected belief in its new answer and proof, rather than just
repeat the same bad belief. Such overrides would then be stored and retrieved
from a persistent memory to use for future questions also. A simple,
hypothetical dialog illustrating this is shown in Figure 8. This is an
exciting avenue made possible by this work, currently used by the TeachMe
systemMishra et al. (2022).

U: Can a magnet attract a penny?
Entailer tries to prove H (A magnet attracts a penny) and the negation neg(H),
and reports the best proof E: A magnet can attract a penny because:
1\. A penny is made of copper
2\. Copper is magnetic
Do you agree?
Here Entailer gives the wrong answer, arising from an incorrect belief #2. The
user offers a correction: U: No. Copper is not magnetic
System tries again, trying to prove H and neg(H) with context C = “Copper is
not magnetic”. C biases the model away from its prior mistake, effectively
changing the model’s belief. The new best proof is reported. E: A magnet
cannot attract a penny because:
1\. A penny is made of copper.
2\. Copper is not magnetic
Do you agree?
The system now gets the right answer, having learned that copper is not
magnetic from the user. C is added to memory for use in future questions, via
information retrieva. U: Yes

Figure 8:  A hypothetical dialog illustrating how a user (U) might identify
and correct Entailer’s (E) incorrect beliefs through interaction. Here
Entailer initially gets the wrong answer due to an incorrect model belief
(“Copper is magnetic”). The user offers a correction, which is then provided
as context when re-asking the question, effectively overriding the prior bad
belief (the user has “taught” the model). By storing such corrections in a
memory, such belief updates persist over time.

##  7 Conclusion

Our goal is a system that can show how its answers systematically follow from
its own internal beliefs, and materialize those beliefs in the process.
Entailer is, to our knowledge, the first system to demonstrate this
capability, achieved using an “over-generate and verify” approach for each
backward-chaining step. Our evaluation suggests that Entailer’s proof-based
answer accuracy is similar to the “direct” QA accuracy, with the additional
benefit of providing a faithful, truthful chain of reasoning showing how its
answers follow from its internal beliefs. The significance of this is that
these chains provide a window into the model’s internal beliefs and their
relationships, allowing users to both verify a system’s answer, or if the
system provides an incorrect answer, to identify the incorrect belief leading
to that error. This in turn offers new opportunities for a user to correct the
system’s misunderstanding by overriding a faulty belief, e.g., by adding a
memory of user corrections/overrides Tandon et al. (2022), or by editing the
model parameters directly Mitchell et al. (2021), a step towards interactive
systems that can both explain to, and learn from, users over time Mishra et
al. (2022). Entailer data and models are available at
https://allenai.org/data/entailer. We look forward to future developments in
these directions.

## Limitations

We have shown how a neural system can expose how its answers systematically
follow from its own internal beliefs, providing a window into the model’s
system of beliefs. While exciting, there are several limitations with the
current work and opportunities for the future.

First, the system is not perfect at generating coherent chains of reasoning,
sometimes producing entailments that are invalid or nearly tautologous
(Section 5.4.1 and Figure 7). Improved proof generation and scoring techniques
would help address this.

Second, like others, we use textual entailment as the basic reasoning
operation, but the definition of entailment remains somewhat imprecise (a
valid entailment is one that “a person would typically infer.” Dagan et al.
(2013)), contributing to noise in the model’s training data. A more precise
characterization of reasoning validity would help in both generation and
evaluation of reasoning chains.

Third, we assume the model is generally consistent about its beliefs, but in
some cases the model may verify contradictory statements, making it less clear
what the model actually believes in such cases. We currently do not handle
such situations. Use of a global notion of belief (rather than per question)
would be a valuable avenue to explore, e.g., Kassner et al. (2021).

Fourth, as a practical matter, recursive construction of proofs is
computationally expensive ($\approx$360 seconds/question for up to depth-3
proofs for 4-way multiple-choice, Appendix A.2). Improvements to the search
algorithm would allow faster experimentation, and also help deploy the model
in a practical setting.

Finally, we have speculated that showing users faithful, truthful chains of
reasoning might be a basis for a conversational system, where users could
correct and teach the system in cases where it was wrong. However, this is
currently just a conjecture - futures explorations into how this might be
realized would be valuable.

## Ethics Statement

Like any other large-scale language model, despite the best intentions, there
is a risk of our model producing biased or offensive statements as part of its
explanations. We release our models for research purposes only.

## Acknowledgements

This research was made possible, in part, by funding from Open Philanthropy.
We also thank Google for providing the TPUs for conducting experiments.

## References

  * Bosselut et al. (2019) Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019.  COMET: Commonsense transformers for automatic knowledge graph construction.  In _ACL_.
  * Bostrom et al. (2022) Kaj Bostrom, Zayne Sprague, Swarat Chaudhuri, and Greg Durrett. 2022.  Natural language deduction through search over statement compositions.  _ArXiv_ , abs/2201.06028.
  * Bostrom et al. (2021) Kaj Bostrom, Xinyu Zhao, Swarat Chaudhuri, and Greg Durrett. 2021.  Flexible generation of natural language deductions.  In _EMNLP_.
  * Brown et al. (2020) Tom B. Brown et al. 2020.  Language models are few-shot learners.  In _NeurIPS_.
  * Clark et al. (2020) Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020.  Transformers as soft reasoners over language.  In _IJCAI’20_.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021.  Training verifiers to solve math word problems.  _ArXiv_ , abs/2110.14168.
  * Creswell et al. (2022) Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022.  Selection-inference: Exploiting large language models for interpretable logical reasoning.  _ArXiv_ , abs/2205.09712.
  * Dagan et al. (2013) Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013.  _Recognizing Textual Entailment: Models and Applications_.  Morgan and Claypool.
  * Dalvi et al. (2021) Bhavana Dalvi, Peter A. Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, and Peter Clark. 2021.  Explaining answers with entailment trees.  In _EMNLP_.
  * Demszky et al. (2018) Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018.  Transforming question answering datasets into natural language inference datasets.  _ArXiv_ , abs/1809.02922.
  * Elazar et al. (2021) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, E. Hovy, H. Schutze, and Yoav Goldberg. 2021.  Measuring and improving consistency in pretrained language models.  _ArXiv_ , abs/2102.01017.
  * Gabriel et al. (2021) Saadia Gabriel, Chandra Bhagavatula, Vered Shwartz, Ronan Le Bras, Maxwell Forbes, and Yejin Choi. 2021.  Paragraph-level commonsense transformers with recurrent memory.  In _AAAI_.
  * Holtzman et al. (2019) Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019.  The curious case of neural text degeneration.  In _ICLR_.
  * Hong et al. (2022) Ruixin Hong, Hongming Zhang, Xintong Yu, and Changshui Zhang. 2022.  Metgen: A module-based entailment tree generation framework for answer explanation.  _ArXiv_ , abs/2205.02593.
  * Kalyanpur et al. (2020) Aditya Kalyanpur, Tom Breloff, and David A. Ferrucci. 2020.  Braid: Weaving symbolic and neural knowledge into coherent logical explanations.  _arXiv: Computation and Language_.
  * Kassner and Schütze (2020) Nora Kassner and H. Schütze. 2020.  Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly.  In _ACL_.
  * Kassner et al. (2021) Nora Kassner, Oyvind Tafjord, Hinrich Schutze, and Peter Clark. 2021.  BeliefBank: Adding memory to a pre-trained language model for a systematic notion of belief.  In _EMNLP_.
  * Li et al. (2019) Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. 2019.  A logic-driven framework for consistency of neural models.  In _EMNLP_.
  * Li et al. (2022) Yujia Li, David H. Choi, et al. 2022.  Competition-level code generation with alphacode.  _ArXiv_ , abs/2203.07814.
  * Manning and MacCartney (2009) Christopher D. Manning and Bill MacCartney. 2009.  _Natural language inference_.  Stanford University.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? A new dataset for open book question answering.  In _EMNLP_.
  * Mishra et al. (2022) Bhavana Dalvi Mishra, Oyvind Tafjord, and Peter Clark. 2022.  Towards teachable reasoning systems: Using a dynamic memory of user feedback for continual system improvement.  In _EMNLP_.
  * Mitchell et al. (2021) Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2021.  Fast model editing at scale.  _ArXiv_ , abs/2110.11309.
  * Nye et al. (2021) Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021.  Show your work: Scratchpads for intermediate computation with language models.  _ArXiv_ , abs/2112.00114.
  * Petroni et al. (2019) F. Petroni, Tim Rocktäschel, Patrick Lewis, A. Bakhtin, Y. Wu, Alexander H. Miller, and S. Riedel. 2019.  Language models as knowledge bases?  In _EMNLP_.
  * Ribeiro et al. (2022) Danilo Neves Ribeiro, Shen Wang, Xiaofei Ma, Rui Dong, Xiaokai Wei, Henry Zhu, Xinchi Chen, Zhiheng Huang, Peng Xu, Andrew O. Arnold, and Dan Roth. 2022.  Entailment tree explanations via iterative retrieval-generation reasoner.  _ArXiv_ , abs/2205.09224.
  * Ribeiro et al. (2019) Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. 2019.  Are red roses red? Evaluating consistency of question-answering models.  In _ACL_.
  * Schwitzgebel (2019) Eric Schwitzgebel. 2019.  Belief.  _Stanford Encyclopedia of Philosophy_.  Https://plato.stanford.edu/entries/belief/.
  * Shwartz et al. (2020) Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020\.  Unsupervised commonsense question answering with self-talk.  In _EMNLP_ , pages 4615–4629.
  * Tafjord and Clark (2021) Oyvind Tafjord and Peter Clark. 2021.  General-purpose question-answering with Macaw.  _ArXiv_ , abs/2109.02593.
  * Tafjord et al. (2019) Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019.  QuaRTz: An open-domain dataset of qualitative relationship questions.  _ArXiv_ , abs/1909.03553.
  * Tafjord et al. (2020) Oyvind Tafjord, B. D. Mishra, and P. Clark. 2020.  ProofWriter: Generating implications, proofs, and abductive statements over natural language.  _ArXiv_ , abs/2012.13048.
  * Talmor et al. (2020) Alon Talmor, Oyvind Tafjord, P. Clark, Y. Goldberg, and Jonathan Berant. 2020.  LeapOfThought: Teaching pre-trained models to systematically reason over implicit knowledge.  In _NeurIPS_.
  * Tandon et al. (2022) Niket Tandon, Aman Madaan, Peter Clark, and Yiming Yang. 2022.  Memory-assisted prompt editing to improve GPT-3 after deployment.  In _ACL Workshop on Commonsense Representation and Reasoning (CSRR’22)_.  (also arxiv:2201.06009).
  * Teso and Kersting (2019) Stefano Teso and Kristian Kersting. 2019.  Explanatory interactive machine learning.  _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_.
  * Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022.  Rationale-augmented ensembles in language models.  _ArXiv_ , abs/2207.00747.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _ArXiv_ , abs/2201.11903.
  * Wiegreffe and Marasović (2021) Sarah Wiegreffe and Ana Marasović. 2021.  Teach me to explain: A review of datasets for explainable NLP.  _ArXiv_ , abs/2102.12060.
  * Xie et al. (2020) Zhengnan Xie, Sebastian Thiem, Jaycie Martin, Elizabeth Wainwright, Steven Marmorstein, and Peter Jansen. 2020.  WorldTree V2: A corpus of science-domain structured explanations and inference patterns supporting multi-hop inference.  In _LREC_.
  * Yang et al. (2022) Kaiyu Yang, Jia Deng, and Danqi Chen. 2022.  Generating natural language proofs with verifier-guided search.  _ArXiv_ , abs/2205.12443.

## Appendix A. Entailer’s Backward Chaining Algorithm

1:procedure prove($H$) $\rightarrow$ score $s(H)$ & proof $tree(H)$:

2: i. find direct score $s_{d}(H)$ & confidence $c_{d}(H)$:

3: direct score $s_{d}(H)$ = call model $H$$\rightarrow$$S_{d}$

4: direct confidence $c_{d}(H)$ = $max(s_{d}(H),(1-s_{d}(H)))$ // confidence
of predicted label

5: ii. find reasoned score $s_{r}(H)$, confidence $c_{r}(H)$, & subtrees
$tree(p_{i})$:

6: call 1step(H) $\rightarrow$$P$ to find best premises $P$ that entail $H$

7: find $s_{e}(P\vdash{}H)$ via model $PH\rightarrow{}S_{e}$ // score the
entailment

8: if $s_{e}(P\vdash{}H)>c_{d}(H)$ or $H$=$H_{0}^{\dagger}$ // if reasoning
conf $c_{r}(H)$ might beat direct conf $c_{d}(H)$ (lines 13,11), backchain

9: then forall $p_{i}\in P$ do

10: call prove($p_{i}$) to find $s(p_{i}),tree(p_{i})$ // get scores and
subtrees for each $p_{i}$

11: reasoned score $s_{r}(H)=(\Pi_{i}s(p_{i})).s_{e}(P\vdash{}H)$ // overall
score for reasoned answer

12: else $s_{r}(H)=0$

13: reasoning confidence $c_{r}(H)=s_{r}(H)$ // reason confidence = score

14: iii. return direct or reasoned answer (pick most confident):

15: if $c_{r}(H)>c_{d}(H)$ or $H$=$H_{0}^{\dagger}$ // if reasoning confidence
is higher (or top level)

16: then return $s(H)$=$s_{r}(H)$ & $tree(H)$=($\\{tree(p_{i})\\}\vdash{}H$)
// reasoned score + subtrees

17: else return $s(H)$=$s_{d}(H)$ & $tree(H)$=$H$ // direct score + H as
terminal node (the subtrees are discarded)

18:

19:procedure 1step($H$) $\rightarrow$ premises $P$ // Find premises $P$ that
together entail $H$

20: repeat $k$ times: // over-generate with nucleus sampling

21: a. find premises $P$ via model $H\rightarrow{}P$ // find premises

22: b. find $\\{s_{d}(p_{i}\in P)\\}$ via model $H(=p_{i})\rightarrow{}S_{d}$
// score those premises

23: c. find $s_{e}(P\vdash{}H)$ via model $PH\rightarrow{}S_{e}$ // score the
entailment itself

24: d. discard if any of $\\{s_{d}(p_{i})\\},s_{e}(P\vdash{}H)<0.5$ // now
filter

25: e. $s_{r\textnormal{-}1deep}(H)=(\Pi_{i}s_{d}(p_{i})).s_{e}(P\vdash{}H)$
// Final score for the 1-deep proof

26: select premises $P$ with highest $s_{r\textnormal{-}1deep}(H)$ // Pick the
best of the $k$ proofs

†We insist on at least a 1-deep tree by, for the top-level hypothesis
$H$=$H_{0}$, ensuring that lines 8 and 15 succeed.

Algorithm 1 Entailer’s backchaining algorithm for searching for the best proof
$tree(H)$ with score $s(H)$ for a hypothesis $H$.

### A.1 Generating One Backward-Chaining Step

The procedure to find a 1-step inference is called 1step($H$) in Algorithm 1
(line 19). Given a hypothesis $H$, we use the angle $H\rightarrow P$ to over-
generate a set of $k$ alternative backward-chaining steps $P\vdash H$ using
nucleus sampling (line 21). We then check that the model believes all the
premises $p_{i}\in P$ using the $H(=p_{i})\rightarrow S_{d}$ angle, and that
it believes the inference step $P\vdash H$ itself is valid (independent of
whether the $p_{i}$ are true or not) using the $PH\rightarrow S_{e}$ angle
(line 24). The overall score, denoting how well the 1-step proof supports the
hypothesis, combines the premise end entailment scores as follows (line 25):
[vskip=0mm,leftmargin=2mm]
$s_{r\textnormal{-}1deep}(H)=(\Pi_{i}s_{d}(p_{i})).s_{e}(P\vdash{}H)$ The
highest-scoring proof $P\vdash H$ is returned.

### A.2 Backward Chaining

Procedure prove($H$) in Algorithm 1 generalizes this to multi-step entailments
by recursively generating support for each premise $p_{i}$ in $P$ (line 10).
The stopping condition is when the model’s direct confidence111111 As the
direct score $s_{d}(p_{i})$ in $p_{i}$ ranges from 0 (definitely false) to 1
(definitely true), we define the corresponding confidence to with respect to
the predicted label, i.e., $c_{d}(p_{i})=1-s_{d}(p_{i})$ when
$s_{d}(p_{i})<0.5$, otherwise $c_{d}(p_{i})=s_{d}(p_{i})$. In contrast, the
proof score ranges from 0 (no proof) to 1 (perfect proof), hence the proof
confidence is simply the proof score, $c_{r}(p_{i})=s_{r}(p_{i})$. Note that
unlike $s_{d}(p_{i})$, $s_{r}(p_{i})=0$ means no proof, not false. in $p_{i}$
is greater than the proof-derived confidence in $p_{i}$, i.e.,
$c_{d}(p_{i})>c_{r}(p_{i})$. When this condition is met, the subtree
$P^{\prime}\vdash p_{i}$ for $p_{i}$ is discarded and $p_{i}$ becomes a leaf
node of the tree (line 17). We also impose a maximum depth $d$ on the tree.

This whole procedure is repeated for each candidate answer hypothesis (Section
3.1). Finally the system selects the answer corresponding to the hypothesis
with the top-scoring proof $s(H)$.

On our datasets, the average runtime per question (4-way multiple-choice) is
$\approx$80 seconds (depth 1 proofs, sample size $k$=6) or $\approx$360
seconds (up to depth 3 proofs, sample size $k$=6) on a 48GB GPU, due to the
large number of candidate inference steps generated during the search.

## Appendix B. Crowdsourcing Instructions for Verifying Entailments (Section
4.1.2)

## Appendix C: Model Training

### C.1 Dataset Preparation

Here we describe in detail how Entailer’s training data is assembled.

  1. 1.

We start with the training partition of the EntailmentBank dataset,
containining 1313 multiple choice questions each with an entailment tree for
their correct answer choice.

  2. 2.

We convert the question + each answer option to a hypothesis using four
different, alternative methods:

     * •

Entailer’s current QA2D model, a reconstruction of that by Demszky et al.
(2018) (Section 3.1).

     * •

An in-house, rule-based QA2D utility

     * •

An earlier version of Entailer’s QA2D model.

     * •

The original hand-written hypothesis supplied in the EntailmentBank dataset.

If a source generates the same hypothesis for two different answer choices, we
choose not to trust it for that question and discard it.

  3. 3.

We “shred” the EntailmentBank entailment trees into individual 1-step P
$\vdash$ H entailments, producing 4175 1-step (valid) entailments using
$\approx$9000 true premises.

  4. 4.

The crowdsourced P $\vdash$ H instances, with annotations on whether premises
are true and entailment is true, are also added (Section 4.1.2).

  5. 5.

Each EntailmentBank proof also comes with a set of associated relevant facts
(sentences), only some of which are used in the entailment proofs. We use
these to create a "relevant context" containing these sentences, sorted into
two buckets:

     * high-relevance:

sentences actually used in the proof of the correct answer

     * medium-relevance:

the remaining sentences

This context is the same across all answer options at this point.

  6. 6.

For every hypothesis and premise appearing in this dataset, we run a simple
BM25 IR search against a larger science text corpus (about 1.5 million
sentences from a science-filtered Wikipedia) to obtain noisier sentences to
use as a low-relevance context.

  7. 7.

For the final training dataset, we create 4 different contexts, using 4
sampling strategies. First we create a "full" context of sentences sorted into
three buckets, with added noise:

     * high:

high-relevance facts + 10% chance of a random sentence from medium/low

     * medium:

medium-relevance facts + 20% chance of random sentences from high/low

     * low:

5 low-relevance facts + 20% chance of random sentence from high/medium

Then, we use four sampling strategies to create actual contexts for the
Entailer training data. Each strategy is defined by the per-sentence chance of
a sentence coming from one of the high/medium/low buckets. For example, a
context sampled with 0.2/0.4/0.6 means that 20% of its sentences came from
high, 40% from medium, an 60% from low. If no sentence is selectable (e.g., a
bucket is exhausted), we use one from low. The final context is syntactically
expressed as “[HIGH] <high sentences> [MEDIUM] <medium sentences> [LOW] <low
sentences>”. The 4 sampling strategies are:

     * 1/1/1

(full context, all available sentences)

     * 0.5/0.5/0.5

(half of sentences from each category)

     * 0.2/0.4/0.6

(more emphasis on lower categories)

     * 0/0/1

(only low sentences)

For the training set we store these as a list of contexts, to be sampled at
random when generating the training instances. We do the same for hypotheses,
so these are also sampled at random.

  8. 8.

The model is trained across the following angles, each sampled equally:
{H$\rightarrow$P, H$\rightarrow$V, HP$\rightarrow$I, QAH$\rightarrow$P,
QAH$\rightarrow$V, QAHP$\rightarrow$I, HC$\rightarrow$P, HC$\rightarrow$V,
HPC$\rightarrow$I, QAHC$\rightarrow$P, QAHC$\rightarrow$V,
QAHPC$\rightarrow$I}

This is just the core angles H$\rightarrow$P, H$\rightarrow$V,
HP$\rightarrow$I, with optional QA and/or C added. The individual premise
verification (leaves) uses the H$\rightarrow$V angle (where H is now a
premise), for those angles we don’t have associated proofs P so we limit to
the angles: H$\rightarrow$V, QAH$\rightarrow$V, HC$\rightarrow$V,
QAHC$\rightarrow$V

The full dataset is provided in the supplementary material, and will be
released publically.

### C.2 Model Details

We train a T5-11B multi-angle model, Entailer, following the multi-task setup
similar to Tafjord and Clark (2021) for the three functionalities described in
Table 1. We used the default hyperparameters (including the Adafactor
optimizer) in the T5 library,fine-tune the model for 20K steps with batch size
of 8, selecting the checkpoint with highest validation score (usually the
final step).

At inference time, Entailer’s primary function is to generate candidate
entailments for each answer choice for a given question, which are then scored
by the same model using its two verification angles. To generate multiple
explanations for a single input (answer hypothesis, IR retrieved context), in
addition to the greedy decoding we use nucleus sampling Holtzman et al.
(2019). For the experiments in this paper we set “temperature= 2.0”, and
“top_p= 0.95”.

## Appendix D. Examples of Macaw Explanations and Entailer Proofs

A random selection of explanations, both good and bad, from the two systems
(Section 5.3).


# Colored Transparent Object Matting from a Single Image Using Deep Learning

Jamal Ahmed Rahim  Kwan-Yee K. Wong
University of Hong Kong
Hong Kong
{jarahim<EMAIL_ADDRESS>

## Abstract

This paper proposes a deep learning based method for colored transparent
object matting from a single image. Existing approaches for transparent object
matting often require multiple images and long processing times, which greatly
hinder their applications on real-world transparent objects. The recently
proposed TOM-Net can produce a matte for a colorless transparent object from a
single image in a single fast feed-forward pass. In this paper, we extend TOM-
Net to handle colored transparent object by modeling the intrinsic color of a
transparent object with a color filter. We formulate the problem of colored
transparent object matting as simultaneously estimating an object mask, a
color filter, and a refractive flow field from a single image, and present a
deep learning framework for learning this task. We create a large-scale
synthetic dataset for training our network. We also capture a real dataset for
evaluation. Experiments on both synthetic and real datasets show promising
results, which demonstrate the effectiveness of our method.

##  1 Introduction

Image matting is an important problem in the field of image processing. It
refers to the process of extracting a foreground matte from an image by
identifying pixels of the foreground object and estimating their opacities.
The extracted matte can be composited onto different backgrounds to produce
new images based on the matting equation:

$I_{i,j}=\alpha_{i,j}F_{i,j}+(1-\alpha_{i,j})B_{i,j},\hskip
14.22636pt\alpha_{i,j}\in[0,1]$ (1)

where $I_{i,j}$, $\alpha_{i,j}$, $F_{i,j}$ and $B_{i,j}$ denote the composite
color, opacity, foreground color, and background color at pixel $(i,j)$
respectively.

Image matting is widely used in image and video editing. However, existing
methods are often designed for opaque objects and cannot handle transparent
objects. Note that the appearance of a transparent object depends on how it
refracts light from the background. Obviously, estimating object opacity alone
is not sufficient to model the appearance of a transparent object. There exist
a number of methods for transparent object matting [10, 11, 12, 13]. They
often require a huge number of images, specially designed background images,
and a long processing time to extract an environment matte for the transparent
object.

Recently, with the great success of deep learning in both low-level and high-
level vision tasks, deep learning framework has been introduced to transparent
object matting. Chen et al. [1] introduced a two-stage framework for learning
transparent object matting from a single image. However, their method assumes
colorless transparent objects and cannot handle colored transparent objects.
This limits its applications on real-world objects.

In this paper, we address the problem of colored transparent object matting
from a single image. We extend TOM-Net to handle colored transparent object by
modeling the intrinsic color of a transparent object with a color filter. We
formulate the problem of colored transparent object matting as simultaneously
estimating an object mask, a color filter, and a refractive flow field from a
single image, and present a two-stage deep learning framework, named CTOM-Net,
for learning this task. We create a large-scale synthetic dataset for training
our network. We also capture a real dataset for evaluation. Promising results
have been achieved on both synthetic and real datasets, which demonstrate the
effectiveness of our method.

##  2 Image Formation Model

In [1], Chen et al. formulated colorless transparent object matting as

$I_{i,j}=(1-m_{i,j})B_{i,j}+m_{i,j}\cdot\rho_{i,j}\cdot M(T,R_{i,j}),$ (2)

where $I_{i,j}$, $m_{i,j}\in\\{0,1\\}$, $B_{i,j}$ and $\rho_{i,j}\in[0,1]$
denote the composite color, foreground ($m_{i,j}=1$) / background
($m_{i,j}=0$) membership, background color and light attenuation index at
pixel $(i,j)$ respectively. $M(T,R_{i,j})$ is a bilinear sampling operation at
location $(i+R_{i,j}^{x},j+R_{i,j}^{y})$ on the background image $T$. The
matte can be estimated by solving $m_{i,j}$, $\rho_{i,j}$ and $R_{i,j}$ for
each pixel in the input image, and these result in an object mask, an
attenuation map and a refractive flow field respectively.

In order to take color into account, we need to introduce color to (2). If we
model the object with a constant color, we will fail to capture color
variations in the object and lose some valuable information. We therefore
propose to estimate a color value for each pixel and cast this problem as
estimating a color filter. In order to keep the solution space small, we
integrate the color filter with the attenuation mask, and formulate colored
transparent object matting as

$I_{i,j}=(1-m_{i,j})B_{i,j}+m_{i,j}\cdot min(C_{i,j},M(T,R_{i,j})),$ (3)

where $C_{i,j}$ denotes the“color filter’ at pixel $(i,j)$. The matte can now
be estimated by solving $m_{i,j}$, $C_{i,j}$ and $R_{i,j}$ for each pixel in
the input image.

Figure 1: Our two-stage framework. The first value under an encoder block is
the number of features extracted and the second value is the resolution
relative to the input image. All filters are of size $3\times 3$.

##  3 Our Regression Framework

Based on the TOM-Net [1] architecture, we propose a framework called CTOM-Net
(see Fig. 1) for learning colored transparent object matting. Like TOM-Net,
CTOM-Net is a two-stage framework composed of the CoarseNet (1st stage) and
the RefineNet (2nd stage). CoarseNet is an encoder-decoder multi-scale
network. It takes a single image as input and outputs an object mask, a color
filter, and a refractive flow field. It can produce a robust object mask, but
the color filter and refractive flow field lack local structural details. This
may result in noisy composites at test time. RefineNet, which is a residual
network [9], is introduced to refine the color filter and refractive flow
field estimated by CoarseNet. It takes the input image and the outputs of the
CoarseNet as input, and outputs a sharper color filter and a more detailed
refractive flow field.

###  3.1 CoarseNet

Inspired by the results of TOM-Net and U-Nets [3, 14] in problems requiring
dense predictions, we adopt a mirror-link CNN [3] for CoarseNet. Specifically,
CoarseNet has one shared encoder and three parallel decoders for regressing an
object mask, a color filter and a refractive flow field respectively (see Fig.
1).

####  3.1.1 Loss Function

The learning of CoarseNet is supervised by the ground-truth matte using an
object mask segmentation loss $L_{m}$, a color filter regression loss $L_{c}$,
and a refractive flow field regression loss $L_{r}$ respectively. Besides, we
also include an image reconstruction loss $L_{i}$ to ensure the reconstructed
image111image generated by compositing an estimated matte on the same
background as the input being as close to the input as possible. The overall
loss function for CoarseNet is given by

$L=\alpha_{m}L_{m}+\alpha_{c}L_{c}+\alpha_{r}L_{r}+\alpha_{i}L_{i},$ (4)

where $\alpha_{m}$, $\alpha_{c}$, $\alpha_{r}$, $\alpha_{i}$ are the weights
for the corresponding loss terms respectively.

Object Mask Segmentation Loss The object mask has a dimension of $2\times
W\times H$, with $W$ and $H$ being the width and height of the input image
respectively. The problem of estimating an object mask is modeled as a binary
classification problem, with the output being normalized by the softmax
function. We compute the object mask segmentation loss using the binary cross-
entropy function as

$L_{m}=\\\
-\frac{1}{HW}\sum_{i,j}(\tilde{M}_{i,j}\log(P_{i,j})+(1-\tilde{M}_{i,j})\log(1-P_{i,j})),$
(5)

where $\tilde{M}_{i,j}\in\\{0,1\\}$ denotes the ground-truth binary mask at
pixel $(i,j)$, and $P_{i,j}\in[0,1]$ denotes the predicted probability of
being foreground at pixel $(i,j)$ .

Color Filter Regression Loss The color filter is a three-channel mask with a
dimension of $3\times W\times H$. Its accuracy can be measured by its
similarity with the ground-truth color filter. We compute the color filter
regression loss using the mean squared error function as

$L_{c}=\frac{1}{HW}\sum_{i,j}||\tilde{C}_{i,j}-C_{i,j}||_{2}^{2},$ (6)

where $\tilde{C}_{i,j}$ and $C_{i,j}$ are the ground-truth and predicted color
filter at pixel $(i,j)$ respectively.

Refractive Flow Field Regression Loss The refractive flow field has a
dimension of $2\times W\times H$, with one channel for horizontal
displacements, and another channel for vertical displacements. To restrict the
horizontal and vertical displacements to the ranges $[-W,W]$ and $[-H,H]$
respectively, we normalize the output by the tanh function and multiply the
result by the width and height of the input image accordingly. We compute the
refractive flow field regression loss using the average end-point error (EPE)
as

$L_{r}=\frac{1}{HW}\sum_{i,j}\sqrt{(\tilde{R}_{i,j}^{x}-R_{i,j}^{x})^{2}+(\tilde{R}_{i,j}^{y}-R_{i,j}^{y})^{2}},$
(7)

where $(\tilde{R}_{i,j}^{x},\tilde{R}_{i,j}^{y})$ and
$(R_{i,j}^{x},R_{i,j}^{y})$ are the ground-truth and predicted displacements
at pixel $(i,j)$ respectively.

Image Reconstruction Loss The composite image has a dimension same as the
input image (i.e., $3\times W\times H$). We compute the image reconstruction
loss using the mean squared error function as

$L_{i}=\frac{1}{HW}\sum_{i,j}||\tilde{I}_{i,j}-I_{i,j}||_{2}^{2},$ (8)

where $\tilde{I}_{i,j}$ and $I_{i,j}$ are the input image and the
reconstructed image at pixel $(i,j)$ respectively.

Implementation Details We empirically set $\alpha_{m}$ to 0.25, $\alpha_{c}$
to 1.1, $\alpha_{r}$ to 0.01 and $\alpha_{i}$ to 1.0. It takes approximately 2
days to train our network on a single GPU (Nvidia GTX 1080 Ti).

###  3.2 RefineNet

Following the TOM-Net architecture, we adopt a residual network to refine our
output. The input to RefineNet is the predicted matte from CoarseNet
concatenated with the input image, and the outputs are a refined color filter
and a refined refractive flow field. The object mask is not refined by the
RefineNet as its prediction by the CoarseNet is already very good and robust.

####  3.2.1 Loss Function

The loss for RefineNet is based only on the color filter and refractive flow
field predictions. The loss functions for color filter $L_{c}$ and refractive
flow field $L_{r}$ remain the same as in CoarseNet. The overall loss function
for the RefineNet is

$L^{r}=\alpha_{c}^{r}L_{c}+\alpha_{r}^{r}L_{r}.$ (9)

In our implementation, we empirically set both $\alpha_{c}^{r}$ and
$\alpha_{r}^{r}$ to 1.0.

##  4 Dataset

Synthetic Dataset We perform supervised learning for colored transparent
object matting, and therefore we need the ground-truth mattes of our inputs
for training. Since it is very difficult and time-consuming to obtain ground-
truth mattes for real images of transparent objects, we created a synthetic
dataset using POV-Ray [7] with shapes taken from the publicly available
dataset created in [1], and background images from the Microsoft COCO dataset
[8]. The shapes were taken from three categories, namely glass, glass with
water and lens. They were rendered in front of a random background image with
random orientation, scale, position, and filter color. In order to make sure
the network does not overfit the synthetic colored transparent objects, we
also include a number of colorless transparent objects in the dataset. Table 1
shows the statistics of our synthetic dataset. The network trains very well on
the synthetic dataset and generalizes well on both colorless and colored
transparent objects.

Table 1: Our Synthetic Dataset Statistics. Split | Colored | Colorless
---|---|---
Glass | G&W | Lens | Glass | G&W | Lens
Train | 52,000 | 26,000 | 20,000 | 10,000 | 10,000 | 10,000
Test | 400 | 400 | 400 | 400 | 400 | 400

Real Dataset In order to test the transferability of our model on real images,
we also captured a real dataset of colored transparent objects. This dataset
contains colored transparent objects from three categories, namely glass,
glass with water and lens. Each example in the dataset has an image of a
transparent object in front of a background and an image of the background
alone. The background images are used for direct comparison as well for the
generation of reconstructed images during testing. We capture 60 images each
for the glass and glass with water category, and 30 images for the lens
category. For colorless transparent objects, we use the real dataset made
publicly available by [1].

Data Augmentation Extensive data augmentation is carried out at runtime during
training. Apart from standard augmentation techniques such as rotation,
scaling, shearing, cropping, and color jitter, we also try to bridge the gap
between synthetic and real data by smoothing the borders of the transparent
objects in the synthetic images and adjusting the ground truth color filter
accordingly. Since real complex objects usually appear to be more colored
towards regions of total internal reflection and less colored in other
regions, they would either require more careful data generation and/or
augmentation. We therefore omit complex objects in training and testing, and
leave it for future works.

##  5 Experimental Results

Our evaluation shows promising results in colored transparent objects matting
from a single image. It is worth noting that if the background and the
transparent object are of similar color, it would be difficult to tell the
extent to which the background color is being filtered. This implies that
there is a limit to which such color filtering can be determined. Our results
on synthetic and real datasets show that we have managed to reach this limit.

###  5.1 Results on Synthetic Data

For testing with the synthetic dataset, we compare our predicted mattes with
the ground-truth mattes. We report the average intersection over union for the
object masks, the average mean-squared error for the color filters, and the
average end-point error for the refractive flow fields. We also report the
average mean-squared error for the reconstructed images. We compare our
results against the background images and the results of TOM-Net in Table 2.
It can be observed that our model outperforms both the background images and
TOM-Net. Figure 2 shows that our model can create visually pleasing composite
images on synthetic data, and can capture the color of the transparent objects
well (something that cannot be handled by TOM-Net).

Input | Background | Ref. Flow (GT / Est) | Mask (GT / Est) | Color Filter (GT / Est) | Reconstruction
---|---|---|---|---|---
Colored Glass |  |  TOM-Net |  |  |
CTOM-Net |  |  |
Colorless Glass |  |  TOM-Net |  |  |
CTOM-Net |  |  |
Colored G&W |  |  TOM-Net |  |  |
CTOM-Net |  |  |
Colorless G&W |  |  TOM-Net |  |  |
CTOM-Net |  |  |
Colored Lens |  |  TOM-Net |  |  |
CTOM-Net |  |  |
Colorless Lens |  |  TOM-Net |  |  |
CTOM-Net |  |  |
Figure 2: Examples of results on synthetic data, compared with TOM-Net as our
baseline. For TOM-Net, we show the ground truth attenuation map and estimated
attenuation map in the color filter column. GT is ground truth, and Est is the
estimate made by the method. (Best viewed in PDF with zoom).

Table 2: Results on synthetic data. F-EPE and F-RoI are EPEs of refractive
flow for whole image and region of interest respectively. M-IoU is the
intersection over union for the object mask. C-MSE is the MSE of the color
filter; and I-MSE is the MSE of the reconstruction. Lower is better, except
for M-IoU.

Method | Colored Glass | Colored Glass With Water | Colored Lens
---|---|---|---
|  F-EPE |  F-RoI |  M-IoU |  C-MSE |  I-MSE |  F-EPE |  F-RoI |  M-IoU |  C-MSE |  I-MSE |  F-EPE |  F-RoI |  M-IoU |  C-MSE |  I-MSE
Background |  2.55 |  28.79 |  0.09 |  4.92 |  1.17 |  4.07 |  44.57 |  0.09 |  4.88 |  1.13 |  5.46 |  33.60 |  0.15 |  6.30 |  1.73
TOM-Net |  2.69 |  28.70 |  0.91 |  - |  0.80 |  3.70 |  38.32 |  0.89 |  - |  0.70 |  2.02 |  12.90 |  0.98 |  - |  0.95
CTOM-Net |  1.55 |  17.02 |  0.98 |  0.64 |  0.26 |  2.53 |  27.94 |  0.97 |  0.75 |  0.33 |  1.05 |  6.53 |  0.99 |  1.55 |  0.67
Method | Colorless Glass | Colorless Glass With Water | Colorless Lens
|  F-EPE |  F-RoI |  M-IoU |  C-MSE |  I-MSE |  F-EPE |  F-RoI |  M-IoU |  C-MSE |  I-MSE |  F-EPE |  F-RoI |  M-IoU |  C-MSE |  I-MSE
Background |  2.53 |  28.83 |  0.09 |  0.94 |  0.35 |  3.84 |  42.86 |  0.09 |  1.54 |  0.53 |  5.57 |  33.81 |  0.15 |  2.06 |  1.06
TOM-Net |  1.90 |  20.05 |  0.91 |  - |  0.20 |  3.19 |  35.05 |  0.92 |  - |  0.28 |  2.03 |  13.00 |  0.99 |  - |  0.35
CTOM-Net |  1.68 |  18.62 |  0.94 |  0.75 |  0.23 |  2.75 |  30.53 |  0.93 |  0.84 |  0.36 |  1.49 |  9.62 |  0.99 |  1.30 |  0.55

Input |  Background |  Ref. Flow |  Mask |  Color Filter |  Reconstruction |  Composite
---|---|---|---|---|---|---
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
Figure 3: Examples of results on real data. Composites are made by rendering
the extracted object against a new background.

###  5.2 Comparison with TOM-Net

In comparison to TOM-Net, our model makes more robust estimates of the
refractive flow field on both colorless and colored transparent objects. Table
2 shows that our method outperforms TOM-Net in refractive flow fields
estimation in all three object categories. The example of colorless glass with
water in Figure 2 shows that TOM-Net completely fails to detect the refractive
flow for the water inside the glass, whereas CTOM-Net can detect the water and
estimates the refractive flow. The example of colored glass with water shows
that TOM-Net detects part of the building as a transparent object. This is
also reflected by the average intersection over union values for the object
mask across all three object categories in Table 2 , where CTOM-Net achieves
higher values than TOM-Net. Note that TOM-Net models only colorless
transparent objects and predicts an attenuation mask, whereas CTOM-Net
considers colored transparent objects and predicts a color filter. This
introduces a small margin for error in image reconstruction, and the
reconstructed image MSE for TOM-Net for colorless objects is therefore
marginally better than that of CTOM-Net. Both methods share the same objective
of producing visually pleasing results, and Figure 2 shows no discernible
difference in the image reconstruction of colorless objects. In summary, CTOM-
Net is a more robust and useful (in the sense that it can handle colored
transparent objects) method for transparent object matting.

###  5.3 Results on Real Data

We test our trained model on the real dataset and compare the reconstructed
images with the background images. Due to the absence of ground-truth values,
we evaluate the results using the peak signal-to-noise ratio (PSNR) and
structural similarity (SSIM) metrics. The average values shown in Table 3 show
that our method outperforms background images in terms of PSNR and SSIM
metrics. Since most of our testing is done on colored objects, and we know
TOM-Net cannot handle such objects, and for brevity, we do not compare with
TOM-Net in this section, but instead focus on showing how well CTOM-Net can
generalize on real images. Figure 3 shows that for real images, our model
predicts mattes that allow rendering visually pleasing and realistic images.
This shows that our model can generalize well on real images.

Table 3: Results for real dataset.

Method | Glass | G&W | Lens
---|---|---|---
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
Background | 16.15 | 0.88 | 15.97 | 0.86 | 11.85 | 0.68
CTOM-Net | 22.63 | 0.90 | 23.21 | 0.91 | 16.36 | 0.75

##  6 Conclusion

We have introduced a simple yet efficient model for colored transparent object
matting, and proposed a two-stage deep learning framework to tackle this
problem. Our method takes a single image as input, and predicts an environment
matte as an object mask, a color filter, and a refractive flow field for a
transparent object. Experiments on both synthetic and real datasets show that
our proposed method achieves promising results, clearly demonstrating the
effectiveness of our method.

Since our method assumes the background image is the only light source, our
method cannot handle transparent objects with specular highlights. In the
future, we will investigate better training datasets and network architectures
to extend our framework to handle objects with specular highlights.

## References

  * [1] Guanying Chen, Kai Han, Kwan-Yee K. Wong: “TOM-Net: Learning Transparent Object Matting from a Single Image,” CVPR, 2018.
  * [2] Y.-Y. Chuang, D. E. Zongker, J. Hindorff, B. Curless, D. H. Salesin, and R. Szeliski: “Environment matting extensions: Towards higher accuracy and real-time capture,” SIGGRAPH, 2000\.
  * [3] J. Shi, Y. Dong, H. Su, and S. X. Yu: “Learning nonlambertian object intrinsics across shapenet categories,” CVPR, 2017.
  * [4] O. Ronneberger, P. Fischer, and T. Brox: “U-net: Convolutional networks for biomedical image segmentation,” International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015.
  * [5] J. Kim, J. Kwon Lee, and K. Mu Lee: “Accurate image superresolution using very deep convolutional networks,” CVPR, 2016\.
  * [6] S. Nah, T. H. Kim, and K. M. Lee: “Deep multi-scale convolutional neural network for dynamic scene deblurring,” CVPR, 2017.
  * [7] Persistence of vision (tm) raytracer.
http://www.povray.org/

  * [8] Microsoft Common Objects in Context (COCO) Dataset. http://cocodataset.org/
  * [9] K. He, X. Zhang, S. Ren, and J. Sun: “Deep residual learning for image recognition,” CVPR, 2016.
  * [10] Y.-Y. Chuang, D. E. Zongker, J. Hindorff, B. Curless, D. H. Salesin, and R. Szeliski: “Environment matting extensions: Towards higher accuracy and real-time capture,” SIGGRAPH, 2000\.
  * [11] Y. Wexler, A. W. Fitzgibbon, A. Zisserman, et al: “Image based environment matting,” Rendering Techniques, 2002.
  * [12] P. Peers and P. Dutre: “Wavelet environment matting,” Eurographics workshop on Rendering, 2003.
  * [13] J. Zhu and Y.-H. Yang: “Frequency-based environment matting,” Computer Graphics and Applications, 2004.
  * [14] Ke Ma, Zhixin Shu, Xue Bai, Jue Wang, Dimitris Samaras: “DocUNet: Document Image Unwarping via a Stacked U-Net,” CVPR, 2018