#  Acorn: Aspect-wise Commonsense Reasoning
Explanation Evaluation

Ana Brassard,αβ Benjamin Heinzerling,αβ Keito Kudo,βα
Keisuke Sakaguchiβα & Kentaro Inuiγβα
αRIKEN, βTohoku University, γMBZUAI
{ana.brassard<EMAIL_ADDRESS>
{keisuke.sakaguchi<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Evaluating free-text explanations is a multifaceted, subjective, and labor-
intensive task. Large language models (LLMs) present an appealing alternative
due to their potential for consistency, scalability, and cost-efficiency. In
this work, we present Acorn, a new dataset of 3,500 free-text explanations and
aspect-wise quality ratings, and use it to gain insights into how LLMs
evaluate explanations. We observed that replacing one of the human ratings
sometimes maintained, but more often lowered the inter-annotator agreement
across different settings and quality aspects, suggesting that their judgments
are not always consistent with human raters. We further quantified this
difference by comparing the correlation between LLM-generated ratings with
majority-voted human ratings across different quality aspects. With the best
system, Spearman’s rank correlation ranged between 0.53 to 0.95, averaging
0.72 across aspects, indicating moderately high but imperfect alignment.
Finally, we considered the alternative of using an LLM as an additional rater
when human raters are scarce, and measured the correlation between majority-
voted labels with a limited human pool and LLMs as an additional rater,
compared to the original gold labels. While GPT-4 improved the outcome when
there were only two human raters, in all other observed cases, LLMs were
neutral to detrimental when there were three or more human raters. We publicly
release the dataset to support future improvements in LLM-in-the-loop
evaluation.111https://github.com/a-brassard/ACORN

##  1 Introduction

Natural language processing systems that not only generate correct output, but
also provide an explanation (Miller, 2017) of why that particular output is
correct, are desirable for several reasons, such as increasing trustworthiness
(Floridi, 2019), compliance with “right to explanation” laws (e.g., European
Parliament and Council of the European Union, ), increasing interpretability
(Jacovi and Goldberg, 2020, but cf. Lipton, 2018), as well as system
improvement and knowledge discovery (Adadi and Berrada, 2018). However, this
immediately raises the question of how to evaluate system-generated
explanations in an efficient and effective manner.

Since explanations are typically free-form text, automatic evaluation of
explanations suffers the well-known, but as of yet unresolved, weaknesses of
automatic evaluation measures (Celikyilmaz et al., 2021), while human
evaluation is characterized by low scalability, high costs, subjectivity, and
inconsistency (Hartmann and Sonntag, 2022). LLM-based evaluation presents an
appealing alternative due to its potential high scalability, low cost, and
consistency.

Here, we investigate whether LLMs (Brown et al., 2020; OpenAI, 2023, i.a.) can
serve as a viable alternative approach to automatically evaluate system-
generated explanations. To verify the feasibility of this approach, we created
Acorn, a new dataset of 3,500 textual explanations with aspect-wise human
ratings of their quality, a first of its kind, and used it to evaluate whether
LLMs are aligned with human judgments (Figure 1).

Specifically, we first considered whether LLMs’ labels deviate from what would
be expected from human raters. We compared inter-annotator agreement between
all-human annotation (five raters) and when one of the raters is replaced by
an LLM. We found that most LLMs reduced the agreement, indicating that their
judgments skew outside of expected human rating.

GPT-4, however, had consistently less impact on the agreement, suggesting that
it may still have potential as a replacement for human judgments. To verify
this, we considered two scenarios: one where the LLM replaces the full human
evaluation, and one where it is used only as an additional rater. In other
words, we compared LLMs to humans _collectively_ and _individually_.

With the best-performing model, Spearman’s rank correlation between majority-
voted gold labels and LLM-generated ones ranged between 0.52 and 0.95,
depending on the aspect, averaging 0.72. This indicates that the LLM’s labels
are not entirely in line with human judgments, but they are not entirely
random either, and their usefulness may depend on the specific evaluation
criteria and usage case.

As for using the LLM as a single rater, we considered whether, in a limited-
budget scenario, it would be beneficial to use an LLM as an _additional_
rater. Specifically, we compared whether the majority-voted labels of a
reduced set of raters and with an added LLM had a higher correlation with the
original majority-voted labels. In most cases, the addition of LLMs either had
no effect or was detrimental, suggesting that they are not a reliable
replacement for human raters. One exception was GPT-4, which improved
correlation in a scenario with only two human raters. However, the resulting
labels are still less reliable than those from three or more human raters.

In summary, we quantified the consequences of using LLMs as a replacement or
addition to human evaluation, concluding that most have rating biases that
diverge from human consensus, leading to different outcomes. GPT-4 had a
beneficial effect when there were only two other human raters, but did not
compensate as much as an additional human rater did. We release Acorn as a
benchmark for future research on automatic evaluation of explanations.

Figure 1: We collected aspect-wise human ratings for 3,500 textual
explanations for commonsense reasoning benchmarks. We compared these against
ratings from large language models (LLMs) to evaluate their alignment with
human judgments.

##  2 Building Acorn: Evaluation Criteria and Data Sources

In a typical commonsense reasoning test, a model selects the most plausible
answer for a multiple-choice question. In a predict-and-explain setting, the
model additionally generates an explanation to justify the selected answer,
where we encounter the challenge of evaluating these explanations. Thus, we
first define a set of rating criteria (§2.1) and collect human ratings for a
selection of existing and newly collected textual explanations (§2.2, §2.3).

###  2.1 Rating Criteria

Criterion | Description | Label Choices
---|---|---
Supports | Which answer does it justify? | a), b), …, none
Overall | How good is the given explanation, overall? | 1 to 5 stars
Well-Written | Coherent, grammatically correct, fluent? | Yes, No
Related | Relevant to the Q and A? | Yes, No
Factual | Stated facts are generally true? | Yes, No, N/A
New Information | How much _new_ information to support the ans.? | None, Some, Sufficient, Ample
Unnecessary Info. | Any unnecessary statements? | Yes, No
Contrastive | Clearly shows the difference between the ans.? | Yes, No

Table 1: Explanation rating criteria used in this study. The first two
criteria target faithfulness and general explanation quality, while the rest
covers specific quality aspects.

As of now, there is no consensus on which criteria to use when evaluating
explanations. Our selection was informed by the following. Explanation
evaluation falls under the umbrella of natural language generation (NLG)
evaluation. We boiled down commonly used criteria (Howcroft et al., 2020) to
two overarching aspects: _surface-level_ criteria and _information/content-
level_ criteria, namely fluency and information sufficiency or minimality.
Additionally, we introduced criteria informed by known challenges of free-text
explanations: an LLM-generated explanation is not necessarily faithful to a
model’s inner workings (_faithfulness_ ; Lipton, 2018), and can include
hallucinated untrue facts (_factuality_ ; Rawte et al., 2023). Finally,
inspired by Miller (2017), we add a criterion aligned with an ideal
explanation’s structure (_contrastiveness_) and a retain a non-specific
_overall_ rating intended to capture potentially informative or useful aspects
that we have not explicitly covered. Our resulting criteria are summarized in
Table 1. During crowdsourcing, workers were given detailed guidelines and
examples to ensure a consistent understanding of these criteria (Appendix B).

###  2.2 Source Datasets

Figure 2: We collected general and aspect-wise ratings for human-written, LLM-
improved, better human-written, and LLM-generated explanations, for BCOPA and
CommonsenseQA, respectively.

Acorn contains ratings for a diverse set of existing, newly-collected, and
generated explanations. Our choice covers two commonsense reasoning
benchmarks, and their respective explanation datasets of varying qualities
(Figure 2). From each, we selected a random subset of 500 explanations for
rating, as well as an additional 500 samples of fluency-improved versions,
resulting in a total of 3,500 explanations. With five raters and eight
criteria per sample, this amounts to 140k ratings in total.

Specifically, as the target commonsense reasoning benchmarks, we selected
BCOPA (Kavumba et al., 2019)222Balanced COPA; a superset of COPA (Gordon et
al., 2012) with added “mirrored” questions that flip the correct label, i.e.,
the originally incorrect choice becomes correct.and CommonsenseQA (Talmor et
al., 2019) based on the simplicity of their tasks and availability of large-
scale explanation datasets (Wiegreffe and Marasović, 2021). Below are the
respective datasets we used to source candidate explanations.

#### CoS-E

(Rajani et al., 2019) A widely-used explanation dataset for CommonsenseQA,
albeit notoriously uninformative to humans (Nauta et al., 2023). A subset is
processed through GPT-3.5333A now deprecated version of text-davinci-003. for
fluency improvement, providing negative samples for higher-level issues like
lack of contrastiveness or adequate information. (500 random samples + 250
fluency-improved versions)

#### ECQA

(Aggarwal et al., 2021) An improved version of explanations for CommonsenseQA,
aligning with our criteria for well-formed explanations. (500 random samples)

#### Generated explanations for CommonsenseQA.

Additional high-quality explanations generated by prompting GPT-3.5 to solve a
subset of CommonsenseQA, though potential issues like irrelevant information
were noted. (500 samples)

#### COPA-SSE

(Brassard et al., 2022) Explanations for BCOPA with a subset processed through
GPT-3.5 for fluency improvement. Since COPA-SSE already contains overall
quality ratings, we selected a random sample of 250 questions and used each
question’s top-rated and bottom-rated explanation. (500 samples + 250 fluency-
improved versions) through gpt-3.5

#### Crowdsourced explanations for BCOPA.

ECQA’s counterpart for BCOPA; a new set of hand-written explanations,
carefully crafted for contrastiveness and thoroughness. (500 samples)

#### Generated explanations for BCOPA.

Similarly to CoS-E, we prompted GPT-3.5 to solve BCOPA questions. (500
samples)

###  2.3 Rating Collection

We crowdsourced ratings for the explanations in Acorn using Amazon Mechanical
Turk (AMT). Each rater was required to pass a qualification test, after which
they were asked to participate in trial rounds, during which we addressed
several clarity issues in the guidelines. The final pool was hand-picked based
on their responses, resulting in 28 participants. More details on the
crowdsourcing protocol are available in Appendix B. In experiments which use
aggregated labels, they were produced with a majority-vote mechanism, with
ties broken using the better label.

##  3 Can LLMs Replace Human Raters?

We answer this question in three steps: we first measure LLMs’ divergence from
expected human label variance (§3.2), then observe the results when using it
to _completely_ replace human evaluation (§3.3), and when using it as an
_additional_ rater instead (§3.4). All experiments follow the settings
described in Section 3.1.

###  3.1 Experimental Settings

#### Models.

We compared four contemporary API-enabled LLMs, namely GPT-3.5, GPT-4, PALM 2,
and Gemini. Each have reported high performance in diverse tasks including
text-based reasoning. We used the gpt-3.5-turbo-0613 and gpt-4-0613 snapshots
of OpenAI’s GPT-3.5 and GPT-4 models (Brown et al., 2020; OpenAI, 2023), and
the text-bison-001 and gemini-1.0-pro versions of Google’s PALM 2 and Gemini
models (Google, 2023; Gemini Team, 2023). In all experiments, temperature is
set to 0.0 with 256 output tokens at most, with all other parameters left at
their default values.

#### Prompting Strategy.

LLMs are notoriously oversensitive to prompt format (Wadhwa et al., 2023). For
the purpose of our analyses, we explored several prompting strategies and
retained the most successful one for each model, as measured by correlation
with majority-voted human ratings. In short, all models worked best with
_verbatim_ prompts, which corresponds to a word-by-word copy of the guidelines
given to humans. The models responded with a structured list of criteria and
their assigned labels for the given target explanation (in most cases). See
Appendix C for a detailed overview of alternative strategies that were
considered.

#### Label Extraction.

Using free-text generation models for a classification task introduces the
problem of extracting said ratings, and presents an information extraction
challenge in itself. This phenomenon, inherent to generative approaches
(Wadhwa et al., 2023), is a source of additional noise that affects all
evaluation pipelines necessitating a non-trivial solution in real
applications. In our experiments, we used a rule-based extraction method,
backed up with LLM-based extraction in case of failure. Finally, we manually
confirmed the remaining failures,444Due to complete failure to follow the
given task, exceeding the output token limit with overly verbose responses,
refusal to answer due to “a lack of explanation,” or triggering content
filters. and counted them as incorrect predictions. With verbatim prompts, the
extraction failure rate was <0.1% for all models but Gemini (1.74%).

###  3.2 Inter-annotator Agreement

Figure 3: Inter-annotator agreement (Krippendorff’s $\alpha$, *100 for
legibility) between human raters (shaded area) and with the LLM’s rating
replacing a random rater.

In subjective tasks, some degree of label variance is expected, resulting in
lower inter-annotator agreement. This disagreement is not necessarily noise,
but a feature of the data, as it can reflect the diversity of human opinions
(Aroyo and Welty, 2015). In this context, regardless of absolute agreement, a
successful LLM-based rater should be harmonious with the range of human labels
rather than deviate from it.

To measure this, we compared the inter-annotator agreement (Krippendorff’s
$\alpha$) between human raters and when a random rater is replaced by an LLM.
There are three possible outcomes: (i) agreement _decreases_ , indicating that
the LLM deviates from human judgments; (ii) agreement _remains the same_ ,
indicating that it is harmonious with human judgments; or (iii) agreement
_increases_ , meaning that the LLM is both harmonious and biased towards a
majority.555The latter may be desirable in use cases that rely on majority-
voted labels as the ground truth, but comes with the trade-off of losing
potentially useful label diversity.

#### Results.

In Figure 3, the shaded area shows the agreement between human raters, while
the bars show the agreement when the respective LLM’s ratings replace a random
human rater. All values are averaged over twenty iterations. In most cases,
agreement decreased, indicating that the LLMs generated labels that conflicted
with human ones, and therefore cannot be considered suitable replacements. A
notable exception is GPT-4, which maintained or improved agreement in all
criteria but well-written and unnecessary information.

###  3.3 LLMs As A Replacement for Human Evaluation

GPT-4 maintained or improved inter-annotator in most criteria, suggesting that
it does not deviate from an expected range of human ratings. Here, we ask if
it can then be used as a replacement for human evaluation. Specifically, we
investigated the degree to which GPT-4’s predictions align with majority-voted
human labels, measured with Spearman’s rank correlation.

Figure 4: Spearman’s ranking correlation between aggregated human labels and
LLM ratings.

#### Results.

Figure 4 shows Spearman’s ranking correlation (*100 for legibility) between
aggregated human labels and LLM predictions. GPT-4 results are annotated, with
others shown for comparison. All values are positive, indicating a positive
correlation with the ratings. The correlations in supports and contrastive
were particularly strong, with values of 0.90 and 0.95, respectively. Others
ranged from 0.52 to 0.78, indicating a moderately high correlation. However,
the model is clearly not a perfect replacement for human raters. Instead,
considering the small change in inter-annotator agreement (§3.2), the model
could potentially be used as an _additional_ rater when human annotation is
scarce, which we explore in the next section.

###  3.4 LLMs As An Additional Rater

The results so far hinted towards GPT-4 acting similarly to an average human
rater. Thus, it may seem appealing as an additional data point when human
raters are scarce or expensive. Here, we verified this potential by measuring
whether using GPT-4 as an additional rater improved the outcome over having
fewer human raters, i.e., whether the majority-voted labels became more
aligned with the original ones when the LLM is added as a rater.

Specifically, we compared Spearman’s rank correlation between the majority-
voted labels with all available raters and in two alternative scenarios: one
where GPT-4 is added as an additional rater and one where it is not. If the
correlation _increases_ when GPT-4 is added, it suggests that its predictions
are in line with the original majority-voted labels, and it is useful as an
additional rater. Otherwise, it would indicate a harmful or negligible effect,
and thus its inclusion should be avoided. We repeated this comparison for
scenarios with four, three, and two random human raters, respectively. All
other models are also included for comparison.

#### Results.

Figure 5 shows Spearman’s rank correlation with the original gold ratings
obtained by aggregating the labels of all five raters. Humans only denotes the
correlation between human majority-voted labels only, and others when
including the respective LLM as an additional rater.

Figure 5: Change in Spearman’s rank correlation when using fewer raters and
when an LLM is added as an additional rater. E.g., 4H+LLM compares the
correlation between the original (five-rater) gold labels, majority-voted
labels with four human raters and the respective LLMs added as a fifth rater,
and four human raters only.

The leftmost cluster compares majority-voted labels between four human raters
(4H) and four raters with an LLM added (4H+LLM). All correlations stayed the
same (0.91), indicating that the LLMs did not compensate for the difference
caused by a missing human rater. In this case, there was no harm nor benefit
to including an LLM.

In the center, we observe the effect when the human raters are reduced to
three (3H), with the influence of the LLM becoming stronger (3H+LLM). Human-
only correlation with the original gold labels dropped to 0.86, and all LLMs
decreased the correlation further. In this case, including an LLM was
_detrimental_ , bringing the majority labels further from the original human
consensus.

Finally, the rightmost cluster shows the case of two human raters (2H) and
adding the respective LLMs (2H+LLM). This represents the minimum possible
ratio of humans to LLMs where humans still retain a majority. Interestingly,
while overall correlation yet again decreased, GPT-4 _improved_ the
correlation (0.81 to 0.83).

Overall, we conclude that when there is already a large pool of workers
(four+), there is no benefit to adding an LLM as a judge. GPT-4 can be a
useful addition if there are only two human raters, but the resulting labels
are still less reliable than those from three or more human raters.

##  4 Discussion

From our results, we conclude that most models we tested are not suitable for
replacing human raters based on their divergence from human rater’s
distributions. The best-performing model, GPT-4, generated labels that were
particularly correlated (>0.90) with human majorities in supports and
contrastive criteria but less so in others (0.52-0.78). Acting as an
additional rater, GPT-4 did not affect the outcome over having one less human
rater, harmed the outcome when there were three human raters, and improved the
outcome when there were only two human raters. This suggests that GPT-4 may
only be useful as an additional rater when human raters are particularly
scarce.

From this, we concluded that LLMs are not a reliable replacement for human
raters, except in isolated cases. However, in this work, we operated under the
assumption that _majority-voted labels are the ground truth._ In practice,
this may not always be the case, and the reliability of LLMs may vary
depending on what information is desired in a given scenario. This opens up a
new avenue for future work, where LLMs can be applied in a more nuanced
manner, considering the context of the task and the intended use of the
labels.

Overall, as a balanced approach, we recommend using LLMs in configurations
that end with targeted human involvement, such as extensively using LLMs in
development or filtering stages and experts for final testing or evaluation.

##  5 Related Work

#### LLM-based Evaluation

LLMs as data labelers, and more broadly humans-and-LLMs-in-the-loop settings,
are an emerging direction in data collection and labeling. E.g., Wiegreffe et
al. (2022) developed a predict-and-explain pipeline that combines GPT-3 with a
supervised filter trained on binary acceptability judgments from humans. More
recently, Chiang and Lee (2023) proposed using LLMs to evaluate text, closely
in line with our work. They, however, found the models successful in their
setting. In contrast, we focus on explanation rating with more complex fine-
grained criteria, and closely scrutinize potential weaknesses; Previous works
brought into question the reliability of LLMs’ predictions, especially in
prompting setups. E.g., Webson and Pavlick (2022) found that instruction-tuned
models often produced good predictions with irrelevant or misleading prompts,
bringing into question their real “understanding” of the task. Others reported
unreliability of LLMs as labelers in various settings (Albrecht et al., 2022;
Shen et al., 2023; Hada et al., 2023, i.a.), a line of work which we join with
findings in the context of explanation evaluation.

#### Explanation Evaluation

For explanations in the form of textual justifications, previous works often
defined their own criteria for evaluation. Automatic evaluation borrowed from
machine learning and measures overlap with “gold standard” text using (a)
word-overlap metrics, e.g., BLEU, METEOR and ROUGE; and (b) embedding-based
metrics, e.g., BERTScore and BLEURT (Clinciu et al., 2021). In contrast,
human-tagged measures are more diverse and explanation-specific. For example,
Clinciu et al. (2021) measured Informativeness and Clarity; Wiegreffe et al.
(2022), inspired by social sciences, measured Acceptability, Generality,
Factuality, Grammar, New Info, Supports Label, and Amount of Information;
while Aggarwal et al. (2021) defined the criteria of Refutation, Complete,
Comprehensive, Minimal, and Coherent. A recent study instead focused on the
_utility_ of explanations, i.e., their helpfulness in answering a question
from a human point of view (Joshi et al., 2023). In our work, we largely
followed these existing criteria, with the addition of a “supports” criterion
indicating which answer the explanation attempts to justify.

#### Explain-and-Predict

Explanations are often generated in predict-and-explain settings, where models
provide justifications for their answers in QA-based benchmarks (e.g., Clinciu
et al., 2021), or an elaborate-then-predict setting, where models output a
prediction guided by its intermediate outputs such as knowledge statements or
reasoning chains (e.g., Marasovic et al., 2022; Wang et al., 2023). In this
work, we evaluate the capabilities of LLMs to evaluate explanations in an
explain-and-predict setting, where models provide justifications for their
answers in commonsense reasoning benchmarks. The following sections provide
background on each aspect.

##  6 Conclusions

Using a newly-built dataset of free-text explanations and crowdsourced aspect-
wise quality ratings, we analyzed the viability of LLMs as explanation
evaluators. We found that LLMs decrease inter-annotator agreement when
replacing human ratings, and that their ratings are moderately to highly
correlated with human ratings, depending on the quality aspect. In a scenario
where LLMs are used as additional raters instead, we found that they can
improve the outcome when there are only two human raters, but are neutral to
detrimental when there are three or more human raters. We conclude that while
LLMs can provide ratings moderately consistent with an average crowdworker,
they are not yet reliable enough for complete replacement.

## Limitations

Explanation evaluation is inherently subjective, and the majority-voted gold
label is not necessarily the “correct” answer. Subjectivity-informed scoring
is a complex task, and we leave its exploration to future work. Furthermore,
criteria may change depending on the intended use. Here, we limited our
analysis to a researcher’s point of view, where explanations are increasingly
used as a diagnostic tool, and aligned our criteria to a list of general
explanatory competencies. While our insights on LLMs’ performance may
potentially be applicable to different evaluation tasks, it should not be
assumed to be universally true.

LLMs are known to be sensitive to prompt format. We somewhat compensated for
this by comparing a prompt-averaged setting, and our focus is not to search
for the optimal setting but rather investigate potential fundamental issues
with LLM-based evaluation. However, for practical applications, we acknowledge
that there is a wealth of tweaks that may improve the performance.
Nevertheless, our findings highlight the need for caution when using LLMs for
explanation evaluation, and we hope that our work will inspire further
research in this direction.

## Ethics Statement

Some of the datasets used in our research have been found to contain biases
that can be further perpetuated through explanations supporting them. As such,
this data should not be used in any downstream applications without first
controlling for potentially harmful samples. Some of the commercial LLMs we
used have built-in filters for potentially harmful content, which were
triggered during several of our experiments.

Crowdworkers play a vital role in dataset creation. We prioritized fair
compensation, transparency in data usage, and respect for their rights and
privacy. Workers were informed on their work’s intended usage and of our
identity as requesters. We did not collect any personal information.

## Acknowledgements

This work was supported by JST CREST Grant No. JPMJCR20D2 and JSPS KAKENHI
Grant No. 15H01702 and 21K17814. We extend sincere thanks to all the
crowdworkers who helped realize this project with their diligent work in
labeling the dataset. A big thanks to Tatsuki Kuribayashi and Jonas F. Lotz
for their detailed feedback on the paper.

This work was written with the help of LLM-powered writing and coding
assistants, however, all ideas and opinions presented in this work are solely
those of the authors.

## References

  * Adadi and Berrada (2018) Amina Adadi and Mohammed Berrada. 2018.  Peeking inside the black-box: A survey on explainable artificial intelligence (xai).  _IEEE Access_ , 6:52138–52160.
  * Aggarwal et al. (2021) Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021.  Explanations for CommonsenseQA: New Dataset and Models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3050–3065, Online. Association for Computational Linguistics.
  * Albrecht et al. (2022) Joshua Albrecht, Ellie Kitanidis, and Abraham J Fetterman. 2022.  Despite "super-human" performance, current LLMs are unsuited for decisions about ethics and safety.
  * Aroyo and Welty (2015) Lora Aroyo and Chris Welty. 2015.  Truth is a lie: Crowd truth and the seven myths of human annotation.  _AI Mag._ , 36(1):15–24.
  * Brassard et al. (2022) Ana Brassard, Benjamin Heinzerling, Pride Kavumba, and Kentaro Inui. 2022.  COPA-SSE: Semi-structured Explanations for Commonsense Reasoning.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 3994–4000, Marseille, France. European Language Resources Association.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners.  pages 1877–1901.
  * Celikyilmaz et al. (2021) Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2021.  Evaluation of text generation: A survey.
  * Chiang and Lee (2023) Cheng-Han Chiang and Hung-Yi Lee. 2023.  Can large language models be an alternative to human evaluations? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15607–15631, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Clinciu et al. (2021) Miruna-Adriana Clinciu, Arash Eshghi, and Helen Hastie. 2021.  A study of automatic metrics for the evaluation of natural language explanations.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , Stroudsburg, PA, USA. Association for Computational Linguistics.
  * (10) European Parliament and Council of the European Union.  Regulation (EU) 2016/679 of the European Parliament and of the Council.
  * Floridi (2019) Luciano Floridi. 2019.  Establishing the rules for building trustworthy ai.  _Nature Machine Intelligence_ , 1(6):261–262.
  * Gemini Team (2023) Gemini Team. 2023.  Gemini: A Family of Highly Capable Multimodal Models.
  * Google (2023) Google. 2023.  PaLM 2 Technical Report.
  * Gordon et al. (2012) Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012.  SemEval-2012 Task 7: Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning.  In _*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_ , pages 394–398, Montréal, Canada. Association for Computational Linguistics.
  * Hada et al. (2023) Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. 2023.  Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?
  * Hartmann and Sonntag (2022) Mareike Hartmann and Daniel Sonntag. 2022.  A survey on improving NLP models with human explanations.  In _Proceedings of the First Workshop on Learning with Natural Language Supervision_ , pages 40–47, Dublin, Ireland. Association for Computational Linguistics.
  * Howcroft et al. (2020) David M Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020.  Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 169–182, Dublin, Ireland. Association for Computational Linguistics.
  * Jacovi and Goldberg (2020) Alon Jacovi and Yoav Goldberg. 2020.  Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4198–4205, Online. Association for Computational Linguistics.
  * Joshi et al. (2023) Brihi Joshi, Ziyi Liu, Sahana Ramnath, Aaron Chan, Zhewei Tong, Shaoliang Nie, Qifan Wang, Yejin Choi, and Xiang Ren. 2023.  Are machine rationales (not) useful to humans? Measuring and improving human utility of free-text rationales.  _Annual Meeting of the Association for Computational Linguistics_.
  * Kavumba et al. (2019) Pride Kavumba, Naoya Inoue, Benjamin Heinzerling, Keshav Singh, Paul Reisert, and Kentaro Inui. 2019.  When Choosing Plausible Alternatives, Clever Hans can be Clever.  In _Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing_ , pages 33–42, Hong Kong, China. Association for Computational Linguistics.
  * Lipton (2018) Zachary C Lipton. 2018.  The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery.  _Queueing Syst._ , 16(3):31–57.
  * Marasovic et al. (2022) Ana Marasovic, Iz Beltagy, Doug Downey, and Matthew Peters. 2022.  Few-Shot Self-Rationalization with Natural Language Prompts.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 410–424, Seattle, United States. Association for Computational Linguistics.
  * Miller (2017) Tim Miller. 2017.  Explanation in Artificial Intelligence: Insights from the Social Sciences.
  * Nauta et al. (2023) Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. 2023.  From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI.  _ACM Comput. Surv._ , 55(13s):1–42.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.
  * Rajani et al. (2019) Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Explain yourself! Leveraging language models for commonsense reasoning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Rawte et al. (2023) Vipula Rawte, Amit Sheth, and Amitava Das. 2023.  A survey of hallucination in Large Foundation Models.
  * Shen et al. (2023) Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. 2023.  Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 4215–4233, Singapore. Association for Computational Linguistics.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Wadhwa et al. (2023) Somin Wadhwa, Silvio Amir, and Byron C Wallace. 2023.  Revisiting Relation Extraction in the era of Large Language Models.
  * Wang et al. (2023) Wenya Wang, Vivek Srikumar, Hannaneh Hajishirzi, and Noah A Smith. 2023.  Elaboration-Generating Commonsense Question Answering at Scale.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1619–1635, Toronto, Canada. Association for Computational Linguistics.
  * Webson and Pavlick (2022) Albert Webson and Ellie Pavlick. 2022.  Do Prompt-Based Models Really Understand the Meaning of Their Prompts? In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2300–2344, Seattle, United States. Association for Computational Linguistics.
  * Wiegreffe et al. (2022) Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022\.  Reframing Human-AI Collaboration for Generating Free-Text Explanations.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 632–658, Seattle, United States. Association for Computational Linguistics.
  * Wiegreffe and Marasović (2021) Sarah Wiegreffe and Ana Marasović. 2021.  Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing.

##  Appendix A Dataset Statistics

###  A.1 Data Sources

Table 2 shows a breakdown of the samples in the test set per source dataset as
described in Section 2.2.

Dataset | #samples
---|---
COPA-SSE (best & worst) | 250 + 250
\+ fluency fix | 250
Generated (BCOPA) | 500
Crowdsourced (BCOPA) | 500
CoS-E | 500
\+ fluency fix | 250
Generated (CommonsenseQA) | 500
ECQA | 500
Total | 3,500
Table 2: Breakdown of explanation data used in our experiments by source
dataset.

###  A.2 Data Examples

Table 4 and Table 5 show examples of best-rated and worst-rated explanations
from each source dataset.

###  A.3 Label Distributions

| -1 | 0 | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---|---|---
Supp. | 11% | 32% | 31% | 8% | 9% | 8% |
Ovr. |  |  | 11% | 14% | 30% | 35% | 10%
Well-wr. |  | 28% | 72% |  |  |  |
Rel. |  | 7% | 93% |  |  |  |
Fact. | 12% | 4% | 85% |  |  |  |
New i. |  | 19% | 46% | 33% | 2% |  |
Unn. i. |  | 17% | 83% |  |  |  |
Contr. |  | 58% | 42% |  |  |  |
Table 3: Label distributions per criterion with human ratings. -1 denotes
"none" for supports and "N/A" for factual. Scalar aspects are rounded to
integer values.

Table 3 shows the label distributions of each criterion in our test set.
Scalar aspects (overall rating, new information) are bucketed. ‘-1’ labels
represent None for supports and N/A for factual.

##  Appendix B Explanation Rating Annotation

Our crowdsourcing protocol for label collection consisted of three phases:
qualification rounds, trial rounds, and main collection rounds. We provided
detailed guidelines showing general instructions, detailed information on each
criterion and their respective labels, three examples, and a FAQ section based
on questions we received from workers. The full document is available upon
request to the first author.

#### Qualifications

In the qualification rounds, we curated a question set of 6 explanations and
manually tagged them with "acceptable" answers, focusing on overall alignment
rather than exact matches. We included a dummy question with strict
instructions for filtering. Out of 700 participants, the top 201 workers, with
a match percentage of 59% or higher, proceeded to trial rounds. We addressed
any concerns or clarifications through email or form feedback. We hand-picked
a final group of 28 workers. Qualifications were open to workers with a HIT
approval rate of 99% or more and 5,000 or more approved HITs. Note that the
location requirement was removed as it was an unnecessary barrier for highly
skilled and motivated workers.

#### Main Rounds

Each of the 3,500 explanations in the test set (§2.2) was rated by five
workers. The ratings were aggregated to create the final gold labels used in
our experiments. Figure 7 shows the crowdsourcing form.

#### Payment Information

For qualifications, each worker was compensated $0.15 per HIT. For the main
rounds, the fee was increased to $0.25 per HIT, roughly matching a payment of
$20.00 per hour.

##  Appendix C Preliminary Experiment: Prompting Strategies

We compared _single_ and _compound_ calling, where the former prompts the
model for a single criterion at a time, and the latter prompts the model for
all criteria at once. We also compared _default_ , _averaged_ , and _verbatim_
prompt formats, corresponding to a simple prompt with the explanation and the
rating criteria, a voting mechanism over several prompt variants (described in
Appendix D), and input identical to the human annotation guidelines,
respectively. For single verbatim calls, only the relevant sections
(guidelines and examples) for the target criterion were included. Default and
averaged prompts were further compared in zero-shot and three-shot settings,
where the latter contained the same examples as shown in the human guidelines.

All models were most successful with verbatim compound prompting,
corresponding to the same guidelines as those shown to humans. Unless
otherwise stated, all results presented in the main text are based on this
setting.

##  Appendix D Prompt Variant Examples

This section shows the prompt variants used for prompt-averaged settings.
Parts identical to the simple prompt are omitted and marked with ‘...’ for
brevity. In few-shot settings, the initial guidelines are only shown once,
followed by $N$ examples with a simple corresponding answer format, unless
otherwise specified. E.g., for the simple prompt:

    QUESTION: <question>
    a) <answer choice>
    b) <answer choice>

    EXPLANATION: <explanation>

    1. a
    2. 2
    3. no
    4. yes
    5. yes
    6. sufficient
    7. no
    8. yes

###  D.1 Simple Prompt

A straightforward prompt outlining the evaluation criteria.

    Evaluate the given explanation according to
        the following criteria:

    1. Which answer does it support? (a, b, c, d, e, none)
    2. Overall rating? (1, 2, 3, 4, 5)
    3. Is it well-written? (no, yes)
    4. Is the explanation related to the question and answers? (no, yes)
    5. Are all contained facts correct? (N/A, no, yes)
    6. How much new information is given? (none, some, sufficient, ample)
    7. Any unnecessary information? (no, yes)
    8. Is it contrastive? (no, yes)

    QUESTION: <question>
    a) <answer choice>
    b) <answer choice>

    EXPLANATION: <explanation>

###  D.2 Shuffled Criteria Prompt

This prompt uses the same wording as the simple prompt but asks for ratings in
a shuffled order.

    Evaluate the given ...

    1. How much new information is given? (none, some, sufficient, ample)
    2. Is it contrastive? (no, yes)
    3. Is the explanation related to the question and answers? (no, yes)
    4. Is it well-written? (no, yes)
    5. Overall rating? (1, 2, 3, 4, 5)
    6. Which answer does it support? (a, b, c, d, e, none)
    7. Are all contained facts correct? (N/A, no, yes)
    8. Any unnecessary information? (no, yes)

    QUESTION: ...

###  D.3 Slightly Paraphrased Prompt

This prompt slightly differs from the simple prompt in wording choice.

    Evaluate the given explanation based on these criteria:

    1. Which answer does it support? (a, b, c, d, e, none)
    2. Rate the overall explanation (1, 2, 3, 4, 5)
    3. Is it well-written? (no, yes)
    4. Is the explanation relevant to the question and answers?  (no, yes)
    5. Are all provided facts accurate? (N/A, no, yes)
    6. How much new information is given? (none, some, sufficient, ample)
    7. Does it contain unnecessary information? (no, yes)
    8. Is it contrastive? (no, yes)

    QUESTION: ...

###  D.4 Verbose Prompt

Here, the rating questions are more elaborated, but without adding more
information compared to the other prompts.

    Evaluate the explanation provided based on the following criteria:

    1. Which answer choice does the explanation support? (a, b, c, d, e, none)
    2. Rate the overall quality of the explanation. (1, 2, 3, 4, 5)
    3. Is the explanation well-written? (no, yes)
    4. Is the explanation related to the question and answer choices? (no, yes)
    5. Verify the accuracy of all facts, if any, mentioned in the explanation. (N/A, no, yes)
    6. How much new information is given in the explanation? (none, some, sufficient, ample)
    7. Does the explanation contain any unnecessary information? (no, yes)
    8. Is the explanation contrastive? (no, yes)

    QUESTION: ...

###  D.5 Repeated Instructions Prompt

This prompt is identical to the simple prompt except for the example
formatting. Instead of showing simple successions of questions, explanations,
and ratings, the full instructions are repeated every time.

###  D.6 Swapped Instructions Prompt

The wording is the same as the simple prompt, but the order in which the
instructions are presented is swapped. I.e., the example is shown first, then
the general instruction text, followed by the rating questions. Similarly to
the repeated instructions prompt (§D.5), the full instructions are repeated
every time.

    QUESTION: ...

    EXPLANATION: ...

    Evaluate the given explanation according to the following criteria:

    1. Which answer does ...

##  Appendix E Crowdsourced Explanations for BCOPA

As a supplement to COPA-SSE, We collected 3,000 hand-written, detailed, and
contrastive explanations for BCOPA. All crowdsourcing was conducted on the
Amazon Mechanical Turk platform.666https://requester.mturk.com/ Generally, we
found that strict qualifications, hand-picking the final worker pool, and
maintaining open communication led to a significant increase in data quality.

#### Qualifications

In the qualification rounds, workers were asked to write three explanations
for an explanation, its mirrored sample,777With the same answer choices but a
different question that makes the alternative correct. and another random
explanation. 397 workers were hand-picked and whitelisted for further rounds
based on the quality of their responses. We limited participation in this
qualification to workers with a HIT approval rate of 99% or more, 1,000 or
more approved HITs, and located in Great Britain or the United States.

#### Main Rounds

The workers were explicitly instructed to write contrastive explanations which
“focus on what the key difference between the options is, and how it leads to
it being the correct choice in one case and not in the other.” This
instruction was inspired by insights from Miller (2017). Figure 6 shows the
crowdsourcing form. We collected two explanations per BCOPA question, totaling
3,000 explanations. Out of these, a random sample of 500 explanations was
rated and included in our test set. Example explanations can be seen in
Appendix A.2.

#### Payment Information

For qualifications, each participating worker was compensated $0.10 per HIT.
For the main rounds, the fee was increased to $0.89 per HIT, roughly matching
a payment of $20.00 per hour.

CoS-E (best) Which effect of stress could cause death if not treated immediately? a) age grieving person b) heart attacks c) depression d) hair loss e) headache Explanation: heart attacks can stop your heart from properly pumping blood, thus leading to death. | Overall: | 4.4 | Well-written: | Yes
---|---|---|---|---
Supports: | b) | Related: | Yes
|  | Factual: | Yes
|  | New info: | 1.8
|  | Unn. info: | No
|  | Contrastive: | No
CoS-E (worst) The college kids weren’t known for their apartment decor choices, they used a folding chair inside their what? a) bathroom b) closet c) school d) room e) church Explanation: after years of being held captive in an isolated | Overall: | 1.0 | Well-written: | No
Supports: | none | Related: | No
|  | Factual: | N/A
|  | New info: | 0.2
|  | Unn. info: | Yes
|  | Contrastive: | No
CoS-E + fluency fixed (best) What kind of driving leads to accidents? a) stressful b) dangerous c) fun d) illegal e) deadly Explanation: Driving dangerously creates accidents. | Overall: | 4.0 | Well-written: | Yes
Supports: | b) | Related: | Yes
|  | Factual: | Yes
|  | New info: | 0.6
|  | Unn. info: | No
|  | Contrastive: | No
CoS-E + fluency fixed (worst) What is a grumpy person likely to have? a) headache b) understand themselves c) compliments d) own house e) happiness Explanation: A peculiar person. | Overall: | 1.0 | Well-written: | No
Supports: | None | Related: | No
|  | Factual: | N/A
|  | New info: | 0.2
|  | Unn. info: | Yes
|  | Contrastive: | No
Generated (best) The cloth’s are not yet placed where they belong, where should we put them? a) linen closet b) kitchen c) on the space heater d) fabric store e) bedroom Explanation: linen closets are typically used to store items such as cloths. | Overall: | 4.8 | Well-written: | Yes
Supports: | a) | Related: | Yes
|  | Factual: | Yes
|  | New info: | 1.4
|  | Unn. info: | No
|  | Contrastive: | No
Generated (worst) Sarah opened his chest and found a second heart. Her patient might not be what? a) istanbul b) animal c) antique shop d) human being e) bedroom Explanation: a human being is the only creature known to have two hearts. | Overall: | 1.2 | Well-written: | Yes
Supports: | d) | Related: | Yes
|  | Factual: | No
|  | New info: | 0.8
|  | Unn. info: | No
|  | Contrastive: | No
ECQA (best) Where would one find a captive monkey? a) rain forest b) tree tops c) toy store d) zoological gardens e) great outdoors Explanation: A monkey can be kept captive at a zoological garden as many animals are kept there for people to see. All the other options are incorrect as they do not have captive monkeys. | Overall: | 4.6 | Well-written: | Yes
Supports: | c) | Related: | Yes
|  | Factual: | Yes
|  | New info: | 1.8
|  | Unn. info: | No
|  | Contrastive: | Yes
ECQA (worst) Where are there lots of seats placed in rows surrounding a court? a) show b) auditorium c) movies d) soccer stadium e) hockey game Explanation: In auditorium, there are lots of seats placed in rows. Auditorium seats are surrounding a court. In show, there are no surrounding court placement of seats. Movies is not a place. In soccer stadium, seats are not placed in rows. In hockey game, there are no proper seats. | Overall: | 1.6 | Well-written: | No
Supports: | b) | Related: | Yes
|  | Factual: | No
|  | New info: | 1.0
|  | Unn. info: | No
|  | Contrastive: | Yes
Table 4: Top-rated and bottom-rated examples of explanations for CommonsenseQA
in our test set. The ratings on the right are aggregated from five
crowdworkers’ annotations. Well-written ranges from $1.0$ to $5.0$, and New
info. from $0.0$ to $3.0$.  COPA-SSE (best) The vehicles travelled slowly.
What was the cause of this? a) It was foggy.
b) It was cloudy. Explanation: Fog is a thick mist which prevents you from driving quickly as you cannot see far in front of your vehicle. Driving slow through is motivated by the desire not to crash. | Overall: | 4.8 | Well-written: | Yes
---|---|---|---|---
Supports: | a) | Related: | Yes
|  | Factual: | Yes
|  | New info: | 2.0
|  | Unn. info: | No
|  | Contrastive: | No
COPA-SSE (worst) The boy found the television show interesting. What happened
as a result? a) He shut it off.
b) He recorded it. Explanation: The result causes the record shut off. | Overall: | 1.0 | Well-written: | No
Supports: | None | Related: | Yes
|  | Factual: | N/A
|  | New info: | 0.4
|  | Unn. info: | Yes
|  | Contrastive: | No
COPA-SSE + Fluency fixed (best) The woman tolerated her friend’s difficult
behavior. What was the cause of this? a) The woman knew her friend was going
through a hard time.
b) The woman felt that her friend took advantage of her kindness. Explanation: Going through a hard time can cause people to act in a difficult way. A friend can be motivated to support their friend when they’re having a tough time. | Overall: | 4.8 | Well-written: | Yes
Supports: | a) | Related: | Yes
|  | Factual: | Yes
|  | New info: | 2.2
|  | Unn. info: | No
|  | Contrastive: | No
COPA-SSE + Fluency fixed (worst) The shirt did not have a hole anymore. What
was the cause of this? a) I patched the shirt.
b) I bleached the shirt. Explanation: I desire to look good. | Overall: | 1.0 | Well-written: | No
Supports: | None | Related: | No
|  | Factual: | N/A
|  | New info: | 0.2
|  | Unn. info: | Yes
|  | Contrastive: | No
Generated (best) The boy was struggling to fall asleep. What was the cause of
this? a) He turned on his nightlight.
b) He watched a horror movie. Explanation: Watching a horror movie can create fear and anxiety that can make it difficult for someone to fall asleep. The nightlight is not likely to be the cause since it is meant to help someone sleep. | Overall: | 5.0 | Well-written: | Yes
Supports: | b) | Related: | Yes
|  | Factual: | Yes
|  | New info: | 2.4
|  | Unn. info: | No
|  | Contrastive: | Yes
Generated (worst) The woman awakened. What was the cause of this? a) She took
sleeping pills.
b) Her husband was snoring. Explanation: Taking sleeping pills is a direct cause of awakening, whereas a husband snoring is an indirect cause. | Overall: | 1.6 | Well-written: | Yes
Supports: | a) | Related: | Yes
|  | Factual: | No
|  | New info: | 1.4
|  | Unn. info: | No
|  | Contrastive: | Yes
Crowdsourced (best) The man was bitten by mosquitoes. What was the cause of
this? a) He fell asleep on his couch.
b) He went camping in the woods. Explanation: Sleeping on a couch would mean you are indoors where there are rarely mosquitoes. Mosquitoes are prevalent in wooded areas, so the man would be more likely to be camping in the woods if he was bit by mosquitoes. | Overall: | 5.0 | Well-written: | Yes
Supports: | b) | Related: | Yes
|  | Factual: | Yes
|  | New info: | 3.0
|  | Unn. info: | No
|  | Contrastive: | Yes
Crowdsourced (worst) I refilled my water bottle. What was the cause of this?
a) I drank all the water in it.
b) I kept it in the fridge. Explanation: If you kept your water in the fridge, it would not need to be refilled. | Overall: | 2.2 | Well-written: | Yes
Supports: | a) | Related: | Yes
|  | Factual: | Yes
|  | New info: | 1.0
|  | Unn. info: | No
|  | Contrastive: | Yes
Table 5: Top-rated and bottom-rated examples of explanations for BCOPA in our
test set. The ratings on the right are aggregated from five crowdworkers’
annotations. Well-written ranges from $1.0$ to $5.0$, and New info. from $0.0$
to $3.0$.  Figure 6: Explanation writing AMT form. Figure 7: Explanation
rating AMT form.


# Development of a CsI Calorimeter for the Compton-Pair (ComPair) Balloon-
Borne Gamma-Ray Telescope

Daniel Shy, Richard S. Woolf, Clio C. Sleator, Eric A. Wulf, Mary Johnson-
Rambert, Emily Kong, J. Mitch Davis, Thomas J. Caligiure, J. Eric Grove, and
Bernard F. Phlips This work was sponsored by NASA-APRA (NNH14ZDA001N-APRA,
NNH15ZDA001N-APRA, and NNH1870A001N-APRA). The ComPair balloon flight is
supported under NNH21ZDA001N-APRAD. Shy is supported by U.S. Naval Research
Laboratory’s Karles Fellowship.D. Shy, R. Woolf, C. Sleator, E. Wulf, M.
Johnson-Rambert, J. M. Davis, J. E. Grove, B. Phlips are with the Space
Science Division, U.S. Naval Research Laboratory, 4555 Overlook Ave., SW,
Washington, DC, 20375, United States of AmericaE. Kong is with the Technology
Service Corporation, Arlington, VA, 22202, United States of AmericaT. Caligure
is with the Naval Research Enterprise Internship Program (NREIP) at the U.S.
Naval Research Laboratory

###### Abstract

There is a growing interest in astrophysics to fill in the observational
gamma-ray MeV gap. We, therefore, developed a CsI:Tl calorimeter prototype as
a subsystem to a balloon-based Compton and Pair-production telescope known as
ComPair. ComPair is a technology demonstrator for a gamma-ray telescope in the
MeV range that is comprised of 4 subsystems: the double-sided silicon
detector, virtual Frisch grid CdZnTe, CsI calorimeter, and a plastic-based
anti-coincidence detector. The prototype CsI calorimeter is composed of thirty
CsI logs, each with a geometry of $1.67\times 1.67\times 10\ \mathrm{cm^{3}}$.
The logs are arranged in a hodoscopic fashion with 6 in a row that alternate
directions in each layer. Each log has a resolution of around $8\%$ full-
width-at-half-maximum (FWHM) at $662\ \mathrm{keV}$ with a dynamic energy
range of around $250\ \mathrm{keV}-30\ \mathrm{MeV}$. A $2\times 2$ array of
SensL J-series SiPMs read out each end of the log to estimate the depth of
interaction and energy deposition with signals read out with an IDEAS ROSSPAD.
We also utilize an Arduino to synchronize with the other ComPair subsystems
that comprise the full telescope. This work presents the development and
performance of the calorimeter, its testing in thermal and vacuum conditions,
and results from irradiation by $2-25\ \mathrm{MeV}$ monoenergetic gamma-ray
beams. The CsI calorimeter will fly onboard ComPair as a balloon experiment in
the summer of 2023.

##  I Introduction

There are several space-based gamma-ray telescopes in the concept and
development phase to address the MeV gap in astronomical observations. These
efforts include the Compton Spectrometer and Imager (COSI) [1], Galactic
Explorer with a Coded aperture mask Compton telescope (GECCO) [2], e-ASTROGAM
[3], Advanced Particle-astrophysics Telescope (APT) [4], SMILE-2+ [5], and the
All-sky Medium Energy Gamma-ray Observatory (AMEGO) [6], which is now adapted
to AMEGO-X [7]. This work presents the development of a thallium-doped cesium
iodide (CsI:Tl) based calorimeter in support of the ComPair balloon-based
telescope [8], which serves as a prototype to AMEGO and will be flown in the
summer of 2023. The AMEGO concept consists of four subsystems: the double-
sided Silicon strip detectors (DSSD) [9], virtual Frisch-grid CdZnTe
calorimeter [10], CsI calorimeter, and a plastic-based anti-coincidence
detector (ACD) [8]. The main concept behind this ‘hybrid’ telescope design is
to maintain high detection sensitivity and good imaging performance through a
large energy range that includes the Compton and pair-creation region. The
tracker layer has the dual objective of measuring the energy of a gamma-ray
Compton scatter as well as tracking electromagnetic showers during a pair-
creation event. Next, the CZT, with its high resolution in the region below 10
MeV could capture the scattered photon. Higher energy events may pass through
the CZT to get detected by the CsI calorimeter. The concept, therefore, has a
dual use of both Compton and Pair imaging combining the COSI/COMPTEL region
and that of the Fermi Large Area Telescope.

Fig. 1 presents a computer aided design (CAD) model of the ComPair balloon
instrument with the different subsystems labeled. The CsI calorimeter box is
at the bottom of the stack colored in orange. The ComPair CsI calorimeter
inherits its design and concept from the calorimeter on board the Fermi Large
Area Telescope (LAT) [11, 12, 13]. The major difference between the LAT
calorimeter and that of ComPair is the usage of SiPMs rather than PIN diodes.
The diodes utilized in the LAT calorimeter did not have the necessary lower
threshold required in ComPair and therefore explored using SiPMs.

Figure 1: Cutaway view of the CAD showing ComPair’s subsystems and their
dimensions. Figure is adopted from [8].

This manuscript presents an expansion and finalization of the work presented
in [14], which contains preliminary results during an early phase of
development. Sec. II presents the hardware and implementation. Sec. III
presents the calorimeter calibration pipeline. Sec. IV presents the
calorimeter’s performance and its characteristics. Sec. V has results from a
thermal-vacuum test. Sec. VI presents results from $2-25\ \mathrm{MeV}$ gamma-
ray beam test.

##  II Hardware

The ComPair CsI calorimeter consists of 30 thallium doped CsI logs that are
$1.67\times 1.67\times 10\ \mathrm{cm}^{3}$ in volume that were manufactured
by Saint-Gobain. Each end contains a $2\times 2$ array of $6\times 6\
\mathrm{mm}^{2}$ SensL J-series SiPMs by onsemi (ARRAYJ-60035-4P-PCB [15]).
This scheme allows for the depth of interaction (DOI) estimation using (1)
while we use (2) to calculate the energy deposited.

$\mathrm{DOI}=\frac{Q_{left}-Q_{right}}{Q_{left}+Q_{right}}.$ (1)

$E_{\mathrm{ADC}}=\sqrt{Q_{left}\times Q_{right}}.$ (2)

In the previous equations, $Q$ represents the SiPM signal recorded by the left
and right SiPM for a given log. Since the DOI is a dimensionless quantity, a
position calibration will need to be applied to resolve the position of
interaction. $E_{\mathrm{ADC}}$ represents the energy deposited that will need
to be energy corrected. The energy and position reconstruction techniques take
heritage from the Fermi LAT calorimeter.

The CsI logs are roughened on all six sides with sandpaper to produce depth-
dependent light output. The logs are then wrapped with two layers of Tetratex
[16]. The SiPMs are then epoxied at each end of the log. The logs are then
arranged hodoscopically in rows of six such that each layer alternates
direction and are held inside a 3D-printed structure. Fig. 1 shows the
assembly located at the right side of the box with the logs shown in yellow.

The SiPMs are then read out by an IDEAS ROSSPAD [17], which is a front-end
that packages four SiPHRA ASICs [18]. We replaced the onboard power supply on
the ROSSPAD with a custom unit to provide the additional current required to
operate the large number of SiPMs. The SiPMs are biased to $27.5\ \mathrm{V}$.
Note that the breakdown voltage for the J-Series is between $24.2-24.7\
\mathrm{V}$. Fig. 2 shows the fully assembled subsystem with the CsI assembly
located in the upper left, the ROSSPAD in the upper right, and the power
supply located in the lower left.

Figure 2: The CsI calorimeter system in its final configuration. Figure is
adapted from [8].

Peripheral hardware that supports the subsystem is located within the lower
right quadrant. It is comprised of two Arduino Dues, which are single-board
microcontrollers with a set of digital and analog input/output (I/O) pins that
are commonly used in the hobbyist community [19]. One Arduino is utilized for
housekeeping: it monitors six thermocouples that are distributed within the
box as well as a real-time clock. The second Arduino is used to synchronize
with the other ComPair subsystems. The synchronization between subsystems
works as follows: when each subsystem detects an event, a trigger is
transmitted to the ComPair trigger module [20]. Then, based on some trigger
logic (e.g., a coincidence between two or more subsystems), the module
responds with a trigger acknowledgment along with a numerical Event ID. The
ROSSPAD cannot receive EventIDs; instead we use a second Arduino Due to
receive trigger acknowledgments and EventIDs from the ComPair TM, in addition
to receiving a trigger signal directly from the ROSSPAD. Thus, the Arduino can
accurately insert the Event IDs from the trigger module into the ROSSPAD data
stream. In addition to maintaining Event IDs, the CsI subsystem also takes in
a GPS-disciplined pulse per second (PPS) signal which allows for the
correction of the on-board clocks. The subsystem communicates to the trigger
module through two TTL-LVDS boards.

The calorimeter is controlled by an external computer that records the ROSSPAD
events as well as data received by the Arduino. During the balloon flight, the
calorimeter will be controlled by a Versalogic BayCat single board computer
[8].

##  III Position and Energy Calibrations

The position and energy calibrations utilize a multitude of variables within
each log. Simply, we apply an energy correction for each DOI voxel within a
log. The detailed procedure is as follows:

  1. 1.

Take flood irradiations with multiple sources
(${}^{22}\mathrm{Na},^{137}\mathrm{Cs},^{228}\mathrm{Th}$), which allows for
corrections across different energies.

  2. 2.

Develop an energy-position density plot like the one in Fig. 3(a). There, we
locate the full-energy peak for a given DOI, i.e., the ADC value associated
with the $511\ \mathrm{keV}$ line at multiple DOIs.

  3. 3.

Calculate the gain correction for that given DOI and repeat that for each
subsequent depth and energy. For each log, we implement 150 depth bins, which
equate to roughly $\sim 1\ \mathrm{mm}$ bins. This results in a DOI-dependent
calibration mask that can be applied to compensate for the different photopeak
centroids.

  4. 4.

Apply the gain calibration to convert from ADC to energy.

  5. 5.

The conversion from the DOI to the true depth is done by bounding the DOI
values for the full-energy peak and calculating the DOI’s distribution
centroid. This is done by creating a histogram of DOI values for a given
energy, and then calculating the min/max of that distribution as well as the
average.

  6. 6.

The DOI values are then scaled and shifted to the appropriate positions and
result in Fig. 3(b). This scheme assumes a linear DOI response across the
length of the log.

  7. 7.

We apply an ADC-to-energy non-linearity correction by studying the gain
calibration value for each given DOI across different energies and fitting a
quadratic curve across them.

((a)) Raw

((b)) Corrected

Figure 3: Depth-energy histograms of a single log’s response to a
${}^{22}\mathrm{Na}$. (a) Presents the raw response clearly showing the
$0.511$ and $1.27\ \mathrm{MeV}$ peak and its distribution along the depth
(DOI). (b) Applies the calibration routine which corrects the energy and
position. The correction clearly shows the alignment of the full-energy peak
across all depths due to the calibration.

In step 7, the non-linearity could originate in the SiPM [21] or the readout
electronics [18]. In addition, variations in the ASIC pedestal from the
calibration and actual measurement could also affect the calorimeter’s
response. Fig. 4 plots the different calibrations for all depths and energies
for each log along with the quadratic fits. As seen in the plot, most of the
calibration fits do match the recorded photopeak centroid and have good
agreements across depth. Finally, we also apply a correction as a function of
energy on the DOI range and centroids by stretching and shifting those values
across energy.

Figure 4: Quadratic calibration curves that plot the recorded ADC
$(E_{\mathrm{ADC}})$ vs. the line energies of the radioactive lab sources for
each log. The scatter points represent experimental data while the red curves
represent quadratic fits. We plot the data for all depths for a given log but
the traces are not very visible in most logs due to their relatively low
variance such that most features are superimposed. The axis for all the
subplots utilizes the same numerical range as that of the lower-leftmost
subplots.

##  IV Calorimeter Performance

Fig. 5 plots the spectra taken with a ${}^{22}\mathrm{Na}$ source flood
irradiation for each log. The resolution averaged over all logs is $7.9\%$
FWHM at $662\ \mathrm{keV}$ and $6.8\%$ FWHM at $1.274\ \mathrm{MeV}$. The
poor performance of several logs is still under investigation. A focused
assessment revealed that their performance is not due to the SiPM or the
readout electronics, but the combination of the two. Our leading theory is
that there might be an issue with the connection between the SiPM and the
ASIC. We note that the resolution does not scale with the inverse square root
of the energy as expected; this is likely due to the accumulation of many
small errors in correcting the response over multiple depths and multiple
logs.

The log’s depth of interaction position resolution is estimated to be around
$\sim 1\ \mathrm{cm}$ FWHM. The position resolution was estimated via a
collimated fan measurement accomplished with two lead bricks separated by
American (¢) pennies ($1.52\ \mathrm{mm}$). Extrapolating from the energy
calibration, we estimate the upper energy range of the system to average
around $30\ \mathrm{MeV}$ using this 12-bit ADC readout [18] with a lower
energy limit of $250\ \mathrm{keV}$. (We show the performance of the system to
25 MeV in Sec. VI).

Figure 5: Corrected ${}^{22}\mathrm{Na}$ spectra for each log in the CsI
calorimeter. Log [3,4] has a very high lower-threshold which results in a high
continuum to $1.274\ \mathrm{MeV}$ peak ratio. The source of this is still
under investigation.

We also studied the muon response of the calorimeter to demonstrate the
ability of the entire calorimeter to measure events at the 10s of MeV. Fig. 6
plots the response from an overnight measurement where (a) presents a spectrum
of a single log while (b) plots the spectrum of the interactions in all logs
summed. In the measurement, only internal triggers were used. Both spectra
present a Landau distribution and continuum with the expected energy response
for muons in the calorimeter [22]. To first order, assuming a
$<-\mathrm{d}E_{\mathrm{min}}/\mathrm{d}x>=1.243\ \mathrm{MeV\ cm^{2}/g}$ (the
minimum in the curve [23]), with a CsI density of $4.51\ \mathrm{g/cm^{3}}$
and a thickness $1.67$, we get $9.1\ \mathrm{MeV}$ deposition, close to the
minimum in the measured distribution. Because the energy deposited by the
muons is a distribution, the muon spectra will be skewed right to higher
energies.

In (b), the spectrum is roughly 5 times the energy of the distribution of (a)
which is an appropriate scale since the calorimeter is 5 layers deep. Note
that we did not implement any event selection criteria in (b).

((a)) Single Bar Response

((b)) All Logs Summed

Figure 6: Energy spectra of muons. (a) Presents the response of a single bar
while (b) sums the interactions that occurred in all logs for a given event.

##  V Environmental Testing

We tested the prototype in a thermal vacuum chamber (TVAC), as is standard in
most balloon-borne instruments [24]. The test intends to verify the
functionality and thermal design of the calorimeter instrument over the
hardware temperature limits expected during a high-altitude balloon flight.
The procedure was an abbreviated version of the NASA General Environmental
Verification Standard (GEVS) standard [25]. Fig. 7 shows the calorimeter in
the chamber.

Fig. 8 plots the environmental conditions that the prototype experienced
during testing. The x-axis plots the time elapsed since the vacuum pump was
turned on. The solid-line trace shows the pressure in the chamber while the
dashed lines show the thermocouple temperatures that are placed on the
chamber’s baseplate and two on the exterior of the calorimeter. We completed
survival tests at $-30^{\circ}\mathrm{C}$ and $40^{\circ}\mathrm{C}$ where the
system was turned off. We then conducted operational tests and characterized
the system in temperatures between $-20^{\circ}\mathrm{C}$ and
$30^{\circ}\mathrm{C}$. The thermal cycle includes one survival (hot and cold)
cycle with one operational cycle; the plateaus lasted for four hours during
the survival cycle and two hours for the operational cycles. Temperature
stabilization is achieved when the control thermocouple is within
$1^{\circ}\mathrm{C}$ of the soak goal temperature and does not change by more
than $3^{\circ}\mathrm{C}$. The thermal portion of the TVAC lasted around 3
days.

Figure 7: The CsI calorimeter located in the environmental chamber. Figure 8:
Environmental chamber conditions during TVAC testing. The solid line plots the
chamber pressure while the dashed lines plot the temperatures experienced by
the external thermocouple.

###  V-A Temperature Response

We placed thoriated welding rods at the top and the bottom of the chamber to
provide a calibration source ($2.614\ \mathrm{MeV}$ line). The major isotope
in the rods is most likely 232Th with a gamma-ray spectrum available in [26].
During the TVAC test, we collected a 15-minute run at several checkpoints to
develop a calibration curve as a function of temperature as we may experience
a variety of temperatures during the balloon flight. We observed no
significant difference when the system was in one atmosphere or under a
vacuum. After the hot and cold survival cycle, beginning at hour 48, we ramped
up from $-20^{\circ}\mathrm{C}$ to $30^{\circ}\mathrm{C}$ in
$10^{\circ}\mathrm{C}$ steps and dwelled at each of those temperatures to soak
for 1.5 hours which was followed by a calibration run. Fig. 9 plots the energy
response of the calorimeter at different temperatures where multiple
interactions for a given event are not summed. The spectra used an ASIC
pedestal subtraction value and energy calibration that was performed at room
temperature before the start of the experiment. The trend shows that the peak
centroid increases as temperature decreases which is consistent with the ASIC
manufacturer’s expected response. In addition, the SiPM’s gain will increase
with the decrease in temperature. Convolved with the response is also the
crystal light output [27].

Figure 9: Thoriated-rod spectra gated at the $2.614\ \mathrm{MeV}$ peak taken
during TVAC testing. In the plot, each interaction is treated individually.

Fig. 10 plots the $2.614\ \mathrm{MeV}$ centroid as a function of different
temperatures. The vertical bars represent the FWHM calculated using only the
upper half of the peak. The red line presents an exponential fit. We intend to
use the fit as a gain correction across all energies for each log. We did not
complete a comprehensive study for each depth-energy bin due to the limited
available time we had with the TVAC chamber.

Figure 10: Calibration curve of the $2.614\ \mathrm{MeV}$ peak as a function
of temperature. The vertical bars represent the FWHM measured using the upper-
half of the peak. The fit of the full-energy peak is $-0.3e^{0.044T}+3.28$.

Finally, the SiPHRA ASIC’s baseline pedestal, which is an electrical offset
that is added to each channel to prevent the value from going negative, also
shows a significant temperature dependence. Fig. 11 plots the baseline as a
function of temperature and shows a linear relationship between the two. The
baseline was measured by force triggering all channels and observing the
baseline in each channel.

Figure 11: Baseline pedestal of the SiPHRA ASICs as a function of temperature.

##  VI Results from High-Energy Gamma-ray Beam Test

The ComPair telescope was taken to the High Intensity Gamma-ray Source (HIGS)
located at the Triangle Universities Nuclear Laboratory (TUNL). This section
only presents CsI Calorimeter results while some full instrument results are
presented in [8]. In the experiment, there are 5 layers of $60\ \mu\mathrm{m}$
DSSDs, $2\ \mathrm{cm}$ of CdZnTe, and additional passive material between the
beam and the CsI calorimeter that equate to almost $13\ \mathrm{g/cm}^{2}$
worth of grammage. This material can be seen in Fig. 1 minus the ACD.

HIGS can produce mono-energetic gamma rays in a $0.5\ \mathrm{cm}$ diameter
collimated beam. The energy resolution of the beam is near $1\%$ FWHM at $5\
\mathrm{MeV}$. Fig. 12 plots the spectrum for a $5.05\ \mathrm{MeV}$ beam with
Fig. 12a showing the beam incident directly normal to the calorimeter while
Fig. 12b has the instrument angled $15^{\circ}$ away from normal. The spectra
show a prominent single escape peak with a hint of a double escape peak. The
full-energy peak manifests as a fall-off feature. The resolution of the
spectra differs in the two configurations, which is most likely due to the
different number of interactions for a given event due to the detector’s
geometry.

((a))

((b))

Figure 12: $5.05\ \mathrm{MeV}$ spectra summing all the interactions. (a)
Shows the spectra for a beam incident normal to instrument while (b) has the
beam angles $15^{\circ}$.

The other energies explored include $2,7,15$ and $25\ \mathrm{MeV}$. Fig. 13
shows their spectra for a normal beam incidence. At $7\ \mathrm{MeV}$ and
above, the spectrum becomes more continuum-like and loses the full energy peak
because of the underlying physics and their cross-sections where the main mode
of interaction shifts from Compton to pair-production. We also observe that
the full-energy efficiency decreases and the ensuing electromagnetic showers
tend to escape the system. This phenomenon has been observed in other
experiments [28]. Appendix A presents several simulations of the conducted
experiment showing relativly good agreement between experiment and simulation.
There, we experimentally show that the system has the dynamic range to measure
gamma rays at 25 MeV.

Figure 13: Spectra of the total energy deposited in the calorimeter. The
beam’s vector is normal to the instrument.

##  VII Conclusion and Future Work

This work presents the development of a CsI-based calorimeter as part of the
ComPair telescope. The instrument specifically serves as a prototype of the
AMEGO/AMEGO-X mission concept.

The calorimeter displays an energy resolution that averages around $8\%$ per
log with an energy dynamic range of $0.25-30\ \mathrm{MeV}$ extrapolated using
the 12-bit IDEAS ROSSPAD. We also demonstrated the calorimeter has the energy
range to measure 25 MeV gamma rays and muon traversing the entire calorimeter
equating to more than 60 MeV deposition. The calorimeter, along with the rest
of the ComPair telescope, is slated to fly as a short-duration high-altitude
balloon experiment out of Fort Sumner, NM in the summer of 2023. The flight
will be hosted by NASA’s Columbia Scientific Balloon Facility.

We are currently developing the next iteration of the prototype which is meant
to resemble an AMEGO-X configuration that will make up a single tower. This
includes utilizing larger logs that are slated to be $388\ \mathrm{mm}$ along
with a $15\times 15\ \mathrm{mm}$ cross-section. The new prototype will
utilize a custom front-end readout developed specifically for the CsI
calorimeter.

The SiPMs in this work did not have the desired energy range of $250\
\mathrm{keV}-500\ \mathrm{MeV}$. Therefore, we are developing custom SiPM
readouts to address energy non-linearity and extend the dynamic range when
reading out the scintillators. The SiPM readouts will contain two species of
SiPMs that are optimized for either low or high energy ranges in terms of
their microcell and overall size. This allows for a “dual-gain” approach that
was inspired by the Fermi-LAT’s “dual-gain” PIN diodes [12]. The development
of dual-gain SiPMs will be reported on in a future publication.

## Acknowledgment

We thank Iker Liceaga-Indart from NASA Goddard Space Flight Center for
developing the calorimeter housing.

We are grateful to the accelerator physics group led by Prof. Y.K. Wu at High
Intensity Gamma-ray Source (HIGS), Triangle Universities Nuclear Laboratory
for their aid in an excellent $\gamma$-ray beam test.

##  Appendix A Simulation of the System

This section presents simulations of the experimental setup as the HIGS
facility using the SoftWare for Optimization of Radiation Detectors (SWORD)
simulation package [29]. Like the experimental setup, we shot a $0.5\
\mathrm{mm}$ pencil beam into the system. Fig. 14 shows the system modeled in
SWORD. Figs. 15, 16, 17, 18, and 19 plots the full summed response of the
calorimeter to 2, 5, 7, 15, and 25 MeV beams. Note that in the simulations, we
did not add any energy blurring due to calorimeter performance as a means to
display the underlying physical process. In the plots, we can see that the
photopeak to continuum ratio decreases with energy and the escape peaks become
more prominent with increasing energy. The experimental $5\ \mathrm{MeV}$
full-energy peak appears less prominent than in simulation is likely due to
the experienced broadening of the escape peaks and sitting on top of the
Compton/pair continuum. Next, at higher energy, we can see the Compton edge
feature disappear and become more of a continuum distribution. This is a
result of the decrease in full-containment and leakage of the electromagnetic
showers.

Figure 14: CAD model of the SWORD simulation modeling the ComPair system and
environment experienced in the HIGS beam test. Figure 15: Simulated response
of the CsI Calorimeter to 2 MeV gamma rays. Figure 16: Simulated response of
the CsI Calorimeter to 5 MeV gamma rays. Figure 17: Simulated response of the
CsI Calorimeter to 7 MeV gamma rays. Figure 18: Simulated response of the CsI
Calorimeter to 15 MeV gamma rays. Figure 19: Simulated response of the CsI
Calorimeter to 25 MeV gamma rays.

## References

  * [1] J. A. Tomsick, S. E. Boggs, A. Zoglauer, E. Wulf, L. Mitchell, B. Phlips, C. Sleator, T. Brandt, A. Shih, J. Roberts, P. Jean, P. von Ballmoos, J. M. Oliveros, A. Smale, C. Kierans, D. Hartmann, M. Leising, M. Ajello, E. Burns, C. Fryer, P. Saint-Hilaire, J. Malzac, F. Tavecchio, V. Fioretti, A. Bulgarelli, G. Ghirlanda, H.-K. Chang, T. Takahashi, K. Nakazawa, S. Matsumoto, T. Melia, T. Siegert, A. Lowell, H. Lazar, J. Beechert, and H. Gulick, “The Compton Spectrometer and Imager Project for MeV Astronomy,” 2021. [Online]. Available: https://arxiv.org/abs/2109.10403
  * [2] E. Orlando _et al._ , “Exploring the MeV sky with a combined coded mask and Compton telescope: the Galactic Explorer with a Coded aperture mask Compton telescope (GECCO),” _Journal of Cosmology and Astroparticle Physics_ , vol. 2022, no. 07, p. 036, jul 2022. [Online]. Available: https://doi.org/10.1088/1475-7516/2022/07/036
  * [3] A. De Angelis and The e-ASTROGAM Collaboration, “The e-ASTROGAM mission,” _Experimental Astronomy_ , vol. 44, no. 1, pp. 25–82, Oct 2017. [Online]. Available: https://doi.org/10.1007/s10686-017-9533-6
  * [4] J. Buckley, L. Bergstrom, B. Binns, J. Buhler, W. Chen, M. Cherry, S. Funk, D. Hooper, J. Mitchell, G. D. Nolfo, S. A. Nussirat, S. Profumo, B. Rauch, D. Stern, G. Varner, S. Wakely, and A. Zink, “The Advanced Particle-astrophysics Telescope (APT),” _Bulletin of the AAS_ , vol. 51, no. 7, sep 30 2019, https://baas.aas.org/pub/2020n7i078.
  * [5] A. Takada, T. Takemura, K. Yoshikawa, Y. Mizumura, T. Ikeda, Y. Nakamura, K. Onozaka, M. Abe, K. Hamaguchi, H. Kubo, S. Kurosawa, K. Miuchi, K. Saito, T. Sawano, and T. Tanimori, “First Observation of the MeV Gamma-Ray Universe with Bijective Imaging Spectroscopy Using the Electron-tracking Compton Telescope on Board SMILE-2+,” _The Astrophysical Journal_ , vol. 930, 2022.
  * [6] C. A. Kierans, “AMEGO: exploring the extreme multimessenger universe,” in _Space Telescopes and Instrumentation 2020: Ultraviolet to Gamma Ray_ , J.-W. A. den Herder, S. Nikzad, and K. Nakazawa, Eds., vol. 11444, International Society for Optics and Photonics. SPIE, 2020, p. 1144431. [Online]. Available: https://doi.org/10.1117/12.2562352
  * [7] R. Caputo _et al._ , “All-sky Medium Energy Gamma-ray Observatory eXplorer mission concept,” _Journal of Astronomical Telescopes, Instruments, and Systems_ , vol. 8, no. 4, p. 044003, 2022. [Online]. Available: https://doi.org/10.1117/1.JATIS.8.4.044003
  * [8] D. Shy, C. Kierans, N. Cannady, R. Caputo, S. Griffin, J. E. Grove, E. Hays, E. Kong, N. Kirschner, I. Liceaga-Indart, J. McEnery, J. Mitchell, A. A. Moiseev, L. Parker, J. S. Perkins, B. Phlips, M. Sasaki, A. J. Schoenwald, C. Sleator, J. Smith, L. D. Smith, S. Wasti, R. Woolf, E. Wulf, and A. Zajczyk, “Development of the ComPair gamma-ray telescope prototype,” in _Space Telescopes and Instrumentation 2022: Ultraviolet to Gamma Ray_ , J.-W. A. den Herder, S. Nikzad, and K. Nakazawa, Eds., vol. 12181, International Society for Optics and Photonics. SPIE, 2022, p. 121812G. [Online]. Available: https://doi.org/10.1117/12.2628811
  * [9] S. Griffin and the AMEGO Team, “Development of a silicon tracker for the all-sky medium energy gamma-ray observatory prototype,” 2019.
  * [10] A. Bolotnikov, J. MacKenzie, E. Chen, F. Kumar, S. Taherion, G. Carini, G. De Geronimo, J. Fried, K. Kim, L. O. Girado, E. Vernon, and R. James, “Performance of 8×8×32 and 10×10×32 mm3 cdznte position-sensitive virtual frisch-grid detectors for high-energy gamma ray cameras,” _Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment_ , vol. 969, p. 164005, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0168900220304587
  * [11] W. Johnson, J. Grove, B. Phlips, J. Ampe, S. Singh, and E. Ponslet, “The construction and performance of the CsI hodoscopic calorimeter for the GLAST beam test engineering module,” _IEEE Transactions on Nuclear Science_ , vol. 48, no. 4, pp. 1182–1189, 2001.
  * [12] J. E. Grove and W. N. Johnson, “The calorimeter of the Fermi Large Area Telescope,” in _Space Telescopes and Instrumentation 2010: Ultraviolet to Gamma Ray_ , M. Arnaud, S. S. Murray, and T. Takahashi, Eds., vol. 7732, International Society for Optics and Photonics. SPIE, 2010, p. 77320J. [Online]. Available: https://doi.org/10.1117/12.857839
  * [13] W. B. Atwood _et al._ , “THE LARGE AREA TELESCOPE ON THE FERMI GAMMA-RAY SPACE TELESCOPE MISSION,” _The Astrophysical Journal_ , vol. 697, no. 2, pp. 1071–1102, may 2009. [Online]. Available: https://doi.org/10.1088/0004-637x/697/2/1071
  * [14] R. S. Woolf, J. E. Grove, B. F. Phlips, and E. A. Wulf, “Development of a CsI:Tl calorimeter subsystem for the All-Sky Medium-Energy Gamma-Ray Observatory (AMEGO),” in _2018 IEEE Nuclear Science Symposium and Medical Imaging Conference Proceedings (NSS/MIC)_ , 2018, pp. 1–6.
  * [15] “ArrayJ-series - silicon photomultiplier (sipm).” [Online]. Available: https://www.onsemi.com/pdf/datasheet/arrayj-series-d.pdf
  * [16] S. P. Stoll, “An Investigation of the Reflective Properties of Tyvek Papers and Tetratex PTFE Film.” [Online]. Available: https://www.fnal.gov/projects/ckm/photon_veto/misc/tyvek_reflection/tyvek_reflection.pdf
  * [17] “Rosspad,” https://ideas.no/products/rosspad/, 2022.
  * [18] A. Ulyanov, D. Murphy, A. Fredriksen, J. Ackermann, D. Meier, N. Nelms, B. Shortt, S. McBreen, and L. Hanlon, “Using the SIPHRA ASIC with an SiPM array and scintillators for gamma spectroscopy,” in _2017 IEEE Nuclear Science Symposium and Medical Imaging Conference (NSS/MIC)_ , 2017, pp. 1–3.
  * [19] “Arduino due.” [Online]. Available: https://store.arduino.cc/products/arduino-due
  * [20] M. Sasaki, “Trigger system for the ComPair instrument,” in _Space Telescopes and Instrumentation 2020: Ultraviolet to Gamma Ray_ , J.-W. A. den Herder, S. Nikzad, and K. Nakazawa, Eds., vol. 11444, International Society for Optics and Photonics. SPIE, 2020, pp. 1023 – 1028. [Online]. Available: https://doi.org/10.1117/12.2561658
  * [21] J. Rosado, “Modeling the nonlinear response of silicon photomultipliers,” _IEEE Sensors Journal_ , vol. 19, no. 24, pp. 12 031–12 039, 2019.
  * [22] L. Landau, “56 - ON THE ENERGY LOSS OF FAST PARTICLES BY IONISATION,” in _Collected Papers of L.D. Landau_ , D. TER HAAR, Ed. Pergamon, 1965, pp. 417–424. [Online]. Available: https://www.sciencedirect.com/science/article/pii/B9780080105864500614
  * [23] D. E. Groom, N. V. Mokhov, and S. I. Striganov, “Muon stopping power and range tables 10 mev–100 tev,” _Atomic Data and Nuclear Data Tables_ , vol. 78, no. 2, pp. 183–356, 2001.
  * [24] J. Ampe, A. Chekhtman, P. Dizon, J. Grove, W. Johnson, B. Leas, D. Sandora, and M. Strickman, “The calibration and environmental testing of the Engineering Module of GLAST CsI calorimeter,” in _2003 IEEE Nuclear Science Symposium. Conference Record (IEEE Cat. No.03CH37515)_ , vol. 3, 2003, pp. 1704–1707 Vol.3.
  * [25] J. S. Milne Jr and D. S. Kaufman, “General Environmental Verification Specification,” in _21st Aerospace Testing Seminar_ , 2003.
  * [26] R. L. Helmer, “232th - gamma-ray spectrometry catalog.” [Online]. Available: https://gammaray.inl.gov/SiteAssets/catalogs/ge/pdf/th232_new.pdf
  * [27] J. D. Valentine, D. K. Wehe, G. F. Knoll, and C. E. Moss, “Temperature dependence of CsI (Tl) absolute scintillation yield,” _IEEE Transactions on Nuclear Science_ , vol. 40, no. 4, pp. 1267–1274, 1993.
  * [28] F. Quarati, A. Owens, P. Dorenbos, J. de Haas, G. Benzoni, N. Blasi, C. Boiano, S. Brambilla, F. Camera, R. Alba, G. Bellia, C. Maiolino, D. Santonocito, M. Ahmed, N. Brown, S. Stave, H. Weller, and Y. Wu, “High energy gamma-ray spectroscopy with labr3 scintillation detectors,” _Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment_ , vol. 629, no. 1, pp. 157–169, 2011. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0168900210024976
  * [29] E. I. Novikova, M. S. Strickman, C. Gwon, B. F. Phlips, E. A. Wulf, C. Fitzgerald, L. S. Waters, and R. C. Johns, “Designing sword–software for optimization of radiation detectors,” in _2006 IEEE Nuclear Science Symposium Conference Record_ , vol. 1, 2006, pp. 607–612.