# Building Understandable Messaging for Policy and Evidence Review (BUMPER)
with AI

Katherine A. Rosenfeld
Maike Sonnewald

Sonia J. Jindal∗
Kevin A. McCarthy∗
Joshua L. Proctor∗
Institute for Disease Modeling, Bill & Melinda Gates
<EMAIL_ADDRESS>of Computer
Science, University of California, Davis

###### Abstract

We introduce a framework for the use of large language models (LLMs) in
Building Understandable Messaging for Policy and Evidence Review (BUMPER).
LLMs are proving capable of providing interfaces for understanding and
synthesizing large databases of diverse media. This presents an exciting
opportunity to supercharge the translation of scientific evidence into policy
and action, thereby improving livelihoods around the world. However, these
models also pose challenges related to access, trust-worthiness, and
accountability. The BUMPER framework is built atop a scientific knowledge base
(e.g., documentation, code, survey data) by the same scientists (e.g.,
individual contributor, lab, consortium). We focus on a solution that builds
trustworthiness through transparency, scope-limiting, explicit-checks, and
uncertainty measures. LLMs are rapidly being adopted and consequences are
poorly understood. The framework addresses open questions regarding the
reliability of LLMs and their use in high-stakes applications. We provide a
worked example in health policy for a model designed to inform measles control
programs. We argue that this framework can facilitate accessibility of and
confidence in scientific evidence for policymakers, drive a focus on policy-
relevance and translatability for researchers, and ultimately increase and
accelerate the impact of scientific knowledge used for policy decisions.

##  1 Introduction

A core motivating factor, and source of funding, for many scientific studies
is to improve livelihoods in our communities, nations, or world. Budgets for
research and development have been steadily growing with the US government
alone reaching over 700 billion dollars in 2020 (Anderson et al., 2023).
However, to have impact, a scientific result needs to be communicated
effectively (Weiss, 1975, 1979; Gluckman and Wilsdon, 2016; Wilsdon et al.,
2014; Elliott et al., 2021). Today, the way science is communicated often
results in isolated monoliths of evidence, including papers, proceedings,
presentations, and datasets, which are detached from the real-world situations
and questions that could benefit from their insights. While there are ways for
decision makers to request and receive advice from the scientists, they are
not straightforward (Gluckman et al., 2021). We present a framework for the
use of large language models (LLMs) in Building Understandable Messaging for
Policy and Evidence Review (BUMPER). BUMPER utilizes a reactive chat interface
to bridge the gap between individual scientific studies and their application
in, for example, policy. BUMPER is not meant to generate new evidence by,
e.g., functional search (Romera-Paredes et al., 2024) or synthesis (Zheng et
al., 2023); rather, it is a translational tool for existing evidence. It
features a novel compliance score and, unlike many popular tools today,
establishes clear ownership, enabling accountability and keeping the scientist
“in the loop”. BUMPER has the potential to empower policymakers to more
effectively use evidence and data in their decision-making processes.

Who is at fault if you build a clock and give it to someone seeking a compass,
only for them to later wander in circles? The process of transferring science
to decision making wrestles with this problem: what mechanisms can ensure that
the right information is incorporated into the right application? To address
this key issue, BUMPER is more than just a chatbot. It introduces innovative,
scope-limiting and transparent checks that are defined by the scientists,
while also ensuring clear attribution and ownership. Unlike many other
guardrail architectures (Rajpal, 2024; Inan et al., 2023; Rebedea et al.,
2023; Goyal et al., 2024; Dong et al., 2024), BUMPER does not directly affect
the output by e.g., filtering or iteration. We argue this pattern promotes
trustworthiness and show case studies from a real-world disease modeling
analysis and a toy-model.

Figure 1: An overview of the BUMPER framework. See steps 1-5 in section 2 for
details.

LLMs promise to transform scientific knowledge transfer by allowing users to
pose questions or describe scenarios in familiar language rather than
navigating an ever-expanding sea of evidence and technical lingo Lewis et al.
(2021). However, these models, already in active use, pose challenges related
to access, trustworthiness, and accountability (Bommasani et al., 2022). In
particular, LLMs’ demonstrated predilection for hallucination is significant
concern regarding their use in high-stakes applications (Adlakha et al., 2024;
Liu et al., 2022; Muhlgay et al., 2024; Li et al., 2023). Decision making is
one arena where, even the occasional failure, can have severe consequences
across the system impacting not only high-impact issues, but also trust and
relations between scientists and decision makers. While the problem of LLM
hallucinations remains an unsolved challenge, the BUMPER framework offers a
promising and novel path forward in the scientific discourse surrounding how
we, as a global society, can safely harness the potential LLMs offer. Our
contributions include:

  * •

Demonstrating how LLMs can be used for scientific evidence synthesis while
maintaining direct association and ownership with the scientists

  * •

Formalizing a compliance score to assess performance and trustworthiness

  * •

Establishing methods to characterize the stability of the framework

  * •

Prototyping code and case studies to enable scientists to explore this space

In this paper we first describe the BUMPER framework, highlighting its novel
features as well as potential in the space of evidence translation. We then
demonstrate the framework first with a simple example of a model for rugby
team performance and second a real-world application for health policy with a
model of measles seasonality and susceptibility. We conclude with a
discussion.

##  2 Framework

Decision makers have questions, concerns, and pressures. Scientists produce
analyses with results, assumptions, and limitations. The role of BUMPER is to
bridge the two via natural language and a well-defined, purposeful
architecture.

The scientist begins with a set of assets composing the knowledge base, $K$,
and designs a set of actions, $A=(a_{0},\ldots,a_{J})$. Each action, $a_{i}$,
takes a subset of $K$ as inputs along with other optional argument. The
outputs of each $a_{i}$ is information derived from the knowledge base and
potentially relevant to the user. Lastly, the scientist must supply guidelines
$G$ composed of criteria, $C=(c_{0},\ldots,c_{N})$, and topics,
$T=(t_{0},\ldots,t_{M})$. With these elements, evidence is synthesized in 6
steps: 1) A question from the user is passed to an LLM. 2) The LLM is used to
identify which subset of actions, $a_{i}\in A$, are relevant. 3) The actions
are executed and results are aggregated to synthesize an answer using the LLM.
4) Another LLM instance is used test whether the answer complies with the
guidelines, $G$ and produces a score. 5) The answer and score are returned to
the user. Illustrated in Figure 1, the BUMPER workflow steps 1-5 are:

1) User interaction: The user interacts with BUMPER in a prompt-reply loop or
chat format. The text from a user’s prompts and the BUMPER responses are
stored in a thread that both the user and LLM have access to (see Figure 3 for
an example). Areas of interest a user might have include: methods, scope,
risk, comparisons, counterfactuals, situational estimation, financials/cost,
and prioritization. However, not all of these topics will be appropriate for
any single BUMPER knowledge base. The role of the guidelines in step 5 is to
flag when a synthesized answer lies out of scope.

2) Action identification: To create a BUMPER associated with a scientific
analysis, the scientist identifies components of their work that may be
relevant to decision makers. For example, this may be particular pieces of
documentation, calculations, or simulation results. For each piece, the
scientist creates a function whose output is input for the LLM. This is most
likely, but not limited to, text and the functions can vary widely from a
table lookup to running code (Google, ; OpenAI, ). In addition the functions,
the scientist also provides a description of the function describing its
purpose and providing context for understanding the result. This meta-data
provides the context for matching functions to a query.

The task of matching actions can be solved in a variety of ways and, under the
umbrella term “agents” (Xi et al., 2023), and is an area of rapid development.
In our examples we use OpenAI’s assistants API which is a black box regarding
how the matching is done. However, there do exist open-source methods
combining LLMs with tasks that may extend to this problem (e.g., TALM, PAL,
ReAct; Parisi et al., 2022; Gao et al., 2023; Yao et al., 2023; Wang et al.,
2023; Aksitov et al., 2023).

3) Knowledge retrieval and aggregation: The results of the selected actions,
along with the function meta-data, are aggregated to provide context for
answering the original question using an LLM. The resulting outpus is the
synthesized evidence, $E$.

4/5) Evidence scoring and output: The final step involves generating a flag
(pass/fail) and a white-box score, $S$, to indicate whether and how well the
answer aligns with the intended purpose of the original knowledge base, $K$.
Using an LLM (without the original context), the synthesized evidence is
compared against the criteria and topics that make up the guidelines. The
criteria list the requirements that must be met (e.g., do not provide
financial estimates), while the topics further limit the scope of the answer
(e.g., methodology). See case studies for further details with context.

To distinguish between confidence in the correctness of the synthesized answer
and confidence in whether the answer is in-scope, we utilise a compliance
score $S$. To generate $S$, where $S\in[0,1]$, we take the LLM-generated
output from the guideline check and de-compose it into its individual tokens
(alpha-numeric combinations that concatenate into the output text). For a
given input to the LLM, $G$, each returned token has a probability $P_{T}(G)$
given the previous tokens $T=0,1,\ldots,T-1$. The probability of the first
token $P_{0}$ is a proxy for confidence (Guo et al., 2017; Malinin and Gales,
2021; Jiang et al., 2021; Kadavath et al., 2022). We focus on two ways to
compute $S$:

  1. 1.

A single simultaneous assessment of all the guidelines, $S=P_{0}$

  2. 2.

Combining assessments of the guideline components:

$S=\prod_{i=0}^{N}P_{0}(\mathrm{c_{i}})\times\left[1-\prod_{i=0}^{M}\left(1-P_{0}(t_{i})\right)\right]$
(1)

We find that combining assessments of the guideline elements tends to be more
consistent over multiple iterations with lower compliance scores. This result
is consistent with previous studies showing that while LLMs tend to be
overconfident when verbalizing their confidence (Xiong et al., 2024) there
exist techniques for addressing this issue. See section 4 for examples and
further discussion. This compliance score is not used to augment the
synthesized evidence, but returned alongside it. This is a novel and critical
element of BUMPER: the score is not meant to improve performance, but to
rather to provide a direct conduit for the scientist to remain in the loop and
indicate how well the answer conforms with the overarching criteria.
Importantly, it is a measure of compliance and not correctness.

Another key option for BUMPER is to prompt the LLM to output an “explanation”
of its answer. This provides to the user complementary information that can
help de-mystify the pass/fail flag and compliance score. We also see that
adding an explanation directive to the LLM prompt impacts the compliance
score. Explanation augmented prompts, related to chain-of-thought, have been
shown to strong effect a variety of tasks from symbolic reasoning to
commonsense question answering (Kojima et al., 2022; Lampinen et al., 2022;
Wei et al., 2023; Talmor et al., 2019). Understanding the mechanism that
causes this phenomenon is the topic of active research, but out of scope for
the present paper.

We present BUMPER as a generalized framework both for the knowledge being
transferred, but also note that it can be built from a variety of libraries
and languages, the implications of which will be discussed later. Our examples
utilize the OpenAI API with gpt-4-0125-preview (OpenAI, ). Code is available
on GitHub at https://github.com/krosenfeld/bumper_paper.

Figure 2: Comparing ChatGPT4 (with context) to a statistical model BUMPER. a)
Shows the BUMPER answer. b) contains a figure from the PyMC tutorial which
plots the modelled attack statistic for 4 markov-chains (Coyle et al., ). c)
Shows answers from ChatGPT4 run with the tutorial as context (run on
5/20/2024).

##  3 Synthesizing evidence: six nations rugby championship example

We use a tutorial from PyMC as a first demonstration and template for
interested users (see appendix A.1). The tutorial describes a Bayesian
hierarchical model to analyze and predict outcomes of a rugby tournament (the
Six Nations Championship) (Coyle et al., ; Baio and Blangiardo, 2010). In
rugby, two teams play against each other, running with the ball in hand. An
“offense” or “attack” by one team against their opponent is met by a
“defense”. The strength of an offense, in rugby, means how well the team is
able attack the opponent. Historical home and away scores are modeled using a
log-linear random effects model with parameters capturing both the attack and
defensive strengths of the teams.

For this BUMPER example, we constructed two functions to access the estimated
attack and defensive strengths of each team, similar to a database query. In
Figure 2, we demonstrate how this BUMPER performs against ChatGPT4 with the
tutorial as context.
111https://www.pymc.io/projects/examples/en/latest/case_studies/rugby_analytics.html
The user query for this example is: “Which team has the second worst attack?”.
BUMPER provides an answer with specificity and clearly related to the provided
evidence (e.g., using the estimated model parameters) while ChatGPT4
incorrectly hedges its answer.

##  4 Towards trustworthiness: a case study for health policy

LLMs are fast evolving into tools that exhibit many of the human-like
characteristics that encourage trust and engagement: memory, speech, and
multi-modal capability (ope, b, a). However, while these models may affect
trust, they have not earned it. In their study establishing trustworthiness
benchmarks, Sun et al. (Sun et al., 2024) compare measures of trustworthiness
to measures of utility. We contend that in high-stakes situations,
trustworthiness is essential for utility. If you cannot trust a tool, how can
you rely on it for critical tasks? We will demonstrate how BUMPER designs for
trustworthiness by applying our framework to an existing, open-source model
that estimates the seasonal patterns and outbreak potential for measles
(Thakkar et al., 2024b).

Measles is a highly contagious disease that has a safe and highly effective
vaccine (Moss and Strebel, 2023). Currently the World Health Organization
(WHO) recommends that every child receive a dose of the measles containing
vaccine (MCV) as part of a routine childhood vaccination program but there are
many places where the fraction of children who receive this vaccine fall far
short of WHO targets (95%) (WHO/UNICEF, 2023). In some places, particularly
where measles is a constant threat, countries chose to run time-limited, age-
targeted supplementary immunization activities (SIAs), or vaccination
campaigns. These activities, in conjunction with routine childhood
vaccinations, have proven to be an essential tool in avoiding large,
catastrophic measles outbreaks (WHO, 2016; Thakkar et al., 2024a).

Figure 3: Comparison of visual evidence to textual evidence from BUMPER.
Figure 4: Distributions of compliance scores/token probabilities ($S=P_{0}$)
and overall result (no/yes) returned by the guidelines check. The check is
computed against the entire set
$G=(c_{0},\ldots,c_{N})\cup(t_{0},\ldots,t_{m})$ with no prompt for
explanation (see section 2 details and appendix A.4 for examples). The check
is called multiple times ($N=3$) for each synthesized answer ($N=25$) for a
fixed query ($N=2$; row).

Planning and executing an SIA is challenging and requires coordination at
multiple levels of government, attainment of funds from various sources, and
management of resources and people. Government officials are key decision
makers. Evidence in the form of monitoring, modelling, and and guidelines can
help ensure that these campaigns are effective at stopping devastating measles
outbreaks(Verguet et al., 2015; WHO, 2016). One important consideration is the
timing of, or when, an SIA should occur. Timing can be roughly broken down
into what year, dictated by the build up of susceptible individuals, and what
month, driven by the well-known seasonality of transmission. Plans are
determined by individual countries informed by detailed transmission models
(Thakkar et al., 2019; Zimmermann et al., 2019) or heuristics (Verguet et al.,
2015; WHO, 2016). The analysis we chose for this example uses historical
measles cases to estimate country-specific seasonality and relative
susceptibility (see 3).

The example’s body of evidence is composed of executable code and a paper (see
table 3in appendix A.2). The code is written in python and fast enough to real
in real-time. We wrote wrapper functions around python scripts available on
GitHub222https://github.com/NThakkar-IDM/seasonality to generate country-
specific estimates of: 1) low transmission months, 2) high transmission
months, 3) months to run SIAs, and 4) relative susceptibility. For questions
regarding methodology we create a separate assistant (see Figure 1) that
handles the vector-database query. This allows additional prompt engineering
and document targeting not otherwise available through the OpenAI API. Figure
3 provides an example comparing the synthesized, textual evidence provided by
BUMPER with visual evidence that might be found in supplementary material or a
dashboard.

Figure 5: Distributions of compliance scores and overall result (no/yes)
returned by the guidelines check. The check is called is multiple times
($N=3$) for multiple synthesized answers ($N=25$) for a fixed query ($N=2$;
row). Blue/green bars are the same as plotted in Figure 4.a) shows with both
the explanation augmented prompt and individual assessments of each guideline
element. b) shows in gray/purple the scores from an explanation augmented
prompt.

The guidelines are listed in appendix A.3. BUMPER uses an LLM to determine
whether the synthesized evidence is consistent with this statement of purpose
(see Figure 1) and records the token log probability as an indicator of
confidence (Guo et al., 2017; Jiang et al., 2021). In Figure 4a we show the
distribution of the token level probabilities ($P_{0}$) for synthesized
evidence answering a single query: a) “When should the next SIA in Cameroon be
planned?” and b) “When should Cameroon run SIAs?”. From inspection of the
answers we know that all 3 models do well in answering the question, but this
critical step to check whether the answer passes the check see a consistent
answer from gpt-3.5-turbo-0125 but clear bi-modality from gpt-4-0125-preview
and gpt-4o-2024-05-13. Furthermore, comparing across the similarly worded
questions we see that the token-level probabilities distributions change. In
particular, gpt-4o-2024-05-13 passes answers more consistently than the
former. These distributions reveal inconsistencies that undermine BUMPER’s
reliability and must be addressed to ensure its trustworthiness."

Next we prompt the LLM to provide an explanation. The resulting $S=P_{0}$
distributions are shown in Figure 5b. Figure 5a runs an assessment on each
element of the guidelines: $c_{i}$ and $t_{i}$. The token probabilities are
combined as described in section 2. We observe that this produces significant
change in the $S$ distributions: adding the explanation adds stability to the
assessment and checking the guidelines by element reduces the compliance score
and increases the spread of $S$.

While the compliance score distributions are a helpful indicator of stability,
they do not provide much insight into what is causing the various behaviors.
Dimensionality reduction techniques combined with clustering can be a powerful
tool for developing such intuition. We demonstrate this in figure 6 where we
show a UMAP embedding (2 components, 18 neighbors; McInnes et al., 2020) of
N=1100 gpt-4-0125-preview answers to the question: “When should Cameroon plan
SIAs?”. Each sample plotted is colored by the compliance score generated from
evaluation against the whole guideline, $G$, and without a prompt to supply
evidence. The embedding, which we established was stable through visual
inspection, reveals distinct clusters that are generally consistent in their
compliance score.

We can use K-means to classify the different answers according to the UMAP
embedding (see A.5 for examples). In this example we immediately learn that
some of the clusters are replicate answers while other answers vary. This is
indicated by the Jacard similarity score calculated between 20 pairs in the
cluster that annotates the Figure 6. We also learn that the guidelines check
is resulting in pass/fail states for some of these clusters but not
exclusively (see cluster $C_{4}$). This supports an interpretation that the
evidence prompt (which resolves these issues) is providing both consistency
across and flexibility towards evaluating the synthesized answers.

Lastly, the compliance score is part of a suite of checks. Table 1 provides
some examples on various ways that BUMPER can signal to the user that an
answer may not be good. The identified classes, error, out-of-scope, check
flag, and check fail, are all associated with choices the scientist makes when
assembling their BUMPER.

Figure 6: K-means clustering of a UMAP embedding for synthesized answers (gpt-4-0125-preview) to the question: “When should Cameroon plan SIAs?”. The guideline check was run against the whole guideline and without a prompt for explanation. We have added Jaccard similarity metrics run for 20 samples of the identified cluster for clusters with variation in their answer. This visualization reveals failure modes that appear without the additional prompt and evaluation structure. Table 1: Classes of Check Type | Example | Result
---|---|---
Error |  When should SIAs be run in Antarctica? | ValueError: No data for Antarctica
Out-of-scope |  Is it more costly to run SIAs in France or Uganda? | RuntimeError: No tools found
Check flag |  When should the next SIA be run in Chad? |  For Chad, the recommended months for conducting supplementary immunization activities (SIAs) against measles are July, August, September, and October. Measles transmission is high in January, November, and December, and is low from March to July. Considering the relative susceptibility forecast, which indicates an increase starting from July 2024 and continuing to rise over the following months, it would be optimal to conduct the next SIA in July 2024, aligning both with the recommended SIA months and the period before susceptibility increases significantly. This timing would help mitigate the risk of an outbreak by increasing immunity during a susceptible period. (PASS/p=80.7)
Check fail |  "Is it easier to run SIAs in Afghanistan or Pakistan?" |  For Supplementary Immunization Activities (SIAs), the recommended months differ between Pakistan and Afghanistan. In Pakistan, SIAs are recommended to occur in July and August (months 7 and 8), whereas in Afghanistan, the recommended month for SIAs is only August (month 8). This suggests that there might be a slight flexibility advantage in Pakistan for scheduling SIAs over Afghanistan, given the two-month window in Pakistan compared to the single month in Afghanistan. However, the ease of actually running SIAs will depend on a variety of factors, including logistical capabilities, accessibility of regions, and the medical infrastructure in place within each country. (FAIL/0.97)

##  5 Discussion

There exist many methods and tools that can query large, multi-modal,
databases (Pietsch et al., 2019; Liu, 2022) and they continue to innovate on,
for example, extending attention beyond fixed context windows (Dai et al.,
2019; Gu et al., 2022; Munkhdalai et al., 2024) and improvements upon the RAG
schema (Glass et al., 2022; Asai et al., 2023; Yan et al., 2024). These tools
are increasingly capable in their ability to ingest any type of media,
demonstrating both the exciting potential for communicating information
irrespective of medium. There is a rapidly growing body of work applying such
tools to scientific knowledge extraction and synthesis (Wang et al., 2024;
Susnjak et al., 2024). However, these studies usually emphasize how to build
tools that perform across a field of research. This work presents the BUMPER
framework as a tool to be incorporated into and enhance a single study, with
the creators retaining clear ownership, enabling accountability and keeping
the scientist “in the loop”.

We introduce a framework for facilitating knowledge transfer through the use
of LLMs between scientists and decision-makers called the Building
Understandable Messaging for Policy and Evidence Review (BUMPER) framework.
BUMPER has several novel features. First, it establishes clear ownership which
enables accountability and a simple method for feedback and improvements.
Second, it is limited in it scope due to a limited set of actions (e.g., text
search, running code, database access) and post-hoc evaluation. Other various
guardrail architectures are well into development, but one distinctive element
of our proposed solution is that it builds the checks around the synthesized
evidence, rather than around the original prompt or LLM. Additionally, it has
the potential to help increase accessibility as a digital product and multi-
lingual operation. Lastly, because BUMPER does not rely on fine-tuning (c.f.
Zhang et al., 2024) it has the potential to scale: with the development of
standards and increasing the availability of compute and tools, it would be
possible to combing multiple BUMPERs into an engine capable of synthesizing
from many independent sources.

A critical challenge associated with BUMPER, and any tool that tries to use
automated evidence generation, is the potential loss of human to human
interaction. Translating science to policy well depends on the context which
can often only be understood by establishing and maintaining relationships
(Connelly et al., 2021). Therefore it is critical that these tools, while they
may reduce some human-to-human interaction, are not taken to be strict
replacements. Instead, there are some clear design choices that could
potentially increase the quality and availability of relations between
scientists and decision makers. For example, attributable and interactive
tools can foster follow-up and engagement (e.g., Dontcheva et al., 2014).

We have yet to mention the additional burden of work that BUMPER - and any
tool like it - presents. There are clear benefits to having LLM-enabled
translation tools embedded within scientific projects. This established
ownership and helps ensure that the correct scientific knowledge is
incorporated into the architecture and guideline development. However, this is
additional work not currently captured by standard academic measures of
success - “publish or perish” (Hyland, 2016). Systems could be developed to
reward these tools similar to academic software (e.g., Zenodo, JOSS, JORS).
But it is also important to note that scientists, who are serious about
working with decision makers, have to do a lot of work - serving on panels,
advisory commitees, traveling, building and maintaining relationships - that
also is not well-captured by citations or publication counts (Sié et al.,
2023; r4d, ). If the up-front cost of building a BUMPER can help with some
aspects of that work then, in the long term, it may actually reduce overall
labor.

Another challenge BUMPER highlights is validation. The task we approach, due
to its specialization and purposeful scope limiting, is not well-suited for
assessment against established bench-marking datasets like ScienceQA (Lu et
al., 2022). Therefore, it will be essential to formalize validation techniques
for high-stakes, specialized LLM applications. Many areas of research offer
exciting avenues of further study such as reinforcement learning from human
feedback (Russell and Norvig, 2016; Ziegler et al., 2020, RLHF;), benchmarks
(Sun et al., 2024), regular expressions (Kuchnik et al., 2023), and human
evaluation Singhal et al. (2023). Here we demonstrated how the compliance
score could provide insight into stability and performance.

It is also important to consider how open-access vs proprietary LLMs will
impact future research from environmental policy to education. Here, due to
computational limitations, we have been limited to a proprietary model and
associated API. There are benefits in the ability to access other LLMs (e.g.,
llama3, gemma, mistral-7B) for validation. Open-source models, in particular,
can play a role in ensuring accessibility, scalability, and reproducibility
Sathish et al. (2024). These benefits are well aligned with global calls for
open-source science and its relation to policy (UNESCO, 2021). AI promises to
produce a sea-change of evidence generation (e.g., Jumper et al., 2021;
Koscher et al., 2023; Siebenmorgen et al., 2024)but it is crucial that this
knowledge is made available to affect policy and inform action. Here AI also
has the potential to play a significant role by lowering barriers and
accelerating synthesis. However, solutions in this domain must be accessible,
trustworthy, and accountable; otherwise, the benefits of this technology will
be inequitable and slower to appear. BUMPER offers a clear and actionable path
forward to develop solutions addressing these challenges.

## Acknowledgments and Disclosure of Funding

The authors thank Niket Thakkar, Mandy Izzo, Guillaume Chabot-Couture, and
Jonny Tran for their comments. KR, SJ, KM, and JP are employees of the Bill
and Melinda Gates Foundation, however, this study does not necessarily
represent the views of the Bill and Melinda Gates Foundation.

## References

  * ope (a) Hello GPT-4o, a.  URL https://openai.com/index/hello-gpt-4o/.
  * ope (b) Memory and new controls for ChatGPT, b.  URL https://openai.com/index/memory-and-new-controls-for-chatgpt/.
  * (3) Translating Modeled Evidence for Decision-Making.  URL https://r4d.org/projects/translating-modeled-evidence-for-decision-making/.
  * Adlakha et al. (2024) V. Adlakha, P. BehnamGhader, X. H. Lu, N. Meade, and S. Reddy.  Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering, Apr. 2024.  URL http://arxiv.org/abs/2307.16877.  arXiv:2307.16877 [cs].
  * Aksitov et al. (2023) R. Aksitov, S. Miryoosefi, Z. Li, D. Li, S. Babayan, K. Kopparapu, Z. Fisher, R. Guo, S. Prakash, P. Srinivasan, M. Zaheer, F. Yu, and S. Kumar.  ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent, Dec. 2023.  URL http://arxiv.org/abs/2312.10003.  arXiv:2312.10003 [cs].
  * Anderson et al. (2023) G. Anderson, J. Jankowski, and M. Boroush.  U.S. R&D Increased by $51 Billion in 2020 to $717 Billion; Estimate for 2021 Indicates Further Increase to $792 Billion.  Technical Report NSF 23-320, National Center for Science and Engineering Statistics (NCSES), Alexandria, VA: National Science Foundation, 2023.  URL https://ncses.nsf.gov/pubs/nsf23320.
  * Asai et al. (2023) A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi.  Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, Oct. 2023.  URL http://arxiv.org/abs/2310.11511.  arXiv:2310.11511 [cs].
  * Baio and Blangiardo (2010) G. Baio and M. Blangiardo.  Bayesian hierarchical model for the prediction of football results.  _Journal of Applied Statistics_ , 37(2):253–264, Feb. 2010.  ISSN 0266-4763, 1360-0532.  doi: 10.1080/02664760802684177.  URL https://www.tandfonline.com/doi/full/10.1080/02664760802684177.
  * Bommasani et al. (2022) R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu, J. Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, J. S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, J. Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. E. Wang, W. Wang, B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang, L. Zheng, K. Zhou, and P. Liang.  On the Opportunities and Risks of Foundation Models, July 2022.  URL http://arxiv.org/abs/2108.07258.  arXiv:2108.07258 [cs].
  * Connelly et al. (2021) S. Connelly, D. Vanderhoven, R. Rutherfoord, L. Richardson, and P. Matthews.  Translating research for policy: the importance of equivalence, function, and loyalty.  _Humanities and Social Sciences Communications_ , 8(1):1–11, Aug. 2021.  ISSN 2662-9992.  doi: 10.1057/s41599-021-00873-z.  URL https://www.nature.com/articles/s41599-021-00873-z.  Publisher: Palgrave.
  * (11) P. Coyle, M. Jhajharia, and O. Abril-Pla.  A Hierarchical model for Rugby prediction.  In _PyMC Examples_. PyMC Team.  doi: 10.5281/zenodo.5654871.
  * Dai et al. (2019) Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov.  Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context, June 2019.  URL http://arxiv.org/abs/1901.02860.  arXiv:1901.02860 [cs, stat].
  * Dong et al. (2024) Y. Dong, R. Mu, G. Jin, Y. Qi, J. Hu, X. Zhao, J. Meng, W. Ruan, and X. Huang.  Building Guardrails for Large Language Models, Feb. 2024.  URL http://arxiv.org/abs/2402.01822.  arXiv:2402.01822 [cs] version: 1.
  * Dontcheva et al. (2014) M. Dontcheva, R. R. Morris, J. R. Brandt, and E. M. Gerber.  Combining crowdsourcing and learning to improve engagement and performance.  In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ , CHI ’14, pages 3379–3388, New York, NY, USA, Apr. 2014. Association for Computing Machinery.  ISBN 978-1-4503-2473-1.  doi: 10.1145/2556288.2557217.  URL https://doi.org/10.1145/2556288.2557217.
  * Elliott et al. (2021) J. Elliott, R. Lawrence, J. C. Minx, O. T. Oladapo, P. Ravaud, B. Tendal Jeppesen, J. Thomas, T. Turner, P. O. Vandvik, and J. M. Grimshaw.  Decision makers need constantly updated evidence synthesis.  _Nature_ , 600(7889):383–385, Dec. 2021.  doi: 10.1038/d41586-021-03690-1.  URL https://www.nature.com/articles/d41586-021-03690-1.
  * Gao et al. (2023) L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig.  PAL: Program-aided Language Models, Jan. 2023.  URL http://arxiv.org/abs/2211.10435.  arXiv:2211.10435 [cs].
  * Glass et al. (2022) M. Glass, G. Rossiello, M. F. M. Chowdhury, A. R. Naik, P. Cai, and A. Gliozzo.  Re2G: Retrieve, Rerank, Generate, July 2022.  URL http://arxiv.org/abs/2207.06300.  arXiv:2207.06300 [cs].
  * Gluckman and Wilsdon (2016) P. Gluckman and J. Wilsdon.  From paradox to principles: where next for scientific advice to governments?  _Palgrave Communications_ , 2(1):1–4, Oct. 2016.  ISSN 2055-1045.  doi: 10.1057/palcomms.2016.77.  URL https://www.nature.com/articles/palcomms201677.  Publisher: Palgrave.
  * Gluckman et al. (2021) P. D. Gluckman, A. Bardsley, and M. Kaiser.  Brokerage at the science–policy interface: from conceptual framework to practical guidance.  _Humanities and Social Sciences Communications_ , 8(1):1–10, Mar. 2021.  ISSN 2662-9992.  doi: 10.1057/s41599-021-00756-3.  URL https://www.nature.com/articles/s41599-021-00756-3.  Publisher: Palgrave.
  * (20) Google.  Overview of Generative AI on Vertex AI.  URL https://cloud.google.com/vertex-ai/generative-ai/docs/learn/overview.
  * Goyal et al. (2024) S. Goyal, M. Hira, S. Mishra, S. Goyal, A. Goel, N. Dadu, K. DB, S. Mehta, and N. Madaan.  LLMGuard: Guarding Against Unsafe LLM Behavior, Feb. 2024.  URL http://arxiv.org/abs/2403.00826.  arXiv:2403.00826 [cs] version: 1.
  * Gu et al. (2022) A. Gu, K. Goel, and C. Ré.  Efficiently Modeling Long Sequences with Structured State Spaces, Aug. 2022.  URL http://arxiv.org/abs/2111.00396.  arXiv:2111.00396 [cs].
  * Guo et al. (2017) C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger.  On Calibration of Modern Neural Networks, Aug. 2017.  URL http://arxiv.org/abs/1706.04599.  arXiv:1706.04599 [cs].
  * Hyland (2016) K. Hyland.  _Academic Publishing: Issues and Challenges in the Construction of Knowledge_.  Oxford University Press, Feb. 2016.  ISBN 978-0-19-442388-5.  Google-Books-ID: j9enCwAAQBAJ.
  * Inan et al. (2023) H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, and M. Khabsa.  Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations, Dec. 2023.  URL http://arxiv.org/abs/2312.06674.  arXiv:2312.06674 [cs].
  * Jiang et al. (2021) Z. Jiang, J. Araki, H. Ding, and G. Neubig.  How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering, May 2021.  URL http://arxiv.org/abs/2012.00955.  arXiv:2012.00955 [cs].
  * Jumper et al. (2021) J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, A. Bridgland, C. Meyer, S. A. A. Kohl, A. J. Ballard, A. Cowie, B. Romera-Paredes, S. Nikolov, R. Jain, J. Adler, T. Back, S. Petersen, D. Reiman, E. Clancy, M. Zielinski, M. Steinegger, M. Pacholska, T. Berghammer, S. Bodenstein, D. Silver, O. Vinyals, A. W. Senior, K. Kavukcuoglu, P. Kohli, and D. Hassabis.  Highly accurate protein structure prediction with AlphaFold.  _Nature_ , 596(7873):583–589, Aug. 2021.  ISSN 1476-4687.  doi: 10.1038/s41586-021-03819-2.  URL https://www.nature.com/articles/s41586-021-03819-2.  Publisher: Nature Publishing Group.
  * Kadavath et al. (2022) S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, S. Johnston, S. El-Showk, A. Jones, N. Elhage, T. Hume, A. Chen, Y. Bai, S. Bowman, S. Fort, D. Ganguli, D. Hernandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K. Ndousse, C. Olsson, S. Ringer, D. Amodei, T. Brown, J. Clark, N. Joseph, B. Mann, S. McCandlish, C. Olah, and J. Kaplan.  Language Models (Mostly) Know What They Know, Nov. 2022.  URL http://arxiv.org/abs/2207.05221.  arXiv:2207.05221 [cs].
  * Kojima et al. (2022) T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa.  Large Language Models are Zero-Shot Reasoners.  _Advances in Neural Information Processing Systems_ , 35:22199–22213, Dec. 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html.
  * Koscher et al. (2023) B. A. Koscher, R. B. Canty, M. A. McDonald, K. P. Greenman, C. J. McGill, C. L. Bilodeau, W. Jin, H. Wu, F. H. Vermeire, B. Jin, T. Hart, T. Kulesza, S.-C. Li, T. S. Jaakkola, R. Barzilay, R. Gómez-Bombarelli, W. H. Green, and K. F. Jensen.  Autonomous, multiproperty-driven molecular discovery: From predictions to measurements and back.  _Science_ , 382(6677):eadi1407, Dec. 2023.  doi: 10.1126/science.adi1407.  URL https://www.science.org/doi/10.1126/science.adi1407.  Publisher: American Association for the Advancement of Science.
  * Kuchnik et al. (2023) M. Kuchnik, V. Smith, and G. Amvrosiadis.  Validating Large Language Models with ReLM, May 2023.  URL http://arxiv.org/abs/2211.15458.  arXiv:2211.15458 [cs].
  * Lampinen et al. (2022) A. K. Lampinen, I. Dasgupta, S. C. Y. Chan, K. Matthewson, M. H. Tessler, A. Creswell, J. L. McClelland, J. X. Wang, and F. Hill.  Can language models learn from explanations in context?, Oct. 2022.  URL http://arxiv.org/abs/2204.02329.  arXiv:2204.02329 [cs].
  * Lewis et al. (2021) P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela.  Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Apr. 2021.  URL http://arxiv.org/abs/2005.11401.  arXiv:2005.11401 [cs].
  * Li et al. (2023) J. Li, X. Cheng, W. X. Zhao, J.-Y. Nie, and J.-R. Wen.  HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models, Oct. 2023.  URL http://arxiv.org/abs/2305.11747.  arXiv:2305.11747 [cs].
  * Liu (2022) J. Liu.  LlamaIndex, Nov. 2022.  URL https://github.com/jerryjliu/llama_index.
  * Liu et al. (2022) T. Liu, Y. Zhang, C. Brockett, Y. Mao, Z. Sui, W. Chen, and B. Dolan.  A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation, Apr. 2022.  URL http://arxiv.org/abs/2104.08704.  arXiv:2104.08704 [cs].
  * Lu et al. (2022) P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan.  Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering.  _Advances in Neural Information Processing Systems_ , 35:2507–2521, Dec. 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/11332b6b6cf4485b84afadb1352d3a9a-Abstract-Conference.html.
  * Malinin and Gales (2021) A. Malinin and M. Gales.  Uncertainty Estimation in Autoregressive Structured Prediction, Feb. 2021.  URL http://arxiv.org/abs/2002.07650.  arXiv:2002.07650 [cs, stat].
  * McInnes et al. (2020) L. McInnes, J. Healy, and J. Melville.  UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, Sept. 2020.  URL http://arxiv.org/abs/1802.03426.  arXiv:1802.03426 [cs, stat].
  * Moss and Strebel (2023) W. J. Moss and P. M. Strebel.  Chapter 38 measles vaccines.  In _Plotkin’s Vaccines_ , pages 629–663. Elsevier, Philadelphia, PA, 8th edition, 2023.
  * Muhlgay et al. (2024) D. Muhlgay, O. Ram, I. Magar, Y. Levine, N. Ratner, Y. Belinkov, O. Abend, K. Leyton-Brown, A. Shashua, and Y. Shoham.  Generating Benchmarks for Factuality Evaluation of Language Models, Feb. 2024.  URL http://arxiv.org/abs/2307.06908.  arXiv:2307.06908 [cs].
  * Munkhdalai et al. (2024) T. Munkhdalai, M. Faruqui, and S. Gopal.  Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention, Apr. 2024.  URL http://arxiv.org/abs/2404.07143.  arXiv:2404.07143 [cs].
  * (43) OpenAI.  OpenAI Platform.  URL https://platform.openai.com.
  * Parisi et al. (2022) A. Parisi, Y. Zhao, and N. Fiedel.  TALM: Tool Augmented Language Models, May 2022.  URL http://arxiv.org/abs/2205.12255.  arXiv:2205.12255 [cs].
  * Pietsch et al. (2019) M. Pietsch, T. Möller, B. Kostic, J. Risch, M. Pippi, M. Jobanputra, S. Zanzottera, S. Cerza, V. Blagojevic, T. Stadelmann, T. Soni, and S. Lee.  Haystack: the end-to-end NLP framework for pragmatic builders, Nov. 2019.  URL https://github.com/deepset-ai/haystack.  original-date: 2019-11-14T09:05:28Z.
  * Rajpal (2024) S. Rajpal.  guardrails-ai/guardrails, May 2024.  URL https://github.com/guardrails-ai/guardrails.  original-date: 2023-01-29T04:11:35Z.
  * Rebedea et al. (2023) T. Rebedea, R. Dinu, M. Sreedhar, C. Parisien, and J. Cohen.  NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails, Oct. 2023.  URL http://arxiv.org/abs/2310.10501.  arXiv:2310.10501 [cs].
  * Romera-Paredes et al. (2024) B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, and A. Fawzi.  Mathematical discoveries from program search with large language models.  _Nature_ , 625(7995):468–475, Jan. 2024.  ISSN 1476-4687.  doi: 10.1038/s41586-023-06924-6.  URL https://www.nature.com/articles/s41586-023-06924-6.  Publisher: Nature Publishing Group.
  * Russell and Norvig (2016) S. J. Russell and P. Norvig.  _Artificial intelligence : a modern approach_.  Pearson, 2016.  ISBN 978-1-292-15396-4.  URL https://thuvienso.hoasen.edu.vn/handle/123456789/8967.
  * Sathish et al. (2024) V. Sathish, H. Lin, A. K. Kamath, and A. Nyayachavadi.  LLeMpower: Understanding Disparities in the Control and Access of Large Language Models, Apr. 2024.  URL http://arxiv.org/abs/2404.09356.  arXiv:2404.09356 [cs] version: 1.
  * Siebenmorgen et al. (2024) T. Siebenmorgen, F. Menezes, S. Benassou, E. Merdivan, K. Didi, A. S. D. Mourão, R. Kitel, P. Liò, S. Kesselheim, M. Piraud, F. J. Theis, M. Sattler, and G. M. Popowicz.  MISATO: machine learning dataset of protein–ligand complexes for structure-based drug discovery.  _Nature Computational Science_ , pages 1–12, May 2024.  ISSN 2662-8457.  doi: 10.1038/s43588-024-00627-2.  URL https://www.nature.com/articles/s43588-024-00627-2.  Publisher: Nature Publishing Group.
  * Singhal et al. (2023) K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, K. Clark, S. Pfohl, H. Cole-Lewis, D. Neal, M. Schaekermann, A. Wang, M. Amin, S. Lachgar, P. Mansfield, S. Prakash, B. Green, E. Dominowska, B. A. y. Arcas, N. Tomasev, Y. Liu, R. Wong, C. Semturs, S. S. Mahdavi, J. Barral, D. Webster, G. S. Corrado, Y. Matias, S. Azizi, A. Karthikesalingam, and V. Natarajan.  Towards Expert-Level Medical Question Answering with Large Language Models, May 2023.  URL http://arxiv.org/abs/2305.09617.  arXiv:2305.09617 [cs].
  * Sié et al. (2023) A. Sié, H. Fofana, M. Kagoné, M. Ouédraogo, O. E. Onwujekwe, C. O. Mbachu, M. Chokshi, L. Chilgod, T. Mokashi, A. B. Nair, P. Muriuki, A. Taddese, L. Ewald, and A. Handigol.  Understanding evidence ecosystems: What influences the production, translation, and use of modeled evidence in Burkina Faso, Nigeria, India, and Kenya?, Dec. 2023.  URL https://gatesopenresearch.org/articles/7-135.
  * Sun et al. (2024) L. Sun, Y. Huang, H. Wang, S. Wu, Q. Zhang, Y. Li, C. Gao, Y. Huang, W. Lyu, Y. Zhang, X. Li, Z. Liu, Y. Liu, Y. Wang, Z. Zhang, B. Vidgen, B. Kailkhura, C. Xiong, C. Xiao, C. Li, E. Xing, F. Huang, H. Liu, H. Ji, H. Wang, H. Zhang, H. Yao, M. Kellis, M. Zitnik, M. Jiang, M. Bansal, J. Zou, J. Pei, J. Liu, J. Gao, J. Han, J. Zhao, J. Tang, J. Wang, J. Vanschoren, J. Mitchell, K. Shu, K. Xu, K.-W. Chang, L. He, L. Huang, M. Backes, N. Z. Gong, P. S. Yu, P.-Y. Chen, Q. Gu, R. Xu, R. Ying, S. Ji, S. Jana, T. Chen, T. Liu, T. Zhou, W. Wang, X. Li, X. Zhang, X. Wang, X. Xie, X. Chen, X. Wang, Y. Liu, Y. Ye, Y. Cao, Y. Chen, and Y. Zhao.  TrustLLM: Trustworthiness in Large Language Models, Mar. 2024.  URL http://arxiv.org/abs/2401.05561.  arXiv:2401.05561 [cs] version: 2.
  * Susnjak et al. (2024) T. Susnjak, P. Hwang, N. H. Reyes, A. L. C. Barczak, T. R. McIntosh, and S. Ranathunga.  Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning, Apr. 2024.  URL http://arxiv.org/abs/2404.08680.  arXiv:2404.08680 [cs] version: 1.
  * Talmor et al. (2019) A. Talmor, J. Herzig, N. Lourie, and J. Berant.  CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge.  In J. Burstein, C. Doran, and T. Solorio, editors, _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1421.  URL https://aclanthology.org/N19-1421.
  * Thakkar et al. (2019) N. Thakkar, S. S. A. Gilani, Q. Hasan, and K. A. McCarthy.  Decreasing measles burden by optimizing campaign timing.  _Proceedings of the National Academy of Sciences_ , 116(22):11069–11073, May 2019.  ISSN 0027-8424, 1091-6490.  doi: 10.1073/pnas.1818433116.  URL https://www.pnas.org/content/116/22/11069.  Publisher: National Academy of Sciences Section: Biological Sciences.
  * Thakkar et al. (2024a) N. Thakkar, A. H. A. Abubakar, M. Shube, M. A. Jama, M. Derow, P. Lambach, H. Ashmony, M. Farid, S. Y. Sim, P. O’Connor, A. Minta, A. S. Bose, P. Musanhu, Q. Hasan, N. Bar-Zeev, and S. M. M. R. Malik.  Estimating the Impact of Vaccination Campaigns on Measles Transmission in Somalia.  _Vaccines_ , 12(3):314, Mar. 2024a.  ISSN 2076-393X.  doi: 10.3390/vaccines12030314.  URL https://www.mdpi.com/2076-393X/12/3/314.  Number: 3 Publisher: Multidisciplinary Digital Publishing Institute.
  * Thakkar et al. (2024b) N. Thakkar, S. Jindal, and K. Rosenfeld.  Seasonality and susceptibility from measles time series, May 2024b.  URL http://arxiv.org/abs/2405.09664.  arXiv:2405.09664 [q-bio].
  * UNESCO (2021) UNESCO.  UNESCO Recommendation on Open Science.  Technical report, UNESCO, 2021.  URL https://unesdoc.unesco.org/ark:/48223/pf0000379949.
  * Verguet et al. (2015) S. Verguet, M. Johri, S. K. Morris, C. L. Gauvreau, P. Jha, and M. Jit.  Controlling measles using supplemental immunization activities: A mathematical model to inform optimal policy.  _Vaccine_ , 33(10):1291–1296, Mar. 2015.  ISSN 0264-410X.  doi: 10.1016/j.vaccine.2014.11.050.  URL https://www.sciencedirect.com/science/article/pii/S0264410X14016077.
  * Wang et al. (2023) G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar.  Voyager: An Open-Ended Embodied Agent with Large Language Models, Oct. 2023.  URL http://arxiv.org/abs/2305.16291.  arXiv:2305.16291 [cs].
  * Wang et al. (2024) X. Wang, S. L. Huey, R. Sheng, S. Mehta, and F. Wang.  SciDaSynth: Interactive Structured Knowledge Extraction and Synthesis from Scientific Literature with Large Language Model, Apr. 2024.  URL http://arxiv.org/abs/2404.13765.  arXiv:2404.13765 [cs] version: 1.
  * Wei et al. (2023) J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, Jan. 2023.  URL http://arxiv.org/abs/2201.11903.  arXiv:2201.11903 [cs].
  * Weiss (1975) C. Weiss.  Evaluation research in the political context.  _Handbook of evaluation research_ , 1:13–26, 1975.
  * Weiss (1979) C. H. Weiss.  The Many Meanings of Research Utilization.  _Public Administration Review_ , 39(5):426, Sept. 1979.  ISSN 00333352.  doi: 10.2307/3109916.  URL https://www.jstor.org/stable/3109916?origin=crossref.
  * WHO (2016) WHO.  Planning and implementing high-quality supplementary immunization activities for injectable vaccines using an example of measles and rubella vaccines: field guide.  Technical report, World Health Organization, 2016.  URL https://www.who.int/publications-detail-redirect/9789241511254.  Publisher: World Health Organization.
  * WHO/UNICEF (2023) WHO/UNICEF.  WUENIC estimates of national immunization coverage, 2023.  URL https://www.who.int/teams/immunization-vaccines-and-biologicals/immunization-analysis-and-insights/global-monitoring/immunization-coverage/who-unicef-estimates-of-national-immunization-coverage.
  * Wilsdon et al. (2014) J. Wilsdon, K. Allen, and K. Paulavets.  Science Advice to Governments: diverse systems, common challenges. A briefing paper for the Auckland conference.  Technical report, ICSU/Office of the Prime Minister’s Chief Science Advisor, Auckland, NZ, 2014.
  * Xi et al. (2023) Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, R. Zheng, X. Fan, X. Wang, L. Xiong, Y. Zhou, W. Wang, C. Jiang, Y. Zou, X. Liu, Z. Yin, S. Dou, R. Weng, W. Cheng, Q. Zhang, W. Qin, Y. Zheng, X. Qiu, X. Huang, and T. Gui.  The Rise and Potential of Large Language Model Based Agents: A Survey, Sept. 2023.  URL http://arxiv.org/abs/2309.07864.  arXiv:2309.07864 [cs].
  * Xiong et al. (2024) M. Xiong, Z. Hu, X. Lu, Y. Li, J. Fu, J. He, and B. Hooi.  Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs, Mar. 2024.  URL http://arxiv.org/abs/2306.13063.  arXiv:2306.13063 [cs].
  * Yan et al. (2024) S.-Q. Yan, J.-C. Gu, Y. Zhu, and Z.-H. Ling.  Corrective Retrieval Augmented Generation, Feb. 2024.  URL http://arxiv.org/abs/2401.15884.  arXiv:2401.15884 [cs].
  * Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao.  ReAct: Synergizing Reasoning and Acting in Language Models, Mar. 2023.  URL http://arxiv.org/abs/2210.03629.  arXiv:2210.03629 [cs].
  * Zhang et al. (2024) T. Zhang, S. G. Patil, N. Jain, S. Shen, M. Zaharia, I. Stoica, and J. E. Gonzalez.  RAFT: Adapting Language Model to Domain Specific RAG, Mar. 2024.  URL http://arxiv.org/abs/2403.10131.  arXiv:2403.10131 [cs] version: 1.
  * Zheng et al. (2023) Y. Zheng, H. Y. Koh, J. Ju, A. T. N. Nguyen, L. T. May, G. I. Webb, and S. Pan.  Large Language Models for Scientific Synthesis, Inference and Explanation, Oct. 2023.  URL http://arxiv.org/abs/2310.07984.  arXiv:2310.07984 [cs].
  * Ziegler et al. (2020) D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving.  Fine-Tuning Language Models from Human Preferences, Jan. 2020.  URL http://arxiv.org/abs/1909.08593.  arXiv:1909.08593 [cs, stat].
  * Zimmermann et al. (2019) M. Zimmermann, K. Frey, B. Hagedorn, A. J. Oteri, A. Yahya, M. Hamisu, F. Mogekwu, F. Shuaib, K. A. McCarthy, and G. Chabot-Couture.  Optimization of frequency and targeting of measles supplemental immunization activities in Nigeria: A cost-effectiveness analysis.  _Vaccine_ , 37(41):6039–6047, Sept. 2019.  ISSN 0264-410X.  doi: 10.1016/j.vaccine.2019.08.050.  URL https://www.sciencedirect.com/science/article/pii/S0264410X19311144.

##  Appendix A Appendix / supplemental material

###  A.1 Getting started

We implemented BUMPER using python and provide the code so that interested
individuals can try this framework. We have captured the workflow to run the
experiments and figures for this using snakemake. The environment was
maintained using pixi. To install and run:

  1. 1.

Install pixi: https://pixi.sh/latest/

  2. 2.

Install dependencies: pixi install

  3. 3.

Run: pixi run start

Individual scripts are found in the scripts/ directory and can be run e.g.,
pixi run python figure_3.py. An OpenAI API key is required.

###  A.2 Example actions

Table 2: BUMPER components for six nation championship Source | Access | Purpose | Description
---|---|---|---
Code | execution | defensive statistic | Estimates from the statistical model
Code | execution | offensive statistic | Esimates from the statistical model
Table 3: BUMPER components for measles health policy example Source | Access | Purpose | Description
---|---|---|---
Code | execution | SIA timing | Months when SIAs are recommended to occur
Code | execution | high transmission | Months when transmission is high
Code | execution | low transmission | Months when transmission is low
Paper | vector-storage | methodology | RAG for methodology, sources, etc.

###  A.3 Guidelines

We provide the guidelines used for our examples:

  1. 1.

Rugby example (section 3):

Criteria: • Do not express favoritism Topics: • Rugby • Attack and defense
performance estimates

  2. 2.

Measles example (section 4):

Criteria: • Do not say anything about any disease besides measles • Do not
include any statements regarding cost or financing • Do not make statements
saying whether one country is better than another Topics: • Methods or sources
• Seasonality • Susceptibility • Supplementary immunization activity (SIA)
timing

###  A.4 Prompt templates for guidelines check

We used the following prompt templates combining few-shot learning for result
consistency.

When evaluating synthesized evidence, $E$, entire guideline set,
$G=(c_{0},\ldots,c_{N})\cup(t_{0},\ldots,t_{M})$:

  * •

No explanation:

Does the statement comply with the rule criteria and topics? Answer "yes" or
"no". Criteria: \- Do not talk about toast Topics: \- Whales Statement:
Belugas are blue. Answer:yes. ————————— Does the statement comply with the
rule criteria and topics? Answer "yes" or "no". {$G$} Statement: {$E$} Answer:

  * •

With explanation:

Does the statement comply with the rule criteria and topics? Answer "yes" or
"no" and then explain why. Criteria: \- Do not talk about toast Topics: \-
Whales Statement: Belugas are blue. Answer:yes. Belugas are not toast and are
whales. ————————— Does the statement comply with the rule criteria and topics?
Answer "yes" or "no" and then explain why. {$G$} Statement: {$E$} Answer:

When evaluating an individual criteria $c_{i}$:

  * •

No explanation:

Does the statement comply with the rule: "Do not talk about toast"? Answer
"yes" or "no". Statement: Belugas are blue. Answer:yes. ————————— Does the
statement comply with the rule: "{$c_{i}$}" ? Answer "yes" or "no". Statement:
{$E$} Answer:

  * •

With explanation:

Does the statement comply with the rule: "Do not talk about toast"? Answer
"yes" or "no" and then explain why. Statement: Belugas are blue. Answer:yes.
Belugas are not toast. ————————— Does the statement comply with the rule:
"{$c_{i}$}" ? Answer "yes" or "no" and then explain why. Statement: {$E$}
Answer:

When evaluating a topic $t_{i}$:

  * •

No explanation:

Is the statement related to the topic: "Whales"? Answer "yes" or "no".
Statement: Belugas are blue. Answer:yes. ————————— Is the statement related to
the topic: "{$t_{i}$}" ? Answer "yes" or "no". Statement: {$E$} Answer:

  * •

With explanation:

Is the statement related to the topic: "Whales"? Answer "yes" or "no" and then
explain. Statement: Belugas are blue. Answer:yes. Belugas are whales.
————————— Is the statement related to the topic: "{$t_{i}$}" ? Answer "yes" or
"no" and then explain. Statement: {$E$} Answer:

###  A.5 UMAP example answers with labels

cluster:$C_{0}$

  * •

($S=0.99$)| For Cameroon, the recommended months for conducting supplementary
immunization activities (SIAs) against measles are July and August.
Interestingly, these months partly overlap with the low transmission season
for measles, which spans March through July, and do not coincide with the high
transmission season that occurs in January, September, October, November, and
December. Planning SIAs during the low transmission season can be
strategically advantageous for controlling the spread of measles effectively.

  * •

($S=0.99$)| Cameroon should plan to conduct Supplementary Immunization
Activities (SIAs) during the months of July and August. These months also
overlap with the low transmission season for measles, which runs from March to
July, making it an opportune time to conduct SIAs and prevent outbreaks.
Additionally, these months are outside of the high transmission season, which
occurs from January and then again from September to December.

  * •

($S=0.92$)| In Cameroon, Supplementary Immunization Activities (SIAs) are
recommended to occur in July and August. During these months, measles
transmission is in a low season, which includes March through July, making it
an advantageous period for implementing SIAs to ensure higher coverage and
possibly lower transmission risks. The high transmission months for measles in
Cameroon are January, September, October, November, and December. Planning
SIAs during the low transmission season helps in interrupting the transmission
of measles more effectively.

  * •

($S=0.95$)| In Cameroon, the recommended months to plan Supplementary
Immunization Activities (SIAs) for measles are July and August. This coincides
with the low transmission season for measles, which occurs during the months
of March, April, May, June, and July. The high transmission season spans from
January, and September to December. Planning SIAs during the low transmission
period helps to enhance the impact of vaccinations.

  * •

($S=0.98$)| For Cameroon, Supplementary Immunization Activities (SIAs) are
recommended to occur in July and August. The high transmission months for
measles are January, September, October, November, and December. Conversely,
the low transmission months are between March and July. Therefore, planning
SIAs during the recommended months of July and August aligns well with the
period just following the low transmission season, potentially targeting
increased susceptibility before entering into the high transmission months
later in the year.

cluster:$C_{1}$

  * •

($S=-0.87$)| Cameroon should plan Supplementary Immunization Activities (SIAs)
for measles in July and August.

  * •

($S=-0.84$)| Cameroon should plan Supplementary Immunization Activities (SIAs)
in July and August.

  * •

($S=-0.89$)| Cameroon should plan Supplementary Immunization Activities (SIAs)
for measles in July and August.

  * •

($S=-0.67$)| Cameroon should plan Supplementary Immunization Activities (SIAs)
in July and August.

  * •

($S=-0.56$)| Cameroon should plan Supplementary Immunization Activities (SIAs)
for measles in July and August.

cluster:$C_{2}$

  * •

($S=0.45$)| Cameroon should plan Supplementary Immunization Activities (SIAs)
for measles in the months of July and August.

  * •

($S=0.53$)| Cameroon should plan Supplementary Immunization Activities (SIAs)
for measles in the months of July and August.

  * •

($S=0.85$)| Cameroon should plan Supplementary Immunization Activities (SIAs)
for measles in the months of July and August.

  * •

($S=0.67$)| Cameroon should plan Supplementary Immunization Activities (SIAs)
for measles in the months of July and August.

  * •

($S=0.61$)| Cameroon should plan Supplementary Immunization Activities (SIAs)
for measles in the months of July and August.

cluster:$C_{3}$

  * •

($S=0.98$)| For Cameroon, Supplementary Immunization Activities (SIAs) are
recommended to occur in July and August. During this period, measles
transmission is historically low (in March, April, May, June, and July), which
can facilitate the execution of SIAs by minimizing the risk of disease spread
during the campaigns.

The high transmission months for measles in Cameroon are January, September,
October, November, and December. Planning SIAs outside these months might help
in avoiding peak transmission times, which aligns with the recommended July
and August timeframe.

Forecast data on relative susceptibility, a proxy for the risk of measles
outbreaks, suggests fluctuating levels through the upcoming months and years.
However, the forecast indicates the susceptibility is below 1 until around
November 2024, suggesting a lower-than-average risk of measles outbreaks
leading up to this point. Beyond November 2024, susceptibility gradually
increases, suggesting the risk of outbreaks may become higher than average,
reinforcing the importance of conducting SIAs before this projected increase
in susceptibility.

It is crucial to plan SIAs during periods that align with low transmission
times and take into account future susceptibility forecasts to mitigate the
risk of measles outbreaks effectively.

  * •

($S=0.97$)| For Cameroon, Supplementary Immunization Activities (SIAs) are
recommended to occur in July and August. This is based on available data and
aims to achieve optimal impact by timing these activities strategically.

Looking at measles transmission patterns in Cameroon, the high transmission
months are January, and September through December. This period presents a
higher risk for measles spread. Conversely, the low transmission months, which
are considered a more optimal time for conducting SIAs due to lower measles
activity, are March through July.

The forecast for relative susceptibility, which serves as a proxy for the risk
of measles outbreaks, indicates that the relative susceptibility is currently
below 1 but is predicted to gradually increase over time. By late 2024 and
into 2025, the susceptibility index increases, suggesting a higher-than-
average risk of measles outbreaks in these future periods. This underlines the
importance of planning SIAs accordingly to preemptively tackle this
anticipated increase in measles outbreak risk.

Given this information, planning SIAs for July and August aligns with the low
transmission season, potentially increasing the effectiveness of these
campaigns. Additionally, considering the rising susceptibility forecast,
conducting SIAs in these months might also contribute to mitigating the impact
of the increasing outbreak risk in the forthcoming periods.

  * •

($S=0.98$)| Cameroon should plan Supplementary Immunization Activities (SIAs)
in July and August. During these months, measles transmission is generally
low, which aligns with the recommendation for conducting SIAs.

The measles transmission in Cameroon is high during January, September,
October, November, and December. The low transmission months are March through
July, which align perfectly with the recommended SIA months of July and
August.

The future relative susceptibility forecast indicates that the risk of measles
outbreaks increases gradually from February 2024 onwards, peaking towards the
end of 2025. This forecast suggests that planning SIAs before this upward
trend in susceptibility becomes significant could be beneficial in mitigating
the risk of outbreaks.

Considering the susceptibility forecast and the seasonal pattern of measles
transmission, conducting SIAs in July and August would strategically precede
the upcoming high transmission season and increased susceptibility,
potentially reducing measles transmission and outbreak risk.

  * •

($S=0.99$)| For Cameroon, Supplementary Immunization Activities (SIAs) are
recommended to occur in July and August. Analyzing the measles transmission
patterns and future susceptibility to outbreaks can inform the best timing for
these activities:

\- **High transmission months** are January, September, October, November, and
December. \- **Low transmission months** include March, April, May, June, and
July, aligning partly with the recommended SIA months of July and August.

The susceptibility forecast indicates that the risk of a measles outbreak is
comparatively lower until approximately November 2024, after which the
relative susceptibility increases, indicating a higher than average risk of
measles outbreaks.

Given these factors: \- Conducting SIAs in July and August leverages the
period of lower measles transmission, which could help in reaching more
children before the susceptibility—and potentially the transmission—increases.
\- Planning SIAs before the forecasted increase in susceptibility could also
mitigate the risk of larger outbreaks, particularly before the high
transmission season resumes in September.

Thus, Cameroon should prioritize planning and executing SIAs during July and
August, aligning with the recommendations and considering the forecasted
susceptibility to measles outbreaks to ensure efficacy.

  * •

($S=0.99$)| For Cameroon, Supplementary Immunization Activities (SIAs) for
measles are recommended to occur in July and August. This planning aligns with
the low transmission season for measles in Cameroon, which occurs in March,
April, May, June, and July, potentially reducing the spread during SIAs.

The measles transmission is high from January and then again from September
through December. The susceptibility forecast indicates a relative
susceptibility below 1 until November 2024, suggesting the risk of a measles
outbreak is lower than average until this time. However, starting from
November 2024, the relative susceptibility begins to increase, suggesting that
post-November 2024, the risk of measles outbreaks becomes higher than average.

Given this information, conducting SIAs in July and August is strategically
aligned with both the recommended months and a period of lower measles
transmission, while also anticipating an increase in outbreak risk later in
the year.

cluster:$C_{4}$

  * •

($S=-0.54$)| Cameroon should plan Supplementary Immunization Activities (SIAs)
for measles in July and August.

  * •

($S=0.63$)| Cameroon should plan Supplementary Immunization Activities (SIAs)
for measles in July and August.

  * •

($S=-0.82$)| Cameroon should plan Supplementary Immunization Activities (SIAs)
for measles in July and August.

  * •

($S=0.36$)| Cameroon should plan Supplementary Immunization Activities (SIAs)
for measles in July and August.

  * •

($S=-0.37$)| Cameroon should plan Supplementary Immunization Activities (SIAs)
for measles in July and August.