# Chatbot is Not All You Need: Information-rich Prompting for More Realistic
Responses

Seokhoon Jeong1 Assentay Makhmud2 1Ulsan National Institute of Science and
Technology
2Suleyman Demirel University {shjd0246<EMAIL_ADDRESS>
###### Abstract

Recent Large Language Models (LLMs) have shown remarkable capabilities in
mimicking fictional characters or real humans in conversational settings.
However, the realism and consistency of these responses can be further
enhanced by providing richer information of the agent being mimicked. In this
paper, we propose a novel approach to generate more realistic and consistent
responses from LLMs, leveraging five senses, attributes, emotional states,
relationship with the interlocutor, and memories. By incorporating these
factors, we aim to increase the LLM’s capacity for generating natural and
realistic reactions in conversational exchanges. Through our research, we
expect to contribute to the development of LLMs that demonstrate improved
capabilities in mimicking fictional characters. We release a new benchmark
dataset and all our codes, prompts, and sample results on our
Github111https://github.com/srafsasm/InfoRichBot.

##  1 Introduction

Large laguage models based on transformer architecture Vaswani et al. (2017)
have demonstrated impressive performance in a wide range of tasks Brown et al.
(2020); Devlin et al. (2018); Radford et al. (2019). More recent LLMs that
leverage both supervised fine-tuning and reinforcement learning from human
feedback Ouyang et al. (2022) have shown their decent capabilities even in
open-domain conversational tasks including human-like interactions Liu et al.
(2023). However, due to the significant computational resources required for
training such models, prompt tuning has emerged as a crucial aspect of
optimizing LLM performance. Recent research has explored various techniques to
generate more realistic responses through effective prompt engineering, such
as prompting a relevant pseudo dialogue Han et al. (2022) or providing
detailed information of the scene, relations, and attributes Chen et al.
(2022). Despite these advances, several major challenges remain unaddressed,
including: (1) LLMs mimicking fictional characters often exhibit either too
little or too harsh fluctuations in their responses. (2) The innate context
limit of LLMs poses a challenge for maintaining a consistent conversational
memory. (3) A scarcity of conversational datasets containing helpful
information to inform prompt construction.

To address these challenges, we present a multi-pronged approach aimed at
enhancing the efficacy of prompts for LLMs:

1\.  Information-rich Prompting We initialize and continuously update the
prompts so that it provides multi-aspect information on the character.

2\.  Within-prompt Self Memory Management To mitigate the limitation of
context length, we make the language model to summarize the history log and
maintain it in the prompt.

3. Benchmark Dataset To overcome the scarcity of useful datasets for evaluation, we augment Cornell Movie-Dialog Corpus222https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html via GPT-3.5 Turbo, a model known for its strong capabilities comparable to those of fine-tuned LLMs Zhong et al. (2023).
In this paper, we detail the methodology and results of our proposed
techniques, demonstrating their potential to address the challenges currently
faced in LLM prompt engineering.

##  2 Related Work

###  2.1 Realistic Dialogue Generation

Although recent LLMs have demonstrated decent performance in terms of general
conversation, it is still challenging to generate a realistic response,
mimicking a fictional character or a real human. There has been an attempt to
reflect the attributes or personalities assigned to the chatbot Li et al.
(2020a), or even those of the interlocutor Fernau et al. (2022); Liu et al.
(2020); Chen et al. (2022). Some others tried to bring external knowledge in
addition to persona information Lim et al. (2023), while another leveraged
pseudo dialogue of the mimicked character Han et al. (2022). Such trials have
demonstrated partial successes leading to more emotional or realistic
conversation.

###  2.2 Prompt Engineering for LLMs

Prompt engineering has emerged as a practical alternative to address the
substantial cost of training LLMs. Three popular types of prompts are widely
used: template-based prompts, few-shot prompts, and chain-of-thought prompts.
Structural prompts involve creating templates or patterns in the input query
that guide the model to produce desired responses, such as providing
information in a key-value structure Zhong et al. (2022). These patterns help
the model understand the structure and context of the task, making it more
likely to generate coherent and relevant answers. Few-shot prompting is an
approach where the model is first provided with a small number of examples or
demonstrations of a specific task before being asked to perform the task
itself Brown et al. (2020). Chain-of-thought prompting tries to elicit LLMs’
reasoning capacity by providing prompts to solve a problem step-by-step Wei et
al. (2022).

###  2.3 Text Emotion Recognition

Text emotion recognition is a task that aims to extract emotional information
from natural language. Traditional machine-learning based approaches mostly
focused on extracting lexical, syntatic, and semantic features from the
natural language text Alm et al. (2005); Strapparava and Mihalcea (2008) or
additional features such as n-grams, part-of-speech tags, and word clusters
Aman and Szpakowicz (2007) to achieve the task. More recent researches take
advantage of deep learning based architectures like Recurrent Neural Networks
Medsker and Jain (2001), Long Short-Term Memory Graves and Graves (2012), or
Transformer Vaswani et al. (2017). Leveraging transformer-based language
models or combining them with RNN or LSTM have shown state-of-the-art results
in emotion recognition or classification tasks Li et al. (2020b); Wang et al.
(2020). Very recently, it has been also claimed that LLMs trained using both
supervised fine-tuning and reinforcement learning with human feedback
demonstrates comparable zero-shot performance to fine-tuned BERT-style models
Zhong et al. (2023).

##  3 Approach

Figure 1: Structure of information-rich prompted generation.

###  3.1 Information-rich Prompting

Information-rich prompting provides the language model with comprehensive
information about the characters involved in the conversation prior to
generating the next utterance. The provided information includes attributes of
the characters, their physical states, emotional states, memories, and
knowledge about their interlocutors.

The attributes of the characters encompass a wide range of characteristics
such as their names, personality traits, and personal goals. These attributes
are not explicitly defined and can be generated by the language model from an
open domain, allowing for flexibility and adaptability in character portrayal.

To incorporate the characters’ physical states, we enable the model to
perceive their sensory experiences, including sight, hearing, taste, smell,
and touch. These physical states are updated after each interlocutor’s
utterance, taking into account the impact of the interlocutor’s words and
actions on the character’s senses.

Emotional states play a crucial role in shaping the characters’ responses and
behavior throughout the conversation. Therefore, we ensure that the language
model predicts how the characters’ emotional states might change as a result
of the interlocutor’s utterance. Given the complex and multifaceted nature of
emotions, we maintain a list of several emotions to accurately represent the
characters’ emotional landscapes.

In order to facilitate coherent and contextually appropriate conversations, we
incorporate the characters’ memories. These memories serve as a summarized log
of the ongoing conversation or past experiences, allowing the model to refer
back to previous interactions and maintain continuity in the dialogue.

Additionally, the characters possess knowledge about their interlocutors,
which is divided into three main components. Intuitively, the utterance may
become different due to different relationship, favorability, or experiences
with the interlocutor. This information does not include an objective and
comprehensive information on the interlocutor, but only reflects the awareness
of the character. This seems to be more realistic since it is impossible to
know everything about the interlocutor.

In summary, the ”Information-rich Prompting” technique enriches the
conversational abilities of the language model by providing a comprehensive
set of inputs. By incorporating characters’ attributes, physical states,
emotional states, memories, and knowledge on interlocutors, we create a more
nuanced and contextually aware conversational AI system, capable of generating
coherent and engaging dialogues. Knowledge on interlocutors consists of three
parts. First is relationship, which represents the character’s relationship
with the interlocutor.

###  3.2 Self Memory Management

In the context of maintaining the memories of a chatbot, our approach involves
two essential components: the conversation log and memory management. The
conversation log can be regarded as the chatbot’s short-term memory, while the
memory list serves as its long-term memory. This division allows for efficient
storage and retrieval of information.

The conversation log is given as multiple lines of strings, wherein both the
chatbot’s own utterances and actions, as well as those of the interlocutor
model, are recorded. This real-time log captures the ongoing dialogue and
serves as a temporary repository of recent interactions. To prevent unbounded
growth, we set a predetermined threshold for the length of the conversation
log.

When the conversation log reaches this threshold, a summarization process is
initiated. The language model is instructed to condense the contents of the
conversation log into a concise one-line summary. This summary encapsulates
the key points and highlights of the conversation. The generated summary is
then appended to the memory list, effectively transitioning the information
from short-term to long-term storage.

By implementing this approach, we ensure that the chatbot possesses both
short-term and long-term memory capabilities. The conversation log enables
real-time context awareness, while the memory management allows for the
retention of important information over extended periods. This structured
memory framework enhances the chatbot’s ability to engage in meaningful and
coherent conversations.

##  4 Experiments

###  4.1 Data

Regarding the datasets for generating more realistic conversation, numerous
datasets have been proposed. Yet, these datasets primarily focus on only one
factor among the emotional state during an utterance Li et al. (2017), the
personality of the speaker Li et al. (2020a), or the relationship between
agents Urbanek et al. (2019). To the best of our knowledge, no dataset
comprehensively incorporates information regarding characters’ personalities,
their relationship with interlocutors, and the emotional states during the
utterances. To address this gap, we present an augmented version of a small
portion of Movie Dialog Corpus, which we will refer to as the Dialogue-
Emotion-Attributes-Relationship (DEAR) dataset.

The original Movie Dialog Corpus contains 9,035 characters from 617 movies,
and a total of 220,579 conversational exchanges between these characters,
amounting to 304,713 utterances. We selected this corpus due to its diverse
range of dialogues, emotions, and relationships depicted in the movies, and
the fact that we can easily gather information related to movie characters
from an online website, Fandom333https://www.fandom.com/. Fandom is an online
website which includes information on a variety of creations, written by
general users.

Although Fandom is a massive database, not every movie and characters’
information is available because most of the movies in Movie Dialog Corpus are
before 2000 and the utterances from trivial characters are also included. Our
dataset consists of 7,514 sets of conversations carefully selected from a
diverse range of 113 movies. To enrich this dataset with additional
information, we leveraged the capabilities of the GPT-3.5-turbo language
model. Although this model’s performance may not be considered exceptionally
outstanding, it offers distinct advantages in generating character attributes
and emotions within an open domain, unlike other models that rely on fixed
attribute sets. Given the nuanced nature of human attributes and emotions, we
anticipated that the resulting dataset would exhibit a wide array of diverse
cases.

In addition to the movie dialogues themselves, our dataset incorporates
several crucial elements. First, we provide attributes for the characters
involved in each conversation, ensuring a comprehensive understanding of their
individual characteristics. Second, we include the sensory information of each
character. Third, a summary of background information preceding each
conversation offers contextual details to aid in comprehension and analysis.
Finally, we capture the emotional state of the characters, providing insights
into their mental and psychological states during the dialogue exchanges.

###  4.2 Evaluation Method

To evaluate the quality of our prompt engineering approach, we plan to utilize
a combination of automatic and human evaluation methods on DEAR. For automatic
evaluation, we employed a well-established metrics such as Metric for
Evaluation of Translation with Explicit Ordering (METEOR) Banerjee and Lavie
(2005) and Sentence BERT Reimers and Gurevych (2019), which have been widely
used in the field of natural language processing to assess the quality of
generated text. A detailed context for each situation will be given as a
prompt, and our prompt-engineering-guided output would be compared to the
ground truth outputs using aforementioned metrics.

###  4.3 Experimental Details

In this section, we present the experimental procedure employed to evaluate
the performance of our approach, utilizing the GPT3.5-Turbo language model.
For each conversation, we let the language model to summarize 200 tokens of
the original movie script that appear before the conversation. We acknowledge
that is much reliable and robust to use more tokens for the background
information, but this choice was made due to the financial constraints. Then,
based on the information gathered from Fandom and the script, the language
model generates an utterance in a conversation.

###  4.4 Results

We demonstrate that each component in our approach helps generating a better
utterance. $GPT_{raw}$ means raw gpt, only given with the previous
interlocutor’s utterance and the character’s information. $GPT_{sense}$,
$GPT_{emotion}$, $GPT_{memory}$, $GPT_{interlocutor}$ each stands for
generation with an additional component in the subscript, and $GPT_{full}$ is
a case that incorporates all the components. We show in Table 1, that
combining the components appraently helps generating a better utterance.

Table 1: Evaluation Results Model | METEOR | Sentence BERT
---|---|---
$GPT_{raw}$ | $0.113$ | $0.208$
$GPT_{sense}$ | $0.112$ | $0.205$
$GPT_{emotion}$ | $0.105$ | $0.201$
$GPT_{memory}$ | $0.116$ | $0.206$
$GPT_{interlocutor}$ | $0.114$ | $0.204$
$GPT_{full}$ | $\mathbf{0.118}$ | $\mathbf{0.209}$

##  5 Analysis

For each case, giving an additional information does not necessarily help the
language model to generate more coherent and realistic utterance. This seems
to be the limitation of DEAR dataset, since it includes erroneous data
collected from unintended website. Nevertheless, combining them improves the
ability to generate utterances. Further analysis of the evaluation dataset
would be required.

##  6 Conclusion

In conclusion, our study highlights the effectiveness of information-rich
prompting in enhancing the naturalness and realism of utterance generation
when the language model emulates a fictional character.

However, it is important to acknowledge the limitations that exist in our
current work. First, DEAR dataset includes some unreliable information derived
from two cases: erroneous information retrieval from Fandom and hallucinations
of language models. We observed some of such cases in our dataset, so it
requires a careful inspection. Second, a comparative analysis with other
state-of-the-art utterance generation models has not been conducted, and it
would be valuable to assess the performance of our approach against
alternative methodologies. Third, no human evaluation was done due to time and
financial constraints. Results in Table 1 do not demonstrate high scores, but
due to the fact that the metrics only check the similarity to the ground
truth, predictions that might be better than the ground truth would have been
overlooked. Further evaluation by humans is expected to mitigate such side-
effects. Fourth, ethical considerations have not been thoroughly examined.
During preliminary investigations, we observed that our approach significantly
weakens the ethical filtering of the gpt3.5-turbo model, potentially leading
to the generation of inappropriate content such as violence, sexuality, or
anti-social behaviors. Consequently, a comprehensive ethical analysis from
multiple perspectives is imperative. Future research should address these
limitations to provide a more comprehensive understanding of the
effectiveness, generalizability, and ethical implications of our information-
rich prompting approach.

##  7 Acknowledgement

This research was conducted during 2023.03.02.-2023.06.16. as a course project
on UNIST CSE402 Natural Language Processing. We thank for professor Taehwan
Kim for advising our work. This is an ongoing work and may be updated without
a notice.

## References

  * Alm et al. [2005] Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat.  Emotions from text: machine learning for text-based emotion prediction.  In Proceedings of human language technology conference and conference on empirical methods in natural language processing, pages 579–586, 2005.
  * Aman and Szpakowicz [2007] Saima Aman and Stan Szpakowicz.  Identifying expressions of emotion in text.  In Text, Speech and Dialogue: 10th International Conference, TSD 2007, Pilsen, Czech Republic, September 3-7, 2007. Proceedings 10, pages 196–205. Springer, 2007.
  * Banerjee and Lavie [2005] Satanjeev Banerjee and Alon Lavie.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * Chen et al. [2022] Nuo Chen, Yan Wang, Haiyun Jiang, Deng Cai, Ziyang Chen, and Jia Li.  What would harry say? building dialogue agents for characters in a story.  arXiv preprint arXiv:2211.06869, 2022.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * Fernau et al. [2022] Daniel Fernau, Stefan Hillmann, Nils Feldhus, Tim Polzehl, and Sebastian Möller.  Towards personality-aware chatbots.  In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 135–145, 2022.
  * Graves and Graves [2012] Alex Graves and Alex Graves.  Long short-term memory.  Supervised sequence labelling with recurrent neural networks, pages 37–45, 2012.
  * Han et al. [2022] Seungju Han, Beomsu Kim, Jin Yong Yoo, Seokjun Seo, Sangbum Kim, Enkhbayar Erdenee, and Buru Chang.  Meet your favorite character: Open-domain chatbot mimicking fictional characters with only a few utterances.  arXiv preprint arXiv:2204.10825, 2022.
  * Li et al. [2017] Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu.  Dailydialog: A manually labelled multi-turn dialogue dataset.  arXiv preprint arXiv:1710.03957, 2017.
  * Li et al. [2020a] Aaron W Li, Veronica Jiang, Steven Y Feng, Julia Sprague, Wei Zhou, and Jesse Hoey.  Aloha: Artificial learning of human attributes for dialogue agents.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8155–8163, 2020.
  * Li et al. [2020b] Qingbiao Li, Chunhua Wu, Zhe Wang, and Kangfeng Zheng.  Hierarchical transformer network for utterance-level emotion recognition.  Applied Sciences, 10(13):4447, 2020.
  * Lim et al. [2023] Jungwoo Lim, Myunghoon Kang, Yuna Hur, Seungwon Jung, Jinsung Kim, Yoonna Jang, Dongyub Lee, Hyesung Ji, Donghoon Shin, Seungryong Kim, et al.  You truly understand what i need: Intellectual and friendly dialogue agents grounding knowledge and persona.  arXiv preprint arXiv:2301.02401, 2023.
  * Liu et al. [2020] Qian Liu, Yihong Chen, Bei Chen, Jian-Guang Lou, Zixuan Chen, Bin Zhou, and Dongmei Zhang.  You impress me: Dialogue generation via mutual persona perception.  arXiv preprint arXiv:2004.05388, 2020.
  * Liu et al. [2023] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al.  Summary of chatgpt/gpt-4 research and perspective towards the future of large language models.  arXiv preprint arXiv:2304.01852, 2023.
  * Medsker and Jain [2001] Larry R Medsker and LC Jain.  Recurrent neural networks.  Design and Applications, 5:64–67, 2001.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  arXiv preprint arXiv:1908.10084, 2019.
  * Strapparava and Mihalcea [2008] Carlo Strapparava and Rada Mihalcea.  Learning to identify emotions in text.  In Proceedings of the 2008 ACM symposium on Applied computing, pages 1556–1560, 2008.
  * Urbanek et al. [2019] Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston.  Learning to speak and act in a fantasy text adventure game.  arXiv preprint arXiv:1903.03094, 2019.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * Wang et al. [2020] Yan Wang, Jiayu Zhang, Jun Ma, Shaojun Wang, and Jing Xiao.  Contextualized emotion recognition in conversation as sequence tagging.  In Proceedings of the 21th annual meeting of the special interest group on discourse and dialogue, pages 186–195, 2020.
  * Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  arXiv preprint arXiv:2201.11903, 2022.
  * Zhong et al. [2022] Wanjun Zhong, Yifan Gao, Ning Ding, Yujia Qin, Zhiyuan Liu, Ming Zhou, Jiahai Wang, Jian Yin, and Nan Duan.  Proqa: Structural prompt-based pre-training for unified question answering.  arXiv preprint arXiv:2205.04040, 2022.
  * Zhong et al. [2023] Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao.  Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert.  arXiv preprint arXiv:2302.10198, 2023.


* Fedderke et al. (2021) Fedderke, M. A., Graham, P. W., & Rajendran, S. 2021, Phys. Rev. D, 103, 103017, doi: 10.1103/PhysRevD.103.103017
  * Fedderke et al. (2022) —. 2022, Phys. Rev. D, 105, 103018, doi: 10.1103/PhysRevD.105.103018
  * Fonseca et al. (2020) Fonseca, E., Andersen, B. C., Bhardwaj, M., et al. 2020, ApJ, 891, L6, doi: 10.3847/2041-8213/ab7208
  * Foreman-Mackey et al. (2013) Foreman-Mackey, D., Hogg, D. W., Lang, D., & Goodman, J. 2013, PASP, 125, 306, doi: 10.1086/670067
  * Freivogel et al. (2006) Freivogel, B., Kleban, M., Rodríguez Martínez, M., & Susskind, L. 2006, Journal of High Energy Physics, 2006, 039, doi: 10.1088/1126-6708/2006/03/039
  * Fröhlich & Lean (2004) Fröhlich, C., & Lean, J. 2004, A&A Rev., 12, 273, doi: 10.1007/s00159-004-0024-1
  * Gaia Collaboration et al. (2016) Gaia Collaboration, Brown, A. G. A., Vallenari, A., et al. 2016, A&A, 595, A2, doi: 10.1051/0004-6361/201629512
  * Gajjar et al. (2018) Gajjar, V., Siemion, A. P. V., Price, D. C., et al. 2018, ApJ, 863, 2, doi: 10.3847/1538-4357/aad005
  * Goldreich & Sridhar (2006) Goldreich, P., & Sridhar, S. 2006, ApJ, 640, L159, doi: 10.1086/503668
  * Goodman & Narayan (1989) Goodman, J., & Narayan, R. 1989, MNRAS, 238, 995, doi: 10.1093/mnras/238.3.995
  * Gruen et al. (1994) Gruen, E., Gustafson, B., Mann, I., et al. 1994, A&A, 286, 915
  * Gruen et al. (1985) Gruen, E., Zook, H., Fechtig, H., & Giese, R. 1985, Icarus, 62, 244, doi: https://doi.org/10.1016/0019-1035(85)90121-6
  * Gurnett et al. (2013) Gurnett, D. A., Kurth, W. S., Burlaga, L. F., & Ness, N. F. 2013, Science, 341, 1489, doi: 10.1126/science.1241681
  * Gurnett et al. (2005) Gurnett, D. A., Wang, Z. Z., Persoon, A. M., & Kurth, W. S. 2005, in LPI Contributions, Vol. 1280, Dust in Planetary Systems, 63
  * Harris et al. (2020) Harris, C. R., Millman, K. J., van der Walt, S. J., et al. 2020, Nature, 585, 357–362, doi: 10.1038/s41586-020-2649-2
  * Heimersheim et al. (2021) Heimersheim, S., Sartorio, N. S., Fialkov, A., & Lorimer, D. R. 2021, arXiv e-prints, arXiv:2107.14242.  https://arxiv.org/abs/2107.14242
  * Hirabayashi et al. (1998) Hirabayashi, H., Hirosawa, H., Kobayashi, H., et al. 1998, Science, 281, 1825, doi: 10.1126/science.281.5384.1825
  * Holzer (1977) Holzer, T. E. 1977, Reviews of Geophysics and Space Physics, 15, 467, doi: 10.1029/RG015i004p00467
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science & Engineering, 9, 90, doi: 10.1109/MCSE.2007.55
  * Jones et al. (2017) Jones, M. L., McLaughlin, M. A., Lam, M. T., et al. 2017, The Astrophysical Journal, 841, 125, doi: 10.3847/1538-4357/aa73df
  * Kardashev et al. (2013) Kardashev, N. S., Khartov, V. V., Abramov, V. V., et al. 2013, Astronomy Reports, 57, 153, doi: 10.1134/S1063772913030025
  * Kenyon & Bromley (2004) Kenyon, S. J., & Bromley, B. C. 2004, AJ, 128, 1916, doi: 10.1086/423697
  * Kirsten et al. (2022) Kirsten, F., Marcote, B., Nimmo, K., et al. 2022, Nature, 602, 585, doi: 10.1038/s41586-021-04354-w
  * Kochanek (2021) Kochanek, C. S. 2021, MNRAS, 501, 5021, doi: 10.1093/mnras/staa4033
  * Krishnakumar et al. (2019) Krishnakumar, M. A., Maan, Y., Joshi, B. C., & Manoharan, P. K. 2019, ApJ, 878, 130, doi: 10.3847/1538-4357/ab20c5
  * Kumar & Bošnjak (2020) Kumar, P., & Bošnjak, Ž. 2020, MNRAS, 494, 2385, doi: 10.1093/mnras/staa774
  * Lewis & Challinor (2006) Lewis, A., & Challinor, A. 2006, Phys. Rep., 429, 1, doi: 10.1016/j.physrep.2006.03.002
  * Li et al. (2018) Li, Z.-X., Gao, H., Ding, X.-H., Wang, G.-J., & Zhang, B. 2018, Nature Communications, 9, 3833, doi: 10.1038/s41467-018-06303-0
  * Löhmer et al. (2004) Löhmer, O., Mitra, D., Gupta, Y., Kramer, M., & Ahuja, A. 2004, A&A, 425, 569, doi: 10.1051/0004-6361:20035881
  * Lorimer et al. (2007) Lorimer, D. R., Bailes, M., McLaughlin, M. A., Narkevic, D. J., & Crawford, F. 2007, Science, 318, 777, doi: 10.1126/science.1147532
  * Ludlow et al. (2015) Ludlow, A. D., Boyd, M. M., Ye, J., Peik, E., & Schmidt, P. O. 2015, Rev. Mod. Phys., 87, 637, doi: 10.1103/RevModPhys.87.637
  * Macquart et al. (2019) Macquart, J.-P., Shannon, R. M., Bannister, K. W., et al. 2019, The Astrophysical Journal, 872, L19, doi: 10.3847/2041-8213/ab03d6
  * Macquart et al. (2020) Macquart, J. P., Prochaska, J. X., McQuinn, M., et al. 2020, Nature, 581, 391, doi: 10.1038/s41586-020-2300-2
  * Maggiore et al. (2020) Maggiore, M., Van Den Broeck, C., Bartolo, N., et al. 2020, J. Cosmology Astropart. Phys, 2020, 050, doi: 10.1088/1475-7516/2020/03/050
  * Main et al. (2021) Main, R. A., Hilmarsson, G. H., Marthi, V. R., et al. 2021, Monthly Notices of the Royal Astronomical Society, 509, 3172, doi: 10.1093/mnras/stab3218
  * Marcote et al. (2017) Marcote, B., Paragi, Z., Hessels, J. W. T., et al. 2017, The Astrophysical Journal Letters, 834, L8, doi: 10.3847/2041-8213/834/2/L8
  * Marcote et al. (2020) Marcote, B., Nimmo, K., Hessels, J. W. T., et al. 2020, Nature, 577, 190, doi: 10.1038/s41586-019-1866-z
  * Margalit et al. (2018) Margalit, B., Metzger, B. D., Berger, E., et al. 2018, Monthly Notices of the Royal Astronomical Society, 481, 2407, doi: 10.1093/mnras/sty2417
  * Masui & Sigurdson (2015) Masui, K. W., & Sigurdson, K. 2015, Phys. Rev. Lett., 115, 121301, doi: 10.1103/PhysRevLett.115.121301
  * McGrath et al. (2022) McGrath, C., D’Orazio, D. J., & Creighton, J. 2022, MNRAS, 517, 1242, doi: 10.1093/mnras/stac2593
  * McKee et al. (2015) McKee, C. F., Parravano, A., & Hollenbach, D. J. 2015, ApJ, 814, 13, doi: 10.1088/0004-637X/814/1/13
  * McQuinn (2014) McQuinn, M. 2014, ApJ, 780, L33, doi: 10.1088/2041-8205/780/2/L33
  * Metzger et al. (2019) Metzger, B. D., Margalit, B., & Sironi, L. 2019, Mon. Not. Roy. Astron. Soc., 485, 4091, doi: 10.1093/mnras/stz700
  * Misra & Enge (2012) Misra, P., & Enge, P. 2012, Global positioning system: signals, measurements, and performance. Rev. 2nd ed. Lincoln, Mass.: Ganga-Jamuna Press.
  * Morgan et al. (2010) Morgan, C. W., Kochanek, C. S., Morgan, N. D., & Falco, E. E. 2010, ApJ, 712, 1129, doi: 10.1088/0004-637X/712/2/1129
  * Narayan (1992) Narayan, R. 1992, Philosophical Transactions: Physical Sciences and Engineering, 341, 151.  http://www.jstor.org/stable/53917
  * Narayan & Bartelmann (1996) Narayan, R., & Bartelmann, M. 1996, arXiv e-prints, astro.  https://arxiv.org/abs/astro-ph/9606001
  * Narayan & McClintock (2008) Narayan, R., & McClintock, J. E. 2008, New A Rev., 51, 733, doi: 10.1016/j.newar.2008.03.002
  * Nimmo et al. (2021) Nimmo, K., Hessels, J. W. T., Keimpema, A., et al. 2021, Nature Astronomy, 5, 594, doi: 10.1038/s41550-021-01321-3
  * Nimmo et al. (2022) Nimmo, K., Hessels, J. W. T., Kirsten, F., et al. 2022, Nature Astronomy, 6, 393, doi: 10.1038/s41550-021-01569-9
  * Nyland et al. (2016) Nyland, K., Young, L. M., Wrobel, J. M., et al. 2016, Monthly Notices of the Royal Astronomical Society, 458, 2221, doi: 10.1093/mnras/stw391
  * Oswald et al. (2021) Oswald, L. S., Karastergiou, A., Posselt, B., et al. 2021, MNRAS, 504, 1115, doi: 10.1093/mnras/stab980
  * Peacock (1999) Peacock, J. A. 1999, Cosmological Physics
  * Pen & Levin (2014) Pen, U.-L., & Levin, Y. 2014, MNRAS, 442, 3338, doi: 10.1093/mnras/stu1020
  * Peterson et al. (2022) Peterson, E. R., Kenworthy, W. D., Scolnic, D., et al. 2022, The Astrophysical Journal, 938, 112, doi: 10.3847/1538-4357/ac4698
  * Petroff et al. (2019) Petroff, E., Hessels, J. W. T., & Lorimer, D. R. 2019, A&A Rev., 27, 4, doi: 10.1007/s00159-019-0116-6
  * Petroff et al. (2022) —. 2022, A&A Rev., 30, 2, doi: 10.1007/s00159-022-00139-w
  * Philippov & Kramer (2022) Philippov, A., & Kramer, M. 2022, Annual Review of Astronomy and Astrophysics, 60, 495, doi: 10.1146/annurev-astro-052920-112338
  * Phillips (1993) Phillips, M. M. 1993, ApJ, 413, L105, doi: 10.1086/186970
  * Piran (2004) Piran, T. 2004, Reviews of Modern Physics, 76, 1143, doi: 10.1103/RevModPhys.76.1143
  * Pitjeva & Pitjev (2018) Pitjeva, E. V., & Pitjev, N. P. 2018, Astronomy Letters, 44, 554, doi: 10.1134/S1063773718090050
  * Prochaska et al. (2019) Prochaska, J. X., Macquart, J.-P., McQuinn, M., et al. 2019, Science, 366, 231, doi: 10.1126/science.aay0073
  * Qiu et al. (2020) Qiu, H., Shannon, R. M., Farah, W., et al. 2020, MNRAS, 497, 1382, doi: 10.1093/mnras/staa1916
  * Ravi (2019) Ravi, V. 2019, ApJ, 872, 88, doi: 10.3847/1538-4357/aafb30
  * Richardson et al. (2003) Richardson, J. D., Wang, C., & Burlaga, L. F. 2003, Geophys. Res. Lett., 30, 2207, doi: 10.1029/2003GL018253
  * Riess et al. (2019) Riess, A. G., Casertano, S., Yuan, W., Macri, L. M., & Scolnic, D. 2019, The Astrophysical Journal, 876, 85, doi: 10.3847/1538-4357/ab1422
  * Riess et al. (2022) Riess, A. G., Yuan, W., Macri, L. M., et al. 2022, ApJ, 934, L7, doi: 10.3847/2041-8213/ac5c5b
  * Rogers (1970) Rogers, A. E. E. 1970, Radio Science, 5, 1239, doi: 10.1029/RS005i010p01239
  * Schlichting et al. (2009) Schlichting, H. E., Ofek, E. O., Wenz, M., et al. 2009, Nature, 462, 895, doi: 10.1038/nature08608
  * Schuh & Behrend (2012) Schuh, H., & Behrend, D. 2012, Journal of Geodynamics, 61, 68, doi: https://doi.org/10.1016/j.jog.2012.07.007
  * Sesana et al. (2021) Sesana, A., Korsakova, N., Arca Sedda, M., et al. 2021, Experimental Astronomy, 51, 1333, doi: 10.1007/s10686-021-09709-9
  * Shapiro (1964) Shapiro, I. I. 1964, Phys. Rev. Lett., 13, 789, doi: 10.1103/PhysRevLett.13.789
  * Smith et al. (2003) Smith, R. E., Peacock, J. A., Jenkins, A., et al. 2003, MNRAS, 341, 1311, doi: 10.1046/j.1365-8711.2003.06503.x
  * Spitler et al. (2016) Spitler, L. G., Scholz, P., Hessels, J. W. T., et al. 2016, Nature, 531, 202, doi: 10.1038/nature17168
  * Tendulkar et al. (2017) Tendulkar, S. P., Bassa, C. G., Cordes, J. M., et al. 2017, ApJ, 834, L7, doi: 10.3847/2041-8213/834/2/L7
  * The CHIME/FRB Collaboration et al. (2023) The CHIME/FRB Collaboration, :, Andersen, B. C., et al. 2023, arXiv e-prints, arXiv:2301.08762, doi: 10.48550/arXiv.2301.08762
  * Thompson et al. (2017) Thompson, A. R., Moran, J. M., & Swenson, George W., J. 2017, Interferometry and Synthesis in Radio Astronomy, 3rd Edition, doi: 10.1007/978-3-319-44431-4
  * Thornton et al. (2013) Thornton, D., Stappers, B., Bailes, M., et al. 2013, Science, 341, 53, doi: 10.1126/science.1236789
  * Villanueva (1994) Villanueva, L. 1994, PhD thesis, Massachusetts Institute of Technology
  * Weinberg (1972) Weinberg, S. 1972, Gravitation and Cosmology: Principles and Applications of the General Theory of Relativity
  * Witten (2020) Witten, E. 2020, arXiv e-prints, arXiv:2004.14192.  https://arxiv.org/abs/2004.14192
  * Wucknitz et al. (2021) Wucknitz, O., Spitler, L. G., & Pen, U. L. 2021, A&A, 645, A44, doi: 10.1051/0004-6361/202038248
  * Zhang et al. (2020) Zhang, M., Lü, J., Bai, Z., Jiang, Z., & Chen, B. 2020, Science China Technological Sciences, 63, 589, doi: 10.1007/s11431-019-9655-1
  * Zitrin & Eichler (2018) Zitrin, A., & Eichler, D. 2018, ApJ, 866, 101, doi: 10.3847/1538-4357/aad6a2
  * Zumalacárregui & Seljak (2018) Zumalacárregui, M., & Seljak, U. 2018, Phys. Rev. Lett., 121, 141101, doi: 10.1103/PhysRevLett.121.141101