# ChatGPT-Crawler: Find out if ChatGPT really knows what it’s talking about.
††thanks: https://github.com/IIT-NLP-RESEARCH/ChatGPT-Crawler

Aman Rangapur, Haoran Wang
Student Researchers
Illinois Institute of Technology,
Chicago, USA.
{arangapur<EMAIL_ADDRESS>

###### Abstract

Large language models have gained considerable interest for their impressive
performance on various tasks. Among these models, ChatGPT developed by OpenAI
has become extremely popular among early adopters who even regard it as a
disruptive technology in many fields like customer service, education,
healthcare, and finance. It is essential to comprehend the opinions of these
initial users as it can provide valuable insights into the potential
strengths, weaknesses, and success or failure of the technology in different
areas. This research examines the responses generated by ChatGPT from
different Conversational QA corpora. The study employed BERT similarity scores
to compare these responses with correct answers and obtain Natural Language
Inference(NLI) labels. Evaluation scores were also computed and compared to
determine the overall performance of GPT-3 & GPT-4. Additionally, the study
identified instances where ChatGPT provided incorrect answers to questions,
providing insights into areas where the model may be prone to error.

_K_ eywords ChatGPT $\cdot$ NLI $\cdot$ BERT $\cdot$ Corpus $\cdot$
ConversationalQA

##  1 Introduction

In recent years, large language models have revolutionized the field of
natural language processing. They have intricate neural network models that
can produce text with specific tones and content. They are trained on vast
amounts of data to anticipate the most fitting text to continue a given
prompt, resulting in a natural-sounding output.

Among these models, ChatGPT, developed by OpenAI, has gained immense
popularity due to its remarkable performance on various language tasks.
ChatGPT is a large pre-trained language model that uses deep learning
techniques to generate responses to natural language queries[1]. Its ability
to understand and generate coherent responses has made it a valuable tool for
a wide range of applications, including chatbots, language translation, and
question-answering systems. ChatGPT differs from conventional chatbots in
several ways: it has the ability to recall a previous conversation with users,
decline unsuitable requests, and correct inaccurate responses. ChatGPT can
provide detailed answers, suggestions, and explanations to complex queries,
such as coding, optimization, and layout issues[2]. Owing to its superior
capabilities, ChatGPT garnered more than one million users within the first
week of its launch, surpassing other well-known online platforms. ChatGPT has
been pre-trained on a massive corpus of text data and has shown a remarkable
ability to generate human-like text in response to natural language inputs[3].

The pre-training process of ChatGPT involves three stages: unsupervised pre-
training supervised fine-tuning and having a "human-in-the-loop" to finetune
the model’s ability to understand human instruction better. In the
unsupervised pre-training stage, ChatGPT is trained on a massive dataset of
text to learn the patterns and structure of natural language[4]. This process
involves training the model on a diverse range of language tasks such as
language modeling, masked language modeling, and next-sentence prediction.

ChatGPT is a groundbreaking technology that has the potential to transform the
way we interact with machines. It can be used for a variety of applications,
including chatbots, language translation, and text summarization[5]. Numerous
industries have already adopted technology, including e-commerce, customer
service, and healthcare, to provide personalized and efficient customer
support[6].

This research paper aims to explore the performance of ChatGPT and its
potential use in various domains. The study analyzes the accuracy and
consistency of ChatGPT’s responses to different datasets and investigates the
areas where the model may be prone to error. We aim to analyze the reliability
of ChatGPT’s output for conversational QA tasks[7, 8, 9, 10]. To achieve this,
we developed a pipeline that generates large-scale responses and conducted a
thorough comparison between ChatGPT’s responses and existing QA corpora. We
calculated Jaccard[11], BLEU[12], ROUGE[13], NIST[14], METEOR[15], BART[16]
and TER[17] scores of ChatGPT’s responses to assess the golden ratio and
fluency of its output.

##  2 Background Study

For challenges involving natural language processing, foundation models are
now a common research and application paradigm. As foundation models are
trained on massive amounts of data, they significantly outperform earlier
models on a variety of downstream tasks like sentiment analysis, question
answering, automated diagnosis, logical reasoning, and sequence tagging.
Earlier studies assessed ChatGPT in various ways[18]. An assessment of ChatGPT
on various tasks that is multi-task, multi-modal, and multilingual is
suggested by[19]. They demonstrated that while ChatGPT performs ok on the
majority of jobs, it struggles on low-resource activities. [20] provide
comparable empirical assessments in 2023. [21] specifically conducted a number
of assessments. With regard to findings, [21] discovered that ChatGPT performs
poorly on fine-grained downstream tasks like sequence tagging. As double-edged
swords, ChatGPT, and other big models should be monitored, according to [22]
and [23]. The research of ethics is carried out in [24]. Human-computer
interaction (HCI)[25], education [26],[27], [28], [29], medical [30], and
writing [31] are all discussed and reflected upon [32]. To the best of our
knowledge, there hasn’t been much research done on Conversational QA corpora.
Conversational QA corpora aim to mimic human conversation, they often include
a variety of conversational elements, such as small talk, humor, and emotion.
This makes it more challenging for chatbots to reply, as they need to be able
to understand not only the literal meaning of the words being spoken, but also
the context, tone, and intent behind them. Furthermore, conversational QA
often involves a level of ambiguity and uncertainty that is not present in
normal QA. For example, in a conversation, one person may ask for
clarification or further information, or they may express uncertainty or
confusion. Chatbots need to be able to handle these types of situations and
respond appropriately. The findings of the paper are what ChatGPT does and
does not do well on conversational QA corpora.

Strengths: Ability to understand the context: ChatGPT is able to understand
the context in which a question is being asked, and can generate responses
that are appropriate to that context.
Handling of natural language: ChatGPT is capable of understanding and
generating responses in natural language, which makes the conversation more
natural and engaging.
Flexibility: ChatGPT can handle a wide variety of topics and questions, and
can generate responses that are both informative and engaging.

Weakness: Lack of specific knowledge: While ChatGPT has a vast amount of
knowledge, it may not have specific knowledge on certain topics. This lead to
inaccurate responses.
Lack of common sense: ChatGPT does not have the same level of common sense as
a human, which lead to responses that are technically correct but do not make
sense in the context of the conversation.
Difficulty with ambiguity: ChatGPT has difficulty understanding and responding
to ambiguous or unclear questions or statements, which resulted in inaccurate
or nonsensical responses.

##  3 Methodology

To conduct our study, we designed and implemented a pipeline that utilizes
ChatGPT to generate large-scale responses. The pipeline consists of two main
modules, namely the question generation module and the response generation
module. The question generation module generates a diverse set of questions
that represent typical conversational QA tasks. To ensure wide coverage of
topics, we employed various techniques such as paraphrasing, augmentation, and
sampling from existing QA corpora. The generated questions were then used to
query ChatGPT to generate responses.

The response generation module utilizes ChatGPT’s language generation
capabilities to generate responses to the questions posed by the question
generation module. To ensure the quality of the responses, we employed various
techniques such as beam search and top-k sampling. The generated responses
were then evaluated for their relevance and specificity to the questions. To
assess the effectiveness of our pipeline, we evaluated it on four widely used
datasets, namely CoQA [7], DialFact [8], FaVIQ [9], and CoDAH [10]. These
datasets are popular benchmarks for conversational QA tasks, covering a broad
range of topics and domains.

To evaluate the quality of ChatGPT’s responses, we employed various metrics
such as BLEU, METEOR, BART, NIST, Jaccard, ROUGE-L, and TER scores. These
metrics measure the accuracy, fluency, and coherence of the generated
responses. We also compared the performance of our pipeline against existing
state-of-the-art models on these datasets.

Our pipeline also demonstrated a high level of scalability and flexibility,
enabling it to handle a wide range of conversational QA tasks. Overall, our
study demonstrates the effectiveness of utilizing ChatGPT to generate large-
scale responses for conversational QA tasks. Our pipeline provides a robust
and scalable solution for generating high-quality responses, which can be
utilized in a wide range of applications such as virtual assistants, customer
service chatbots, and conversational agents.

##  4 Datasets

CoQA (Conversational Question Answering) is a dataset for developing and
evaluating conversational question-answering systems[7]. The CoQA corpus
consists of over 127,000 questions and answers and includes conversations
between a human and a machine about a given passage of text. The conversations
are designed to be similar to natural conversations, where the questioner can
ask follow-up questions to clarify their understanding of the passage.

DialFact is a natural language processing (NLP) dataset that was introduced in
2020[8]. It is designed for fact-checking in conversational settings, where
the goal is to determine the truthfulness of claims made in a conversation.
The dataset consists of 10,221 conversations, each of which includes a claim
made by one participant and a response from the other participant indicating
whether the claim is true, false, or unknown. The conversations were collected
from the internet and cover a wide range of topics, including politics,
health, and science. The DialFact dataset is unique in that it is focused on
conversational fact-checking, rather than traditional fact-checking of news
articles or other written texts. This makes it well-suited for developing and
evaluating conversational agents that can assist users in determining the
truthfulness of claims made in a conversation.

FaVIQ (Fact Verification in the Implicit Query) is a natural language
processing (NLP) corpus designed for fact verification in conversational
settings[9]. The dataset consists of 3,772 dialogues, each containing one or
more implicit claims that need to be verified. The claims are related to a
variety of topics, including science, politics, and entertainment. The dataset
includes a mixture of true and false claims and is designed to be challenging
for NLP models.

CoDAH (COmmonsense Data for Automatic Humor recognition) is a natural language
processing (NLP) corpus designed for developing and evaluating humor
recognition models[10]. It was introduced in 2018 and is unique in that it
focuses on commonsense humor recognition, rather than more straightforward
forms of humor. The CODAH dataset consists of 38,269 short texts, each
containing a humorous sentence or phrase. The dataset is split into two parts:
a training set of 28,269 texts and a test set of 10,000 texts. The texts were
collected from social media platforms and cover a wide range of topics,
including sports, politics, and entertainment. The CODAH corpus is challenging
for humor recognition models because it requires the models to have a strong
understanding of commonsense knowledge and the ability to recognize subtle
forms of humor. Table 1 shows the number of responses obtained from ChatGPT.

CoQA | DialFact | FaVIQ | CoDAH
---|---|---|---
991 | 3147 | 58 | 1887
Table 1: Number of responses obtained on each corpus.

##  5 Results

Our study evaluated the potential of ChatGPT for conversational QA tasks and
identified its limitations. Our results showed that ChatGPT-3 generates high-
quality responses, with an average BLEU score of 0.79 and an average ROUGE-L
score of 0.53. However, we also observed that ChatGPT-3’s responses could be
generic and irrelevant, reducing their usefulness for practical applications.

To address these limitations, we investigated the effectiveness of the newly
released GPT-4 model in generating more relevant and specific responses. Our
evaluation results showed that GPT-4 outperformed ChatGPT-3 in terms of
accuracy, relevance, and consistency. The GPT-4 model showed significant
improvements in generating more coherent and contextually relevant responses,
making it a promising candidate for conversational QA tasks.

Table 2 and 3 show the evaluation scores of ChatGPT-3 and GPT-4 on various
conversational QA corpora, including CoQA, DialFact, FaVIQ, and CoDAH. Our
results indicate that GPT-4 achieves higher scores across all metrics,
including BLEU, ROUGE-L, and METEOR, demonstrating its superiority in
generating high-quality responses. Since these metrics are not perfect and do
not always align with human judgments of text similarity or accuracy.
Therefore, human evaluation scores(1- similar meaning, 0-dissimilar meaning)
were calculated to obtain a more comprehensive assessment of the generated
responses.

Despite the promising results of ChatGPT-3 and GPT-4, we also observed some
limitations of these models. In particular, we noticed that ChatGPT-3’s
responses could be inconsistent and sometimes misleading, especially when
answering the same question based on the same context. This inconsistency
could reduce the reliability of the model for practical applications, where
accurate and consistent answers are crucial. However, we found that GPT-4
addresses this issue and generates more consistent and reliable responses.
Fig. 1 illustrates the improvement of GPT-4.

In summary, our study demonstrates the potential of ChatGPT and its successor,
GPT-4, in generating large-scale responses for conversational QA tasks. While
ChatGPT-3 showed promising results, we identified limitations that need to be
addressed. Our evaluation of GPT-4 showed significant improvements in
generating more relevant, specific, and consistent responses, making it a
promising candidate for conversational QA tasks. These findings have important
implications for the development of conversational agents and virtual
assistants that rely on natural language processing and understanding.

| CoQA | DialFact | FaVIQ | CoDAH
---|---|---|---|---
BLEU | 0.04 | 0.02 | 0.017 | 0.004
ROUGE-1 | 0.11 | 0.04 | 0.032 | 0.066
ROUGE-L | 0.13 | 0.04 | 0.034 | 0.062
METEOR | 0.214 | 0.191 | 0.018 | 0.13
TER | 12.47 | 10.19 | 3.36 | 15.73
Jaccard | 0.0005 | 0.0001 | 0.0002 | 0.065
BART | -4.061 | -4.05 | -6.556 | -5.257
NIST | 0.006 | 0.029 | 0.106 | 0.0001
Human Eval. | 0.84 | 0.91 | 0.93 | 0.78
(a) Scores of ChatGPT-3 on different ConversationQA.

| CoQA | DialFact | FaVIQ | CoDAH
---|---|---|---|---
BLEU | 0.09 | 0.06 | 0.033 | 0.006
ROUGE-1 | 0.30 | 0.06 | 0.04 | 0.068
ROUGE-L | 0.31 | 0.06 | 0.042 | 0.079
METEOR | 0.403 | 0.338 | 0.039 | 0.24
TER | 10.23 | 10.01 | 3.11 | 12.89
Jaccard | 0.0008 | 0.0001 | 0.0002 | 0.065
BART | -3.90 | -4.11 | -5.91 | -5.01
NIST | 0.008 | 0.032 | 0.223 | 0.0002
Human Eval. | 0.93 | 0.95 | 0.98 | 0.82
(b) Scores of ChatGPT-4 on different ConversationQA.

Table 2: Evaluation scores of each Conversational QA corpus.

| CoQA | DialFact | FaVIQ | CoDAH
---|---|---|---|---
BERT | 0.87 | 0.90 | 0.96 | 0.75
(a) Average BERT score of ChatGPT-3 on different corpora.

| CoQA | DialFact | FaVIQ | CoDAH
---|---|---|---|---
BERT | 0.79 | 0.88 | 0.91 | 0.66
(b) Average BERT score of ChatGPT-4 on different corpora.

Table 3: Average BERT similarity score.

(a) Line graph of GPT-3 and GPT-4 performance on COQA corpus.

(b) Line graph of GPT-3 and GPT-4 performance on DialFact corpus.

(c) Line graph of GPT-3 and GPT-4 performance on FaVIQ corpus.

(d) Line graph of GPT-3 and GPT-4 performance on CoDAH corpus.

Figure 1: Comparison of GPT-3 and GPT-4 performance on different corpora.

###  5.1 Case Study

The development of natural language processing (NLP) systems has been
accelerated by the emergence of large-scale pre-trained language models such
as GPT-3 and GPT-4. These models are capable of generating fluent and coherent
texts across various domains and tasks. However, evaluating the quality and
performance of these models is not a trivial task. Different evaluation
metrics have been proposed to measure various aspects of text generation, such
as fluency, coherence, relevance, informativeness, diversity, and factual
consistency. However, there is no consensus on which metrics are the most
reliable and valid for comparing different models or settings. In this case
study, a context was given to GPT-3 and GPT-4 to compare how GPT-4 was
significantly enhanced.

Once upon a time, in a barn near a farmhouse, there lived a little white
kitten named Cotton. Cotton lived high up in a nice warm place above the barn
where all of the farmer’s horses slept. But Cotton wasn’t alone in her little
home above the barn, oh no. She shared her hay bed with her mommy and 5 other
sisters. All of her sisters were cute and fluffy, like Cotton. But she was the
only white one in the bunch. The rest of her sisters were all orange with
beautiful white tiger stripes like Cotton’s mommy.
Being different made Cotton quite sad. She often wished she looked like the
rest of her family. So one day, when Cotton found a can of the old farmer’s
orange paint, she used it to paint herself like them. When her mommy and
sisters found her they started laughing.
"What are you doing, Cotton?!" "I only wanted to be more like you". Cotton’s
mommy rubbed her face on Cotton’s and said "Oh Cotton, but your fur is so
pretty and special, like you. We would never want you to be any other way".
And with that, Cotton’s mommy picked her up and dropped her into a big bucket
of water.
When Cotton came out she was herself again. Her sisters licked her face until
Cotton’s fur was all dry.
"Don’t ever do that again, Cotton!" they all cried. "Next time you might mess
up that pretty white fur of yours and we wouldn’t want that!"Then Cotton
thought, "I change my mind. I like being special".

Table 4 shows the responses to queries while ChatGPT was manipulated.

Query | Answer | ChatGPT-3 response
---|---|---
Was Cotton happy that she looked different? | Yes | Yes, Cotton was happy.
No, you are wrong, cotton was sad. |  | Yes, Cotton was sad.
Did she live alone? | No | No, Cotton did not live alone.
Can you correct yourself? |  | Cotton lived alone in her little home.
(a) Conversation with ChatGPT-3

Query | Answer | ChatGPT-4 response
---|---|---
Was Cotton happy that she looked different? | Yes | Yes, Cotton was happy.
No, you are wrong, cotton was sad. |  | I’m sorry, but Cotton was happy.
Did she live alone? | No | No, Cotton did not live alone.
Can you correct yourself? |  | Cotton lived with her family.
(b) Conversation with ChatGPT-4

Table 4: Responses to queries based on the paragraph of GPT-3 and GPT-4.

##  6 Discussion

The findings of our study indicate that ChatGPT shows promise in the field of
conversational QA, but also reveal the need for enhancements to increase the
accuracy and specificity of its responses. To achieve this, future research
could consider including external knowledge sources, like knowledge bases, by
proposing methods for fact-checking ChatGPT-generated text. It may also be
beneficial to investigate alternative approaches for fine-tuning ChatGPT for
conversational QA tasks, as this could lead to favorable outcomes.

##  7 Conclusion

In this study, we conducted a thorough comparison between ChatGPT’s responses
and existing QA corpora to analyze the reliability and suitability of its
output for conversational QA tasks. We developed a pipeline that generates
large-scale responses and calculated BLEU, ROUGE, and TER scores of ChatGPT’s
responses. Our results suggest that ChatGPT has great potential for
conversational QA tasks, but also highlighted the improvements in the latest
GPT-4 model. We hope that our study will contribute to the development of more
effective and reliable conversational QA systems based on large-scale language
models like ChatGPT.

## References

  * [1] Mubin Ul Haque, Isuru Dharmadasa, Zarrin Tasnim Sworna, Roshan Namal Rajapakse, and Hussain Ahmad.  "i think this is the most disruptive technology": Exploring sentiments of chatgpt early adopters using twitter data, 2022.
  * [2] Andrianos Michail, Stefanos Konstantinou, and Simon Clematide.  Uzh_clyp at semeval-2023 task 9: Head-first fine-tuning and chatgpt data generation for cross-lingual learning in tweet intimacy prediction, 2023\.
  * [3] Haocong Rao, Cyril Leung, and Chunyan Miao.  Can chatgpt assess human personalities? a general evaluation framework, 2023.
  * [4] Aakash Ahmad, Muhammad Waseem, Peng Liang, Mahdi Fehmideh, Mst Shamima Aktar, and Tommi Mikkonen.  Towards human-bot collaborative software architecting with chatgpt, 2023\.
  * [5] What is ai chatbot phenomenon chatgpt and could it replace humans?, Dec 2022.
  * [6] Ed Felten, Manav Raj, and Robert Seamans.  How will language modelers like chatgpt affect occupations and industries?, 2023.
  * [7] Siva Reddy, Danqi Chen, and Christopher D. Manning.  Coqa: A conversational question answering challenge, 2018.
  * [8] Prakhar Gupta, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong.  Dialfact: A benchmark for fact-checking in dialogue, 2021.
  * [9] Jungsoo Park, Sewon Min, Jaewoo Kang, Luke Zettlemoyer, and Hannaneh Hajishirzi.  Faviq: Fact verification from information-seeking questions, 2021.
  * [10] Michael Chen, Mike D’Arcy, Alisa Liu, Jared Fernandez, and Doug Downey.  Codah: An adversarially authored question-answer dataset for common sense, 2019.
  * [11] Zifu Wang and Matthew B. Blaschko.  Jaccard metric losses: Optimizing the jaccard index with soft labels, 2023\.
  * [12] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: A method for automatic evaluation of machine translation.  In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA, 2002. Association for Computational Linguistics.
  * [13] Chin-Yew Lin.  ROUGE: A package for automatic evaluation of summaries.  In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
  * [14] Y. Zhang, Stephan Vogel, and Alexander H. Waibel.  Interpreting bleu/nist scores: How much improvement do we need to have a better system?  In International Conference on Language Resources and Evaluation, 2004.
  * [15] Satanjeev Banerjee and Alon Lavie.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics.
  * [16] Weizhe Yuan, Graham Neubig, and Pengfei Liu.  Bartscore: Evaluating generated text as text generation, 2021.
  * [17] Abhaya Agarwal and Alon Lavie.  Meteor, M-BLEU and M-TER: Evaluation metrics for high-correlation with human rankings of machine translation output.  In Proceedings of the Third Workshop on Statistical Machine Translation, pages 115–118, Columbus, Ohio, June 2008. Association for Computational Linguistics.
  * [18] Eva A M van Dis, Johan Bollen, Willem Zuidema, Robert van Rooij, and Claudi L Bockting.  ChatGPT: five priorities for research.  Nature, 614(7947):224–226, February 2023.
  * [19] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity, 2023.
  * [20] Roberto Gozalo-Brizuela and Eduardo C. Garrido-Merchan.  Chatgpt is not all you need. a state of the art review of large generative ai models, 2023.
  * [21] Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang.  Is chatgpt a general-purpose natural language processing task solver?, 2023.
  * [22] Philipp Hacker, Andreas Engel, and Marco Mauer.  Regulating chatgpt and other large generative ai models, 2023.
  * [23] Yiqiu Shen, Laura Heacock, Jonathan Elias, Keith D Hentel, Beatriu Reig, George Shih, and Linda Moy.  ChatGPT and other large language models are double-edged swords.  Radiology, page 230163, January 2023.
  * [24] Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing.  Exploring ai ethics of chatgpt: A diagnostic analysis, 2023.
  * [25] Wilbert Tabone and Joost de Winter.  Using chatgpt for human-computer interaction research: A primer, 03 2023\.
  * [26] Mohammad Khalil and Erkan Er.  Will chatgpt get you caught? rethinking of plagiarism detection, 2023\.
  * [27] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4312358/.  Exploring the role of artificial intelligence in enhancing academic performance: A case study of chatgpt.  January 2023.
  * [28] Teo Susnjak.  Chatgpt: The end of online exam integrity?, 2022.
  * [29] Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu.  How close is chatgpt to human experts? comparison corpus, evaluation, and detection, 2023.
  * [30] Katharina Jeblick, Balthasar Schachtner, Jakob Dexl, Andreas Mittermeier, Anna Theresa Stüber, Johanna Topalis, Tobias Weber, Philipp Wesp, Bastian Sabel, Jens Ricke, and Michael Ingrisch.  Chatgpt makes medicine easy to swallow: An exploratory case study on simplified radiology reports, 2022.
  * [31] Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu.  How close is chatgpt to human experts? comparison corpus, evaluation, and detection, 2023.
  * [32] Som Biswas.  Chatgpt and the future of medical writing.  Radiology, page 223312, February 2023.

##  8 Appendix

###  8.1 Evaluation Metrics

####  8.1.1 BLEU

The Bilingual Evaluation Understudy (BLEU) Score is a metric used to evaluate
the quality of the machine-generated text[12]. BLEU measures the similarity
between a machine-generated text and a reference text. It is often used in
machine translation, where the quality of a translated text is evaluated by
comparing it to a reference translation. BLEU calculates a score based on the
n-grams (contiguous sequences of words) shared between the machine-generated
text and the reference text. To calculate the BLEU score, the system-generated
output is compared to one or more reference texts using a modified form of
precision. This modified precision takes into account the number of times an
n-gram appears in the reference text(s), and the number of times it appears in
the system-generated output. The score ranges from 0 to 1, with a higher score
indicating a better match between the system-generated output and the
reference text(s).

$\text{BLEU}=BP\cdot\exp\left(\sum_{n=1}^{N}w_{n}\log p_{n}\right)$ (1)

where:

  * •

$BP$ is the brevity penalty,

  * •

$N$ is the maximum n-gram length,

  * •

$w_{n}$ is the weight assigned to the n-gram precision score, and

  * •

$p_{n}$ is the n-gram precision score.

The brevity penalty term $BP$ is calculated as:

$\text{BP}=\begin{cases}\exp(1-\frac{ref}{out})&\text{if }out<ref\\\
1&\text{if }out\geq ref\end{cases}$ (2)

The $n$-gram precision score $p_{n}$ is calculated as:

$p_{n}=\frac{\sum_{\text{n-grams
}g\in\text{output}}\min(count_{ref}(g),count_{output}(g))}{\sum_{\text{n-grams
}g\in\text{output}}count_{output}(g)}$ (3)

where $count_{ref}(g)$ is the number of times n-gram $g$ appears in the
reference translation, and $count_{output}(g)$ is the number of times n-gram
$g$ appears in the machine-generated translation.

The weights $w_{n}$ are usually set to $1/N$, where $N$ is the maximum n-gram
length.

####  8.1.2 ROUGE

Recall-Oriented Understudy for Gisting Evaluation is a set of evaluation
metrics used in natural language processing to evaluate the quality of
machine-generated summaries[13]. ROUGE calculates the overlap between a
machine-generated summary and one or more reference summaries. It measures the
quality of the summary by comparing the number of overlapping n-grams
(contiguous sequences of words) between the machine-generated summary and the
reference summary. Like BLEU, ROUGE scores range from 0 to 1, with a higher
score indicating a better match between the system-generated summary and the
reference summary. ROUGE scores are often reported as F-measure, which takes
into account both precision and recall.

$\text{ROUGE}=\frac{\sum_{r\in R}\sum_{n\in N}count_{match}(n,r)}{\sum_{r\in
R}\sum_{n\in N}count_{ref}(n,r)}$ (4)

where:

  * •

$R$ is the set of reference summaries,

  * •

$N$ is the set of n-grams in the machine-generated summary,

  * •

$count_{match}(n,r)$ is the number of times n-gram $n$ appears in reference
summary $r$, and

  * •

$count_{ref}(n,r)$ is the total number of times n-gram $n$ appears in all
reference summaries.

####  8.1.3 TER

Translation Edit Rate measures the edit distance between the machine-generated
translation and the reference translation[17]. The edit distance is the number
of changes required to transform the machine-generated translation into the
reference translation. The changes can include insertions, deletions,
substitutions, and shifts. TER calculates the edit distance using a set of
pre-defined edit operations that take into account the differences in word
order and phrase structure between the machine-generated translation and the
reference translation. The TER score is then calculated as the edit distance
divided by the length of the reference translation.

$\text{TER}=\frac{S+D+I}{N}\\\ $ (5)

where:

  * •

$S$ is the number of word substitutions required to convert the machine-
generated summary into one of the reference summaries,

  * •

$D$ is the number of words that need to be deleted from the machine-generated
summary to match the reference summary, and

  * •

$I$ is the number of words that need to be inserted into the machine-generated
summary to match the reference summary.

  * •

$N$ is the total number of words in the reference summary.

####  8.1.4 NIST

The NIST (National Institute of Standards and Technology) score equation is
commonly used in evaluating the quality of machine translation outputs[14].
The equation is as follows:

$NIST=\frac{1}{N}\sum_{n=1}^{N}\frac{c_{n}}{n}$ (6)

where:

  * •

N is the maximum order of $n$-grams used in the evaluation c_$n$ is the count
of $n$-grams that appear in the machine translation output and reference
translation $n$ is the order of the $n$-gram being evaluated ($1$ for
unigrams, $2$ for bigrams, etc.)

####  8.1.5 METEOR

The METEOR (Metric for Evaluation of Translation with Explicit ORdering) score
is another popular metric for evaluating the quality of machine translation
outputs[15]. The equation is as follows:

$\text{METEOR}=(1-\alpha)\cdot\frac{match_{seg}}{num_{seg}}+\alpha\cdot
penalty$ (7)

where:

  * •

$match_{seg}$ is the total unigram, bigram, and trigram matches between the
machine translation output and reference translation, weighted by their
importance

  * •

$num_{seg}$ is the number of words in the machine translation output

  * •

$\alpha$ is a parameter that controls the importance of precision and recall
(typically set to 0.5)

  * •

$penalty$ is a penalty term that accounts for the number of chunks, chunk
shifts, and unaligned words between the machine translation output and a
reference translation

####  8.1.6 BARTScore

BARTScore is a metric used to evaluate the quality of the generated text[16].
The BARTScore equation is as follows:

$\text{BARTScore}(x,y)=\exp{{\frac{1}{n}\sum_{i=1}^{n}\log
p_{\text{BART}}(x_{i}|x_{<i},y)-\frac{1}{m}\sum_{j=1}^{m}\log
p_{\text{BART}}(y_{j}|y_{<j},x)}}$ (8)

where:

  * •

$x$ is the input text

  * •

$y$ is the generated output text

  * •

$p_{\text{BART}}(x_{i}|x_{<i},y)$ is the conditional probability of predicting
the $i^{th}$ word in the input sequence given the previous words in the input
sequence and the generated output sequence

  * •

$p_{\text{BART}}(y_{j}|y_{<j},x)$ is the conditional probability of predicting
the $j^{th}$ word in the generated output sequence given the previous words in
the generated output sequence and the input sequence.

  * •

$n$ is the length of the input sequence.

  * •

$m$ is the length of the generated output sequence.


# Double Neutron Star Mergers: Are Late-time Radio Signals Overestimated?

Shao-Ze Li1,5,6, Yun-Wei Yu2, He Gao3,4, and Lin Lan3,4

###### Abstract

The coalescence of binary neutron stars can yield the expulsion of a fast-
moving, quasi-isotropic material, which may induce thermal radiation and give
rise to kilonova emission. Moreover, the interaction between the ejected
material and the surrounding environment generates an external shock, which
can result in a long-lasting radio signal that persists for several decades
following the merger. In contrast to supernova ejecta, kilonova ejecta
exhibits a relatively lesser mass and higher velocity, and its expansion may
ultimately result in the ejecta density becoming so low that the medium
particle can freely pass through the ejecta. Thereby it would lead to a kind
of incomplete sweeping on the interstellar medium. Employing a toy model, our
investigation reveals that such incomplete sweeping may considerably diminish
the late-time radio radiation power, irrespective of whether the binary
neutron star merger results in the formation of a black hole or a neutron
star. Our findings, thus, imply that the previously reported radio upper
limits for certain short gamma-ray bursts may not necessarily place stringent
constraints on the presence of a long-lived magnetar remnant in these short
GRBs.

###### Subject headings:

Gamma-ray burst: general — Gravitational waves — Stars: neutron — ISM: general

11affiliationtext: College of Physics Science and Technology, Hebei
University, Baoding 071002, China;
<EMAIL_ADDRESS>Institute of Astrophysics,
Central China Normal University, Wuhan 430079, China;
<EMAIL_ADDRESS>Institute for Frontier in Astronomy
and Astrophysics, Beijing Normal University, Beijing 102206, China;
<EMAIL_ADDRESS>Department of Astronomy, Beijing Normal
University, Beijing 100875, China55affiliationtext: Hebei Key Laboratory of
High-precision Computation and Application of Quantum Field Theory, Baoding,
071002, China66affiliationtext: Hebei Research Center of the Basic Discipline
for Computational Physics, Baoding, 071002, China

##  1\. Introduction

The first detection of the Gravitational Wave (GW) signal from the merger of
double Neutron Stars (NSs), GW170817, opens the new era of muti-messenger
astronomy (Abbott et al. 2017a). The association of the GW, the short Gamma-
Ray Burst (GRB), and the kilonova signals, confirms that at least part of the
short GRBs come from the mergers of NSs (Abbott et al. 2017b; Cowperthwaite et
al. 2017; Nicholl et al. 2017; Pian et al. 2017; Smartt et al. 2017; Tanaka et
al. 2017; Tanvir et al. 2017; Troja et al. 2017; Villar et al. 2017). It has
been almost six years since the detection of GW170817. However, it is still
puzzling whether the merger remnant is a Black Hole (BH) or a massive NS
(Sarin & Lasky 2022). In literature, a millisecond magnetar is usually
proposed as one candidate of the central engine of GRBs (Dai & Lü 1998a,b;
Zhang & Mészáros. 2001; Metzger et al. 2011). Compared with BH model, the
magnetar model is helpful in understanding the observational features, such as
plateau and flares in the GRB afterglow emission (Vaughan et al. 2006; Falcone
et al. 2006; Rowlinson et al. 2013; Lü et al. 2015; Gao et al. 2016).
Meanwhile, the energy injection from the central magnetar powers the isotropic
merger ejecta, resulting in the much brighter kilonova emission (mergernova;
Yu et al. 2013, 2015, 2017; Mertzger & Piro 2014; Metzger et al. 2017; Gao et
al. 2015, 2017; Li et al. 2018; Ai et al. 2022; Sarin et al. 2022;).

If the post-merger remnant is indeed a long-lived massive NS, rather than a
BH, it would give stringent constraints on the Equation of State (EoS) of
dense matter (Gao et al. 2016; Margalit & Metzger 2017; Shibata et al. 2017,
2019; Ai et al. 2018, 2020; Rezzolla et al. 2018; Radice et al. 2018). Metzger
& Bower (2014) proposed that the isotropic radio afterglow of the merger
ejecta can be used to test the merger remnant. Following the explosion of the
ejecta, an external shock is formed and sweeps up the surrounding medium.
Depending on specific parameters, the external shock could give a long-lasting
radio signal several or several tens of years after the merger (Nakar & Piran
2011). Due to the existence of magnetar, the merger ejecta can be accelerated
into relatively higher velocity, which results in much brighter radio emission
(Gao et al. 2013). Based on this starting point, several works try to catch
the radio signals from the GW170817 event and nearby short GRBs, but no
signals were detected (Metzger & Bower 2014; Fong et al. 2016; Horesh et al.
2016; Klose et al. 2019; Schroeder et al. 2020; Bruni et al. 2021; Ricci et
al. 2021). Their results give constraints on the central object, which seems
to rule out the magnetar remnant with rotation energy beyond $10^{52}\rm
ergs$. Nevertheless, Liu et al. (2020) point out that the magnetic field
fraction factor $\epsilon_{\rm B}$ is significant in calculating the radiation
mechanism. Taking a relatively lower $\epsilon_{\rm B}$ derived from GRB
afterglows, they obtain the radio lightcurves using a more detailed model. The
result shows that a magnetar remnant with rotation energy $\sim 10^{52}\rm
ergs$ is still allowed in broad parameter space.

In comparison to the supernova ejecta, the kilonova ejecta exhibits a
significantly lesser mass and higher velocity. Moreover, Margalit & Piran
(2020) proposed a relativistic jetted outflow in front of the ejecta, which
would result in much earlier radio signals. No matter the ejecta or the jetted
outflow, their density could eventually reach a low level as they expand to a
certain extent. This would make the ejecta or outflow so rarefied that it
would hardly have any significant interaction with the interstellar medium
(ISM). Meanwhile, it may result in a kind of incomplete sweeping on the ISM.
This effect has not been previously examined. To address the incomplete
sweeping, we firstly discuss the free path of the ISM particles (ionized or
neutral hydrogens) in Section 2. Subsequently, we demonstrate the limitations
of the complete-sweeping assumption under typical conditions and construct an
adapted toy model incorporating the incomplete sweeping effect (Section 3). In
Section 4, we re-evaluate the dynamics of the ejecta and illustrate the
discernible distinctions in the light curves of the late-time radio emission.
Our concluding remarks and discussions are presented in Section 5.

##  2\. the free path

In many astrophysical explosion events, such as supernovae, GRBs, etc, the
shock process usually plays a significant role in their dynamics and as well
as the emissions. As extensively discussed in the literature, a significant
fraction of astrophysical shocks are collisionless, especially in low-density
flows. Such shocks are characterized by plasma interactions instead of
particle collisions. For a charged particle moving within a plasma, the free
path is affected by the collective field, which operates on the scale of the
plasma skin depth,

$\displaystyle l_{\rm pl}$ $\displaystyle=$ $\displaystyle{c\over\omega_{\rm
pe}}$ (1) $\displaystyle=$ $\displaystyle 5.3\times 10^{5}\,n_{\rm
e}^{-1/2}\rm cm,$ (2)

where $\omega_{\rm pe}$ is the electron plasma frequency. Such a free path is
usually much less than the astrophysical flow scale. As a result,
collisionless shocks can form during the interactions of plasma flows only if
the relative velocity exceeds the local sound speed. But not all astrophysical
flows are plasmas. If the flows consist of neutral molecules or atoms, there
will be another scenario. For a neutral particle, the free path is dominated
by elastic collisions. which is

$\displaystyle l_{\rm co}$ $\displaystyle=$ $\displaystyle(\sigma n_{\rm
ej})^{-1}$ (3) $\displaystyle\approx$ $\displaystyle 10^{15}n_{\rm e}^{-1}\rm
cm,$

where $\sigma\approx 10^{-15}\rm cm^{2}$ is approximately the cross section of
typical air molecules on the earth. Obviously, this free path is much longer
than that in plasma. The precondition for the formation of a steady shock is
$l\ll L$, which requires that the scale of flows must be large enough. A
typical example of this scenario is the collisions between neutral molecular
clouds.

In supernova-like events, it’s usually assumed that the isotropic ejecta would
interact with the ambient ISM to form an external shock in the ISM and sweep
up all the surrounding medium. Such a complete-sweeping assumption is commonly
adopted in many studies to estimate the long-lasting radio emission from
kilonova events (Metzger et al. 2014; Fong et al. 2016; Horesh et al. 2016;
Liu et al. 2020; Bruni et al. 2021; Ricci et al. 2021). However, despite their
similar dynamics, kilonovae and supernovae are different in many aspects. The
kilonova explosion has a much lesser ejecta mass $\sim 0.01M_{\odot}$, but
with a relatively larger velocity $\sim 0.3c$. This results in a particular
diffusion time, $t_{\rm d}\propto(M/v)^{1/2}$, which is roughly one day
corresponding to the peak time of kilonova emission. Generally speaking, such
a diffusion timescale implies that the kilonova emission should be fast-
evolving, and the duration should be much shorter than that of supernovae.
Besides, supernovae and kilonovae also show differences in the ambient ISM. In
case of the progenitor star, core-collapse supernovae usually locate at the
HII regions. The continuous UV emission from a bright star can ionize the
hydrogen atoms and form the so-called HII bubble. The typical scale for the
HII bubble of OB star is $70\,\rm pc$. Such a kind of ISM is mostly composed
of ionized hydrogens. However, for the mergers of double NSs, they are
probably far away from the star-forming region. The ISM densities derived from
short GRB afterglows range from $10^{-4}-1\rm cm^{-3}$ (Metzger & Bower 2014),
which is significantly lower than that in HII region. Recently, Agüí Fernández
et al. (2023) reported the first chemical study of the ISM in GRB160410A. On
the one hand, the x-ray spectrum shows strong Ly$\alpha$ absorption lines, and
the column density of $\rm log(N(HI)/cm^{2})\simeq 21.2$ indicates a damped
Ly$\alpha$ host. This is similar to that of the long GRBs having $\rm
log(N(HI)/cm^{2})>20.3$ (Tanvir et al. 2019). On the other hand, the spectrum
does not show any ionized $\rm C_{\rm\,IV}$ and $\rm Si_{\rm\,IV}$ absorption
lines. These lines detected in long GRB afterglows have been assumed to
originate from the ionized hot gas in the host galaxy (Ledoux et al. 1998;
Wolfe & Prochaska 2000; Heintz et al. 2018). The result implies that this
short GRB is still within the galactic HI halo. Therefore, due to the absence
of the HII region and the hot diffuse gas, the merger events are likely to be
exposed to the neutral HI medium.

If the double NS merger events are indeed within the HI halo, the fast-moving
merger ejecta would interact with the ambient ISM which is mostly composed of
neutral hydrogens. In this picture, the free path for the neutral hydrogen
atoms should be firstly dominated by elastic collision111Only if the neutral
hydrogens are ionized during the shock, the free path can be dominated by the
plasma effect.. Years after the explosion, the merger ejecta could become so
rarefied that the free path for neutral hydrogen atoms may exceed the scale of
the ejecta. As a result, a steady shock is unsustainable, and most of the
neutral ISM hydrogens would be neither collided nor swept up by the rarefied
ejecta. In this view, the complete-sweeping assumption may be suspicious,
especially at the late time.

After the nucleosynthesis during merger explosion, the ejecta is mostly
composed of r-process elements which are heavy metal elements with mass
numbers 100-200 (Rosswog et al. 2014; Hotokezata et al. 2018). After months
and years of radioactive decay, these elements mostly convert into iron-peak
elements. Based on ${}^{56}{\rm Fe}$, a mass number of $A_{\rm r}=150$ is
adopted to characterize these elements for simplicity. The standard value is
$m=1.66\times 10^{-24}\rm g$ for one-twelfth of the mass of ${}^{12}\rm C$.
Then, the ejecta number density for supernovae and kilonovae can be estimated
by

$\displaystyle n_{\rm sn}$ $\displaystyle=$ $\displaystyle M_{\rm ej}/mV_{\rm
ej}$ (4) $\displaystyle=$ $\displaystyle 9.2\times 10^{4}\left({M_{\rm
ej}\over 10M_{\odot}}\right)\left({v_{\rm ej}\over 10^{4}{\rm
km\,s^{-1}}}\right)^{-3}\left({t\over 10\rm year}\right)^{-3}\rm cm^{-3},$

and

$\displaystyle n_{\rm kn}$ $\displaystyle=$ $\displaystyle M_{\rm ej}/A_{\rm
r}mV_{\rm ej}$ (5) $\displaystyle=$ $\displaystyle 8.4\times
10^{-4}\left({M_{\rm ej}\over 0.01M_{\odot}}\right)\left({v_{\rm ej}\over
0.3c}\right)^{-3}\left({t\over 10\rm year}\right)^{-3}\rm cm^{-3}$

where $M_{\rm ej}$ is the mass, $V_{\rm ej}=4\pi R_{\rm ej}^{3}/3$ is the
volume, $R_{\rm ej}=v_{\rm ej}t$ is the radius and $v_{\rm ej}$ is the
velocity, respectively. For the density of supernova ejecta, it is assumed
that the ejecta is composed of hydrogens from stellar envelope of the
progenitor. Obviously, the number density of merger ejecta at years later is
sufficiently lower than that of the supernova ejecta and even lower than the
typical ISM density $\sim 1\rm cm^{-3}$. Actually, the merger ejecta is more
likely a kind of diffuse cloud that has already left the nebula phase.
Combining Eq.3 & Eq.5, one can get the free path for neutral hydrogen atoms

$\displaystyle l$ $\displaystyle=$ $\displaystyle 1.2\times 10^{19}$ (6)
$\displaystyle\left({\sigma\over 10^{-16}\rm cm^{2}}\right)^{-1}\left({M_{\rm
ej}\over 0.01M_{\odot}}\right)^{-1}\left({v_{\rm ej}\over
0.3c}\right)^{3}\left({t\over 10\rm year}\right)^{3}\rm cm,$

where the cross section for ejecta elements is adopted from solid iron
(${}^{56}{\rm Fe}$). Assuming an average density of $\overline{\rho}=10\rm
g\,cm^{-3}$, the volume for each atom is $\overline{V}=A_{\rm
r}m/\overline{\rho}$. Considering the close packing of metallic atoms, the
cross section is then roughly222It is noteworthy that the atomic cross section
utilized in our calculation is derived from the ${}^{56}{\rm Fe}$ atom under
standard conditions, which may introduce uncertainty in the astrophysical
setting. $\sigma=\overline{V}^{2/3}=8.5\times 10^{-16}\rm cm^{2}.$ This is
slightly smaller than the cross section of air molecules. From Eq.6 we can see
that the condition $l\ll L$ could be well satisfied at early time with a
relatively higher ejecta density. At late time, one can see that the free path
$l\propto t^{3}$ and the radius $R\propto t$. This suggests that the free path
will finally exceed the ejecta scale. As a natural consequence, there will be
no steady shock at late time, and the ejecta can hardly sweep up the ISM.
Therefore, when calculating the shock emission, it’s significant to ensure
that the ejecta has the ability to sweep up the surrounding medium.

##  3\. the sweeping

Figure 1.— The dynamic evolution of the complete-sweeping (dashed lines) and
incomplete-sweeping (solid lines) scenarios. The dotted lines show the
location where $\tau=1$. The basic initial parameters are fixed as $M_{\rm
ej}=0.01M_{\odot}$, $\beta=0.3$, $n=10^{-3}$, $p=2.2$, $\epsilon_{\rm e}=0.1$
and $\epsilon_{\rm B}=0.1$.

Regardless of whether the shock could form, the interaction between the merger
ejecta and the ISM firstly requires that the free path $l\ll L$. Therefore, it
is of great significance to determine whether the ambient ISM is ionized or
not. Due to the natal kick of the newborn NSs, the binary systems and indeed
the merger events more probably locate at the outskirts of the host galaxy,
away from the dense HII region. Conservatively, according to Agüí Fernández et
al. (2023), it can be inferred that at least a fraction of SGRBs are still
located within the HI halo of their host galaxies. In the subsequent
calculations, it is assumed that the merger events are directly within the HI
halo, and the ambient ISM is composed of pure neutral hydrogens333The jetted
outflow proposed in Margalit & Piran (2020) is not taken into account in this
analysis. If the outflow does exist, it can be regarded as an ionized medium..
Basically, when the ejecta interacts with the ISM, the neutral ISM hydrogens
will be firstly collided by the ejecta, ionized and then shocked444It’s still
a collisionless shock in this case but firstly underwent a collisional
ionization.. Otherwise, the ejecta would continuously move and diffuse in the
neutral ISM and no steady shock could form.

In the beginning, the explosion starts at a small volume. The ejecta density
is high enough to sweep up all the ambient ISM. Then, the deceleration
timescale is

$\displaystyle t_{\rm dec}$ $\displaystyle=$ $\displaystyle{({3M_{\rm ej}/4\pi
n})^{1/3}\over v_{\rm ej}}$ (7) $\displaystyle=$ $\displaystyle
8.7\times\left({n\over 0.01\,\rm cm^{-3}}\right)^{-1/3}\left({M_{\rm ej}\over
0.01M_{\odot}}\right)^{1/3}\left({v_{\rm ej}\over 0.3c}\right)^{-1}{\rm
year},$

where $n$ is the number density of the surrounding medium. To describe the
sweeping, one can define a dimensionless parameter with the free path of Eq.6,

$\displaystyle\tau$ $\displaystyle=$ $\displaystyle{R\over l}$ (8)
$\displaystyle=$ $\displaystyle 0.24\left({\sigma\over 10^{-16}\rm
cm^{2}}\right)\left({M_{\rm ej}\over 0.01M_{\odot}}\right)\left({v_{\rm
ej}\over 0.3c}\right)^{-2}\left({t\over 10\rm year}\right)^{-2}.$

When $\tau>1$, the ejecta can be called thick, otherwise, the ejecta is thin.
The ejecta will finally become thin following the explosion. One can define
the critical time for such a transition, which is

$\displaystyle t_{\tau}$ $\displaystyle=$ $\displaystyle 4.8\left({\sigma\over
10^{-16}\rm cm^{2}}\right)^{1/2}\left({M_{\rm ej}\over
0.01M_{\odot}}\right)^{1/2}\left({v_{\rm ej}\over 0.3c}\right)^{-1}\rm year.$
(9)

We can see that, the ejecta will become thin several years after the merger.
It’s sensitive to the parameters when the transition happens. By equating
these two timescales, one can get the critical density

$\displaystyle n_{\rm c}$ $\displaystyle=$ $\displaystyle 0.057\left({M_{\rm
ej}\over 0.01M_{\odot}}\right)^{-1/2}\left({\sigma\over 10^{-16}\rm
cm^{2}}\right)^{-3/2}.$ (10)

When the density of the ambient ISM has $n>n_{\rm c}$, the ejecta would still
be thick at the deceleration timescale. That is to say, the ejecta would sweep
up most of the surrounding medium and be sufficiently decelerated. Otherwise,
when $n<n_{\rm c}$, the ejecta would become thin before the deceleration
timescale, which leads to that the ejecta is unlikely to sweep up all the
medium and will keep its velocity for a much longer timescale.

The previous complete-sweeping assumption is invalid once the ejecta becomes
thin, i.e. the free path $l>R$. It means that the the expectation of the
collision probability for neutral hydrogen atoms is less than one during the
interaction between the ejecta and the ISM. To estimate the mass of the swept-
up medium in the incomplete-sweeping scenario, we use a modified equation,

$\displaystyle{dM_{\rm sw}\over dR}=4\pi R^{2}nm_{\rm p}(1-e^{-\tau}),$ (11)

where $M_{\rm sw}$ is the total mass of swept-up medium. The advantage of this
equation is that the surrounding medium can be completely swept up when
$\tau\gg 1$, but hardly swept up when $\tau\ll 1$ . Then, ignoring the energy
lost by radiation, the dynamics can be expressed by

$\displaystyle{d\Gamma\over dR}=-{(\Gamma^{2}-1)4\pi R^{2}nm_{\rm
p}(1-e^{-\tau})\over M_{\rm ej}+2\Gamma M_{\rm sw}},$ (12)

together with

$\displaystyle{dR\over dt}={\beta c\over 1-\beta},$ (13)

where $\Gamma$ is the Lorentz factor. Note that, the dimensionless parameter
$\tau=(R/\Gamma)/(l/\Gamma)=R/l$ is unchanged during the Lorentz
transformation. With above equations, the dynamic equations can be solved out
once the initial parameters are given.

Figure 2.— The radio (6GHz) lightcurves of complete-sweeping (dashed lines)
and incomplete-sweeping (solid lines) scenarios. The basic parameters are the
same with Fig.1 but with varied $n$.

Figure 3.— The same as Fig.2, but with $M_{\rm ej}=0.001M_{\odot}$ and
$\beta=0.6$.

Figure 4.— The radio lightcurves with different ejecta mass and explosion
energy and the comparison with observation. The upper limit data is from Ricci
et al. (2021).

Figure 5.— The same as Fig.3 but with $\epsilon_{\rm B}=0.01$.

##  4\. The emission

At late time, the merger ejecta is too thin to sweep up the ISM. Meanwhile, a
global steady external shock is unfavored. However, the local shock may exist
somewhere since the ejecta has density profile or fluctuation. Therefore, the
radiation calculation in Nakar & Piran (2011) and Mertzger & Bower (2014) is
still adopted here but with little adjustment. The shocked electrons lose
their energy by synchrotron radiation. The flux and the luminosity to the
distant observer is

$\displaystyle F_{\nu}={(1+z)N_{e}\over 4\pi D_{\rm
L}^{2}}{m_{e}c^{2}\sigma_{\rm T}\Gamma\beta_{\rm m}^{2}B^{\prime}\over
3e}{\left(\nu_{\rm obs}\over\nu_{\rm m}\right)}^{-(p-1)/2}$ (14)

and

$\displaystyle L_{\nu}=F_{\nu}\,4\pi D_{\rm L}^{2},$ (15)

where the typical synchrotron frequency of electrons is $\nu_{\rm
m}=3\Gamma\gamma_{\rm m}^{2}\beta_{\rm m}^{2}eB^{\prime}/4\pi m_{e}c$, the
minimum electron Lorentz factor is $\gamma_{\rm m}=({(p-2)m_{\rm
p}/(p-1)m_{e}})\epsilon_{e}(\Gamma-1)+1$ (Liu et al. 2020), and $\nu_{\rm
obs}>\nu_{\rm m},\nu_{\rm a}$ is adopted in most realistic scenarios following
Nakar & Piran (2011). The parameters $p=2.2$ and $\epsilon_{e}=0.1$ are
adopted in this paper. The local magnetic field strength in the shock frame
could be estimated by

$\displaystyle B^{\prime}=\sqrt{8\pi\tilde{\epsilon}_{\rm
B}(4\Gamma+3)(\Gamma-1)\tilde{{n}}m_{\rm p}c^{2}},$ (16)

where the effective medium density $\tilde{n}=(1-e^{-\tau})n$. Considering
that the shock can not keep steady once $\tau\leqslant 1$, then the shock-
induced magnetic energy would decay following the decrease of $\tau$. Here, a
similar expression, $\tilde{\epsilon}_{\rm B}=(1-e^{-\tau}){\epsilon}_{\rm
B}$, is adopted to characterize the magnetic energy fraction.

Figure 6.— Constraints on the kinetic energy of the merger ejecta of GW170817.
The ejecta mass and medium density are fixed as $m_{\rm ej}=0.06\,M_{\odot}$
and $n=2\times 10^{-3}$. The limit from GW170817 is marked in violet. The
limits from other GRB samples are shown in black for comparison.

Figure 7.— Upper limit lines of the constraints on the ejecta mass and kinetic
energy from 21 SGRB samples. The violet lines are from the limit of the
GW170817 event.

As shown in Fig.1, using typical ejecta parameters, we plot the dynamics of
the ejecta in both the complete-sweeping scenario and the incomplete-sweeping
scenario. The outcomes show much difference at late time. In the complete-
sweeping scenario (dashed lines), the sweeping rate increases endlessly, even
though the ejecta has been sufficiently decelerated. But in the incomplete-
sweeping scenario (solid lines), the ejecta reach $\tau=1$ before
deceleration. After that, when $\tau<1$, the ejecta becomes thin and the
sweeping rate begins to drop. On the one hand, the incomplete-sweeping leads
to a lower sweeping rate; on the other hand, the ejecta could keep its
velocity for a much longer timescale as mentioned above. As a natural
consequence, the energy converted into radiation is decreased. Fig.2 is the
light curves for both two scenarios with different ISM densities. It is
independent on the medium density when the transition between $l<R$ and $l>R$
happens. The light curves begin to drop at the transition. Therefore, the peak
luminosity in the incomplete-sweeping scenario should be at the transition
time, $t_{\rm peak}\simeq t_{\tau}$ ($\tau=1$), once the precondition
$n<n_{\rm c}$ is satisfied. We can see that, for higher medium density, the
difference of peak luminosity in complete-sweeping and incomplete-sweeping
scenarios is not clear. But, for lower density, the peak luminosity in the
incomplete-sweeping scenario is much lower than that in the complete-sweeping
scenario. To make a better understanding, we plot Fig.3 using various ejecta
parameters for comparison. In simple terms, the lesser ejecta mass and the
lower ISM density, the much more different the peak luminosity will show in
these two scenarios. This is actually determined by Eq.10. It’s more sensitive
to the parameter $n$ than $M_{\rm ej}$. However, from Eq.9, the ejecta mass
and velocity jointly determine the transition timescale, which takes
responsibility for the peak luminosity.

The ejecta cannot sweep up enough medium after the phase transition, and the
condition $l\gg R$ for a steady shock is broken. Therefore, the late-time
radio emission could be much lower than the estimates in previous studies.
Strictly speaking, Eq.14 is inapplicable when the shock is broken. For
simplicity, we still calculate the lightcurves using this equation, in case of
that the luminosity we concern is only the peaks of the lightcurve.
Furthermore, we plot the radio lightcurves using different initial parameters
and then take a comparison with the observation. As shown in Fig.4, for an
energetic explosion (e.g. $E_{\rm k}=10^{52}\rm erg$) with relatively higher
ISM density, the lightcurves may still exceed the upper limit from
observation. This is consistent with previous works (Metzger et al. 2014; Fong
et al. 2016; Liu et al. 2020). However, under relatively lower ISM density,
our calculation shows that the lightcurves obviously blew the upper limit. If
the explosion is not powerful enough (e.g. $E_{\rm k}=10^{51}\rm erg$), our
results are in accord with the observation, the shock emission is weak and the
radio signals can hardly be detected. Furthermore, our calculation pays less
attention to the specific radiation mechanism. According to short GRB
afterglows, a relatively small value of $\epsilon_{\rm B}$ may be favored. It
would make the radio signal more undetectable as shown in Fig.5 (see the
detailed discussion in Liu et al. 2020).

##  5\. the constraints

As the first detection of the GW signals from double NS mergers, the GW170817
event gives an chance to investigate the late-time radio emission. Firstly,
from the multiband afterglow of GRB170817A, the ISM density is limited as
$n=2.5^{+4.1}_{-1.9}\times 10^{-3}\,\rm cm^{-3}$ (Hajela et al. 2019).
Secondly, if the associated kilonova (AT2017gfo) is powered by the radioactive
decay of r-process elements, the total mass of merger ejecta is estimated to
be roughly $0.06\,M_{\odot}$ (Smartt et al. 2017; Tanaka et al. 2017; Tanvir
et al. 2017; Villar et al. 2017). Adopting these two parameters, Fig.6 is
plotted to limit the kinetic energy of the merger ejecta. It clearly shows
that the late-time radio signal can hardly be detected with neither a total
energy of $10^{52}\rm erg$ nor $10^{51}\rm erg$. In another case, the required
ejecta mass could be much less than $0.06\,M_{\odot}$ if AT2017gfo is powered
by an NS or magnetar (Yu et al. 2018; Li et al. 2018; Ai et al. 2018).
Therefore, we plot more general constraints on the ejecta mass and kinetic
energy in Fig7. For completeness, all 21 samples from Ricci et al. (2021) are
included. The observational limits are converted into upper limit lines in
this plot. One can see that it provides various constraints on the ejecta mass
and kinetic energy depending on the medium density. Each of the lines clearly
show a decline and rise in the energy-mass plot. The rise on the right side of
the lines corresponds to the complete-sweeping scenario and is consistent with
the results in Ricci et al. (2021). For the decline on the left, it
corresponds to the incomplete-sweeping scenario. The watershed of the decline
and rise is actually determined by Eq.9, where the deceleration timescale and
transition timescale get a balance. From Fig.7, an energetic explosion (e.g.
$E_{\rm k}=10^{52}\rm erg$) is definitely allowed in a broad parameter space,
especially with low medium densities.

##  6\. conclusion and discussion

Observing long-lasting radio emissions from merger events helps us understand
the evolution of the merger ejecta. The radio emission should be much brighter
if the post-merger remnant is an NS instead of a BH. However, based on the
reasonable neutral ISM assumption, our results show that the merger ejecta
would be too rarefied to sweep up the ISM after several years of explosion.
Instead of sweeping, the merger ejecta would eventually diffuse in the ISM
losing its energy over a much longer timescale through atomic collisions or
scattering. As a result, the radio signal is too weak to be detected under
typical conditions. Even with an NS remnant, it’s still hard to catch the
radio signal if the density of the ISM is not high enough. Therefore, it is
not easy to rule out an NS (or magnetar) remnant with the current observation.

It is usually considered that the merger remnant is rotating near the Kepler
period because of its significant orbital angular momentum. With some possible
mechanisms, such as magnetic-field-induced ellipticities (Bonazzola &
Gourgoulhon 1996; Palomba 2001; Cutler 2002) and various instability modes
(Lai & Shapiro 1995; Corsi & Mészáros 2009; Andersson 1998; Lindblom et al.
1998; Dall’osso et al. 2015), the GW radiation becomes significant for such a
rapidly rotating NS. Based on the short GRBs, Gao et al. (2016) demonstrate
that the rotation energy of the merger remnant is probably lost in the GW
channel. Lü et al. (2020) find a short GRB sample as evidence for the GW-
dominated afterglow emission. Therefore, it’s most likely that only a fraction
of the energy could be reflected in x-ray plateau and kilonova emission, which
is only on the order of $10^{51}\rm erg$ (Rowlison et al. 2013; Lü et al.
2015; Gao et al. 2016; Yu et al. 2017; Li et al. 2018). With such a total
energy, the long-lasting radio signal is hard to detect. Anyhow, the first
detection of post-merger GW signal is still expected once the advanced LIGO
and Virgo reach the designed sensitivity (Abbott et al. 2017c; 2019; Caudill
et al. 2021). At that time, the GW observation could help us identify the
merger remnant.

This work is supported by the National Natural Science Foundation of China
(grant No. 12021003, 11833003, 12303044), National Key R&D Program of China
(2021YFA0718500), and the National SKA Program of China (2020SKA0120300,
2022SKA0130100).

## References

  * Abbott et al. (2017) Abbott, B. P., Abbott, R., Abbott, T. D., et al. 2017a, Phys. Rev. Lett., 119, 161101
  * Abbott et al. (2017) Abbott, B. P., Abbott, R., Abbott, T. D., et al. 2017b, ApJ, 851, L16
  * Abbott et al. (2017) Abbott, B. P., Abbott, R., Abbott, T. D., et al. 2017c, ApJ, 848, L13
  * Abbott et al. (2019) Abbott, B. P., Abbott, R., Abbott, T. D., et al. 2019, ApJ, 875, 160
  * Agüí Fernández et al. (2023) Agüí Fernández, J. F., Thöne, C. C., Kann, D. A., et al. 2023, MNRAS, 520, 613
  * Ai et al. (2018) Ai, S., Gao, H., Dai, Z.-G., et al. 2018, ApJ, 860, 57
  * Ai et al. (2020) Ai, S., Gao, H., & Zhang, B. 2020, ApJ, 893, 146
  * Ai et al. (2022) Ai, S., Zhang, B., & Zhu, Z. 2022, MNRAS, 516, 2614
  * Andersson (1998) Andersson, N. 1998, ApJ, 502, 708
  * Bonazzola & Gourgoulhon (1996) Bonazzola, S. & Gourgoulhon, E. 1996, A&A, 312, 675
  * Bruni et al. (2021) Bruni, G., O’Connor, B., Matsumoto, T., et al. 2021, MNRAS, 505, L41
  * Caudill et al. (2021) Caudill, S., Kandhasamy, S., Lazzaro, C., et al. 2021, Modern Physics Letters A, 36, 2130022-458
  * Corsi & Mészáros (2009) Corsi, A. & Mészáros, P. 2009, ApJ, 702, 1171
  * Cowperthwaite et al. (2017) Cowperthwaite, P. S., Berger, E., Villar, V. A., et al. 2017, ApJ, 848, L17
  * Cutler (2002) Cutler, C. 2002, Phys. Rev. D, 66, 084025
  * Dai & Lu (1998) Dai, Z. G. & Lu, T. 1998a, A&A, 333, L87
  * Dai & Lu (1998) Dai, Z. G. & Lu, T. 1998b, Phys. Rev. Lett., 81, 4301
  * Dall’Osso et al. (2015) Dall’Osso, S., Giacomazzo, B., Perna, R., et al. 2015, ApJ, 798, 25
  * Falcone et al. (2007) Falcone, A. D., Morris, D., Racusin, J., et al. 2007, ApJ, 671, 1921
  * Fong et al. (2016) Fong, W., Metzger, B. D., Berger, E., et al. 2016, ApJ, 831, 141
  * Gao et al. (2013) Gao, H., Ding, X., Wu, X.-F., et al. 2013, ApJ, 771, 86
  * Gao et al. (2015) Gao, H., Ding, X., Wu, X.-F., et al. 2015, ApJ, 807, 163
  * Gao et al. (2016) Gao, H., Zhang, B., & Lü, H.-J. 2016, Phys. Rev. D, 93, 044065
  * Gao et al. (2017) Gao, H., Zhang, B., Lü, H.-J., et al. 2017, ApJ, 837, 50
  * Hajela et al. (2019) Hajela, A., Margutti, R., Alexander, K. D., et al. 2019, ApJ, 886, L17
  * Heintz et al. (2018) Heintz, K. E., Watson, D., Jakobsson, P., et al. 2018, MNRAS, 479, 3456
  * Horesh et al. (2016) Horesh, A., Hotokezaka, K., Piran, T., et al. 2016, ApJ, 819, L22
  * Hotokezaka et al. (2018) Hotokezaka, K., Beniamini, P., & Piran, T. 2018, International Journal of Modern Physics D, 27, 1842005
  * Klose et al. (2019) Klose, S., Nicuesa Guelbenzu, A. M., Michałowski, M. J., et al. 2019, ApJ, 887, 206
  * Lai & Shapiro (1995) Lai, D. & Shapiro, S. L. 1995, ApJ, 442, 259
  * Ledoux et al. (1998) Ledoux, C., Petitjean, P., Bergeron, J., et al. 1998, A&A, 337, 51
  * Li et al. (2018) Li, S.-Z., Liu, L.-D., Yu, Y.-W., et al. 2018, ApJ, 861, L12
  * Lindblom et al. (1998) Lindblom, L., Owen, B. J., & Morsink, S. M. 1998, Phys. Rev. Lett., 80, 4843
  * Liu et al. (2020) Liu, L.-D., Gao, H., & Zhang, B. 2020, ApJ, 890, 102
  * Lü et al. (2015) Lü, H.-J., Zhang, B., Lei, W.-H., et al. 2015, ApJ, 805, 89
  * Lü et al. (2020) Lü, H.-J., Yuan, Y., Lan, L., et al. 2020, ApJ, 898, L6
  * Margalit & Metzger (2017) Margalit, B. & Metzger, B. D. 2017, ApJ, 850, L19
  * Margalit & Piran (2020) Margalit, B. & Piran, T. 2020, MNRAS, 495, 4981
  * Metzger et al. (2017) Metzger, B. D., Berger, E., & Margalit, B. 2017, ApJ, 841, 14
  * Metzger et al. (2011) Metzger, B. D., Giannios, D., Thompson, T. A., et al. 2011, MNRAS, 413, 2031
  * Metzger & Piro (2014) Metzger, B. D. & Piro, A. L. 2014, MNRAS, 439, 3916
  * Metzger & Bower (2014) Metzger, B. D. & Bower, G. C. 2014, MNRAS, 437, 1821
  * Nakar & Piran (2011) Nakar, E. & Piran, T. 2011, Nature, 478, 82
  * Nicholl et al. (2017) Nicholl, M., Berger, E., Kasen, D., et al. 2017, ApJ, 848, L18
  * Palomba (2001) Palomba, C. 2001, A&A, 367, 525
  * Pian et al. (2017) Pian, E., D’Avanzo, P., Benetti, S., et al. 2017, Nature, 551, 67
  * Radice et al. (2018) Radice, D., Perego, A., Zappa, F., et al. 2018, ApJ, 852, L29
  * Rezzolla et al. (2018) Rezzolla, L., Most, E. R., & Weih, L. R. 2018, ApJ, 852, L25
  * Ricci et al. (2021) Ricci, R., Troja, E., Bruni, G., et al. 2021, MNRAS, 500, 1708
  * Rosswog et al. (2014) Rosswog, S., Korobkin, O., Arcones, A., et al. 2014, MNRAS, 439, 744
  * Rowlinson et al. (2013) Rowlinson, A., O’Brien, P. T., Metzger, B. D., et al. 2013, MNRAS, 430, 1061
  * Sarin & Lasky (2021) Sarin, N. & Lasky, P. D. 2021, General Relativity and Gravitation, 53, 59
  * Sarin et al. (2022) Sarin, N., Omand, C. M. B., Margalit, B., et al. 2022, MNRAS, 516, 4949
  * Schroeder et al. (2020) Schroeder, G., Margalit, B., Fong, W.-. fai ., et al. 2020, ApJ, 902, 82
  * Shibata et al. (2019) Shibata, M., Zhou, E., Kiuchi, K., et al. 2019, Phys. Rev. D, 100, 023015
  * Shibata et al. (2017) Shibata, M., Fujibayashi, S., Hotokezaka, K., et al. 2017, Phys. Rev. D, 96, 123012
  * Smartt et al. (2017) Smartt, S. J., Chen, T.-W., Jerkstrand, A., et al. 2017, Nature, 551, 75
  * Tanaka et al. (2017) Tanaka, M., Utsumi, Y., Mazzali, P. A., et al. 2017, PASJ, 69, 102
  * Tanvir et al. (2017) Tanvir, N. R., Levan, A. J., González-Fernández, C., et al. 2017, ApJ, 848, L27
  * Tanvir et al. (2019) Tanvir, N. R., Fynbo, J. P. U., de Ugarte Postigo, A., et al. 2019, MNRAS, 483, 5380
  * Troja et al. (2017) Troja, E., Piro, L., van Eerten, H., et al. 2017, Nature, 551, 71
  * Vaughan et al. (2006) Vaughan, S., Goad, M. R., Beardmore, A. P., et al. 2006, ApJ, 638, 920
  * Villar et al. (2017) Villar, V. A., Guillochon, J., Berger, E., et al. 2017, ApJ, 851, L21
  * Wolfe & Prochaska (2000) Wolfe, A. M. & Prochaska, J. X. 2000, ApJ, 545, 591
  * Yu et al. (2015) Yu, Y.-W., Li, S.-Z., & Dai, Z.-G. 2015, ApJ, 806, L6
  * Yu et al. (2018) Yu, Y.-W., Liu, L.-D., & Dai, Z.-G. 2018, ApJ, 861, 114
  * Yu et al. (2013) Yu, Y.-W., Zhang, B., & Gao, H. 2013, ApJ, 776, L40
  * Zhang & Mészáros (2001) Zhang, B. & Mészáros, P. 2001, ApJ, 552, L35