# Generating Synthetic Documents for Cross-Encoder Re-Rankers: A Comparative
Study of ChatGPT and Human Experts

Arian Askari<EMAIL_ADDRESS>Leiden UniversityThe Netherlands ,
Mohammad Aliannejadi  m.aliannejadi,e.kanoulas University of AmsterdamThe
Netherlands ,  Evangelos Kanoulas<EMAIL_ADDRESS>University of
AmsterdamThe Netherlands and  Suzan Verberne<EMAIL_ADDRESS>Leiden UniversityThe Netherlands

###### Abstract.

We investigate the usefulness of generative Large Language Models (LLMs) in
generating training data for cross-encoder re-rankers in a novel direction:
generating synthetic documents instead of synthetic queries. We introduce a
new dataset, ChatGPT-RetrievalQA, and compare the effectiveness of models
fine-tuned on LLMgenerated and human-generated data. Data generated with
generative LLMs can be used to augment training data, especially in domains
with smaller amounts of labeled data. We build ChatGPT-RetrievalQA based on an
existing dataset, human ChatGPT Comparison Corpus (HC3), consisting of public
question collections with human responses and answers from ChatGPT. We fine-
tune a range of cross-encoder re-rankers on either human-generated or ChatGPT-
generated data. Our evaluation on MS MARCO DEV, TREC DL’19, and TREC DL’20
demonstrates that cross-encoder re-ranking models trained on ChatGPT responses
are statistically significantly more effective zero-shot re-rankers than those
trained on human responses. In a supervised setting, the human-trained re-
rankers outperform the LLM-trained re-rankers. Our novel findings suggest that
generative LLMs have high potential in generating training data for neural
retrieval models. Further work is needed to determine the effect of factually
wrong information in the generated responses and test our findings’
generalizability with open-source LLMs. We release our data, code, and cross-
encoders checkpoints for future work. 111https://github.com/arian-
askari/ChatGPT-RetrievalQA

††journalyear: 2023††copyright: None††conference: ; May 3rd, 2023;

##  1\. Introduction

Generative large language models (LLMs) such as GPT-3 (Brown et al., 2020) and
GPT-3.5 (including ChatGPT) have shown remarkable performance in generating
realistic text outputs for a variety of tasks such as summarization (Zhang et
al., 2023), machine translation (Peng et al., 2023), sentiment analysis
(Susnjak, 2023; Wang et al., 2023a), retrieval interpretability (Llordes et
al., 2023), and stance detection (Zhang et al., 2022). Although ChatGPT can
produce impressive answers, it is not immune to errors or hallucinations
(Goldstein et al., 2023). Furthermore, the lack of transparency in the source
of information generated by ChatGPT can be a bigger concern in domains such as
law, medicine, and science, where accountability and trustworthiness are
critical (Cascella et al., 2023; Sun, 2023; Sallam et al., 2023).

Retrieval models, as opposed to generative models, retrieve the actual (true)
information from sources and search engines provide the source of each
retrieved item (Sanderson and Croft, 2012). This is why information retrieval
(IR) — even when generative LLMs are available — remains an important
application, especially in situations where reliability is vital. One
potential purpose of generative LLMs in IR is to generate training data for
retrieval models. Data generated with generative LLMs can be used to augment
training data, especially in domains with smaller amounts of labeled data. In
this paper, we present the ChatGPT-RetrievalQA dataset to address two research
questions:

RQ1: How does the effectiveness of cross-encoder re-rankers fine-tuned on
ChatGPT-generated responses compare to those fine-tuned on human-generated
responses in both supervised and zero-shot settings?

RQ2: How does the effectiveness of using ChatGPT for generating relevant
documents differ between specific and general domains?

By answering these questions, we aim to shed light on the potential of using
LLMs for data augmentation in cross-encoder re-rankers and the domain
dependency of their effectiveness. As shown in Figure 1, our primary
experimental setup involves using CEChatGPT222We refer to the cross-encoders
fine-tuned on ChatGPT-generated and human-generated responses as CEChatGPT and
CEhuman, respectively. for inference (i.e., re-ranking task) on human-
generated responses.

Our dataset and analysis provide insights into the benefits and limitations of
using generative LLMs for augmenting training data for retrieval models.

Our main contributions in this work are three-fold:  (i) We release the
ChatGPT-RetrievalQA dataset, which is based on the HC3 dataset (Guo et al.,
2023) but is designed specifically for information retrieval tasks in both
full-ranking and re-ranking setups. This dataset contains $24,322$ queries,
$26,882$ responses generated by ChatGPT, and $58,546$ human-generated
responses.  (ii) We fine-tune cross-encoder re-rankers on both the human- and
ChatGPT-generated responses, evaluating their performance on our dataset in a
supervised setting. We also show the effectiveness of the ChatGPT-trained
models in a zero-shot evaluation on the MS MARCO-passage collection and the
TREC Deep Learning tracks.  (iii) We conduct an analysis of the effectiveness
of ChatGPT-trained cross-encoders on different domains and show that human-
trained models are slightly more effective in domain-specific tasks, e.g., in
the medicine domain.  Our novel findings highlight the potential of using
generative LLMs like ChatGPT for generating high-quality responses in
information retrieval tasks in order to create training datasets.

##  2\. Related Work

InPars (Bonifacio et al., 2022), Promptagator (Dai et al., 2022), and
InPars-v2 (Jeronymo et al., 2023) have utilized LLMs to generate synthetic
queries given documents. Particularly, InPars-v2 (Jeronymo et al., 2023)
achieves state-of-the-art results on the BEIR dataset in a zero-shot setting
by using an open-source language model, GPT-J-6B (Wang and Komatsuzaki, 2021)
and a powerful external re-ranker, MonoT5-MSMARCO (Nogueira et al., 2020) to
filter the top-10k high-quality pairs of synthetic query-document pairs for
data augmentation. In contrast, we use documents (passages) generated by
ChatGPT given a query – the reverse from InPars-v2. Document generation for
given queries as a source for data augmentation has not been explored in prior
work.

We believe that exploring this reverse direction is important as it allows us
to augment training data with a focus on user behavior and query logs rather
than the (static) document collection itself. This can improve the
effectiveness of re-rankers by augmenting the training data with synthetic
documents according to the queries that actual search engine users are
searching for, increasing the diversity of the training data, while allowing
the rankers to better generalize to new queries.

Guo et al. (2023) use public question-answering datasets (see below) and
prompt the questions to the ChatGPT user interface for generating answers. The
goal of the HC3 dataset is to linguistically compare human and ChatGPT
responses and explore the possibility of distinguishing between responses
generated by ChatGPT and those written by humans. The HC3 dataset contains
questions (i.e., queries) from four different domains: medicine (Medical
Dialog (Chen et al., 2020)), finance (FiQA (Maia et al., 2018)), Wikipedia
(WikiQA (Yang et al., 2015) and Wiki_csai (Guo et al., 2023)), and Reddit
(ELI5 (Fan et al., 2019)). While there is no study on generating documents to
augment training data, a more recent study, QuerytoDoc (Wang et al., 2023b),
generates documents for query expansion, which is out of the scope of
augmenting data for information retrieval. Furthermore, there are various
recent studies on ChatGPT with a focus on ranking and retrieval but to the
best of our knowledge, none of them focus on data augmentation by generating
relevant documents. Examples of recent studies are the one by Faggioli et al
(Faggioli et al., 2023), who study if ChatGPT can be used for generating
relevance labels, and Sun et al. (2023), who assess whether ChatGPT is good at
searching by giving it a query and set of candidate documents and asking for
re-ranking.

Table 1. Statistic on the size of Train, Validation, and Test sets across domains for evaluation of cross-Encoders. Domain | # of queries
---|---
Train set | Validation set | Test set
All | 16788 | 606 | 6928
Medicine: Meddialog (Chen et al., 2020) | 862 | 31 | 355
Finance: FiQA (Maia et al., 2018) | 2715 | 98 | 1120
Reddit: ELI5 (Fan et al., 2019) | 11809 | 427 | 4876
Wikipedia: openQA (Yang et al., 2015) | 820 | 29 | 338
Wikipedia: csai (Guo et al., 2023) | 582 | 21 | 239

Figure 1. Our main experimental setup involves above steps.

##  3\. Dataset

Our ChatGPT-RetrievalQA dataset is based on the HC3 dataset produced by Guo et
al. (2023), which contains $24,322$ queries and $26,882$ ChatGPT-generated
responses, as well as $58,546$ human-generated responses. There is on average
one ChatGPT-generated and $2.4$ human-generated response per query. To build
the ChatGPT-RetrievalQA dataset for retrieval and ranking experiments (an
experimental setup different from (Guo et al., 2023)), we convert the dataset
files to a format similar to the MSMarco dataset (Nguyen et al., 2016), in
both full-ranking and re-ranking setups.333This allows for easy reuse of
available scripts on MS MARCO. We divide the data into training, validation,
and test sets.

Table 2. Comparing the effectiveness of cross-encoder re-rankers fine-tuned on human and ChatGPT responses in supervised and zero-shot settings. $\dagger$ indicates that a CE achieves statistically significant improvement for that dataset among all of the cross-encoder re-rankers and BM25 on that dataset. Statistical significance was measured with a paired t-test ($p<0.05$) with Bonferroni correction for multiple testing. The cutoff for MAP, NDCG, and MRR are 1000, 10, and 10. | Zero-shot setting | Supervised setting
---|---|---
Model | TREC DL’19 | TREC DL’20 | MS MARCO DEV | ChatGPT-RetrievalQA (Ours)
MAP | NDCG | MRR | MAP | NDCG | MRR | MAP | NDCG | MRR | MAP | NDCG | MRR
BM25 | .377 | .506 | .858 | .286 | .480 | .819 | .195 | .234 | .187 | .143 | .184 | .240
MiniLMhuman | .326 | .451 | .833 | .269 | .376 | .913 | .130 | .155 | .118 | .310$\dagger$ | .384$\dagger$ | .460$\dagger$
MiniLMChatGPT | .342$\dagger$ | .510$\dagger$ | .903 | .344$\dagger$ | .539$\dagger$ | .978$\dagger$ | .226$\dagger$ | .267$\dagger$ | .218$\dagger$ | .294 | .362 | .444
TinyBERThuman | .294 | .360 | .741 | .277 | .364 | .791 | .128 | .154 | .116 | .244 | .310 | .367
TinyBERTChatGPT | .328 | .488 | .942$\dagger$ | .303 | .460 | .972 | .194 | .231 | .185 | .231 | .291 | .358

To facilitate training, we provide training triples files in TSV format,
including both textual and ID-based representations, where the structure of
each line is composed of ‘query, positive passage, negative passage’ and ‘qid,
positive pid, negative pid’. We consider the actual response by ChatGPT or
human as the relevant answer and we randomly sample $1000$ negative answers
for each query similar to MS MARCO. In addition, we provide the top 1000
documents, ranked by BM25, per query to enable re-ranking studies. Table 1
shows the size of the train, validation, and test sets for each domain.

##  4\. Methods

###  4.1. First-stage ranker: BM25

Lexical retrievers use word overlap to produce the relevance score between a
document and a query. Several lexical approaches have been developed in the
past, such as vector space models, Okapi BM25 (Robertson and Walker, 1994),
and query likelihood. We use BM25 as the first-stage ranker because of its
popularity and effectiveness. BM25 calculates a score for a query–document
pair based on the statistics of the words that overlap between them:

(1)

$s_{lex}(q,d)=BM25(q,d)=\sum_{t\in q\cap
d}{rsj_{t}.\frac{tf_{t,d}}{tf_{t,d}+k_{1}\\{(1-b)+b\frac{|d|}{l}\\}}}$

where $t$ is a term, $tf_{t,d}$ is the frequency of $t$ in document $d$,
$rsj_{t}$ is the Robertson-Spärck Jones weight (Robertson and Walker, 1994) of
$t$, and $l$ is the average document length. $k_{1}$ and $b$ are parameters.

###  4.2. Cross-encoder re-rankers

The common approach to employ pre-trained Transformer models with a cross-
encoder architecture in a re-ranking setup is by concatenating the input
sequences of query and passage. This method, known as MonoBERT or CECAT, is
illustrated in Figure 1 and has been utilized in several studies. In CECAT,
the sequences of query words $q_{1}:q_{m}$ and passage words $p_{1}:p_{n}$ are
joined with the [SEP] token, and the ranking model of CECAT calculates the
score for the representation of the [CLS] token obtained by cross-encoder (CE)
using a single linear layer Ws:

(2)

$CE_{CAT}(q_{1:m},p_{1:n})=CE([CLS]\,q\,[SEP]\,p\,[SEP])*W_{s}$

We use CECAT as our cross-encoder re-ranker with a re-ranking depth of $1000$.
In our experiments, both CEChatGPT and CEhuman follow the above design.

##  5\. Experimental design

Evaluation setup. We conduct our zero-shot evaluation experiments on the MS
MARCO-passage collection (Nguyen et al., 2016) and the two TREC Deep Learning
tracks (TREC-DL’19 and DL’20) (Craswell et al., 2020; Craswell et al., 2021).
The evaluation metrics used are MAP@1000, NDCG@10, and MRR@10, which are
standard for these datasets, to make our results comparable to previously
published and upcoming research (Craswell et al., 2021; Craswell et al.,
2020). The MS MARCO-passage dataset contains about $8.8$ million passages
(average length: $73.1$ words) and about $1$ million natural language queries
(average length: $7.5$ words) and has been extensively used to train deep
language models for ranking. Following prior work on MS MARCO (Khattab and
Zaharia, 2020; Lin et al., 2021; MacAvaney et al., 2020; Zhuang and Zuccon,
2021; Zhuang et al., 2021), we only use the dev set ($\sim 7k$ queries) for
our empirical evaluation. The TREC DL’19 and DL’20 collections share the
passage corpus of MS MARCO and have $43$ and $54$ queries respectively with
much more relevant documents per query. We measure the same metrics in the
supervised setting on the test set of ChatGPT-RetrievalQA. The average length
of human responses is $142.5$ words, and $198.1$ words for ChatGPT in the
ChatGPT-RetrievalQA dataset.

Training configuration. We use the Huggingface library (Wolf et al., 2019),
and PyTorch (Paszke et al., 2017) for the cross-encoder re-ranking training
and inference. Following prior work (Hofstätter et al., 2020), we use the Adam
(Kingma and Ba, 2014) optimizer with a learning rate of $7*10^{-6}$ for all
cross-encoder layers, regardless of the number of layers trained. We use a
training batch size of 32. For all cross-encoder re-rankers, we use the cross-
entropy loss (Zhang and Sabuncu, 2018). For the lexical retrieval with BM25,
we use the similarity function of Elasticsearch (Elasticsearch, 2023). We cap
the query length at $30$ tokens and the passage length at $200$ tokens
following prior work (Hofstätter et al., 2020; Askari et al., 2023).

Table 3. Comparing the effectiveness of CEC and CEH in supervised setting across different domains where CE, C, and H refer to the MiniLM, human, and ChatGPT. The OpenQA and Wiki_csai datasets are in Wikipedia domain. Domain | Model | MAP@1K | NDCG@10 | MRR@10
---|---|---|---|---
All | CEH | .310 | .384 | .460
CEC | .294 | .362 | .444
Medicine (Chen et al., 2020) | CEH | .397 | .419 | .395
CEC | .379 | .400 | .377
Finance (Maia et al., 2018) | CEH | .257 | .399 | .251
CEC | .250 | .368 | .245
Reddit (Fan et al., 2019) | CEH | .323 | .418 | .543
CEC | .302 | .391 | .522
OpenQA (Yang et al., 2015) | CEH | .322 | .345 | .320
CEC | .331 | .341 | .328
Wiki_csai (Guo et al., 2023) | CEH | .149 | .152 | .135
CEC | .163 | .159 | .144

##  6\. Results

###  6.1. Main results (RQ1)

Table 2 shows a comparison of the effectiveness of CEhuman and CEChatGPT.
Please note that for both models, during inference, we evaluate their
effectiveness in retrieving human responses in the supervised or zero-shot
settings. We choose MiniLM (12 layers version) (Wang et al., 2020) for the
experiments due to its competitive results in comparison to BERT re-ranker
(Askari et al., 2023) while being three times smaller and six times faster. In
addition, we conduct experiments with TinyBERT (2 layers version) (Jiao et
al., 2020) to assess the generalizability of our results.

In the supervised setting where we evaluate the on test set queries with human
documents of our ChatGPT-RetrievalQA dataset, MiniLMhuman significantly
outperforms all of the other cross-encoders re-rankers. Although lower than
the human-trained models, MiniLMChatGPT and TinyBERTChatGPT still outperform
the strong baseline (Bonifacio et al., 2022), BM25 (Robertson and Walker,
1994), statistically significantly by a large margin in this setting.

In the zero-shot setting, the MiniLMChatGPT consistently outperforms the other
cross-encoder re-rankers including MiniLMhuman and BM25 significantly across
the TREC DL’20 and MS MARCO DEV. However, on TREC DL’19, BM25 achieves the
highest effectiveness for MAP@1000, MiniLMChatGPT for NDCG@10, and
TinyBERTChatGPT for MRR@10. Overall, we can see the models fine-tuned on
ChatGPT-generated responses are significantly more effective in the zero-shot
setting compared to those fine-tuned on human-generated responses.

###  6.2. Domain-level re-ranker effectiveness (RQ2)

Table 3 shows the effectiveness of MiniLMhuman and MiniLMChatGPT re-rankers in
the supervised settings – on the test set of our dataset – across all of the
domains including Medicine, Finance, Reddit, and Wikipedia. Overall, the
results show that MiniLMhuman achieves higher effectiveness than MiniLMChatGPT
for all domains except Wikipedia. However, the difference in effectiveness is
small, and MiniLMChatGPT still achieves a reasonable level of effectiveness.
In the Finance domain, both MiniLMhuman and MiniLMChatGPT achieve relatively
low effectiveness compared to other domains. In the Wikipedia domain,
MiniLMhuman and MiniLMChatGPT achieve relatively similar levels of
effectiveness. In the Medicine domain, the CEhuman shows the highest
effectiveness. Overall, these results suggest that MiniLMhuman performs more
effective in supervised settings, particularly in specific domains such as
Medicine, even though the difference in performance is small.

##  7\. Discussion

Data overlap. It is worth noting that in the supervised setting, the
collection of documents used for training and testing is shared for CEhuman
re-rankers. Therefore, some documents may be seen during both training and
evaluation. This setup is very common when working with human-assessed data,
and similar to the MS MARCO dataset (Nguyen et al., 2016). The shared
collection could be a potential benefit for CEhuman re-rankers in the
supervised setting, as the models may have already seen some of the documents
during training. To further investigate this hypothesis, it would be worth
exploring a different setup in future work in which the collection of
documents is completely separated between the training and test sets.

Effectiveness of BM25. Table 4 shows an analysis of the effectiveness of BM25
on human and ChatGPT-generated responses in the train, and test sets. BM25 is
less effective for human-generated responses than for ChatGPT-generated
responses on the train and test sets, as evidenced by lower scores for all
metrics. We observed the same pattern for the validation set. These results
suggest that the task of retrieving human-generated responses is more
challenging for BM25 than for ChatGPT-generated responses. This is probably
related to the lexical overlap discussed below.

Queries without label. In Table 5, we investigate a common scenario in real-
world search engines where query logs and a collection of human-generated
documents are available, and there are not any judged documents for part or
all of the query logs. To simulate and analyze this situation, we evaluate
CEChatGPT on the seen queries of the train set and unseen documents of the
human-generated documents collection. Table 5 shows that CEChatGPT rankers are
fairly effective in this scenario. Especially, they are more effective than
BM25 in the same setup, in that the NDCG@10 for MiniLMChatGPT is 0.388 and
0.202 for BM25 (see the third row of Table 4). This suggests the potential of
augmenting training data with generative LLMs for fine-tuning models to
effectively re-rank sourced and reliable human-generated documents from the
corpus given the query logs where there are no judged documents for the
queries.

Table 4. Analyzing the effectiveness of BM25 on human/ChatGPT responses in train, validation, and test set. Split | Source | MAP@1K | NDCG@10 | Recall@1K
---|---|---|---|---
Test | human | .143 | .184 | .520
ChatGPT | .370 | .396 | .898
Train | human | .158 | .202 | .560
ChatGPT | .413 | .443 | .903

Lexical overlap. Our data analysis reveals that ChatGPT-generated responses
have a slightly higher lexical overlap than human-generated responses with the
queries. The average percentage of query words that occur in ChatGPT-generated
responses is 34.6%, compared to 25.5% for humans. The Q1, median, and Q3 are
also on average 7% points higher for ChatGPT compared to human responses. We
suspect that this higher lexical overlap compared to the human response
happens because ChatGPT often repeats the question or query in the response,
and it tends to generate lengthier responses compared to human which increase
the chance of having mutual words with the query. It is noteworthy to mention
that lexical overlap is not the best indicator of response quality for fine-
tuning effective cross-encoders, as there may be cases where responses with
low lexical overlap are still relevant and informative, especially in
question-answering tasks.

Table 5. Analyzing the effectiveness of CEsChatGPT on the seen queries of the train set and unseen documents of human-generated documents collection. Model | MAP@1K | NDCG@10 | MRR@10
---|---|---|---
MiniLMChatGPT | .318 | .388 | .510
TinyBERTChatGPT | .254 | .318 | .420

##  8\. Conclusion

In this paper, we have presented an analysis of the effectiveness of fine-
tuning cross-encoders on human-generated responses versus ChatGPT-generated
responses. Our results show that the cross-encoderChatGPT is more effective
than cross-encoderhuman in the zero-shot setting while MiniLM human is
slightly more effective in the supervised setting and this is consistent
across different domains. Furthermore, we show that BM25 is less effective on
human-generated responses than on ChatGPT-generated responses, indicating that
human-generated responses are more challenging to match with queries than
ChatGPT-generated responses. Overall, our findings suggest that ChatGPT-
generated responses are more useful than human-generated responses for
training effective zero-shot re-ranker, at least based on our dataset and
experiments, and highlight the potential of using generative LLMs for
generating effective and useful responses for creating training datasets in
natural language processing tasks. Our study can be particularly advantageous
for domain-specific tasks where relying on LLM-generated output as a direct
response to a user query can be risky. Our results confirm that it is possible
to train effective cross-encoder re-rankers by training them on ChatGPT-
generated responses even for domain-specific queries. Further work is needed
to determine the effect of factually wrong information in the generated
responses and to test the generalizability of our findings on open-source
LLMs.

## References

  * (1)
  * Askari et al. (2023) Arian Askari, Amin Abolghasemi, Gabriella Pasi, Wessel Kraaij, and Suzan Verberne. 2023\.  Injecting the BM25 Score as Text Improves BERT-Based Re-rankers. In _Advances in Information Retrieval_. Springer Nature Switzerland, Cham, 66–83.
  * Bonifacio et al. (2022) Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022.  Inpars: Data augmentation for information retrieval using large language models.  _arXiv preprint arXiv:2202.05144_ (2022).
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Cascella et al. (2023) Marco Cascella, Jonathan Montomoli, Valentina Bellini, and Elena Bignami. 2023.  Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios.  _Journal of Medical Systems_ 47, 1 (2023), 1–5.
  * Chen et al. (2020) Shu Chen, Zeqian Ju, Xiangyu Dong, Hongchao Fang, Sicheng Wang, Yue Yang, Jiaqi Zeng, Ruisi Zhang, Ruoyu Zhang, Meng Zhou, et al. 2020\.  Meddialog: a large-scale medical dialogue dataset.  _arXiv preprint arXiv:2004.03329_ (2020).
  * Craswell et al. (2021) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021.  Overview of the TREC 2020 deep learning track.  _arXiv preprint arXiv:2102.07662_ (2021).
  * Craswell et al. (2020) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020.  Overview of the TREC 2019 deep learning track.  _arXiv preprint arXiv:2003.07820_ (2020).
  * Dai et al. (2022) Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B Hall, and Ming-Wei Chang. 2022\.  Promptagator: Few-shot dense retrieval from 8 examples.  _arXiv preprint arXiv:2209.11755_ (2022).
  * Elasticsearch (2023) BV Elasticsearch. 2023\.  Elasticsearch.  _software], version_ 6, 1 (2023).
  * Faggioli et al. (2023) Guglielmo Faggioli, Laura Dietz, Charles Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, et al. 2023\.  Perspectives on Large Language Models for Relevance Judgment.  _arXiv preprint arXiv:2304.09161_ (2023).
  * Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019\.  ELI5: Long form question answering.  _arXiv preprint arXiv:1907.09190_ (2019).
  * Goldstein et al. (2023) Josh A Goldstein, Girish Sastry, Micah Musser, Renee DiResta, Matthew Gentzel, and Katerina Sedova. 2023\.  Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations.  _arXiv preprint arXiv:2301.04246_ (2023).
  * Guo et al. (2023) Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023\.  How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection.  _arXiv preprint arXiv:2301.07597_ (2023).
  * Hofstätter et al. (2020) Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020\.  Improving efficient neural ranking models with cross-architecture knowledge distillation.  _arXiv preprint arXiv:2010.02666_ (2020).
  * Jeronymo et al. (2023) Vitor Jeronymo, Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, Roberto Lotufo, Jakub Zavrel, and Rodrigo Nogueira. 2023.  InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrieval.  _arXiv preprint arXiv:2301.01820_ (2023).
  * Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020\.  TinyBERT: Distilling BERT for Natural Language Understanding. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 4163–4174.  https://doi.org/10.18653/v1/2020.findings-emnlp.372
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  Colbert: Efficient and effective passage search via contextualized late interaction over bert. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_. 39–48.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Lin et al. (2021) Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021\.  Pretrained transformers for text ranking: Bert and beyond.  _Synthesis Lectures on Human Language Technologies_ 14, 4 (2021), 1–325.
  * Llordes et al. (2023) Michael Llordes, Debasis Ganguly, Sumit Bhatia, and Chirag Agarwal. 2023.  Explain like I am BM25: Interpreting a Dense Model’s Ranked-List with a Sparse Approximation.  _arXiv preprint arXiv:2304.12631_ (2023).
  * MacAvaney et al. (2020) Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. 2020.  Expansion via prediction of importance with contextualization. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_. 1573–1576.
  * Maia et al. (2018) Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018.  Www’18 open challenge: financial opinion mining and question answering. In _Companion proceedings of the the web conference 2018_. 1941–1942.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated machine reading comprehension dataset. In _CoCo@ NIPs_.
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020.  Document ranking with a pretrained sequence-to-sequence model.  _arXiv preprint arXiv:2003.06713_ (2020).
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017\.  Automatic differentiation in pytorch.  (2017).
  * Peng et al. (2023) Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023\.  Towards Making the Most of ChatGPT for Machine Translation.  _arXiv preprint arXiv:2303.13780_ (2023).
  * Robertson and Walker (1994) Stephen E Robertson and Steve Walker. 1994.  Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In _SIGIR’94_. Springer, 232–241.
  * Sallam et al. (2023) Malik Sallam, Nesreen Salim, Muna Barakat, and Alaa Al-Tammemi. 2023.  ChatGPT applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limitations.  _Narra J_ 3, 1 (2023), e103–e103.
  * Sanderson and Croft (2012) Mark Sanderson and W Bruce Croft. 2012.  The history of information retrieval research.  _Proc. IEEE_ 100, Special Centennial Issue (2012), 1444–1451.
  * Sun et al. (2023) Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023\.  Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent.  _arXiv preprint arXiv:2304.09542_ (2023).
  * Sun (2023) Zhongxiang Sun. 2023\.  A short survey of viewing large language models in legal aspect.  _arXiv preprint arXiv:2303.09136_ (2023).
  * Susnjak (2023) Teo Susnjak. 2023\.  Applying BERT and ChatGPT for Sentiment Analysis of Lyme Disease in Scientific Literature.  _arXiv preprint arXiv:2302.06474_ (2023).
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wang et al. (2023b) Liang Wang, Nan Yang, and Furu Wei. 2023b.  Query2doc: Query Expansion with Large Language Models.  arXiv:2303.07678 [cs.IR]
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  _Advances in Neural Information Processing Systems_ 33 (2020), 5776–5788.
  * Wang et al. (2023a) Zengzhi Wang, Qiming Xie, Zixiang Ding, Yi Feng, and Rui Xia. 2023a.  Is ChatGPT a Good Sentiment Analyzer? A Preliminary Study.  _arXiv preprint arXiv:2304.04339_ (2023).
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019\.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_ (2019).
  * Yang et al. (2015) Yi Yang, Wen-tau Yih, and Christopher Meek. 2015.  Wikiqa: A challenge dataset for open-domain question answering. In _Proceedings of the 2015 conference on empirical methods in natural language processing_. 2013–2018.
  * Zhang et al. (2022) Bowen Zhang, Daijun Ding, and Liwen Jing. 2022.  How would Stance Detection Techniques Evolve after the Launch of ChatGPT?  _arXiv preprint arXiv:2212.14548_ (2022).
  * Zhang et al. (2023) Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023.  Extractive Summarization via ChatGPT for Faithful Summary Generation.  _arXiv preprint arXiv:2304.04193_ (2023).
  * Zhang and Sabuncu (2018) Zhilu Zhang and Mert Sabuncu. 2018.  Generalized cross entropy loss for training deep neural networks with noisy labels.  _Advances in neural information processing systems_ 31 (2018).
  * Zhuang et al. (2021) Shengyao Zhuang, Hang Li, and Guido Zuccon. 2021.  Deep query likelihood model for information retrieval. In _European Conference on Information Retrieval_. Springer, 463–470.
  * Zhuang and Zuccon (2021) Shengyao Zhuang and Guido Zuccon. 2021.  TILDE: Term independent likelihood moDEl for passage re-ranking. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1483–1492.


# Supplemental material for: Extraction of the interaction parameters for
$\alpha$-RuCl3 from neutron data using machine learning

Anjana M. Samarakoon  Shull-Wollan Center, Oak Ridge National Laboratory, Oak
Ridge, Tennessee 37831, USA  Neutron Scattering Division, Oak Ridge National
Laboratory, Oak Ridge, Tennessee 37831, USA  Materials Science Division,
Argonne National Laboratory, Lemont, Illinois 60439, USA  Pontus Laurell
Center for Nanophase Materials Sciences, Oak Ridge National Laboratory, Oak
Ridge, Tennessee 37831, USA  Computational Sciences and Engineering Division,
Oak Ridge National Laboratory, Oak Ridge, Tennessee 37831, USA  Department of
Physics and Astronomy, University of Tennessee, Knoxville, Tennessee 37996,
USA.  Christian Balz  Neutron Scattering Division, Oak Ridge National
Laboratory, Oak Ridge, Tennessee 37831, USA  ISIS Neutron and Muon Source,
Rutherford Appleton Laboratory, Didcot, OX11 0QX, United Kingdom  Arnab
Banerjee  Neutron Scattering Division, Oak Ridge National Laboratory, Oak
Ridge, Tennessee 37831, USA  Department of Physics and Astronomy, Purdue
University, West Lafayette, Indiana 47906, USA  Paula Lampen-Kelley
Department of Materials Science and Engineering, University of Tennessee,
Knoxville, Tennessee 37996, USA  Materials Science and Technology Division,
Oak Ridge National Laboratory, Oak Ridge, Tennessee 37831, USA  David Mandrus
Department of Materials Science and Engineering, University of Tennessee,
Knoxville, Tennessee 37996, USA  Materials Science and Technology Division,
Oak Ridge National Laboratory, Oak Ridge, Tennessee 37831, USA  Stephen E.
Nagler  Neutron Scattering Division, Oak Ridge National Laboratory, Oak Ridge,
Tennessee 37831, USA  Quantum Science Center, Oak Ridge National Laboratory,
Oak Ridge, Tennessee 37831, USA  Satoshi Okamoto  Materials Science and
Technology Division, Oak Ridge National Laboratory, Oak Ridge, Tennessee
37831, USA  Quantum Science Center, Oak Ridge National Laboratory, Oak Ridge,
Tennessee 37831, USA  D. Alan Tennant  Materials Science and Technology
Division, Oak Ridge National Laboratory, Oak Ridge, Tennessee 37831, USA
Quantum Science Center, Oak Ridge National Laboratory, Oak Ridge, Tennessee
37831, USA  Shull-Wollan Center, Oak Ridge National Laboratory, Oak Ridge,
Tennessee 37831, USA  Department of Physics and Astronomy, University of
Tennessee, Knoxville, Tennessee 37996, USA.

###### Abstract

This PDF file includes:

  * •

Details on numerical methods and training of networks

  * •

More information on Neutron data analysis and treatments

  * •

Real-space spin configurations

  * •

Phase map generation and SSF predictions using machine-learning

  * •

Additional predictions of dynamical structure factor

  * •

Details of ED calculations and comparisons with MC results

  * •

Formulation of the optimal region

  * •

Comparison of machine-learned parameter set with some of the previously
reported values

  * •

Magnetic specific heat

Supplementary Figures 1-13
Supplementary Tables I-III

##  I Details on numerical methods and training of networks

###  I.1 Monte Carlo and Landau-Lifshitz Solver for scattering

Neutron scattering from the magnetic system directly probes the magnetic two-
point correlations of the material and the scattering cross section can be
formulated as:

$\frac{d^{2}\sigma}{d\hbar\omega\Omega}=\frac{k_{f}}{k_{i}}r_{m}^{2}\sum_{\alpha,\beta}\frac{g_{\alpha}g_{\beta}}{4}\left(\delta_{\alpha\beta}-\frac{q_{\alpha}q_{\beta}}{q^{2}}\right)|F(\textbf{Q})|^{2}\mathcal{S}^{\alpha\beta}\left(\textbf{Q},\omega\right)$
(S1)

where Q and $\omega$ are the wavevector and energy transfer in the scattering
process, $k_{i}$ and $k_{f}$ are the initial and final wavevectors of the
neutrons, $r_{m}$ is a scattering factor, $\alpha,\beta=x,y,z$ are Cartesian
coordinates indicating initial and final spin polarization of the neutron,
$F(\textbf{Q})$ is the magnetic form factor and
$\mathcal{S}^{\alpha\beta}\left(\textbf{Q},\omega\right)$ is the spin
correlation function. The pre-factors are set by measurement conditions or
atomic properties of the material and so are straightforwardly evaluated
allowing quantitative experimental determination of
$\mathcal{S}\left(\textbf{Q},\omega\right)$.

The dynamical correlation function
$\mathcal{S}^{\alpha\beta}(\textbf{Q},\omega)$ is equal to the Fourier
transform of the spin-spin correlation functions in space and time:

$\mathcal{S}^{\alpha\beta}(\textbf{Q},\omega)=\frac{1}{2\pi
N}\sum_{i,j}e^{i\textbf{Q}.\left(\textbf{R}_{j}-\textbf{R}_{i}\right)}\int_{-\infty}^{\infty}e^{-i\omega
t}\langle S_{i}^{\alpha}(t_{0})S_{j}^{\beta}(t_{0}+t)\rangle dt.$ (S2)

Figure S1:  The nuclear structure of RuCl3 as reported in Refs. [1, 2]. The
space group is R$\bar{3}$ (148). Note that there are 6 Ru-ions per unit-cell
corresponding to three stacked honeycomb layers. We used this structure in our
MC/LLD simulations even though the Hamiltonian of this study does not contain
out-of-plane exchanges. Thus, our simulation setup can be readily generalized
to 3D models with inter-layer exchange interactions.

The Monte Carlo solver used in Ref. [3] is deployed for a network of
Heisenberg spins on a honeycomb lattice. Although we neglect out-of-plane
exchanges in this study, the unit cell with three honeycomb layers shown in
Fig. S1 was used throughout the simulations. The spins are treated as vectors
$\mathbf{S}_{i}=[S_{i}^{x},S_{i}^{y},S_{i}^{z}]$ of fixed length of
$\sqrt{S(S+1)}(=\sqrt{3}/2)$ at positions $\mathbf{R}_{i}$. The energy due to
interactions, $\\{p\\}$ is given by the spin Hamiltonian,
$\mathcal{H}=\mathcal{H}(\\{p\\},\mathbf{S},\mathbf{R})$ given in Eq. (1) of
the main text. Spin configurations which are representative of the system in
thermal equilibrium at chosen temperature $T$ are found using simulated
annealing based on the Metropolis algorithm, a Markov Chain Monte Carlo method
[4]. From these configurations a full range of physical properties can be
calculated including $S^{\rm sim}(\textbf{Q})$. The time development of a
thermally equilibrated spin configuration, $\mathbf{S}(t)$ can be evaluated
using the Landau-Lifshitz (LL) equations of motion which are readily derived
through Poisson brackets involving $\mathbf{S}_{i}$ and $\mathcal{H}$.

$\frac{d\mathbf{S}_{i}}{dt}=\gamma\left[S_{i}\times\frac{\partial\mathcal{H}}{\partial\mathbf{S}_{i}}\right]$
(S3)

The LL equation is solved numerically using a fourth-order Runge-Kutta
algorithm with adaptive step size [5]. The $\mathbf{S}(t)$ is calculated for
an appropriately spaced set of discrete times from $t_{0}$ to $t_{N}$ with
spacing $\Delta t$ and used to calculate
$\mathcal{S}^{\alpha\beta}(\textbf{Q},\omega)$. The $t_{N}$ is determined by
the target energy resolution of 0.5 meV. For a given $\\{p\\}$, the
$S(\textbf{Q},\omega)$ was calculated from 256 independent simulations and
averaged to yield good statistics. The actual measured cross section
$\mathcal{S}^{\rm exp}\left(\nu\right)$ depends on experimental conditions
including resolution and the MC/LLD solver then undertakes the transformation
$\\{\mathcal{H}(p)\\}\rightarrow\mathcal{S}^{\rm sim}(\textbf{Q},\\{p\\})$ to
replicate expected scattering.

###  I.2 Parameterization of the Hamiltonian and optimization

The original Hamiltonian space is re-parameterized as follows to cover the
entire parameter-space (both positive and negative sides of each parameter):

$\begin{split}J_{1}&=\left(\frac{U}{3}\right){\rm sin}(\phi){\rm
sin}(\theta){\rm sin}(\beta){\rm sin}(\gamma),\\\ K&=\left(U\right){\rm
cos}(\phi){\rm sin}(\theta){\rm sin}(\beta){\rm sin}(\gamma),\\\
\Gamma&=\left(\frac{U}{2}\right){\rm cos}(\theta){\rm sin}(\beta){\rm
sin}(\gamma),\\\ J_{2}&=\left(\frac{U}{6}\right){\rm cos}(\beta){\rm
sin}(\gamma),\\\ J_{3}&=\left(\frac{U}{3}\right){\rm cos}(\gamma),\end{split}$
(S4)

where $U$ is the overall energy scale. The hyper-parameter space of
$\\{p\\}=\\{\phi,\theta,\beta,\gamma\\}$ was explored with fixed $U=$ 10 meV,
using the iterative mapping algorithm (IMA), which is a variant of the
Efficient Global Optimization algorithm as described in Ref. [3]. The IMA is
employed to minimize the multi-experiment error measure,
${\hat{\chi}}^{2}_{\rm
com}={\hat{\chi}}^{2}_{S(\textbf{Q})}\times{\hat{\chi}}^{2}_{S(\textbf{Q},\omega)}$.
The ${\hat{\chi}}^{2}_{S(\textbf{Q})}$ and
${\hat{\chi}}^{2}_{S(\textbf{Q},\omega)}$ are the low-cost estimators of the
mean squared error, $\chi^{2}_{S(\nu)}=\sum_{\nu}[S^{\rm exp}(\nu)-S^{\rm
sim}(\nu)]^{2}$ with respect to static and dynamical structure factors. In
contrast to the Gaussian process regression in Ref. [3], we here calculate a
low cost estimators using the surrogate networks to determine the squared
distance between prediction and experimental data, (Eq. S5).

$\begin{split}{\hat{\chi}}^{2}_{S(\textbf{Q})}&=\sum_{\textbf{Q}}[S^{\rm
exp}(\textbf{Q})-S^{\rm sur}(\textbf{Q})]^{2}\\\
{\hat{\chi}}^{2}_{S(\textbf{Q},\omega)}&=\sum_{\\{\textbf{Q},\omega\\}}[S^{\rm
exp}(\textbf{Q},\omega)-S^{\rm
sur}(\textbf{Q},\epsilon\omega)]^{2}\end{split}$ (S5)

Here, $\epsilon$ renormalizes the energy scale $U$ and is found by minimizing
${\hat{\chi}}^{2}_{S(\textbf{Q},\omega)}$ within the interval of
$\epsilon\in[1/3,3]$ for a given Hamiltonian parameter set $\\{p\\}$. The
$\epsilon$ is necessary to account for the overall energy scale.

The IMA iteratively samples parameter space subject to the condition that
${\hat{\chi}}^{2}_{\rm com}$ is below the error tolerance threshold, $C_{\rm
com}$, and the surrogates are retrained. As more data is collected, the
prediction accuracy of the surrogates towards minimum of the
${\hat{\chi}}^{2}_{\rm com}$ becomes reliable.

###  I.3 Training of Autoencoders

In this work, we have trained two Autoencoders; for static and dynamic
structure factors. The training includes $S^{\rm Sim}(\textbf{Q})$ and $S^{\rm
Sim}(\textbf{Q},\omega)$ generated from the MC/LLD solver for random sets of
Hamiltonian parameter produced by the IMA. By the last iteration of IMA, we
have employed 10000 models for Hamiltonian Eq. (1). For each model, both
$S^{\rm Sim}(\textbf{Q})$ and $S^{\rm Sim}(\textbf{Q},\omega)$ are calculated.
The training data includes simulated data of 90$\%$ of randomly selected
existing samples and the remaining data was used as ML training validation.
The autoencoder tries to minimize the deviation between its input $S^{\rm
sim}(\textbf{Q})$ and filtered output, summed over all random models in the
dataset.

The Autoencoder corresponding to $S(\textbf{Q})$, is constructed with three
hidden layers of logistic neurons and latent space of 3-dimensions. The number
of neurons for the three layers are empirically found to be 300-3-300. A
single hidden layer autoencoder architecture of 200 linear neurons is found to
be performing reasonably well with $S(\textbf{Q},\omega)$ data.

Training the autoencoder corresponds to determining the weight matrix $(W)$
and bias $(b)$ for each layer including the output layer. A loss function,
$\mathcal{L}$

$\displaystyle\mathcal{L}$ $\displaystyle=$
$\displaystyle\sum_{\\{p\\}}\left[\frac{1}{N_{\nu}}\sum_{\nu}(S^{\rm
sim}(\nu)-S^{\rm sim}_{\rm AE}(\nu))^{2}+\mathcal{L}_{R}\right]$ (S6)

is minimized using the Adam optimization algorithm as readily accessible in
Keras [6], a deep learning API written in Python. The term $\mathcal{L}_{R}$
is relatively weak, and includes two types of regularization: An $L_{2}$
regularization on the weight matrices, and an activity regularizer for
uncorrelated feature constraint. This regularization seems to improve the
physical interpretability of the latent space representation. Despite having
millions of trainable parameters in the neural network, the autoencoder does
not seem prone to overfitting; the low-dimensionality of the latent space
itself acts as a strong regularizer.

###  I.4 Training of the generative model

The RBN is constructed with two layers: A layer of radial basis (RB) neurons
followed by an output layer of logistic neurons. The latent space predictions
for a given parameter set $\\{p\\}$ are defined as:

$\displaystyle\mathcal{L}_{i}$ $\displaystyle=$ $\displaystyle
f_{2}(w^{(2)}_{ij}.h_{j}(\\{p\\})+b_{2})$ (S7) $\displaystyle
h_{j}(\\{p\\})=exp\left[-\frac{\sum_{k}(p_{k}-c_{jk})^{2}}{\sigma^{2}}\right]$
(S8)

where $f_{2}$ is the logistic activation function $f(x)=1/(1+e^{-x})$ similar
to the output layer of the NLAE Encoder. The weight matrix $w^{(2)}_{ij}$, and
bias vector $b_{2}$ of the output layer and the clustering centers $c_{jk}$ of
the RB layer are to be determined from the training process. The spread of RB
function, $\sigma$ is preset to 0.05. The network is trained with the $S^{\rm
sim}(L)$ as the target and the corresponding $\\{p\\}$ as the input. Thus the
input and the output dimensionality is set by the dimensionality of the
$\\{H(p)\\}$ and the $\mathcal{L}_{Q}$. The number of neutrons in the RB layer
is determined during the training process. The training starts with no neurons
in the hidden layer and iteratively adding neurons to minimize the error
between output and the target.

##  II More information on Neutron data analysis and treatments

Fig. S2(a) shows a 2D slice of 4D raw data from the inelastic neutron
experiment executed at the Fine-Resolution Fermi Chopper Spectrometer
(SEQUOIA), SNS, ORNL, under temperature of $\sim$ 4 K and zero field
conditions. The level of experimental noise is comparable to the magnetic
signal even though we have integrated over a wide range of $l=0\pm 3.5$ . We
therefore perform a 3D Gaussian filtering on the $l$-integrated 3D data volume
as a pre-processing step to the ML integrated workflow. As shown in Fig.
S2(b), the Gaussian filter helps to get rid of statistical noise up to some
extent and makes the spectrum more clearer. A Gaussian kernel with standard
deviation ($\sigma$) of 1 was used here. A lower value of $\sigma$ will not do
better on statistical noise while a higher value would make the features
broader than instrument resolution.

Figure S2:  Pre-processing treatment on SEQUOIA data. (a) the raw data sliced
in $[-h,h,0]$ and $\hbar\omega$ plane with the integration of$[k,k,0]:k=0\pm
0.02$, $[0,0,l]:l=0\pm 3.5$ and (b) the corresponding slice of smoothed data
are shown here. The data was collected at 4 K with a single crystal of mass
0.7 g. The data was smoothed by filtering $l$-integrated 3D image with a 3D
Gaussian smoothing kernel with standard deviation of 1.0. The Gaussian filter
helps to get rid of statistical noise and sharpen up the features in the raw
data.

##  III Real-space spin configurations

Fig. S3 compares the real-space spin configurations of the 3D and planar
zigzag phases. Fig. S4 shows spin configurations for some other phases
identified in Fig. 2.

Figure S3:  Magnetic structures of the two zig-zag (Z.Z.) phases mentioned in
the main text. Panel (a), (b) and (c) show the three of the six magnetic
domains for the 3D Z.Z. phase. Panel (d), (e) and (f) are the three of the six
magnetic domains for the 2D Z.Z. phase. The yellow spheres and red arrows
represents the Ru-ions and its magnetic moments respectively. The green plane
cuts through the vertical center of the lattice for better perspective on the
out of plane spin components. Both phases can be found in parameter space
plane shown in Fig. 2(a). $\Gamma>0$ favors 3D Z.Z., while $\Gamma<0$ favors
the 2D Z.Z. phase. The magnetic configuration of both phases consist of zig-
zag chains, but the spin orientations different. The directions of zig-zag
chains in 3D Z.Z. phase aligns with the principal axis of the anisotropic
models while it is in-plane and perpendicular to the propagation vector of the
chains. The corresponding labels to map the $S(\textbf{Q})$ shown in Fig. 2
are also displayed. Due to the difference in spin orientations, neutron
structure factor is also distinguishable.  Figure S4:  Magnetic structure of
some of the phases shown in Fig. 2. The spin configuration for the
$S(\textbf{Q})$ labeled as (a) 6 , (b) 3, (c) 7 and (d) 4 in Fig. 2 are shown
here. The yellow spheres and red arrows represents the Ru-ions and its
magnetic moments respectively. The green plane cuts through the vertical
center of the lattice for better perspective on the out of plane spin
components. The corresponding phases for label 6, 3 and 7 can be easily
identify as anti-ferromagnetic, ferromagnetic and 120° orders respectively.
The spin-structure corresponding to the label 4 [panel (d)] is planar with
“6-in” type plaquettes forming a triangular network.

##  IV Phase map generation and SSF predictions using machine-learning

As explained in the main text, an Autoencoder with latent space of $N_{L}=3$
is trained to compress $S(\textbf{Q})$ data and used to predict phase maps of
magnetic ordering (see Fig. S5(a)). The $S(\textbf{Q})$ (and consequently
$S(L)$) encodes correlations of the systems and provides natural
classification with an adequate size of reciprocal space. Parameter sets
corresponding to the same structure would cluster together in either Q or $L$
space. (see Fig. S5(b)). Thus, it is technically possible to perform a
clustering analysis to identify and label different clusters. However, such
clustering analysis would fail when the system undergoes continuous
transitions or crossovers rather than abrupt first-order like changes. In this
case it is still possible to easily construct a graphical phase diagram. If
the Q space can be reduced to an $L$ space with $N_{L}=3$, then the latent
vectors can be treated as the RGB color components of a phase map.

Figure S5:  Latent space representations and generative model predictability.
(a) A predicted map of magnetic orderings in the $J_{1}-K-\Gamma$ space on the
hyper-surface of $J_{1}^{2}+K^{2}+\Gamma^{2}=1$ at a fixed temperature of
$T/|K|=0.02$. The phase map is generated for $\rm\Gamma>0$ and depicted
according to $\rm\Gamma=cos(\theta)$. (b) 3D histogram (density map) of the
$\\{p\\}$ manifold in latent space. The surfaces are the 3D contours of the
histogram and their color represents the bin-height (density) used to generate
the contour. In this latent space representation, a broad phase in parameter
space appears as a dense point and a higher order phase transition / crossover
appears as lines of points connecting dense clusters. First-order phase
transitions appear as isolated clusters and discontinuities in the latent
space projection [7]. (c) $S(L)$ for (filled triangles) the simulated
structure factors calculated along the white dashed line shown in panel (a) at
$\theta=\pi/4$ along with (solid lines) the prediction from the GN. (d) The
high-symmetry-plane slices of simulated and surrogate-predicted
$S(\textbf{Q})$ data at multiple places in parameter space as indexed on panel
(c).

In order to construct high-dimensional phase maps, we used a Generator network
(see Fig. 3(b)) to predict $S_{\rm GN}(L)$ for the parameters set located in
between existing $\\{p\\}$samples. Figure S5(a) is generated in such fashion
in the hyper plane of $\sqrt{J_{1}^{2}+K^{2}+\Gamma^{2}}=1$ and
$J_{2}=J_{3}=0$ and Figure S5(c) shows the $S_{\rm GN}(L)$ in comparison to
$S(L)$ not included in GN training, along the dashed circle in panel (a).

As shown in Fig. 3 (b) and (c), a faster surrogate for the MC/LLD solver can
be constructed by connecting the GN with the Decoder network of the
Autoencoder. Figure S5 (d) shows the surrogate-predicted structure factors,
$S^{\rm sur}(\textbf{Q})$ in comparison to the simulated for the parameter
sets, not used as a part of training data. The surrogate-predictions are worse
in some regions where the the $S^{\rm sim}(\textbf{Q})$ change rapidly and the
$\\{p\\}$ samples are taken sparsely. The predictability of both Decoder and
the Generator contribute the total prediction error of the surrogate. As more
$\\{p\\}$ are evaluated on demand in the regions of interest, the surrogate
predictions will be more reliable. Even though the surrogate predictions are
not accurate in some regions, it can be used to do guide parameter space
searches and even draw quick conclusions for experiment planning. However, any
prediction leading to critical decisions or conclusion, should always be
validated with direct calculations.

##  V Additional predictions of dynamical structure factor

Fig. S6 shows predicted dynamical structure factors for the parameter sets
specified in Table 1.

Figure S6:  The dynamical structure factors for multiple parameter sets. All the panels shows $[-h,h,0]-\hbar\omega$ slices of $S^{\rm Sur}(\textbf{Q},\omega)$ at $[k,k,0]:k=0$, $[0,0,l]:l=0$ for the parameter sets listed in the Table 1. The ratios between parameter for the $S^{\rm Sur}(\textbf{Q},\omega)$ shown in panel (a)-(e) also match for the sets labeled as 1-5 in the Fig. 2(a).  Set $\\#$ |  $J_{1}$ (meV) |  $K$ (meV) |  $\Gamma$ (meV) |  $J_{2}$ (meV) |  $J_{3}$ (meV)
---|---|---|---|---|---
1 |  -0.28 |  -1.39 |  0.87 |  0 |  1.75
2 |  -0.47 |  -2.33 |  1.89 |  -0.07 |  0.04
3 |  -0.21 |  -2.50 |  0.99 |  0.08 |  -1.58
4 |  0.12 |  -3.40 |  -2.23 |  0.01 |  0.43
5 |  -0.28 |  -1.74 |  -0.83 |  -0.04 |  1.71
6 |  -0.04 |  -1.66 |  0.99 |  0.04 |  0.11
7 |  -0.34 |  -7.72 |  2.34 |  -0.1 |  0.74
8 |  -0.52 |  -5.02 |  1.46 |  -0.12 |  0.74
9 |  -0.84 |  -4.45 |  0.92 |  -0.08 |  1.05
10 |  -0.18 |  -5.89 |  0.15 |  -0.23 |  1.45
Table 1: The corresponding parameter sets for the dynamical structure factors
shown in Fig. S6.

##  VI Details of ED calculations and comparisons with MC results

Lanczos exact diagonalization calculations were performed at zero temperature
using the $\mathcal{H}\Phi$ library [8] on the $24$-site $C_{3}$-symmetric
finite size cluster shown in Fig. 2. It has the same rotational symmetry as
the full honeycomb lattice. Static structure factors were evaluated directly
as ground state expectation values, whereas dynamical structure factors were
calculated using the continued fraction expansion method (CFE) [9]. As in Ref.
[10] we used 500 Lanczos steps in the CFE. We used a Lorentzian broadening
$\eta$ of $0.05$ meV (half width at half maximum) for the optimized
parameters, or $0.05/sqrt(3)$ meV for parameter sets 1-4 defined in Table 2
and represented visually in Fig. S8. The broadening was intentionally made
small to allow experimental resolution effects to be implemented as in the LLD
calculations. The frequency step size was $\Delta\omega=\eta/5$.

Comparisons of $S(\mathbf{Q})$ and $S(\mathbf{Q},\omega)$ are shown in Fig.
S9. A comparison of the individual matrix elements, $\langle
S^{\alpha}S^{\beta}\rangle$ is also shown in Fig. S10, S11 and S12 for
parameter sets 1,2,4 listed in Table 2.

Figure S7:  Finite size cluster with periodic boundary conditions used for Lanczos ED calculations. Sides with matching symbols are identified. This figure is reproduced without changes from Ref. [10]; licensed under a Creative Commons Attribution (CC BY) license 222http://creativecommons.org/licenses/by/4.0/.  Set $\\#$ |  $J_{1}$ (meV) |  $K$ (meV) |  $\Gamma$ (meV) |  $J_{2}$ (meV) |  $J_{3}$ (meV)
---|---|---|---|---|---
1 |  -2.1 |  -13.6 |  -1.0 |  -0.2 |  1.7
2 |  -2.5 |  -13.3 |  0.6 |  -0.2 |  1.6
3 |  -2.0 |  -13.8 |  -1.1 |  -0.2 |  1.6
4 |  -2.5 |  -13.9 |  -0.7 |  -0.1 |  0.6
a |  -1.98 |  -11.29 |  0.49 |  -0.19 |  1.55
b |  -1.45 |  -9.29 |  0.38 |  -0.195 |  1.49
c |  -0.93 |  -7.28 |  0.26 |  -0.193 |  1.435
Opt. |  -0.4 |  -5.27 |  0.15 |  -0.19 |  1.38
Table 2: Parameter sets used to compare ED calculation with LLD.  Figure S8:
The parameters sets listed in Table 2 are shown in the machine-learned phase
map varying $\Gamma/|K|$ and $J_{3}/|K|$ through the optimal solution for
$\alpha$-RuCl3 at fixed $J_{1}/|K|=-0.1$ and $J_{2}/|K|=0$, with $K<0$.
Parameter sets 1-4 were chosen to be located close to phase transitions, and
parameter sets a-c were chosen to lie on a line connecting the optimized
parameter set with parameter set 2.  Figure S9:  The validation of MC/LLD with
Lanczos ED for four different parameter sets listed in Table 2. The
comparisons of (a - d) static and (e - h) dynamic structure factors from (left
side) Monte-Carlo simulations/LLD and (right side) ED are shown. The ED
calculations use a 24-site cluster as shown in Fig. 2 and structure factors
are calculated up to the 2nd Brillouin Zone of the honeycomb lattice. The
MC/LLD structure factors are calculated on a super cell of $20\times 20$ RuCl3
nuclear unit cells (equivalent to 2400 sites). Even though the the
Q-resolutions of two calculations are not comparable, the magnetic peak
structure in $S(\textbf{Q})$ agrees well for sets (b)2 and (d)4 but not for
set (a)1 or (c)3. According to the MC/LLD calculations, the corresponding
magnetic structure for set 1 and 3 is 2D Z.Z.. For this phase, M-point peaks
in the 1st B.Z. are not allowed by the polarization factor, but M-point peaks
appear in higher Brillouin zones, which are not covered in the ED calculation.
Thus, the diffuse patterns in the $S_{\rm ED}(\textbf{Q})$ for set 1 and 3 are
due to the finite-size effect.

Figure S10:  The comparison of the individual SSF matrix elements, $\langle
S^{\alpha}S^{\beta}\rangle$ for the parameter set 1 listed in Table 2.

Figure S11:  The comparison of the individual SSF matrix elements, $\langle
S^{\alpha}S^{\beta}\rangle$ for the parameter set 2 listed in Table 2.

Figure S12:  The comparison of the individual SSF matrix elements, $\langle
S^{\alpha}S^{\beta}\rangle$ for the parameter set 4 listed in Table 2.

##  VII Formulation of the optimal region

The five-dimensional optimal region for which ${\hat{\chi}}_{\rm
Com}^{2}<C_{\rm Com}^{2}$ as illustrated in Fig. 4 (blue contours) can be
formulated by fitting the region to a minimum volume ellipsoid [11] as,

$\displaystyle V\times M\times V^{\dagger}\leq 1,$ (S9)

where,

$M=\begin{bmatrix}0.0108&0.0011&-0.0202&0.0062&0.0069\\\
0.0011&0.0004&0.0003&-0.0011&0.0007\\\
-0.0202&0.0003&0.1101&-0.0269&-0.0131\\\
0.0062&-0.0011&-0.0269&0.0508&0.0239\\\ 0.0069&0.0007&-0.0131&0.0239&0.0213\\\
\end{bmatrix},$

$V=\begin{bmatrix}J_{1}+0.4481&K+5.5136&\Gamma-0.1627&J_{2}+0.2228&J_{3}-1.4543\end{bmatrix},$

##  VIII Comparison of machine-learned parameter set with some of the
previously reported values

In Table 3 we show $\chi^{2}_{\mathrm{Com}}$ evaluated for some models in the
literature. By this metric the optimized parameter set obtained by our machine
learning method performs best with a value of $\chi^{2}_{\mathrm{Com}}\approx
5$ — but it is expected to do so by design. However, no other model produces
$\chi^{2}_{\mathrm{Com}}<10$, while three models are found to produce
$\chi^{2}_{\mathrm{Com}}$ values in the $[10,20]$ range. It is remarkable that
our optimized parameter set and the model of Ref. [12] have significantly
different $\chi^{2}_{\mathrm{Com}}$ values despite appearing close in terms of
interaction parameters—perhaps $J_{2}$ plays an important role? We stress that
our treatment here neglects other small terms — both in some of the models in
Table 3 and in determination of the fitness landscape. This may be too drastic
an approximation, as suggested particularly by models with larger
$\Gamma^{\prime}$ values such as in Ref. [13]. For this reason we suggest the
$\chi^{2}_{\mathrm{Com}}$ values be interpreted with care. In addition, our
SCS predicts a ferromagnetic ground state for the model of Ref. [14] rather
than zigzag, which likely contributes substantially to the high
$\chi^{2}_{\mathrm{Com}}$ value.

Reference |  $J_{1}$ |  $K$ |  $\Gamma$ |  $J_{2}$ |  $J_{3}$ |  $\hat{\chi}^{2}_{\rm Com}$
---|---|---|---|---|---|---
Optimal ML parameters (this work) |  -0.3961 |  -5.2731 |  0.15 |  -0.1935 |  1.3761 |  4.95
Hou et al. [15] |  -1.87 |  -10.7 |  3.8 |  0 |  1.27 |  13.78
Suzuki et al. [16] |  -3 |  -5 |  2.5 |  0 |  0.75 |  18.6
Winter et al. PRB [17] |  -5.5 |  7.6 |  8.4 |  0 |  2.3 |  19.58
Winter et al. NC [12] |  -0.5 |  -5 |  2.5 |  0 |  0.5 |  26.96
Wu et al. [18] |  -0.35 |  -2.8 |  2.4 |  0 |  0.34 |  28.70
Ozel et al. [19] |  -0.95 |  1.15 |  3.8 |  0 |  0 |  35.75
Kim and Kee [20] |  -3.5 |  4.6 |  6.42 |  0 |  0 |  36.27
Kim et al. [21, 22] |  -12 |  17 |  12 |  0 |  0 |  38.23
Li et al. [14] |  -2.5 |  -25 |  7.5 |  0 |  0 |  38.424
Banerjee et al. [23] |  -4.6 |  7 |  0 |  0 |  0 |  38.54
Laurell and Okamoto [10] |  -1.3 |  -15.1 |  10 |  0 |  0.9 |  38.65
Cookmeyer and Moore [24] |  -0.5 |  -5 |  2.5 |  0 |  0.1125 |  38.70
Ran et al. (2017) [25] |  0 |  -6.8 |  9.5 |  0 |  0 |  39.04
Suzuki and Suga [26, 27] |  -1.53 |  -24.4 |  5.25 |  0 |  0 |  40.03
Ozel et al. [19] |  0.46 |  -3.5 |  2.35 |  0 |  0 |  40.12
Ran et al. (2022) [28] |  0 |  -7.2 |  5.6 |  0 |  0 |  40.339
Wang et al. [29] |  -0.3 |  -10.9 |  6.1 |  0 |  0.03 |  40.65
Table 3: $\chi^{2}_{\mathrm{Com}}$ for different models approximately
described by Eq. (1) in the main text. For some of the models we have
neglected additional small terms not captured by our model, such as
$\Gamma^{\prime}$. Those models are those of Refs. [10, 17, 26, 15, 27, 20].
It is possible that inclusion of those terms and/or usage of parameters that
have not been bond-averaged would improve those models’
$\chi^{2}_{\mathrm{Com}}$ performance.

##  IX Magnetic specific heat

Magnetic specific heat for some models was calculated using the microcanonical
thermal pure quantum state (mTPQ) method [30] implemented in the
$\mathcal{H}\Phi$ library [8]. Similarly to the ED calculations, the $24$-site
cluster in Fig. 2 was used. To reduce statistical error, results were averaged
over 15 initial random vectors. The results are shown in Fig. S13. These
findings are not compared with SCS results since the specific heat of
classical and quantum models are fundamentally different. In future work one
may co-optimize parameters to fit both neutron data and other observables,
e.g. $C(T)$.

Figure S13: (a) Magnetic specific heat for some of the parameter sets listed
in Table 2 and (b) the corresponding $S(\mathbf{Q},\omega)$ are shown. In
panel (a), the black line shows experimentally determined
$C_{\mathrm{mag}}(T)$ from Ref. [31], which features a peak near $6.5$ K due
to onset of magnetic order and a high-temperature peak around $70$ K. The
sharp peak at $\approx 170$ K is not a feature of the specific heat, but an
artifact of the background subtraction due to a structural transition not
present in the nonmagnetic analog. Shown are also mTPQ calculations for five
parameters sets near the machine-learning-predicted optimal region. The other
lines indicates the statistical average over initial vectors. We see that our
optimized parameters (Opt. Set) do not capture the high-temperature peak well,
whereas Set 2 [see Table 2] performs better in this regard. This matches the
general trend noticed in Ref. [10] that the $C(T)$ behavior is largely
controlled by the overall energy scale of the spin Hamiltonian, whereas
$S(\mathbf{Q},\omega)$ depends crucially on the balance of different
interactions. Note that Sets a,b and c are linearly connected points between
Set 2 and Opt. Set in parameter space as shown in Fig. S8. Even though, Set 02
is doing well on $C(T)$, the corresponding $S(\mathbf{Q},\omega)$ does not
capture the dispersion at $M-$points as shown in panel (b). Evidently, it is
unlikely to find a single region to capture both experimental observations in
the current parameter space. However the two regions are not far away in
parameter space as shown in Fig. S8 and it might be possible to find a single
solution in a further extended parameter space with appropriate extra
exchanges.

## References

  * Glamazda  _et al._ [2017] A. Glamazda, P. Lemmens, S.-H. Do, Y. S. Kwon, and K.-Y. Choi, Phys. Rev. B 95, 174429 (2017).
  * Mu  _et al._ [2022] S. Mu, K. D. Dixit, X. Wang, D. L. Abernathy, H. Cao, S. E. Nagler, J. Yan, P. Lampen-Kelley, D. Mandrus, C. A. Polanco, L. Liang, G. B. Halász, Y. Cheng, A. Banerjee, and T. Berlijn, Phys. Rev. Research 4, 013067 (2022).
  * Samarakoon  _et al._ [2020] A. M. Samarakoon, K. Barros, Y. W. Li, M. Eisenbach, Q. Zhang, F. Ye, V. Sharma, Z. L. Dun, H. Zhou, S. A. Grigera, C. D. Batista, and D. A. Tennant, Nat. Commun. 11, 892 (2020).
  * Metropolis  _et al._ [1953] N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller, J. Chem. Phys. 21, 1087 (1953).
  * Huberman  _et al._ [2008] T. Huberman, D. A. Tennant, R. A. Cowley, R. Coldea, and C. D. Frost, J. Stat. Mech.: Theory Exp. 2008 (05), P05017.
  * Chollet  _et al._ [2015] F. Chollet _et al._ , Keras (2015), github.com/keras-team/keras.
  * Tennant and Samarakoon [2021] A. Tennant and A. Samarakoon, J. Phys. Condens. Matter 34, 044002 (2021).
  * Kawamura  _et al._ [2017] M. Kawamura, K. Yoshimi, T. Misawa, Y. Yamaji, S. Todo, and N. Kawashima, Comp. Phys. Comms. 217, 180 (2017).
  * Dagotto [1994] E. Dagotto, Rev. Mod. Phys. 66, 763 (1994).
  * Laurell and Okamoto [2020] P. Laurell and S. Okamoto, npj Quantum Mater. 5, 2 (2020).
  * Moshtagh  _et al._ [2005] N. Moshtagh _et al._ , Convex optimization 111, 1 (2005).
  * Winter  _et al._ [2017] S. M. Winter, K. Riedl, P. A. Maksimov, A. L. Chernyshev, A. Honecker, and R. Valentí, Nat. Commun. 8, 1152 (2017).
  * Maksimov and Chernyshev [2020] P. A. Maksimov and A. L. Chernyshev, Phys. Rev. Research 2, 033011 (2020).
  * Li  _et al._ [2021] H. Li, H.-K. Zhang, J. Wang, H.-Q. Wu, Y. Gao, D.-W. Qu, Z.-X. Liu, S.-S. Gong, and W. Li, Nat. Commun. 12, 4007 (2021).
  * Hou  _et al._ [2017] Y. S. Hou, H. J. Xiang, and X. G. Gong, Phys. Rev. B 96, 054410 (2017).
  * Suzuki  _et al._ [2021] H. Suzuki, H. Liu, J. Bertinshaw, K. Ueda, H. Kim, S. Laha, D. Weber, Z. Yang, L. Wang, K. F. H. Takahash and, M. Minola, B. V. Lotsch, B. J. Kim, H. Yavaş, M. Daghofer, J. Chaloupka, G. Khaliullin, H. Gretarsson, and B. Keimer, Nat. Commun. 12, 4512 (2021).
  * Winter  _et al._ [2016] S. M. Winter, Y. Li, H. O. Jeschke, and R. Valentí, Phys. Rev. B 93, 214431 (2016).
  * Wu  _et al._ [2018] L. Wu, A. Little, E. E. Aldape, D. Rees, E. Thewalt, P. Lampen-Kelley, A. Banerjee, C. A. Bridges, J.-Q. Yan, D. Boone, S. Patankar, D. Goldhaber-Gordon, D. Mandrus, S. E. Nagler, E. Altman, and J. Orenstein, Phys. Rev. B 98, 094425 (2018).
  * Ozel  _et al._ [2019] I. O. Ozel, C. A. Belvin, E. Baldini, I. Kimchi, S. Do, K.-Y. Choi, and N. Gedik, Phys. Rev. B 100, 085108 (2019).
  * Kim and Kee [2016] H.-S. Kim and H.-Y. Kee, Phys. Rev. B 93, 155143 (2016).
  * Kim  _et al._ [2015] H.-S. Kim, V. S. V., A. Catuneanu, and H.-Y. Kee, Phys. Rev. B 91, 241110 (2015).
  * Janssen  _et al._ [2017] L. Janssen, E. C. Andrade, and M. Vojta, Phys. Rev. B 96, 064430 (2017).
  * Banerjee  _et al._ [2016] A. Banerjee, C. A. Bridges, J.-Q. Yan, A. A. Aczel, L. Li, M. B. Stone, G. E. Granroth, M. D. Lumsden, Y. Yiu, J. Knolle, S. Bhattacharjee, D. L. Kovrizhin, R. Moessner, D. A. Tennant, D. G. Mandrus, and S. E. Nagler, Nat. Mat. 15, 733 (2016).
  * Cookmeyer and Moore [2018] J. Cookmeyer and J. E. Moore, Phys. Rev. B 98, 060412 (2018).
  * Ran  _et al._ [2017] K. Ran, J. Wang, W. Wang, Z.-Y. Dong, X. Ren, S. Bao, S. Li, Z. Ma, Y. Gan, Y. Zhang, J. T. Park, G. Deng, S. Danilkin, S.-L. Yu, J.-X. Li, and J. Wen, Phys. Rev. Lett. 118, 107203 (2017).
  * Suzuki and Suga [2018] T. Suzuki and S.-i. Suga, Phys. Rev. B 97, 134424 (2018).
  * Suzuki and Suga [2019] T. Suzuki and S.-i. Suga, Phys. Rev. B 99, 249902 (2019).
  * Ran  _et al._ [2022] K. Ran, J. Wang, S. Bao, Z. Cai, Y. Shangguan, Z. Ma, W. Wang, Z.-Y. Dong, P. Čermák, A. Schneidewind, S. Meng, Z. Lu, S.-L. Yu, J.-X. Li, and J. Wen, Chin. Phys. Lett. 39, 027501 (2022).
  * Wang  _et al._ [2017] W. Wang, Z.-Y. Dong, S.-L. Yu, and J.-X. Li, Phys. Rev. B 96, 115103 (2017).
  * Sugiura and Shimizu [2012] S. Sugiura and A. Shimizu, Phys. Rev. Lett. 108, 240401 (2012).
  * Widmann  _et al._ [2019] S. Widmann, V. Tsurkan, D. A. Prishchenko, V. G. Mazurenko, A. A. Tsirlin, and A. Loidl, Phys. Rev. B 99, 094415 (2019).


# Semi-Supervised Domain Adaptation for Cross-Survey Galaxy Morphology
Classification and Anomaly Detection

Aleksandra Ćiprijanović
Fermi National Accelerator Laboratory
Kirk Rd and Pine St
Batavia, IL 60510
<EMAIL_ADDRESS>
&Ashia Lewis
Fermi National Accelerator Laboratory
Kirk Rd and Pine St
Batavia, IL 60510
<EMAIL_ADDRESS>
Kevin Pedro
Fermi National Accelerator Laboratory
Kirk Rd and Pine St
Batavia, IL 60510
<EMAIL_ADDRESS>
&Sandeep Madireddy
Mathematics and Computer Science Division,
Argonne National Laboratory
Lemont, IL 60439, USA
<EMAIL_ADDRESS>
Brian Nord
Fermi National Accelerator Laboratory;
Kavli Institute for Cosmological Physics &
Department of Astronomy and Astrophysics,
University of Chicago;
Laboratory for Nuclear Physics, MIT
<EMAIL_ADDRESS>
&Gabriel N. Perdue
Fermi National Accelerator Laboratory
Kirk Rd and Pine St
Batavia, IL 60510
<EMAIL_ADDRESS>
Stefan M. Wild
Mathematics and Computer Science Division,
Argonne National Laboratory
Lemont, IL 60439, USA
<EMAIL_ADDRESS>

###### Abstract

In the era of big astronomical surveys, our ability to leverage artificial
intelligence algorithms simultaneously for multiple datasets will open new
avenues for scientific discovery. Unfortunately, simply training a deep neural
network on images from one data domain often leads to very poor performance on
any other dataset. Here we develop a Universal Domain Adaptation method
DeepAstroUDA, capable of performing semi-supervised domain alignment that can
be applied to datasets with different types of class overlap. Extra classes
can be present in any of the two datasets, and the method can even be used in
the presence of unknown classes. For the first time, we demonstrate the
successful use of domain adaptation on two very different observational
datasets (from SDSS and DECaLS). We show that our method is capable of
bridging the gap between two astronomical surveys, and also performs well for
anomaly detection and clustering of unknown data in the unlabeled dataset. We
apply our model to two examples of galaxy morphology classification tasks with
anomaly detection: 1) classifying spiral and elliptical galaxies with
detection of merging galaxies (three classes including one unknown anomaly
class); 2) a more granular problem where the classes describe more detailed
morphological properties of galaxies, with the detection of gravitational
lenses (ten classes including one unknown anomaly class).

##  1 Introduction

With big datasets from astronomical surveys like the Dark Energy Survey [DES;
5] or the Vera Rubin Legacy Survey of space and time [LSST; 9], development of
artificial intelligence (AI) algorithms capable of combining knowledge from
different telescopes will open doors to many new insights. Unfortunately,
standard deep learning algorithms are not well suited for work on multiple
datasets. Domain adaptation (DA) research includes the development of methods
designed to enable work on multiple datasets at the same time, by allowing the
model to learn and use only features present in both datasets [4, 18, 20],
thus aligning two latent data distributions. In astronomy, DA has first been
used in combination with active learning in [16], for Supernova Ia
classification and identification of Mars landforms. Then, in [2], authors use
DA for identifying merging galaxies in simulated Sloan Digital Sky Survey
[SDSS; 21] mock images and and real SDSS data. In [3], authors use DA for
galaxy morphology classification in simulated mock LSST data, and as a tool to
mitigate possible catastrophic perturbation-driven errors. Unfortunately, even
when two datasets are quite similar, like simulated mock images mimicking some
telescope and real observations from the same telescope, DA methods can be
quite hard to fine tune. Because of this, the development of DA methods that
can be used on multiple observational datasets exhibiting even larger
differences has not previously been attempted. Furthermore, when working with
real astronomical data, researchers will not always be able to work with fully
curated datasets; there may be partial overlap of classes, unknown classes or
anomalies in one or both domains, etc. Most standard DA methods try to align
the entire data distributions, so the presence of any kind of non-overlapping
classes will not allow those DA methods to be applied successfully.

In this work we develop a semi-supervised universal DA method DeepAstroUDA
that can handle the presence of non-overlapping classes in any of the two data
domains and can even detect and cluster unknown classes. As with all DA
methods, it requires two datasets: the source domain, which contains labeled
images, and the target domain, which can be unlabeled as the labels are not
used during model training. Our aim is the development of a universal DA
method that can be applied to a plethora of astronomical tasks and can
successfully perform DA on both simulated and observational data. We apply our
method to two observational datasets (available via the Galaxy Zoo project
[12, 11, 19]) and show that it is capable of successfully bridging even two
substantially different datasets and even discovering new unknown classes in
the unlabeled target domain. We focus on galaxy morphology classification with
anomaly detection: 1) classification of spiral and elliptical galaxies, with
the discovery of merging galaxies, and 2) more granular galaxy morphology
classification with sub-classes that more closely describe galaxy shapes
(based on attributes such as ellipticity, bulge prominence, presence of a bar
in spiral galaxies etc.) and detection of gravitationally lensed galaxies.

##  2 Methods

In this work, we introduce a semi-supervised universal DA method DeepAstroUDA.
Domain alignment and clustering of similar objects into classes is performed
via two loss functions: adaptive clustering and entropy separation111Code is
publicly available at: https://github.com/deepskies/DeepAstroUDA.

Adaptive Clustering (AC) Loss: The main idea of this type of semi-supervised
clustering [10]is to group unlabeled target domain samples into clusters by
computing pairwise similarities among their extracted features, then force the
class labels predicted by the classifier for samples with large pairwise
feature similarities to be consistent. For a pair of unlabeled target samples
$x_{1}$ and $x_{2}$ we predict a pairwise similarity label by using the output
prediction vectors $\mathbf{p}_{1}$ and $\mathbf{p}_{2}$ and rank ordering
their feature elements [7]. We then require the top $k$ elements ($k=3$ for
3-class problem and $k=7$ for 10-class problem, for computational efficiency)
to decide that the paired samples belong to the same class, which is denoted
by similarity label $s_{12}=1$; otherwise, $s_{12}=0$. For the labeled source
domain images, we use their class labels to generate similarity labels. We
also calculate a similarity score between samples as
$\mathbf{p}^{\top}_{1}\mathbf{p}_{2}$. Finally, we can write the AC loss as a
binary cross-entropy loss, where similarity labels are used as ground truth
labels:

${\cal L}_{\mathrm{AC}}=-\sum_{i\in B}\sum_{j\in
b_{t}}s_{ij}\mathrm{log}(\mathbf{p}^{\top}_{i}\mathbf{p}_{j})+(1-s_{ij})\mathrm{log}(1-\mathbf{p}^{\top}_{i}\mathbf{p}_{j}),$
(1)

where $B$ is the bank that contains samples from all previous source and
target batches, and $b_{t}$ is the current target batch [13]. By comparing
similarities between unlabeled target samples from the current target batch to
all elements stored in the bank, current target samples are pushed towards
source and target samples with which they share the most similar features.

Entropy Separation (ES) Loss: The objective of the ES loss is to align classes
present in both datasets, while pushing away the classes present only in one
of the domains [13]. This is possible because unknown samples often do not
share features with known samples, which leads to larger entropies for unknown
samples compared to entropies between shared classes [22]. Therefore, the
entropy can be used to decide the boundary between known and unknown samples.
If we denote the mean entropy of the classifier output $\mathbf{p}_{i}$ of
sample $i$ from the target batch $b_{t}$ as $H(\mathbf{p}_{i})$, we can define
a boundary $\rho$ around the entropy value so that ${\cal
L}_{\mathrm{ES}}(\mathbf{p}_{i})=-|H(\mathbf{p}_{i})-\rho|$ when
$|H(\mathbf{p}_{i})-\rho|>m$, and ${\cal L}_{\mathrm{ES}}=0$ otherwise. Here
$m$ is a confidence threshold around the boundary $\rho$, which is used to
decide if we are confident about whether a particular sample belongs in a
known or unknown class. Only those samples that are far enough from the
entropy boundary $\rho$ will be moved towards known classes or pushed away as
an unknown class. The boundary $\rho$ and confidence threshold $m$ start from
preset values (determined from experiments), but are actively fine-tuned
during training. Finally, the total ES loss is:

${\cal L}_{\mathrm{ES}}=\frac{1}{|b_{t}|}\sum_{i\in b_{t}}{\cal
L}_{\mathrm{ES}}(\mathbf{p}_{i}).$ (2)

Total Loss: The main classification loss for the labeled source domain is the
weighted cross-entropy (CE) loss. Combining this with the other terms
described above gives the total loss, which is the objective of the model
training: ${\cal L}={\cal L}_{CE}+\lambda({\cal L}_{\mathrm{AC}}+{\cal
L}_{\mathrm{ES}})$. The importance of the DA loss terms is governed by the
weight parameter $\lambda$. We find that $\lambda=0.005$ achieves the best
model performance on our datasets.

We use ResNet50 [8] network and train it with early stopping that monitors the
change in accuracy and stops the training when there is no improvement in 12
epochs. Domain-specific batch normalization is used to eliminates domain style
information leakage. The model is trained using stochastic gradient descent
with Nesterov momentum [14] and an initial learning rate of $0.001$ (with an
inverse learning rate scheduler, whereby the learning rate is decayed by a
factor of $0.1$ every 10 epochs). We train our models on 4 NVIDIA RTX A6000
GPUs (available from Google Colab and LambdaLabs), and on average the training
converges in ${\approx}5$ hours.

##  3 Data and Experiments

In this work we apply our method to 3-class and 10-class experiments. We use
two datasets from the Galaxy Zoo project [GZ; 12, 19], which used crowd
sourcing to create labels for millions of galaxies through a web-based
interface: source domain dataset from GZ2 SDSS [19] images, and target domain
dataset from GZ3 DECaLS222Current publicly available GZ datasets can be found
at https://data.galaxyzoo.org [17], that uses DESI Legacy Imaging Surveys [6].
Specifically, we use a subset of this dataset (with images that passed more
rigorous vote filtering) named Galaxy10 DECaLS333Galaxy10 data is available at
https://astronn.readthedocs.io/en/latest/galaxy10.html. For our 10-class
experiment, we use 9 classes from this dataset (disturbed, merging, round
smooth, cigar shaped smooth, barred spiral, unbarred tight spiral, unbarred
loose spiral, edge-on without bulge, edge-on with bulge) and add one more
class from the full GZ3 DECaLS dataset, gravitationally lensed galaxies, which
we will treat as an unknown class. We use the same labels for our source GZ2
SDSS dataset. Furthermore, we also test our method on a simple 3-class problem
with spiral, elliptical (known classes), and merging galaxies (unknown class).
These correspond, respectively, to the barred spiral, round smooth and merging
classes from the 10-class experiment. In Figure 1 we show example images from
both source and target domains. In this work, we demonstrate the performance
of our model on two Open DA problems, where the unlabeled target domain
contains one unknown class, not present in the labeled source domain. Both
SDSS and DECaLS data include three filter images ($i$, $r$, $g$). We use
SExtractor [1] to determine the center and radius of objects in the downloaded
images, and then crop images to $256\times 256$ pixels, using the extracted
object properties to ensure no pertinent parts of the image have been
inappropriately removed. The source and target datasets each contain
${\approx}6.5$k images for our 3-class problem, and ${\approx}20$k images for
out 10-class problem. All data in our experiments are normalized to pixel
values $\left[0,1\right]$ and then divided into training, validation, and test
sets in proportions $60\%:20\%:20\%$.

Figure 1: Example images from the 3-class problem. Source domain SDSS data
(spiral and elliptical classes) is given in the top row, while the target
domain DECaLS data (spiral, elliptical, and merging classes) is in the bottom
row. We give true labels in the top left corner of each image and predicted
labels in the top right corner (when trained with DA). These examples show the
most often correctly and incorrectly classified objects in our experiments.

##  4 Results

In Table 1 we report accuracies for the source and target test sets: normal
training without any DA, i.e. training with just ${\cal L}_{CE}$ on the source
domain (top row), and then DeepAstroUDA training with DA (bottom row). Using
DA improves performance in both data domains. Aligning the data distributions
allows the model to work well on unlabeled target data. The use of domain-
invariant features makes training harder, preventing the model from
overfitting on the source domain to allow it to train for longer and increase
performance even on the labeled source data. In the top row of Figure 2, we
show how the target domain accuracies in the 3-class problem change during
training, as well as how the three different loss terms evolve. The bottom row
shows the training target accuracies for the 10-class problem and a target
test set confusion matrix that indicates the similarities between some known
galaxy classes. Finally, in Figure 3, we show t-SNE plots [15] of the latent
space of the model trained on the 3-class problem, to better illustrate the
effects of DA training. Our results show that DeepAstroUDA can: 1)
successfully be used on difficult domain shift problems such as astronomical
data originating from different surveys, and 2) handle any type of domain
overlap and perform in the presence of unknown classes, which can be used for
anomaly detection tasks like searching for merging galaxies, gravitational
lenses, etc. While we focus on cross-survey Open DA problems (unknown class
present in the target domain) in this paper, we will explore and present
results of our method on different types of domain shift problems in our
future work.

Table 1: Accuracies for ResNet50 on SDSS (source) and DECaLS (target) test data (3-class and 10-class problem) when training without DA (top row) and with DA (bottom row). Inclusion of DA increases accuracy for both source and target data in both experiments.  | 3-class | 10-class
---|---|---
Train. type | Source | Target | Source | Target
No DA | $0.81$ | $0.56$ | $0.77$ | $0.43$
DA | $\bm{0.84}$ | $\bm{0.82}$ | $\bm{0.82}$ | $\bm{0.79}$

Figure 2: Top (3-class): left: target domain accuracies during training
(elliptical in violet, spiral in navy and unknown merger class in yellow);
right: loss functions during training (CE loss in yellow, AC loss in green, ES
loss in dark green). Vertical dashed lines on the left plot show epochs in
which hyperparameters of the ES loss were fine-tuned. Bottom (10-class): left:
target domain mean known class accuracy (blue), unknown gravitational lens
class accuracy (red), and mean accuracy for all classes (black dashed line);
right: target test set confusion matrix to illustrate confusion between
morphologically similar classes (disturbed (0), merging (1), round smooth (2),
cigar shaped smooth (3), barred spiral (4), unbarred tight spiral (5),
unbarred loose spiral (6), edge-on without bulge (7), edge-on with bulge (8),
lenses (9)).
Figure 3: t-SNE plots of the latent space of the model trained on the 3-class
problem. Without DA (left) the target domain (empty circles) remains
completely separate from the source (filled circles), while using DA (right)
aligns classes correctly, which allows the model to perform well in both
domains. Furthermore, we can see that the unknown merger class (yellow) in the
target domain is separated from the rest and moved to the outskirts.

## Broader Impact

Our paper introduces a universal DA method that can handle any type of overlap
between different astronomical datasets and is applicable not only to multiple
types of problems in astrophysics and cosmology, but also to other scientific
domains and beyond. This research will impact the wider scientific community,
since work with data from simulations and multiple telescopes or experiments
is often present in many scientific applications. This is also relevant for
the development of methods trained on simulations or old observations that
work during new observations in real time, which is crucial for fast detection
of transient phenomena, anomalies or real-time data cataloging and dataset
creation.

## Acknowledgments and Disclosure of Funding

This manuscript has been supported by Fermi Research Alliance, LLC under
Contract No. DE-AC02-07CH11359 with the U.S. Department of Energy (DOE),
Office of Science, Office of High Energy Physics. This research has been
partially supported by the High Velocity Artificial Intelligence grant as part
of the DOE High Energy Physics Computational HEP program. It has also been
partially supported by the DOE Office of Science, Office of Advanced
Scientific Computing Research, applied mathematics and SciDAC programs under
Contract No. DE-AC02-06CH11357. This research used resources of the Argonne
Leadership Computing Facility at Argonne National Laboratory, which is a user
facility supported by the DOE Office of Science.

The authors of this paper have committed themselves to performing this work in
an equitable, inclusive, and just environment, and we hold ourselves
accountable, believing that the best science is contingent on a good research
environment. We acknowledge the Deep Skies Lab as a community of multi-domain
experts and collaborators who have facilitated an environment of open
discussion, idea-generation, and collaboration. This community was important
for the development of this project.

Furthermore, we also thank the anonymous referees who helped improve this
manuscript.

Author Contributions: A. Ćiprijanović: Conceptualization, Data curation,
Formal analysis, Investigation, Methodology, Project administration,
Resources, Software, Supervision, Visualization, Writing of original draft; A.
Lewis: Formal analysis, Investigation, Methodology, Resources, Software,
Visualization, Writing of original draft; K. Pedro: Conceptualization,
Methodology, Project administration, Resources, Software, Supervision, Writing
(review & editing); S. Madireddy: Conceptualization, Methodology, Resources,
Software, Supervision, Writing (review & editing); B. Nord: Methodology,
Supervision, Writing (review & editing); G. N. Perdue: Conceptualization,
Methodology, Project administration, Resources, Software, Supervision, Writing
(review & editing); S. M. Wild: Conceptualization, Methodology, Writing
(review & editing).

## References

[1] E. Bertin and S. Arnouts. SExtractor: Software for source extraction.
A&AS, 117:393–404, June 1996.

[2] A. Ćiprijanovi ć, D. Kafkes, K. Downey, S. Jenkins, G. N. Perdue, S.
Madireddy, T. Johnston, G. F. Snyder, and B. Nord. DeepMerge - II. Building
robust deep learning algorithms for merging galaxy identification across
domains. MNRAS, 506(1):677–691, September 2021.

[3] Aleksandra Ćiprijanovi ć, Diana Kafkes, Gregory Snyder, F. Javier Sánchez,
Gabriel Nathan Perdue, Kevin Pedro, Brian Nord, Sandeep Madireddy, and Stefan
M. Wild. DeepAdversaries: examining the robustness of deep learning models for
galaxy morphology classification. Machine Learning: Science and Technology,
3(3):035007, September 2022.

[4] Gabriela Csurka. A comprehensive survey on domain adaptation for visual
applications. In Domain Adaptation in Computer Vision Applications, pages
1–35. Springer International Publishing, Cham, 2017.

[5] Dark Energy Survey Collaboration, T. Abbott, F. B. Abdalla, J. Aleksi ć,
S. Allam, A. Amara, D. Bacon, E. Balbinot, M. Banerji, and et al. The Dark
Energy Survey: more than dark energy - an overview. MNRAS, 460(2):1270–1299,
August 2016.

[6] DESI Collaboration. Overview of the DESI Legacy Imaging Surveys. AJ,
157(5):168, May 2019.

[7] Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and
Andrew Zisserman. Automatically Discovering and Learning New Visual Categories
with Ranking Statistics. arXiv e-prints, page arXiv:2002.05714, February 2020.

[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual
learning for image recognition. In 2016 IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), pages 770–778, 2016.

[9] Željko Ivezić, Steven M. Kahn, J. Anthony Tyson, Bob Abel, Emily Acosta,
Robyn Allsman, David Alonso, Yusra AlSayyad, and et al. LSST: From science
drivers to reference design and anticipated data products. ApJ, 873(2):111,
March 2019.

[10] Jichang Li, Guanbin Li, Yemin Shi, and Yizhou Yu. Cross-domain adaptive
clustering for semi-supervised domain adaptation. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2505–2514, June 2021.

[11] Chris Lintott, Kevin Schawinski, Steven Bamford, Anže Slosar, Kate Land,
Daniel Thomas, Edd Edmondson, Karen Masters, Robert C. Nichol, M. Jordan
Raddick, Alex Szalay, Dan Andreescu, Phil Murray, and Jan Vandenberg. Galaxy
Zoo 1: data release of morphological classifications for nearly 900000
galaxies. MNRAS, 410(1):166–178, January 2011.

[12] Chris J. Lintott, Kevin Schawinski, Anže Slosar, Kate Land, Steven
Bamford, Daniel Thomas, M. Jordan Raddick, Robert C. Nichol, Alex Szalay, Dan
Andreescu, Phil Murray, and Jan Vandenberg. Galaxy Zoo: morphologies derived
from visual inspection of galaxies from the Sloan Digital Sky Survey. MNRAS,
389(3):1179–1189, September 2008.

[13] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal
Domain Adaptation through Self Supervision. arXiv e-prints, page
arXiv:2002.07953, February 2020.

[14] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the
importance of initialization and momentum in deep learning. In Sanjoy Dasgupta
and David McAllester, editors, Proceedings of the 30th International
Conference on Machine Learning, volume 28 of Proceedings of Machine Learning
Research, pages 1139–1147, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR.

[15] L. van der Maaten and G. Hinton. Visualizing data using t-SNE. Journal of
Machine Learning Research, 9(86):2579–2605, 2008.

[16] Ricardo Vilalta, Kinjal Dhar Gupta, Dainis Boumber, and Mikhail M.
Meskhi. A General Approach to Domain Adaptation with Applications in
Astronomy. PASP, 131(1004):108008, October 2019.

[17] Mike Walmsley, Chris Lintott, Tobias Géron, Sandor Kruk, Coleman
Krawczyk, Kyle W. Willett, Steven Bamford, Lee S. Kelvin, Lucy Fortson, Yarin
Gal, William Keel, Karen L. Masters, Vihang Mehta, Brooke D. Simmons, Rebecca
Smethurst, Lewis Smith, Elisabeth M. Baeten, and Christine Macmillan. Galaxy
Zoo DECaLS: Detailed visual morphology measurements from volunteers and deep
learning for 314 000 galaxies. MNRAS, 509(3):3966–3988, January 2022.

[18] Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey.
Neurocomputing, 312:135–153, 2018.

[19] Kyle W. Willett, Chris J. Lintott, Steven P. Bamford, Karen L. Masters,
Brooke D. Simmons, Kevin R. V. Casteels, Edward M. Edmondson, Lucy F. Fortson,
Sugata Kaviraj, William C. Keel, Thomas Melvin, Robert C. Nichol, M. Jordan
Raddick, Kevin Schawinski, Robert J. Simpson, Ramin A. Skibba, Arfon M. Smith,
and Daniel Thomas. Galaxy Zoo 2: detailed morphological classifications for
304 122 galaxies from the Sloan Digital Sky Survey. MNRAS, 435(4):2835–2860,
November 2013.

[20] Garrett Wilson and Diane J. Cook. A survey of unsupervised deep domain
adaptation. ACM Transactions on Intelligent Systems and Technology, 11(5),
July 2020.

[21] Donald G. York, J. Adelman, Jr. Anderson, John E., Scott F. Anderson,
James Annis, Neta A. Bahcall, J. A. Bakken, Robert Barkhouser, Steven Bastian,
Eileen Berman, et al., and SDSS Collaboration. The Sloan Digital Sky Survey:
Technical Summary. AJ, 120(3):1579–1587, September 2000.

[22] Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I.
Jordan. Universal domain adaptation. In 2019 IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), pages 2715–2724, 2019.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [No] We do not discuss
limitations in this short preliminary study, but we will discuss it in our
more detailed future work.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [N/A] We
do not see any obvious negative social impact of our work, which is related to
domain adaptation for astronomy.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A] We
do not include any theoretical results.

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
We provide URL for our GitHub repository, where the code used in this work is
publicly available.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] Details are given in Section 2 and Section 3.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [No] In this preliminary study we only report
results from a single run without multiple random seeds.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We give
details in Section 2.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes] In Section
3 we reference papers related to the datasets we use, as well as provide links
where the data is available.

    2. (b)

Did you mention the license of the assets? [N/A]

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [No]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A] We are using publicly available Galaxy Zoo data.

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A] Astronomical data does
not contain any personally identifiable information.

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A] We did not use any crowdsourcing. Even
though datasets were labeled via crowdsourcing, all work was performed
beforehand, through the Galaxy Zoo project, which we were not involved in.

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]