# Parameter-Efficient Low-Resource
Dialogue State Tracking by Prompt Tuning

Mingyu Derek Ma† Jiun-Yu Kao‡ Shuyang Gao§ Arpit Gupta‡
Di Jin‡ Tagyoung Chung‡ Nanyun Peng†‡
†University of California, Los Angeles ‡Amazon Alexa AI
{ma<EMAIL_ADDRESS>
{jiunyk, guparpit, djinamzn<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Dialogue state tracking (DST) is an important step in dialogue management to
keep track of users’ beliefs. Existing works fine-tune all language model (LM)
parameters to tackle the DST task, which requires significant data and
computing resources for training and hosting. The cost grows exponentially in
the real-world deployment where dozens of fine-tuned LM are used for different
domains and tasks. To reduce parameter size and better utilize cross-task
shared information, we propose to use soft prompt token embeddings to learn
task properties. Without tuning LM parameters, our method drastically reduces
the number of parameters needed to less than 0.5% of prior works while
achieves better low-resource DST performance.††§Work done while at Amazon.

##  1 Introduction

Dialogue state tracking (DST) that extracts structured conversation progress
in a list of slot-value pairs from unstructured dialogue utterances is an
essential component of a dialogue system Wang and Lemon (2013). Unlike
classification-based models that pick the slot value from given candidate Ye
et al. (2021); Chen et al. (2020), recent works formulate DST as a conditional
generation task Gao et al. (2019); Lin et al. (2020), where the concatenation
of dialogue history and a slot-specific prompt are fed to generative models
and the text generation output are decoded to predicted slot values Ham et al.
(2020); Hosseini-Asl et al. (2020). This formulation enjoys the benefit of
generalizability to unseen domains and slot types beyond a defined dialogue
ontology Li et al. (2021); Peng et al. (2021).

General prompting methods use a textual prompt to provide task information to
the LM Liu et al. (2021); Ma et al. (2023b). Prior works have variations that
update different parameter combinations such as both LM and prompt token
embeddings Gao et al. (2021); Li and Liang (2021); Xu et al. (2023), only the
token embeddings of the LM Zhu et al. (2021), or only the prompt token
embeddings Lester et al. (2021); Gu et al. (2022); Vu et al. (2022).

While there are some existing prompt-based approaches for DST with different
designs of prompts such as using slot name Lee and Jha (2019); Zhao et al.
(2021); Lee et al. (2021); Su et al. (2022), slot description Rastogi et al.
(2020), slot type Lin et al. (2021b), possible values Lin et al. (2021b),
priming examples Gupta et al. (2022) and/or slot-specific question Gao et al.
(2019); Zhou and Small (2019); Gao et al. (2020); Lin et al. (2021a); Li et
al. (2021); Wu et al. (2022) in prompt sentences, they all fine-tune the
entire LM along with the prompt tokens for a new domain, which requires a
significant amount of training time, system resources, and annotated data
Clarke et al. (2022); Sauer et al. (2022). The computing and data resource-
hungry issues are more severe in the real-world deployment where LMs tuned for
different domains and tasks need to be trained and hosted, and a typical
dialogue system has to serve dozens of such LMs Maronikolakis and Schütze
(2021); Strubell et al. (2019); Lacoste et al. (2019). This leads to a high
cost of the development and service of dialogue systems and constrains offline
deployment. In addition, limited data is available for a new domain or task.

Figure 1: Model design. The snow icon indicates non-trainable parameters.
Absolute positional embeddings are added together with segment embeddings and
sequence embeddings, we omit it for simplicity in the illustration.

We propose a parameter-efficient and data-efficient DST model for low-resource
settings, which only needs to update 0.08% of parameters compared with the
previous best model, by keeping LM parameters frozen and introducing soft
prompt tokens to represent task properties of different slots. Figure 1 gives
an overview of our model. The only prior work we are aware of that only
updates prompt token embeddings and thus parameter-efficient is Zhu et al.
(2022), but it focuses on continual domain adaptation and with a significant
amount of training data.

Our design introduces three techniques that are generalizable to other
generative-based information extraction models. 1) Task-specific parameters:
task prompt tokens are introduced to specifically learn domain, slot and slot
type information so that the model behaves according to the task; word-mapping
prompt tokens enable us to obtain task knowledge contained in natural language
instruction and optimize human-created prompts with continuous embedding
space. 2) Task metadata in objective: we introduce the reiteration technique
in the target sequence in order to include explicit task signals in the text
generation objective. 3) Distinguishing segments: segment embeddings help the
model identify the prompt segment, dialogue speakers, and question partition.
Our proposed method enables much more efficient dialogue system deployment as
only one LM needs to be hosted and inference for different domains could be
realized by feeding domain-specific prompt token embeddings into the
transformer stack.

Experiments on MultiWOZ 2.0 show that our method achieves better performance
on low-resource DST with orders of magnitude fewer parameters. We further
conduct ablation studies, error analysis, and examine the semantic information
shown in the prompt tokens. We observe that our model is more specialized in
predicting categorical slot values, is more conservative for slots with free
output space and introduces more hallucination errors for categorical slots.

##  2 Method

We introduce task definition (Section 2.1), overall framework (Section 2.2)
and soft prompt designs (Section 2.3) in this section.

###  2.1 Task Definition

The goal is to construct a belief state with $|S|$ pairs of slot and value at
a certain turn in a multi-turn conversation. All the turns up to the query
turn are dialogue history, and slot-specific information (i.e. name,
description, value candidates, question and type of the slot) is
provided.111We show slot-specific info in Appendix A.1 and Appendix A.2. Value
candidates are from dialogue ontology.

###  2.2 Generative Seq2seq Framework

We use a decoder-only pre-trained language model (PLM) GPT-2 Radford et al.
(2019) as the backbone to provide language and commonsense knowledge, rather
than an encoder-decoder model because of its superior performance Li et al.
(2021). To get a belief state at a certain turn, we create $|S|$ data
instances to predict the slot value for each slot. Figure 1 demonstrates the
design and a sample query.

#### Input sequence.

We construct the input sequence by concatenating the following segments: 1)
Task prompt tokens for domain, slot and type, each has $k$ prompt tokens and
they are shared among instances with the same domain, slot or type; 2) Prefix,
a short sentence containing slot description, names of domain, slot, and type,
and all possible candidates if the query slot is categorical; 3) Dialogue
history, in which [sys] and [usr] tokens are used to indicate the speaker; and
4) Question, human-written question about the slot.

#### Target sequence and reiteration.

We introduce the reiteration technique in the target sequence as shown in
Figure 1 and generate task information before the answer phrase. We include
the verbalized slot information as a “domain is domain name, slot is slot
name, type is type name” phrase in the expected output sequence. By doing so,
we require the model to optimize to remember the task information explicitly
before generating the answer phrase, while using a consistent text generation
cross-entropy loss. This technique allows the model to optimize upon both the
answer and the sentence containing slot metadata, and explicitly learn the
task information.

Model | Params# | 5 | 10 | 20 | 1% | 5% | 10% | 5 | 10 | 20 | 1% | 5% | 10% | 5 | 10 | 20 | 1% | 5% | 10%
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Attraction (3 slots, 1% = 27 conv.) | Hotel (10 slots, 1% = 33 conv.) | Restaurant (7 slots, 1% = 38 conv.)
TRADE |  | — | — | — | — | 52.19 | 58.46 | — | — | — | — | 31.93 | 41.29 | — | — | — | — | 47.31 | 53.65
DSTQA |  | — | — | — | — | 51.58 | 61.77 | — | — | — | — | 33.08 | 49.69 | — | — | — | — | 35.33 | 54.27
T5DST | 60M | 4.77 | 21.93 | 30.57 | 40.68 | 52.12 | 60.13 | 8.19 | 13.46 | 17.94 | 18.63 | 38.76 | 46.13 | 13.80 | 19.51 | 22.79 | 29.47 | 53.32 | 58.44
Lee et al. | 60M | 6.33 | 19.12 | 34.53 | 37.56 | 54.34 | 58.75 | 9.31 | 15.76 | 22.07 | 24.41 | 40.11 | 42.98 | 15.87 | 19.66 | 22.15 | 30.96 | 48.94 | 58.59
Li et al. | 335M | 7.90 | 27.09 | 35.63 | 42.18 | 49.13 | 60.85 | 12.49 | 15.15 | 19.44 | 24.04 | 37.88 | 46.47 | 17.27 | 22.30 | 25.68 | 30.70 | 49.75 | 58.50
Ours | 271K | 33.56 | 39.41 | 45.75 | 47.28 | 56.99 | 63.61 | 15.63 | 18.18 | 22.50 | 33.01 | 38.24 | 45.60 | 19.76 | 25.72 | 27.65 | 34.40 | 50.81 | 55.79
|  | Taxi (4 slots, 1% = 15 conv.) | Train (6 slots, 1% = 29 conv.) | Average
TRADE | — | — | — | — | 59.03 | 60.51 | — | — | — | — | 48.82 | 59.65 | — | — | — | — | 47.86 | 54.71
DSTQA |  | — | — | — | — | 58.25 | 59.35 | — | — | — | — | 50.36 | 61.28 | — | — | — | — | 45.72 | 57.27
T5DST | 60M | 48.22 | 53.74 | 58.27 | 58.19 | 59.23 | 69.03 | 12.31 | 21.93 | 36.45 | 43.93 | 69.27 | 69.48 | 17.46 | 26.11 | 33.20 | 38.18 | 54.54 | 60.64
Lee et al. | 60M | 45.32 | 49.93 | 58.58 | 58.52 | 60.77 | 71.23 | 13.57 | 25.02 | 38.52 | 50.26 | 69.32 | 69.72 | 18.08 | 25.90 | 35.17 | 40.34 | 54.70 | 60.25
Li et al. | 335M | 50.99 | 57.47 | 58.49 | 58.26 | 61.68 | 69.23 | 17.56 | 27.42 | 39.27 | 45.32 | 71.69 | 73.45 | 21.24 | 29.89 | 35.70 | 40.10 | 54.03 | 61.70
Ours | 271K | 51.11 | 59.63 | 60.89 | 60.33 | 61.63 | 63.00 | 18.95 | 30.95 | 50.34 | 52.05 | 69.51 | 75.00 | 27.80 | 34.78 | 41.43 | 45.41 | 55.44 | 60.60

Table 1:  Overall performance. Detailed parameter counts are in Appendix A.3,
variances are in Appendix B.5.

#### Segment embeddings.

The input sequence contains segments with diverse formats and they are quite
different from the format used in the pre-training phase of the LM. We divide
the input sequence into segments, including five prompt segments, the system
turns, the user turns and the answer segment. Tokens within a segment are
assigned the same segment ID. Segment embeddings, which have the same length
as the input sequence, are added with sequence embeddings and positional
embeddings. We randomly initialize the embeddings of segment IDs and update
them during training.

#### Training and inference.

We pass the combined embeddings to the decoder stack to calculate the
likelihood over the vocabulary. We use the cross-entropy loss with a
regularization term to constrain the scale of prompt token embeddings
following $L=CE+\lambda\|PE^{\prime}-PE\|_{2}^{2}$ where $\lambda$ is a
weighting factor, and $PE^{\prime}$ and $PE$ are updated and initialized
prompt token embeddings Müller et al. (2022). Parameters of the PLM are
frozen, and only prompt and segment embeddings are updated with Adam
optimizer. During inference, we generate the output autoregressively with
greedy decoding, and extract the answer with a rule-based function.

###  2.3 Soft Prompt Tokens

#### Prompt segments.

We use two kinds of prompt tokens. Task prompt tokens are chosen according to
the task’s metadata, and used in the domain, slot and type prompt segments.
Word-mapping prompt tokens are mapped from existing tokens in the prefix and
question parts and used to replace normal tokens. In other words, task and
word-mapping prompt tokens are shared across instances with the same task and
instances using the same words respectively. We concatenate embeddings of each
prompt segment (obtained by separate embedding matrices) with dialogue history
embeddings (obtained by the frozen token embedding matrix) to form sequence
embeddings.

#### Prompt initialization.

To boost the performance in the low-resource setting, we use the pre-trained
token embeddings to initialize the soft prompt token embeddings. The token
embeddings from PLM are used to represent word semantics for language
understanding, while the soft prompt tokens are used to represent task
information initialized by task-related semantic meanings. We initialize a
task prompt token by embedding of a randomly chosen token from its domain,
slot or slot type name. Word-mapping prompt tokens are initialized with the
embedding of the mapped word.

##  3 Experimental Setup

#### Dataset.

We experiment on dialogues of five domains (i.e. attraction, hotel,
restaurant, train, taxi) in MultiWOZ 2.0 Budzianowski et al. (2018).

#### Settings.

We evaluate using the low-resource few-shot DST task. We take 5, 10, 20, 1%,
5% and 10% of training conversations to train, and evaluate on the full test
set of each target domain.222Appendix C.3 and C.4 show experimental setting
details.

#### Evaluation metrics.

Joint Goal Accuracy (JGA) represents the proportion of turns with all slots
predicted correctly, and Slot Accuracy (SA) reflects the proportion of correct
slots. If a slot is empty at a certain turn (for example, no related
information is mentioned), the model needs to predict “none”. A slot value is
only correct if it matches exactly with the ground-truth value.

#### Baseline models.

We compare with the following works.333We are not comparing with prompt-based
DST works that jointly train with other tasks for a fair comparison. 1) TRADE
Wu et al. (2019): GRU-based model with copy mechanism; 2) DSTQA Zhou and Small
(2019): QA-style model using ELMo representation; 3) T5DST Lin et al. (2021b):
T5-based generative model with slot type as prompt; 4) Lee et al. (2021):
T5-based generative model with slot description and possible slot values as
prompt; 5) Li et al. (2021): GPT-2 based QA-style generative model with
manually created questions. The entire language model is updated for T5DST,
Lee et al. and Li et al., and they represent the performance of prompt-based
DST works. Appendix B.6 shows comparison with baselines’ frozon LM variation.

##  4 Experimental Results

#### Overall results.

We show the overall few-shot experimental results in Table 1. Although our
model uses only 0.08% and 0.45% of parameters compared with baselines, it
still achieves higher JGA than all baseline models when using 1% or less
training data across all domains. Especially we observe around 5, and 9 points
JGA increases for the attraction and hotel domains compared with existing best
models with 1% training data. In the attraction domain with 3 unique slots,
our model trained using 5 dialogues performs on par with the previous best
model using 20 dialogues. Our model shows its superiority especially when the
amount of unique tasks is small. Using 5% and 10% data, our model performs
comparably with existing best models with small gaps.

Figure 2: Slot accuracies across slot types using 1% training data, each dot
represents a unique slot.

We demonstrate the performance of slots with different types in Figure 2
across all five domains in Figure 2 compared with two generative baselines Li
et al. and T5DST Lin et al. (2021b). These slot types are defined in Table 4.
We observe the worst performance in open slots, which could be explained by
the larger output candidate space.444A SA vs ontology size analysis is in
Appendix B.2. Breaking down slot type to more fine-grained type lead to a
better result (considering day as a separate type rather than categorical
type, number and time as separate types rather than open type). Compared with
baselines, our model performs comparably on open and time slots, but is more
superior for categorical, number and day slots.555SA for each slot and
comparisons are in Appendix B.3.

#### Ablation study.

In Table 2, removing the slot segment (Line 2) leads to the largest
performance drop among the three task prompt segments (L1-3), as slot is the
most fine-grained task categorization. Prefix (L5) is more important than the
question prompt (L4), which contains more metadata and parameters. The model
without segment embedding (L6) has on average 7.8 points JGA drop, indicating
the effectiveness of the segment embedding. We also observe an almost 2 points
JGA drop (and an even larger drop with fewer training dialogues shown in
Appendix B.1) without reiteration (L7), which shows the helpfulness of
including explicit task information in the learning objective. Note that even
without reiteration, our model performs better than all baselines using 1%
training data.

# | Model |  | Attr. | Hotel | Rest. | Taxi | Train | Avg
---|---|---|---|---|---|---|---|---
1 | w/o domain |  | 44.22 |  | 28.16 |  | 29.78 |  | 60.27 |  | 50.01 |  | 42.49
2 | w/o slot |  | 46.64 |  | 26.55 |  | 24.35 |  | 51.11 |  | 45.11 |  | 38.75
3 | w/o type |  | 45.30 |  | 25.26 |  | 33.65 |  | 59.89 |  | 51.91 |  | 43.20
4 | w/o question |  | 45.08 |  | 32.26 |  | 33.30 |  | 59.63 |  | 51.60 |  | 44.37
5 | w/o prefix |  | 42.98 |  | 28.78 |  | 31.54 |  | 57.72 |  | 47.00 |  | 41.60
6 | w/o segment emb. |  | 34.35 |  | 23.18 |  | 27.33 |  | 59.69 |  | 43.30 |  | 37.57
7 | w/o reiteration |  | 45.08 |  | 27.57 |  | 33.48 |  | 59.89 |  | 51.08 |  | 43.42
8 | Full model |  | 47.28 |  | 33.01 |  | 34.40 |  | 60.33 |  | 52.05 |  | 45.41

Table 2:  Ablation study using 1% training data (JGA).

#### Error and qualitative analysis.

We categorize error cases as: 1) hallucination: predicting value for an empty
slot; 2) omission: predicting “none” for a non-empty slot; 3) wrong value:
predicting wrong real value for a non-empty slot Gao et al. (2020). Figure 3
shows the error distribution in terms of the proportion of each error
category. The general open slots (including time and number) have relatively
more omission errors, while the general categorical slots have relatively more
hallucination errors. Our model is more conservative for open slots compared
with Li et al..666Our model produces relatively larger proportion of omission
error than Li et al., but it generates a reasonable amount of not-none values
for non-empty slots as explained in Appendix B.7.

Figure 3: Error distribution across slot types

We then investigate semantic information contained in the learned prompt
tokens by selecting the most changed prompt tokens and producing the closest
tokens with the smallest cosine similarity between the learned prompt token
embedding and frozen token embeddings of the PLM. We show the result for the
attraction domain in Table 3, and for all domains in Appendix B.4. The closest
tokens are mostly variations or semantically similar tokens of the expected
meanings of prompt tokens.

Prompt token | Closest tokens
---|---
<domain_attraction_4> | raction; ractions; racted
<slot_name_2> | name; Name; names
<type_open_3> | open; Open; opened
special | special; Special; statistical
Q | answer; Answer; answered

Table 3:  Closest tokens for the most changed prompt tokens in five prompt
segments for the attraction domain.

##  5 Related Works

Two lines of work are closely related to our work.

#### Dialogue State Tracking.

DST aims to extract slots and values to capture user intents from dialogues.
In terms of model design, classification-based models pick the slot value from
candidates Ye et al. (2021); Gao et al. (2020); Ma et al. (2019); Chen et al.
(2020), which assumes that the dialogue ontology is pre-defined and cannot be
generalized Chen et al. (2020). Generation-based models directly generate slot
values to handle unseen domains and values Ham et al. (2020); Hosseini-Asl et
al. (2020); Gao et al. (2019, 2020); Lin et al. (2020); Peng et al. (2021). In
terms of knowledge sources, some methods create synthesized dialogues with
human heuristics to do data augmentation for the target domain Campagna et al.
(2020); Hou et al. (2020) which require expensive human costs. Recent works
transfer knowledge from data-rich domains with slot description Lee and Jha
(2019); Rastogi et al. (2020), slot type, possible values Lin et al. (2021b),
possible values Lee et al. (2021), task constraint Mi et al. (2022),
similarity functions between slots and values Wu et al. (2020), and meta-
learning Dingliwal et al. (2021), while the availability of related source
domains constrains their generalizability. Some works transfer from other
tasks like Reading Comprehension Gao et al. (2019, 2020); Lin et al. (2021a).
We take inspiration and use a transformer-based generative model with slot
metadata but using much less trainable parameters.

#### Prompting methods.

Recent works use a textual prompt as a part of the input sequence to provide
task information to the LM Liu et al. (2021). The prompt can be chosen by
experts Radford et al. (2019); Lu et al. (2023, 2022), learned as discrete
readable tokens Shin et al. (2020) or continuous embeddings Qin and Eisner
(2021). The textual prompt can also contain a few examples known as “in-
context learning” without tuning LM Brown et al. (2020). Some works fine-tune
both LM and prompt parameters Gao et al. (2021); Qin and Eisner (2021); Li and
Liang (2021); Ma et al. (2023a); Hsu et al. (2022) or only token embeddings
Zhu et al. (2021). Works like Lester et al. (2021); Gu et al. (2022); Vu et
al. (2022) show that freezing PLM and only tuning learnable soft prompts,
known as “prompt tuning”, is competitive with fine-tuning while using much
less parameters. For the DST task, Lee et al. (2021) use slot information as
prompt and fine-tune PLM, Zhu et al. (2022) use prompt tuning for continual
domain adaptation, both requiring a significant amount of training data. For
the low-resource DST task, Yang et al. (2022) introduce two-way prompts but
need to fine-tune LM, which is not parameter-efficient.

##  6 Conclusion and Future Work

We propose a parameter-efficient DST model using prompt tuning, and it
represents tasks with soft prompt tokens with segment awareness and
reiteration. Our model achieves state-of-the-art low-resource DST performance
with less than 0.5% parameters compared with fine-tuning LM. We plan to
further investigate prompt aggregation.

## Limitations

There are several limitations to our work. Firstly, the proposed model is more
sensitive to hyper-parameters such as the number of prompt tokens and learning
rate than existing methods that fine-tune LM. Therefore, it would require
additional parameter searching efforts to obtain the best performance.
Secondly, our model is designed for and evaluated in English-only
conversations, and applying our technique to other languages or code-switching
scenarios might lead to performance decay. Finally, our experimental result
shows that our proposed prompt tuning method works better than fine-tuning LM
when there are fewer unique tasks to be optimized. Therefore, our method might
not work well on a more diverse dataset.

## Ethics Statement

We do not see an immediate negative impact of the proposed method and do not
see biased predictions made by our model. Our method is based on a pre-trained
generative language model and trained on an open DST dataset, thus bias
contained in the corpus for pre-training and the DST dataset might propagate
to prediction outputs of our model. Human validation of the prediction results
and their fairness needs to be conducted before our proposed model is used in
production and other real-world applications. Our proposed model does not
increase energy and carbon costs but will potentially reduce them due to its
data and parameter efficiency.

## Acknowledgments

Many thanks to Sidi Lu, Tanmay Parekh, and Sarik Ghazarian for internal
reviews, to members at Amazon Alexa AI, PLUS lab and UCLA-NLP for suggestions,
and to the anonymous reviewers for their feedback.

## References

  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5016–5026, Brussels, Belgium. Association for Computational Linguistics.
  * Campagna et al. (2020) Giovanni Campagna, Agata Foryciarz, Mehrad Moradshahi, and Monica Lam. 2020.  Zero-shot transfer learning with synthesized data for multi-domain dialogue state tracking.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 122–132, Online. Association for Computational Linguistics.
  * Chen et al. (2020) Lu Chen, Boer Lv, Chi Wang, Su Zhu, Bowen Tan, and Kai Yu. 2020.  Schema-guided multi-domain dialogue state tracking with graph attention neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 7521–7528.
  * Clarke et al. (2022) Christopher Clarke, Joseph Peper, Karthik Krishnamurthy, Walter Talamonti, Kevin Leach, Walter Lasecki, Yiping Kang, Lingjia Tang, and Jason Mars. 2022.  One agent to rule them all: Towards multi-agent conversational AI.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3258–3267, Dublin, Ireland. Association for Computational Linguistics.
  * Dingliwal et al. (2021) Saket Dingliwal, Shuyang Gao, Sanchit Agarwal, Chien-Wei Lin, Tagyoung Chung, and Dilek Hakkani-Tur. 2021.  Few shot dialogue state tracking using meta-learning.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1730–1739, Online. Association for Computational Linguistics.
  * Gao et al. (2020) Shuyang Gao, Sanchit Agarwal, Di Jin, Tagyoung Chung, and Dilek Hakkani-Tur. 2020\.  From machine reading comprehension to dialogue state tracking: Bridging the gap.  In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_ , pages 79–89, Online. Association for Computational Linguistics.
  * Gao et al. (2019) Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagyoung Chung, and Dilek Hakkani-Tur. 2019.  Dialog state tracking: A neural reading comprehension approach.  In _Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 264–273, Stockholm, Sweden. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021.  Making pre-trained language models better few-shot learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3816–3830, Online. Association for Computational Linguistics.
  * Gu et al. (2022) Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2022.  PPT: Pre-trained prompt tuning for few-shot learning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8410–8423, Dublin, Ireland. Association for Computational Linguistics.
  * Gupta et al. (2022) Raghav Gupta, Harrison Lee, Jeffrey Zhao, Yuan Cao, Abhinav Rastogi, and Yonghui Wu. 2022.  Show, don’t tell: Demonstrations outperform descriptions for schema-guided task-oriented dialogue.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4541–4549, Seattle, United States. Association for Computational Linguistics.
  * Ham et al. (2020) Donghoon Ham, Jeong-Gwan Lee, Youngsoo Jang, and Kee-Eung Kim. 2020.  End-to-end neural pipeline for goal-oriented dialogue systems using GPT-2.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 583–592, Online. Association for Computational Linguistics.
  * Hosseini-Asl et al. (2020) Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020.  A simple language model for task-oriented dialogue.  _Advances in Neural Information Processing Systems_ , 33:20179–20191.
  * Hou et al. (2020) Yutai Hou, Sanyuan Chen, Wanxiang Che, Cheng Chen, and Ting Liu. 2020.  C2c-genda: Cluster-to-cluster generation for data augmentation of slot filling.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, page 13.
  * Hsu et al. (2022) I-Hung Hsu, Kuan-Hao Huang, Elizabeth Boschee, Scott Miller, Prem Natarajan, Kai-Wei Chang, and Nanyun Peng. 2022.  DEGREE: A data-efficient generation-based event extraction model.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1890–1908, Seattle, United States. Association for Computational Linguistics.
  * Lacoste et al. (2019) Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019\.  Quantifying the carbon emissions of machine learning.  _arXiv preprint arXiv:1910.09700_.
  * Lee et al. (2021) Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. 2021.  Dialogue state tracking with a language model using schema-driven prompting.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4937–4949, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lee and Jha (2019) Sungjin Lee and Rahul Jha. 2019.  Zero-shot adaptive transfer for conversational language understanding.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 6642–6649.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li et al. (2021) Shuyang Li, Jin Cao, Mukund Sridhar, Henghui Zhu, Shang-Wen Li, Wael Hamza, and Julian McAuley. 2021.  Zero-shot generalization in dialog state tracking through generative question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1063–1074, Online. Association for Computational Linguistics.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Lin et al. (2021a) Zhaojiang Lin, Bing Liu, Andrea Madotto, Seungwhan Moon, Zhenpeng Zhou, Paul Crook, Zhiguang Wang, Zhou Yu, Eunjoon Cho, Rajen Subba, and Pascale Fung. 2021a.  Zero-shot dialogue state tracking via cross-task transfer.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7890–7900, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lin et al. (2021b) Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Andrea Madotto, Eunjoon Cho, and Rajen Subba. 2021b.  Leveraging slot descriptions for zero-shot cross-domain dialogue StateTracking.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5640–5648, Online. Association for Computational Linguistics.
  * Lin et al. (2020) Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, and Pascale Fung. 2020.  MinTL: Minimalist transfer learning for task-oriented dialogue systems.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3391–3405, Online. Association for Computational Linguistics.
  * Liu et al. (2021) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _arXiv preprint arXiv:2107.13586_.
  * Lu et al. (2022) Keming Lu, I-Hung Hsu, Wenxuan Zhou, Mingyu Derek Ma, and Muhao Chen. 2022.  Summarization as indirect supervision for relation extraction.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 6575–6594, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Lu et al. (2023) Keming Lu, I-Hung Hsu, Wenxuan Zhou, Mingyu Derek Ma, and Muhao Chen. 2023.  Multi-hop evidence retrieval for cross-document relation extraction.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , Toronto, Canada. Association for Computational Linguistics.
  * Ma et al. (2019) Mingyu Derek Ma, Kevin Bowden, Jiaqi Wu, Wen Cui, and Marilyn Walker. 2019.  Implicit discourse relation identification for open-domain dialogues.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 666–672, Florence, Italy. Association for Computational Linguistics.
  * Ma et al. (2023a) Mingyu Derek Ma, Alexander K. Taylor, Wei Wang, and Nanyun Peng. 2023a.  DICE: Data-efficient clinical event extraction with generative models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics_ , Toronto, Canada. Association for Computational Linguistics.
  * Ma et al. (2023b) Mingyu Derek Ma, Xiaoxuan Wang, Po-Nien Kung, P. Jeffrey Brantingham, Nanyun Peng, and Wei Wang. 2023b.  STAR: Boosting low-resource event extraction by structure-to-text data generation with large language models.  _arXiv preprint arXiv:2305.15090_.
  * Maronikolakis and Schütze (2021) Antonis Maronikolakis and Hinrich Schütze. 2021.  Multidomain pretrained language models for green NLP.  In _Proceedings of the Second Workshop on Domain Adaptation for NLP_ , pages 1–8, Kyiv, Ukraine. Association for Computational Linguistics.
  * Mi et al. (2022) Fei Mi, Yitong Li, Yasheng Wang, Xin Jiang, and Qun Liu. 2022.  Cins: Comprehensive instruction for few-shot learning in task-oriented dialog systems.
  * Müller et al. (2022) Thomas Müller, Guillermo Pérez-Torró, and Marc Franco-Salvador. 2022\.  Few-shot learning with Siamese networks and label tuning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8532–8545, Dublin, Ireland. Association for Computational Linguistics.
  * Peng et al. (2021) Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. 2021.  Soloist: Building task bots at scale with transfer learning and machine teaching.  _Transactions of the Association for Computational Linguistics_ , 9:807–824.
  * Qin and Eisner (2021) Guanghui Qin and Jason Eisner. 2021.  Learning how to ask: Querying LMs with mixtures of soft prompts.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5203–5212, Online. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rastogi et al. (2020) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020.  Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8689–8696.
  * Sauer et al. (2022) Anna Sauer, Shima Asaadi, and Fabian Küch. 2022.  Knowledge distillation meets few-shot learning: An approach for few-shot intent classification within and across domains.  In _Proceedings of the 4th Workshop on NLP for Conversational AI_ , pages 108–119, Dublin, Ireland. Association for Computational Linguistics.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020.  Eliciting knowledge from language models using automatically generated prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4222–4235.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in NLP.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3645–3650, Florence, Italy. Association for Computational Linguistics.
  * Su et al. (2022) Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang. 2022.  Multi-task pre-training for plug-and-play task-oriented dialogue system.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4661–4676, Dublin, Ireland. Association for Computational Linguistics.
  * Vu et al. (2022) Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou’, and Daniel Cer. 2022.  SPoT: Better frozen model adaptation through soft prompt transfer.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5039–5059, Dublin, Ireland. Association for Computational Linguistics.
  * Wang and Lemon (2013) Zhuoran Wang and Oliver Lemon. 2013.  A simple and generic belief tracking mechanism for the dialog state tracking challenge: On the believability of observed information.  In _Proceedings of the SIGDIAL 2013 Conference_ , pages 423–432, Metz, France. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu et al. (2020) Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020.  TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 917–929, Online. Association for Computational Linguistics.
  * Wu et al. (2019) Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019.  Transferable multi-domain state generator for task-oriented dialogue systems.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 808–819, Florence, Italy. Association for Computational Linguistics.
  * Wu et al. (2022) Yangjun Wu, Han Wang, Dongxiang Zhang, Gang Chen, and Hao Zhang. 2022.  Incorporating instructional prompts into a unified generative framework for joint multiple intent detection and slot filling.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 7203–7208, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Xu et al. (2023) Jiashu Xu, Mingyu Derek Ma, and Muhao Chen. 2023.  Can NLI provide proper indirect supervision for low-resource biomedical relation extraction? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics_ , Toronto, Canada. Association for Computational Linguistics.
  * Yang et al. (2022) Yuting Yang, Wenqiang Lei, Juan Cao, Jintao Li, and Tat-Seng Chua. 2022.  Prompt learning for few-shot dialogue state tracking.  _arXiv preprint arXiv:2201.05780_.
  * Ye et al. (2021) Fanghua Ye, Jarana Manotumruksa, Qiang Zhang, Shenghui Li, and Emine Yilmaz. 2021\.  Slot self-attentive dialogue state tracking.  In _Proceedings of the Web Conference 2021_ , pages 1598–1608.
  * You et al. (2020) Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2020\.  Large batch optimization for deep learning: Training bert in 76 minutes.  In _International Conference on Learning Representations_.
  * Zang et al. (2020) Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020.  MultiWOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines.  In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_ , pages 109–117, Online. Association for Computational Linguistics.
  * Zhao et al. (2021) Jeffrey Zhao, Mahdis Mahdieh, Ye Zhang, Yuan Cao, and Yonghui Wu. 2021.  Effective sequence-to-sequence dialogue state tracking.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7486–7493, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhou and Small (2019) Li Zhou and Kevin Small. 2019.  Multi-domain dialogue state tracking as dynamic knowledge graph enhanced question answering.  _ArXiv_ , abs/1911.06192.
  * Zhu et al. (2022) Qi Zhu, Bing Li, Fei Mi, Xiaoyan Zhu, and Minlie Huang. 2022.  Continual prompt tuning for dialog state tracking.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1124–1137, Dublin, Ireland. Association for Computational Linguistics.
  * Zhu et al. (2021) Yaoming Zhu, Jiangtao Feng, Chengqi Zhao, Mingxuan Wang, and Lei Li. 2021.  Counter-interference adapter for multilingual machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2812–2823, Punta Cana, Dominican Republic. Association for Computational Linguistics.

##  Appendix A Design Details

###  A.1 Slot Type Definitions

The slot types are defined according to output space and the number of
possible answers. The slot types are defined in Table 4.

Slot Types |  Slots
---|---
Categorical |  attraction-area, hotel-area, hotel-internet, hotel-parking, hotel-price range, hotel-type, restaurant-area, restaurant-price range, train-day
Day |  hotel-book day, restaurant-book day
Number |  hotel-book people, hotel-book stay, hotel-stars, restaurant-book people, train-book people
Open |  attraction-open, attraction-type, hotel-name, restaurant-food, taxi-departure, taxi-destination, train-departure, train-destination
Time |  restaurant-book time, taxi-arrive by, taxi-leave at, train-leave at
Table 4: Slot type definitions.

###  A.2 Question Prompt and Description

We show questions (used as question prompts) and description (as part of
prefix prompt) for each slot in Table 12. Slot descriptions are from MultiWOZ
2.2 dataset Zang et al. (2020).

###  A.3 Detailed Parameter Count

The average parameter count across all domains is 271K. We show detailed
parameter count for each domain in Table 5. The parameters needed for each
domain vary because the question and prefix prompt can map to a different set
of prompt tokens for each domain. The parameters needed for each domain are
calculated by adding prompt token embedding size ($prompt\ token\ count\times
1024$) with segment embedding size ($8\times 1024$ given 8 segments).

| Attr. | Hotel | Rest. | Taxi | Train
---|---|---|---|---|---
Domain | 5 | 20 | 20 | 10 | 10
Slot | 15 | 200 | 140 | 40 | 60
Type | 10 | 80 | 100 | 20 | 40
Question | 20 | 46 | 36 | 19 | 27
Prefix | 60 | 117 | 84 | 29 | 76
All | 110 | 463 | 380 | 118 | 213
Params # | 120832 | 482304 | 397312 | 129024 | 226304
Table 5:  The number of parameters needed for each domain. We list the number
of prompt tokens needed for each prompt segment, all prompt tokens needed and
the ultimate parameter count.

##  Appendix B Additional Experimental Results

###  B.1 Additional Ablation Study for Reiteration

We show an additional ablation study to investigate the effect of introducing
the reiteration technique in Table 6. We observe that the reiteration
technique can lead to a significant increase in performance, especially with
fewer amount of training dialogues. When there are limited training data,
reiteration can help the model learn task boundaries among each slot faster
and better.

Few-shot | Model |  | Attr. | Hotel | Rest. | Taxi | Train | Avg
---|---|---|---|---|---|---|---|---
5 | w/o reit. |  | 22.16 |  | 12.09 |  | 16.67 |  | 47.68 |  | 4.97 |  | 20.71
w/ reit. |  | 33.56 |  | 15.63 |  | 19.76 |  | 51.11 |  | 18.95 |  | 27.80
10 | w/o reit. |  | 23.08 |  | 12.39 |  | 13.75 |  | 56.39 |  | 9.26 |  | 22.97
w/ reit. |  | 39.41 |  | 18.18 |  | 24.72 |  | 59.63 |  | 30.95 |  | 34.58
1% | w/o reit. |  | 45.08 |  | 27.57 |  | 33.48 |  | 59.89 |  | 51.08 |  | 43.42
w/ reit. |  | 47.28 |  | 33.01 |  | 34.40 |  | 60.33 |  | 52.05 |  | 45.41
Table 6:  Ablation study for the reiteration technique.

###  B.2 Performance vs Ontology Size

We investigate the relationship between performance and the number of unique
candidate answers (ontology size) using 1% target domain training data and
Figure 4 demonstrates the result with trendlines created by expanding average
algorithm for each model. We also show the performance of two generative
baseline models for comparison. We observe that the performance of all three
models drops when the ontology size grows. For most ontology sizes, our model
outperforms Li et al. and T5DST Lin et al. (2021b).

Figure 4: Performance for slots with different ontology sizes

###  B.3 Performance by Slots

Figure 5: Slot accuracy difference between our model and Li et al. (2021), a
positive value indicates our model is better.

To better understand the pros and cons of the prompt tuning method compared
with fine-tuning LM, we show the slot accuracy difference for all unique slots
training with 1% target domain data in Figure 5. Our model outperforms Li et
al. (2021) the most in categorical-type “area” slots, and number-type slots
“book people”, all with at least 10% higher accuracy. Our model falls behind
in 9 out of 30 slots, especially for the “restaurant: book time” slots.

Figure 6: Slot accuracy for each slot across different domains

We show the slot accuracy for each unique slot in all five domains using 1%
target domain training data with comparisons to generative baselines in Figure
6.

###  B.4 Closest Tokens of Learned Prompt Tokens

Table 13 shows the full list of closest tokens for the most updated prompt
tokens of each prompt segment in all five domains. We produce the closest
tokens with the smallest cosine similarity between the learned prompt token
embedding and frozen token embeddings of the PLM.

###  B.5 Variances of the Few-Shot Experimental Results

The variances of the experimental results reported in Table 1 (in the order of
using 5, 10, 20 conversations and 1%, 5%, 10% of training data):

  * •

Attraction: 0.27, 0.33, 0.35, 0.30, 0.22, 0.32

  * •

Hotel: 0.52, 0.49, 0.55, 0.57, 0.61, 0.58

  * •

Restaurant: 0.63, 0.72, 0.81, 0.79, 0.83, 0.80

  * •

Taxi: 0.54, 0.61, 0.54, 0.48, 0.52, 0.69

  * •

Train: 0.68, 0.72, 0.73, 0.52, 0.49, 0.55

###  B.6 Comparison with Frozen LM Version of Baselines

Since there are many design choices to make to create a frozen LM version of
the baselines (such as whether to add prefix prompt tokens, how to map tokens
in the prompt segment to the underlying parameters etc), such variations would
almost become new models. In our experiments, we show that our model
outperforms existing models (optimizing all parameters) in low-resource
settings, and we are confident that our model outperforms their frozen LM
version with even larger gaps given the assumption that simply removing
trainable parameters hurts the performance. We quantify such gaps by comparing
the frozen and unfrozen version of the baseline Li et al. with our model in
Table 7.

Model | Attr. | Hotel | Rest. | Taxi | Train
---|---|---|---|---|---
Li et al. (frozen LM) | 29.16 | 14.81 | 15.14 | 47.56 | 35.77
Li et al. | 42.18 | 24.04 | 30.70 | 58.26 | 45.32
Ours | 47.28 | 33.01 | 34.40 | 60.33 | 52.05
Table 7:  Comparison with the frozen LM variation of the baseline. JGA (%)
using 1% training data for each domain.

###  B.7 More about Error Analysis

Figure 3 shows the error distribution in terms of the proportion of each error
category, rather than the absolute error case counts. Though in Figure 3, the
omission error produced by our model takes the larger proportions in all five
slot types compared with Li et al., our model actually makes fewer absolute
omission errors than Li et al. in the “categorical” and “day” slot types, as
shown in Table 8.

Model | Open | Time | Number | Categorical | Day
---|---|---|---|---|---
Li et al. | 12.9 | 6.2 | 2.5 | 7.2 | 2.8
Ours | 16.4 | 8.3 | 4.8 | 5.6 | 2.3
Table 8:  Omission error counts divided by all testing instances (%) when
training with 1% data.

We additionally show Slot Accuracy (SA) on the non-empty testing instances in
Table 9. The result suggests that our model performs better than Li et al. in
4 out of 5 domains except for the “Taxi” domain, which is the most “none”
dominated one.

Model | Attr. | Hotel | Rest. | Taxi | Train
---|---|---|---|---|---
Li et al. | 55.34 | 66.37 | 72.17 | 24.96 | 84.70
Ours | 61.58 | 75.63 | 78.06 | 19.74 | 85.61
Table 9:  Slot Accuracy (%) on non-empty instances when training with 1% data.

Both observations indicate that even if omission error occupies more relative
proportions of the error cases, our model is able to generate a reasonable
amount of not-none values for non-empty slots compared with Li et al. in most
domains.

##  Appendix C Details of Implementation and Experiments

###  C.1 Implementation Details

We apply different learning rate optimization for the parameters of each
prompt segment. We use separate prompt embeddings for each prompt segment,
meaning even if the same token appears in the prefix and question segments
during initialization, it maps to different prompt embeddings for a larger
optimization space. We use GPT2-medium with 1024 hidden states as our default
model. We use the BPE tokenizer to convert the input sequence to tokens. We
set the maximum sequence length to 1024. If the input sequence exceeds the
maximum length, we cut the earlier part of the dialogue history while keeping
the full other partition. Only the exact match between the generated sequence
and the ground-truth slot value counts as a correct prediction. We use greedy
decoding to generate the predicted sequence, and we stop the generation either
when <|endoftext|> token is generated or the output length reaches 20. We
choose the best epoch by monitoring the JGA of the development set.

Our entire codebase is implemented in PyTorch.777https://pytorch.org/ The
implementations of the transformer-based models are extended from the
Huggingface888https://github.com/huggingface/transformers codebase Wolf et al.
(2020).

###  C.2 Number of Task Prompt Tokens

We explore various values for the number of task prompt tokens used by the
domain, slot and type prompt segments, and we show the hyper-parameters that
lead to the best performance in Table 10. We observe that the domains with
fewer unique slots (such as the attraction domain with just 3 unique slots)
need much fewer prompt tokens than the domains with more unique slots (such as
hotel and restaurant with 10 and 7 unique slots respectively). The more
special prompt tokens needed, the more the parameter numbers are.

Model | Attr. | Hotel | Rest. | Taxi | Train
---|---|---|---|---|---
w/ reiteration | 5 | 20 | 20 | 10 | 10
w/o reiteration | 5 | 5 | 20 | 5 | 20
Table 10:  Best prompt numbers for each domain.

###  C.3 Experiment Details

We report the averaged result for three runs with different random seeds for
each experiment. In the ablation study shown in Table 2, for lines 1-3, we
directly remove the corresponding prompt segment from the input sequence; for
lines 4-5, we keep the prefix and question text in the input sequence but use
token embeddings rather than prompt embeddings to get initial token
representation. In the prompt token semantic analysis in Table 3, we select
the most changed prompt tokens by calculating the L2 norm of the difference of
the learned and initialized prompt token embeddings.

All the models in this work are trained on a single Nvidia A6000 GPU on a
Ubuntu 20.04.2 operating system. We show the hyperparameter search range and
best hyperparameter setting in Table 11.

Hyperparameter | Search Range | Best
---|---|---
Number of task prompt tokens | 1, 2, 3, 5, 10, 15, 20, 25, 30 | See Table 10
Prompt initialization | random, token embedding of task name | token embedding of task name
Batch size | 1, 2, 3, 4 | 4
Learning rate | 1e-2, 5e-3, 1e-3, 5e-4, 1e-4, 5e-5, 1e-5 | 1e-3
Decoding method | beam search, greedy | greedy
Surface form for empty slot | “none”, “not mentioned”, “” | “none”
Optimizer | Adam, Lamb You et al. (2020) | Adam
Early stopping patience epochs |  | 8
Learning rate scheduler |  | ReduceLROnPlateau with 5 patience epochs
Max epochs |  | 100
Table 11: Hyperparameter search range and the best setting.

###  C.4 Details of the Baseline Models

We produce the result of Li et al. and Lee et al. with our own
reimplementation with our experimental setting and obtain the results of T5DST
Lin et al. (2021b) by running their codebase with our setting. We verify the
correctness of our reproduction and we are able to reproduce the performance
claimed in their papers under their settings. We report performance for TRADE
and DSTQA from their papers. For Li et al., we use GPT2-medium as the backbone
PLM and do not use DSTC8 for transfer learning as it would introduce
additional data resources and make the comparison not fair. For T5DST, we use
the best setting concluded by the authors that includes slot type information
in the input sequence. We use T5-small with 60M parameters which has 6
encoder-decoder layers and the hidden size of 512 as the backbone PLM. For Lee
et al., we use T5-small as the backbone, we include slot description from
MultiWoZ 2.2, possible slot values from dialogue ontology and no domain
description in the natural language augmented prompt.

Domain | Slot | Question | Description
---|---|---|---
Attraction | area | In what area is the user looking for an attraction? | area to search for attractions
name | What is the name of the attraction the user prefers? | name of the attraction
type | What type of attraction does the user prefer? | type of the attraction
Hotel | area | In what area is the user looking for a hotel? | area or place of the hotel
book day | The user is looking for a hotel starting what day of the week? | day of the hotel booking
book people | How many people does the user need a hotel booking for? | number of people for the hotel booking
book stay | How many days does the user prefer to stay at a hotel? | length of stay at the hotel
internet | Does the user want internet in their hotel? | whether the hotel has internet
name | What is the name of the hotel the user prefers? | name of the hotel
parking | Does the user need the hotel to have parking? | whether the hotel has parking
price range | What is the price range of the hotel the user prefers? | price budget of the hotel
stars | The user prefers a hotel with what star rating? | star rating of the hotel
type | What type of hotel does the user prefer? | what is the type of the hotel
Restaurant | area | In what area is the user looking for a restaurant? | area or place of the restaurant
book day | The user is looking for a restaurant for what day of the week? | day of the restaurant booking
book people | How many people does the user need a restaurant booking for? | how many people for the restaurant reservation
book time | What time does the user want to book a restaurant? | time of the restaurant booking
food | The user prefers a restaurant serving what type of food? | the cuisine of the restaurant you are looking for
name | What is the name of the restaurant the user prefers? | name of the restaurant
price range | What is the price range of the restaurant the user prefers? | price budget for the restaurant
Taxi | arrive by | What time does the user want to arrive by taxi? | arrival time of taxi
departure | Where does the user want the taxi to pick them up? | departure location of taxi
destination | Where does the user want to go by taxi? | destination of taxi
leave at | What time does the user want the taxi to pick them up? | leaving time of taxi
Train | arrive by | What time does the user want to arrive by train? | arrival time of the train
book people | How many people does the user need train bookings for? | how many train tickets you need
day | What day does the user want to take the train? | day of the train
departure | Where does the user want to leave from by train? | departure location of the train
destination | Where does the user want to go by train? | destination of the train
leave at | What time does the user want the train to leave? | leaving time for the train

Table 12:  Question and description used in the input sequence for each slot  Domain | Prompt segment | Prompt token | Closest tokens
---|---|---|---
Attraction | Domain | <domain_attraction_4> | raction; ractions; racted; ract; ractive
<domain_attraction_0> | att; Att; ATT; atts; atten
Slot | <slot_name_2> | name; Name; names; NAME; named
<slot_type_4> | type; types; Type; style; TYPE
Type | <type_open_3> | open; Open; opened; opens; opening
<type_categorical_3> | ateg; orical; orically; ategy; ategic
Prefix | special | special; Special; statistical; SPECIAL; remarkable
site | site; sites; website; Site; webpage
Question | Q | answer; Answer; answered; answers; Q
attraction | attraction; attractions; fascination; attractiveness; attracted
Hotel | Domain | <domain_hotel_11> | cogn; izoph; nostalg; contrad; Alas
<domain_hotel_19> | enment; Alas; ishy; ridic; minent
Slot | <slot_parking_13> | Pear; Aqua; Icar; Mermaid; Omega
<slot_internet_17> | internet; Wi; Internet; WiFi; VPN
Type | <type_number_14> | regex; NUM; abulary; pmwiki; printf
<type_open_3> | open; Dar; Ezek; Zur; Citiz
Prefix | ). | .).; ).; ].; .}; .</; .]; .); .’; }.; .)
yes | yes; Yes; YES; yeah; ye
Question | hotel | cannabis; sushi; Tinder; whiskey; booze
days | days; hours; consequences; minutes; Days
Restaurant | Domain | <domain_restaurant_7> | rest; Rest; urnal; restrial; restling
<domain_restaurant_16> | rest; Rest; Rest; Text; Funds
Slot | <slot_name_3> | name; Name; names; named; NAME
<slot_area_14> | area; Area; But; At; <|endoftext|>
Type | <type_categorical_4> | orical; brut; oric; ateg;
<type_day_15> | day; Day; week; DAY; month
Prefix | west | west; West; southwest; Southwest; northwest
booking | booking; reservation; insult; reverence; audition
Question | name | exting; bookstore; describ; mascara; homepage
restaurant | Deadpool; Bitcoin; Veg; steak; Hollywood
Taxi | Domain | <domain_taxi_6> | i; a; o; I; in
<domain_taxi_9> | tax; Tax; taxes; Taxes; taxed
Slot | <slot_destination_8> | dest; Dest; destination; Destination
<slot_departure_4> | ure; ures; URE; ured; uring
Type | <type_time_0> | time; Time; TIME; timer; year
<type_open_6> | open; Open; opens; opening; closed
Prefix | open | open; Open; opened; opens; OPEN
rival | rival; rivals; quickShip
Question | taxi | taxi; taxis; Taxi; cab; Uber
does | does; is; doesn; has; isn
Train | Domain | <domain_train_8> | train; Train; genre; disciplinary; trained
<domain_train_7> | train; Train; trainers; trains; trained
Slot | <slot_destination_8> | dest; kosher; GMO; mill; JFK; okemon
<slot_arriveby_3> | by; By; BY; While; from
Type | <type_time_0> | time; groupon; times; TIME; wasteful
<type_day_6> | day; Day; week; month; year
Prefix | dest | dest; goto; inion; externalTo; Destination
location | location; locations; geographic; geography; geographical
Question | from | graphs; ancestry; statistics; backstory; stats
train | train; plane; railway; subway; highway
Table 13:  Closest tokens for the learned prompt tokens.


# A programmable $\bm{k\cdot p}$ Hamiltonian method and application to
magnetic topological insulator MnBi2Te4

Guohui Zhan1, Minji Shi1, Zhilong Yang1, Haijun Zhang1,2∗ 1 National
Laboratory of Solid State Microstructures, School of Physics, Nanjing
University, Nanjing 210093, China
2 Collaborative Innovation Center of Advanced Microstructures, Nanjing
University, Nanjing 210093, China<EMAIL_ADDRESS>
###### Abstract

In the band theory, first-principles calculations, the tight-binding method
and the effective $\bm{k\cdot p}$ model are usually employed to investigate
the electronic structure of condensed matters. The effective $\bm{k\cdot p}$
model has a compact form with a clear physical picture, and first-principles
calculations can give more accurate results. Nowadays, it has been widely
recognized to combine the $\bm{k\cdot p}$ model and first-principles
calculations to explore topological materials. However, the traditional method
to derive the $\bm{k\cdot p}$ Hamiltonian is complicated and time-consuming by
hand. In this work, we independently developed a programmable algorithm to
construct effective $\bm{k\cdot p}$ Hamiltonians for condensed matters.
Symmetries and orbitals are used as the input information to produce the
one-/two-/three-dimensional $\bm{k\cdot p}$ Hamiltonian in our method, and the
open-source code can be directly downloaded online. At last, we also
demonstrated the application to MnBi2Te4-family magnetic topological
materials.

Recently, the study of topological states and topological materials, such as,
topological insulators, topological semimetals and topological
superconductors, has become an important topic in condensed matter physics,
and great progresses have been achieved in both experiments and theoriesHasan
and Kane (2010); Qi and Zhang (2011); Armitage  _et al._ (2018). The developed
topological band theory with first-principles calculations played a key role
in the past years. For example, topological invariants, such as
$\mathcal{Z}_{2}$, were defined through the band theoryFu and Kane (2007). The
concept of band inversion of topological states was birth from the band
theoryBernevig  _et al._ (2006); Zhang and Zhang (2013). Also, the topological
boundary states were successfully predicted through first-principles
calculationsZhang  _et al._ (2009). However, the first-principles band
structures seem like a black box with much-hidden information, which results
in a barrier to deeply understand the essential physical pictures of
topological states. Differently, the $\bm{k\cdot p}$ Hamiltonian has a simple
form and a clear physical picture, which is a necessary and useful supplement
for the first-principles calculationsLuttinger and Kohn (1955); Kane (1966);
Voon and Willatzen (2009); Liu  _et al._ (2010). With the $\bm{k\cdot p}$
Hamiltonian, Fu predicted an unconventional hexagonal warping term in surface
states of topological insulator Bi2Te3Fu (2009). Therefore, the combination of
first-principles calculations and the $\bm{k\cdot p}$ Hamiltonian has become a
standard paradigm to theoretically study topological states and topological
materials Zhang and Zhang (2013); Zhang  _et al._ (2009); Xu  _et al._ (2011).

Based on the group theory and the $\bm{k\cdot p}$ perturbation theory, Liu et
al. provided an instructive demonstration showing how to construct the
$\bm{k\cdot p}$ Hamiltonian of Bi2Se3-family three-dimensional topological
insulatorsLiu  _et al._ (2010). However, though the derivation is a standard
method, the detailed process is quite troublesome and time-consuming by hand,
especially for dealing with high-order terms of $\bm{k\cdot p}$ Hamiltonians.
Therefore, it is practicable and highly necessary to develop automated
programmable methods to efficiently construct $\bm{k\cdot p}$ Hamiltonians. In
this context, there have been some innovative proposes, such as, kdotp-
symmetry code developed by Gresch $et$ $al.$Gresch (2018, ), Qsymm Python
package developed by Varjas $et$ $al.$et al (2018); Varjas . In this letter,
based on group theory, we independently developed a programmable algorithm to
construct $\bm{k}\cdot\bm{p}$ Hamiltonians for all one-/two-/three-dimensional
materials (the open-source code can be download from
https://github.com/shimj/Model-Hamiltonian). With this automated algorithm,
only the crystal symmetry and atomic orbitals involved are needed to produce
the $\bm{k\cdot p}$ Hamiltonian, which effectively avoids time-consuming
calculations and latent mistakes. The schematic of our programmable method is
shown in Fig. 1. We have successfully applied our method to MnBi2Te4-family
magnetic topological materialsZhang  _et al._ (2019, 2020); Wang  _et al._
(2020).

Basic $\bm{k}\cdot\bm{p}$ Hamiltonian theory. In the band theory, the
wavefunction in a periodic lattice is written as the Bloch wavefunction,
$\psi_{\bm{k}}(\bm{r})=u_{\bm{k}}(\bm{r})e^{i\bm{k}\cdot\bm{r}}$. Here,
$u_{\bm{k}}(\bm{r})$ is a periodic function with
$u_{\bm{k}}(\bm{r})=u_{\bm{k}}(\bm{r}+\bm{R})$ where $\bm{R}$ indicates the
lattice vector. By substituting the Bloch wavefunction $\psi_{\bm{k}}$ into
the Schrödinger equation, we have

$\mathcal{H}_{\bm{k}}u_{\bm{k}}=E_{\bm{k}}u_{\bm{k}},$ (1)

where the Hamiltonian has the $\bm{k}$-dependent form
$\mathcal{H}_{\bm{k}}=p^{2}/2m+V+\hbar\bm{k\cdot p}/m_{0}+\hbar^{2}k^{2}/2m$.
Generally, the representation matrix of Hamiltonian $\mathcal{H}_{\bm{k}}$
near a $\bm{k_{0}}$, with $\bm{k}=\bm{k_{0}}+\delta\bm{k}$, can be approached
by Taylor series as

$\mathcal{H}_{\bm{k_{0}}+\delta\bm{k}}\doteq\sum_{\alpha+\beta+\gamma\leq
n}(\delta k_{x})^{\alpha}(\delta k_{y})^{\beta}(\delta
k_{z})^{\gamma}\Gamma_{\alpha\beta\gamma},$ (2)

where $\Gamma_{\alpha\beta\gamma}$ are constant matrices, $n$ indicates the
order of Taylor series expansion. In the following, we will derive the
representation matrix of $\mathcal{H}_{\bm{k}}$ from the commutation relation
between Hamiltonian and symmetry operations.

Figure 1:  Workflow for constructing the $\bm{k}\cdot\bm{p}$ effective
Hamiltonian. The selected generators of the group and atomic orbitals are the
input information to run the program. In the process, the key point is getting
the representation matrix $\mathcal{F}$ and $\mathcal{M}$. By solving
$(\mathcal{F}\otimes\mathcal{M}-I)\alpha=0$, the Hermitian matrices can be
obtained from the solution $A$.

When a periodic condensed matter system preserves a $S$ symmetry, its
Hamiltonian $\mathcal{H}$ satisfies the commutation relation
$[\mathcal{H},S]=0$. In $\bm{k}$-space, we have

$S\mathcal{H}_{\bm{k}}S^{-1}=\mathcal{H}_{\gamma\bm{k}},$ (3)

where

$\gamma=\begin{cases}\tilde{g}&S=Q\\\ -1&S=\mathcal{T}\\\
-\tilde{g}&S=Q\mathcal{T}\end{cases}.$ (4)

Here, $Q$ represents an operator corresponding to the spatial operation
$g=\\{\tilde{g}|\tau\\}$ and $\mathcal{T}$ (= $(-i\sigma_{y}\otimes
I)\mathcal{K}$) is the time reversal operator, $\tilde{g}$ is a rotation or
reflection operator and $\tau$ is a translation vector. It is noteworthy that
$2\pi$-rotation for a spinful system is not included in our discussion, since
it leads to an identity $(-1)\mathcal{H}_{\bm{k}}(-1)=\mathcal{H}_{\bm{k}}$
without giving any constraint to the Hamiltonian $\mathcal{H}_{\bm{k}}$. In
principle, any complete set of lattice-periodic functions, for example
$\\{e^{i\bm{G}\cdot\bm{r}}\\}$, can be used as basis to accurately solve the
Schrödinger equation
$\mathcal{H}_{\bm{k}}|u_{\bm{k}}\rangle=E_{\bm{k}}|u_{\bm{k}}\rangle$. Since
we are interested in the bands around a certain point $\bm{k}_{0}$ in the
Brillouin Zone (BZ), it allows us to use $\\{|u_{i}^{\bm{k}_{0}}\rangle\\}$ as
a basis to approximately solve the Schrödinger equation in the framework of
the perturbation theory, where $i$ indicates the band index. Once we know the
symmetry transformations of these states, the matrix representation of
$\mathcal{H}_{\bm{k}}$ under this basis can then be determined in form, thus
$E_{\bm{k}}$ and the vector $|u_{\bm{k}}\rangle$.

Around a high-symmetry point $\bm{k}_{0}$, for a symmetry operation $S$
meeting $\gamma\bm{k}_{0}=\bm{k}_{0}$, we can rewrite Eq. (3) as

$S\mathcal{H}_{\bm{k}_{0}+\delta\bm{k}}S^{-1}=\mathcal{H}_{\bm{k}_{0}+\gamma\delta\bm{k}}.$
(5)

First of all, the symmetry operations under consideration have to keep the
linear space spanned by $\\{|u_{i}^{\bm{k}_{0}}\rangle\\}$ invariant.
Otherwise, we must either drop some symmetry operations or add more states
into the basis. With the representation matrix $D(S)$ for symmetry operations
$S$ defined as

$S^{u}(|u^{\bm{k}_{0}}_{1}\rangle,|u^{\bm{k}_{0}}_{2}\rangle,...)=(|u^{\bm{k}_{0}}_{1}\rangle,|u^{\bm{k}_{0}}_{2}\rangle,...)D(S^{u}),$
(6a)
$S^{a}(|u^{\bm{k}_{0}}_{1}\rangle,|u^{\bm{k}_{0}}_{2}\rangle,...)=(|u^{\bm{k}_{0}}_{1}\rangle,|u^{\bm{k}_{0}}_{2}\rangle,...)D(S^{a})\mathcal{K},$
(6b)

where we have explicitly marked the symmetry operation $S$ by $u$ for
“unitary” and $a$ for “antiunitary”, Eq. (5) immediately gives

$D(S^{u})\mathcal{H}_{\bm{k}_{0}}(\gamma^{-1}\delta\bm{k})D(S^{u})^{-1}=\mathcal{H}_{\bm{k}_{0}}(\delta\bm{k}),$
(7a)
$D(S^{a})\mathcal{H}_{\bm{k}_{0}}(\gamma^{-1}\delta\bm{k})^{*}D(S^{a})^{-1}=\mathcal{H}_{\bm{k}_{0}}(\delta\bm{k}).$
(7b)

where $\mathcal{H}_{\bm{k}_{0}}(\delta\bm{k})$ denotes
$\mathcal{H}_{\bm{k}_{0}+\delta\bm{k}}$.

LCAO basis. In group theory, for a linear space invariant under a group, one
can always find a set of basis in which all the vectors can form irreducible
representations of that group and are orthogonal with each other. For a
general vector, through analyzing its components, one can easily find all
other vectors related by the group operations and all the irreducible
representations. Since the first-principles calculations can give the atomic-
orbital projection of the wavefuction, it would be useful to set up a database
containing the representations of all low-level linear combinations of atomic
orbitals (LCAOs).

First of all, we ignore the spin. The periodic part of linear combination of
atomic orbitals has the form

$\displaystyle u^{\bm{k}}_{m\alpha}(\bm{r})$
$\displaystyle=\frac{1}{\sqrt{N}}\sum_{\bm{R}_{n}}e^{-i\bm{k}\cdot(\bm{r}-\bm{r}_{\alpha}-\bm{R}_{n})}\phi_{m}(\bm{r}-\bm{r}_{\alpha}-\bm{R}_{n}),$
(8)

where $\alpha$ and $m$ respectively refers to the $\alpha$-th atom in a unit
cell and its $m$-th atomic orbital. Here we also defined
$\phi^{\prime}_{m}(\bm{r})=e^{-i\bm{k}\cdot\bm{r}}\phi_{m}(\bm{r})/\sqrt{N}$.
It should be noted that if we collect the periodic wavefunctions for all the
same atoms in a unit cell and all the atomic orbitals with the same quantum
number $\ell$ to span a linear space, it must be invariant under symmetry
operations.

To simplify the calculation, we analyze the indices $m$ and $\alpha$
separately. To do that, we need two linear spaces: a complex linear space
$V_{1}$ spanned by $\\{\phi^{\prime}_{m}(\bm{r})\\}$ and
$V_{2}=\mathbb{R}^{\alpha_{\text{max}}}$, where $\alpha_{\text{max}}$ means
the number of atoms in our consideration of a unit cell and the vectors in it
indicate the spatial distribution of atoms. Then with the correspondence as

$\displaystyle\phi^{\prime}_{m}(\bm{r})\otimes(\overbrace{0,...,0}^{\alpha-1},1,0,...,0)^{T}$
$\displaystyle\leftrightarrow$
$\displaystyle\sum_{\bm{R}_{n}}\phi^{\prime}_{m}(\bm{r}-\bm{r}_{\alpha}-\bm{R}_{n}),$
(9)

we know that the tensor product of these two spaces $V_{1}\otimes V_{2}$ is
isomorphic to the space $V_{0}$ spanned by
$\\{u^{\bm{k}}_{m\alpha}(\bm{r})\\}$ for a certain $\bm{k}$. Finally, with the
straightforward correspondence

$\displaystyle
S\sum_{\bm{R}_{n}}\phi^{\prime}_{m}(\bm{r}-\bm{r}_{\alpha}-\bm{R}_{n})$
$\displaystyle\leftrightarrow$ $\displaystyle
S^{P}\phi^{\prime}_{m}(\bm{r})\otimes
S(\overbrace{0,...,0}^{\alpha-1},1,0,...,0)^{T},$ (10)

where $S^{P}$ is the translation-deleted part of $S$. One can easily find the
resulting state of any symmetry operations on $u^{\bm{k}}_{m\alpha}(\bm{r})$.
Above all, the direct product between the represenations of $V_{1}$ and
$V_{2}$ is that of $V_{0}$.

When taking the spin into account, we could regard $V$ as $V_{1}\otimes
V_{2}\otimes V_{\text{spin}}$ . In this way, the discussion above can be
immediately generalized to the situation with spin, where we must consider the
effect of $S^{P}$ on the spin part.

For different systems, the representations formed by $V_{2}$ differ from each
other, but they are mostly direct sum of pure one-dimensional representations.
Thus in the database, we only collect matches between representations of the
atomic orbital with or without spin and all one-dimensional representations.

Expansion with Hermitian matrices. Considering the hermiticity of
$\mathcal{H}_{\bm{k}_{0}}(\delta\bm{k})$, it can be expanded by $n^{2}$
linearly independent Hermitian matrices with real coefficients
$h(\delta\bm{k})$, and then we can obtain $n^{2}$ independent equations. We
denote these Hermitian matrices by $\\{B_{i},i=1,2,...,n^{2}\\}$. They span a
linear space on real number field. Then
$\mathcal{H}_{\bm{k}_{0}}(\delta\bm{k})$ can be expressed as

$\mathcal{H}_{\bm{k}_{0}}(\delta\bm{k})=\sum_{i}h_{i}(\delta\bm{k})B_{i},$
(11)

Therefore, the Eq. (7) can be expressed as

$\sum_{j}h_{j}(\gamma^{-1}\delta\bm{k})D(S^{u})B_{j}D(S^{u})^{-1}=\sum_{i}h_{i}(\delta\bm{k})B_{i},$
(12a)
$\sum_{j}h_{j}(\gamma^{-1}\delta\bm{k})D(S^{a})B^{*}_{j}D(S^{a})^{-1}=\sum_{i}h_{i}(\delta\bm{k})B_{i},$
(12b)

Now we define two new matrices $\mathcal{M}$ and $\mathcal{N}$ as follows,

$D(S^{u})B_{j}D(S^{u})^{-1}=\sum_{m}\mathcal{M}_{mj}B_{m},$ (13a)
$D(S^{a})B^{*}_{j}D(S^{a})^{-1}=\sum_{m}\mathcal{M}_{mj}B_{m},$ (13b)
$h_{j}(\gamma^{-1}\delta\bm{k})=\sum_{n}\mathcal{N}_{nj}h_{n}(\delta\bm{k}),$
(14)

Comparing the coefficients of $B_{i}$ on both sides of the Eq. (12), it yields
$\sum_{n}(\mathcal{MN}^{T})_{in}h_{n}(\delta\bm{k})=h_{i}(\delta\bm{k})$ for
$i=1,2,...,n^{2}$, that is

$\mathcal{N}^{T}h(\delta\bm{k})=\mathcal{M}^{-1}h(\delta\bm{k})$ (15)

On one hand, if the basis $\\{|u^{\bm{k}_{0}}_{i}\rangle,i=1,2,...,n\\}$ is
orthonormal, it is obvious that $D$ are unitary matrices
($D^{-1}=D^{\dagger}$), which implies that $D(S^{u})B_{j}D(S^{u})^{-1}$ are
vectors in Hermitian matrix space on real number field, Thus $\mathcal{M}$
must be real. On the other hand, if we define the inner product of Hermitian
matrices as $\text{tr}(B_{i}B_{j}^{\dagger})$. It can be proved that

$\displaystyle\text{tr}(D(S^{u})B_{i}D(S^{u})^{-1}(D(S^{u})B_{j}D(S^{u})^{-1})^{\dagger})$
$\displaystyle=$ $\displaystyle\text{tr}(B_{i}B_{j}^{\dagger}),$ (16)

which means the inner product remains unchanged (the same for $S^{a}$). This
implies that if $\\{B_{i}\\}$ is an orthonormal basis, $\mathcal{M}$ must be
orthogonal ($\mathcal{M}^{-1}=\mathcal{M}^{T}$).

Especially, if the combination of inversion and time reversal symmetry
$\mathcal{PT}$ is in consideration, then we have

$\mathcal{M}(\mathcal{PT})h(\delta\bm{k})=h(\delta\bm{k}),$ (17)

which allows us to simplify the calculation. If we assume the solution of
$[\mathcal{M}(\mathcal{PT})-I]X=0$ is $\\{b_{i},i=1,2,...,m\\}$, where
$m<n^{2}$ always holds, then
$h(\delta\bm{k})=\sum_{i}h^{\prime}_{i}(\delta\bm{k})b_{i}$ and
$H_{\bm{k}_{0}}(\delta\bm{k})=\sum_{i}h^{\prime}_{i}(\delta\bm{k})B^{\prime}_{i}$,
where we denoted $\sum_{j}(b_{i})_{j}B_{j}$ by $B^{\prime}_{i}$. Now we can
recalculate $\mathcal{M}^{\prime}$ with $\\{B^{\prime}_{i},i=1,2,...,m\\}$,
but this time only the $m\times m$ submatrix on the top left corner of
$\mathcal{M}^{\prime}$ is meaningful and actually we can only get this
submatrix with the incomplete set of Hermitian matrices
$\\{B^{\prime}_{i}\\}$.

Expansion of $h(\delta\bm{k})$. Since $h(\delta\bm{k})$ is a polynomial of
$\delta\bm{k}$, we could expand $h(\delta\bm{k})$ by a monomial basis of
$\delta\bm{k}$. Suppose $\\{f_{i}(\delta\bm{k}),i=1,2,...,d\\}$ is the basis,
then $h(\delta\bm{k})$ can be expanded as

$h_{i}(\delta\bm{k})=\sum_{j}f_{j}(\delta\bm{k})A_{ji}.$ (18)

Then with the definition of matrix $\mathcal{F}$ below,

$f_{i}(\gamma^{-1}\delta\bm{k})=\sum_{j}\mathcal{F}_{ji}f_{j}(\delta\bm{k})$
(19)

Eq. (14) gives $\mathcal{F}A=A\mathcal{N}$, and then Eq. (15) gives
$A=\mathcal{F}A\mathcal{M}^{T}$. If defining $\alpha$ as a vector from
stacking up columns of $A^{T}$, we obtain

$(\mathcal{F}\otimes\mathcal{M}-I)\alpha=0.$ (20)

Assuming that the solution is $\alpha=\sum_{l}c_{l}\tilde{\alpha}(l)$, where
$\\{\tilde{\alpha}(l)\\}$ is a basis of the solution space and $c_{l}$ are
real coefficients, then $A=\sum_{l}c_{l}\tilde{A}(l)$, and finally

$H_{\bm{k}_{0}}(\delta\bm{k})=\sum_{i}h_{i}(\delta\bm{k})B_{i}=\sum_{ijl}c_{l}f_{j}(\delta\bm{k}){\tilde{A}(l)}_{ji}B_{i}$
(21)

Note that $\mathcal{F}$ is a block diagonal matrix, since polynomials with
different orders cannot transform to each other. We could separately calculate
$\mathcal{F}^{(n)}$ and $A^{(n)}$ for each Taylor series expansion order
$f^{(n)}(\delta\bm{k})$. Finally, we stack $f^{(n)}(\delta\bm{k})$ and
$A^{(n)}$ respectively to get $f(\delta\bm{k})$ and $A$, that is

$f(\delta\bm{k})=\begin{pmatrix}f^{(0)}(\delta\bm{k})\\\
f^{(1)}(\delta\bm{k})\\\ f^{(2)}(\delta\bm{k})\\\ \vdots\end{pmatrix}$ (22)

and

$\displaystyle\tilde{A}=\sum_{l=1}^{l^{(0)}}c_{l}\begin{pmatrix}\tilde{A}^{(0)}(l)\\\
0\\\ 0\\\
\vdots\end{pmatrix}+\sum_{l=l^{(0)}+1}^{l^{(0)}+l^{(1)}}c_{l}\begin{pmatrix}0\\\
\tilde{A}^{(1)}(l)\\\ 0\\\ \vdots\end{pmatrix}$
$\displaystyle+\sum_{l=l^{(0)}+l^{(1)}+1}^{l^{(0)}+l^{(1)}+l^{(2)}}c_{l}\begin{pmatrix}0\\\
0\\\ \tilde{A}^{(2)}(l)\\\ \vdots\end{pmatrix}+...$ (23)

The case of $\bm{k}_{0}$ on the BZ boundary under a symmetry operation
$\gamma$ with $\gamma\bm{k}_{0}-\bm{k}_{0}\neq 0$. When the $\bm{k}_{0}$ is on
the BZ boundary, the following equation should be taken into account,

$\mathcal{H}_{\bm{k}+\bm{G}}=e^{-i\bm{G}\cdot\bm{r}}\mathcal{H}_{\bm{k}}e^{i\bm{G}\cdot\bm{r}},$
(24)

where $\bm{G}$ is an arbitrary reciprocal lattice vector. With this equation,
the symmetry operation meeting $\gamma\bm{k}_{0}-\bm{k}_{0}=\bm{G}\neq 0$, can
also limit the form of the Hamiltonian around $\bm{k}_{0}$, since

$\displaystyle
S\mathcal{H}_{\bm{k}_{0}+\delta\bm{k}}S^{-1}=\mathcal{H}_{\gamma\bm{k}_{0}+\gamma\bm{\delta}k}=e^{-i\bm{G}\cdot\bm{r}}\mathcal{H}_{\bm{k}_{0}+\gamma\delta\bm{k}}e^{i\bm{G}\cdot\bm{r}}.$
(25)

We can define $\tilde{S}$ as
$\tilde{S}=e^{i\bm{(}\gamma\bm{k}_{0}-\bm{k}_{0})\cdot\bm{r}}S$, which leads
to an equation with the same form as Eq. (5),

$\tilde{S}\mathcal{H}_{\bm{k}_{0}+\delta\bm{k}}\tilde{S}^{-1}=\mathcal{H}_{\bm{k}_{0}+\gamma\delta\bm{k}}.$
(26)

Although the set of symmetry operations $\\{\tilde{S}\\}$ actually describes
the full symmetry of $\mathcal{H}_{\bm{k}_{0}}$, it is not always a group. The
product of two arbitrary elements $\tilde{S}_{1}$ and $\tilde{S}_{2}$ is

$\tilde{S}_{1}\tilde{S}_{2}=e^{i\gamma_{1}(\bm{k}_{0}-\gamma_{2}\bm{k}_{0})\cdot\bm{\tau}_{1}}e^{i(\gamma_{1}\gamma_{2}\bm{k}_{0}-\bm{k}_{0})\cdot\bm{r}}S_{1}S_{2},$
(27)

which is still in $\\{\tilde{S}\\}$ and corresponding to $S_{1}S_{2}$ only if
$e^{i\gamma_{1}(\bm{k}_{0}-\gamma_{2}\bm{k}_{0})\cdot\bm{\tau}_{1}}=1$. Due to
the existence of the symmetry $\gamma$ with $\gamma\bm{k}_{0}\neq\bm{k}_{0}$,
only if all symmetries in consideration are symmorphic ($\tau=0$),
$\\{\tilde{S}\\}$ becomes a group. Otherwise, we either drop the symmetry
operation $\gamma$ with $\gamma\bm{k}_{0}\neq\bm{k}_{0}$ or the nonsymmorphic
symmetry operations in our code.

Figure 2: Crystal structure and band structure of magnetic topological
insulator MnBi2Te4. (a) The unit cell of AFM MnBi2Te4 consists of two SLs. The
red arrows represent the spin moment of Mn atom. The green arrow denotes for
the half translation operator $\tau_{1/2}$. (b)(c)The fat band structures of
FM and AFM states . The blue and red dots denote the characters of Te and Bi
$p_{z}$-orbitals respectively. (d) The unit cell of FM MnBi2Te4 has one SL.
(e) The process of construct FM MnBi2Te4 $\bm{k\cdot p}$ Hamiltonian.

Application to magnetic topological material MnBi2Te4. MnBi2Te4 is a versatile
magnetic topological materials to realize to quantum anomalous Hall (QAH)
state, antiferromagnetic (AFM) topological insulator, magnetic axion
insulator, tunable dynamical axion field and ferromagnetic (FM) Weyl
semimetalZhang  _et al._ (2019); Li  _et al._ (2019); Gong  _et al._ (2019);
Otrokov  _et al._ (2019a, b); Deng  _et al._ (2020); Liu  _et al._ (2020);
Chen  _et al._ (2019); Klimovskikh  _et al._ (2020). Here, with the above
method, we take MnBi2Te4 as an example to demonstrate how to construct the
$\bm{k\cdot p}$ Hamiltonian.

As shown in Fig. 2(a), MnBi2Te4 has a layered crystal structure with a
triangle lattice. The trigonal axis (threefold rotation symmetry $C_{3z}$) is
defined as the $z$ axis, a binary axis (twofold rotation symmetry $C_{2x}$) is
defined as the $x$ axis and a bisectrix axis (in the reflection plane) is
defined as the $y$ axis for the coordinate system. The material consists of
septuple layers (SL) (e.g., Te-Bi-Te-Mn-Te-Bi-Te) arranged along the $z$
direction. The structural of nonmagnetic MnBi2Te4 is described by the space
group $D^{5}_{3d}$ ($R\bar{3}m$) whose generators are $C_{3z}$, $P$, $C_{2x}$
and pure translations which can be ignored since their representations formed
by LCAOs are identities. The first-principles calculations of FM and AFM
states(Fig. 2(b)(c)) show that four bands at the $\Gamma$ point near the
Fermi-level all contains $p_{z}$ orbitals of Bi and Te atom, the basis for
both FM and AFM states can be expressed as
$\\{|\text{Bi},p_{z},\uparrow\rangle^{+},i|\text{Te},p_{z},\uparrow\rangle^{-},|\text{Bi},p_{z},\downarrow\rangle^{+},-i|\text{Te},p_{z},\downarrow\rangle^{-}\\}$,
where the superscripts ‘$+/-$’ indicate parities.

For the FM state, the symmetries are three-fold rotation $C_{3z}$, inversion
symmetry $P$, and the combination of two-fold rotation and time reversal
symmetry $C_{2x}\Theta$, the representation matrices can be obtained by our
code:

$\begin{cases}D(C_{3z})=\text{exp}(-i\frac{\pi}{3}\sigma_{z}\otimes 1_{2\times
2})\\\ D(P)=1_{2\times 2}\otimes\tau_{z}\\\
D(C_{2x}\Theta)=\text{exp}(i\frac{\pi}{2}\sigma_{z}\otimes 1_{2\times
2})\mathcal{K}\end{cases}$ (28)

where $\mathcal{K}$ is the complex conjugation operator.

By using our $\bm{k\cdot p}$ method, we first get 0-order $\bm{k}\cdot\bm{p}$
Hamiltonian around $\Gamma$ point:

$\displaystyle\mathcal{H}(\bm{k})=c_{1}\sigma_{0}\otimes\sigma_{0}+c_{4}\sigma_{0}\otimes\sigma_{z}+c_{13}\sigma_{z}\otimes\sigma_{0}+c_{16}\sigma_{z}\otimes\sigma_{z}$

where $c_{i}(i=1,4,13,16)$ are coefficients.

Further, we can also get the $\bm{k}\cdot\bm{p}$ Hamiltonian up to 2 order as
follows,

$\mathcal{H}_{\text{FM}}(\bm{k})=\begin{pmatrix}M_{1}(\bm{k})&A_{1}k_{z}&0&A_{2}k_{-}\\\
A_{1}k_{z}&M_{2}(\bm{k})&A_{4}k_{-}&0\\\
0&A_{4}k_{+}&M_{3}(\bm{k})&A_{3}k_{z}\\\
A_{2}k_{+}&0&A_{3}k_{z}&M_{4}(\bm{k})\end{pmatrix},$ (30)

where $k_{\pm}=k_{x}\pm ik_{y}$ and
$M_{i}(\bm{k})=M_{0}^{i}+B_{1}^{i}k_{z}^{2}+B_{2}^{i}(k_{x}^{2}+k_{y}^{2})$.
Note that the off-diagonal two-order terms are omitted, since they contribute
higher order of $\bm{k}$ to the energy.

To show the change of the Hamiltonian induced by the FM order, we now assume
both $C_{2x}$ and $\Theta$ are preserved. In addition to $D(C_{3z}),D(P)$, the
representation matrices also include
$D(C_{2x})=\text{exp}[-i(\pi/2)\sigma_{x}\otimes 1_{2\times 2}]$,
$D(\Theta)=i\sigma_{y}\otimes 1_{2\times 2}\mathcal{K}$. Then we can get the
$\bm{k}\cdot\bm{p}$ Hamiltonian of nonmagnetic state

$\mathcal{H}_{\text{NM}}(\bm{k})=\epsilon_{0}(\bm{k})+\begin{pmatrix}M_{\gamma}(\bm{k})&A_{1}k_{z}&0&A_{2}k_{-}\\\
A_{1}k_{z}&-M_{\gamma}(\bm{k})&A_{2}k_{-}&0\\\
0&A_{2}k_{+}&M_{\gamma}(\bm{k})&-A_{1}k_{z}\\\
A_{2}k_{+}&0&-A_{1}k_{z}&-M_{\gamma}(\bm{k})\end{pmatrix},$ (31)

where $\epsilon_{0}(\bm{k})=C+D_{1}k_{z}^{2}+D_{2}(k_{x}^{2}+k_{y}^{2})$ and
$M_{\gamma}(\bm{k})=M_{0}^{\gamma}+B_{1}^{\gamma}k_{z}^{2}+B_{2}^{\gamma}(k_{x}^{2}+k_{y}^{2})$.
Comparing these two Hamiltonian, we find the perturbative term induced by the
FM magnetic structure

$\delta\mathcal{H}_{\text{FM}}(\bm{k})=\begin{pmatrix}M_{+}(\bm{k})&A_{3}k_{z}&0&A_{4}k_{-}\\\
A_{3}k_{z}&M_{-}(\bm{k})&A_{4}k_{-}&0\\\
0&-A_{4}k_{+}&-M_{+}(\bm{k})&A_{3}k_{z}\\\
A_{4}k_{+}&0&A_{3}k_{z}&-M_{-}(\bm{k})\end{pmatrix}$ (32)

where $M_{\pm}(\bm{k})=M_{\alpha}(\bm{k})\pm M_{\beta}(\bm{k})$,
$M_{j}(\bm{k})=M_{0}^{j}+B_{1}^{j}k_{z}^{2}+B_{2}^{j}(k_{x}^{2}+k_{y}^{2})$
with $j=\alpha,\beta$.

For the A-type AFM state, the time reversal symmetry $\Theta$ is broken, but
the combination operation $\Theta\tau_{1/2}$ of $\Theta$ and a translation
$\tau_{1/2}$ (shown in Fig. 2(a)) is preserved. Compared with nonmagnetic
state, the representation of the combination symmetry $\Theta\tau_{1/2}$ in
AFM state is the same as that of the time reversal symmetry $\Theta$. Thus
their Hamiltonian are the same in form.

###### Acknowledgements.

This work is supported by the Fundamental Research Funds for the Central
Universities (Grant No. 020414380149), Natural Science Foundation of Jiangsu
Province (No. BK20200007), the Natural Science Foundation of China (Grants No.
12074181, No. 11674165 and NO. 11834006) and the Fok Ying-Tong Education
Foundation of China (Grant No. 161006). G. Zhan and M. Shi contributed equally
to this work.

##  Appendix A Point Operations on Orbits

In our method, an important step is getting the representation matrices of a
symmetry operation formed by LCAOs. There are some representation matrix under
three orbits $s,p,d$.

(1) The $s$ orbital is invariant under any point operations. We denote this
representation matrix as $D_{s}$: $D_{s}=1$.

(2) The $p$ orbitals $|p_{0}\rangle$, $|p_{\pm 1}\rangle$ can be combined into
several states whose transformation under point operations are clearer as
follow:

$\begin{cases}|p_{x}\rangle=\frac{1}{\sqrt{2}}(|p_{1}\rangle+|p_{-1}\rangle)\\\
|p_{y}\rangle=\frac{1}{i\sqrt{2}}(|p_{1}\rangle-|p_{-1}\rangle)\\\
|p_{z}\rangle=|p_{0}\rangle\end{cases}$ (33)

It is well-known that
$\langle\bm{r}|p_{x}\rangle=xF_{p}(\bm{r}),\langle\bm{r}|p_{y}\rangle=yF_{p}(\bm{r}),\langle\bm{r}|p_{z}\rangle=zF_{p}(\bm{r})$,
which results in

$S(|p_{x}\rangle,|p_{y}\rangle,|p_{z}\rangle)=(|p_{x}\rangle,|p_{y}\rangle,|p_{z}\rangle)g.$
(34)

this $3\times 3$ matrix $g$ can be easily calculated by
$R_{x}(\alpha)R_{y}(\beta)R_{z}(\gamma)$, and we denote this representation
matrix as $D_{p}$: $D_{p}=g$.

(3) The $d$ orbitals $|d_{0}\rangle,|d_{\pm 1}\rangle,|d_{\pm 2}\rangle$ can
also form a new set of states as follows:

$\begin{cases}|d_{xz}\rangle=\frac{1}{\sqrt{2}}(|d_{1}\rangle+|d_{-1}\rangle)\\\
|d_{yz}\rangle=\frac{1}{i\sqrt{2}}(|d_{1}\rangle-|d_{-1}\rangle)\\\
|d_{xy}\rangle=\frac{1}{i\sqrt{2}}(|d_{2}\rangle-|d_{-2}\rangle)\\\
|d_{x^{2}-y^{2}}\rangle=\frac{1}{\sqrt{2}}(|d_{2}\rangle+|d_{-2}\rangle)\\\
|d_{z^{2}}\rangle=|d_{0}\rangle\end{cases}$ (35)

whose wavefunctions are of the form
$\langle\bm{r}|d_{xz}\rangle=2xzF_{d}(\bm{r})$,$\langle\bm{r}|d_{yz}\rangle=2yzF_{d}(\bm{r})$,$\langle\bm{r}|d_{xy}\rangle=2xyF_{d}(\bm{r})$,$\langle\bm{r}|d_{x^{2}-y^{2}}\rangle=(x^{2}-y^{2})F_{d}(\bm{r})$,$\langle\bm{r}|d_{z^{2}}\rangle=(2z^{2}-x^{2}-y^{2})/\sqrt{3}F_{d}(\bm{r})$.
The representation matrix formed by these five orbitals as $D_{d}$.

When considering the spin, the representation matrices must take account into
the spin part: $D=D_{s,p,d}\otimes D_{spin}$. Furthermore, if there exists
spin-orbit interaction, the magnetic quantum numbers $m_{l}$ and $m_{s}$ are
no longer good ones. The states with good quantum number $j$ and $m$ can be
obtained by combining $|m_{l},m_{s}\rangle$ with $CG$ coefficients. For
example,

$|\frac{3}{2},\frac{3}{2}\rangle=|p_{1},+\rangle=\frac{1}{\sqrt{2}}(|p_{x},+\rangle+i|p_{y},+\rangle).$
(36)

The representation matrix formed by a new basis is just similar to the old:
$D_{new}=C^{-1}D_{old}C$, where $C$ is the transition matrix.

##  Appendix B MnBi2Te4 $\bm{k}\cdot\bm{p}$ Hamiltonian

There are more detailed derivation to construct MnBi2Te4 $\bm{k}\cdot\bm{p}$
Hamiltonian in our method.

Firstly, for the FM state, both $C_{2x}$ and $\Theta$ are broken, but their
combination $C_{2x}\Theta$ preserves. We find that the top and bottom bands
contain the $|p_{z},\uparrow\rangle$ and $|p_{z},\downarrow\rangle$ orbitals
of Bi atoms. Representations in $V_{1}$ and $V_{2}$ space are respectively

$\begin{cases}C_{3z}=\text{exp}(-i\frac{\pi}{3}\sigma_{z})\\\ P=-1_{2\times
2}\\\ C_{2x}\Theta=\text{exp}(-i\frac{\pi}{2}\sigma_{z})\end{cases},\hskip
10.00002pt\begin{cases}C_{3z}=1_{2\times 2}\\\ P=-1_{2\times 2}\\\
C_{2x}\Theta=-1_{2\times 2}\end{cases}$ (37)

Thus the representation in $V_{1}\otimes V_{2}$ space is

$\begin{cases}C_{3z}=\text{exp}(-i\frac{\pi}{3}\sigma_{z})\\\ P=1_{2\times
2}\\\ C_{2x}\Theta=\text{exp}(i\frac{\pi}{2}\sigma_{z})\end{cases}$ (38)

The two bands around the Fermi-energy contains $|p_{z},\uparrow\rangle$ and
$|p_{z},\downarrow\rangle$ orbitals of Te atoms, and representations in
$V_{1}$ and $V_{2}$ space are respectively

$\begin{cases}C_{3z}=\text{exp}(-i\frac{\pi}{3}\sigma_{z})\\\ P=-1_{2\times
2}\\\ C_{2x}\Theta=\text{exp}(-i\frac{\pi}{2}\sigma_{z})\end{cases},\hskip
10.00002pt\begin{cases}C_{3z}=1_{2\times 2}\\\ P=1_{2\times 2}\\\
C_{2x}\Theta=1_{2\times 2}\end{cases}$ (39)

Thus the representation in $V_{1}\otimes V_{2}$ space is

$\begin{cases}C_{3z}=\text{exp}(-i\frac{\pi}{3}\sigma_{z})\\\ P=-1_{2\times
2}\\\ C_{2x}\Theta=\text{exp}(-i\frac{\pi}{2}\sigma_{z})\end{cases}$ (40)

Finally, the representation formed by the four states
$\\{|\text{Bi},p_{z},\uparrow\rangle^{+},i|\text{Te},p_{z},\uparrow\rangle^{-},|\text{Bi},p_{z},\downarrow\rangle^{+},-i|\text{Te},p_{z},\downarrow\rangle^{-}\\}$
(where the superscripts indicate parities) can be obtained by summing up the
two representation and making a similarity transformation:

$\begin{cases}D(C_{3z})=\text{exp}(-i\frac{\pi}{3}\sigma_{z}\otimes 1_{2\times
2})\\\ D(P)=1_{2\times 2}\otimes\tau_{z}\\\
D(C_{2x}\Theta)=\text{exp}(i\frac{\pi}{2}\sigma_{z}\otimes 1_{2\times
2})\mathcal{K}\end{cases}$ (41)

where $\mathcal{K}$ is the complex conjugation operator. To construct a
$\bm{k}\cdot\bm{p}$ Hamiltonian, we firstly need a basis of orthonormal
Hermitian matrices. $\Gamma$ matrices are well-known $4\times 4$ Hermitian
matrices which are pairwise orthogonal and can be adopted directly. This base
can be expressed as
$\\{B_{i}|i=1,2,...,16\\}=\\{\sigma_{i}\otimes\sigma_{j}|i,j=0,x,y,z\\}$, and
the $M$ matrices can be obtained in this base as follows,

$\begin{cases}\mathcal{M}(C_{3z})=(1\oplus(-\frac{1}{2}-i\frac{\sqrt{3}}{2}\sigma_{y})\oplus
1)\otimes 1_{4\times 4}\\\ \mathcal{M}(P)=(\sigma_{z}\oplus 1_{2\times
2})\otimes(1_{2\times 2}\oplus-\sigma_{z})\\\
\mathcal{M}(C_{2x}\Theta)=(\sigma_{z}\oplus 1_{2\times 2})\otimes(1_{2\times
2}\oplus-\sigma_{z})\end{cases}$ (42)

For the 0-order polynomial of $\bm{k}$, that is ${1}$, we have
$\mathcal{F}=1$, then $(\mathcal{F}\otimes\mathcal{M}-I)\alpha=0$ for each
operation can be solved, the results are

$\begin{cases}\alpha({C_{3}z})=\sum_{l=1,2,3,4,13,14,15,16}c_{l}v(l)\\\
\alpha({P})=\sum_{l=1,4,5,8,9,12,13,16}c_{l}v(l)\\\
\alpha({C_{2x}\Theta})=\sum_{l=1,2,4,9,11,13,14,16}c_{l}v(l)\end{cases}$ (43)

where $c_{l}$ are any real number, and

$v(l)=(\underbrace{0,...,0}_{l-1},1,0,...,0)^{T}.$ (44)

By solving the intersection of these three solutions we find

$\tilde{A}=\sum_{l=1,4,13,16}c_{l}v(l)^{T},$ (45)

and finally we get the 0-order $\bm{k}\cdot\bm{p}$ Hamiltonian around $\Gamma$
point

$\displaystyle\mathcal{H}(\bm{k})=$
$\displaystyle\sum_{i}\sum_{l=1,4,13,16}c_{l}v(l)_{i1}B_{i}$ $\displaystyle=$
$\displaystyle
c_{1}\sigma_{0}\otimes\sigma_{0}+c_{4}\sigma_{0}\otimes\sigma_{z}+c_{13}\sigma_{z}\otimes\sigma_{0}+c_{16}\sigma_{z}\otimes\sigma_{z}$

Similarly, we can get the $\bm{k}\cdot\bm{p}$ Hamiltonian up to 2 order as
follows,

$\mathcal{H}_{\text{FM}}(\bm{k})=\begin{pmatrix}M_{1}(\bm{k})&A_{1}k_{z}&0&A_{2}k_{-}\\\
A_{1}k_{z}&M_{2}(\bm{k})&A_{4}k_{-}&0\\\
0&A_{4}k_{+}&M_{3}(\bm{k})&A_{3}k_{z}\\\
A_{2}k_{+}&0&A_{3}k_{z}&M_{4}(\bm{k})\end{pmatrix},$ (47)

where $k_{\pm}=k_{x}\pm ik_{y}$ and
$M_{i}(\bm{k})=M_{0}^{i}+B_{1}^{i}k_{z}^{2}+B_{2}^{i}(k_{x}^{2}+k_{y}^{2})$.
Note that the off-diagonal two-order terms are omitted since they contribute
higher order of $\bm{k}$ to the energy. To show the change of the Hamiltonian
induced by the broken of symmetry, we now assume both $C_{2x}$ and $\Theta$
are preserved. Their representations for top and bottom bands in $V_{1}$ and
$V_{2}$ space are respectively

$\begin{cases}C_{2x}=\text{exp}(i\frac{\pi}{2}\sigma_{x})\\\
\Theta=i\sigma_{y}\end{cases},\hskip 10.00002pt\begin{cases}C_{2x}=-1_{2\times
2}\\\ \Theta=1_{2\times 2}\end{cases}$ (48)

and for the conduction band and valence band are

$\begin{cases}C_{2x}=\text{exp}(i\frac{\pi}{2}\sigma_{x})\\\
\Theta=i\sigma_{y}\end{cases},\hskip 10.00002pt\begin{cases}C_{2x}=1_{2\times
2}\\\ \Theta=1_{2\times 2}\end{cases},$ (49)

where the matrices of $C_{3z}$ and $P$ are omitted since they keep unchanged.
Thus representation matrices are $C_{2x}=\text{exp}(-i(\pi/2)\sigma_{x}\otimes
1_{2\times 2}]$, $\Theta=i\sigma_{y}\otimes 1_{2\times 2}K$. We finally get
the $\bm{k}\cdot\bm{p}$ Hamiltonian of nonmagnetic state

$\mathcal{H}_{\text{NM}}(\bm{k})=\epsilon_{0}(\bm{k})+\begin{pmatrix}M_{\gamma}(\bm{k})&A_{1}k_{z}&0&A_{2}k_{-}\\\
A_{1}k_{z}&-M_{\gamma}(\bm{k})&A_{2}k_{-}&0\\\
0&A_{2}k_{+}&M_{\gamma}(\bm{k})&-A_{1}k_{z}\\\
A_{2}k_{+}&0&-A_{1}k_{z}&-M_{\gamma}(\bm{k})\end{pmatrix},$ (50)

where $\epsilon_{0}(\bm{k})=C+D_{1}k_{z}^{2}+D_{2}(k_{x}^{2}+k_{y}^{2})$ and
$M_{\gamma}(\bm{k})=M_{0}^{\gamma}+B_{1}^{\gamma}k_{z}^{2}+B_{2}^{\gamma}(k_{x}^{2}+k_{y}^{2})$.
Comparing these two Hamiltonian, we find the perturbative term induced by the
FM magnetic structure

$\delta\mathcal{H}_{\text{FM}}(\bm{k})=\begin{pmatrix}M_{+}(\bm{k})&A_{3}k_{z}&0&A_{4}k_{-}\\\
A_{3}k_{z}&M_{-}(\bm{k})&A_{4}k_{-}&0\\\
0&-A_{4}k_{+}&-M_{+}(\bm{k})&A_{3}k_{z}\\\
A_{4}k_{+}&0&A_{3}k_{z}&-M_{-}(\bm{k})\end{pmatrix}$ (51)

where $M_{\pm}(\bm{k})=M_{\alpha}(\bm{k})\pm M_{\beta}(\bm{k})$,
$M_{j}(\bm{k})=M_{0}^{j}+B_{1}^{j}k_{z}^{2}+B_{2}^{j}(k_{x}^{2}+k_{y}^{2})$
with $j=\alpha,\beta$.

For the AFM state, $\Theta$ is broken, but the combination of $\Theta$ and a
translation $\tau_{1/2}$ (shown in Fig. 2(d)), that $\Theta\tau_{1/2}$ is
preserved. The two bands below Fermi energy contain $|p_{z},\uparrow\rangle$
and $|p_{z},\downarrow\rangle$ orbitals of Bi atoms and their representation
in $V_{2}$ space is

$\begin{cases}C_{3z}=1_{2\times 2}\\\ P=-1_{2\times 2}\\\ C_{2x}=-1_{2\times
2}\\\ \Theta\tau_{1/2}=1_{2\times 2}\end{cases}$ (52)

whereas the two bands above the Fermi energy contains $|p_{z},\uparrow\rangle$
and $|p_{z},\downarrow\rangle$ orbitals of Te atoms, and their representation
in $V_{2}$ space is

$\begin{cases}C_{3z}=1_{2\times 2}\\\ P=1_{2\times 2}\\\ C_{2x}=1_{2\times
2}\\\ \Theta\tau_{1/2}=1_{2\times 2}\end{cases}$ (53)

As we can see, they are the same as that of the nonmagnetic state. Since the
representations in $V_{1}$ space are obviously the same too, the Hamiltonian
expression must also be the same.

## References

  * Hasan and Kane (2010) M. Z. Hasan and C. L. Kane, Rev. Mod. Phys. 82, 3045 (2010).
  * Qi and Zhang (2011) X.-L. Qi and S.-C. Zhang, Rev. Mod. Phys. 83, 1057 (2011).
  * Armitage  _et al._ (2018) N. P. Armitage, E. J. Mele, and A. Vishwanath, Rev. Mod. Phys. 90, 015001 (2018).
  * Fu and Kane (2007) L. Fu and C. L. Kane, Phys. Rev. B 76, 045302 (2007).
  * Bernevig  _et al._ (2006) B. A. Bernevig, T. L. Hughes, and S.-C. Zhang, Science 314, 1757 (2006).
  * Zhang and Zhang (2013) H. Zhang and S.-C. Zhang, Phys. Status solidi RRL 7, 72 (2013).
  * Zhang  _et al._ (2009) H. Zhang, C.-X. Liu, X.-L. Qi, X. Dai, Z. Fang, and S.-C. Zhang, Nat. Phys. 5, 438 (2009).
  * Luttinger and Kohn (1955) J. M. Luttinger and W. Kohn, Phys. Rev. 97, 869 (1955).
  * Kane (1966) E. Kane,  _Semiconductors and Semimetals_, Semiconductors and Semimetals, Vol. 1 (Elsevier, 1966) pp. 75 – 100.
  * Voon and Willatzen (2009) L. C. L. Y. Voon and M. Willatzen,  _The kp method: electronic properties of semiconductors_ (Springer Science & Business Media, 2009).
  * Liu  _et al._ (2010) C.-X. Liu, X.-L. Qi, H. Zhang, X. Dai, Z. Fang, and S.-C. Zhang, Phys. Rev. B 82, 045122 (2010).
  * Fu (2009) L. Fu, Phys. Rev. Lett. 103, 266801 (2009).
  * Xu  _et al._ (2011) G. Xu, H. Weng, Z. Wang, X. Dai, and Z. Fang, Phys. Rev. Lett. 107, 186806 (2011).
  * Gresch (2018) D. Gresch,  _Identifying Topological Semimetals_ , Ph.D. thesis (2018).
  * (15) D. Gresch, “kdotp-symmetry code,” http://z2pack.ethz.ch/kdotp-symmetry.
  * et al (2018) D. V. et al, New J. Phys. 20, 093026 (2018).
  * (17) Varjas, “Qsymm code,” https://gitlab.kwant-project.org/qt/qsymm.
  * Zhang  _et al._ (2019) D. Zhang, M. Shi, T. Zhu, D. Xing, H. Zhang, and J. Wang, Phys. Rev. Lett. 122, 206401 (2019).
  * Zhang  _et al._ (2020) J. Zhang, D. Wang, M. Shi, T. Zhu, H. Zhang, and J. Wang, Chinese Physics Letters 37, 077304 (2020).
  * Wang  _et al._ (2020) H. Wang, D. Wang, Z. Yang, M. Shi, J. Ruan, D. Xing, J. Wang, and H. Zhang, Phys. Rev. B 101, 081109 (2020).
  * Li  _et al._ (2019) J. Li, Y. Li, S. Du, Z. Wang, B.-L. Gu, S.-C. Zhang, K. He, W. Duan, and Y. Xu, Science Advances 5 (2019), 10.1126/sciadv.aaw5685.
  * Gong  _et al._ (2019) Y. Gong, J. Guo, J. Li, K. Zhu, M. Liao, X. Liu, Q. Zhang, L. Gu, L. Tang, X. Feng, D. Zhang, W. Li, C. Song, L. Wang, P. Yu, X. Chen, Y. Wang, H. Yao, W. Duan, Y. Xu, S.-C. Zhang, X. Ma, Q.-K. Xue, and K. He, Chinese Physics Letters 36, 076801 (2019).
  * Otrokov  _et al._ (2019a) M. M. Otrokov, I. I. Klimovskikh, H. Bentmann, D. Estyunin, and E. V. Chulkov, Nature 576, 416 (2019a).
  * Otrokov  _et al._ (2019b) M. M. Otrokov, I. P. Rusinov, M. Blanco-Rey, M. Hoffmann, A. Y. Vyazovskaya, S. V. Eremeev, A. Ernst, P. M. Echenique, A. Arnau, and E. V. Chulkov, Phys. Rev. Lett. 122, 107202 (2019b).
  * Deng  _et al._ (2020) Y. Deng, Y. Yu, M. Z. Shi, Z. Guo, Z. Xu, J. Wang, X. H. Chen, and Y. Zhang, Science 367, 895 (2020).
  * Liu  _et al._ (2020) C. Liu, Y. Wang, H. Li, Y. Wu, Y. Li, J. Li, K. He, Y. Xu, J. Zhang, and Y. Wang, Nat. Mater. 19, 522 (2020).
  * Chen  _et al._ (2019) B. Chen, F. Fei, D. Zhang, B. Zhang, W. Liu, S. Zhang, P. Wang, B. Wei, Y. Zhang, Z. Zuo, _et al._ , Nat. Commun. 10, 4469 (2019).
  * Klimovskikh  _et al._ (2020) I. I. Klimovskikh, M. M. Otrokov, D. Estyunin, S. V. Eremeev, and E. V. Chulkov, npj Quantum Materials 5 (2020).