# Enriched Pre-trained Transformers
for Joint Slot Filling and Intent Detection

Momchil Hardalov1 Ivan Koychev1 Preslav Nakov2
1Sofia University “St. Kliment Ohridski”, Bulgaria,
2Qatar Computing Research Institute, HBKU, Qatar,
{hardalov<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Detecting the user’s intent and finding the corresponding slots among the
utterance’s words are important tasks in natural language understanding. Their
interconnected nature makes their joint modeling a standard part of training
such models. Moreover, data scarceness and specialized vocabularies pose
additional challenges. Recently, the advances in pre-trained language models,
namely contextualized models such as ELMo and BERT have revolutionized the
field by tapping the potential of training very large models with just a few
steps of fine-tuning on a task-specific dataset. Here, we leverage such model,
namely BERT, and we design a novel architecture on top it. Moreover, we
propose an intent pooling attention mechanism, and we reinforce the slot
filling task by fusing intent distributions, word features, and token
representations. The experimental results on standard datasets show that our
model outperforms both the current non-BERT state of the art as well as some
stronger BERT-based baselines.

##  1 Introduction

With the proliferation of portable devices, smart speakers, and the evolution
of personal assistants, such as Amazon’s Alexa, Apple’s Siri, Google’s
Assistant, and Microsoft’s Cortana, a need for better natural language
understanding (NLU) has emerged. The major challenges such systems face are
(i) finding the intention behind the user’s request, and (ii) gathering the
needed information to complete it via slot filling, while (iii) engaging in a
dialogue with the user. Table 1 shows a user request collected from a personal
voice assistant. Here, the intent is to play music by the artist Justin
Broadrick from year 2005. The slot filling task naturally arises as a sequence
tagging task. Conventional neural network architectures, such as RNNs or CNNs
are appealing approaches to tackle the problem.

Intent | PlayMusic |
---|---|---
Words | play | music | from | 2005 | by | justin | broadrick
| $\downarrow$ | $\downarrow$ | $\downarrow$ | $\downarrow$ | $\downarrow$ | $\downarrow$ | $\downarrow$
Slots | O | O | O | B-year | O | B-artist | I-artist
Table 1: Example from the SNIPS dataset with slots encoded in the BIO format.
The utterance’s intent is PlayMusic, and the given slots are year and artist.

Various extensions thereof can be found in previous work (Xu and Sarikaya,
2013a; Goo et al., 2018; Hakkani-Tür et al., 2016; Liu and Lane, 2016; E et
al., 2019; Gangadharaiah and Narayanaswamy, 2019). Finally, sequence tagging
approaches such as Maximum Entropy Markov model (MEMM) (Toutanova and Manning,
2000; McCallum et al., 2000) and Conditional Random Fields (CRF) (Lafferty et
al., 2001; Jeong and Lee, 2008; Huang et al., 2015) have been added on top to
enforce better modeling of the slot filling task. Recently, some interesting
work has been done with hierarchical structured capsule networks (Xia et al.,
2018; Zhang et al., 2019).

Here, we investigate the usefulness of pre-trained models on the task of NLU.
Our approach is based on BERT (Devlin et al., 2019). That model offer two main
advantages over previous work (Hakkani-Tür et al., 2016; Xu and Sarikaya,
2013a; Gangadharaiah and Narayanaswamy, 2019; Liu and Lane, 2016; E et al.,
2019; Goo et al., 2018): (i) they are based on the Transformer architecture
(Vaswani et al., 2017), which allows them to use bi-directional context when
encoding the tokens instead of left-to-right (as in RNNs) or limited windows
(as in CNNs), and (ii) the model is trained on huge unlabeled text
collections, which allows it to leverage relations learned during pre-
training, e.g., that Justin Broadrick is connected to music or that San
Francisco is a city.

We further adapt the pre-trained models for the NLU tasks. For the intent, we
introduce a pooling attention layer, which uses a weighted sum of the token
representations from the last language modelling layer. We further reinforce
the slot representation with the predicted intent distribution, and word
features such as predicted word casing, and named entities. To demonstrate its
effectiveness, we evaluate it on two publicly available datasets: ATIS
(Hemphill et al., 1990) and SNIPS (Coucke et al., 2018)111The source code will
be made available..

Our contributions are as follows:

  * •

We enrich pre-trained language model, e.g. BERT (Devlin et al., 2019), to
jointly solve intent classification and slot filling.

  * •

We introduce an additional pooling network from the intent classification
task, allowing the model to obtain the hidden representation from the entire
sequence.

  * •

We use the predicted user intent as an explicit guide for the slot fitting
layer rather than just depending on the language model.

  * •

We reinforce the slot learning with named entity and true case
annotations222http://stanfordnlp.github.io/CoreNLP/truecase.html.

The rest of this paper is organized as follows: Section 2 describes our
approach. Details about the dataset, baselines and model details are shown in
Section 3. All experiments are described in Section 4. Section 5 presents
related work. Finally, Section 6 concludes and points to possible directions
for future work.

##  2 Proposed Approach

We propose a joint approach for intent classification and slot filling built
on top of a pre-trained language model, i.e., BERT (Devlin et al., 2019). We
further improve the base model in three directions: (i) for intent
classification, we obtain a pooled representation from the last hidden states
for all tokens (Section 2.1), (ii) we obtain predictions for the word case and
named entities for each token in the utterance (word features), and (iii) we
feed the predicted intent distribution vector, BERT’s last hidden
representations, and word features into a slot filling layer (see Section
2.2). The complete architecture of the model is shown in Figure 1b.

(a) BERT-Joint.

(b) Transformer-NLU (ours).

Figure 1: Model architectures for joint learning of intent and slot filling:
LABEL:sub@subfig:base:bert classical joint learning with BERT, and
LABEL:sub@subfig:bert:ours proposed enhanced version of the model.

###  2.1 Intent Pooling Attention

Traditionally, BERT and subsequent BERT-style models use a special token
([CLS]) to denote the beginning of a sequence. In the original paper (Devlin
et al., 2019), the authors attach a binary classification loss to it for
predicting whether two sequences follow each other in the text (next sentence
prediction, or NSP). Adding such an objective forces the last residual block
to pool a contextualized representation for the whole sentence from the
penultimate layer, which should have a more semantic, rather then task-
specific meaning. The latter strives to improve downstream sentence-level
classification tasks. However, its effectiveness has been recently debated in
the literature (Lample and Conneau, 2019; Joshi et al., 2019; Yang et al.,
2019; Lan et al., 2019). It has been even argued that it should be removed
(Liu et al., 2019).

Here, the task is to jointly learn two strongly correlated tasks, as using
representations from the last, task-specific layer is more natural compared to
using the pooled one from the [CLS] token. Therefore, we introduce a pooling
attention layer to better model the relationship between the task-specific
representations for each token and for the intent. We further adopt a global
concat attention (Luong et al., 2015) as a throughput mechanism. Namely, we
learn a function ($f_{attn}$) to predict the attention weights $\alpha_{int}$
for each token. We obtain $f_{attn}$ by multiplying the outputs from the
language model $H\in\mathbb{R}^{N\times d_{h}}$ by a latent weight matrix
$W_{int\\_e}\in\mathbb{R}^{d_{h}\times d_{h}}$, where $N$ is the number of
tokens in an example and $d_{h}$ is the hidden size of the Transformer. This
is followed by a non-linear $tanh$ activation. In order to obtain importance
logit for each token, we multiply the latter by a projection vector
$v\in\mathbb{R}^{d_{h}}$ (shown in Eq. 1). We further normalize and scale
(Vaswani et al., 2017) to obtain the attention weights.

Finally, we gather a hidden representation $h_{int}$ as a weighted sum of all
attention inputs, and we pass it through a $tanh$ activation (see Eq. 3). For
the final prediction, we use a linear projection on top of $h_{int}$. Finally,
we apply dropouts on $h_{int}$, and as proposed in Vaswani et al. (2017), on
the attention weights.

$\displaystyle{f_{attn}(h_{1},\dots,h_{n})=v\cdot\tanh(W_{int\\_e}\cdot
H^{T})}$ (1)
$\displaystyle\alpha_{int}=softmax(\frac{f_{attn}}{\sqrt{d_{h}}})$ (2)
$\displaystyle h_{int}=tanh(\sum_{i=1}^{N}{\alpha_{int}^{i}h_{enc}^{i}})$ (3)
$\displaystyle y_{int}=W_{int}h_{int}^{T}+b_{int}$ (4)

###  2.2 Slots Modeling

The task of slot filling is closely related to other well-known tasks in the
field, e.g., part-of-speech (POS) tagging, named entity recognition (NER),
etc. Also, it can benefit from knowing the interesting entities in the text.
Therefore, we reinforce the slot filling with tags found by a named entity
recognizer (word features). Next, we combine the intent prediction, the
language model’s hidden representations, and some extracted word features into
a single vector used for token slot attribution. Details about all components
are discussed below.

#### Word Features

A major shortcoming of having free-form text as an input is that it tends not
to follow basic grammatical principles or style rules. The casing of words can
also guide the models while filling the slots, i.e., upper-case words can
refer to names or to abbreviations. Also, knowing the proper casing enabled
the use of external NERs or other tools that depend on the text quality.

As a first step, we improve the text casing using a True Case model. The model
maps the words into the following classes: UPPER, LOWER, INIT_UPPER, and O,
where O is for mixed-case words such as _McVey_. With the text re-cased, we
further extract the named entities with a NER annotator. Named entities are
recognized using a combination of three CRF sequence taggers trained on
various corpora. Numerical entities are recognized using a rule-based system.
Both the truecaser and the NER model are part of the Stanford CoreNLP toolkit
(Manning et al., 2014).

Finally, we merge some entities ((job) TITLE, IDEOLOGY, CRIMINAL_CHARGE,
CAUSE_OF_DEATH) into a special category OTHER as they do not correlate
directly to the domains of either dataset. Moreover, we add a custom regex-
matching entry for AIRPORT_CODES, which are three-letter abbreviations of the
airports. The latter is specially designed for the ATIS (Tur et al., 2010)
dataset. While, marking the proper terms, some of the codes introduce noise,
e.g., the proposition for could be marked as an AIRPORT_CODE because of FOR
(Aeroporto Internacional Pinto Martins, Fortaleza, CE, Brazil). In order to
mitigate this effect, we do a lookup in a dictionary of English words, and if
a match is found, we trigger the O class for the token.

$\displaystyle s_{w}^{i}=W_{w}[ners;cases]+b_{w}$ (5) $\displaystyle
h_{w}^{i}=max(0,s_{w}^{i})+\alpha*min(0,s_{w}^{i})$ (6) $\displaystyle
f_{words}(ners,cases)=W_{proj}{h_{w}^{i}}^{T}+b_{proj}$ (7)

In order to allow the network to learn better feature representations for the
named entities and the casing, we pass them through a two-layer feed-forward
network. The first layer is shown in Eq. 5 followed by a non-linear PReLU
activation (Eq. 6), where $W_{w}\in\mathbb{R}^{23\times 32}$. The second one
is a linear projection denoted as $f_{words}$ (Eq. 7), where
$W_{proj}\in\mathbb{R}^{32\times 32}$.

#### Sub-word Alignment

Modern NLP approaches suggest the use of sub-word units (Sennrich et al.,
2016; Wu et al., 2016; Kudo and Richardson, 2018), which mitigate the effects
of rare words, while preserving the efficiency of a full-word model. Although
they are a flexible framework for tokenization, sub-word units require
additional bookkeeping for the models in order to maintain the original
alignment between words and their labels.

In our approach, we first split the sentences into the original word-tag
pairs, we then disassemble each one into word pieces. Next, the original slot
tag is assigned to the first word piece, while each subsequent is marked with
a special tag (X). Still, the word features from the original token are copied
to each unit. To align the predicted labels with the input tags, we keep a
binary vector for active positions.

#### Slot Filling as Token Classification

As in (Devlin et al., 2019), we treat the slot filling as a token
classification problem, where we apply a shared layer on top of each token’s
representations in order to predict tags.

Furthermore, we assemble the feature vector for the $i^{th}$ slot by
concatenating together the predicted intent probabilities, the word features,
and the contextual representation from the language model. Afterwards, we add
a dropout followed by a linear projection to the proper number of slots, as
shown in Eq. 8.

$y^{i}_{s}=W_{s}[softmax(y_{int});f_{words}^{i};h_{LM}^{i}]+b_{s}\vspace{-0.15in}$
(8)

###  2.3 Interaction and Learning

To train the model, we use a joint loss function $\mathcal{L}_{joint}$) for
the intent and for the slots. For both tasks, we apply cross-entropy over a
softmax activation layer, except in the case of CRF tagging. In those
experiments, the slot loss $\mathcal{L}_{slot}$ will be the negative log-
likelihood (NLL) loss. Moreover, we introduce a new hyper-parameter $\gamma$
to balance the objectives of the two tasks (see Eq. 9). Finally, we propagate
the loss from all the non-masked positions in the sequence, including word
pieces, and special tokens ([CLS], $<$s$>$ etc.). Note that we do not freeze
any weights during fine-tuning.

$\mathcal{L}_{joint}=\gamma*\mathcal{L}_{intent}+(1-\gamma)*\mathcal{L}_{slot}\vspace{-0.15in}$
(9)

##  3 Experimental Setup

###  3.1 Dataset

In our experiments, we use two publicly available datasets, the Airline Travel
Information System (ATIS) (Hemphill et al., 1990), and SNIPS (Coucke et al.,
2018). The ATIS dataset contains transcripts from audio recordings of flight
information requests, while the SNIPS dataset is gathered by a custom-intent-
engine for personal voice assistants. Albeit both are widely used in NLU
benchmarks, ATIS is substantially smaller – almost three times in terms of
examples, and it contains fifteen times less words. However, it has a richer
set of labels, 21 intents and 120 slot categories, as opposed to the 7 intents
and 72 slots in SNIPS. Another key difference is the diversity of domains –
ATIS has only utterances from the flight domain, while SNIPS covers various
subjects, including entertainment, restaurant reservations, weather forecasts,
etc. Furthermore, ATIS allows multiple intent labels. As they only form about
2% of the data, we do not extend our model to multi-label classification. Yet,
we add a new intent category for combinations seen in the training dataset,
e.g., the class for utterance with intents flight and also airfare, would be
marked as atis_airfare#atis_flight. A comparison between the two datasets is
shown in Table 2.

| ATIS | SNIPS
---|---|---
Vocab Size | 722 | 11,241
Average Sentence Length | 11.28 | 9.05
#Intents | 21 | 7
#Slots | 120 | 72
#Training Samples | 4,478 | 13,084
#Dev Samples | 500 | 700
#Test Samples | 893 | 700
Table 2: Statistics about the ATIS and SNIPS datasets.

###  3.2 Measures

We evaluate our models with three well-established evaluation metrics. The
intent detection performance is measured in terms of accuracy. For the slot
filling task, we use F1-score. Finally, the the joint model is evaluated using
sentence-level accuracy, i.e., proportion of examples in the corpus, whose
intent and slots are both correctly predicted. Here, we must note that during
evaluation we consider only the predictions for aligned words, and we omit
special tokens, and word pieces.

###  3.3 Baselines

For our baseline models, we chose BERT (Devlin et al., 2019) due to its state-
of-the-art performance in various NLP tasks. The model’s architecture is shown
in Figure 1a.

#### BERT

For training the model, we follow the fine-tuning procedure proposed by Devlin
et al. (2019). We train a linear layer over the pooled representation of the
special [CLS] token to predict the utterance’s intent. The latter is optimized
during pre-training using the next sentence prediction (NSP) loss to encode
the whole sentence. Moreover, we add a shared layer on top of the last hidden
representations of the tokens in order to obtain a slot prediction. Both
objectives are optimized using a cross-entropy loss.

#### State-of-the-art Models

We further compare our approach to some other benchmark models:

  * •

Joint Seq. (Hakkani-Tür et al., 2016) uses a Recurrent Neural Network (RNN) to
obtain hidden states for each token in the sequence for slot filling, and uses
the last state to predict the intent.

  * •

Atten.-Based (Liu and Lane, 2016) treats the slot filling task as a generative
one, applying sequence-to-sequence RNN to label the input. Further, an
attention weighted sum over the encoder’s hidden states is used to detect the
intent.

  * •

Slotted-Gated (Goo et al., 2018) introduces a special gated mechanism to an
LSTM network, thus reinforcing the slot filling with the hidden representation
used for the intent detection.

  * •

Capsule-NLU (Zhang et al., 2019) adopts Capsule Networks to exploit the
semantic hierarchy between words, slots, and intents using dynamic routing-by-
agreement schema.

  * •

Interrelated (E et al., 2019) uses a Bidirectional LSTM network with attentive
sub-networks for the slot and the intent modeling, and an interrelated
mechanism to establish a direct connection between the two. SF (slot), and ID
(intent) prefixes indicate which sub-network is to be executed first.

  * •

BERT-Joint (Chen et al., 2019) adapts the standard BERT classification, and
token classification pipeline to jointly model the slot and intent, in
addition they experiments with a CRF layer on top. However, they are
evaluating the slot filling task using per-token F1-score (micro averaging),
rather than per-slot entry, as it is standard, leading to higher results.

###  3.4 Model Details

We use the PyTorch implementation of BERT from the Transformers library of
(Wolf et al., 2019) as a base for our models. We fine-tune all models for 50
epochs with hyper-parameters set as follows: batch size of 64 examples,
maximum sequence length of 50 word pieces, dropout set to 0.1 (for both
attentions and hidden layers), and weight decay of 0.01. For optimization, we
use Adam with a learning rate of 8e-05, $\beta_{1}$ 0.9, $\beta_{2}$ 0.999,
$\epsilon$ 1e-06, and warm-up proportion of 0.1. Finally, in order to balance
between the intent and the slot losses, we set the parameter $\gamma$ (Eq. 9)
to 0.6. In order to tackle the problem with random fluctuations for BERT, we
ran the experiments three times and we used the best-performing model on the
development set. We define the latter as the highest sum from all three
measures described in Section 3.2. All the above-mentioned hyper-parameter
values were tuned on the development set, and then used for the final model on
the test set.

| ATIS | SNIPS
---|---|---
Model |  Intent |  |  |  |  |
|  Sent. |  |  |  |  |
|  Slot |  |  |  |  |
|  Intent |  |  |  |  |
|  Sent. |  |  |  |  |
|  Slot |  |  |  |  |
| (F1) |  |  |  |  |
Joint Seq. Hakkani-Tür et al. (2016) | $92.60$ | $80.70$ | $94.30$ | $96.90$ | $73.20$ | $87.30$
Atten.-Based (Liu and Lane, 2016) | $91.10$ | $78.90$ | $94.20$ | $96.70$ | $74.10$ | $87.80$
Sloted-Gated (Goo et al., 2018) | $95.41$ | $83.73$ | $95.42$ | $96.86$ | $76.43$ | $89.27$
Capsule-NLU (Zhang et al., 2019) | $95.00$ | $83.40$ | $95.20$ | $97.30$ | $80.90$ | $91.80$
Interrelated SF-First (E et al., 2019) | $97.76$ | $86.79$ | $95.75$ | $97.43$ | $80.57$ | $91.43$
Interrelated ID-First (E et al., 2019) | $97.09$ | $86.90$ | $95.80$ | $97.29$ | $80.43$ | $92.23$
BERT-Joint (Chen et al., 2019) * | $97.5$ | $88.2$ | $96.1$ | $98.6$ | $92.8$ | $97.0$
BERT-Joint Ours | $97.42$ | $87.57$ |  $95.74$ (98.1) | $98.71$ | $91.57$ |  $96.27$ (97.9)
Transformer-NLU:BERT | $97.87$ | $88.69$ |  $96.25$ (98.2) | $98.86$ | $91.86$ |  $96.57$ (98.0)
Transformer-NLU:BERT w/o Slot Features | $97.87$ | $88.35$ | $95.97$ | $98.86$ | $91.57$ | $96.25$
Transformer-NLU:BERT w/ CRF | $97.42$ | $88.26$ | $96.14$ | $98.57$ | $92.00$ | $96.54$

Table 3: Intent detection and slot filling results measured on the SNIPS and
the ATIS dataset. Highest results in each category are written in bold. Models
used in the analysis are in _italic_. * Chen et al. (2019)’s BERT-Joint uses
per token F1 with micro averaging (which is not standard and inflates their
score by several points absolute). For comparison we also report the per token
F1 for our models (the numbers in parenthesis).

##  4 Results

Here, we discuss the results of our model and we compare them to state-of-the-
art and to BERT-based baselines. We further present an exhaustive analysis of
the model’s components.

#### Evaluation results

Table 3.4 presents quantitative evaluation results in terms of (i) intent
accuracy, (ii) sentence accuracy, and (iii) slot F1 (see Section 3.2). The
first part of the table refers to previous works, whereas the second part
presents our experiments and it is separated with a double horizontal line.
The evaluation results confirm that our model performs constantly better then
current state-of-the-art baselines, which supports the effectiveness of the
approach.

While, models become more accurate, the absolute difference between two
experiments becomes smaller and smaller, thus a better measurement is needed.
Hereby, we introduce a fine-grained measure, i.e., Relative Error Reduction
(RER) percentage, which is defined as the proportion of absolute error reduced
by a $model_{a}$ compared to $model_{b}$.

$RER=100-\frac{Error_{model_{a}}}{Error_{model_{b}}}$ (10)

Table 4 shows the error reducction by our model compared to the current state
of the art (SOTA), and to a BERT-based baseline. Since there is no single best
model from the SOTA, we take the per-column maximum among all, albeit they are
not recorded in a single run. For the ATIS dataset, we see a reduction of
13.66% (1.79 points absolute) for sentence accuracy, and 10.71% (0.45 points
absolute) for slot F1, but just 4.91% for intent accuracy. Such a small
improvement can be due to the quality of the dataset and to its size. For the
SNIPS dataset, we see major increase in all measures and over 55% error
reduction. In absolute terms, we have 1.57 for intent, 10.96 for sentence, and
4.34 for slots. This effects cannot be only attributed to the better model
(discussed in the analysis below), but also to the implicit information that
BERT learned during its extensive pre-training. This is especially useful in
the case of SNIPS, where fair amount of the slots in categories like
SearchCreativeWork, SearchScreeningEvent, AddToPlaylist, PlayMusic are names
of movies, songs, artists, etc.

(a) atis_flight (ATIS).

(b) AddToPlaylist (SNIPS).

Figure 2: Intent pooling attention weights ($\alpha_{int}$) for one sample per
dataset. The larger the number, the higher the word’s contribution. The
example intent is shown in the sub-figure’s caption.

#### Transformer-NLU Analysis

We dissect the proposed model by adding or removing prominent components to
outline their contributions. The results are shown in the second part of Table
3.4. First, we compare the results of BERT-Joint and the enriched model
Transformer-NLU:BERT. We can see a notable reduction of the intent
classification error by 17.44% and 11.63% for the ATIS and the SNIPS dataset,
respectively. Furthermore, we see a 19.87% (ATIS) and 17.35% (SNIPS) error
reduction in slot’s F1, and 11.43% (ATIS) and 11.63% (SNIPS) for sentence
accuracy.

Metric | Relative Error Reduction
---|---
| ATIS
Intent (Acc) | 4.91% | 17.44%
Sent. (Acc) | 13.66% | 11.43%
Slot (F1) | 10.71% | 19.87%
| SNIPS
Intent (Acc) | 55.64% | 11.63 %
Sent. (Acc) | 57.38% | 12.38%
Slot (F1) | 55.86% | 17.35%
Transf.-NLU:BERT | vs. SOTA | vs. BERT
Table 4: Relative error reduction (Eq. 10) comparison between the proposed
model (Transformer-NLU:BERT) and the two baselines: i) current state of the
art for each measure, and ii) conventionally fine-tuned BERT-Joint without any
improvements.

Next, we remove the additional slot features – predicted intent, word casing,
and named entities. The results are shown as Transformer-NLU:BERT w/o Slot
Features. As expected, the intent accuracy remains unchanged for both
datasets, since we retain the pooling attention layer, while the F1-score for
the slots decreases. For SNIPS, the model achieved the same score as for BERT-
Joint, while for ATIS it was within 0.2 points absolute.

Finally, we added a CRF layer on top of the slot network, since it had shown
positive effects in earlier studies (Xu and Sarikaya, 2013a; Huang et al.,
2015; Liu and Lane, 2016; E et al., 2019). We denote the experiment as
Transformer-NLU:BERT w/ CRF. However, in our case it did not yield the
expected improvement. The results for slot filling are close to the highest
recorded, while a drastic drop in intent detection accuracy is observed, i.e.,
-17.44% for ATIS, and -20.28% for SNIPS. We attribute this degradation to the
large gradients from the NLL loss. The effect is even stronger in the case of
smaller datasets, making the optimization unstable for parameter-rich models
such as BERT. We tried to mitigate this issue by increasing the $\gamma$
hyper-parameter, effectively reducing the contribution of the slot’s loss
$\mathcal{L}_{slot}$ to the total, which in turn harmed the slot’s F1.

#### Intent Pooling Attention Visualization

To demonstrate the effects of the intent pooling attention, we visualize the
learned attention weights for two examples from the benchmark datasets. On the
left subplot of Figure 2, a request from the ATIS dataset is presented: i want
fly from baltimore to dallas round trip. The utterance’s intent is marked as
atis_flight, and we can clearly see that the attention had successfully picked
the key tokens from the text, i.e., I, want, fly, from, and to, whereas
supplementary words such as names, locations, dates, etc. have a lesser
contribution to the final sentence-level representation. Moreover, when
trained on the ATIS dataset the layer tends to set the weights in the two
extremes — equally high for important tokens, and towards zero for others. We
attribute this behaviour to the limited domain and vocabulary.

Another example, from the SNIPS dataset, is shown on Figure 2b. Here, the
intent is to add a song to a playlist (AddToPlaylist). In this example, we see
a more diverse spread of attention weights. The model again assigns the
highest weight to the most relevant tokens add, to, the, and play. Also, the
model learned that the first wordpiece has the highest contribution, while the
subsequent ones are supplementary.

Finally, we let the pooling attention layer to take in consideration the
special tokens marking the start and the end ([CLS], and [SEP]) of a sequence,
since they are expected to learn semantic sentence-level representations from
the penultimate layer. Also, the model learns to exploit them as it assigns
high attention weights to both.

##  5 Related Work

#### Intent Classification

Modeling user’s intent had been an area of interest itself. Several approaches
exists that focus only on the utterance’s intent, and omit slot information,
i.e., Hu et al. (2009) maps each intent domain and user’s queries into
Wikipedia representation space, Kim et al. (2017) and Xu and Sarikaya (2013b)
use log-linear models with multiple-stages and word features, while (Ravuri
and Stolcke, 2015) investigate word and character n-gram language models based
on Recurrent Neural Networks and LSTMs (Hochreiter and Schmidhuber, 1997) ,
Xia et al. (2018) proposed a zero-shot transfer thought Capsule Networks
(Sabour et al., 2017) and semantic features for detecting user intents where
no labeled data exists. Moreover, some work extend the problem to a multi-
class multi-label one (Xu and Sarikaya, 2013b; Kim et al., 2017; Gangadharaiah
and Narayanaswamy, 2019).

#### Slot Filling

Before the rise of deep learning models, sequential ones such as Maximum
Entropy Markov model (MEMM) (Toutanova and Manning, 2000; McCallum et al.,
2000), and Conditional Random Fields (CRF) Lafferty et al. (2001); Jeong and
Lee (2008) were the state-of-the-art choice. Recently, several combinations
between these frameworks and different neural network architecture were
proposed (Xu and Sarikaya, 2013a; Huang et al., 2015; E et al., 2019).
However, a steer away from sequential models is observed in favour of self-
attentive ones such as the Transformer (Devlin et al., 2019; Liu et al., 2019;
Radford et al., 2018, 2019). They compose a contextualized representation of
both the sentences, and each word, though a sequence of intermediate non-
linear hidden layers, usually followed by a projection layer in order to
obtain per-token tags. Such models had been successfully applied to closely
related tasks, e.g., as named entity recognition (Devlin et al., 2019), POS
tagging (Tsai et al., 2019), etc.

Approaches modeling the intent or the slot as independent of each other suffer
from uncertainty propagation due the absence of shared knowledge between the
tasks. To overcome this limitation, we learn both tasks using a joint model.

#### Joint Models

Given the strong correlation between the intent and word-level slot tags,
using a joint model is the natural way to address the error propagation in
both tasks. It is no surprise that this has been the trend in natural language
understanding. On the other hand, deep neural networks with their back-
propagation mechanism propose a convenient framework for learning multiple
objectives at once. Moreover, a variety of state-of-the-art approaches for
jointly modeling the intent and slot lay on this fundamentals. Xu and Sarikaya
(2013a) introduced a shared intent and slot hidden state Convolutional Neural
Network (CNN) followed by a globally normalized CRF (TriCRF) for sequence
tagging. Since then, Recurrent Neural Networks have been the dominating force
in the field, e.g., Hakkani-Tür et al. (2016) used bidirectional LSTM cells
for slot filling and the last hidden state for intent classification, Liu and
Lane (2016) introduced a shared attention weights between the slot and intent
layer. Goo et al. (2018) integrated the intent via a gating mechanism into the
context vector of LSTM cells used for slot filling.

Recently, researchers started to explore new directions for jointly modeling
beyond sequential reading models. E et al. (2019) introduced a novel bi-
directional interrelated model, utilizing an iterative mechanism to correct
the predicted intent and slot by multiple step refinement. Further, Zhang et
al. (2019) tried to exploit the semantic hierarchical relationship between
words, slots, and intent via a dynamic routing-by-agreement schema in a
Capsule Network (Sabour et al., 2017).

In this work, we utilize a pre-trained Transformer model to bootstrap the
learning, when having a small dataset. Also, instead of depending only on the
language model’s hidden state to learn the interaction between the slot and
the intent, we fuse the two tasks together. Namely, we guide the slot filling
with the predicted intent, and use a pooled representation from the task-
specific outputs of BERT for intent detection. Finally, unlike others, we
leverage additional information from external sources: (i) from explicit NER
and true case annotations, (ii) from implicit information learned by the
language model during its extensive pre-training.

##  6 Conclusions and Future Work

We studied the two main challenges in natural language understanding, i.e.,
intent detection and slot filling. In particular, we proposed an enriched pre-
trained language model to jointly model the two tasks, i.e., Transformer-NLU.
We designed a pooling attention layer in order to obtain intent representation
beyond just the pooled one from a the special start token. Further, we
reinforced the slot filling with word specific features, and the predicted
intent distribution. We evaluated our approach on two real-world datasets,
also comparing it to baselines such as current state-of-the-art, and BERT-
based models. Our experiments showed that Transformer-NLU outperforms other
alternatives in all standard measures used to evaluate NLU tasks.
Surprisingly, both the usage of the robustly pre-trained version of BERT, and
adding a CRF layer on top of the slot filling network did not bring
improvements. Finally, the Transformer-NLU:BERT had achieved intent accuracy
of 97.87 (ATIS) and 98.86 (SNIPS). Or in relative error reduction – almost 5%
for ATIS, and over 55% for SNIPS, compared to the current state of the art. In
terms of slot filling’s F1, our approach scored 96.25 (+13.66%) for ATIS, and
96.57 (+55.86%) for SNIPS.

In future work we plan to investigate the natural hierarchy of the slots, e.g.
B-toloc.city can be split into B, toloc, and city. Further, to experiment with
better named entity recognition framework such as FLAIR (Akbik et al., 2019a,
b).

## References

  * Akbik et al. (2019a) Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019a.  FLAIR: An easy-to-use framework for state-of-the-art NLP.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 54–59, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Akbik et al. (2019b) Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019b.  Pooled contextualized embeddings for named entity recognition.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 724–728, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Chen et al. (2019) Qian Chen, Zhu Zhuo, and Wen Wang. 2019.  Bert for joint intent classification and slot filling.  _arXiv preprint arXiv:1902.10909_.
  * Coucke et al. (2018) Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018.  Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces.  _arXiv:1805.10190_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * E et al. (2019) Haihong E, Peiqing Niu, Zhongfu Chen, and Meina Song. 2019.  A novel bi-directional interrelated model for joint intent detection and slot filling.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5467–5471, Florence, Italy. Association for Computational Linguistics.
  * Gangadharaiah and Narayanaswamy (2019) Rashmi Gangadharaiah and Balakrishnan Narayanaswamy. 2019.  Joint multiple intent detection and slot labeling for goal-oriented dialog.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 564–569, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Goo et al. (2018) Chih-Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Keng-Wei Hsu, and Yun-Nung Chen. 2018.  Slot-gated modeling for joint slot filling and intent prediction.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 753–757, New Orleans, Louisiana. Association for Computational Linguistics.
  * Hakkani-Tür et al. (2016) Dilek Hakkani-Tür, Gokhan Tur, Asli Celikyilmaz, Yun-Nung Vivian Chen, Jianfeng Gao, Li Deng, and Ye-Yi Wang. 2016.  Multi-domain joint semantic frame parsing using bi-directional RNN-LSTM.  In _Proceedings of The 17th Annual Meeting of the International Speech Communication Association_ , INTERSPEECH ’16.
  * Hemphill et al. (1990) Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990.  The ATIS spoken language systems pilot corpus.  In _Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Computation_ , 9(8):1735–1780.
  * Hu et al. (2009) Jian Hu, Gang Wang, Fred Lochovsky, Jian-tao Sun, and Zheng Chen. 2009.  Understanding user’s query intent with Wikipedia.  In _Proceedings of the 18th International Conference on World Wide Web_ , WWW ’09, pages 471–480, New York, NY, USA. ACM.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional LSTM-CRF models for sequence tagging.  _arXiv:1508.01991_.
  * Jeong and Lee (2008) Minwoo Jeong and Gary Geunbae Lee. 2008.  Triangular-chain conditional random fields.  _IEEE Transactions on Audio, Speech, and Language Processing_ , 16(7):1287–1302.
  * Joshi et al. (2019) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019.  SpanBERT: Improving pre-training by representing and predicting spans.  _arXiv:1907.10529_.
  * Kim et al. (2017) Byeongchang Kim, Seonghan Ryu, and Gary Geunbae Lee. 2017.  Two-stage multi-intent detection for spoken language understanding.  _Multimedia Tools and Applications_ , 76(9):11377–11390.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Lafferty et al. (2001) John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  In _Proceedings of the Eighteenth International Conference on Machine Learning_ , ICML ’01, pages 282–289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
  * Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019.  Cross-lingual language model pretraining.  _arXiv:1901.07291_.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A lite BERT for self-supervised learning of language representations.  _arXiv:1909.11942_.
  * Liu and Lane (2016) Bing Liu and Ian Lane. 2016.  Attention-based recurrent neural network models for joint intent detection and slot filling.  In _Proceedings of The 17th Annual Meeting of the International Speech Communication Association_ , INTERSPEECH ’16, pages 685–689.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv:1907.11692_.
  * Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015.  Effective approaches to attention-based neural machine translation.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1412–1421, Lisbon, Portugal. Association for Computational Linguistics.
  * Manning et al. (2014) Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014.  The Stanford CoreNLP natural language processing toolkit.  In _Association for Computational Linguistics (ACL) System Demonstrations_ , pages 55–60.
  * McCallum et al. (2000) Andrew McCallum, Dayne Freitag, and Fernando C. N. Pereira. 2000.  Maximum entropy markov models for information extraction and segmentation.  In _Proceedings of the Seventeenth International Conference on Machine Learning_ , ICML ’00, pages 591–598, San Francisco, CA, USA.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI Blog_.
  * Ravuri and Stolcke (2015) Suman Ravuri and Andreas Stolcke. 2015.  Recurrent neural network and LSTM models for lexical utterance classification.  In _Sixteenth Annual Conference of the International Speech Communication Association_.
  * Sabour et al. (2017) Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017.  Dynamic routing between capsules.  In _Advances in neural information processing systems_ , pages 3856–3866.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Toutanova and Manning (2000) Kristina Toutanova and Christopher D. Manning. 2000.  Enriching the knowledge sources used in a maximum entropy part-of-speech tagger.  In _2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora_ , pages 63–70, Hong Kong, China. Association for Computational Linguistics.
  * Tsai et al. (2019) Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. 2019.  Small and practical BERT models for sequence labeling.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3630–3634, Hong Kong, China. Association for Computational Linguistics.
  * Tur et al. (2010) Gokhan Tur, Dilek Hakkani-Tür, and Larry Heck. 2010.  What is left to be understood in ATIS? In _2010 IEEE Spoken Language Technology Workshop_ , pages 19–24. IEEE.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the Annual Conference on Neural Information Processing Systems_ , NIPS ’17, pages 5998–6008, Long Beach, CA, USA.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  HuggingFace’s transformers: State-of-the-art natural language processing.  _arXiv:1910.03771_.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv:1609.08144_.
  * Xia et al. (2018) Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip Yu. 2018.  Zero-shot user intent detection via capsule neural networks.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3090–3099, Brussels, Belgium. Association for Computational Linguistics.
  * Xu and Sarikaya (2013a) Puyang Xu and Ruhi Sarikaya. 2013a.  Convolutional neural network based triangular CRF for joint intent detection and slot filling.  In _2013 IEEE Workshop on Automatic Speech Recognition and Understanding_ , pages 78–83. IEEE.
  * Xu and Sarikaya (2013b) Puyang Xu and Ruhi Sarikaya. 2013b.  Exploiting shared information for multi-intent natural language sentence classification.  In _Interspeech_ , pages 3785–3789.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019.  XLNet: Generalized autoregressive pretraining for language understanding.  _arXiv:1906.08237_.
  * Zhang et al. (2019) Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and Philip Yu. 2019.  Joint slot filling and intent detection via capsule neural networks.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5259–5267, Florence, Italy. Association for Computational Linguistics.


# The random first-order transition theory of active glass in the high-
activity regime

Rituparno Mandal  Institute for Theoretical Physics, Georg-August-Universität
Göttingen, 37077 Göttingen, Germany.  Saroj Kumar Nandi  TIFR Centre for
Interdisciplinary Sciences, 36/P, Gopanpally Village, Serilingampally Mandal,
Hyderabad 500046, India.  Chandan Dasgupta  Department of Physics, Indian
Institute of Science, Bangalore 560012, India.  Peter Sollich  Institute for
Theoretical Physics, Georg-August-Universität Göttingen, 37077 Göttingen,
Germany.  Department of Mathematics, King’s College London, London WC2R 2LS,
UK  Nir S. Gov  Department of Chemical and Biological Physics, Weizmann
Institute of Science, Rehovot 7610001, Israel.

###### Abstract

Dense active matter, in the fluid or amorphous-solid form, has generated
intense interest as a model for the dynamics inside living cells and
multicellular systems. An extension of the random first-order transition
theory (RFOT) to include activity was developed, whereby the activity of the
individual particles was added to the free energy of the system in the form of
the potential energy of an active particle, trapped by a harmonic potential
that describes the effective confinement by the surrounding medium. This
active-RFOT model was shown to successfully account for the dependence of the
structural relaxation time in the active glass, extracted from simulations, as
a function of the activity parameters: the magnitude of the active force
($f_{0}$) and its persistence time ($\tau_{p}$). However, significant
deviations were found in the limit of large activity (large $f_{0}$ and/or
$\tau_{p}$). Here we extend the active-RFOT model to high activity using an
activity-dependent harmonic confining potential, which we solve self-
consistently. The extended model predicts qualitative changes in the high
activity regime, which agree with the results of simulations in both three-
dimensional and two-dimensional models of active glass.

_Introduction._ Active glass is a condensed phase of matter that has internal
sources of active (non-thermal) forces and extremely slow dynamics resembling
in many ways the dynamics of passive glass-forming liquids. It has attracted a
significant amount of interest as an abstract model for many biological
systems Angelini  _et al._ (2011); Nnetu  _et al._ (2012); Zhou  _et al._
(2009); Parry  _et al._ (2014); Garcia  _et al._ (2015); Nishizawa  _et al._
(2017); Nandi (2018) or synthetic soft active matter systems Klongvessa  _et
al._ (2019a, b); Geyer  _et al._ (2019), and as a new challenge for non-
equilibrium physics Janssen (2019); Berthier  _et al._ (2019). Realizations of
active glass in numerical simulations are mostly in the form of a dense
aggregate of interacting particles that are self-propelled; the self-
propulsion appears in the form of random forces applied to each particle,
characterized by a force amplitude $f_{0}$ and a persistence time $\tau_{p}$
Ni  _et al._ (2013); Berthier (2014); Mandal  _et al._ (2016a). A
dimensionless quantity that is often used Fily  _et al._ (2014) to
characterize the strength of activity is the “active Peclet number”
$\mathrm{Pe}\equiv f_{0}\tau_{p}/(\gamma\sigma)$ where $\gamma$ is a friction
coefficient and $\sigma$ is a microscopic length related to the particle size.
Thus, the strength of activity can be increased by increasing $f_{0}$ or
$\tau_{p}$. Theories of equilibrium glasses have been extended for active
systems and the resulting descriptions provide insights into many aspects of
how activity affects the glassy properties Berthier and Kurchan (2013); Szamel
(2016); Liluashvili  _et al._ (2017); Feng and Hou (2017); Nandi and Gov
(2017); Nandi  _et al._ (2018). However, these theories are applicable in a
regime where the activity is weak, i.e. $\mathrm{Pe}$ is small. Our aim in
this work is to develop a theory for the regime of high activity.

We have recently presented an active random first order transition (active-
RFOT) theory of an active glass Nandi  _et al._ (2018), where we proposed that
the additional term in the free energy (or the “effective temperature”), due
to activity, is in the form of the potential energy of a single particle
trapped in a harmonic potential Ben-Isaac  _et al._ (2015); Wexler  _et al._
(2020). This “effective medium” model treats the particles that surround the
test particle as a confining potential with spring constant $k$ and friction
coefficient $\gamma$. The additional free energy term is given by Nandi  _et
al._ (2018)

$\Delta F=\frac{Hf_{0}^{2}\tau_{p}}{1+k\tau_{p}/\gamma}$ (1)

where $H=\tilde{H}/\gamma$, $\tilde{H}=T_{K}\kappa_{a}/\Delta C_{p}$, $T_{K}$
is the Kauzmann temperature, defined as the temperature where the
configurational entropy of the passive system vanishes Kauzmann (1948),
$\Delta C_{p}$ is the jump in specific heat from the liquid to the crystalline
state Kauzmann (1948), and $\kappa_{a}$ is an active fragility parameter that
quantifies the sensitivity of the configurational entropy to changes in the
active force. Using this modification we obtained an expression for the
$\alpha$-relaxation time $\tau$Nandi  _et al._ (2018),

$\log{\left(\tau/\tau_{0}\right)}=\frac{E}{\left(T-T_{K}\right)+\Delta F}$ (2)

where $\tau_{0}$ is a microscopic time scale and $E$ represents the surface
reconfiguration energy governing the relaxation dynamics of a region. In the
active-RFOT explored in Nandi  _et al._ (2018) both $\tau_{0}$ and $E$ were
assumed to be independent of the activity parameters $f_{0},\tau_{p}$.

The theoretical expression (Eq. 2) was compared to simulation data for a
three-dimensional active glass model Nandi  _et al._ (2018), and found to
correctly predict the qualitative dependence of the relaxation time on the
activity parameters (see Fig. 1(a)). In the limit of large $\tau_{p}$, we find
that the active term $\Delta F$ approaches a constant, but for large $f_{0}$
the predicted plateau values deviate significantly from the simulation results
(see Fig. 1(a)). Understanding the origin of this discrepancy at high
activities is the central subject of this paper.

We assume that the basic RFOT phenomenology remains valid even at high
activity and consider what modifications to the active term (Eq. 1) can
account for this discrepancy. More specifically, at large activity, i.e. large
active force $f_{0}$ and large persistence time $\tau_{p}$, the effective
medium parameters $(k,\gamma)$ may become dependent on the activity. Since in
the large $\tau_{p}$ limit the parameter $\gamma$ cancels out, we focus on the
effective confinement parameter $k$. The surface reconfiguration energy, $E$,
should also depend on activity, but fits of simulation data suggest that the
modification of $E$ due to activity remains small even in this regime and we
treat $E$ as a constant.

_Activity-induced correction to the confinement._ Inside a dense active glass
each particle is confined within a potential well formed by its neighbors. The
active motion can lead to persistent squeezing of the particles against each
other, leading to a stronger effective confinement due to the steep repulsive
part of the inter-particle interaction potential. We now estimate the change
in the effective confinement $k$ of a test particle in the glass due to the
active fluctuations of the neighboring particles, using a density functional
theory (DFT) formalism Ramakrishnan and Yussouff (1979); Hoell  _et al._
(2019).

For simplicity we do the calculation in one dimension, where the effective
potential in DFT at $x$ has the form

$u(x)=\int_{-\infty}^{+\infty}d{x}^{\prime}v(x-x^{\prime})[\rho(x^{\prime})-\rho_{l}].$
(3)

Here, $v(x)$ is the inter-particle potential (it is a function of $|x|$),
$\rho(x)$ is the local number density field and $\rho_{l}$ is the density of
the uniform liquid. In writing this equation, we have approximated the direct
pair correlation function of the uniform liquid that appears in the
Ramakrishnan-Yussouff form Ramakrishnan and Yussouff (1979) of the free-energy
functional by $-v/(k_{B}T)$. The term involving $\rho_{l}$ is a constant that
is ignored in the rest of the analysis. We consider two particles located at
$x=\pm a$ and calculate $u(x)$ near $x=0$. The density near $x=\pm a$ is
assumed to have a Gaussian form with variance $\delta$. The effective
potential is given by

$u(x)=\frac{1}{\sqrt{2\pi\delta}}\int_{-\infty}^{+\infty}dx^{\prime}v(x-x^{\prime})\\{\exp[-(x^{\prime}-a)^{2}/(2\delta)]+\exp[-(x^{\prime}+a)^{2}/(2\delta)]\\}.$
(4)

Clearly, $u(x)=u(-x)$, so that all odd derivatives of $u(x)$ at $x=0$ are
zero. We need to calculate the second derivative of $u(x)$ at $x=0$: this
gives the effective “spring constant”. Consider the first term in Eq. (4):

$\displaystyle u_{1}(x)$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{2\pi\delta}}\int_{-\infty}^{+\infty}dx^{\prime}v(x-x^{\prime})\exp[-(x^{\prime}-a)^{2}/(2\delta)]$
(5) $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{2\pi\delta}}\int_{-\infty}^{+\infty}dy\,v(x-y-a)\exp[-y^{2}/(2\delta)]$

where we have set $x^{\prime}-a=y$. The second derivative of $u_{1}(x)$ at
$x=0$ is given by

$u_{1}^{\prime\prime}(x=0)=\frac{1}{\sqrt{2\pi\delta}}\int_{-\infty}^{+\infty}dy\,v^{\prime\prime}(-y-a)\exp[-y^{2}/(2\delta)].$
(6)

Now we make use of the fact that the exponential function is sharply peaked at
$y=0$ because $\delta$ is small $(\delta\ll a^{2})$, so that the main
contribution to the integral comes from values of $y$ close to zero. This
allows us to consider only the first non-vanishing term of the Taylor series
expansion of $v^{\prime\prime}(-y-a)=v^{\prime\prime}(y+a)$ near $y=0$, giving

$\displaystyle u_{1}^{\prime\prime}(0)$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{2\pi\delta}}\int_{-\infty}^{+\infty}dy\,[v^{\prime\prime}(a)+\frac{1}{2}v^{\prime\prime\prime\prime}(a)y^{2}]\exp[-y^{2}/(2\delta)].$
(7) $\displaystyle=$ $\displaystyle
v^{\prime\prime}(a)+\frac{1}{2}v^{\prime\prime\prime\prime}(a)\delta$

Terms involving higher powers of $\delta$ can be obtained by retaining higher
order terms in the Taylor series expansion of $v^{\prime\prime}(y+a)$.
Combining this with the contribution of the second term in Eq. (4), we get

$u^{\prime\prime}(0)=2v^{\prime\prime}(a)+v^{\prime\prime\prime\prime}(a)\delta.$
(8)

The effective confinement parameter therefore becomes

$k=k_{0}+k_{1}\delta$ (9)

where $k_{0}=2v^{\prime\prime}(a)$ and
$k_{1}=v^{\prime\prime\prime\prime}(a)$.

Figure 1: (a) Comparison between the measured $\alpha$-relaxation time $\tau$
from simulations of a three-dimensional active glass (at $T=0.45$) Nandi  _et
al._ (2018) (points) as function of $\tau_{p}$, and for different values of
$f_{0}$: from top to bottom $f_{0}=0.1,0.25,0.5,0.75,1.0,2.0$. The dashed
lines give the calculated relaxation time according to the active-RFOT (Eqs.
1, 2), using a constant $k=k_{0}$, while the solid lines use the mARFOT
expression for the effective confinement $k=k_{0}+k_{1}\delta_{sc}$. (b,c)
Plots of the simulated relaxation data of (a) as a function of $f_{0}$,
plotted as the inverse of Eq. 2 (red circles). Solid and dashed lines are the
mARFOT and active-RFOT results, respectively. (b) $\tau_{p}=0.05$, so that
$f_{c}\sim 70$ and we are in the quadratic regime of $f_{0}\ll f_{c}$. (c)
$\tau_{p}=100$, so that $f_{c}\sim 3$, and indeed we see a clear deviation
from the quadratic dependence, in agreement with the predicted $f_{0}^{4/3}$
power law. The values used in the fit are as in Nandi  _et al._ (2018):
$\tau_{0}=0.135$, $T_{K}=0.29$, $E=1.55$, $H=0.042$, $k_{0}/\gamma=0.316$,
with also $k_{1}=0.04$,$\gamma=1$.

_Self-consistent calculation of the confinement._ The mean-square dispersion
of the particles $\langle x^{2}\rangle=\delta$ can now be calculated self-
consistently. The potential energy of the confined active particle (Eq. 1) is
related to its mean-square displacement: $\delta=2\Delta F/k$. Substituting
Eq. 9 into Eq. 1 results in the following implicit equation for $\delta$

$\delta=\frac{2}{k_{0}+k_{1}\delta}\,\frac{Hf_{0}^{2}\tau_{p}}{1+\left(k_{0}+k_{1}\delta\right)\tau_{p}/\gamma}$
(10)

The solutions of Eq. 10 are the roots of a cubic polynomial in $\delta$

$\displaystyle a\delta^{3}+b\delta^{2}+c\delta$ $\displaystyle=$
$\displaystyle d$ (11) $\displaystyle\begin{aligned}
a&=&k_{1}^{2}\tau_{p}/\gamma\\\ b&=&k_{1}(1+2k_{0}\tau_{p}/\gamma)\\\
c&=&k_{0}(1+k_{0}\tau_{p}/\gamma)\\\ d&=&2Hf_{0}^{2}\tau_{p}\end{aligned}$
(12)

In the limit of large activity (force and persistence time), the discriminant
of Eq. 12 is negative, and one obtains one real (and positive) solution for
$\delta$, as well as two irrelevant complex solutions. For smaller activities,
we have a single positive solution and two negative solutions that are
discarded. Explicitly, we find that the self-consistent solution $\delta_{sc}$
initially increases linearly with $f_{0}^{2}$ for small $f_{0}$, but for large
forces it now increases as $f_{0}^{2/3}$. The transition between these limits
depends on the value of $\tau_{p}$:

$\delta_{sc}\sim\left\\{\begin{array}[]{c
l}\frac{d}{c}=\frac{2Hf_{0}^{2}\tau_{p}}{k_{0}\left(1+k_{0}\tau_{p}/\gamma\right)}&,\
\quad f_{0}<f_{c}\\\
\left(\frac{d}{a}\right)^{1/3}=\left(\frac{2f_{0}^{2}H\gamma}{k_{1}^{2}}\right)^{1/3}&,\quad
f_{0}>f_{c}\end{array}\right.$ (13)

where the crossover force is given explicitly by
$f_{c}=(k_{0}(\gamma+k_{0}\tau_{p})/\tau_{p})^{3/4}/(\sqrt{2Hk_{1}})$.

For $\tau_{p}\rightarrow\infty$ the critical force has the limiting value
$f_{c}\rightarrow(k_{0})^{3/2}/(\sqrt{2Hk_{1}})$. In this limit, the mean-
square displacement of the particles can be written relatively compactly as

$\delta_{\infty}=\frac{k_{0}}{3k_{1}}\frac{(\Lambda-1)^{2}}{\Lambda},\quad\Lambda=\left(1+d^{\prime}+\sqrt{d^{\prime}(2+d^{\prime})}\right)^{1/3}$
(14)

where we have defined the dimensionless parameter $d^{\prime}=27k_{1}H\gamma
f_{0}^{2}/k_{0}$. In the limit of small $f_{0}$ we thus recover the quadratic
dependence of $\delta_{\infty}$ on $f_{0}$, while at large $f_{0}$ the
$f_{0}^{2/3}$ power-law of Eq. 13 is obtained.

The self-consistent solution $\delta_{sc}$ can now be used in the modified
confinement spring constant (Eq. 9), $k=k_{0}+k_{1}\delta_{sc}$, and in the
active contribution to the free energy. With this modification we find that
$\Delta F$ increases as $f_{0}^{2}$ for small $f_{0}$, but at large activity
varies more slowly as $f_{0}^{4/3}$. We next compare this modified version of
the active-RFOT (mARFOT) to simulation results.

_Comparison to simulations._ Substituting the self-consistent solution we can
compare to the simulation data of a three-dimensional active glassy system,
shown in Fig. 1 Nandi  _et al._ (2018). We find that the mARFOT greatly
diminishes the discrepancy with the simulation data at large $\tau_{p}$ and
$f_{0}$ (see Fig. 1(a)). Note that these calculations use for the parameters
$E,T_{K},\tau_{0}$ values that are obtained from fits of $\tau$ for the
passive system as a function of $T$, to the form of Eq. 2 with $\Delta F=0$
Mandal  _et al._ (2016b); Nandi  _et al._ (2018). The values of the active
parameters $k_{0}/\gamma,H$ are taken as in Nandi  _et al._ (2018), determined
by the fit at large $\tau_{p}$ and small $f_{0}$. This leaves only
$k_{1},\gamma$ as free fitting parameters.

Figure 2: (color online) (a) Comparison between the measured
$\alpha$-relaxation time $\tau$ from simulations of a two-dimensional active
glass (at $T=0.4$, points) as a function of $\tau_{p}$, and for different
values of $f_{0}$: from top to bottom $f_{0}=0.1,0.2,0.4,0.5,0.7,1,1.5,2,3,4$.
The lines are for the active-RFOT (dashed) and mARFOT (solid). (b,c) Plots of
the simulated relaxation data of (a) as a function of $f_{0}$, plotted as the
inverse of Eq. 2 (red circles). Solid and dashed lines are the mARFOT and
active-RFOT results, respectively. (b) $\tau_{p}=0.01$, so that $f_{c}\sim 43$
and we are in the quadratic regime of $f_{0}\ll f_{c}$. (c) $\tau_{p}=1000$,
so that $f_{c}\sim 1.4$, and indeed we see a clear deviation from the
quadratic dependence, in agreement with the predicted $f_{0}^{4/3}$ power law.
The other values used in the fit are: $\tau_{0}=0.5$, $T_{K}=0.2$, $E=1.99$,
$H=0.4$, $k_{0}/\gamma=1.1$, with $k_{1}=0.8$,$\gamma=1$.

Another way to expose the qualitative change at large $\tau_{p}$ is to plot
$\log{\left(\tau_{0}/\tau\right)}$, i.e. the inverse of Eq. 2, as a function
of $f_{0}$ for different values of $\tau_{p}$. This is shown for two values of
$\tau_{p}$ in Fig. 1(b)-(c). It is clear that at low $\tau_{p}$ this function
increases quadratically with $f_{0}$ (see Fig. 1(b)), as predicted by the
original active-RFOT expression (Eq. 1). For this value of $\tau_{p}=0.05$ we
expect a crossover force $f_{c}\sim 70$ (Eq. 13), so we never enter the regime
where the mARFOT is distinguishable. At a larger value of $\tau_{p}=100$ (see
Fig. 1(c)) we clearly find that the simulation data indicates an increase that
is lower than quadratic, and is in good agreement with the predicted
$f_{0}^{4/3}$ dependence of the mARFOT. For this value of $\tau_{p}$ the
cross-over force is predicted to be $f_{c}\sim 3$, so the range of simulated
forces does indeed enter the regime where mARFOT effects are significant.

In Fig. 2 we compare the model to new simulation data for a two-dimensional
active glass (a finite temperature version of the model studied in Mandal  _et
al._ (2020)). As in the three-dimensional case, we fit the $\alpha$-relaxation
time of the passive system as a function of temperature to extract the
parameters $T_{K}$, $\tau_{0}$ and $E$ (see SI, Fig.S1). The discrepancy
between the simulation data and the active-RFOT is less strong compared to the
data from the three-dimensional system (see Fig. 1a), but it is clearly
observable. The mARFOT is seen to resolve the major part of the discrepancy
(see Fig. 2(a)). It also captures the transition from quadratic (Fig.2(b)) to
non-quadratic dependence of the active term in the free energy on the force
amplitude at high $\tau_{p}$ (Fig.2(c)).

_Discussion._ We provide here a self-consistent extension of the active-RFOT
model that incorporates the renormalization of the effective confinement due
to active fluctuations of the particles. This treatment gives a non-analytic
modification of the power-law dependence of the active contribution to the
free energy (“effective temperature”) on the active force magnitude. The
predicted non-quadratic dependence, with power-law $f_{0}^{4/3}$, is in good
agreement with our simulation data, both in three and two dimensions. This
result may offer an explanation for similar puzzling observations in
configurational entropy calculations Preisler and Dijkstra (2016), where at
high active forces a clear lower-than-quadratic dependence of the “effective
temperature” on the active force was found.

Although our extension of the active-RFOT description improves the agreement
with simulation results for high activity, there are regions in the parameter
space $(f_{0},\tau_{p},T)$ where this theory is not expected to provide a very
good description of the actual behavior. A recent study Mandal  _et al._
(2020) of an athermal ($T=0$) active system found a jamming transition as
$f_{0}$ is reduced below a critical value $f_{J}$ in the $\tau_{p}\to\infty$
limit. For $f<f_{J}$ and large but finite values of $\tau_{p}$, the relaxation
time is found to increase with increasing $\tau_{p}$. It is not clear whether
this jamming transition would persist at moderate temperatures. However, our
simulations for small values of $T$ and $f<f_{J}$ show a trend of increasing
$\tau$ with $\tau_{p}$ for very large values ($\sim 1000$) of $\tau_{p}$. This
behavior cannot be reproduced in the mARFOT description, which always predicts
a decrease of $\tau$ with increasing $\tau_{p}$. Also, the active-RFOT and
mARFOT descriptions cannot be used to describe the behavior for $T<T_{K}$.
This is because the factor $T-T_{K}$ in Eq. 2, arising from the temperature
dependence of the configurational entropy for $T>T_{K}$, should be replaced by
zero for $T<T_{K}$. Eq. 2 would then predict a temperature-independent
relaxation time $\tau$ for $T<T_{K}$, but the (presumably weak) temperature
dependence of parameters such as $\kappa_{a},\gamma$ and $E$, which we have
neglected in our treatment, would lead to values of $\tau$ that depend weakly
on $T$. The temperature dependence of these parameters needs to be explored
further. Finally, the effect of activity on the surface reconfiguration
energy, ignored in our analysis, may become important for values of $f_{0}$
substantially higher than those considered here. Further examination of this
effect would improve our present understanding of the properties of dense
active matter in the high-activity regime. Exploration of other effects of
activity that are absent within our current framework, such as the one leading
to motility induced phase separation Cates and Tailleur (2015), would also be
interesting.

###### Acknowledgements.

N.S.G. is the incumbent of the Lee and William Abramowitz Professorial Chair
of Biophysics, and acknowledges that this work is made possible through the
historic generosity of the Perlman family. This project has received funding
from the European Union’s Horizon 2020 research and innovation programme under
Marie Skłodowska-Curie grant agreement No. 893128. SKN acknowledges support of
the Department of Atomic Energy, Government of India, under Project
Identification No. RTI 4007

## References

  * Angelini  _et al._ (2011) T. E. Angelini, E. Hannezo, X. Trepat, M. Marquez, J. J. Fredberg, and D. A. Weitz, Proc. Natl. Acad. Sci. (USA) 108, 4717 (2011).
  * Nnetu  _et al._ (2012) K. D. Nnetu, M. Knorr, J. Käs, and M. Zink, New J. Phys. 14, 115012 (2012).
  * Zhou  _et al._ (2009) E. H. Zhou, X. Trepat, C. Y. Park, G. Lenormand, M. N. Oliver, S. M. Mijailovich, C. Hardin, D. A. Weitz, J. P. Butler, and J. J. Fredberg, Proc. Natl. Acad. Sci. (USA) 106, 10632 (2009).
  * Parry  _et al._ (2014) B. Parry, I. Surovtsev, M. Cabeen, C. O’Hern, E. Dufresne, and C. Jacobs-Wagner, Cell 156, 183 (2014).
  * Garcia  _et al._ (2015) S. Garcia, E. Hannezo, J. Elgeti, J. F. Joanny, P. Silberzan, and N. S. Gov, Proc. Natl. Acad. Sci. (USA) 112, 15314 (2015).
  * Nishizawa  _et al._ (2017) K. Nishizawa, K. Fujiwara, M. Ikenaga, N. Nakajo, M. Yanagisawa, and D. Mizuno, Sci. Rep. 7, 15143 (2017).
  * Nandi (2018) S. K. Nandi, Phys. Rev. E 97, 052404 (2018).
  * Klongvessa  _et al._ (2019a) N. Klongvessa, F. Ginot, C. Ybert, C. Cottin-Bizonne, and M. Leocmach, Phys. Rev. Lett. 123, 248004 (2019a).
  * Klongvessa  _et al._ (2019b) N. Klongvessa, F. Ginot, C. Ybert, C. Cottin-Bizonne, and M. Leocmach, Phys. Rev. E 100, 062603 (2019b).
  * Geyer  _et al._ (2019) D. Geyer, D. Martin, J. Tailleur, and D. Bartolo, Phys. Rev. X 9, 031043 (2019).
  * Janssen (2019) L. M. Janssen, Journal of Physics: Condensed Matter 31, 503002 (2019).
  * Berthier  _et al._ (2019) L. Berthier, E. Flenner, and G. Szamel, J. Chem. Phys. 150, 200901 (2019).
  * Ni  _et al._ (2013) R. Ni, M. A. C. Stuart, and M. Dijkstra, Nat. Commun 4, 2704 (2013).
  * Berthier (2014) L. Berthier, Phys. Rev. Lett. 112, 220602 (2014).
  * Mandal  _et al._ (2016a) R. Mandal, P. J. Bhuyan, M. Rao, and C. Dasgupta, Soft Matter 12, 6268 (2016a).
  * Fily  _et al._ (2014) Y. Fily, S. Henkes, and M. C. Marchetti, Soft Matter 10, 2132 (2014).
  * Berthier and Kurchan (2013) L. Berthier and J. Kurchan, Nature Physics 9, 310 (2013).
  * Szamel (2016) G. Szamel, Phys. Rev. E 93, 012603 (2016).
  * Liluashvili  _et al._ (2017) A. Liluashvili, J. Ónody, and T. Voigtmann, Phys. Rev. E 96, 062608 (2017).
  * Feng and Hou (2017) M. Feng and Z. Hou, Soft Matter 13, 4464 (2017).
  * Nandi and Gov (2017) S. K. Nandi and N. S. Gov, Soft Matter 13, 7609 (2017).
  * Nandi  _et al._ (2018) S. K. Nandi, R. Mandal, P. J. Bhuyan, C. Dasgupta, M. Rao, and N. S. Gov, Proceedings of the National Academy of Sciences 115, 7688 (2018).
  * Ben-Isaac  _et al._ (2015) E. Ben-Isaac, É. Fodor, P. Visco, F. van Wijland, and N. S. Gov, Physical Review E 92, 012716 (2015).
  * Wexler  _et al._ (2020) D. Wexler, N. Gov, K. Ø. Rasmussen, and G. Bel, Physical Review Research 2, 013003 (2020).
  * Kauzmann (1948) W. Kauzmann, Chemical reviews 43, 219 (1948).
  * Ramakrishnan and Yussouff (1979) T. V. Ramakrishnan and M. Yussouff, Physical Review B 19, 2775 (1979).
  * Hoell  _et al._ (2019) C. Hoell, H. Löwen, and A. M. Menzel, The Journal of Chemical Physics 151, 064902 (2019).
  * Mandal  _et al._ (2016b) R. Mandal, P. J. Bhuyan, M. Rao, and C. Dasgupta, Soft Matter 12, 6268 (2016b).
  * Mandal  _et al._ (2020) R. Mandal, P. J. Bhuyan, P. Chaudhuri, C. Dasgupta, and M. Rao, Nature communications 11, 1 (2020).
  * Preisler and Dijkstra (2016) Z. Preisler and M. Dijkstra, Soft Matter 12, 6043 (2016).
  * Cates and Tailleur (2015) M. Cates and J. Tailleur, Annu. Rev. Condens. Matt. Phys. 6, 219 (2015).