# Discourse Probing of Pretrained Language Models

Fajri Koto Jey Han Lau Timothy Baldwin
School of Computing and Information Systems
The University of Melbourne
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Existing work on probing of pretrained language models (LMs) has predominantly
focused on sentence-level syntactic tasks. In this paper, we introduce
document-level discourse probing to evaluate the ability of pretrained LMs to
capture document-level relations. We experiment with 7 pretrained LMs, 4
languages, and 7 discourse probing tasks, and find BART to be overall the best
model at capturing discourse — but only in its encoder, with BERT performing
surprisingly well as the baseline model. Across the different models, there
are substantial differences in which layers best capture discourse
information, and large disparities between models.

##  1 Introduction

The remarkable development of pretrained language models Devlin et al. (2019);
Lewis et al. (2020); Lan et al. (2020) has raised questions about what precise
aspects of language these models do and do not capture. Probing tasks offer a
means to perform fine-grained analysis of the capabilities of such models, but
most existing work has focused on sentence-level analysis such as syntax
Hewitt and Manning (2019); Jawahar et al. (2019); de Vries et al. (2020),
entities/relations Papanikolaou et al. (2019), and ontological knowledge
Michael et al. (2020). Less is known about how well such models capture
broader discourse in documents.

Rhetorical Structure Theory is a framework for capturing how sentences are
connected and describing the overall structure of a document Mann and Thompson
(1986). A number of studies have used pretrained models to classify discourse
markers Sileo et al. (2019) and discourse relations Nie et al. (2019); Shi and
Demberg (2019), but few Koto et al. (to appear) have systematically
investigated the ability of pretrained models to model discourse structure.
Furthermore, existing work relating to discourse probing has typically focused
exclusively on the BERT-base model, leaving open the question of how well
these findings generalize to other models with different pretraining
objectives, for different languages, and different model sizes.

Model Type #Param #Data Objective BERT Enc 110M 16GB MLM+NSP RoBERTa 110M
160GB MLM ALBERT 12M 16GB MLM+SOP ELECTRA 110M 16GB MLM+DISC GPT-2 Dec 117M
40GB LM BART Enc+Dec 121M 160GB DAE T5 110M 750GB DAE

Table 1: Summary of all English pretrained language models used in this work.
“MLM” = masked language model, “NSP” = next sentence prediction, “SOP” =
sentence order prediction, “LM” = language model, “DISC” = discriminator, and
“DAE” = denoising autoencoder.

Probing Task English Chinese German Spanish (1) 4-way NSP (2) Sentence
Ordering XSUM articles Narayan et al. (2018) Split: 8K/1K/1K Wikipedia (ZH)
Split: 8K/1K/1K Wikipedia (DE) Split: 8K/1K/1K Wikipedia (ES) Split: 8K/1K/1K
(3) Discourse Connective Sampled DisSent dataset Nie et al. (2019) #Labels: 15
Split: 10K/1K/1K CDTB Li et al. (2014) #Labels: 22 Split: 1539/76/168 Potsdam
Commentary Bourgonje and Stede (2020) #Labels: 15 Split: 900/148/159 N/A (4)
RST Nuclearity (5) RST Relation RST-DT Carlson et al. (2001) #Labels
(nuc/rel): 3/18 Split: 16903/1943/2308 CDTB Li et al. (2014) #Labels
(nuc/rel): 3/4 Split: 6159/353/809 Potsdam Commentary Bourgonje and Stede
(2020) #Labels (nuc/rel): 3/31 Split: 1892/289/355 RST-Spanish Treebank da
Cunha et al. (2011) #Labels (nuc/rel): 3/29 Split: 2042/307/421 (6) RST EDU
Segmentation RST-DT Carlson et al. (2001) Split: 312/35/38 docs CDTB Li et al.
(2014) Split: 2135/105/241 p’graphs Potsdam Commentary Bourgonje and Stede
(2020) Split: 131/20/25 docs RST-Spanish Treebank da Cunha et al. (2011)
Split: 200/34/30 docs (7) Cloze Story Test Mostafazadeh et al. (2016) Split:
1683/188/1871 N/A N/A N/A

Table 2: A summary of probing tasks and datasets for each of the four
languages. “Split” indicates the number of train/development/test instances.

Our research question in this paper is: How much discourse structure do layers
of different pretrained language models capture, and do the findings
generalize across languages?

There are two contemporaneous related studies that have examined discourse
modelling in pretrained language models. Upadhye et al. (2020) analyzed how
well two pretrained models capture referential biases of different classes of
English verbs. Zhu et al. (2020) applied the model of Feng and Hirst (2014) to
parse IMDB documents Maas et al. (2011) into discourse trees. Using this
(potentially noisy) data, probing tasks were conducted by mapping attention
layers into single vectors of document-level rhetorical features. These
features, however, are unlikely to capture all the intricacies of inter-
sentential abstraction as their input is formed based on discourse
relations111For example, they only consider discourse relation labels and
ignore nuclearity. and aggregate statistics on the distribution of discourse
units.

To summarize, we introduce 7 discourse-related probing tasks, which we use to
analyze 7 pretrained language models over 4 languages: English, Mandarin
Chinese, German, and Spanish. Code and public-domain data associated with this
research is available at https://github.com/fajri91/discourse_probing.

##  2 Pretrained Language Models

We outline the 7 pretrained models in Table 1. They comprise 4 encoder-only
models: BERT Devlin et al. (2019), RoBERTa Liu et al. (2019), ALBERT Lan et
al. (2020), and ELECTRA Clark et al. (2020); 1 decoder-only model: GPT-2
Radford et al. (2019); and 2 encoder–decoder models: BART Lewis et al. (2020)
and T5 Raffel et al. (2019). To reduce the confound of model size, we use
pretrained models of similar size ($\sim$110m model parameters), with the
exception of ALBERT which is designed to be lighter weight. All models have 12
transformer layers in total; for BART and T5, this means their encoder and
decoder have 6 layers each. Further details of the models are provided in the
Supplementary Material.

Figure 1:  Illustration of the RST discourse probing tasks (Tasks 4–6). Figure
2: Probing task performance on English for each of the seven tasks, plus the
average across all tasks. For BART and T5, layers 7–12 are the decoder layers.
All results are averaged over three runs, and the vertical line for each data
point denotes the standard deviation (noting that most results have low s.d.,
meaning the bar is often not visible).

##  3 Probing Tasks for Discourse Coherence

We experiment with a total of seven probing tasks, as detailed below. Tasks
4–6 are component tasks of discourse parsing based on rhetorical structure
theory (RST; Mann and Thompson (1986)). In an RST discourse tree, EDUs are
typically clauses or sentences, and are hierarchically connected with
discourse labels denoting: (1) nuclearity = nucleus (N) vs. satellite
(S);222The satellite is a supporting EDU for the nucleus. and (2) discourse
relations (e.g. elaborate). An example of a binarized RST discourse tree is
given in Figure 1.

  1. 1.

Next sentence prediction. Similar to the next sentence prediction (NSP)
objective in BERT pretraining, but here we frame it as a 4-way classification
task, with one positive and 3 negative candidates for the next sentence. The
preceding context takes the form of between 2 and 8 sentences, but the
candidates are always single sentences.

  2. 2.

Sentence ordering. We shuffle 3–7 sentences and attempt to reproduce the
original order. This task is based on Barzilay and Lapata (2008) and Koto et
al. (2020), and is assessed based on rank correlation relative to the original
order.

  3. 3.

Discourse connective prediction. Given two sentences/clauses, the task is to
identify an appropriate discourse marker, such as while, or, or although Nie
et al. (2019), representing the conceptual relation between the
sentences/clauses.

  4. 4.

RST nuclearity prediction. For a given ordered pairing of (potentially
complex) EDUs which are connected by an unspecified relation, predict the
nucleus/satellite status of each (see Figure 1).

  5. 5.

RST relation prediction. For a given ordered pairing of (potentially complex)
EDUs which are connected by an unspecified relation, predict the relation that
holds between them (see Figure 1).

  6. 6.

RST elementary discourse unit (EDU) segmentation. Chunk a concatenated
sequence of EDUs into its component EDUs.

  7. 7.

Cloze story test. Given a 4-sentence story context, pick the best ending from
two possible options Mostafazadeh et al. (2016); Sharma et al. (2018). This
task is harder than NSP, as it requires an understanding of commonsense and
storytelling Chaturvedi et al. (2017); Liu et al. (2018).

##  4 Experimental Setup

We summarize all data (sources, number of labels, and data split) in Table 2.
This includes English, Chinese, German, and Spanish for each probing task. For
NSP and sentence ordering, we generate data from news articles and Wikipedia.
For the RST tasks, we use discourse treebanks for each of the four languages.

We formulate all probing tasks except sentence ordering and EDU segmentation
as a classification problem, and evaluate using accuracy. During fine-tuning,
we add an MLP layer on top of the pretrained model for classification, and
only update the MLP parameters (all other layers are frozen). We use the [CLS]
embedding for BERT and ALBERT following standard practice, while for other
models we perform average pooling to obtain a vector for each sentence, and
concatenate them as the input to the MLP.333BERT and ALBERT performance with
average pooling is in included in the Appendix.

For sentence ordering, we follow Koto et al. (2020) and frame it as a
sentence-level sequence labelling task, where the goal is to estimate
$P(r|s)$, where $r$ is the rank position and $s$ the sentence. The task has 7
classes, as we have 3–7 sentences (see Section 3). At test time, we choose the
label sequence that maximizes the sequence probability. Sentence embeddings
are obtained by average pooling. The EDU segmentation task is also framed as a
binary sequence labelling task (segment boundary or not) at the (sub)word
level. We use Spearman rank correlation and macro-averaged F1 score to
evaluate sentence ordering and EDU segmentation, respectively.

We use a learning rate $1e-3$, warm-up of 10% of total steps, and the
development set for early stopping in all experiments. All presented results
are averaged over three runs.444More details of the training configuration are
given in the Appendix.

Figure 3: Discourse performance of BERT across different languages. All
results are averaged over three runs, and a vertical line is used to denote
the standard deviation for each data point (most of which are not visible, due
to the low standard deviation).

##  5 Results and Analysis

In Figure 2, we present the probing task performance on English for all models
based on a representation generated from each of the 12 layers of the model.
First, we observe that most performance fluctuates (non-monotonic) across
layers except for some models in the NSP task and some ALBERT results in the
other probing tasks. We also found that most models except ALBERT tend to have
a very low standard deviation based on three runs with different random seeds.

We discover that all models except T5 and early layers of BERT and ALBERT
perform well over the NSP task, with accuracy $\geq$ 0.8, implying it is a
simple task. However, they all struggle at sentence ordering (topping out at
$\rho\sim 0.4$), suggesting that they are ineffective at modelling discourse
over multiple sentences; this is borne out in Figure 4, where performance
degrades as the number of sentences to re-order increases.

Figure 4: Sentence ordering task breakdown based on the best layer of each
model.

Interestingly, for Discourse Connectives, RST Nuclearity, and RST Relation
Prediction, the models produce similar patterns, even though the discourse
connective data is derived from a different dataset and theoretically divorced
from RST. BART outperforms most other models in layers 1–6 for these tasks (a
similar observation is found for NSP and Sentence Ordering) with BERT and
ALBERT struggling particularly in the earlier layers. For EDU segmentation,
RoBERTa and again the first few layers of BART perform best. For the Cloze
Story Test, all models seem to improve as we go deeper into the layers,
suggesting that high-level story understanding is captured deeper in the
models.

We summarize the overall performance by calculating the averaged normalized
scores in the last plot in Figure 2.555Given a task, we perform min–max
normalization for all model-layer scores (7$\times$12 scores in total), and
then compute the average over all tasks for each model’s layer. RoBERTa and
BART appear to be the best overall models at capturing discourse information,
but only in the encoder layers (the first 6 layers) for BART. We hypothesize
that the BART decoder focuses on sequence generation, and as such is less
adept at language understanding. This is supported by a similar trend for T5,
also a denoising autoencoder. BERT does surprisingly well (given that it’s the
baseline model), but mostly in the deeper layers (7–10), while ELECTRA
performs best at the three last layers.

In terms of the influence of training data, we see mixed results. BART and
RoBERTa are the two best models, and both are trained with more data than most
models (an order of magnitude more; see Table 1). But T5 (and to a certain
extent GPT-2) are also trained with more data (in fact T5 has the most
training data), but their discourse modelling performance is underwhelming. In
terms of training objectives, it appears that a pure decoder with an LM
objective (GPT-2) is less effective at capturing discourse structure. ALBERT,
the smallest model (an order of magnitude less parameters than most), performs
surprisingly well (with high standard deviation), but only at its last layer,
suggesting that discourse knowledge is concentrated deep inside the model.

Lastly, we explore whether these trends hold if we use a larger model (BERT-
base vs. BERT-large) and for different languages (again based on monolingual
BERT models for the respective languages). Results are presented in Figure 3.
For model size (“English (large)” vs. “English”), the overall pattern is
remarkably similar, with a slight uplift in absolute results with the larger
model. Between the 4 different languages (English, Chinese, German, and
Spanish), performance varies for all tasks except for NSP (e.g. EDU
segmentation appears to be easiest in Chinese, and relation prediction is the
hardest in German), but the shape of the lines is largely the same, indicating
the optimal layers for a particular task are consistent across languages.

##  6 Conclusion

We perform probing on 7 pretrained language models across 4 languages to
investigate what discourse effects they capture. We find that BART’s encoder
and RoBERTa perform best, while pure language models (GPT-2) struggle.
Interestingly, we see a consistent pattern across different languages and
model sizes, suggesting that the trends we found are robust across these
dimensions.

## Acknowledgements

We are grateful to the anonymous reviewers for their helpful feedback and
suggestions. The first author is supported by the Australia Awards Scholarship
(AAS), funded by the Department of Foreign Affairs and Trade (DFAT),
Australia. This research was undertaken using the LIEF HPC-GPGPU Facility
hosted at The University of Melbourne. This facility was established with the
assistance of LIEF Grant LE170100200.

## References

  * Barzilay and Lapata (2008) Regina Barzilay and Mirella Lapata. 2008.  Modeling local coherence: An entity-based approach.  _Computational Linguistics_ , 34(1):1–34.
  * Bourgonje and Stede (2020) Peter Bourgonje and Manfred Stede. 2020.  The Potsdam commentary corpus 2.2: Extending annotations for shallow discourse parsing.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 1061–1066, Marseille, France. European Language Resources Association.
  * Carlson et al. (2001) Lynn Carlson, Daniel Marcu, and Mary Ellen Okurovsky. 2001.  Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory.  In _Proceedings of the Second SIGdial Workshop on Discourse and Dialogue_.
  * Chaturvedi et al. (2017) Snigdha Chaturvedi, Haoruo Peng, and Dan Roth. 2017.  Story comprehension for predicting what happens next.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1603–1614, Copenhagen, Denmark. Association for Computational Linguistics.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020\.  ELECTRA: Pre-training text encoders as discriminators rather than generators.  In _ICLR 2020: Eighth International Conference on Learning Representations_.
  * da Cunha et al. (2011) Iria da Cunha, Juan-Manuel Torres-Moreno, and Gerardo Sierra. 2011.  On the development of the RST Spanish Treebank.  In _Proceedings of the 5th Linguistic Annotation Workshop_ , pages 1–10.
  * de Vries et al. (2020) Wietse de Vries, Andreas van Cranenburgh, and Malvina Nissim. 2020.  What’s so special about BERT’s layers? A closer look at the NLP pipeline in monolingual and multilingual models.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4339–4350, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Feng and Hirst (2014) Vanessa Wei Feng and Graeme Hirst. 2014.  A linear-time bottom-up discourse parser with constraints and post-editing.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 511–521, Baltimore, Maryland. Association for Computational Linguistics.
  * Hewitt and Manning (2019) John Hewitt and Christopher D. Manning. 2019.  A structural probe for finding syntax in word representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does BERT learn about the structure of language? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  * Koto et al. (to appear) Fajri Koto, Jey Han Lau, and Timothy Baldwin. to appear.  Top-down discourse parsing via sequence labelling.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics_.
  * Koto et al. (2020) Fajri Koto, Afshin Rahimi, Jey Han Lau, and Timothy Baldwin. 2020.  IndoLEM and IndoBERT: A benchmark dataset and pre-trained language model for Indonesian NLP.  In _Proceedings of the 28th International Conference on Computational Linguistics (COLING 2020)_.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  ALBERT: A lite BERT for self-supervised learning of language representations.  In _ICLR 2020: Eighth International Conference on Learning Representations_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2014) Yancui Li, Wenhe Feng, Jing Sun, Fang Kong, and Guodong Zhou. 2014.  Building Chinese discourse corpus with connective-driven dependency tree structure.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2105–2114, Doha, Qatar. Association for Computational Linguistics.
  * Liu et al. (2018) Fei Liu, Trevor Cohn, and Timothy Baldwin. 2018.  Narrative modeling with memory chains and semantic supervision.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 278–284, Melbourne, Australia. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Mann and Thompson (1986) William C. Mann and Sandra A. Thompson. 1986.  Assertions from discourse structure.  In _Strategic Computing - Natural Language Workshop: Proceedings of a Workshop Held at Marina del Rey, California, May 1-2, 1986_.
  * Michael et al. (2020) Julian Michael, Jan A. Botha, and Ian Tenney. 2020.  Asking without telling: Exploring latent ontologies in contextual representations.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6792–6812, Online. Association for Computational Linguistics.
  * Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016.  A corpus and cloze evaluation for deeper understanding of commonsense stories.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 839–849, San Diego, California. Association for Computational Linguistics.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Nie et al. (2019) Allen Nie, Erin Bennett, and Noah Goodman. 2019.  DisSent: Learning sentence representations from explicit discourse relations.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4497–4510, Florence, Italy. Association for Computational Linguistics.
  * Papanikolaou et al. (2019) Yannis Papanikolaou, Ian Roberts, and Andrea Pierleoni. 2019.  Deep bidirectional transformers for relation extraction without supervision.  In _Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)_ , pages 67–75, Hong Kong, China. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Sharma et al. (2018) Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. 2018.  Tackling the story ending biases in the story cloze test.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 752–757, Melbourne, Australia. Association for Computational Linguistics.
  * Shi and Demberg (2019) Wei Shi and Vera Demberg. 2019.  Next sentence prediction helps implicit discourse relation classification within and across domains.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5790–5796, Hong Kong, China. Association for Computational Linguistics.
  * Sileo et al. (2019) Damien Sileo, Tim Van De Cruys, Camille Pradel, and Philippe Muller. 2019.  Mining discourse markers for unsupervised sentence representation learning.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3477–3486, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Upadhye et al. (2020) Shiva Upadhye, Leon Bergen, and Andrew Kehler. 2020.  Predicting reference: What do language models learn about discourse models? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 977–982, Online. Association for Computational Linguistics.
  * Voita et al. (2019) Elena Voita, Rico Sennrich, and Ivan Titov. 2019.  The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4396–4406, Hong Kong, China. Association for Computational Linguistics.
  * Zhu et al. (2020) Zining Zhu, Chuer Pan, Mohamed Abdalla, and Frank Rudzicz. 2020.  Examining the rhetorical capacities of neural language models.  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pages 16–32, Online. Association for Computational Linguistics.

##  Appendix A Pretrained Language Models

The pretrained models are sourced from Huggingface (https://huggingface.co/),
as detailed in Tables 3 and 4.

Model Huggingface model BERT bert-base-uncased BERT (large) bert-large-uncased
RoBERTa roberta-base ALBERT albert-base-v2 ELECTRA electra-base-discriminator
GPT-2 gpt2 BART bart-base T5 t5-small

Table 3: List of English pretrained language models

Language Huggingface model Chinese bert-base-chinese German bert-base-german-
dbmdz-uncased Spanish bert-base-spanish-wwm-uncased

Table 4: List of non-English BERT models.

##  Appendix B Data Construction, Examples, and Training Configuration

###  B.1 Next Sentence Prediction

#Sentence (context) Total 2 2500 4 2500 6 2500 8 2500 Total 10000

Table 5: NSP data based on the number of sentences.

We use spaCy (https://spacy.io/) to perform sentence tokenization, and ensure
that the distractor options in the training set do not overlap with the test
set. For all languages and models, the training configurations are similar:
the maximum tokens in the context and the next sentence are 450 and 50,
respectively. If the token lengths are more than this, we truncate the context
from the beginning of the sequence, and truncate the next sentence at the end
of the sequence. We concatenate context with each option, and perform binary
classification.

Other training configuration details: learning rate = 1e-3, Adam epsilon
=1e-8, maximum gradient norm = 1.0, maximum epochs = 20, warmup = 10% of the
training steps, and patience for early stopping = 5 epochs.

Context s1: The Eastern Star, mostly carrying elderly tourists, capsized on 1
June near Jianli in Hubei province. s2: Just 14 of the 456 passengers and crew
are known to have survived. Options 0: The channel recently said its signal
was carried by 22 satellites 0: That step has become a huge challenge for
opposition candidates 0: Six men were convicted and then acquitted of the
atrocity and no-one has since been convicted of involvement in the bombing 1:
A search is continuing for eight people who remain missing.

Table 6: Example of English NSP data with 2-sentence context. 1 indicates the
correct next sentence.

###  B.2 Sentence Ordering

#Sentence Total 3 2000 4 2000 5 2000 6 2000 7 2000 Total 10000

Table 7: Sentence ordering data based on number of sentence.

Context s0: West Mercia Police said the police do not encourage members of the
public to pursue their own investigations. s1: David John Poole, from
Hereford, poses online as a 14-year-old girl and says he has been sent
hundreds of explicit messages. s2: He says his work has led to two arrests in
four weeks. Correct order: 2–0–1

Table 8: Example of English sentence ordering data

In generating sentence ordering data, we once again use spaCy
(https://spacy.io/) to perform sentence tokenization. For all languages and
models, the training configurations are similar, with the maximum tokens in
each sentence = 50, learning rate = 1e-3, Adam epsilon = 1e-8, maximum
gradient norm = 1.0, training epochs = 20, warmup = 10% of the training steps,
and patience for early stopping = 10 epochs.

###  B.3 Discourse Connective Prediction

Figure 5: Discourse connective word distribution.

As our Chinese and German data is extracted from discourse treebanks, the
number of distinct connective words varies. For instance, in the Chinese
discourse treebank, we find 246 unique connective words. To simplify this, we
set the connective word to OTHER if its word frequency is less than 12.

For all languages and models, the training configurations are: maximum token
length of each sentence = 50, learning rate = 1e-3, Adam epsilon = 1e-8,
maximum gradient norm = 1.0, maximum epochs = 20, warmup = 10% of the training
steps, and patience for early stopping = 10 epochs.

Figure 6: Discourse connective: data examples Figure 7: Nuclearity label
distribution. Figure 8: Relation label distribution.

###  B.4 RST-related Tasks

In Figures 7 and 8, we present the distribution of the nuclearity and relation
labels for the 4 different discourse treebanks. The English treebank is
significantly larger, with a strong preference for the NS (nuclear–satellite)
relationship. Unlike other languages, the proportion of NN (nuclear–nuclear)
relationships in the Chinese discourse treebank (CDTB) is the highest. We also
notice that the relation label set in CDTB is the simplest, with only 4
labels.

Most of the training details for nuclearity and relation prediction are the
same as for the NSP task, except we set the maximum token length of each
sentence to 250. Particularly for EDU segmentation, we set the maximum token
length in a document to 512.

###  B.5 Cloze Story Test

As discussed in Table 2, we use cloze story test version-1 Mostafazadeh et al.
(2016). Although version-2 Sharma et al. (2018) is better in terms of story
biases, the gold labels for the test set are not publicly available, which
limited our ability to explore different layers of a broad range of pretrained
language models (due to rate limiting of test evaluation).

For the data split, we followed previous work Liu et al. (2018) in splitting
the development set into a training and validation set. We perform binary
classification similar to the NSP task, by first merging all 4-sentence
stories into a single text (context). We limit the context to a maximum of 450
tokens, and each candidate sentence (as the story ending) is limited to 50
tokens. Other training details are the same as for the NSP task.

##  Appendix C Full Experimental Results

Layer NSP Sent. Ord. Discourse Conn. Nuclearity Relation EDU segment. Cloze
ST. BERT (English); std = 0.00 – 0.02 1 0.36 0.28 0.28 0.62 0.38 0.30 0.58 2
0.45 0.28 0.31 0.64 0.41 0.42 0.61 3 0.79 0.28 0.30 0.62 0.40 0.49 0.63 4 0.95
0.32 0.35 0.65 0.44 0.52 0.60 5 0.97 0.31 0.44 0.75 0.54 0.55 0.66 6 0.97 0.35
0.53 0.78 0.60 0.56 0.72 7 0.96 0.33 0.57 0.80 0.65 0.56 0.72 8 0.96 0.32 0.57
0.81 0.65 0.54 0.73 9 0.96 0.34 0.59 0.80 0.65 0.52 0.72 10 0.97 0.33 0.58
0.80 0.64 0.47 0.75 11 0.97 0.31 0.59 0.79 0.63 0.44 0.76 12 0.99 0.32 0.56
0.76 0.59 0.39 0.76 RoBERTa (English); std = 0.00 – 0.02 1 0.78 0.29 0.46 0.72
0.55 0.68 0.72 2 0.86 0.31 0.48 0.73 0.56 0.92 0.73 3 0.88 0.30 0.49 0.75 0.58
0.90 0.74 4 0.95 0.34 0.51 0.77 0.59 0.88 0.75 5 0.96 0.37 0.51 0.79 0.60 0.91
0.78 6 0.96 0.37 0.52 0.79 0.61 0.86 0.78 7 0.96 0.39 0.52 0.78 0.61 0.85 0.83
8 0.95 0.37 0.54 0.78 0.61 0.87 0.86 9 0.94 0.37 0.54 0.79 0.61 0.87 0.86 10
0.94 0.36 0.54 0.78 0.61 0.88 0.86 11 0.93 0.35 0.53 0.77 0.59 0.87 0.85 12
0.90 0.31 0.48 0.75 0.56 0.73 0.82 ALBERT (English); std = 0.00 – 0.03 1 0.34
0.29 0.29 0.63 0.40 0.47 0.56 2 0.85 0.33 0.33 0.63 0.44 0.54 0.66 3 0.91 0.30
0.32 0.64 0.45 0.53 0.68 4 0.93 0.30 0.35 0.67 0.46 0.51 0.69 5 0.96 0.30 0.35
0.67 0.47 0.47 0.70 6 0.97 0.29 0.37 0.67 0.48 0.44 0.71 7 0.97 0.29 0.40 0.68
0.48 0.40 0.73 8 0.98 0.26 0.40 0.68 0.49 0.34 0.73 9 0.97 0.23 0.40 0.68 0.49
0.32 0.75 10 0.98 0.21 0.41 0.70 0.49 0.25 0.76 11 0.98 0.17 0.43 0.73 0.52
0.18 0.77 12 0.99 0.13 0.53 0.79 0.63 0.11 0.85 ELECTRA (English); std = 0.00
– 0.02 1 0.86 0.27 0.47 0.72 0.54 0.42 0.72 2 0.90 0.31 0.48 0.72 0.55 0.47
0.74 3 0.90 0.31 0.49 0.73 0.55 0.45 0.74 4 0.94 0.31 0.51 0.74 0.57 0.50 0.75
5 0.96 0.35 0.52 0.77 0.59 0.54 0.76 6 0.96 0.36 0.53 0.78 0.60 0.57 0.78 7
0.96 0.37 0.53 0.79 0.61 0.54 0.78 8 0.97 0.39 0.55 0.80 0.63 0.51 0.82 9 0.97
0.41 0.56 0.80 0.63 0.48 0.86 10 0.97 0.43 0.58 0.80 0.63 0.49 0.89 11 0.97
0.42 0.57 0.80 0.64 0.52 0.89 12 0.96 0.40 0.57 0.79 0.60 0.48 0.88

Table 9: Full results for BERT, RoBERTa, ALBERT, and ELECTRA over English.

Layer NSP Sent. Ord. Discourse Conn. Nuclearity Relation EDU segment. Cloze
ST. GPT-2 (English); std = 0.00 – 0.02 1 0.86 0.26 0.47 0.72 0.55 0.35 0.73 2
0.87 0.26 0.48 0.73 0.56 0.37 0.73 3 0.88 0.28 0.48 0.73 0.56 0.40 0.74 4 0.90
0.30 0.51 0.75 0.57 0.40 0.76 5 0.91 0.32 0.51 0.75 0.57 0.41 0.75 6 0.93 0.33
0.52 0.77 0.59 0.42 0.76 7 0.93 0.33 0.52 0.76 0.60 0.42 0.77 8 0.92 0.34 0.51
0.77 0.59 0.41 0.77 9 0.92 0.33 0.50 0.76 0.59 0.41 0.77 10 0.91 0.31 0.49
0.75 0.58 0.41 0.77 11 0.91 0.30 0.49 0.74 0.57 0.42 0.75 12 0.85 0.28 0.47
0.73 0.55 0.38 0.72 BART (English); Layers 7–12 are the decoder; std = 0.00 –
0.01. 1 0.86 0.30 0.48 0.73 0.55 0.79 0.73 2 0.92 0.34 0.49 0.76 0.58 0.88
0.76 3 0.95 0.35 0.51 0.76 0.58 0.89 0.76 4 0.96 0.38 0.52 0.78 0.60 0.86 0.78
5 0.97 0.39 0.53 0.78 0.62 0.82 0.79 6 0.96 0.41 0.52 0.80 0.62 0.62 0.78 7
0.94 0.32 0.51 0.77 0.59 0.10 0.76 8 0.95 0.39 0.54 0.79 0.61 0.23 0.77 9 0.95
0.40 0.54 0.79 0.62 0.32 0.78 10 0.95 0.40 0.54 0.80 0.62 0.31 0.81 11 0.96
0.38 0.52 0.78 0.60 0.34 0.80 12 0.95 0.36 0.52 0.77 0.59 0.47 0.82 T5
(English); Layers 7–12 are the decoder; std = 0.00 – 0.03. 1 0.77 0.27 0.39
0.71 0.50 0.26 0.71 2 0.80 0.30 0.43 0.74 0.54 0.38 0.70 3 0.82 0.32 0.45 0.75
0.55 0.40 0.73 4 0.84 0.33 0.46 0.76 0.57 0.37 0.74 5 0.87 0.33 0.45 0.76 0.57
0.33 0.72 6 0.86 0.35 0.46 0.76 0.58 0.28 0.72 7 0.77 0.28 0.41 0.73 0.54 0.24
0.71 8 0.77 0.26 0.44 0.74 0.55 0.27 0.71 9 0.77 0.24 0.46 0.75 0.56 0.27 0.71
10 0.74 0.22 0.45 0.75 0.55 0.20 0.72 11 0.70 0.22 0.44 0.73 0.54 0.11 0.72 12
0.68 0.20 0.42 0.73 0.52 0.00 0.72

Table 10: Full results for GPT-2, BART, and T5 over English.

Layer NSP Sent. Ord. Discourse Conn. Nuclearity Relation EDU segment. Chinese;
std = 0.00 – 0.02. 1 0.30 0.38 0.36 0.53 0.58 0.70 2 0.47 0.38 0.36 0.53 0.58
0.75 3 0.47 0.42 0.36 0.53 0.58 0.79 4 0.83 0.42 0.36 0.53 0.58 0.79 5 0.90
0.44 0.35 0.59 0.60 0.81 6 0.93 0.44 0.39 0.60 0.64 0.83 7 0.94 0.45 0.42 0.64
0.67 0.83 8 0.94 0.44 0.43 0.64 0.66 0.83 9 0.94 0.43 0.46 0.63 0.68 0.83 10
0.96 0.43 0.44 0.64 0.68 0.83 11 0.96 0.42 0.46 0.61 0.66 0.81 12 0.98 0.40
0.44 0.62 0.66 0.78 German; std = 0.00 – 0.07. 1 0.43 0.58 0.44 0.48 0.16 0.55
2 0.60 0.58 0.43 0.49 0.18 0.56 3 0.77 0.59 0.43 0.47 0.17 0.67 4 0.76 0.61
0.43 0.44 0.21 0.71 5 0.98 0.63 0.38 0.48 0.26 0.73 6 0.99 0.65 0.43 0.54 0.29
0.74 7 1.00 0.65 0.43 0.58 0.31 0.76 8 0.99 0.64 0.43 0.60 0.35 0.75 9 1.00
0.64 0.44 0.59 0.33 0.69 10 0.99 0.64 0.45 0.58 0.35 0.65 11 1.00 0.63 0.43
0.58 0.33 0.58 12 1.00 0.63 0.38 0.58 0.33 0.59 Spanish; std = 0.00 – 0.02. 1
0.39 0.49 — 0.50 0.29 0.43 2 0.55 0.52 — 0.56 0.31 0.50 3 0.56 0.53 — 0.58
0.31 0.52 4 0.96 0.55 — 0.62 0.37 0.57 5 0.98 0.56 — 0.64 0.41 0.59 6 0.99
0.56 — 0.68 0.45 0.62 7 1.00 0.57 — 0.68 0.47 0.64 8 1.00 0.58 — 0.75 0.49
0.69 9 1.00 0.58 — 0.74 0.51 0.66 10 1.00 0.58 — 0.77 0.56 0.62 11 1.00 0.57 —
0.77 0.55 0.59 12 1.00 0.56 — 0.76 0.54 0.50

Table 11: Full results for the BERT monolingual models over Chinese, German,
and Spanish.

Layer NSP Sent. Ord. Discourse Conn. Nuclearity Relation EDU segment. Cloze
ST. BERT-Large (English); std = 0.00 – 0.02. 1 0.34 0.26 0.29 0.60 0.37 0.23
0.61 2 0.53 0.25 0.33 0.62 0.40 0.28 0.67 3 0.57 0.26 0.32 0.63 0.42 0.29 0.66
4 0.60 0.29 0.32 0.63 0.42 0.40 0.66 5 0.64 0.30 0.35 0.65 0.43 0.43 0.69 6
0.82 0.31 0.37 0.66 0.44 0.45 0.68 7 0.87 0.31 0.39 0.66 0.45 0.44 0.69 8 0.95
0.30 0.39 0.66 0.45 0.44 0.72 9 0.96 0.32 0.40 0.67 0.47 0.43 0.70 10 0.96
0.32 0.40 0.67 0.46 0.49 0.71 11 0.97 0.32 0.39 0.68 0.47 0.53 0.70 12 0.97
0.33 0.47 0.77 0.57 0.58 0.71 13 0.97 0.34 0.52 0.78 0.61 0.56 0.71 14 0.97
0.34 0.57 0.81 0.65 0.59 0.73 15 0.97 0.34 0.61 0.82 0.67 0.58 0.75 16 0.97
0.35 0.60 0.83 0.67 0.55 0.75 17 0.97 0.34 0.62 0.82 0.68 0.53 0.82 18 0.98
0.36 0.63 0.82 0.68 0.54 0.82 19 0.99 0.37 0.63 0.82 0.67 0.50 0.83 20 0.99
0.34 0.63 0.81 0.67 0.48 0.84 21 0.99 0.35 0.63 0.81 0.65 0.41 0.83 22 0.99
0.35 0.61 0.81 0.65 0.37 0.84 23 0.99 0.34 0.59 0.80 0.63 0.36 0.82 24 0.99
0.33 0.57 0.77 0.58 0.31 0.81

Table 12: Full results of English BERT-large.

##  Appendix D Frozen vs. Fine-tuned BERT Layers

Figure 9: A comparison of BERT with frozen vs. fine-tuned layers.

##  Appendix E Full Results of Models with Average Pooling

Figure 10: Full results of all models over English with average pooling on all
tasks except in EDU segmentation (with the only differences over Figure 2
being for BERT and ALBERT, where we originally used [CLS] embeddings on two-
text classification probing tasks).

##  Appendix F [CLS] vs. Average Pooling in English BERT-base Model

Average pooling generally performs worse than [CLS] embeddings in the last
layers of BERT.

Figure 11: Comparison of [CLS] vs. average pooling embeddings for BERT-base
across the five tasks for English. Please note that sentence ordering and EDU
segmentation are always performed with average pooling embeddings and sequence
labelling at the (sub)word level, respectively.


left (right) columns in the figure show data taken in the vicinity of
$\gamma=\pi$ ($\gamma=0$). The axes are chosen such that a straight line
indicates scaling according to Eq. (37). For all datasets we see a straight
line over at least two decades, which renders the fit results trustworthy, and
in agreement with our simplified theoretical model. That said, in the
simulated data, we observe a slight upward bending in the small
$\varepsilon$-regime. This is to be expected as Eq. (36) constitutes only a
minimal approximate model, which completely disregards the interplay between
the two constituent drives that are likely to contribute an additional
$\varepsilon$ dependence. However, these interplay effects are expected to
fade away as time-scale separation becomes more pronounced in the limit $N\gg
1$. This behavior is corroborated by the experimental data in Fig. S14 (c) and
(d) obtained with $N=300$, where almost no bending is visible anymore. A few
experimental data points for small $\varepsilon$ in (d) were left out of
consideration when computing the fitting line, since the corresponding time-
traces did not reach the $1/\mathrm{e}$ threshold by the time the experiment
ended. Overall, simulation and experimental data yield very similar heating
rates, described by a power-law with an exponent close to $2$, consistent with
our theoretical analysis.

###  VIII.4 Frequency dependence of heating timescales

In Floquet systems, heating is predominantly controlled by the frequency of
the employed drive, where increased frequencies lead to a suppression of
heating rates and a corresponding increase of heating timescales. In our
system the frequency of the Floquet drive is given by
$\omega_{F}=2\pi/(N\tau)$. An increase of driving frequency can thus be
accomplished but tuning either $\tau$ or $N$. Since, finite $\varepsilon$ has
vanishing influence on the overall frequency of the Floquet drive, let us
first investigate the limit $\varepsilon\rightarrow 0$. Then, the model of Eq.
(37) predicts heating timescales (in units of Floquet cycles) to scale as
$\Gamma^{-1}/N\sim 1/(N(J\tau)^{2})$ which we confirm numerically (Fig. S15).
With increasing $\varepsilon$, the increase of $\Gamma^{-1}/N$ as a function
of $N$ is slowed down until a break-even point is reached where $\varepsilon$
becomes the dominant source of heating (Fig. S15 (a)). Notice that as
$N\approx 10$ timescale separation is gradually lost while interplay effects
arising from interference/resonance between the two drives start becoming
relevant; this implies that Eq. (37) does not capture the heating dynamics
anymore in the non-timescale-separated regime. In fact, in this regime, the
drive induces new effective Hamiltonians whose properties depend sensitively
on the specific value of $N$.

Fig. S15: Numerical simulation – (a) $N$-dependence of the heating time
$\Gamma^{-1}$ in units of Floquet cycles for 4 different values of
$\varepsilon$. The dashed lines correspond to a least-square fit. We find a
scaling of $\Gamma^{-1}/N\sim N^{-0.88}$ (b) $J\tau$-dependence of the heating
time $\Gamma^{-1}$ in units of Floquet cycles for 4 different $N$. In both
panels all points are averaged over 10 random graph realizations. Dashed lines
correspond to least-square fits: we find a scaling of We find a scaling of
$\Gamma^{-1}/N\sim(J\tau)^{-\alpha}$ with $\alpha=2.18$ ($N=15$),
$\alpha=2.20$ ($N=50$), $\alpha=2.39$ ($N=100$) and $\alpha=2.50$ ($N=300$).
The system size is $L=16$. Remaining parameters are as in Fig. S8.

##  IX Two-frequency Floquet engineering

Fig. S16: Experimental characterisation of the heating dynamics at small N.
Complex interplay between the fast and slow periodic drives is seen when the
condition $N=T/\tau\gg 1$ is not met. (A) Movie showing time-series data from
103 experiments with different values of flip angle $\gamma$ in $[0,1.1\pi]$
and $N=8$ $\hat{\textbf{x}}$-pulses between $\hat{\textbf{y}}$-pulses. Colors
represent absolute value of signal $|\langle\mathcal{I}_{x}\rangle|$ (see
colorbar). Floquet cycle number $M$ runs vertically on a logarithmic scale.
Data are taken past $10^{4}$ Floquet cycles. (B) Movie similar to (A) but with
$N=9$ $\hat{\textbf{x}}$-pulses between the $\hat{\textbf{y}}$-pulses. In both
movies (A) and (B), $\hat{\textbf{x}}$-pulse flip angle was calibrated to
$\vartheta=\pi/2$. (C-D) Line cut of (A), (B), respectively, at
$\gamma=0.244\pi$. In (C), different plateaus correspond to micromotion within
a Floquet cycle. The data shown in (A) and (B) is identical to the data shown
in the main text Fig. 5. Fig. S17: Numerical simulation – Magnetization
$\langle\mathcal{I}_{x}\rangle$ as a function of the number of kicks and
$\gamma$ for $N=8$ (a) and $N=9$ (b). The system size is $L=16$. Remaining
parameters are as in Fig. S8. Fig. S18: Numerical simulation – Same as Fig.
S17 for $N=2$ (a) and $N=3$ (b). The system size is $L=16$. Remaining
parameters are as in Fig. S8.

When timescale separation between the fast and slow drive is lost, both drives
mutually affect each other, and new effective Hamiltonians arise. To
understand the relevant mechanisms let us investigate once more the effective
Hamiltonian that captures the daynamics generated by the fast
$\hat{\textbf{x}}$ drive (Eq. (10)).

For large $N$, $\hat{\textbf{x}}$ magnetization conservation is approximately
present in the effective Hamiltonian, as terms that violate it are suppressed
as $1/N$. In the small $N$ regime, this is generically not true. As a
consequence $\left[U_{0},U_{x}\right]\neq 0$ (cf. Eqs. (19) and (20)) and the
dynamics over $M$ Floquet cycles is governed by

$\displaystyle
U_{F}^{M}\simeq\left(\mathrm{e}^{-iT\bar{\mathcal{H}}^{(0)}}U_{x}^{N}U_{z}\right)^{M}=\left(\mathrm{e}^{-iT\bar{\mathcal{H}}^{(0)}}U_{\vec{n}}\right)^{M},$
(38)

where $\bar{\mathcal{H}}^{(0)}$ (given in Eq. (10)) depends on $N$. During
each driving period we now effectively rotate the system around a new axis
$\vec{n}$ which is composed of an $\hat{\textbf{x}}$\- and a
$\hat{\textbf{z}}$-rotation:
$U_{\vec{n}}=U_{x}^{N}U_{z}\equiv\mathrm{e}^{-i\alpha\sum_{j}\vec{n}\cdot\vec{I}_{j}}$,
where

$\displaystyle\alpha$ $\displaystyle=$
$\displaystyle\frac{c}{\sin(c)}\sqrt{1+\sin(\vartheta/2N)^{2}\cos(\gamma/2)^{2}+\sin(\gamma/2)^{2}},$
$\displaystyle\vec{n}\\!$ $\displaystyle=$
$\displaystyle\\!\left[\\!\sin(\vartheta\\!/2N)\\!\cos(\gamma\\!/2),-\\!\sin(\\!\vartheta\\!/2N)\\!\sin(\\!\gamma\\!/2),\sin(\\!\gamma\\!/2)\\!\cos(\\!\vartheta\\!/2N)\\!\right]^{T}$
(39) $\displaystyle\times$ $\displaystyle
1/\sqrt{1+\sin(\vartheta/2N)^{2}\cos(\gamma/2)^{2}+\sin(\gamma/2)^{2}}$

with $c=\mathrm{arccos}(\cos(\vartheta/2N)\cos(\gamma/2))$. Note how a change
in $\gamma$ now modifies both the value of $\alpha$ and the direction
$\vec{n}$ of the rotation. Thus, the (single- and two-cycle) effective
Hamiltonian around $\gamma=\pi$ can be significantly different from that
obtained close to $\gamma=0$. Remember that especially the two-cycle effective
Hamiltonian for $\gamma=0$ and $\gamma=\pi$ emerged to be identical in the
time-scale separated regime leading to very similar heating characteristics
around these two points of interest. To illustrate the difference and the
sensitivity of the effective Hamiltonian as a function of $\gamma$ and $N$ in
the non-time-scale separated regime, let us exemplary discuss the two cases
$N=8$ and $N=9$.

For $N=8$, the total accumulated kick angle $\vartheta/2N=2\pi$ so that
$\alpha=\gamma$ and $\vec{n}=(0,0,1)^{T}$. Moreover, $N=8$ yields
$\bar{\mathcal{H}}^{(0)}=\overline{\mathcal{H}}$. Thus, the toggling frame
expansion of Eq. (38) for the two-cycle effective Hamiltonian is identical (to
leading order) at $\gamma=0,\pi$ eventually inducing similar heating behaviour
around these two points (see Fig. S17 (a)).

In contrast, for $N=9$ we obtain $\alpha\rightarrow\pi/2$ and
$\vec{n}\rightarrow(1,0,0)^{T}$ as $\gamma\rightarrow 0$, whereas for
$\gamma\rightarrow\pi$ we find $\alpha\rightarrow\pi$ with $\vec{n}\rightarrow
1/\sqrt{2}(0,-1,1)^{T}$. A single such rotation induces the following
transformation:

$\displaystyle I_{jz}\rightarrow I_{jy},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ I_{jy}\rightarrow
I_{jz},\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
I_{jx}\rightarrow-I_{jx}.$ (40)

The toggling-frame expansion of Eq. (38) now yields the toggling frame
Hamiltonians

$\displaystyle\mathcal{H}_{m}$ $\displaystyle=$
$\displaystyle\sum_{j<k}b_{jk}\bigg{(}\frac{3}{2}\bigg{[}\mathcal{H}_{\mathrm{ff}}+(-1)^{m}\mathcal{G}_{\mathrm{c}}(N,\pi/2)\mathcal{H}_{\mathrm{dq}}$
$\displaystyle-\tilde{\mathcal{H}}_{\mathrm{ff}}\mathcal{G}_{\mathrm{s}}(N,\pi/2)\bigg{]}-\vec{I}_{j}\vec{I}_{k}\bigg{)}$
$\displaystyle+\sum_{j}c_{j}\left(\mathcal{F}_{m,+}(N,\pi/4)I_{jz}+\mathcal{F}_{m,-}(N,\pi/4)I_{jy}\right),$

where

$\displaystyle\mathcal{F}_{m,\pm}(N,\pi/4)$ $\displaystyle=$
$\displaystyle\frac{(-1)^{m}}{2}\left(\mathcal{G}_{\mathrm{c}}(N,\pi/4)-\mathcal{G}_{\mathrm{s}}(N,\pi/4)\right)$
(42)
$\displaystyle\pm\frac{1}{2}\left(\mathcal{G}_{\mathrm{c}}(N,\pi/4)+\mathcal{G}_{\mathrm{s}}(N,\pi/4)\right).$

The leading order two-cycle effective Hamiltonian is readily obtained from
$\mathcal{H}_{1}+\mathcal{H}_{2}$ and yields

$\mathcal{H}_{1}+\mathcal{H}_{2}=\overline{\mathcal{H}}+\frac{1}{2N}\sum_{j}\\!c_{j}\left(I_{jz}\\!-I_{jy}\right).$
(43)

Although $\overline{\mathcal{H}}$ preserves $\hat{\textbf{x}}$-magnetization,
the single particle terms
$\frac{1}{2N}\sum_{j}\\!c_{j}\left(I_{jz}\\!-I_{jy}\right)$ do not. Thus,
around $\gamma=\pi$, we expect to find a decreased lifetime of the
corresponding DTC order for $N=9$ as compared to $N=8$. We confirm this result
numerically (cf. Fig. S17). Contrasting Fig. S17 (a) with (b) allows to
visualize the differences in the heating dynamics induced by a minimal change
in $N$ and the corresponding change of the effective Hamiltonian not only
around $\gamma=\pi$ but in particular also around $\gamma=0$. In comparing
Fig. S17 to the corresponding experimental results in Fig. S16 and Fig. 5 of
the main text, we find a qualitative agreement also for small values of $N$.
The reduced lifetime of the DTC order in the experimental data for $N=8$ as
compared to the simulations might be attributed to drive imperfections: due to
the small region of rigidity, already small imperfections might induce a decay
of the DTC order.

Note that for our choice of $\vartheta=\pi/2$ we can engineer $8$ different
families of effective Hamiltonians characterised by the $8$ different
rotations $U_{x}^{N}U_{z}$. Other choices of $\vartheta$ can yield even more
different families. Interestingly, the different effective Hamiltonians show
quite different behaviour with respect to their prethermal properties as well
as the formation of DTC order, which not only concerns the lifetime but also
the rigidity away from $\gamma=\pi$. Most prominently, this can be observed
for $N=2$ (cf. Fig. S18 (a)) where the region of rigidity is significantly
increased as compared to (for instance) $N=3$ (Fig. S18 (b)).

In conclusion, our experimental and numerical studies show that control over
$N$ and $\gamma$ is sufficient to case-specifically design effective
Hamiltonians, heating dynamics and non-equilibrium order.

Fig. S19: Numerical simulation – Random graph dependence of the heating time
$\Gamma^{-1}$ as a function of $\varepsilon$ for $\gamma=\pi+\varepsilon$ in
(a) and $\gamma=0+\varepsilon$ in (b) for $N=255$. We show $20$ different
realizations. Different colors represent different random graphs, each with
$L=16$, generated using $r_{\mathrm{min}}=0.7$ and $r_{\mathrm{min}}=0.8$. For
each graph we independently determine $J$ and set $\tau J=0.2$. Solid lines
serve as guide to the eye.

##  X  Random graph independence

To ensure that our results are not affected by the choice of the random graph
realization, we show in Fig. S19 the heating times for $20$ different random
graph realizations around the two special points of interest $\gamma\approx 0$
and $\gamma\approx\pi$. In both regimes, the heating rates show no significant
dependence on the specific random graph realization. Instead the long-range
interaction terms appear to have a self-averaging effect already at moderate
system sizes. This is additionally enhanced by the ergodicity of the dynamics,
promoted by adding small random noise to $\tau$.