Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation
Under review as a conference paper at ICLR 2025

SCALING INSTRUCTION-TUNED LLMS TO MILLION-
TOKEN CONTEXTS VIA HIERARCHICAL SYNTHETIC
DATA GENERATION

Anonymous authors
Paper under double-blind review

ABSTRACT

Large Language Models (LLMs) struggle with long-context reasoning, not only
due to the quadratic scaling of computational complexity with sequence length but
also because of the scarcity and expense of annotating long-context data. There
has been barely any open-source work that systematically ablates long-context
data, nor is there any openly available instruction tuning dataset with contexts sur-
passing 100K tokens. To bridge this gap, we introduce a novel post-training syn-
thetic data generation strategy designed to efficiently extend the context window
of LLMs while preserving their general task performance. Our approach scalably
extends to arbitrarily long context lengths, unconstrained by the length of avail-
able real-world data, which effectively addresses the scarcity of raw long-context
data. Through a step-by-step rotary position embedding (RoPE) scaling training
strategy, we demonstrate that our model, with a context length of up to 1M tokens,
performs well on the RULER benchmark and InfiniteBench and maintains robust
performance on general language tasks.

1

INTRODUCTION

The capabilities of Large Language Models (LLMs) have significantly advanced, enabling impres-
sive performance across a wide range of natural language processing tasks (Wu et al., 2023; Jiang
et al., 2023; Wei et al., 2022). However, managing long contexts remains a major challenge, which
limits the practical utility of LLMs in tasks such as document comprehension and summarization,
code generation, lifelong conversations, and complex agent scenarios (Liu et al., 2023; Meng et al.,
2023). Extending context lengths to 1M tokens marks a critical breakthrough for applications re-
quiring processing beyond a 128K token limit. For instance, company-wide document retrieval
benefits from efficiently analyzing extensive organizational histories stored in unstructured formats,
while interconnected project timelines and legal documents gain from enhanced reasoning across
multi-document datasets.

To extend the context length of LLMs, current approaches focus on either architectural innova-
tions like efficient attention mechanisms (Katharopoulos et al., 2020; Gu & Dao, 2024) or scaling
positional embeddings (Chen et al., 2023; Peng et al., 2023) and continual pretraining on natural
long-form data, such as books and web data. However, the RULER benchmark (Hsieh et al., 2024)
shows that many models struggle to maintain consistent performance as context length increases,
even when claiming to support longer contexts. This highlights the need for high-quality instruction
data to fully utilize the nuances of long-form content. Acquiring such data is challenging and costly,
as open-source datasets often fall short in document length, relevance, and tasks requiring genuine
long-range understanding. To date, no open-source instruction-tuning datasets exceed 100K tokens,
creating a significant gap between theoretical and practical long-context capabilities of LLMs (Li
et al., 2024; Zhao et al., 2024).

To address limitations in extending LLM context length, we propose an effective long-context in-
struction data generation pipeline, as illustrated in Figure 1. Our pipeline leverages short-context
models to create long-context instruction data using three key methods: (a) Hierarchical question
ordering: structuring questions in a logical sequence to ensure coherent reasoning across contexts;
(b) Diverse question type pool: maintaining a wide range of question types, including hierarchical-

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: High-level overview over our approach to automatically generate QA pairs for long context
documents. (1) In the first step, we split a document into small and medium chunks which are then
(2) summarized by an off-the-shelf LLM requiring only smaller context windows. In (3) we sample
summaries at different localities in a hierarchical manner, balancing local and global views of the
original document. In (4) we generate questions based on the sampled summaries. In the right
panel, we show a subset of prompts used to generate diverse and complex questions, given the
sampled summaries.

aware, multi-hop, local-specific, and other complex types to handle varied tasks; and (c) Multi-
document integration:
incorporating multiple documents to generate data with arbitrary context
lengths. The contributions of this paper are threefold:

1. Extensive and scalable long-context data generation strategy: We present, to the best of our
knowledge, the first extensive strategy for synthetically generating long-context data with com-
prehensive ablation tests and evaluations. Our highly scalable approach is unconstrained by the
length of available real-world data, effectively combining multiple documents with diverse, com-
plex questions. This hierarchical method ensures logical coherence and sequence integrity.

2. Extensive evaluation of core strategies: We conduct extensive evaluations on shorter context
lengths (100K and 180K) to demonstrate the effectiveness of our hierarchical strategy, multi-
document combinations, and diverse question-answer pair generation. These evaluations validate
that our core strategies work well across various tasks and context lengths.

3. Scaling to 1M context length: We successfully extend LLaMA-3.1-8B-Instruct to a context
length of 1 million tokens. Our model significantly outperforms the LLaMA-3.1-8B-Instruct
model in zero-shot RoPE scaling to a 1M context window on the RULER benchmark and sur-
passes the gradientai/Llama-3-8B-Instruct-Gradient-1048k model trained by Gradient AI. Ad-
ditionally, our model outcompetes LLaMA-3.1-8B-Instruct on InfiniteBench while maintaining
strong performance on LongBench and MMLU.

The remainder of this work is organized as follows. In Section 2 we place our work in the land-
scape of existing literature around methods to address long context capabilities of LLMs. Section 3
presents our method for generating long-context instruction tuning data. Our approach is then val-
idated in Section 4 with a series of extensive and representative experiments. Finally, we conclude
in Section 5.

2 RELATED WORK

Adapting transformers to enable longer context capabilities is a critical area of research in natural
language processing. This effort primarily focuses on three key directions: (1) architectural modifi-
cations to the transformer model itself, (2) improvements in positional embedding techniques, and
(3) the development and utilization of more extensive long-context datasets.

Efficient Attention Mechanisms. To address the quadratic computational and memory demands
of standard transformer self-attention, researchers have developed various architectural modifica-
tions to improve efficiency and extend context lengths. Notable examples include Longformer
(Beltagy et al., 2020), which combines sliding window and global attention, and BlockTransformer
(Ho et al., 2024), which employs hierarchical global-to-local modeling. Linear Attention methods

2

📘Long Context DocumentLLM QA GeneratorLocality-guidedHierachical Summary Sampling📄📃📘134Depth vs. Breadth QA GenerationGiven the following context and full summary of a book, generate a question-answer pair that relates to the full summary but can be answered better with knowledge from the given context.Context: <local_summary>Full Summary: <full_summary>...Hierarchy-aware QA<summary_1><summary_2><summary_3>You are a Professor designing afinal exam for an advancedinterdisciplinary course. Create 1complex question requiring deepanalysis and synthesis ofinformation from all three chunks. ...Multi-Hop QA<local_summary>Given the context information and noprior knowledge, generate content basedon the below query.You are a Teacher/Professor. Create 1specific question about the details,events, characters, and settings fromthe context provided. This questionshould have an exact, unambiguous answerthat can be directly found in the giveninformation....Local Specific QA<local_summary>Given the context information and noprior knowledge.Generate content based on the belowquery. You are a Teacher/Professor.Your task is to set up 1 diversetemporal question about...Complex & Diverse QA📄📄📄Pool of Local Summaries📄📃📃📄2📘Under review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

(Katharopoulos et al., 2020) reformulate self-attention for linear complexity, while InfiniteTrans-
former (Munkhdalai et al., 2024) incorporates unbounded long-term memory through continuous-
space attention. State space models like Mamba (Gu & Dao, 2024) capture long-range dependencies
efficiently without explicit attention mechanisms. Despite these advancements, bridging the gap
with high-quality data remains a critical challenge and is the focus of this work.

Position Embedding Extension. Advances in positional encoding methods have enabled language
models to handle longer sequences effectively. Techniques like RoPE (Su et al., 2023), ALiBi
(Press et al., 2022), and xPos (Sun et al., 2022) have emerged as prominent solutions. RoPE has
gained widespread adoption in LLaMA (Touvron et al., 2023), b) and PaLM (Anil et al., 2023),
due to its ability to represent relative positions and its theoretical grounding in the complex plane.
A breakthrough showed that RoPE’s embeddings could extend to longer contexts with minimal
or no fine-tuning (Men et al., 2024), leading to two key approaches: Positional Interpolation (PI)
(Chen et al., 2023) which linearly scales positional indices to extend context length, and NTK-aware
Scaling RoPE (Peng et al., 2023) which combines high-frequency extrapolation with low-frequency
interpolation. While these developments improve model performance with longer inputs, they rely
heavily on limited long-context data for fine-tuning.

Long Context Data. Recent work, such as LongT5 (Guo et al., 2022) and LongAlpaca (Chen
et al., 2024), has shown the benefits of additional pretraining on long sequences, enabling models to
better capture extended context. Methods like combining multiple short-context sequences (Xiong
et al., 2023) have also emerged as promising ways to efficiently extend context lengths. However,
a significant gap remains in generating high-quality instruction-tuning data exceeding 100K context
lengths. Few open-source efforts address this need. Our work introduces a scalable pipeline for
generating long-context instruction-tuning data by systematically combining multiple documents,
diverse questions, and a hierarchical strategy to ensure coherence and structure.

Synthetic Data Generation. Synthetic data generation offers a promising path for scaling language
models across diverse tasks and complex instructions. AutoEvol-Instruct (Zeng et al., 2024), au-
tomates the evolution of instruction datasets using large language models, reducing the need for
extensive human intervention. WizardLM (Xu et al., 2023) employs Evol-Instruct to iteratively
evolve and scale instruction complexity, achieving strong results on benchmarks like MT-Bench and
Vicuna’s evaluation set. Auto Evol-Instruct (Zeng et al., 2024) further refines this process with an
iterative evolution strategy, while Self-Instruct (Wang et al., 2023) enhances instruction-following
performance through data synthesis. Our work extends this research by generating long-context data
tailored for instruction tuning.

3 METHOD

In this section, we describe our methodology for generating coherent instructions from a single
document and scaling it to multiple documents to curate long-context datasets beyond the context
length of available raw data. Section 3.1 outlines our strategy for ensuring (1) quality and complexity
and (2) coherent ordering of generated question-answer pairs. Section 3.2 expands on scaling to
longer context lengths using multiple documents. Figure 1 provides an overview of our long-context
synthetic data generation pipeline.

3.1 COHERENT INSTRUCTIONS FROM A SINGLE DOCUMENT
The quality of long-context instruction-tuning datasets is driven by two key factors: (1) the com-
plexity and diversity of the generated instructions, and (2) the structured ordering of questions and
instructions. To address these, we devised a bifurcated strategy targeting each component.

Quality, Diversity, and Complexity of Instructions. As illustrated in Figure 1, our methodology
for generating rich, diverse, and complex instructions leverages the key insight that short-context
models can be used to generate long-context instruction-tuning data. The core approach involves
dividing the input document into smaller chunks, typically 4K tokens, enabling models optimized
for shorter contexts to process these segments with greater precision and clarity. We curate an
initial set of prompts covering multiple dimensions of instruction complexity, such as temporal
reasoning, thematic inquiry, and character-based scenarios (full set in Appendix B). During question-
answer pair generation, a small chunk and one question are randomly selected to generate a pair. To
ensure broader contextual understanding, we incorporate multi-hop questions spanning 2–4 chunks,
enabling cross-chunk question-answer pairs.

3

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Figure 2: High-level overview over our approach to generate order-following QAs. (1) Input a raw
long context document. (2) Split the document into small, medium, and global chunks, and generate
summaries at each level. (3) The first QA is based on the global summary. (4) We randomly select
a medium chunk to generate a QA, (5) then delve deeper by selecting a small chunk within it for
another QA. (6) To continue, the process alternates between exploiting the same small chunk or
exploring new medium or small chunks to generate further QAs.

Figure 3: High-level overview over our approach to curate long context data using multiple doc-
uments. (1) Diverse and hierarchical QAs are generated at different levels of granularity for each
document. (2) N hierarchical and diverse QAs are sampled and extracted from each document.
(3) QAs from different documents are combined, maintaining a balance of hierarchical and diverse
questions across the entire set. N = 5 in our algorithm, and when we revisit previous documents
in step (3), we sample 3 hierarchial questions for each document with 60 % probability as well as 9
total diverse questions from all previous documents.

Ensuring Coherent Order. To ensure logical and coherent QA generation, we use a hierarchical
strategy to split, summarize, and generate questions from long documents (see Figure 2), balancing
exploration and exploitation. The document is first divided into large sections of 12K tokens, then
into smaller 4K-token chunks linked hierarchically to connect broader and granular segments. The
first QA is based on the global summary to give a high-level overview of the document. Then,
we randomly select a medium chunk to generate a QA, and then delve deeper by selecting a small
chunk within it for another QA. To continue, the process alternates between exploiting the same
small chunk or exploring new medium or small chunks to generate further QAs. This iterative
process ensures a balance between specificity and diversity, covering both localized details and
broader document sections. The hierarchical structure ensures logical progression from broad QAs
to detailed ones. The detailed algorithm and pseudocode are provided in Appendix A.

3.2 EXTENDING TO LONGER CONTEXT LENGTHS USING MULTIPLE DOCUMENTS
Here we extend our methodology to handle longer contexts by concatenating multiple documents
and generating coherent hierarchical and diverse QA pairs across them. The workflow is visualized
in Figure 3 and the detailed algorithm is provided in Appendix A. Below, we clearly define the pa-
rameters N1, N2, and N3, which govern the selection of hierarchical and diverse QA pairs, ensuring
logical continuity and broad reasoning across documents. For each document, the process proceeds
as follows:

1. N1 hierarchical QA pairs and N1 diverse QA pairs: After processing each document, N1 =
5 hierarchical follow-up questions are added. These questions are designed to capture
contextually related information within the document, creating a logical order of reasoning
and flow across sections. Moreover, another N1 = 5 diverse QA pairs for this document is
added as well, designed to capture specific details of the document.

4

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

2. N2 diverse QA pairs: Next, N2 = 9 diverse QA pairs are added. These questions are
sampled from all previously visited documents where diverse QA pairs have not already
been sampled. This approach ensures cross-referencing between documents.

3. N3 revisiting hierarchical QA pairs: For every previously visited document, there is a 60%
probability of sampling N3 = 3 hierarchical follow-up questions. These are added to revisit
earlier contexts, fostering a richer and interconnected understanding of the content.

This process is repeated iteratively for all K documents in the dataset to create a comprehensive
instruction-tuning dataset that balances within-document reasoning, cross-document relationships,
and revisiting earlier content for contextual continuity. We also present an example of a concatenated
data example in Appendix C.

4 EXPERIMENTS

In this section, we validate our long-context data generation approach through a series of exper-
iments.
In Section 4.2, we extend LLaMA-3.1-8B-Instruct to a 1M context-length model using
stepwise RoPE scaling and hierarchical, diverse QA data generated by Qwen-2-72B. Our 1M model
delivers excellent results on ultra-long contexts while maintaining strong performance on short and
medium-length contexts. In Section 4.3, we evaluate robustness using smaller and same-sized gen-
erator models (Qwen-2.5-7B and LLaMA-3.1-8B-Instruct), confirming our models achieve strong
performance across ultra-long, short, and medium contexts. These findings highlight the scalability
and effectiveness of our approach across generator model sizes. In Section 4.4, we present abla-
tion studies showing how our hierarchical strategy and diversified questions significantly improve
long-context instruction tuning, focusing on 180K with two documents.

4.1 SETUP

Models. We use LLaMA-3.1-8B-Instruct as the base model for instruction-tuning, given its capa-
bility as a leading open-source LLM. To validate robustness, we employ various generator models
for synthetic data: Qwen-2-72B-Instruct (large, high-quality data), Qwen-2.5-7B-Instruct (smaller),
and LLaMA-3.1-8B-Instruct (same size). This demonstrates that our improvements are not reliant on
very large models and that smaller models can achieve similar gains. We also benchmark against the
Gradient AI model (gradientai/Llama-3-8B-Instruct-Gradient-1048k), a 1M context-length model
trained on 1.4 billion tokens, showing that our method outperforms existing baselines.

Hardware. We fine-tuned our models on a SLURM cluster using 8 to 32 H100 GPUs across up to 4
nodes, connected via InfiniBand for efficient multinode training. We used FSDP to shard the model
across GPUs and implemented DeepSpeed Ulysses sequence parallelism for long-context training.

Datasets. Our primary dataset is the Together long books dataset1, processed into approximately
1.4 billion tokens, distributed across these stages: 2000 samples of 180K tokens, 1280 samples of
350K tokens, 600 samples of 650K tokens, and 200 samples of 1M tokens. We generated 582,900
QA pairs with hierarchical and diverse questions for robust instruction-tuning using the Together AI
inference API 2. By sending 32 simultaneous API requests, it took about two days to create our full
long-context instruction dataset, comprising 7,772 books. For each book, we generated 25 hierar-
chical and 50 diverse questions, resulting in 582,900 QA pairs alongside global summaries. During
training, we calculate loss solely on answers, masking out questions and context to ensure the model
focuses on reasoning and generating accurate answers without being penalized for reproducing input
content.

Evaluation Protocol. We evaluated our models using: 1) InfiniteBench (Zhang et al., 2024): De-
signed for LLMs on extended contexts, it includes tasks like key-value retrieval, summarization,
and QA on data exceeding 100K tokens. We evaluated the first 150 samples per task, excluding
coding tasks as our data lacks code. 2) LongBench (Bai et al., 2024): Focused on medium-context
tasks (10K tokens), it assesses summarization, QA, and fact-checking across multiple domains, of-
fering a measure of general capabilities. We excluded coding tasks. 3) RULER (Hsieh et al., 2024):
RULER is a synthetic benchmark designed to evaluate how well models handle complex, real-world
tasks in long contexts. Unlike traditional retrieval-based tasks like Needle-in-a-Haystack (NIAH),
which focus on extracting specific pieces of information from distractor texts, RULER tests models’

1https://huggingface.co/datasets/togethercomputer/Long-Data-Collections
2https://api.together.xyz/

5

Under review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

(a) Context length 350K

(b) Context length 650K

(c) Context length 1M

Figure 4: Effective context length up 1M tokens using Qwen-2-72B-Instruct as generator on RULER.

Table 1: Model performance on InfiniteBench (100K tokens) using Qwen-2-72B-Instruct as generator.

Metric

Retrieve.PassKey
Retrieve.Number
Retrieve.KV
En.Sum
En.QA
En.MC
En.Dia
Math.Find

Average

LLaMA-3.1-
8B-Instruct

gradient-
ai-model

100.00
95.33
42.66
27.63
24.83
68.00
16.66
35.33

51.31

100.00
99.83
15.60
17.02
14.31
57.20
5.00
19.42

41.04

180K

350K

650K

1M

100.00
99.33
88.66
24.01
34.26
74.00
18.00
37.33

100.00
100.00
92.00
23.51
33.23
72.00
18.00
35.33

100.00
100.00
63.33
23.68
31.72
75.33
22.00
36.00

100.00
100.00
57.33
23.06
31.97
74.00
16.00
36.00

59.45

59.26

56.51

54.80

ability to comprehend deeper relationships and manage long-range dependencies. Given a specified
context length, RULER generates synthetic tasks across multiple categories, including multi-hop
reasoning and document tracing, and measures the model’s accuracy. In our evaluation, we sampled
130 tasks for each context length across 13 categories, totaling over 150 million tokens. 4) MMLU
(Hendrycks et al., 2021): This benchmark evaluates general model performance across multiple
domains, assessing both breadth and depth of understanding. It includes tasks spanning STEM, hu-
manities, and social sciences, with varying difficulty levels. MMLU ensures that improvements in
handling long-context tasks do not cause regression in overall model capabilities.

4.2 MAIN RESULTS: SCALING UP TO LONGER CONTEXT LENGTHS (350K, 650K, 1M)

To extend Llama-3.1-8B-Instruct to a 1M context model, we applied stepwise rope scaling. Training
started with 180K tokens and progressed through checkpoints at 350K, 650K, and 1M tokens, con-
catenating 4, 8, and 12 documents as per the algorithm in Section 3.2. We compiled 2000 samples at
180K, 1280 at 350K, 600 at 650K, and 200 at 1M tokens. Data was generated using Qwen-2-72B,
fine-tuned on Llama-3.1-8B-Instruct with rope scaling at a 6e-5 learning rate for 1 epoch. Training
the 650K model took 30 hours, and the 1M model took an additional 52.5 hours.

An earlier ablation test combining two documents (Section 4.4) showed that combining hierarchical
and diverse questions with a fixed number of QAs and global summarization is optimal for handling
long contexts. We extended this setup for ultra-long context data, with each document followed by
N1 = 5 hierarchical and N1 = 5 diverse questions. When revisiting previous documents, there is
a 60% chance of extracting N2 = 3 hierarchical question from each document and N3 = 9 diverse
questions sampled from all prior documents.

Figure 4 shows the effective context lengths of the 350K, 650K, and 1M models on the RULER
benchmark. For comparison, we performed zero-shot rope scaling on the LLaMA-3.1-8B-Instruct
model and included results using input truncation for context lengths above 128K as an additional
baseline. On contexts shorter than 128K, our models performed comparably to LLaMA-3.1-8B-
Instruct and surpassed it with zero-shot rope scaling. This demonstrates the robustness of our mod-
els on short and medium contexts. For contexts longer than 128K, our models significantly outper-
formed both baselines, with their strengths becoming more evident as context length increased. Raw
evaluation results are in Appendix D.

6

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Table 2: Model performance on LongBench (10K tokens) using Qwen-2-72B-Instruct as generator.

LLaMA-3.1-
8B-Instruct

Gradient-
AI-Model

Single Document

Multi-Document

Summarization

Few-shot Learning

Synthetic Tasks

All

46.91

41.45

26.10

63.48

67.48

48.11

30.71

12.45

21.72

59.69

55.50

35.89

180K 350K 650K

1M

45.83

45.88

45.24

45.15

41.71

41.75

41.13

41.29

25.14

24.97

24.26

24.98

62.22

61.66

60.00

59.27

68.17

67.50

65.00

66.42

47.58

47.34

46.18

46.42

Table 3: Model performance on MMLU using Qwen-2-72B-Instruct as the generator.

Category

mmlu
humanities
other
social sciences
stem

LLaMA-3.1-
8B-Instruct

68.21 ± 0.37
64.23 ± 0.67
73.03 ± 0.77
77.48 ± 0.74
60.36 ± 0.83

gradient-
ai-model

60.48 ± 0.39
55.75 ± 0.69
67.04 ± 0.82
70.46 ± 0.80
51.32 ± 0.86

350K-model

650K-model

1M-model

66.29 ± 0.38
61.51 ± 0.68
72.84 ± 0.77
76.81 ± 0.74
59.44 ± 0.84

65.80 ± 0.38
61.02 ± 0.68
71.84 ± 0.78
75.27 ± 0.76
57.72 ± 0.84

65.08 ± 0.38
61.02 ± 0.68
71.84 ± 0.78
75.27 ± 0.76
57.72 ± 0.84

To further validate our approach, we compared it to the Gradient AI model (gradientai/Llama-3-8B-
Instruct-Gradient-1048k), a 1M context model, on InfiniteBench, LongBench, and MMLU bench-
marks. Table 1 compares models across context lengths on InfiniteBench, while Table 2 focuses
on LongBench. All our models (180K, 350K, 650K, 1M) consistently outperforms the Gradient
AI model on InfiniteBench, showcasing the effectiveness of our hierarchical, diversified QA-based
data-generation strategy. The 180K and 350K models scored 59.45 and 59.26, significantly exceed-
ing the LLaMA-3.1-8B-Instruct baseline of 51.31. The 650K model scored 56.51, and the 1M model
achieved a strong 54.80. 3

Notably, while the Retrieve.KV task shows the most significant improvement, tasks like Re-
trieve.Number, En.MC, and Math.Find also display meaningful gains. The improvement on Re-
trieve.KV stems from our data-generation methodology, which uses a structured mix of hierarchical
and diverse questions while revisiting prior documents. This encourages the model to associate
relevant sections, aligning with the demands of key-value retrieval and RAG techniques, where ac-
curate context memorization is critical. Beyond key-value retrieval, our model excels on other tasks:
on En.MC, the 650K model scored 75.33, surpassing the baseline (68.00) and Gradient AI model
(57.20). On Math.Find, it scored 36.00 at 650K, outperforming the Gradient AI model (19.42),
showcasing improved reasoning capabilities.

As shown in Table 2, , our models maintain robust short-context performance on LongBench, despite
being trained for significantly longer contexts (up to 1M tokens). For example, our 1M context-
length model achieves an average score of 46.42, comparable to the baseline LLaMA-3.1-8B-
Instruct model (48.11). This demonstrates that while optimized for ultra-long contexts, the model
generalizes effectively to shorter contexts, such as those on LongBench. Minor regressions in tasks
like summarization are due to trade-offs when training for extended contexts. As the model adapts
to handle extremely long contexts, small task-specific adjustments may impact short-context perfor-
mance. However, these regressions are minimal and expected, given the differences between short-
and long-context tasks. Despite these trade-offs, our model consistently outperforms the Gradient
AI model (35.89) on all LongBench tasks, demonstrating the effectiveness of our hierarchical and
diversified instruction-tuning approach.

As detailed in Table 3, our model demonstrated minimal regression in general task performance
despite significant improvements in ultra-long-context tasks. For instance, our model retained com-

3The results dropped likely due to multi-node training, as we believe our 650K and 1M models are under-

trained because of the extended time required to train and the communication overhead from NCCL.

7

Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

(a) 350K model using Qwen-2.5-7B-Instruct as generator.

(b) 650K model using Qwen-2.5-7B-Instruct as generator.

(c) 350K model using Llama-3.1-8B-Instruct as generator.

(d) 650K model using Llama-3.1-8B-Instruct as generator.

Figure 5: Effective context length using Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct as generators on
RULER.

Table 4: InfiniteBench performance with Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct as generators.

Task

Retrieve.PassKey
Retrieve.Number
Retrieve.KV
En.Sum
En.QA
En.MC
En.Dia
Math.Find

Average

LLaMA-3.1-
8B-Instruct

gradient-
ai-model

180K-
llama-gen

350K-
llama-gen

650K-
llama-gen

180K-
qwen-gen

350K-
qwen-gen

650K-
qwen-gen

100.00
95.33
42.66
27.63
24.83
68.00
16.66
35.33

51.31

100.00
99.33
13.33
17.02
15.84
61.33
4.00
26.66

42.19

100.00
99.04
85.47
25.68
33.39
58.00
19.50
36.66

57.22

100.00
100.00
89.33
26.85
35.67
60.66
14.66
32.66

57.48

100.00
100.00
42.14
26.64
33.37
66.00
20.00
35.33

52.94

100.00
99.76
89.52
26.97
32.30
63.33
27.33
30.00

58.65

100.00
100.00
85.33
27.70
29.55
61.33
21.33
34.66

57.49

100.00
100.00
52.66
26.74
29.67
64.66
23.33
38.00

54.38

petitive MMLU scores (e.g., 68.21 ± 0.37 for the baseline and 65.08 ± 0.38 for the 1M model),
whereas the Gradient AI model showed marked degradation on both MMLU and LongBench. This
reinforces the robustness of our method, ensuring that gains in ultra-long-context performance do
not compromise broader capabilities. In conclusion, our models excel at ultra-long-context tasks
on RULER and InfiniteBench, outperforming the base LLaMA-3.1-8B-Instruct and Gradient AI
models while maintaining strong performance on general tasks like MMLU and LongBench.

4.3 VALIDATING ROBUSTNESS ACROSS GENERATOR MODELS
To validate that observed improvements are not solely due to using a large generator model (e.g.,
Qwen-2-72B), we trained and evaluated models with Qwen-2.5-7B and LLaMA-3.1-8B-Instruct as
generators. By employing smaller or similarly sized models, we demonstrated the robustness and
generalizability of our hierarchical QA data-generation strategy. Additionally, we benchmarked
against the Gradient AI model (gradientai/Llama-3-8B-Instruct-Gradient-1048k), a 1M context
model trained on 1.4 billion tokens. While our models were trained only up to 650K tokens to
validate the approach, the same method can seamlessly scale to 1M tokens. Our models outper-
formed the Gradient AI baseline across all long-context benchmarks, achieving higher accuracy on
InfiniteBench and RULER, while preserving general task performance on MMLU and LongBench.

Figure 5 highlights effective context length using Llama-3.1-8B-Instruct and Qwen-2.5-7B as gen-
erators on RULER. On all settings (350K, 650K), our hierarchical approach outperformed the Gra-
dient AI model and the zero-shot baselines across context lengths. Table 4 summarizes results on
InfiniteBench (100K context length). Our approach again consistently outperformed both the base
LLaMA-3.1-8B-Instruct model and the Gradient AI model. This demonstrates that even smaller
generator models produce high-quality data for instruction-tuning.

8

Under review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Table 5: LongBench performance with Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct as generators.

Task

single-document
multi-document
summarization
few-shot learning
synthetic tasks

Average

LLaMA-3.1-
8B-Instruct

gradient-
ai-model

180K-
llama-gen

350K-
llama-gen

650K-
llama-gen

180K-
qwen-gen

350K-
qwen-gen

650K-
qwen-gen

46.91
41.45
26.10
63.48
67.48

48.11

30.75
12.45
21.72
59.70
55.50

35.89

46.48
38.69
25.28
61.56
66.17

47.23

46.64
38.75
25.10
62.79
67.75

47.72

46.53
37.54
24.68
60.50
66.00

46.20

46.20
40.76
25.05
61.92
67.11

47.95

46.70
41.90
24.83
61.56
67.60

47.97

46.28
39.31
24.90
60.69
67.10

47.00

Table 6: MMLU performance with Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct as generators.

Category

mmlu
humanities
other
social sciences
stem

LLaMA-3.1-
8B-Instruct

68.21 ± 0.37
64.23 ± 0.67
73.03 ± 0.77
77.48 ± 0.74
60.36 ± 0.83

gradient-
ai-model

180K-
llama-gen

350K-
llama-gen

650K-
llama-gen

180K-
qwen-gen

350K-
qwen-gen

650K-
qwen-gen

60.48 ± 0.39
55.75 ± 0.69
67.04 ± 0.82
70.46 ± 0.80
51.32 ± 0.86

66.99 ± 0.38
62.32 ± 0.67
72.90 ± 0.77
76.70 ± 0.74
58.67 ± 0.84

66.74 ± 0.38
61.38 ± 0.68
73.03 ± 0.76
76.93 ± 0.74
58.61 ± 0.84

65.93 ± 0.38
60.57 ± 0.68
72.87 ± 0.76
75.53 ± 0.75
57.72 ± 0.84

67.33 ± 0.38
62.81 ± 0.67
73.51 ± 0.76
76.76 ± 0.74
58.77 ± 0.84

65.78 ± 0.38
59.68 ± 0.68
73.00 ± 0.76
75.66 ± 0.75
58.14 ± 0.84

64.60 ± 0.38
59.45 ± 0.68
73.45 ± 0.77
71.87 ± 0.77
56.49 ± 0.85

Table 5 evaluates model performance on LongBench (10K context length). Despite being optimized
for ultra-long contexts, our approach retains strong performance on shorter contexts, comparable
to LLaMA-3.1-8B-Instruct. For example, with Qwen-2.5-7B-Instruct as the generator, our model
scored 47.00 at 650K, closely matching LLaMA-3.1-8B-Instruct’s 48.11. Our model also outper-
forms Gradient AI (35.89) across all LongBench tasks. Table 6 shows our models’ minimal regres-
sion in MMLU performance. The 650K trained using LLaMA-3.1-8B-Instruct as generator scored
65.93 ± 0.38, close to LLaMA-3.1-8B-Instruct (68.21 ± 0.37). In contrast, Gradient AI showed
notable regression. This underscores our hierarchical approach’s ability to support long-context
learning while maintaining general task performance.

4.4 ABLATION STUDIES

Our 100K context length single-document ablation studies, detailed in Appendix E, demonstrate that
hierarchical ordering significantly boosts performance, particularly when combined with diverse
question sets. Configurations with hierarchical ordering consistently outperformed those without,
highlighting its importance for structuring instruction-tuning data. These findings provide a solid
foundation for extending our experiments to larger context lengths and exploring the interaction
of hierarchical and diverse question compositions. Building on these results, we expanded our
experimentation to a 180K context length combining two documents, aiming to determine whether
the patterns observed at 100K scale effectively with rope scaling. We also explore which question
types (hierarchical or diverse and complex) perform best for questions directly following documents
or referencing previous ones.

For each experiment, we generated 300–600 training samples of 180K tokens (concatenating two
documents) using Qwen-2-72B and fine-tuned the data on LLaMA-3.1-8B-Instruct with a learning
rate of 6e-5 for 1 epoch. As the 180K context length exceeds LLaMA-3.1-8B-Instruct’s native 128K
window, we applied rope scaling. The following compositions were tested: a) Random vs. fixed
number of questions: Follow-up questions were either randomized (2–10) or fixed (6 main and
4 follow-up) to maintain consistency. b) Hierarchical vs. diverse and complex questions: We
tested hierarchical ordering questions (h) against questions targeting specific, diverse, and complex
reasoning (s). Each experiment is labeled as x-y-z, where x refers to questions following the first
document, y the second, and z to questions referencing the first document after the second is pro-
cessed. For instance, h-h-s-fixed includes 6 hierarchical questions for each document and 4 diverse
follow-ups referencing the first document after the second. c) Summarization: Some experiments
excluded global summarization at the start to assess its impact on model comprehension.

Table 7 shows the ablation results on InfiniteBench. Notably: 1) All experiments outperformed
the baseline LLaMA-3.1-8B-Instruct model by a significant margin, demonstrating the effective-
ness of our strategy with rope scaling. 2) Fixed questions outperform randomized ones: hs-hs-hs-
fixed scored 59.45, surpassing hs-hs-hs-randomized (58.51). 3) Hierarchical questions paired with
diverse questions achieve the best performance: hs-hs-hs-fixed yielded the highest score (59.45),

9

Under review as a conference paper at ICLR 2025

Table 7: Ablation study on InfiniteBench with 180K context length. Each experiment is labeled as
x-y-z, where x is the type of question after the first document, y is the type of question after the
second document, and z is the type of question referencing after the second document is processed.
For example, h-h-s-fixed is the dataset with 6 hierarchical questions following the first document, 6
hierarchical questions following the second document, and 4 follow-up diverse questions referencing
the first document after the second document is processed. Randomized signifies that the number of
questions sampled is randomized, and no-sum signifies that the global summary is removed.

Task

Retrieve.PassKey
Retrieve.Number
Retrieve.KV
En.Sum
En.QA
En.MC
En.Dia
Math.Find

Average

Task

Retrieve.PassKey
Retrieve.Number
Retrieve.KV
En.Sum
En.QA
En.MC
En.Dia
Math.Find

Average

LLaMA-3.1-
8B-Instruct

hs-hs-hs-
randomized

hs-hs-hs-
fixed

h-h-s-
randomized

100.00
95.33
42.66
27.63
24.83
68.00
16.66
35.33

51.31

h-h-s-fixed-
no-sum

100.00
99.33
84.00
24.11
32.81
70.66
16.66
36.66

58.03

100.00
100.00
82.66
23.42
33.32
71.33
18.00
39.33

58.51

h-h-s-
fixed

100.00
99.33
83.33
24.74
33.88
73.33
14.66
39.33

58.58

100.00
99.33
88.66
24.01
34.26
74.00
18.00
37.33

59.45

100.00
100.00
84.66
24.33
31.84
73.33
14.00
36.66

58.10

h-h-
randomized

h-h-h-
randomized

100.00
98.66
76.66
24.33
30.69
72.00
15.33
35.33

56.63

100.00
99.33
84.66
23.86
31.97
72.00
18.00
35.33

58.14

highlighting the benefits of structuring and diverse, complex questions. 4) Summarization improves
performance: hs-hs-fixed-no-sum scored 58.03, slightly below hs-hs-hs-fixed (58.58). Based on
these findings, for longer context lengths (Section 4.2, we retain summarization, fix the number
of questions/answers, and ensure both hierarchical and diverse questions are generated after direct
documents and for those referencing previous ones.

5 CONCLUSION

This paper presents a novel strategy to generate high-quality,
long-context instruction-tuning
datasets that exceed the typical raw data context length. It incorporates hierarchical ordering to en-
sure logical coherence while maintaining diversity and complexity in questions. Systematic ablation
studies show that combining diverse questions with hierarchical ordering enhances performance,
particularly in long-context scenarios. Our 1M model demonstrates strong capabilities, outper-
forming LLaMA-3.1-8B-Instruct on InfiniteBench and significantly surpassing it on RULER, while
maintaining robust performance on shorter-context tasks, as shown by LongBench and MMLU. Our
data curation strategy is highly scalable, enabling efficient creation of instruction-tuning datasets ex-
ceeding 1 million tokens and scaling up to 10 million or more. With sufficient resources and a strong
training stack, our method supports increasingly longer context lengths, potentially unlimited.

While our approach has significantly improved instruction tuning for long-context scenarios, a
promising direction for future work is developing a self-evolutionary strategy that diversifies and
adapts prompts. A short-context model could autonomously generate long-context instruction data
using our methodology and evolve independently, creating diverse and adaptable prompts for various
scenarios. This could enable models to progressively evolve into longer-context models. Addition-
ally, combining our data-centric approach with architectural optimizations offers another promising
avenue for future research.

10

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

Ethics Statement. In conducting this research, we ensured adherence to the highest ethical stan-
dards in the development and testing of our models. No human subjects were involved in data col-
lection, ensuring that there are no privacy concerns or risks associated with the handling of personal
information.

Reproducibility. We included the code to generate a bunch of hierarchical questions and di-
verse questions for a single document (see Section 3.1) in supplementary material (see generating-
data.py). We also included the code to concatenate multiple documents (see Section 3.2) in supple-
mentary material (see concatenate-350K.py). To enable long context training, we described detailed
hardware setup in Section 4.1. Details about evaluations are also mentioned in in Section 4.1.

REFERENCES

Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos,
Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark,
Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark
Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang,
Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Brad-
bury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christo-
pher A. Choquette-Choo, Aakanksha Chowdhery, Cl´ement Crepy, Shachi Dave, Mostafa De-
hghani, Sunipa Dev, Jacob Devlin, Mark D´ıaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxi-
aoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez,
Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui,
Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Ke-
nealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li,
Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Freder-
ick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Mous-
salem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Mar-
tin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley,
Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone,
Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan,
Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai
Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng,
Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report,
2023. URL https://arxiv.org/abs/2305.10403.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao
Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, mul-
titask benchmark for long context understanding, 2024. URL https://arxiv.org/abs/
2308.14508.

Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer,

2020. URL https://arxiv.org/abs/2004.05150.

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window
of large language models via positional interpolation, 2023. URL https://arxiv.org/
abs/2306.15595.

Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora:
Efficient fine-tuning of long-context large language models, 2024. URL https://arxiv.
org/abs/2309.12307.

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024.

URL https://arxiv.org/abs/2312.00752.

Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and
Yinfei Yang. Longt5: Efficient text-to-text transformer for long sequences, 2022. URL https:
//arxiv.org/abs/2112.07916.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja-
cob Steinhardt. Measuring massive multitask language understanding, 2021. URL https:
//arxiv.org/abs/2009.03300.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Namgyu Ho, Sangmin Bae, Taehyeon Kim, Hyunjik Jo, Yireun Kim, Tal Schuster, Adam Fisch,
James Thorne, and Se-Young Yun. Block transformer: Global-to-local language modeling for
fast inference, 2024. URL https://arxiv.org/abs/2406.02657.

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang
Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language
models?, 2024. URL https://arxiv.org/abs/2404.06654.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap-
lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier,
L´elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril,
Thomas Wang, Timoth´ee Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
//arxiv.org/abs/2310.06825.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc¸ois Fleuret. Transformers are
rnns: Fast autoregressive transformers with linear attention, 2020. URL https://arxiv.
org/abs/2006.16236.

Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language
models understand long contexts?, 2024. URL https://arxiv.org/abs/2311.04939.

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni,
and Percy Liang. Lost in the middle: How language models use long contexts, 2023. URL
https://arxiv.org/abs/2307.03172.

Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and Weipeng
Chen. Base of rope bounds context length, 2024. URL https://arxiv.org/abs/2405.
14591.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual

associations in gpt, 2023. URL https://arxiv.org/abs/2202.05262.

Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient
infinite context transformers with infini-attention, 2024. URL https://arxiv.org/abs/
2404.07143.

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window
extension of large language models, 2023. URL https://arxiv.org/abs/2309.00071.

Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases
enables input length extrapolation, 2022. URL https://arxiv.org/abs/2108.12409.

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En-
hanced transformer with rotary position embedding, 2023. URL https://arxiv.org/abs/
2104.09864.

Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaud-
hary, Xia Song, and Furu Wei. A length-extrapolatable transformer, 2022. URL https:
//arxiv.org/abs/2212.10554.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee
Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar-
mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation
language models, 2023. URL https://arxiv.org/abs/2302.13971.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and
Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions,
2023. URL https://arxiv.org/abs/2212.10560.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo-
gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol
Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models,
2022. URL https://arxiv.org/abs/2206.07682.

12

Under review as a conference paper at ICLR 2025

Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prab-
hanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model
for finance, 2023. URL https://arxiv.org/abs/2303.17564.

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin,
Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar
Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis,
Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models, 2023. URL
https://arxiv.org/abs/2309.16039.

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin
Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023. URL
https://arxiv.org/abs/2304.12244.

Weihao Zeng, Can Xu, Yingxiu Zhao, Jian-Guang Lou, and Weizhu Chen. Automatic instruction
evolving for large language models, 2024. URL https://arxiv.org/abs/2406.00770.

Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han,
Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. ∞bench: Extending long context
evaluation beyond 100k tokens, 2024. URL https://arxiv.org/abs/2402.13718.

Liang Zhao, Tianwen Wei, Liang Zeng, Cheng Cheng, Liu Yang, Peng Cheng, Lijie Wang, Chenxia
Li, Xuejie Wu, Bo Zhu, Yimeng Gan, Rui Hu, Shuicheng Yan, Han Fang, and Yahui Zhou.
Longskywork: A training recipe for efficiently extending context length in large language models,
2024. URL https://arxiv.org/abs/2406.00605.

13

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Under review as a conference paper at ICLR 2025

A APPENDIX: ADDITIONAL DETAILS ON DATA GENERATION ALGORITHMS

In this section, we present the pseudocode for the hierarchical QA generation strategy described in
Section 3.1, along with the algorithm for combining multiple documents, as outlined in Section 3.2.

Algorithm 1 Hierarchical Question Generation Strategy (Single Document)

if first iteration then

return first small chunk of current medium

end for
return conversations

return random medium chunk
else if no small chunk selected then

chunks ← HierarchicalSplit(document.text)
summaries, full summary ← SummarizeHierarchical(chunks)
conversations ← [InitialSummary(document.text, full summary)]
for i = 1 to N Questions To Generate do

context, summary ← SelectContext(chunks, summaries, last medium, last small, i)
qa pair ← GenerateQAPair(context, summary)
AppendToConversations(conversations, qa pair)
UpdateLastChunks(last medium, last small)

1: procedure GENERATEEXTENDEDCONTEXT(document, N Questions To Generate)
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12: end procedure
13: procedure SELECTCONTEXT(chunks, summaries, last medium, last small, iteration index)
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
end if
27:
28: end procedure
29: procedure GENERATEQAPAIR(context, summary)
30:
31:
32:
33:
end if
34:
35: end procedure

random choice ← RandomChoice([0, 1, 2])
if random choice = 0 then

return GenerateGeneralQAPair(context, summary)

return deeper content of current small chunk

return next small chunk in current medium

return GenerateSpecificQAPair(context)

if ContextIsSpecific(context) then

else if random choice = 1 then

return next medium chunk

end if

else

else

else

▷ Equal 1/3 probability for each

B ADDITIONAL INFORMATION ON DATA GENERATION PROMPTS

Here we list all prompts used in the different stages of our synthetic data generation pipeline.

Document Summarization

"""Summarize the following text concisely in no

more than {word_limit} words:

{chunk}"""

14

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Under review as a conference paper at ICLR 2025

Algorithm 2 Concatenating Multiple Documents

Input: Set of K documents, each with hierarchical and diverse questions
Initialize: conversation list C ← ∅
for each document Di where i = 1, 2, . . . , K do
Hi ← GenerateHierarchicalQuestions(Di)
Si ← RandomlySampleSpecificQuestions(Di)
C ← C ∪ InitialHierarchicalQuestions(Hi)
C ← C ∪ RandomlySampleDiverseQuestions(Si)
Store remaining unselected diverse questions from Si

end for
for each document Di where i = 2, 3, . . . , K do
C ← C ∪ NextHierarchicalQuestions(Hi−1)
C ← C ∪ RandomlySampleUnselectedDiverse(Si−1)
Update hierarchical index for document Di

end for
for each document Di where i = 1, 2, . . . , K − 1 do

if RandomCondition(0.6) then

C ← C ∪ FollowUpHierarchicalQuestions(Hi)

end if

end for
Process remaining specific and diverse questions:
x ← Length(Si)
if x ≥ ThresholdForSpecificQuestions then

2

Select and append follow-up specific questions to C
Remove selected follow-up specific questions from pool

end if
Output: Final conversation list C

Diverse Questions

"""Context information is below.
---------------------
${context}
---------------------
Given the context information and not prior knowledge.
Generate content based on the below query.
You are a Teacher/Professor. Your task is to
set up 1 diverse temporal question about the
context for an upcoming quiz/examination. The question
should cover different time periods and events
described in the context. Restrict the question
to the context information provided. You must
return the result in JSON: {’question’: <question>,
’answer’: <answer>}"""

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

Diverse Questions

"""Context information is below.
---------------------
${context}
---------------------
Given the context information and not prior knowledge.
Generate content based on the below query.
You are a Teacher/Professor. Your task is to
create 1 character-based question from the context
for an upcoming quiz/examination. The question should
explore different aspects of the characters, such
as their motivations, actions, and relationships. Restrict
the question to the context information provided.
You must return the result in JSON:
{’question’: <question>, ’answer’: <answer>}"""

"""Context information is below.
---------------------
${context}
---------------------
Given the context information and not prior knowledge.
Generate content based on the below query.
Formulate 1 complex question that requires analysis
of multiple aspects from the context for
an upcoming quiz/examination. The question should encourage
critical thinking and synthesis of different pieces
of information within the context. Restrict the
question to the context information provided. You
must return the result in JSON: {’question’:
<question>, ’answer’: <answer>}"""

"""Context information is below.
---------------------
${context}
---------------------
Given the context information and not prior knowledge.
Generate content based on the below query.
You are a Teacher/Professor. Ask 1 question
about the main themes or messages of
the text for an upcoming quiz/examination. The
question should cover different aspects of the
themes and how they are developed in
the context. Restrict the question to the
context information provided. You must return the
result in JSON: {’question’: <question>,
’answer’: <answer>}"""

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

Diverse Questions

"""Context information is below.
---------------------
${context}
---------------------
Given the context information and not prior knowledge.
Generate content based on the below query.
You are a Teacher/Professor. Create 1 question
that compare different elements within the context
for an upcoming quiz/examination. The question should
highlight similarities and differences between various
elements such as characters, events, and themes. Restrict
the question to the context information provided.
You must return the result in JSON:
{’question’: <question>, ’answer’: <answer>}"""

"""Context information is below.
---------------------
${context}
---------------------
Given the context information and not prior knowledge.
Generate content based on the below query.
You are a Teacher/Professor. Develop 1 question
that explore the cause and effect relationships
within the context for an upcoming quiz/examination.
The question should focus on understanding the
reasons behind events and their outcomes. Restrict
the question to the context information provided.
You must return the result in JSON:
{’question’: <question>, ’answer’: <answer>}"""

"""Context information is below.
---------------------
${context}
---------------------
Given the context information and not prior knowledge.
Generate content based on the below query.
You are a Teacher/Professor. Create 1 hypothetical
question based on the context for an
upcoming quiz/examination. The question should explore
what-if scenarios and possible alternate outcomes. Restrict
the question to the context information provided. You
must return the result in JSON: {’question’:
<question>, ’answer’: <answer>}"""

17

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Under review as a conference paper at ICLR 2025

Diverse Questions

"""Context information is below.

---------------------
${context}
---------------------
Given the context information and not prior knowledge.
Generate content based on the below query.
You are a Teacher/Professor. Formulate 1 question
that require interpretation of the context for
an upcoming quiz/examination. The question should encourage
students to provide their own insights and
interpretations based on the information given. Restrict
the question to the context information provided.
You must return the result in JSON:
{’question’: <question>, ’answer’: <answer>}"""

"""Context information is below.
---------------------
${context}
---------------------
Given the context information and not prior knowledge.
Generate content based on the below query.
You are a Teacher/Professor. Ask 1 detail-oriented
question about the context for an upcoming
quiz/examination. These question should focus on specific
details, facts, and figures mentioned in the
context. Restrict the question to the context
information provided. You must return the result
in JSON: {’question’: <question>, ’answer’: <answer>}"""

"""Context information is below.
---------------------
${context}
---------------------
Given the context information and not prior knowledge.
Generate content based on the below query.
You are a Teacher/Professor. Create 1 question
that explore different perspectives or viewpoints within
the context for an upcoming quiz/examination. The
question should examine how different characters or
groups might view events or themes differently.
Restrict the questions to the context information
provided. You must return the result in
JSON: {’question’: <question>, ’answer’: <answer>}"""

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Under review as a conference paper at ICLR 2025

Multi-Hop Questions

"""Context information is below.
${selected_chunk_1}
${selected_chunk_2}
${selected_chunk_3}
You are a Professor designing a final exam
for an advanced interdisciplinary course. Create 1
complex question requiring deep analysis and synthesis of
information from all three chunks. Do not mention
that there are three chunks/your questions. Do not
mention excerpts either. For example, instead of a
question that says "Analyze the theme of justice
and its various forms as portrayed in the
three provided literary excerpts. How do the characters’
actions and the outcomes of their situations reflect
or challenge traditional notions of justice? Consider the
legal, personal, and societal implications of justice in
each excerpt and discuss the role of power
dynamics in shaping justice." You should say: "Analyze
the theme of justice and its various forms
as portrayed. How do the characters’ actions and
the outcomes of their situations reflect or challenge
traditional notions of justice? Consider the legal, personal,
and societal implications of justice and discuss the
role of power dynamics in shaping justice."
Question Guidelines:
1. The question must integrate and require reasoning
across all three chunks.
2. The question should be multi-layered, promoting analysis,
synthesis, and evaluation.
Answer Guidelines:
1. Provide a comprehensive answer addressing all question
aspects.
2. Reference and interconnect information from each chunk.
Return 1 question-answer pair in JSON format:
{ "question": <question>, "answer": <answer> }
"""

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

19

Under review as a conference paper at ICLR 2025

1026
1027
1028
1029
1030

1031
1032
1033
1034
1035
1036

1037
1038
1039
1040
1041
1042

1043
1044
1045
1046
1047
1048

1049
1050
1051
1052
1053

1054
1055
1056
1057
1058
1059

1060
1061
1062
1063
1064
1065

1066
1067
1068
1069
1070
1071

1072
1073
1074
1075
1076

1077
1078
1079

Specific Detail Question

"""Context information is below.
${context}
Given the context information and not prior knowledge,
generate content based on the below query. You
are a Teacher/Professor. Create 1 specific question about
the details, events, characters, and settings from the
context provided. This question should have an exact,
unambiguous answer that can be directly found in
the given information. The question should be similar
in style to the following examples:
"Where does Character A meet Character B for
the first time?"
"What is Character C’s religion?"
"Where does Character D live for the majority
of the story?"
"Which of the following is NOT one of
Character E’s responsibilities?"
"Which among [list of names] is not [Character
F]’s child?"
"Who among [list of names] is the final
to perish?"
"What’s the name of [Family name]’s summer home?"
"Who accompanied [Character G] [specific activity] at last?"
Ensure that the question and answer are strictly
based on the context information provided. The question
may include multiple-choice options when appropriate. You
must return the result in JSON: {’question’: <question>,
’answer’: <answer>}"""

C COMPLETE CONVERSATION DATA

Here we present an example of our instruction tuning data that is developed by concatenating four
long documents
documents (350K context length). The data consists of three types of content:
and their summarizations (yellow) (constituting QA pairs), QA pairs extracted from each document
(blue), and QA pairs that reference past documents (green). These are color-coded in the following
example for clarity.

{"conversations": [

"role": "user",
"content": "Produced by Charles Bowen from page scans provided by the

Web Archive\n\n\n\n\nTranscriber’s Notes: ... Please give me a

summary of the book"

"role": "assistant",
"content": "Charles Bowen created a digital book from Web Archive

scans, ..."

"role": "user",
"content": "What are the main reasons for the tensions and conflicts

among the characters in ’The Patricians’?"

{

},
{

},

{

},
{

"role": "assistant",

20

Under review as a conference paper at ICLR 2025

1080
1081
1082
1083
1084

1085
1086
1087
1088
1089
1090

1091
1092
1093
1094
1095
1096

1097
1098
1099
1100
1101
1102

1103
1104
1105
1106
1107

1108
1109
1110
1111
1112
1113

1114
1115
1116
1117
1118
1119

1120
1121
1122
1123
1124
1125

1126
1127
1128
1129
1130

1131
1132
1133

"content": "The tensions and conflicts in ’The Patricians’ arise from

issues of pride, ..."

"role": "user",
"content": "What class in ancient Rome did the term ’Patricians’

typically refer to?"

"role": "assistant",
"content": "The term ’Patricians’ typically referred to the

aristocratic class in ancient Rome, who were affluent, powerful,
and held considerable political sway in the Roman Republic."

"role": "user",
"content": "Who among the following did not attend Christopher’s

banquet? A) Tausdorf B) Althea C) Schindel D) Francis"

"role": "assistant",
"content": "D) Francis"

"role": "user",
"content": "Analyze the theme of justice and its various forms as
portrayed. How do the characters’ actions and the outcomes of
their situations reflect or challenge traditional notions of
justice? Consider the legal, personal, and societal implications
of justice and discuss the role of power dynamics in shaping
justice."

},
{

},
{

},
{

},
{

},
{

},
{

"role": "assistant",
"content": "The theme of justice is explored through the complex
interplay of legal, personal, and societal implications,
reflecting and challenging traditional notions of justice. ..."

},
...

{

},
{

},

{

},
{

},
{

"role": "user",
"content": "Produced by D Alexander and the Online Distributed

Proofreading Team at http://www.pgdp.net ... + Please give me a
summary of the book"

"role": "assistant",
"content": "The text acknowledges D Alexander ..."

"role": "user",
"content": "Who does the professor and his party meet at the sulphur

spring?"

"role": "assistant",
"content": "Jranvin, the chief of the islanders"

"role": "user",

21

Under review as a conference paper at ICLR 2025

1134
1135
1136
1137
1138

1139
1140
1141
1142
1143
1144

1145
1146
1147
1148
1149
1150

1151
1152
1153
1154
1155
1156

1157
1158
1159
1160
1161

1162
1163
1164
1165
1166
1167

1168
1169
1170
1171
1172
1173

1174
1175
1176
1177
1178
1179

1180
1181
1182
1183
1184

1185
1186
1187

"content": "What does Jim recognize that lifts his spirits while

confined
on the Marjorie?"

"role": "assistant",
"content": "Jim recognizes his own ship, which lifts his spirits

while he is

confined on the Marjorie."

"role": "user",
"content": "How did the professor and his party find out about the

location

of the Marjorie and what did they discover upon investigating the

harbor?"

},
{

},
{

},
{

"role": "assistant",
"content": "The professor and his party, including Tom, Jo, and

Berwick,

hiked to a cliff and climbed to the top where they spotted the

Marjorie

and another ship in a harbor five miles away. Upon investigating the

harbor,

they found the Sea Eagle leaving and chaos on the Marjorie. They also

captured

the steward who revealed he helped Jim escape."

},
...

{

"role": "user",
"content": "Analyze the theme of justice and its various forms as

portrayed.

How do the characters’ actions and the outcomes of their situations

reflect

or challenge traditional notions of justice? Consider the legal,

personal,

and societal implications of justice and discuss the role of power

dynamics

in shaping justice."

},
{

"role": "assistant",
"content": "The theme of justice is intricately woven through
the narrative, ..."

},
...

{

"role": "user",
"content": "\n\n\n\nProduced by Marius Masi, Juliet Sutherland and

the Online

\nDistributed Proofreading Team at ...

Please give me a summary of

the book"

"role": "assistant",
"content": "The text describes a collection of illustrated books

about Japan, ..."

},
{

},

22

Under review as a conference paper at ICLR 2025

1188
1189
1190
1191
1192

1193
1194
1195
1196
1197
1198

1199
1200
1201
1202
1203
1204

1205
1206
1207
1208
1209
1210

1211
1212
1213
1214
1215

1216
1217
1218
1219
1220
1221

1222
1223
1224
1225
1226
1227

1228
1229
1230
1231
1232
1233

1234
1235
1236
1237
1238

1239
1240
1241

{

},
{

},
{

},
{

"role": "user",
"content": "How does the text compare Japanese and Western art in

terms

of their integration into daily life and appreciation by the public?"

"role": "assistant",
"content": "The text emphasizes that Japanese art, unlike Western art

,

is deeply integrated into daily life and universally appreciated,
similar to the vitality of art in ancient Greece..."

"role": "user",
"content": "How does the book contrast Japanese and Western art in

terms of

their integration into daily life and societal appreciation?"

"role": "assistant",
"content": "The book highlights that Japanese art is more integrated

into daily

life and universally appreciated, ..."

},
...

{

"role": "user",
"content": "Analyze the character development of Tausdorf throughout

the

passage. How does his perception of Bona change, and what factors

contribute

to this change? Additionally, discuss the role of the setting and the

use of

imagery in shaping Tausdorf’s actions and emotions."

"role": "assistant",
"content": "Tausdorf’s character development is complex and

multifaceted, ..."

"role": "user",
"content": "What is the name of the yacht that Jim and Berwick

consider hiring?"

},
{

},
{

},
{

"role": "assistant",
"content": "Storm King"

},
...

{"role": "user", "content": "\n\n\n\nProduced by David Edwards and the
Online Distributed ... Please give me a summary of the book"},
{"role": "assistant", "content": "\"The Curlytops Snowed In\" by Howard
R. Garis follows siblings Ted and Janet Martin, known as the

Curlytops, ..."},

{"role": "user", "content": "What challenge did Nicknack, the goat,

face during the winter adventures, and how did the children address

it?"},

23

Under review as a conference paper at ICLR 2025

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

1M model
Context Length Accuracy (%)

650K model
Context Length Accuracy (%)

350K model
Context Length Accuracy (%)

1 000 000 62.95%
524 288 76.88%
262 144 76.24 %
131 072 82.05 %
65 536 83.73 %
32 768 86.19 %
16 384 90.06 %
8 192 92.96 %
4 096 95.59 %

650 000 57.24 %
524 288 74.88 %
262 144 78.26 %
131 072 82.29 %
65 536 81.81 %
32 768 87.96 %
16 384 92.86 %
8 192 91.92 %
4 096 95.49 %

350 000 71.15 %
262 144 78.26 %
131 072 82.72 %
65 536 82.63 %
32 768 87.26 %
16 384 91.45 %
8 192 92.62 %
4 096 96.13 %

Table 8: Context length accuracy of RULER for different models

{"role": "assistant", "content": "Nicknack had difficulty standing on

ice due to the unsuitability of his hooves..."},

{"role": "user", "content": "What causes the snow house to collapse in

Chapter VI, and what do the children decide to do after the
incident?"},

{"role": "assistant", "content": "The snow house collapses when

Nicknack the goat ..."},

...

{"role": "user", "content": "Compare and contrast the portrayal of art
and its integration into daily life in Japan and the West, as
described in the text. How does the Japanese approach to art,
particularly in the context of theater, reflect broader cultural
values and philosophies? Discuss the role of realism, balance, and
the environment in shaping the artistic experience in both cultures
."},

{"role": "assistant", "content": "The text illustrates a stark contrast

between the Japanese and Western approaches to art, particularly
in theater, which reflects deeper cultural values and philosophies.
In Japan, art is not merely a conscious effort to achieve beauty
but an instinctive expression of inherited taste that permeates all

aspects of life..."},

{"role": "user", "content": "Analyze the theme of survival and its

various forms as portrayed in the narratives. How do the characters
’ actions and the outcomes of their situations reflect or challenge
traditional notions of survival? Consider the natural, social, and
personal implications of survival and discuss the role of

resourcefulness in shaping survival strategies."},

{"role": "assistant", "content": "The theme of survival is intricately
woven through the narratives, reflecting the characters’ resilience

and adaptability in the ..."},

...

]}

D RULER NUMERICAL RESULTS

E 100K CONTEXT LENGTH ABLATION STUDIES

The 100K ablation studies aim to assess whether hierarchical ordering and diverse question types
improve results on single-document instruction tuning data. We also aim to identify which of these
factors most significantly influences overall performance.
In particular, we want to explore (1)
whether hierarchical ordering enhances outcomes, (2) whether diverse question sets contribute pos-
itively, and (3) whether the use of multi-hop questions further boosts results.

24

Under review as a conference paper at ICLR 2025

Table 9: Context length of RULER for LLaMA-3.1-8B-Instruct models

LLaMA-3.1-8B-Instruct
Context Length Percentage (%)

Zero-shot Rope Scaling to 1M
Context Length Percentage (%)

524 288 28.93 %
262 144 30.34 %
131 072 83.06 %
65 536 82.26 %
32 768 88.44 %
16 384 93.13 %
8 192 92.08 %
4 096 95.49 %

1 000 000 48.81 %
524 288 62.53 %
262 144 66.44 %
131 072 72.68 %
65 536 77.81 %
32 768 84.01 %
16 384 87.36 %
8 192 90.73 %
4 096 95.94 %

(a) Context length of RULER
of LLaMA-3.1-8B-Instruct

(b) Context length of RULER with
zero-shot rope scaling to 1M context length

Each experiment uses 300-600 data samples, each with 100K tokens, fine-tuned on LLaMA-3.1-
8B-Instruct for 1 epoch at a 6e-5 learning rate. The specific ablation tests we included are 1) 4
hierarchies: from a single document, we generated hierarchical ordering data using the algorithm
specified in Section 3.1. 2) 4 hierarchies with multi-hop reasoning: In addition to the 4 hierachies
set up in Section 3.1, every time we generate a new QA pair, there is a 20 % chance that a multi-hop
question-answer pair will follow. 3) 4 hierarchies without order: hierarchical questions were gen-
erated without enforcing the order from Section 3.1, testing if strict hierarchy enforcement improves
outcomes. 4) Diverse questions: this setup generated various question types to test if diversity
improves performance, as outlined in Section 3.1.

The results of these ablation studies on InfiniteBench are summarized in Table 11. The key find-
ings include: 1) Multi-Hop Reasoning Improves Performance: Among all configurations, multi-hop
reasoning achieved the highest average score of 54.70, demonstrating the importance of captur-
ing cross-document relationships and broader reasoning capabilities. 2) Diverse Questions Pro-
vide Broad Improvements: The diverse questions setup achieved the second-highest score of 52.41,
highlighting the value of introducing variety in QA generation for instruction-tuning data. 3) Hier-
archical Ordering Boosts Performance: Both the strict hierarchical model (52.08) and the random
hierarchical model (50.69) outperformed the base LLaMA-3.1-8B-Instruct (51.31), validating the
effectiveness of hierarchical structuring, even when not strictly ordered.

The LongBench results (presented in Table 10) provide additional insights, though the differences
between configurations are relatively minor. This is likely because LongBench evaluates models on
short contexts (up to 10K tokens), which do not fully leverage the strengths of hierarchical or multi-
hop structures designed for longer contexts. In summary, the ablation tests show that hierarchical
ordering, multi-hop reasoning, and diverse questions are key to optimizing performance on long-
context tasks.

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

25

Under review as a conference paper at ICLR 2025

Table 10: Ablation study on LongBench with 100K context length.

Task

NarrativeQA
Qasper
MultiFieldQA-en
MultiFieldQA-zh

Single Document

HotpotQA
2WikiMQA
Musique
DuReader

Multi-Document

GovReport
QMSum
MultiNews
VCSUM

Summarization

TREC
TriviaQA
SAMSum
LSHT

Few-shot Learning

Passage Count
PassageRetrieval-e
PassageRetrieval-z

Synthetic Tasks

All

LLaMA-3.1-8B-
Instruct

4
hierarchies

4 hierarchies
multi-hop

4 hierarchies
random

diverse
questions

25.48
45.33
54.98
61.83

46.91

55.00
44.95
31.76
34.10

41.45

35.07
25.13
27.08
17.10

26.10

72.50
91.65
43.77
46.00

63.48

6.55
99.50
96.38

67.48

48.11

25.89
47.02
54.86
55.75

45.88

56.67
52.19
29.15
36.83

43.71

34.39
25.15
27.34
16.12

25.75

73.00
92.28
43.81
46.00

63.77

4.00
99.00
98.50

67.17

48.31

25.10
44.79
53.96
54.87

44.68

56.91
52.96
28.55
36.32

43.69

33.72
25.27
27.48
16.75

25.81

73.00
92.25
43.98
47.00

64.06

3.00
99.00
100.00

67.33

48.15

25.04
46.00
54.86
59.89

46.45

55.83
48.74
29.85
35.57

42.50

35.31
25.52
27.29
16.13

26.06

73.00
91.87
44.49
47.00

64.09

7.56
98.50
94.63

66.90

48.27

27.91
46.25
53.75
56.14

46.01

58.34
52.71
28.10
36.74

43.97

35.33
25.38
27.46
16.40

26.14

72.00
91.83
45.48
48.00

64.33

5.00
98.50
99.50

67.67

48.67

Table 11: Ablation study on InfiniteBench with 100K context length.

LLaMA-3.1-8B-
Instruct

4 hierarchies

diverse
questions

4 hierarchies
random

4 hierarchies
multi-hop

Retrieve.PassKey
Retrieve.Number
Retrieve.KV
En.Sum
En.QA
En.MC
En.Dia
Math.Find

Average

100.00
95.33
42.66
27.63
24.83
68.00
16.66
35.33

51.31

86.66
86.00
58.00
24.11
32.50
72.00
23.33
36.66

52.41

86.66
85.33
58.66
22.77
25.40
70.00
20.66
36.00

50.69

100.00
96.66
57.33
22.67
30.25
70.66
26.00
34.00

54.70

86.66
86.66
60.00
23.02
29.66
70.66
24.66
35.33

52.08

26

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

