# Principled Instructions Are All You Need for Questioning LLaMA-1/2,
GPT-3.5/4

Sondos Mahmoud Bsharat∗, Aidar Myrzakhan∗, Zhiqiang Shen∗
∗joint first author & equal contribution
VILA Lab, Mohamed bin Zayed University of AI

( )

###### Abstract

This paper introduces 26 guiding principles designed to streamline the process
of querying and prompting large language models. Our goal is to simplify the
underlying concepts of formulating questions for various scales of large
language models, examining their abilities, and enhancing user comprehension
on the behaviors of different scales of large language models when feeding
into different prompts. Extensive experiments are conducted on LLaMA-1/2 (7B,
13B and 70B), GPT-3.5/4 to verify the effectiveness of the proposed principles
on instructions and prompts design. We hope that this work provides a better
guide for researchers working on the prompting of large language models.
Project page is available at https://github.com/VILA-Lab/ATLAS.

##  1 Introduction

  * “

Prompt engineering is the art of communicating with a generative large
language model.

”

ChatGPT,  _2023_

Large language models (LLMs) like ChatGPT [13] have shown impressive abilities
in various domains and tasks, such as answering questions [7], mathematical
reasoning [5], code generating [11, 9], etc. However, their application and
usage, especially on designing the optimal instructions or prompts, can
sometimes be unclear to the common users. In this work, we aim to reveal these
mysteries for developers or general users when inquiring and interacting with
LLMs, and further enhance the quality of the responses from the pretrained
LLMs by simply curating better prompts.

Figure 1: Illustration example of prompts and corresponding responses before
and after applying principles. Left is the original promotes and their
responses from GPT-4, right is the principled prompts and the associated
responses. Principles 5 and 6 are utilized.

Given that directly fine-tuning LLMs for particular tasks tends to be
impractical or unattainable for the majority of users and developers due to
inefficiency, the research community has turned its attention to the
optimization of prompts. The technique of prompt engineering, which entails
the crafting of precise, task-specific instructions in natural language,
either manually or through automated means, and the careful selection of
representative examples for inclusion in the prompt, has become a central area
of investigation for LLMs. Despite these dedicated efforts, the task of
reliably guiding LLMs to produce specific responses and making full use of the
capability of pretrained LLMs continues to pose a considerable challenge.

In this work, we present comprehensive principled instructions to improve the
quality of prompts for LLMs. Specifically, we investigate a wide range of
behaviors when feeding into different types and formulations of prompts, such
as integrating the intended audience in the prompt, e.g., add “the audience is
an expert in the field”, or “the audience is the 5-year-old child”, as well as
other multiple aspects of the characteristics of LLMs. Our findings indicate
that larger models possess a considerable capacity for simulation. The more
precise the task or directive provided, the more effectively the model
performs, aligning its responses more closely with our expectations. This
suggests that LLMs do not merely memorize training data but are capable of
adapting this information to suit varying prompts, even when the core
inquiries remain constant. Therefore, it proves beneficial to assign a
specific role to LLMs as a means to elicit outputs that better match our
intended results.

We elaborate the principled instructions for LLM prompting, provide further
motivation, and detail several specific designing principles in Section 3. In
Section 4 we show experimentally that the proposed principles can produce
higher quality, more concise, factual and less complicated or intricate
responses than standard prompts for LLMs. Specifically, with the manually-
designed ATLAS benchmark, which includes multiple questions for each
principle, the specialized prompts we introduced have enhanced both the
quality and accuracy of the LLM responses by an average of 57.7% and 67.3%,
respectively, when applied to GPT-4. Furthermore, the improvements are more
pronounced with the increase in model size, for example, the performance gains
when moving from LLaMA-2-7B to GPT-4 exceed 40%.

##  2 Related Work

Large Language Models. The evolution of large language models (LLMs) has been
pivotal in advancing natural language processing (NLP). This section reviews
key developments in LLMs, providing a foundation for the current study.
Beginning with Google’s BERT [3] revolutionized context understanding through
its bidirectional training approach, while T5 [17] further advanced the field
by unifying various NLP tasks into a single framework. Concurrently, GPT-1
[14] introduced a pioneering model leveraging transformer architectures for
unsupervised learning. This was followed by its successor, GPT-2 [15] which
significantly expanded its parameter count to 1.5 billion, demonstrating
remarkable capabilities in text generation. Then, GPT-3 [2] marked a
substantial leap in scale and capability, boasting 175 billion parameters and
showcasing proficiency across a wide range of language tasks.

Regarding other recently proposed LLMs, Gopher [16], not only advanced
language processing capabilities with its 280-billion parameter model but also
brought ethical considerations to the forefront. Meta’s LLaMA series [21, 22]
highlighted the importance of efficiency, suggesting powerful performance with
fewer resources, a concept also advocated by Chinchilla [4], which proposed
that smaller, optimally trained models could achieve exceptional results. The
latest in this series of innovations is Mistral [6] excels in efficiency and
performance, outperforming larger models. The most recent milestones in this
trajectory are OpenAI’s GPT-4 [13] and Google’s Gemini family [20]. They
represent another significant advancement in the field with their enhanced
understanding and generative capabilities, setting new benchmarks for the
application of LLMs in various domains.

Prompting. Prompting, as a distinct aspect of interacting with language models
and its simplicity with no need to fine-tune the model, has evolved into a
nuanced field of study, highlighting the intricate relationship between user
inputs and LLM responses. Early explorations, such as those by [19], delved
into how varying prompt designs could dramatically influence the performance
and outputs of language models, marking the birth of prompt engineering. This
area rapidly expanded, uncovering the critical role of prompts in few-shot and
zero-shot learning scenarios, exemplified by [2] work with GPT-3, where
strategically crafted prompts enabled the model to perform tasks with minimal
prior examples. Beyond mere task instruction, recent studies have shifted
towards understanding the semantic and contextual nuances in prompts,
examining how subtle changes can lead to significantly different responses
from the LLM.

Ask-Me-Anything [1] prompting introduced focusing on using multiple imperfect
prompts and aggregating them to improve model performance, particularly in
question-answering formats. Another one, Chain-of-Thought method [23], where
the model generates a series of intermediate reasoning steps to improve
performance on complex tasks. Also, least-to-most prompting [26] a novel
strategy to break down complex problems into simpler subproblems,
significantly enhancing the model’s capability to tackle more challenging
problems than those presented in the prompts. The effectiveness of explanation
was explored [8], finding that explanations can enhance LLM’s learning
capabilities on complex tasks. Furthermore, a catalog of prompt engineering
techniques was examined with ChatGPT [24], emphasizing the importance of
prompt engineering in enhancing LLM applications in software development and
education. It also highlighted that effective prompt design is crucial in
improving LLM performance, particularly in coding practices and learning
experiences. Lastly, Directional Stimulus Prompting [12] presents a novel
framework that uses a tunable policy model to generate auxiliary prompts,
guiding LLMs towards specific desired outcomes. This diversity in prompting
strategies underscores the rapidly evolving landscape of LLMs, offering
multiple directions to harness their capabilities more effectively.

##  3 Principles

###  3.1 Motivation

Since the quality of the responses generated by a pretrained and aligned LLM
is directly relevant to the quality of the prompts or instructions provided by
the users, it is essential to craft prompts that the LLM can comprehend and
respond to effectively. The prompts delivered to an LLM serve as a way to
program the interaction between a user and the LLM, enhancing its ability to
address a diverse range of tasks. The primary focus of this work is on the
methodology of crafting and customizing prompts to enhance output quality.
This necessitates a comprehensive grasp of the functioning and behaviors of
LLMs, their underlying mechanisms, and the principles governing their
responses. In this work, we achieve this goal through elaborating 26
principles for comprehensive prompts in different scenarios and circumstances,
examples are shown in Fig. 1.

###  3.2 Overview

The overview of principles is presented in Table 1. According to their unique
nature, we group them into five categories as in Table 2: (1) Prompt Structure
and Clarity, e.g., integrate the intended audience in the prompt such as the
audience is an expert in the field; (2) Specificity and Information, e.g., Add
to your prompt the following phrase “Ensure that your answer is unbiased and
does not rely on stereotypes.”; (3) User Interaction and Engagement, e.g.,
Allow the model to elicit precise details and requirements from you by asking
you questions until he has enough information to provide the needed output
“From now on, I would like you to ask me questions to…”. (4) Content and
Language Style, e.g., No need to be polite with LLM so there is no need to add
phrases like “please”, “if you don’t mind”, “thank you”, “I would like to”,
etc., and get straight to the point; (5) Complex Tasks and Coding Prompts,
e.g., Break down complex tasks into a sequence of simpler prompts in an
interactive conversation.

#Principle |  Prompt Principle for Instructions
---|---
1 |  No need to be polite with LLM so there is no need to add phrases like “please”, “if you don’t mind”, “thank you”, “I would like to”, etc., and get straight to the point.
2 |  Integrate the intended audience in the prompt, e.g., the audience is an expert in the field.
3 |  Break down complex tasks into a sequence of simpler prompts in an interactive conversation.
4 |  Employ affirmative directives such as ‘do,’ while steering clear of negative language like ‘don’t’.
5 |  When you need clarity or a deeper understanding of a topic, idea, or any piece of information, utilize the following prompts: o Explain [insert specific topic] in simple terms. o Explain to me like I’m 11 years old. o Explain to me as if I’m a beginner in [field]. o Write the [essay/text/paragraph] using simple English like you’re explaining something to a 5-year-old.
6 |  Add “I’m going to tip $xxx for a better solution!”
7 |  Implement example-driven prompting (Use few-shot prompting).
8 |  When formatting your prompt, start with ‘###Instruction###’, followed by either ‘###Example###’ or ‘###Question###’ if relevant. Subsequently, present your content. Use one or more line breaks to separate instructions, examples, questions, context, and input data.
9 |  Incorporate the following phrases: “Your task is” and “You MUST”.
10 |  Incorporate the following phrases: “You will be penalized”.
11 |  use the phrase ”Answer a question given in a natural, human-like manner” in your prompts.
12 |  Use leading words like writing “think step by step”.
13 |  Add to your prompt the following phrase “Ensure that your answer is unbiased and does not rely on stereotypes”.
14 |  Allow the model to elicit precise details and requirements from you by asking you questions until he has enough information to provide the needed output (for example, “From now on, I would like you to ask me questions to…”).
15 |  To inquire about a specific topic or idea or any information and you want to test your understanding, you can use the following phrase: “Teach me the [Any theorem/topic/rule name] and include a test at the end, but don’t give me the answers and then tell me if I got the answer right when I respond”.
16 |  Assign a role to the large language models.
17 |  Use Delimiters.
18 |  Repeat a specific word or phrase multiple times within a prompt.
19 |  Combine Chain-of-thought (CoT) with few-Shot prompts.
20 |  Use output primers, which involve concluding your prompt with the beginning of the desired output. Utilize output primers by ending your prompt with the start of the anticipated response.
21 |  To write an essay /text /paragraph /article or any type of text that should be detailed: “Write a detailed [essay/text /paragraph] for me on [topic] in detail by adding all the information necessary”.
22 |  To correct/change specific text without changing its style: “Try to revise every paragraph sent by users. You should only improve the user’s grammar and vocabulary and make sure it sounds natural. You should not change the writing style, such as making a formal paragraph casual”.
23 |  When you have a complex coding prompt that may be in different files: “From now and on whenever you generate code that spans more than one file, generate a [programming language ] script that can be run to automatically create the specified files or make changes to existing files to insert the generated code. [your question]”.
24 |  When you want to initiate or continue a text using specific words, phrases, or sentences, utilize the following prompt:
o I’m providing you with the beginning [song lyrics/story/paragraph/essay…]:
[Insert lyrics/words/sentence]’. Finish it based on the words provided. Keep
the flow consistent.
25 |  Clearly state the requirements that the model must follow in order to produce content, in the form of the keywords, regulations, hint, or instructions
26 |  To write any text, such as an essay or paragraph, that is intended to be similar to a provided sample, include the following instructions: o Please use the same language based on the provided paragraph[/title/text /essay/answer].
Table 1: Overview of 26 prompt principles. Category |  Principles | #Principle
---|---|---
Prompt Structure and Clarity |  Integrate the intended audience in the prompt. | 2
Employ affirmative directives such as ‘do’ while steering clear of negative language like ‘don’t’. | 4
Use Leading words like writing “think step by step.” | 12
Use output primers, which involve concluding your prompt with the beginning of the desired output. by ending your prompt with the start of the anticipated response. | 20
Use Delimiters. | 17
When formatting your prompt, start with ‘###Instruction###’, followed by either ‘###Example###’ or ‘###Question###’ if relevant. Subsequently, present your content. Use one or more line breaks to separate instructions, examples, questions, context, and input data. | 8
Specificity and Information |  Implement example-driven prompting (Use few-shot prompting). | 7
When you need clarity or a deeper understanding of a topic, idea, or any piece of information, utilize the following prompts: o Explain [insert specific topic] in simple terms. o Explain to me like I’m 11 years old o Explain to me as if I’m a beginner in [ field ] o “Write the [essay/text/paragraph] using simple English like you’re explaining something to a 5-year-old” | 5
Add to your prompt the following phrase “Ensure that your answer is unbiased and does not rely on stereotypes.” | 13
To write any text intended to be similar to a provided sample, include specific instructions: o “Please use the same language based on the provided paragraph.[/title/text /essay/answer]” | 26
When you want to initiate or continue a text using specific words, phrases, or sentences, utilize the provided prompt structure: o I’m providing you with the beginning [song lyrics/story/paragraph/essay…]: [Insert lyrics/words/sentence]. Finish it based on the words provided. Keep the flow consistent. | 24
Clearly state the model’s requirements that the model must follow in order to produce content, in form of the keywords, regulations, hint, or instructions. | 25
To inquire about a specific topic or idea and test your understanding g, you can use the following phrase [16]: o “Teach me the [Any theorem/topic/rule name] and include a test at the end, but don’t give me the answers and then tell me if I got the answer right when I respond” | 15
To write an essay/text/paragraph/article or any type of text that should be detailed: o “Write a detailed [essay/text/paragraph] for me on [topic] in detail by adding all the information necessary.” | 21
User Interaction and Engagement |  Allow the model to elicit precise details and requirements from you by asking you questions until he has enough information to provide the needed output o “From now on, I would like you to ask me questions to…”. | 14
To write an essay /text /paragraph /article or any type of text that should be detailed: “Write a detailed [essay/text/paragraph] for me on [topic] in detail by adding all the information necessary”. | 21
Content and Language Style |  To correct/change specific text without changing its style: “Try to revise every paragraph sent by users. You should only improve the user’s grammar and vocabulary and make sure it sounds natural. You should not change the writing style, such as making a formal paragraph casual.” | 22
Incorporate the following phrases: “Your task is” and “You MUST.” | 9
Incorporate the following phrases: “You will be penalized.” | 10
Assign a role to the language model. | 16
Use the phrase “Answer a question given in natural language form” in your prompts. | 11
No need to be polite with LLM so there is no need to add phrases like “please”, “if you don’t mind”, “thank you”, “I would like to”, etc., and get straight to the point. | 1
Repeat a specific word or phrase multiple times within a prompt. | 18
Add “I’m going to tip $xxx for a better solution!” | 6
Complex Tasks and Coding Prompts |  Break down complex tasks into a sequence of simpler prompts in an interactive conversation. | 3
When you have a complex coding prompt that may be in different files : o “From now and on whenever you generate code that spans more than one file, generate a [programming language ] script that can be run to automatically create the specified files or make changes to existing files to insert the generated code. [your question].” | 23
Combine Chain-of-thought (Cot) with few-shot prompts. | 19
Table 2: Prompt principle categories.

###  3.3 Design Principles

In this study, a number of guiding principles are established for formulating
prompts and instructions to elicit high-quality responses from pre-trained
large language models:

Conciseness and Clarity: Generally, overly verbose or ambiguous prompts can
confuse the model or lead to irrelevant responses. Thus, the prompt should be
concise, avoiding unnecessary information that does not contribute to the task
while being specific enough to guide the model. This is the basic principle
guidance for prompt engineering.

Contextual Relevance: The prompt must provide relevant context that helps the
model understand the background and domain of the task. Including keywords,
domain-specific terminology, or situational descriptions can anchor the
model’s responses in the correct context. We highlight this design philosophy
in our presented principles.

Task Alignment: The prompt should be closely aligned with the task at hand,
using language and structure that clearly indicate the nature of the task to
the model. This may involve phrasing the prompt as a question, a command, or a
fill-in-the-blank statement that fits the task’s expected input and output
format.

Example Demonstrations: For more complex tasks, including examples within the
prompt can demonstrate the desired format or type of response. This often
involves showing input-output pairs, especially in “few-shot” or “zero-shot”
learning scenarios.

Avoiding Bias: Prompts should be designed to minimize the activation of biases
inherent in the model due to its training data. Use neutral language and be
mindful of potential ethical implications, especially for sensitive topics.

Incremental Prompting: For tasks that require a sequence of steps, prompts can
be structured to guide the model through the process incrementally. Break down
the task into a series of prompts that build upon each other, guiding the
model step-by-step. Also, prompts should be adjustable based on the
performance of the model and iterative feedback, i.e., it needs to be well
prepared to refine the prompt based on initial outputs and model behaviors.
Moreover, prompts should be adjustable based on the performance and response
of the model, and iterative human feedback and preference.

Finally, more advanced prompts may incorporate programming-like logic to
achieve complex tasks. For instance, use of conditional statements, logical
operators, or even pseudo-code within the prompt to guide the model’s
reasoning process. The design of prompts is an evolving field, especially as
LLMs become more sophisticated. As researchers continue to explore the limits
of what can be achieved through prompt engineering, these principles will
likely be refined and expanded.

##  4 Experiments

###  4.1 Setup and Implementation Details

All our evaluation is performed on ATLAS [18], a manually crafted benchmark
for principled prompt evaluation. For each principle, it contains 20 human-
selected questions with and without the principled prompts. Following [10,
25], we evaluate the various scales of LLM outputs by human evaluation.

Figure 2: Boosting example of LLM response after using the principle 13 on
prompts. Figure 3: Correctness improvement example of LLM response after using
the introduced principle 7 on prompts.

Figure 4: Boosting of LLM response quality after employing the introduced
principles on prompts. small-scale indicates the 7B models, medium-scale
indicates the 13B models and large-scale indicates the 70B and GPT-3.5/4
models.

Figure 5: Correctness improvement of LLM response quality after employing the
introduced principles on prompts. small-scale indicates the 7B models, medium-
scale indicates the 13B models and large-scale indicates the 70B and GPT-3.5/4
models.

###  4.2 Models and Metrics

We use instruction finetuned LLaMA-1-{7, 13}, LLaMA-2-{7, 13}, off-the-shelf
LLaMA-2-70B-chat, GPT-3.5 (ChatGPT) and GPT-4 as our base models. We group
these models into different scales: small-scale (7B models), medium-scale
(13B) and large-scale (70B, GPT-3.5/4). We evaluate these models in two
settings: Boosting and Correctness. They are employed together to provide a
comprehensive understanding of a model’s performance.

  * •

Boosting. We assess the enhancement in the quality of responses from different
LLMs via human evaluation after applying the outlined prompt principles. The
original, unmodified prompts act as a benchmark for measuring this
enhancement. Demonstrating boosting confirms that a model’s performance has
improved due to the use of structured, principled instructions, as shown in
Fig. 2.

  * •

Correctness. The concept of correctness refers to the precision of the model’s
outputs or responses, ensuring they are accurate, relevant, and devoid of
errors. Human evaluators are utilized to gauge this aspect, which is crucial
for verifying the model’s accuracy. Correctness is a testament to the model’s
ability to generate outputs that align with the expected standards of
accuracy, as shown in Fig. 3.

###  4.3 Results

Figure 6: Boosting score across various LLMs on the ATLAS dataset. Figure 7:
Correctness improvement score on the ATLAS dataset. Figure 8: Illustration of
heatmap for LLMs improvement percentages. Figure 9: Illustration of heatmap
for LLMs correctness percentages.

####  4.3.1 Results on small, medium and large-scale LLMs

Boosting. The results of improvement after employing the introduced principles
are shown in Fig. 5. Generally, all principles can bring a significant
improvement on the three scales of LLMs. In the cases of principles 2, 5, 15,
16, 25 and 26, the large-scale models get the most improvement by the
principled prompts.

Correctness. As shown in Fig. 5, the employment of all principles typically
results in improvements exceeding 20% on the averaged various models. In
particular, for small and medium scale models, the improvement can basically
reach between 20% and 30%, and for large models, the improvement can reach
more than 50%.

####  4.3.2 Results on individual LLMs

Boosting. Fig. 6 illustrates the improvement of response quality on individual
model and principle after using the revised prompts. On average, there is a
stable 50% improvement across different LLMs. Fig. 8 further provides the
detailed results of improvement for each principle with different LLMs.

Correctness. Fig. 7 illustrates the enhancements in accuracy across different
sizes of LLMs. From LLaMA-2-13B, LLaMA-2-70B-chat to GPT-3.5 and GPT-4, there
is a noticeable trend: the larger the model, the greater the increase in
correctness. Fig. 9 further presents the correctness enhancements by each
principle.

####  4.3.3 More examples on various scales of LLMs

We present additional examples for both small and medium-scale LLMs, as
illustrated in Fig. 10 and 11 for the small-scale LLaMA-2-7B, and Fig. 12 and
13 for the medium-scale LLaMA-2-13B. Empirically, the use of the proposed
principles on prompts has demonstrably enhanced the accuracy of the responses
generated by these models.

Figure 10: Correctness improvement on small-scale LLaMA-2-7B model after using
the introduced principle on prompts.

Figure 11: Correctness improvement on small-scale LLaMA-2-7B model after using
the introduced principle on prompts. Figure 12: Correctness improvement on
medium-scale LLaMA-2-13B model after using the introduced principle on
prompts.

Figure 13: Correctness improvement on medium-scale LLaMA-2-13B model after
using the introduced principle on prompts.

##  5 Conclusion

We presented 26 principles through an exhaustive analysis that enhances the
LLM ability to focus on the crucial elements of the input context, leading to
the generation of quality responses. By guiding the LLM with these
meticulously crafted principles before the input is processed, we can
encourage the model towards producing better responses. Our empirical results
demonstrate that this strategy can effectively reformulate contexts that might
otherwise compromise the quality of the output, thereby enhancing the
relevance, brevity, and objectivity of the responses.

There are numerous directions for future exploration. In our experiments, we
utilized a constrained shot prompting approach to apply these principles.
There is potential to refine our base models to align with our principled
instructions further with alternative strategies, such as fine-tuning,
reinforcement learning, direct preference optimization, or different prompting
methods using our generated dataset. Moreover, the strategies that prove
successful could be integrated into standard LLM operations, for instance, by
fine-tuning with the original/principled prompts as inputs and the polished,
principled responses as targets for training.

##  6 Limitations and Discussion

While the proposed 26 principles are designed to improve and enhance the
quality of responses of LLMs across a diverse array of queries, the
effectiveness of these principles may diminish when dealing with questions
that are very complex or highly specialized. This limitation can mainly depend
on the reasoning capabilities and training of each model. To address these
variations, we have tested the principles across different scales to measure
their effectiveness comprehensively.

Despite our efforts in evaluating these principles on seven distinct language
models, it is crucial to acknowledge that models with architectures different
from those tested might respond in different ways to these principles.
Additionally, our assessment of improvement and correctness percentages was
based on a limited selection of questions. Expanding the question set in
future research could yield more generalized findings and offer deeper
insights into the applicability of each principle.

## References

  * [1] Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré.  Ask me anything: A simple strategy for prompting language models, 2022\.
  * [2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  CoRR, abs/1810.04805, 2018.
  * [4] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.  Training compute-optimal large language models, 2022.
  * [5] Shima Imani, Liang Du, and Harsh Shrivastava.  Mathprompter: Mathematical reasoning using large language models.  arXiv preprint arXiv:2303.05398, 2023.
  * [6] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7b, 2023.
  * [7] Ehsan Kamalloo, Nouha Dziri, Charles LA Clarke, and Davood Rafiei.  Evaluating open-domain question answering in the era of large language models.  arXiv preprint arXiv:2305.06984, 2023.
  * [8] Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill.  Can language models learn from explanations in context?  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 537–563, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.
  * [9] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al.  Starcoder: may the source be with you!  arXiv preprint arXiv:2305.06161, 2023.
  * [10] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Alpacaeval: An automatic evaluator of instruction-following models.  https://github.com/tatsu-lab/alpaca_eval, 2023.
  * [11] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al.  Competition-level code generation with alphacode.  Science, 378(6624):1092–1097, 2022.
  * [12] Zekun Li, Baolin Peng, Pengcheng He, Michel Galley, Jianfeng Gao, and Xifeng Yan.  Guiding large language models via directional stimulus prompting.  arXiv preprint arXiv:2302.11520, 2023.
  * [13] OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.  Gpt-4 technical report, 2023.
  * [14] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * [15] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [16] Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving.  Scaling language models: Methods, analysis & insights from training gopher.  CoRR, abs/2112.11446, 2021.
  * [17] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  CoRR, abs/1910.10683, 2019.
  * [18] Zhiqiang Shen, Sondos Mahmoud Bsharat, and Aidar Myrzakhan.  Atlas: A llm inquiry principle benchmark.  Preprint, 2024.
  * [19] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV au2, Eric Wallace, and Sameer Singh.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts, 2020.
  * [20] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp, Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalie Clay, Phil Crone, Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad, Ale Jakse Hartman, Martin Chadwick, Gaurav Singh Tomar, Xavier Garcia, Evan Senter, Emanuel Taropa, Thanumalayan Sankaranarayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomènech Badia, David Reitter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq Iqbal, Gabriela Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Yujing Zhang, Ravi Addanki, Antoine Miech, Annie Louis, Laurent El Shafey, Denis Teplyashin, Geoff Brown, Elliot Catt, Nithya Attaluri, Jan Balaguer, Jackie Xiang, Pidong Wang, Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming-Wei Chang, Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Ahn, Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sébastien M. R. Arnold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, Srivatsan Srinivasan, Hyeontaek Lim, Sarah Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li, Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipanjan Das, Dominika Rogozińska, Vitaly Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Giménez, Legg Yeung, Hanzhao Lin, James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fernando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang, Shuo yiin Chang, Paul Komarek, Ross McIlroy, Mario Lučić, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yong Cheng, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina Butterfield, Justin Chung, Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lenc, Timothy Chung, Aedan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko Iinuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphaël Lopez Kaufman, Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Aditya Siddhant, Nenad Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma, Angelos Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih-Kuan Yeh, Soravit Changpinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar Samuel, Mingqiu Wang, Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Léonard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang, Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, Víctor Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thornton, Fangxiaoyu Feng, Wojciech Stokowiec, Ce Zheng, Phoebe Thacker, Çağlar Ünlü, Zhishuai Zhang, Mohammad Saleh, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowski, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinberg, James Cobon-Kerr, Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, YaGuang Li, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Paganini, Jean-Baptiste Lespiau, Alexandre Moufarek, Samer Hassan, Kaushik Shivakumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Nemanja Rakićević, Mostafa Dehghani, Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Gamaleldin Elsayed, Ed Chi, Mahdis Mahdieh, Ian Tenney, Nan Hua, Ivan Petrychenko, Patrick Kane, Dylan Scandinaro, Rishub Jain, Jonathan Uesato, Romina Datta, Adam Sadovsky, Oskar Bunyan, Dominik Rabiej, Shimu Wu, John Zhang, Gautam Vasudevan, Edouard Leurent, Mahmoud Alnahlawi, Ionut Georgescu, Nan Wei, Ivy Zheng, Betty Chan, Pam G Rabinovitch, Piotr Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nino Vieillard, Sahitya Potluri, Jane Park, Elnaz Davoodi, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhijit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Zhe Chen, Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Chris Gorgolewski, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Paul Suganthan, Evan Palmer, Geoffrey Irving, Edward Loper, Manaal Faruqui, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Michael Fink, Alfonso Castaño, Irene Giannoumis, Wooyeol Kim, Mikołaj Rybiński, Ashwin Sreevatsa, Jennifer Prendki, David Soergel, Adrian Goedeckemeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Diana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu Li, Chimezie Iwuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Marin Georgiev, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang, Jeff Piper, Nathan Ie, Minnie Lui, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Lam Nguyen Thiet, Daniel Andor, Pedro Valenzuela, Cosmin Paduraru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somer Greene, Duc Dung Nguyen, Paula Kurylowicz, Sarmishta Velury, Sebastian Krause, Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal, Tejasi Latkar, Mingyang Zhang, Quoc Le, Elena Allica Abellan, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Sid Lall, Ken Franko, Egor Filonov, Anna Bulanova, Rémi Leblond, Vikas Yadav, Shirley Chung, Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Hao Zhou, Alek Dimitriev, Hannah Forbes, Dylan Banarse, Zora Tung, Jeremiah Liu, Mark Omernick, Colton Bishop, Chintu Kumar, Rachel Sterneck, Ryan Foley, Rohan Jain, Swaroop Mishra, Jiawei Xia, Taylor Bos, Geoffrey Cideron, Ehsan Amid, Francesco Piccinno, Xingyu Wang, Praseem Banzal, Petru Gurita, Hila Noga, Premal Shah, Daniel J. Mankowitz, Alex Polozov, Nate Kushman, Victoria Krakovna, Sasha Brown, MohammadHossein Bateni, Dennis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Anhad Mohananey, Matthieu Geist, Sidharth Mudgal, Sertan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee-Thorp, Christopher Yew, Quan Yuan, Sumit Bagri, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Aliaksei Severyn, Jonathan Lai, Kathy Wu, Heng-Tze Cheng, David Miller, Nicolas Sonnerat, Denis Vnukov, Rory Greig, Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang, Rui Zhu, Mark Geller, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Nejc Trdin, Andrei Sozanschi, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papamakarios, Rupert Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Abhimanyu Goyal, Diane Wu, Denese Owusu-Afriyie, Cosmo Du, Chloe Thornton, Jordi Pont-Tuset, Pradyumna Narayana, Jing Li, Sabaer Fatehi, John Wieting, Omar Ajmeri, Benigno Uria, Tao Zhu, Yeongil Ko, Laura Knight, Amélie Héliou, Ning Niu, Shane Gu, Chenxi Pang, Dustin Tran, Yeqing Li, Nir Levine, Ariel Stolovich, Norbert Kalb, Rebeca Santamaria-Fernandez, Sonam Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Balaji Lakshminarayanan, Charlie Deck, Shyam Upadhyay, Hyo Lee, Mike Dusenberry, Zonglin Li, Xuezhi Wang, Kyle Levin, Raphael Hoffmann, Dan Holtmann-Rice, Olivier Bachem, Summer Yue, Sho Arora, Eric Malmi, Daniil Mirylenka, Qijun Tan, Christy Koh, Soheil Hassas Yeganeh, Siim Põder, Steven Zheng, Francesco Pongetti, Mukarram Tariq, Yanhua Sun, Lucian Ionita, Mojtaba Seyedhosseini, Pouya Tafti, Ragha Kotikalapudi, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszcz, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Chenkai Kuang, Vinod Koverkathu, Christopher A. Choquette-Choo, Yunjie Li, TJ Lu, Abe Ittycheriah, Prakash Shroff, Pei Sun, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Ishita Dasgupta, Guillaume Desjardins, Marco Cornero, Brona Robenek, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rivière, Alanna Walton, Clément Crepy, Alicia Parrish, Yuan Liu, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Latorre-Chimoto, Hanna Klimczak-Plucińska, David Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Ivo Penchev, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yadlowsky, Amy Shen, Amir Globerson, Adam Kurzrok, Lynette Webb, Sahil Dua, Dong Li, Preethi Lahoti, Surya Bhupatiraju, Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Rammohan Belle, Lei Wang, Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru, John Aslanides, Nidhi Vyas, Martin Wicke, Xiao Ma, Taylan Bilal, Evgenii Eltyshev, Daniel Balle, Nina Martin, Hardie Cate, James Manyika, Keyvan Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, Warren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Adams Yu, Christof Angermueller, Xiaowei Li, Weiren Wang, Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurumurthy, Mark Goldenson, Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando, Kevin Brooks, Ken Durden, Harsh Mehta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinhyuk Lee, Komal Jalan, Dinghua Li, Ginger Perng, Blake Hechtman, Parker Schuh, Milad Nasr, Mia Chen, Kieran Milan, Vladimir Mikulik, Trevor Strohman, Juliana Franco, Tim Green, Demis Hassabis, Koray Kavukcuoglu, Jeffrey Dean, and Oriol Vinyals.  Gemini: A family of highly capable multimodal models, 2023.
  * [21] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [22] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * [23] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.  Chain-of-thought prompting elicits reasoning in large language models, 2023.
  * [24] Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt.  A prompt pattern catalog to enhance prompt engineering with chatgpt, 2023\.
  * [25] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  arXiv preprint arXiv:2306.05685, 2023.
  * [26] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi.  Least-to-most prompting enables complex reasoning in large language models, 2023.


caption captionlabel disposition publishers subject mvbookbookvolumes

Talk presented at the International Workshop on Future Linear Colliders
(LCWS2019), Sendai, Japan, October 30th 2019. Deutsches
Elektronen/Synchrotron, DESY

# Recent Performance Studies of the GEM-based TPC Readout
(DESY Module)

Ties Behnke  Ralf Diener  Ulrich Einhaus  Uwe Krämer
Paul Malek  Oliver Schäfer  Mengqing Wu

###### Abstract

For the International Large Detector (ILD) at the planned International Linear
Collider (ILC) a Time Projection Chamber (TPC) is foreseen as the main
tracking detector. To achieve the required point resolution, Micro/Pattern
Gaseous Detectors (MPGD) will be used in the amplification stage. A readout
module using a stack of three Gas Electron Multipliers (GEM) for gas
amplification was developed at DESY and tested at the DESY II Test Beam
Facility.

After introducing the readout module and the infrastructure at the test beam
facility, the performance related to single point and double/hit resolution of
three of these modules is presented. This is followed by results on the
particle identification capabilities of the system, using the specific energy
loss $\mathrm{d}E/\mathrm{d}x$, and simulation studies, aimed to investigate
and quantify the impact of high granularity on $\mathrm{d}E/\mathrm{d}x$
resolution.

In addition, a new and improved TPC field cage and the LYCORIS Large/Area
Silicon/Strip Telescope for the test beam are described. The LYCORIS beam
telescope is foreseen to provide a precise reference of the particle
trajectory to validate the momentum resolution measured with a large TPC
prototype. For this purpose, it is being installed and tested at the test beam
facility within the so/called PCMAG (Persistent Current Magnet).

###### Contents

  1. 1 Introduction
  2. 2 The LCTPC Test Beam at DESY
    1. 2.1 The DESY GridGEM Module
    2. 2.2 The LCTPC Setup at the DESY II Test Beam Facility
    3. 2.3 Established System Performance
    4. 2.4 Determination of $\mathrm{d}E/\mathrm{d}x$ Resolution
  3. 3 Impact of High Granularity on $\mathrm{d}E/\mathrm{d}x$ resolution
    1. 3.1 The Cluster Counting Method
    2. 3.2 The ROPPERI System
  4. 4 The Large Prototype TPC Field Cage
  5. 5 The LYCORIS Beam Telescope
  6. 6 Summary
  7. 7 Acknowledgments

##  1 Introduction

This project is part of the design effort for the International Large Detector
(ILD) at the future International Linear Collider experiment (ILC) [1]. The
FLC-TPC group at DESY is part of the LCTPC collaboration [2], which is driving
the efforts of developing a TPC for the ILD. Different readout and
amplification technologies based on Micro/Pattern Gaseous Detectors (MPGD) are
currently studied within the collaboration. FLC-TPC is committed to designing
a self-supporting readout composed of GEMs [3] mounted on thin ceramic grids
[4].

##  2 The LCTPC Test Beam at DESY

###  2.1 The DESY GridGEM Module

In order to maximize the sensitive area while minimizing the gaps and the
material budget, the DESY GridGEM module has been developed. The mechanical
base of the module is an aluminium back frame that is used to mount the module
in the TPC prototype end plate. Onto this a printed circuit board (PCB) is
glued, which contains the readout pad plane. This structure also provides the
gas tightness of the module. The pad plane contains 4832 readout pads with a
pitch of $1.26\text{\,}\mathrm{mm}\text{/}$ arranged in 28 concentric rows
with a pitch of $5.85\text{\,}\mathrm{mm}\text{/}$ in radial direction. On top
of the pad board the amplification structure is built. It consists of three
GEMs supported by $1\text{\,}\mathrm{mm}\text{/}$ high alumina/ceramic frames.
Additional frames are used as spacers between the GEM foils to define transfer
and induction gaps, as shown in Figure 1(a). A fully assembled module is shown
in Figure 1(b). Since the frame bars are only $\sim
1.4\text{\,}\mathrm{mm}\text{/}$ wide, the modules have an active area of
about $95\text{\,}\mathrm{\char 37\relax}\text{/}$. The data is read out by a
modified version of the ALICE TPC Readout (ALTRO) [5, 6].

(a) Explosion view of the GEM module.

(b) Picture of an assembled GEM module.

Figure 1: The DESY GridGEM module.

###  2.2 The LCTPC Setup at the DESY II Test Beam Facility

The LCTPC readout prototypes are tested in an electron beam at the DESY II
Test Beam Facility [7]. The beam line delivers electrons in the momentum range
from
$1\text{\,}\mathrm{GeV}\text{/}\mathrm{\text{{c}}}6\text{\,}\mathrm{GeV}\text{/}\mathrm{\text{{c}}}$
at rates from $\mathcal{O}($\mathrm{kHz}\text{/}$)$ to $\sim
100\text{\,}\mathrm{Hz}\text{/}$, respectively.

The setup consists of a large TPC prototype (LPTPC) and a superconducting
solenoid magnet, the so/called Persistent Current Magnet (PCMAG), shown in
Figure 2(a). A coincidence signal from four scintillator planes placed in the
beam path in front of the PCMAG is used as the trigger signal for the main
setup. Slow control is implemented using the Distributed Object/Oriented
Control System (DOOCS). The solenoid produces a magnetic field of up to
$1\text{\,}\mathrm{T}\text{/}$. It is mounted on a movable stage installed in
the experimental area T24/1 of the test beam. The stage can be translated
vertically and horizontally perpendicular to the beam axis and rotated around
the vertical axis. Additionally, the TPC can be rotated around its axis inside
the magnet. This allows to scan the beam over the full volume of the TPC and
to change both the polar and azimuthal angle of the beam relative to the TPC.
The usable inner diameter of the PCMAG is $85\text{\,}\mathrm{cm}\text{/}$.
Due to a lightweight construction without a field return yoke, the magnet wall
contributes only about $20\text{\,}\mathrm{\char 37\relax}\text{/}$ of a
radiation length $X_{0}$ [8].

The LPTPC has a maximum drift length of about
$570\text{\,}\mathrm{mm}\text{/}$ and an inner diameter of
$75\text{\,}\mathrm{cm}\text{/}$. The anode end plate contains slots for 7
modules, arranged in three staggered rows. Since all module slots are of
identical shape, the rows are not concentric. Unused slots can be filled with
dummy modules for a uniform termination of the drift field. This is shown in
Figure 2(b), where the active modules can be distinguished by their more
reflective surface.

(a) The PCMAG.

(b) The LPTPC anode end-plate [9].

Figure 2: The PCMAG solenoid and the LPTPC setup at the DESY II Test Beam
Facility.

###  2.3 Established System Performance

(a) Comparison of measurements.

(b) Extrapolation to the ILD.

Figure 3: The transverse resolution $\sigma_{r\varphi}$ 3(a) as measured in
the two beam test campaigns. The markers represent the data and the lines show
fits of the expected behaviour as described in [10]. 3(b) An extrapolation to
the size and magnetic field of the ILD TPC. The colored bands show the
$2\text{\,}\sigma$ confidence intervals.

In two beam test periods in 2013 and 2016, respectively, the point resolution
of the system has been determined. Generally good agreement was found between
the data of both tests. A small discrepancy was found in the $r\varphi$
/̄resolution at small drift distances, as shown in Figure 3(a), visible as a
larger intrinsic resolution $\sigma_{r\varphi,0}$ of the system and a smaller
transverse diffusion constant $D_{\mathrm{t}}$ in the newer data. Various
systematic effects have been studied, including gas pressure and temperature,
gas composition and contamination as well as settings of high voltages and
electric and magnetic fields. Within reasonable fluctuations, none of these
effects can explain the observed discrepancy and further investigations are
required.

Using the measurement data, the point resolution can be extrapolated to the
conditions of the ILD TPC. Since the transverse diffusion is reduced due to
the higher magnetic field of $3.5\text{\,}\mathrm{T}\text{/}$, the expected
diffusion constant is simulated using Magboltz [11]. Figure 3(b) shows the
resulting $r\varphi$ /̄resolution for the 2013 data set. The orange line
represents the direct extrapolation of the measurement, including the measured
electron attachment rate. Since the attachment rate is expected to be
negligible in the ILD TPC, due to much lower gas contamination, for the green
line the attachment rate was set to zero. In this case both data sets result
in a resolution around
$\sigma_{r\varphi}=$100\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$$ at the
full drift length of the ILD TPC, fulfilling the requirement set in [1].

Figure 4: The two/hit separation efficiency versus the distance between the
respective tracks. The three different marker color represent different steps
of the algorithm. Red: no attempted hit separation; green: intermediate step;
blue: full algorithm.

A related quantity is the double/hit resolution, which is important in the
case of close/by or partially overlapping tracks. Here, an algorithm was
developed that first identifies potential double/hit candidates based on
preliminary track fits. It then performs fits to the charge distribution of
both a single/hit and a double/hit hypothesis. If the double/hit hypothesis
performs significantly better than the single/hit hypothesis, the charge
distribution is split into two hits based on the fit. The blue dots in Figure
4 show the two/hit separation efficiency of this algorithm versus the distance
between the two reconstructed tracks at the location of the hits. The other
colors represent intermediate steps of the process. The solid lines represent
fits of an error function to the respective data. The efficiency reaches
$50\text{\,}\%$ at distances around and below $2\text{\,}\mathrm{mm}\text{/}$,
slightly depending on the drift distance due to diffusion. This fulfills the
requirement specified in [1].

###  2.4 Determination of $\mathrm{d}E/\mathrm{d}x$ Resolution

(a) Track $\mathrm{d}E/\mathrm{d}x$ distribution.

(b) Optimization of the truncated mean.

Figure 5: The determination of the $\mathrm{d}E/\mathrm{d}x$ resolution. 5(a)
The distribution of the truncated mean of $\mathrm{d}E/\mathrm{d}x$ per track.
The red line represents the Gaussian fit used to extract the resolution. 5(b)
The $\mathrm{d}E/\mathrm{d}x$ resolution as a function of the truncation
fraction. The blue marks show the resolution for the measured LPTPC tracks and
the green marks an extrapolation to the full ILD TPC.

The average energy loss on a track is calculated from the charge distribution
of all hits that were assigned to this track by a track finding algorithm [12,
13]. On average, each track in the large TPC prototype contains about 55 hits
that are considered valid for the $\mathrm{d}E/\mathrm{d}x$ calculation. The
mean $\mathrm{d}E/\mathrm{d}x$ per track in the LPTPC is shown in Figure 5(a).
The relative resolution is extracted from a Gaussian fit as $\sigma$ divided
by the mean. To achieve a stable estimate of the average energy loss from the
Landau/like primary ionization distribution the mean of a truncated
distribution is calculated, where only a certain fraction of the lowest charge
hits is taken into account. As shown in Figure 5(b), a shallow optimum of the
resolution is found around a truncation fraction of $75\text{\,}\mathrm{\char
37\relax}\text{/}$. For tracks extrapolated to the full ILD TPC as described
below the same optimum is found. Using the optimal truncation fraction the
resolution for the measured tracks is
$\sigma_{\mathrm{d}E/\mathrm{d}x}=$8.62\pm 0.14\text{\,}\mathrm{\char
37\relax}\text{/}$$.

Figure 6: $\mathrm{d}E/\mathrm{d}x$ resolution obtained using pseudo-tracks of
various length. The red line represents the power/law fit described in the
text.

The final goal of this analysis is to estimate the $\mathrm{d}E/\mathrm{d}x$
resolution of the full size ILD TPC. Therefore, the results need to be
extrapolated to the greater track length, i.e. the greater number of possible
hits. Naively, one could expect the resolution to scale with the number of
hits $N$ as $1/\sqrt{N}$ but experience of previous experiments shows, that it
rather follows a power/law

$\sigma_{\mathrm{d}E/\mathrm{d}x}\propto N^{-k}$

with an exponent $k$ in the range of $0.430.48$, depending on experiment and
$\mathrm{d}E/\mathrm{d}x$ calculation methods [14]. This is caused by the
Laundau/like shape of the probability distribution of the primary ionization
process. Therefore, the exponent needs to be determined for this setup
individually. Since the maximum number of valid hits in the prototype TPC is
limited, a simple fit of the power/law to the resolution for tracks up to that
length would introduce large uncertainties in the extrapolation. Instead,
randomly selected hits from multiple tracks are combined to form pseudo/tracks
of arbitrary length. Since the software processes events sequentially, this
process is limited to hits from consecutive events [12, 15, 16]. To avoid
biases from this method, care is taken that each hit is used only in a single
pseudo/track. The mean $\mathrm{d}E/\mathrm{d}x$ value for each pseudo/track
is then calculated as above. Samples of pseudo/tracks with various numbers of
hits are created and the resolution calculated for each. Figure 6 shows the
resulting resolution versus the chosen track length. The power/law fit returns
an exponent of $k=$0.47\pm 0.01$$. The $\mathrm{d}E/\mathrm{d}x$ resolution at
220 hits as calculated from the fit is equal to
$\sigma_{\mathrm{d}E/\mathrm{d}x}=$4.4\pm 0.1\text{\,}\mathrm{\char
37\relax}\text{/}$$, fulfilling the goal given in [1].

##  3 Impact of High Granularity on $\mathrm{d}E/\mathrm{d}x$ resolution

Simulation studies have been performed in order to investigate and quantify
the impact of high granularity on the $\mathrm{d}E/\mathrm{d}x$ resolution.

###  3.1 The Cluster Counting Method

Traditionally, the energy loss is computed by summing all the electrons
generated from ionization by the incident particle. The number of electrons
produced in the ionization process is distributed according to a Landau
distribution, which has a long tail towards large values. The relatively large
width of the distribution is reflected in a degradation of the correlation
between the measured energy and the momentum of the particle. An alternative
method consists in counting the number of ionizing interactions of the
incident particles. This follows a Poisson distribution with a significantly
smaller width, compared to the Landau, therefore providing a better
correlation and particle identification power [17]. A comparison between the
two methods has been performed studying the pion/kaon separation power as a
function of the pad size, as shown in Figure 7, where the stars and the
circles represent the simulation performed using the charge summation method
and the cluster counting method, respectively. They are compared to results
from test beam measurements represented by the blue squares. Below a pad size
of about $500\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ the cluster
counting method performs better than the charge summation method.

Figure 7: Pion/kaon separation power as a function of the pad size. Blue
squares represent results from test beam measurements; stars and circles
represent the simulations using the charge summation and cluster counting
methods, respectively.

###  3.2 The ROPPERI System

In order to exploit the cluster counting capability, a high granularity
readout system is needed. The ROPPERI (Readout Of a Pad Plane with ElectRonics
designed for pIxels) system represents a possibility to combine high
granularity, flexibility and large anode coverage of the pad-plane readout
[18].

Figure 8: The ROPPERI board.

The idea of the ROPPERI board implementation is sketched in Figure 8. GEMs are
used for amplification, small pads on a PCB form the anode and they are read
out by a Timepix ASIC, which has a matrix of $256\times 256=65536$ pixels with
a $55\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ pitch. The connections
from the pads are routed through the PCB to the ASIC which is bump bonded to
the PCB surface. The Timepix power and communication pads are on the same side
of the ASIC as the pixels, which makes it necessary to also connect them via
bump bonds back into the PCB, where they are routed to a macroscopic connector
plug (VHDCI).

A prototype board had been designed to study different pad sizes and line
lengths, and the influence of the capacitance on the signal to noise ratio.
Several boards have been produced out of which seven have been bump bonded in
different tries. Out of these seven boards, only three worked completely.
During the data taking, different thresholds have been used in runs with
$100200$ frames. The bump bonding tests showed issues connected to temperature
during and after the bump bonding. Due to temperature difference and differing
expansion coefficients of the materials of the chip and PCB, the bump
connections broke. Until this happened, only a limited number of test runs
were possible.

In the test runs, the worst pads and lines were about 11 times noisier than
the bare Timepix chip with a known equivalent noise charge of about 90
electrons. So, under the assumption of a noise level of about 1000 electrons
and $1\text{\cdot}{10}^{4}2\text{\cdot}{10}^{4}$ signal electrons arriving on
a pad after the GEM amplification, a signal to noise ratio higher than 10 can
be estimated.

##  4 The Large Prototype TPC Field Cage

(a) The current prototype field cage.

(b) Sketch of the structure of the current field cage wall [19].

Figure 9: The existing large prototype TPC field cage.

DESY is working on the construction of a new large prototype TPC field cage. A
high precision TPC field cage for the ILD TPC has to guarantee a low material
budget, a high voltage stability and high mechanical precision. Since the
current large prototype field cage, shown in Figure 9(a), does not fully meet
the requirements on the mechanical precision [20, 21], it was decided to build
a new one. This field cage is build in the DESY workshop, so it also
represents an opportunity to gain in/house experience in the design and
construction of such a composite structure. The prototype is supposed to fit
into the PCMAG, therefore its dimensions are restricted by the magnet
geometry, as was the case for the previous field cage. The outer diameter is
foreseen to be $77\text{\,}\mathrm{cm}\text{/}$ leaving a gap of
$4\text{\,}\mathrm{cm}\text{/}$ to the inner wall of the magnet, which has a
diameter of $85\text{\,}\mathrm{cm}\text{/}$.

As stated above, the $61\text{\,}\mathrm{cm}\text{/}$ long field cage should
be lightweight and at the same time mechanically very stable. Therefore, its
structure is made of composite materials, already used in the construction of
previous prototype field cages. Figure 9(b) shows a sketch of the wall
structure, which is estimated to contribute $1.31\text{\,}\mathrm{\char
37\relax}\text{/}$ of a radiation length $X_{0}$. An optimized design of field
strips and mirror strips, implemented on a double/sided copper/coated
polyimide foil, is used to generate a homogeneous field inside the cage. A
resistor chain linearly degrades the voltage along the length of the field
cage, creating a constant field. At the time of the construction of the
current field cage, no producer was found to manufacture the
$61\text{\times}226\text{\,}\mathrm{cm}\text{/}$ large, two/sided field strip
foil in one piece. Therefore, it was produced from two about
$30\text{\,}\mathrm{cm}\text{/}$ wide foils which were connected during the
construction. Now, the production of such a large foil in one piece is
possible and a set of foils is being produced at the CERN workshop.

(a)

(b)

Figure 10: Schematic view of cuts through the field strip foils perpendicular
to the field strips. 10(a) shows the current and 10(b) the new design. The
field strips are depicted in black, the polyimide foil in red, the white boxes
represent the resistors. Figures from [21].

After the construction of the current field cage, the design studies of the
field strip layout have been carried on and a simpler design delivering a
comparable field quality was found. In the new design, each mirror strip spans
over two gaps so only half of the resistors and vias are needed compared to
the previous design. Details can be found in [21]. Figure 10 shows sketches of
the cross sections of the old and new designs of the field strips.

##  5 The LYCORIS Beam Telescope

Figure 11: Sketch of the functionality of the LYCORIS telescope (here denoted
by “strip telescope”) installed within the PCMAG solenoid together with a
large TPC prototype under test.

The LYCORIS Large Area Silicon Strip Telescope [22, 23] is foreseen to be
installed within the PCMAG at the DESY II Test Beam Facility. The system
requirements described here are for the use case of a TPC within the PCMAG.
The telescope is designed to provide a precise reference measurement of the
particle trajectory, as shown in the sketch in Figure 11. These reference
measurements can be used to study and correct for potential inhomogeneities of
the electric field within the TPC volume, which otherwise limit the achievable
momentum resolution.

To meet these criteria, the telescope needs to provide a spatial single point
resolution of better than
$\sigma_{\mathrm{bend}}=$10\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$$
along the bending direction within the PCMAG and a resolution better than
$\sigma_{\mathrm{drift}}=$1\text{\,}\mathrm{mm}\text{/}$$ along the drift axis
of the TPC.

Hybrid-Less silicon strip sensors, designed by SLAC, are used for the LYCORIS
telescope. They are characterized by an active area of
$10\text{\times}10\text{\,}\mathrm{cm}\text{/}$ with a strip pitch of
$25\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$ and contribute
$0.3\text{\,}\mathrm{\char 37\relax}\text{/}$ of a radiation length. The
sensors are read out by an integrated digital readout chip called KPiX, also
designed at SLAC, which is directly bump bonded onto the sensors and routed to
the strips within the silicon sensor.

Figure 12: The LYCORIS telescope mounted in the PCMAG solenoid together with a
Mimosa telescope.

The telescope has been fully assembled and successfully tested in several test
beam campaigns, also in combination with EUDET-type Mimosa pixel telescopes
[24]. A picture of the LYCORIS telescope, used in combination with a Mimosa
telescope inside the PCMAG solenoid is shown in Figure 12. The system is
undergoing final adjustments to be integrated into the infrastructure of the
DESY II Test Beam Facility.

##  6 Summary

The DESY GridGEM modules have been shown to be able to fulfill the
requirements set in [1] for several important parameters: point resolution,
double/hit resolution and $\mathrm{d}E/\mathrm{d}x$ resolution. Simulation
studies have shown that the $\mathrm{d}E/\mathrm{d}x$ resolution may be
improved with an optimized pad size of
$\mathcal{O}($500\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$)$. The
ROPPERI system indicates a possible way to achieve such readouts with very
high granularity.

A new, improved field cage for the large TPC prototype is in preparation. The
Lycoris beam telescope has been fully assembled and was successfully tested.
Some more work is needed to enable integration into the infrastructure of the
DESY II Test Beam Facility.

##  7 Acknowledgments

Figure 2(b) is used with permission from the LCTPC Collaboration as authors of
the article “A time projection chamber with GEM-based readout”, Nuclear
Instruments and Methods in Physics Research A 856 (2017) 109–118, Elsevier, ©
2016.
This material is based upon work supported by the National Science Foundation
under Grant No. 0935316 and was supported by JSPS KAKENHI Grant No. 23000002.
The research leading to these results has received funding from the European
Commission under the 6th Framework Programme “Structuring the European
Research Area”, contract number RII3-026126, and under the FP7 Research
Infrastructures project AIDA, grant agreement no. 262025.
Special thanks go to Y. Makida, M. Kawai, K. Kasami and O. Araoka of the KEK
IPNS cryogenic group, and A. Yamamoto of the KEK cryogenic centre for their
support in the configuration and installation of the superconducting PCMAG
solenoid.
The measurements leading to these results have been performed at the Test Beam
Facility at DESY Hamburg (Germany), a member of the Helmholtz Association.
The authors would like to thank the technical team at the DESY II accelerator
and test beam facility for the smooth operation of the test beam and the
support during the test beam campaign. The contributions to the experiment by
the University of Lund, KEK, Nikhef and CEA are gratefully acknowledged.

## References

  * [1] “Detectors”  Linear Collider Collaboration, 2013  arXiv: https://www.linearcollider.org/
  * [2] “Homepage of the LCTPC collaboration”  URL: http://www.lctpc.org
  * [3] Fabio Sauli  “GEM: A new concept for electron amplification in gas detectors”  In _Nuclear Instruments and Methods in Physics Research_ 386.2, 1997, pp. 531–534  DOI: 10.1016/S0168-9002(96)01172-2
  * [4] Ties Behnke, Ralf Diener, Christoph Rosemann and Lea Steder  “A Novel Self-supporting GEM-based Amplification Structure for a Time Projection Chamber at the ILC”  In _Journal of Instrumentation_ 8, 2013, pp. 12009  DOI: 10.1088/1748-0221/8/12/P12009
  * [5] R. Esteve Bosch, A. Jimenez de Parga, B. Mota and L. Musa  “The ALTRO chip: A 16-channel A/D converter and digital processor for gas detectors”  In _IEEE Transactions on Nuclear Science_ 50.6, 2003, pp. 2460–2469  DOI: 10.1109/TNS.2003.820629
  * [6] Leif Jönsson and Ulf Mjörnmark  “Front-end electronics and data acquisition for the LCTPC”, 2007  URL: http://www.eudet.org/e26/e28/e182/e605/eudet-memo-2007-53.pdf
  * [7] R. Diener et al.  “The DESY II test beam facility”  In _Nuclear Instruments and Methods in Physics Research_ 922, 2019, pp. 265–286  DOI: 10.1016/j.nima.2018.11.133
  * [8] A. Yamamoto et al.  “Balloon-borne experiment with a superconducting solenoidal magnet spectrometer”  In _Advances in Space Research_ 14, 1994, pp. 2  DOI: 10.1016/0273-1177(94)90071-X
  * [9] David Attié et al.  “A Time projection chamber with GEM-Based readout”  In _Nuclear Instruments and Methods in Physics Research_ 856, 2017, pp. 109–118  DOI: 10.1016/j.nima.2016.11.002
  * [10] Felix Johannes Müller  “Development of a Triple GEM Readout Module for a Time Projection Chamber $\&$ Measurement Accuracies of Hadronic Higgs Branching Fractions in $\nu\nu$H at a 350 GeV ILC” Dissertation, Universität Hamburg, 2016, DESY-THESIS  Hamburg: Verlag Deutsches Elektronen-Synchrotron, 2016, pp. 245  DOI: 10.3204/PUBDB-2016-02659
  * [11] S.F. Biagi  “Monte Carlo simulation of electron drift and diffusion in counting gases under the influence of electric and magnetic fields”  In _Nuclear Instruments and Methods in Physics Research_ 421.1 Elsevier, 1999, pp. 234–240  DOI: 10.1016/S0168-9002(98)01233-9
  * [12] Jason Abernathy et al.  “MarlinTPC: A Marlin based common TPC software framework for the LC-TPC Collaboration.”, 2007  URL: http://bib-pubdb1.desy.de/record/83142
  * [13] Claus Kleinwort  “A Track Finding Method for a TPC Based on fast Hough Transformation”, 2014  URL: http://bib-pubdb1.desy.de/record/193046
  * [14] W. Blum, W. Riegler and L. Rolandi  “Particle Detection with Drift Chambers”, Particle Acceleration and Detection  Springer, 2008  DOI: 10.1007/978-3-540-76684-1
  * [15] Ties Behnke and Frank Gaede  “Software for the international linear collider: Simulation and reconstruction frameworks”  In _Pramana_ 69.6, 2007, pp. 1089–1092  Deutsches Elektronen-Synchrotron, DESY  DOI: 10.1007/s12043-007-0233-z
  * [16] F. Gaede  “Marlin and LCCD: Software tools for the ILC”  In _Nuclear Instruments and Methods in Physics Research_ 559, 2006, pp. 177–180  DOI: 10.1016/j.nima.2005.11.138
  * [17] M. Hauschild  “dE/dx and Particle ID Performance with Cluster Counting”, 2006  URL: https://agenda.linearcollider.org/event/1049/contributions/506/
  * [18] U.Einhaus, J.Kaminski and M.Caselle  “ROPPERI - A TPC readout with GEMs, pads and Timepix”, 2018  arXiv:1801.07178 [physics.ins-det]
  * [19] Peter Schade and Jochen Kaminski  “A large TPC prototype for a linear collider detector”  In _Nuclear Instruments and Methods in Physics Research_ 628.1, 2011, pp. 128–132  DOI: 10.1016/j.nima.2010.06.300
  * [20] T Behnke et al.  “A lightweight field cage for a large TPC prototype for the ILC”  In _Journal of Instrumentation_ 5.10, 2010, pp. 10011  DOI: 10.1088/1748-0221/5/10/P10011
  * [21] Peter Schade  “Development and Construction of a Large TPC Prototype for the ILC and Study of $\tau$ Polarisation in $\tilde{\tau}$ Decays with the ILD Detector”  Deutsches Elektronen-Synchrotron, DESY, 2009  DOI: 10.3204/DESY-THESIS-2009-040
  * [22] M. Wu et al.  “Lycoris: A large area beam telescope based on hybrid-less strip silicon sensors”  IEEE, 2018  DOI: 10.1109/NSSMIC.2018.8824667
  * [23] Mengqing Wu et al.  “Development of a large active area beam telescope based on the SiD microstrip sensor”  In _Nuclear Instruments and Methods in Physics Research_ , 2019, pp. 162864  DOI: 10.1016/j.nima.2019.162864
  * [24] Hendrik Jansen et al.  “Performance of the EUDET-type beam telescopes”  In _EPJ Techniques and Instrumentation_ 3.1, 2016, pp. 7  DOI: 10.1140/epjti/s40485-016-0033-2