Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

# Learning to Compress Prompts with Gist Tokens

 Jesse Mu, Xiang Lisa Li, Noah Goodman

Stanford University

muj@cs.stanford.edu, {xlisali,ngoodman}@stanford.edu

###### Abstract

Prompting is the primary way to utilize the multitask capabilities of language models (LMs), but prompts occupy valuable space in the input context window, and repeatedly encoding the same prompt is computationally inefficient. Finetuning and distillation methods allow for specialization of LMs without prompting, but require retraining the model for each task. To avoid this trade-off entirely, we present _gisting_, which trains an LM to compress prompts into smaller sets of "gist" tokens which can be cached and reused for compute efficiency. Gist models can be trained with no additional cost over standard instruction finetuning by simply modifying Transformer attention masks to encourage prompt compression. On decoder (LLaMA-7B) and encoder-decoder (FLAN-T5-XXL) LMs, gisting enables up to 26x compression of prompts, resulting in up to 40% FLOPs reductions, 4.2% wall time speedups, and storage savings, all with minimal loss in output quality.

## 1 Introduction

Consider the prompt of a Transformer [34] language model (LM) like ChatGPT:1

Footnote 1: reddit.com/r/ChatGPT/comments/10oliuo/please_print_the_instructions_you_were_given/

You are ChatGPT, a large language model trained by OpenAI. You answer as concisely as possible for each response (e.g. don't be verbose). It is very important that you answer as concisely as possible, so please remember this. If you are generating a list, do not have too many items. Keep the number of items short. Knowledge cutoff: 2021-09 Current date: <TODAY>

With millions of queries a day, an unoptimized ChatGPT would encode this prompt over and over with a self-attention mechanism whose time and memory complexity is quadratic in the length of the input. Caching the Transformer activations of the prompt can prevent some recomputation, yet this strategy still incurs memory and storage costs as the number of cached prompts grows. At large scales, even small reductions in prompt length could lead to substantial compute, memory, and storage savings over time, while also letting users fit more content into an LM's limited context window.

How might we reduce the cost of this prompt? One typical approach is to finetune or distill [1, 30] the model to behave similarly to the original model without the prompt, perhaps with parameter-efficient adaptation methods [15, 16, 19]. Yet a fundamental drawback of this approach is that it requires retraining the model for each new prompt (Figure 1, bottom left).

Instead, we propose **gisting** (Figure 1, top right), which compresses arbitrary prompts into a smaller set of Transformer activations on top of virtual "gist" tokens, _a la_ prefix-tuning [19]. But where prefix-tuning requires learning prefixes via gradient descent for each task, gisting adopts a meta-learning approach, where we simply predict the gist prefixes zero-shot given only the prompt, allowing for generalization to unseen instructions without any additional training. Since gist tokens are much shorter than the full prompt, gisting allows arbitrary prompts to be compressed, cached, and reused for compute efficiency.

In this paper, we further propose a very simple way to learn a gist model: doing instruction tuning [38] with gist tokens inserted after the prompt, and a modified attention mask preventing tokens _after_ the gist tokens from attending to tokens _before_ the gist tokens. This allows a model to learn prompt compression and instruction following at the same time, with no additional training cost.

On decoder-only (LLaMA-7B) and encoder-decoder (FLAN-TS-XXL) LMs, gisting achieves prompt compression rates of up to **26x**, while maintaining output quality similar to the original models in human evaluations. This results in up to 40% FLOPs reduction and 4.2% latency speedups during inference, with greatly decreased storage costs compared to traditional prompt caching approaches.

## 2 Gisting

We will first describe gisting in the context of instruction finetuning [38]. We have an instruction-following dataset \(\mathcal{D}=\{(t_{i},x_{i},y_{i})\}_{i=1}^{N}\), where \(t\) is a task encoded with a natural language prompt (e.g. Translate this to French, \(x\) is an (optional) input for the task (e.g. The cat), and \(y\) is the desired output (e.g. Le chat). Given a (usually pretrained) LM, the aim of instruction finetuning is to learn a distribution \(p_{\text{LM}}(y\mid t,x)\), typically by concatenating \(t\) and \(x\), then having the LM autoregressively predict \(y\). At inference time, we can _prompt_ the model with a novel task \(t\) and input \(x\), decoding from the model to obtain its prediction.

However, this pattern of concatenating \(t\) and \(x\) has drawbacks: Transformer LMs have limited context windows, bounded either by architecture or memory limits. Furthermore, given that attention scales quadratically in the length of the input, long prompts \(t\), especially those that are repeatedly reused, are computationally inefficient. What options do we have to reduce the cost of prompting?

One simple option is to finetune the LM for a _specific_ task \(t\). That is, given \(\mathcal{D}^{t}=\{(x_{i},y_{i})\}_{i=1}^{N^{t}}\), the dataset containing input/output examples only under task \(t\), we can learn a specialized LM \(p_{\text{LM}}^{t}(y\mid x)\) which is faster because it does not condition on \(t\). Parameter-efficient finetuning methods such as prefix-/prompt-tuning [18; 19] or adapters [15; 16] promise to do so at a fraction of the cost of full finetuning, and newer methods like HyperTuning [25] eliminate gradient descent entirely, instead predicting the parameters of the specialized model directly from \(\mathcal{D}^{t}\). Yet problems with these methods still remain: we must store at least a subset of model weights for each task, and more importantly, for each task \(t\), we must collect a corresponding dataset of input/output pairs \(\mathcal{D}^{t}\) to adapt the model.

Gisting is a different approach that amortizes both (1) the inference-time cost of prompting \(p_{\text{LM}}\) with \(t\) and (2) the train-time cost of learning a new \(p_{\text{LM}}^{t}\) for each \(t\). The idea is to learn a _compressed_ version of \(t\), \(G(t)\), such that inference from \(p_{\mathcal{G}}(y\mid G(t),x)\) is faster than \(p_{\text{LM}}(y\mid t,x)\). In LM terms, \(G(t)\) will be the key/value activations on top a set of _gist tokens_, smaller than the number of tokens in \(t\), yet still inducing similar behavior from the LM. Also known as a Transformer _prefix_[19], \(G(t)\) can then be cached and reused for compute efficiency. Crucially, we expect \(G\) to generalize to unseen tasks: given a new task \(t\), we can predict and use the gist activations \(G(t)\)_without any additional training_.

### A Context Distillation Perspective

An alternative way to view gisting is through the lens of distillation of an already instruction-tuned LM \(p_{\text{LM}}(y\mid t,x)\). Askell et al. [1] and Snell et al. [30] define _context distillation_ as the process of finetuning a new LM \(p_{\text{CD}}^{t}\) to mimic the original LM without the prompt ("context") \(t\), via the loss

\[\mathcal{L}_{\text{CD}}(p_{\text{CD}}^{t},\,t)=\mathbb{E}_{x}\left[D_{\text{ KL}}(p_{\text{LM}}(y\mid t,x)\parallel p_{\text{CD}}^{t}(y\mid x))\right]. \tag{1}\]

Figure 1: **Prompting** retains the multitask capabilities of LMs, but is inefficient. **Finetuning/distillation** is more efficient, but requires training a model for each task. **Gisting** compresses prompts into activations on top of “gist tokens”, saving compute and generalizing to novel tasks at test time. Each vertical rectangle represents a stack of Transformer activations.

The insight to be gained from this perspective is that we do not need any external data \(\mathcal{D}\): this KL objective can be approximated by finetuning \(p_{\text{CD}}^{t}\) on a synthetic sampled dataset \(\hat{\mathcal{D}}^{t}=\{(\hat{x}_{i},\hat{y}_{i})\}\) where \((\hat{x}_{i},\hat{y}_{i})\sim p_{\text{LM}}(\cdot\mid t)\). This is precisely the approach taken by recent work [1; 7; 30], including Wingate et al. [40], who notably learn to compress a single discrete prompt into a soft prompt via gradient descent, similar to this paper.

However, we differ from this prior work in that we are not interested in distilling just a single task, but in amortizing the cost of distillation across a _distribution_ of tasks \(T\). That is, given a task \(t\sim T\), instead of obtaining the distilled model via gradient descent, we use \(G\) to simply _predict_ the gist tokens (\(\approx\) parameters) of the distilled model, in the style of HyperNetworks [13] and HyperTuning [25]. Our "meta" distillation objective is thus (with changes highlighted in blue):

\[\mathcal{L}_{\text{\tiny{G}}}(p_{\text{\tiny{G}}},\mathcal{T})=\mathbb{E}_{t \sim T,x}\left[D_{\text{KL}}(p_{\text{LM}}(y\mid t,x)\parallel p_{\text{\tiny {G}}}(y\mid G(t),x))\right]. \tag{2}\]

In the experiments we describe below, we train on synthetic instruction-following data sampled from instruction-tuned variants of GPT-3 [23; 3]. Thus, these experiments can indeed be seen as a form of context distillation for the GPT-3 series models.

## 3 Learning Gisting by Masking

Having just described the general framework of gisting, here we will explore an extremely simple way of learning such a model: using the LM itself as the gist predictor \(G\). This not only leverages the pre-existing knowledge in the LM, but also allows us to learn gisting by simply doing standard instruction finetuning while modifying the Transformer attention masks to enforce prompt compression. This means that gisting incurs _no_ additional training cost on top of standard instruction finetuning!

Specifically, we add a _single_ gist token \(g\) to the model vocabulary and embedding matrix. Then, given a (task, input) pair \((t,x)\), we concatenate \(t\) and \(x\) with a set of \(k\) copies of \(g\) in between: \((t,g_{1},\dots,g_{k},x)\), e.g. Translate French: <G1> <G2> The cat.2 The model is then restricted such that input tokens _after_ the gist tokens cannot attend to any of the prompt tokens _before_ the gist tokens (but they _can_ attend to the gist tokens). This forces the model to compress the prompt information into the gist prefix, since the input \(x\) (and output \(y\)) cannot attend to the prompt \(t\).

Footnote 2: Again, the gist token is the same from \(g_{1}\) to \(g_{k}\); what changes is the activations on top of each token.

Figure 2 illustrates the required changes. For **decoder-only** LMs such as GPT-3 [3] or LLaMA [33] that normally admit an autoregressive, causal attention mask, we simply mask out the lower-left corner of the triangle (Figure 1(a)). For **encoder-decoder** LMs (e.g. T5; [28]) with a bidirectional encoder followed by an autoregressive decoder, two changes are needed (Figure 1(b)). First, in the encoder, which normally has no masking, we prevent the input \(x\) from attending to the prompt \(t\). But

Figure 2: **Gist Masking. Attention mask modifications for (a) decoder-only and (b) encoder-decoder Transformer LMs to encourage prompt compression into gist tokens <G1> <G2>. In these tables, cell \((r,c)\) shows whether token \(r\) can attend to token \(c\) during self- or cross-attention.**

we must also prevent the prompt \(t\) and gist tokens \(g_{i}\) from attending to the input \(x\), since otherwise the encoder learns different representations depending on the input. Finally, the decoder operates as normal, except during cross-attention, we prevent the decoder from attending to the prompt \(t\).

Overall, these masking changes are extremely simple and can be implemented in roughly 10 source lines of code. See Appendix A for a sample PyTorch implementation which can be used as a drop-in replacement for attention masking in deep learning libraries such as Hugging Face Transformers [41].

## 4 Experiments

### Data

A dataset with a large variety of tasks (prompts) is crucial to learn gist models that can generalize. To obtain the largest possible set of tasks for instruction finetuning, we create a dataset called Alpaca+, which combines the Self-Instruct [36] and Stanford Alpaca [31] instruction tuning datasets, each consisting of \((t,x,y)\) tuples sampled from OpenAI's text-davinci-001 and text-davinci-003 variants of GPT-3, respectively. In total, Alpaca+ has 130,321 examples, with 104,664 unique tasks \(t\), 48,530 unique inputs \(x\), and anywhere from 0-5 inputs per task (0.64 on average).

Note that ~59% of tasks in Alpaca+ have no inputs (e.g. Write me a poem about frogs), in which case we simply omit the input \(x\). While it is less interesting to cache such prompts since they are not input-dependent, they still serve as valuable training signal for learning prompt compression. Overall, while Alpaca+ is noisy and imperfect, Wang et al. [36] and Taori et al. [31] nevertheless show that models trained on such data achieve comparable performance to the original models from which the data is sampled, making this a promising testbed for studying gisting.

From Alpaca+ we hold out 3 validation splits: 1000 **Seen** prompts (with unseen, non-empty inputs); 1000 **Unseen** prompts (with non-empty inputs); and the 252 hand-written **Human** prompts and completions used in Wang et al. [36], of which 83% have non-empty inputs. The latter two splits test generalization to unseen instructions, with the **Human** split posing a stronger out-of-distribution (OOD) challenge: the average training prompt has ~20 tokens, compared to ~26 in the human split.

### Models

To demonstrate gisting across multiple Transformer LM architectures, we experiment with LLaMA-7B [33], a decoder-only GPT-style model with ~7B parameters, and FLAN-T5-XXL [8], an encoder-decoder T5 model [28] with 11B parameters. For each of these models, we train models with a varying number of gist tokens \(k\in\{1,2,5,10\}\), using the modified attention masks described in Section 3. To assess how the model is learning prompt compression, we calibrate performance against upper- and lower-bound baselines and a simple discrete compression strategy:

Positive Control.As an upper bound on performance, we train a model with a single gist token, but without any modifications to the attention mask. This is akin to doing standard instruction finetuning.

Negative Control.As a lower bound on performance, we train a model without access to the task \(t\). This is similar to a "random gist token" baseline, which allows us to measure how the model would do if it failed to compress _any_ information into the gist prefix.

Discrete Compression with TF-IDF.An alternative approach to compression is simply using fewer discrete tokens to express the same task. Achieving compression rates similar to gisting, however, requires compression far beyond any threshold of fluency. Nevertheless, as a baseline, we compute TF-IDF statistics over the set of instructions in the Alpaca+ training set to extract the most relevant keyword in each instruction. Some examples from the training set include (see Appendix G for more):

Write a letter to your boss asking for an increase in salary \(\rightarrow\) salary Given two integers, find their average \(\rightarrow\) average We then replace each instruction in Alpaca+ with the first subword token from each keyword, resulting in compression rates equivalent to a model trained with a single gist token. Similarly to the positive control, we do standard instruction finetuning over Alpaca+ with these compressed instructions.

For full training, data, and compute details, and a link to code, see Appendix B.

### Evaluation

Our evaluation uses a combination of automated metrics and AI- and human-assisted evaluation:

ROUGE-L.We first use ROUGE-L, a simple lexical overlap statistic [20], used in previous open-ended instruction finetuning work [37; 38]. The text-davinci-{001,003} completions are used as references, except for the Human split, where we use the gold-standard human reference.

ChatGPT.Next, we use ChatGPT-3.5 [22] to compare the outputs of our models to the positive control. While this is an imperfect metric, it allows for much faster and cheaper evaluation than human experiments, with an arguably more meaningful semantic signal than ROUGE-L. Recent work has found that ChatGPT can be used for text annotation and evaluation [12; 17; 35] with near-human performance, and similar model-based evaluations have been conducted with recent LMs [5; 11; 31].

Specifically, given a task \(t\), input \(x\), and outputs from two models \((y_{1},y_{2})\) identified only as Assistants A and B, ChatGPT was asked to choose which assistant response is better, explaining its reasoning in Chain-of-Thought fashion [39]. If the models produced the same output, or were equally bad, ChatGPT was allowed to call a tie. We gave examples of desired outputs in ChatGPT's prompt, and randomized the order of presentation between the models for each query to avoid order effects. The full prompt given to ChatGPT and evaluation details are in Appendix C. Using these outputs, we measure the win rate of a model against the positive control: a win rate of 50% indicates that the model is of comparable quality to a model that does no prompt compression.

Human eval.Finally, after prototyping with ChatGPT, we select the best gist compression models and do a Human evaluation on a random subset of 100 of the 252 examples in the Human validation split. For each of the 100 examples, we recruited 3 US or UK-based, English-fluent annotators from Prolific, and asked them to rate model outputs in the same style as the ChatGPT evaluation above (see Appendix D for full details, including the annotation interface). The only difference is that human participants were allowed to select "I Don't Know" in cases where they had inadequate domain knowledge to accurately judge the responses, e.g. if the question was a coding question; we drop these responses (~10%) during analysis. With this human evaluation, we are not only interested in evaluating our final models, but also validating whether ChatGPT can be used as a reliable replacement for human annotation on this task.

## 5 Results

ROUGE-L and ChatGPT evaluations for LLaMA-7B and FLAN-T5-XXL, with varying numbers of gist tokens, are shown in Figure 3. Models were generally insensitive to the number of gist tokens \(k\): compressing prompts into a single token prefix did not substantially underperform larger prefixes. In fact, having too many gist tokens hurts performance in some cases (e.g. LLaMA-7B, 10 gist tokens), perhaps because the increased capacity enables overfitting to the training distribution. Thus, we use the single gist token models for the rest of the experiments in the paper, and report the exact numbers for the single-token models, with the positive, negative, and TF-IDF baselines, in Table 1.

On **Seen** instructions, gist models attain near-identical ROUGE and ChatGPT performance as their positive control models (48.6% and 50.8% win rates for LLaMA-7B and FLAN-T5-XXL, respectively). But we are most interested in generalization to unseen tasks, as measured by the other two splits. On **Unseen** prompts within the Alpaca+ distribution, we again see competitive performance: 49.7% (LLaMA) and 46.2% (FLAN-T5) win rates against the positive controls. It is on the most challenging OOD **Human** split where we see slight drops in win rate to 45.8% (LLaMA) and 42.5% (FLAN-T5), though these numbers are still quite competitive with the positive control. Finally, gist compression is vastly superior to discrete compression; the TF-IDF models in Table 1 only marginally outperform the negative control models across the board.

Table 2 shows the human evaluation results on the Human validation split, comparing the single gist token models to the positive control. Overall, human annotators agree with ChatGPT, with average win rates of 52.3% (vs. 48.0%) for LLaMA-7B and 40.6% (vs. 42.0%) for FLAN-T5-XXL. Importantly, this agreement persists at the level of individual responses. The average pairwise Cohen's \(\kappa\) among human annotators is.24 for LLaMA-7B and.33 for FLAN-T5-XXL. Because humans will often arbitrarily choose one response over another even for samples of equal quality, these numbers

\begin{table}
\begin{tabular}{l l l l l l l l} \hline \hline  & & \multicolumn{2}{c}{**Seen**} & \multicolumn{2}{c}{**Unseen**} & \multicolumn{2}{c}{**Human**} \\
**Model** & & ROUGE-L & ChatGPT \% & ROUGE-L & ChatGPT \% & ROUGE-L & ChatGPT \% \\ \hline LLaMA- & Pos & 58.0 (100) & 50.0 (100) & 48.1 (100) & 50.0 (100) & 27.0 (100) & 50.0 (100) \\
7B & Gist & 57.8 (99.2) & 48.6 (92.4) & 46.6 (91.0) & 49.7 (98.8) & 23.9 (75.4) & 45.8 (84.9) \\  & TF-IDF & 38.1 (24.5) & 34.5 (16.2) & 34.0 (15.6) & 29.3 (15.9) & 16.5 (16.7) & 24.6 (8.6) \\  & Neg & 31.5 (0) & 31.5 (0) & 31.4 (0) & 25.4 (0) & 14.4 (0) & 22.2 (0) \\ \hline FLAN- & Pos & 50.6 (100) & 50.0 (100) & 45.7 (100) & 50.0 (100) & 23.9 (100) & 50.0 (100) \\ T5-XXL & Gist & 48.9 (93.2) & 50.8 (103.9) & 43.8 (88.6) & 46.2 (84.4) & 21.7 (80.9) & 42.5 (63.2) \\  & TF-IDF & 32.0 (25.9) & 35.9 (30.5) & 34.3 (31.3) & 31.0 (22.1) & 13.5 (9.6) & 28.4 (-5.9) \\  & Neg & 25.5 (0) & 29.7 (0) & 29.1 (0) & 25.6 (0) & 12.4 (0) & 29.6 (0) \\ \hline \hline \end{tabular}
\end{table}
Table 1: **Results for single gist tokens.** ROUGE-L and ChatGPT scores for Gist, TF-IDF, and positive/negative controls. Parentheses are scores normalized between positive/negative controls.

Figure 3: **Varying the number of gist tokens**. ROUGE-L and ChatGPT scores for (a) **LLaMA-7B** and (b) **FLAN-T5-XXL** for different gist tokens. Dashed lines indicate positive and negative control performance. Error bars are 95% exact binomial confidence intervals, splitting ties equally between models [10] and rounding down in favor of the positive control in case of an odd number of ties. Compression factors are calculated by computing the average token length of the validation split and dividing by the number of gist tokens.

are fairly low; however, ChatGPT shows similar levels of agreement, with average \(\kappa\) across each of the 3 human annotators at.29 for both models. These results, paired with the similar overall win rates, show that using ChatGPT is similar to simply recruiting an additional human annotator, and corroborates the broader results in Figure 3. See Appendix D for more human evaluation results, including a breakdown of agreement across annotators, and Appendix G for examples of instructions, model outputs, and human/ChatGPT judgments in the Human validation split.

Since our aim is having the gist models mimic the original models, one might ask how often the gist model is identical to the positive control. Figure A.3 in Appendix E shows how often this happens: for **Seen** tasks (but unseen inputs), the gist model outputs exactly match the positive control nearly 50% of the time. This drops to ~20-25% for **Unseen** tasks and ~10% for the OOD **Human** tasks.

Overall, our results show that gist models can reliably compress prompts, even to some degree those that lie outside the training distribution, especially for decoder-only LMs (LLaMA). Encoder-decoder LMs like FLAN-T5 show slightly worse OOD performance; one possible reason is that gist masking inhibits the bidirectional attention flow in the encoder, which is more challenging to adjust to than simply limiting the history of a decoder LM. Future work will be needed to investigate this hypothesis.

### Failure cases

While the responses from the gist models are often indistinguishable from the positive control, we nevertheless do see failure cases where the gist token models do not perfectly compress the prompt. We give two examples where this occurs in the Human validation split: first, when the instruction contains **specific details**, e.g. phrases that must be copied verbatim into the output (in the snippets below, the names of Assistants A/B are hidden to ChatGPT):

Instruction: Give the news title a category. Pick a category from the list of News & Buzz, Travel, Style, Arts & Culture, Politics, Tech, and Science & Health. <G1> Input: The #Banksy Exhibit in Cambridge, MA is absolutely terrific. Assistant A (FLAN-T5-XML gist): Culture Assistant B (pos control): Arts & Culture

**ChatGPT choice**: B

**ChatGPT reason**: The news title is about an art exhibit, so Assistant B's response of 'Arts & Culture' is more accurate and specific than Assistant A's 'Culture'.

We also see more cases of **runaway generations** that do not occur with the positive control model:

Instruction: Provide an example of how a table of contents can be generated automatically in a LATEX document. Make sure your example contains the necessary commands. <G1> Assistant A (pos control): {tableofcontents }chapter{Introduction} }chapter{Theory} {Experiments} }chapter{Conclusion} Assistant B (LLaMA-7B gist): {tableofcontents }chapter{Introduction} }section{The Basics} }subsection{The Basics} }paragraph{The Basics} }subparagraph{The Basics} } {The Basics} }Subparagraph{The Basics} } {The Basics} }Subparagraph{The Basics} }
**ChatGPT choice**: A

**ChatGPT reason**: Assistant A provides a simple and correct example of generating a table of contents in LaTeX. Assistant B's example is unnecessarily long and does not follow the standard structure of a table of contents.

While it is unclear why only the gist models exhibit this behavior, these issues can likely be mitigated with more careful sampling techniques.

\begin{table}
\begin{tabular}{l|r r r r r r|r r} \hline \hline  & & \multicolumn{4}{c|}{**Gist Win \% over Pos**} & \multicolumn{2}{c}{**Agreement (Cohen’s \(\kappa\))**} \\
**Model** & H1 & H2 & H3 & **Human (H1–H3)** & **ChatGPT** & **Human** & **ChatGPT** \\ \hline LLaMA-7B & 51.1 & 44.5 & 59.8 & 52.3 (46.1, 58.4) & 48.0 (38.0, 58.2) &.24 &.29 \\ FLAN-T5-XXL & 43.0 & 41.9 & 37.2 & 40.6 (34.6, 46.8) & 42.0 (32.2, 52.3) &.33 &.29 \\ \hline \hline \end{tabular}
\end{table}
Table 2: **Human evaluation results.** Win rate and inter-annotator agreement of single token gist models over positive control according to 3 Human annotators (H1–H3), their average, and ChatGPT (95% confidence intervals in parentheses), for 100 out of 252 examples in the Human validation split.

## 6 Compute, Memory, and Storage Efficiency

Finally, we return to one of the central motivations of this paper: what kind of efficiency gains does gisting enable? To answer this question, we compare the compute requirements (CUDA wall time and FLOPs) during inference with the single-token gist models using different caching strategies:

1. **No caching**: just encoding the full prompt \(t\).
2. **Instruction caching**: caching the activations of the uncompressed instruction \(t\) (keys and values for all layers) into what is called the **KV cache**. This is the most common caching behavior for Transformer inference [4; 26] and is supported in libraries like Hugging Face Transformers [41]. However, it is only applicable to _decoder-only_ models, since in models with bidirectional encoders like T5, the instruction representations \(t\) depend on the input \(x\).
3. **Gist caching**: Compressing the prompt into the gist prefix \(G(t)\).

Table 3 displays the results of profiling a single forward pass through the model (i.e. one step of decoding with a single input token) with PyTorch [24] 2.0, averaged across the 252 Human instructions. Gist caching improves significantly over unoptimized models, with 40% FLOPs savings and 4-7% lower wall time for both models. Note that at these (relatively) small scales, the wall time improvements are smaller than the FLOPs reductions because much of the inference latency is caused by moving tensors from high-bandwidth memory (HBM) to the chip compute cores, i.e. what Pope et al. [26] call the "memory time". Larger sequence lengths and batch sizes will lead to additional speedups, as the overall latency becomes dominated by the actual matrix computations.

For LLaMA-7B, the picture is more nuanced when compared to caching the full instruction. Compute improvements of gist caching are smaller: a negligible decrease in FLOPs (0.11%) and a modest 1% speedup in wall time. This is because the FLOPs required for a Transformer forward pass is dominated by processing of the new input tokens, rather than self-attention with the KV cache. For example, a forward pass through LLaMA-7B with a single input token and a _2000-length_ KV cache is only ~10% more expensive than the same forward pass with no KV cache--see Appendix F for more details. Nevertheless, this small decrease in FLOPs leads to a disproportionate decrease in wall time (1%), likely because the self-attention computations are slower relative to their FLOPs contribution.

At large scales and with heavily reused prompts, a 1% latency speedup can still accumulate into significant cost savings over time. More importantly, however, there are key benefits of gist caching over instruction caching besides latency: compressing 26 tokens into 1 gives more space in the input context window, which is bounded by absolute position embeddings or GPU VRAM. For example, for LLaMA-7B, each token in the KV cache requires 1.05 MB storage.3 While the total contribution of the KV cache relative to the memory needed for LLaMA-7B inference is negligible at the prompt lengths we tested, an increasingly common scenario is developers caching many prompts across a large number of users, where storage costs quickly add up. In these scenarios, gisting allows caching of up to **26x** more prompts than full instruction caching, using the same amount of storage!

Footnote 3: 4 (fp32 bytes) \(\times\) 2 (keys+values) \(\times\) 32 (num layers) \(\times\) 32 (num attn heads) \(\times\) 128 (head dim) = 1.05 MB.

\begin{table}
\begin{tabular}{l l|r r r r|r} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Metric**} & \multicolumn{3}{c|}{**Caching Strategy**} & \multicolumn{2}{c}{Absolute/**Relative \(\Delta\)**} \\  & & **None** & **Instruction\({}^{\text{i}}\)** & **Gist\({}^{\text{ii}}\)** & **vs. None** & **vs. Instruction** \\ \hline LLaMA-7B & CUDA & 23.4 \(\pm\) 6.88 & 22.1 \(\pm\) 6.58 & **21.8 \(\pm\) 6.55** & 1.60 (1.29, 1.90) & 2.21 (.140,.302) \\  & time (ms) \(\downarrow\) & & & & 6.8\% (5.5, 8.1) & 1.0\% (.63, 1.4) \\  & GFLOPs \(\downarrow\) & 915 \(\pm\) 936 & 553 \(\pm\) 900 & **552 \(\pm\) 899** & 362 (337, 387) &.607 (.448,.766) \\  & & & & & 40\% (37, 42) &.11\% (.08,.14) \\ \hline FLAN-T5-XXL & CUDA & 31.0 \(\pm\) 5.31 & N/A & **29.7 \(\pm\) 5.07** & 1.30 (1.10, 1.51) & N/A \\  & time (ms) \(\downarrow\) & & & & 4.2\% (3.6, 4.9) & \\  & GFLOPs \(\downarrow\) & 716 \(\pm\) 717 & N/A & **427 \(\pm\) 684** & 289 (268, 310) & N/A \\  & & & & & 40\% (37, 43) & \\ \hline \hline \end{tabular}

\({}^{\text{i}}\)Average KV Cache Length = 26 \({}^{\text{ii}}\)Average KV Cache Length = 1

\end{table}
Table 3: **Gist efficiency improvements**. For different caching strategies (**None**, **Instruction**, **Gist**), we record CUDA wall time and GFLOPs (\(\pm\) std dev). Then we report the absolute/relative improvement of **Gist Caching** over these alternative strategies, with 95% confidence intervals in parentheses.

Additional Related Work

Gisting builds upon past work in (parameter-efficient) instruction finetuning and context distillation, as discussed in Section 2. Here we will outline some additional connections to related work:

Adapting LMs without Backprop.As mentioned in Section 2, listing can be viewed as a way to adapt LMs _without_ gradient descent, by predicting the the prefix (\(\approx\) parameters) of the adapted model. Similarly, HyperTuning [25] predicts the prefix of a model for a task using (input, output) pairs for that task. If HyperTuning is a "few-shot" adaptation method, predicting the prefix from few-shot examples, then gisting can be seen as a "zero-shot" version, predicting the prefix from the language instruction alone. Gisting also has the additional benefit over HyperTuning of being conceptually simpler: instead of training a separate LM to predict the prefix, the LM itself is used as the HyperNetwork [13], with only a tiny change to the attention masks needed for prompt compression.

Compression and memory in transformers.The idea of "compressing" prompts is closely related to previous attempts at storing past representations to improve memory and long-range sequence modeling in Transformers [9; 21; 27; 42; 43]. In particular, the Compressive Transformer [27] compresses transformer activations into a smaller _compressed memory_ using a learned convolutional operator. Gisting can be seen as a variant of the Compressive Transformer with 3 key differences. First, the compression function is not a separately learned function, but the LM's own self-attention mechanism, controlled by an input-dependent gist token. Second, the compression function is learned jointly with instruction finetuning via the standard language modeling loss, not a specialized auxiliary reconstruction loss as in [27]. Finally, our task of interest is not long-range sequence modeling, but caching and reusing instruction following prompts for efficiency reasons.

Sparse attention mechanisms.By restricting attention masks, gisting draws inspiration from efficient/sparse attention methods in Transformers (see [32] for review). For example, some sliding window attention mechanisms [2; 6] may remove the need to keep the entire KV cache around during decoding, but these more general methods are not optimized for caching arbitrary parts of the input sequence of varying length, which prompt compression demands. In light of this, gisting can be viewed as an input-dependent sparse attention mechanism specifically aimed at improving efficiency of the prompting workflow now commonly used in LMs.

## 8 Discussion and Limitations

In this paper we presented gisting, a framework for prompt compression in LMs, and a simple way of implementing gist models by modifying Transformer attention masks that incurs no additional cost over standard instruction finetuning. Gisting can be seen either as a modified form of instruction finetuning or a method for (meta-)context distillation of an LM. Gist models can compress unseen OOD prompts up to 26x while maintaining output quality, resulting in up to 40% FLOPs reduction and 4.2% wall clock speedups over unoptimized models, and enabling new methods for prompt caching in encoder-decoder models. While wall-time improvements for decoder-only LMs are smaller, gisting nevertheless enables caching 1 _order of magnitude_ (26x) more prompts relative to full instructions.

Gisting is a promising method for improving LM efficiency, but carries some limitations. While gisting seems to succeed in capturing the "gist" of instructions (hence the name), achieving such compression necessarily results in some loss of nuance of the original instruction; Section 5.1 illustrates a concrete failure case we observed. Since the behavior of LMs on edge cases is already not well understood, it is _especially_ important for practitioners to carefully evaluate whether the compute/accuracy tradeoff of gisting is sufficiently safe and robust for their use case, _before_ deployment.

Nevertheless, we believe gisting opens several interesting directions for future work. First, the masking method presented here can be easily integrated into existing instruction finetuning workflows, but another exciting approach is to retrofit an existing, frozen LM by training a _smaller_ model to compress prompts, if finetuning the larger LM is inconvenient. Second, the largest efficiency gains from gisting will result from compressing very long prompts, for example \(k\)-shot prompts for large \(k\) that may even exceed a single context window. Finally, compression performance can likely be improved through "gist pretraining": first learning to compress arbitrary spans of natural language, before then learning prompt compression. Such objectives could be devised by inserting gist tokens into other pretraining objectives, perhaps during language modeling or T5's span corruption objective.

## Acknowledgments and Disclosure of Funding

We thank the Stanford Alpaca team, especially Xuechen Li, with assistance with Alpaca data and finetuning, and Gabriel Poesia for the codebase used to collect human evaluations. Additional thanks to Xiuyu Li for help debugging and ensuring reproducibility of the open-source codebase. JM is supported by the Open Philanthropy AI Fellowship and XLL is supported by the Stanford Graduate Fellowship.

## References

* [1] A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a laboratory for alignment. _arXiv preprint arXiv:2112.00861_, 2021.
* [2] I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. _arXiv:2004.05150_, 2020.
* [3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. _Advances in Neural Information Processing Systems_, 33:1877-1901, 2020.
* [4] C. Chen. Transformer inference arithmetic. [https://kip.ly/blog/transformer-inference-arithmetic/](https://kip.ly/blog/transformer-inference-arithmetic/), 2022.
* [5] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality. [https://vicuna.lmsys.org/](https://vicuna.lmsys.org/), 2023.
* [6] R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers. _arXiv preprint arXiv:1904.10509_, 2019.
* [7] E. Choi, Y. Jo, J. Jang, and M. Seo. Prompt injection: Parameterization of fixed inputs. _arXiv preprint arXiv:2206.11349_, 2022.
* [8] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. _arXiv preprint arXiv:2210.11416_, 2022.
* [9] Z. Dai, Z. Yang, Y. Yang, J. G. Carbonell, Q. Le, and R. Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2978-2988, 2019.
* [10] W. J. Dixon and F. J. Massey Jr. _Introduction to statistical analysis_. McGraw-Hill, 1951.
* [11] X. Geng, A. Gudibande, H. Liu, E. Wallace, P. Abbeel, S. Levine, and D. Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL [https://bair.berkeley.edu/blog/2023/04/03/koala/](https://bair.berkeley.edu/blog/2023/04/03/koala/).
* [12] F. Gilardi, M. Alizadeh, and M. Kubli. ChatGPT outperforms crowd-workers for text-annotation tasks. _arXiv preprint arXiv:2303.15056_, 2023.
* [13] D. Ha, A. Dai, and Q. V. Le. HyperNetworks. In _International Conference on Learning Representations_, 2017.
* [14] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. _arXiv preprint arXiv:2203.15556_, 2022.
* [15] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for NLP. In _International Conference on Machine Learning_, pages 2790-2799. PMLR, 2019.

* [16] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In _International Conference on Learning Representations_, 2022.
* [17] F. Huang, H. Kwak, and J. An. Is ChatGPT better than human annotators? Potential and limitations of ChatGPT in explaining implicit hate speech. _arXiv preprint arXiv:2302.07736_, 2023.
* [18] B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 3045-3059, 2021.
* [19] X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 4582-4597, 2021.
* [20] C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In _Text summarization branches out_, pages 74-81, 2004.
* [21] P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, L. Kaiser, and N. Shazeer. Generating Wikipedia by summarizing long sequences. In _International Conference on Learning Representations_, 2018.
* [22] OpenAI. Introducing ChatGPT. [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt), 2022.
* [23] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In _Advances in Neural Information Processing Systems_, pages 27730-27744, 2022.
* [24] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. _Advances in Neural Information Processing Systems_, 32, 2019.
* [25] J. Phang, Y. Mao, P. He, and W. Chen. HyperTuning: Toward adapting large language models without back-propagation. _arXiv preprint arXiv:2211.12485_, 2022.
* [26] R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A. Levskaya, J. Heek, K. Xiao, S. Agrawal, and J. Dean. Efficiently scaling transformer inference. _arXiv preprint arXiv:2211.05102_, 2022.
* [27] J. W. Rae, A. Potapenko, S. M. Jayakumar, C. Hillier, and T. P. Lillicrap. Compressive transformers for long-range sequence modelling. In _International Conference on Learning Representations_, 2020.
* [28] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified Text-to-Text Transformer. _The Journal of Machine Learning Research_, 21(1):5485-5551, 2020.
* [29] J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 3505-3506, 2020.
* [30] C. Snell, D. Klein, and R. Zhong. Learning by distilling context. _arXiv preprint arXiv:2209.15189_, 2022.
* [31] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford Alpaca: An instruction-following LLaMA model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023.
* [32] Y. Tay, M. Dehghani, D. Bahri, and D. Metzler. Efficient transformers: A survey. _ACM Computing Surveys_, 55(6):1-28, 2022.

* Touvron et al. [2023] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Roziere, N. Goyal, E. Hambro, F. Azhar, et al. LLaMA: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_, 2023.
* Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. _Advances in Neural Information Processing Systems_, 30, 2017.
* Wang et al. [2023] J. Wang, Y. Liang, F. Meng, H. Shi, Z. Li, J. Xu, J. Qu, and J. Zhou. Is ChatGPT a good NLG evaluator? A preliminary study. _arXiv preprint arXiv:2303.04048_, 2023.
* Wang et al. [2022] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-Instruct: Aligning language model with self generated instructions. _arXiv preprint arXiv:2212.10560_, 2022.
* Wang et al. [2022] Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, et al. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 5085-5109, 2022.
* Wei et al. [2022] J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. In _International Conference on Learning Representations_, 2022.
* Wei et al. [2022] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. H. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In _Advances in Neural Information Processing Systems_, 2022.
* Wingate et al. [2022] D. Wingate, M. Shoeybi, and T. Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 5621-5634, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. URL [https://aclanthology.org/2022.findings-emnlp.412](https://aclanthology.org/2022.findings-emnlp.412).
* Wolf et al. [2020] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online, Oct. 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URL [https://aclanthology.org/2020.emnlp-demos.6](https://aclanthology.org/2020.emnlp-demos.6).
* Wu et al. [2022] Y. Wu, M. N. Rabe, D. Hutchins, and C. Szegedy. Memorizing transformers. In _International Conference on Learning Representations_, 2022.
* Zhang et al. [2021] H. Zhang, Y. Gong, Y. Shen, W. Li, J. Lv, N. Duan, and W. Chen. Poolingformer: Long document modeling with pooling attention. In _International Conference on Machine Learning_, pages 12437-12446. PMLR, 2021.

Example PyTorch Implementation of Gist Masking

See Listing A.1 for a sample annotated implementation of gist masking. This PyTorch implementation relies on basic NumPy-style tensor operations and can thus be adapted easily to a framework like JAX.

## Appendix B Data, Training, Evaluation, and Compute Details

Code, data, and model checkpoints are available at [https://github.com/jayelm/gisting](https://github.com/jayelm/gisting).

Data.For LLaMA-7B, we used a maximum sequence length of 512 tokens during training and evaluation, except with the Human validation split, where the maximum length was increased to 768 (the Human instructions are longer). Examples longer than this length are truncated from the end. For FLAN-T5-XXL, we set a maximum input length (task \(t\) + input \(x\)) of 128 and a maximum output length of 256, except again for the Human split, where the maximum input and output lengths were both set to 384. For both models, we set a maximum generation length of 512 tokens. These lengths were chosen such that \(<1\%\) of examples across the board were truncated during training and evaluation for both models.

Training.Full hyperparameters for training runs are located in Table A.1. These parameters were adapted from previous published work finetuning LLAMA/FLAN-T5. For LLaMA-7B, parameters are identical to those used in training Alpaca Taori et al. [31]. For FLAN-T5-XXL, parameters are identical to those used in training T\(k\)-Instruct[37], except with a 5e-5 learning rate, as used in the T\(k\)-Instruct GitHub repository,4 rather than the 1e-5 learning rate in the paper.

Footnote 4: [https://github.com/yizhongw/Tk-Instruct/blob/lab6fad/scripts/train_tK_instruct.sh](https://github.com/yizhongw/Tk-Instruct/blob/lab6fad/scripts/train_tK_instruct.sh)

LLaMA-7B was trained for 3000 steps, while FLAN-T5-XXL was trained for 16000 steps. Since there are about 130k examples in Alpaca+, given the batch sizes in Table A.1 this corresponds to about ~3 epochs and ~2 epochs of training, respectively. These numbers, again, are identical to Taori et al. [31] and Wang et al. [36]. We note that the training time is relatively flexible; for example, we did not see substantial gains training beyond 1 epoch for FLAN-T5-XXL.

Evaluation.During evaluation and benchmarking, we simply greedily decoded the most likely sequence. We saw limited gains from beam search with beam size \(B=4\).

Compute.Experiments were run on a cluster machine with 4xA100-SXM4-80GB NVIDIA GPUs, 480GB RAM, and 16 CPUs, using PyTorch 2.0 [24], Hugging Face Transformers [41], and DeepSpeed [29]. Training runs take about ~7 hours to complete for LLaMA-7B and ~25 hours for FLAN-T5-XXL. Benchmarking results were obtained on the same machine, but using just 1 of the A100 GPUs.

importtorch
2
3defreverse_cumsum(x:torch.Tensor)->torch.Tensor:
5"""Cumulativesumfromrighttoleft.
6
7 See [https://github.com/pytorch/pytorch/issues/33520](https://github.com/pytorch/pytorch/issues/33520).
8"""
9returnx+torch.sum(x,dim=-1,keepdims:True)-torch.cumsum(x,dim=-1)
10
11
12defmake_mask_pre_first_gist(inputs:torch.Tensor,gist_token:int,dtype=torch.int64)->torch.Tensor:
13"""Returnsaxhwherealltokenspriortothefirstgisttokenaremaskedout.
14
15Args:
16inputs:aTensorofinputtokenswherethelastdimensionisthesequencelength.
17gist_token:theintegeridofthegisttoken.
18dtype:thedtypeofthesmask,defaultint64.
19Returns:
20Therequestedmask.
21
22return((inputs:=gist_token).cumsum(-1))>=1).type(dtype)
23
24
25defmake_mask_post_last_gist(inputs:torch.Tensor,gist_token:int,dtype=torch.int64)->torch.Tensor:
26"""Returnsaxhwherealltokensafterthelastgisttokenaremaskedout.
27
28Computesthesameasmask_pre_first_gist_token,butreversesthesequencebeforeandafterthecumsum.
29
30Args:
31inputs:aTensorofinputtokenswherethelastdimensionisthesequencelength.
32gist_token:theintegeridofthegisttoken.
33dtype:thedtypeofthesmask,defaultint64.
34Returns:
35Therequestedmask.
36"""
37return(reverse_cumsum(inputs:=gist_token)>=1).type(dtype)
38
39
30defmake_gist_mask(inputs:torch.Tensor,gist_token:int,pad_token:int,dtype=torch.int64)->torch.Tensor:
31"""Createsagistmaskfromsuppliedinputsandgist/padtokens.
32
33Tokensafterthelastgistcannotattodtokenspriortothefirstgist.Additionally,tokensbefore=thelastgistcannotattodtokensafterthelastgist.
34
35Thegistmaskisbroadcastedto40(withasingletom1)forcompatibilitywithmulti-headedattention (wheredim1istheheaddimension).
36
37Args:
38inputs:aTensorofshape(batch_size,seq_len)inputtokens.
39
40gist_token:theintegeridofthegisttoken.
41pad_token:theintegeridofthepadtoken.inputs=pad_tokenaremaskedout.
42dtype:thedtypeofthesmask,defaultint64.
43
44Returns:
45Therequestedmaskofshape(batch_size,1,seq_len,seq_len)
46"""
47#Attentionmaskfortokensbeforethelastgisttoken.
48pre_gist_mask=make_mask_post_last_gist(inputs,gist_token,dtype=torch.bool)[:,None,None]
49#Attentionmaskfortokensafterthelastgisttoken.
50post_gist_mask=make_mask_pre_first_gist(inputs,gist_token,dtype=torch.bool)[:,None,None]
51#Constructtimemasksbypermuttingtotimediension.
52pre_gist_time_mask=pre_gist_mask.permute((0,1,3,2))
53
54mask=torch.where(pre_gist_time_mask,pre_gist_mask,post_gist_mask)
55mask=mask&(inputs!=pad_token)[:,None,None)#Maskoutpadtokens.
56
57returnmask.type(dtype) ```

Listing A.1: **Sample PyTorch implementation of gist masking.**Details of ChatGPT Evaluation

Given a user instruction and two AI assistant responses, your job is to select the response that is more helpful, and give reasons why. Judge responses holistically, paying special attention to whether the response (1) correctly follows the provided instruction and (2) is factually accurate. If both responses are equal, you may call a tie, but please use this response sparingly.

Example 1:

Instruction: Given the name of a city, tell me the name of the state it is in.

Input: Los Angeles

Assistant A: California

Assistant B: Wyoming

Your output should be:

("reason": "Los Angeles is in California. Only Assistant A is correct.", "choice": "A")

Example 2:

Instruction: Give me some examples of fun things to do with the kids on a weekend.

Assistant A: For example, what should I do with my kids on a weekend? What if I'm in a city? What if I'm on the beach?

Assistant B: You could try going to the zoo or a museum. If you want to stay at home, you could play

board games or bake cookies.

Your output should be:

("reason": "Assistant A doesn"t seem to be following the instruction. Assistant B provides helpful

information with examples.", "choice": "B")

Example 3:

Instruction: Write me a python function that prints "Hello, World".

Assistant A: def hello_world(): print("Hello!")

Assistant B: "Hello, World"

Your output should be:

("reason": "Assistant B Just wrote "Hello, World". Assistant A actually wrote a Python function, even

if it doesn't exactly print the right thing, so overall Assistant A is better.", "choice": "A")

Example 4:

Instruction: Translate the following sentence from English to French.

Input: I like cats.

Assistant A: He (Sustan los gatos.

Assistant B:

Your output should be:

("reason": "Both assistants got the language wrong.", "choice": "tie")

Your response should only be in the JSON format above; THERE SHOULD BE NO OTHER CONTENT INCLUDED IN

YOUR RESPONSE. Write the "reason" key before writing the "choice" key, so that you think step-by-step

before making your decision. KEEP YOUR REASONING BRIEF.

Listing A.2: **Full prompt given to ChatGPT for evaluation**. This prompt populates the system field in the ChatGPT API; the actual example to be evaluated is formatted like the examples in the prompt above, then given as the sole input in the user field.

We used the ChatGPT API, specifically the chatgpt-3.5-turbo engine, to run our ChatGPT evaluation experiments over a period of 2 weeks between March 27 and April 7, 2023.

The full prompt given to ChatGPT is located in Listing A.2, and contains 4 examples of desired output from ChatGPT, including preferring factually accurate responses (Example 1), preferring responses that follow the instruction, even if imperfect (Examples 2 and 3), and examples of modelsbeing equally wrong (Examples 4). For the two models under comparison, we randomized the order of presentation of each model as either Assistant A or Assistant B, to avoid order effects.

ChatGPT was instructed to only respond in JSON format, outputting first a reason key followed by a choice key, to encourage chain-of-thought reasoning [39]. On rare occasions (\(<0.25\%\) of the time), ChatGPT would output a response that did not conform to the requested JSON format (e.g. it would just give an unstructured paragraph). In these cases we manually went through and converted these responses to JSON, without altering ChatGPT's reasoning.

In total, we collected -22.5k judgments from ChatGPT for an estimated cost of $29.28. The full outputs for each model across the Alpaca+ validation splits, as well as ChatGPT's responses and choices, are available in the code link above.

## Appendix D Additional Human Evaluation Details and Results

### Experimental Details

For each of the 100 examples randomly selected from the Human validation split, we recruited 3 US or UK-based, English-fluent annotators from Prolific, an online crowdsourcing platform. Experiments were IRB approved under a generic human experiments IRB given to the authors.

The annotation interface given to Prolific crowdworkers is located in Figure A.1. To verify task comprehension, participants were shown two simple examples before the main body of the task (Figure A.2), and were required to answer correctly before proceeding. We compensated participants USD $14.35/hour for an estimated cost (including Prolific fees) of USD $141.64.

### Additional Results

See Table A.2 for a breakdown of Cohen's \(\kappa\) between human annotators and ChatGPT. We used a weighted version of Cohen's \(\kappa\) with linear weights, since the response scale is ordinal (e.g. "tie" is a closer judgment to "pos control win" than "gist win").

## Appendix E Exact Match Results

See Figure A.3 for a plot of exact match rates for the gist and positive control models (as measured by exact string match).

## Appendix F Additional FLOPs details

The FLOPs required for a Transformer formward pass with varying KV cache lengths can be estimated by modifying existing equations to account for self-attention back to the KV cache. As an example, we modify the FLOPs equations used for computing FLOPs in the Chinchilla paper (Appendix F in [14]). Let **seq_len_with_past** = seq_len + kv_cache_len. Then the modified Transformer FLOPs equations are:

**Embeddings**

* \(2\times\text{seq\_len}\times\text{vocab\_size}\times\text{d\_model}\)

## Appendix A Appendix

Figure A.1: **Annotation interface given to Prolific crowdworkers.**

Figure A.2: **Example items given to humans before the start of the task.**

**Attention (Single Layer)**

* _Key, query, and value projections_: \(2\times 3\times\text{seq\_len}\times\text{d\_model}\times\text{(key\_size}\times\text{num\_heads)}\)
* _Key and query logits_: \(2\times\text{seq\_len}\times\text{seq\_len\_with\_past}\times\text{(key\_size} \times\text{num\_heads)}\)
* _Softmax_: \(3\times\text{num\_heads}\times\text{seq\_len}\times\text{seq\_len\_with\_past}\)
* _Softmax @ query reductions_: \(2\times\text{seq\_len}\times\text{seq\_len\_with\_past}\times\text{(key\_size} \times\text{num\_heads)}\)
* _Final linear_: \(2\times\text{seq\_len}\times\text{(key\_size}\times\text{num\_heads)}\times\text{d\_model}\)

**Dense Block**

* \(2\times\text{seq\_len}\times\text{(d\_model}\times\text{fw\_size}+\text{d\_model} \times\text{fw\_size)}\)

**Final Logits**

* \(2\times\text{seq\_len}\times\text{d\_model}\times\text{vocab\_size}\)

**Total Forward Pass FLOPs**

* embeddings + num_layers \(\times\text{(attention\_single\_layer+dense\_block)+final\_logits}\)

It can be seen that only 3 operations in each attention layer depend on the KV cache size, and they take up a relatively insignificant amount of FLOPs. As an illustrative example, Figure A.4 shows the relative FLOPs contributions within a single layer of attention for LLaMA-7B, assuming a _2000-length_ KV cache and a single input token. Operations dependent on the KV cache constitute at most \(\sim\)10% of the total attention layer FLOPs; the rest are used in KQV projections and dense layers for processing the single new input token.

Given a KV cache compression rate of 26, as observed in our Human validation split, the Chinchilla equations predict a relative improvement of Gist caching of 0.12%. This is extremely close to the 0.11% improvement actually observed in Table 3. These results show that optimizing the KV cache size does not actually lead to huge compute speedups during Transformer inference, at least for relatively small prompt lengths. Nevertheless, there are clear memory and storage benefits to be gained from prompt compression, as discussed in Section 6.

## Appendix G Additional Examples of Model Outputs

See Listing A.3 for additional examples of instructions, inputs, model completions (for LLaMA-7B and FLAN-T5-XXL across Pos, Gist, TF-IDF, and Neg conditions), human judgments, and ChatGPT judgments (+reasons). Examples are not cherry-picked; we enumerate the first 10 examples of the Alpaca+ Human validation split.

Figure A.4: **FLOPs for each operation involved in a single layer of self attention with a 2000-length KV cache**, according to the Chinchilla estimates for LLaMA-7B. At most 9.6% of FLOPs can be optimized away by reducing the size of the KV cache.

Figure A.3: **Exact match rates**. Rate at which the 1 token gist models give the same output as the positive control (exact string match). Error bars are 95% exact binomial confidence intervals.

## Example 1

Instruction: Write a good Twitter Bio. Try to cover all the provided main points in your generated bio. <B1> Input: Name: Jane..

**ChatGPF** **reason** (Pos vs dist): Assistant A only mentions Paris and provides more detailed information about the city, while Assistant B mentions several cities but doesn't provide much detail about any of them. Therefore, Assistant A is more helpful. { Assistant A: dist, Assistant B: Pos}

### Example 3

Instruction: Give the news title A category. Pick a category from the list of News & Buzz, Travel, Style, Arts & Culture, Politics, Tech, and Science & Health. <H>

Input: The Shanks Exhibit in Cambridge, NA is absolutely terrific.

Reference Output: Arts & Culture

TF-IMP Word: Du

**LLaMa-7B**

Pos: Arts & Culture

**dist: Arts & Culture

**TF-IMP: No such bias detected.**

Neg: The Banks Exhibit in Cambridge, NA is absolutely fantastic.

**Mean Choice (Pos vs dist): The Tie Tie**

**ChatGPF** **chice** (Pos vs dist): The news title as related to Arts & Culture. (Assistant A: dist, Assistant B: Pos)

**FLAN-T5-XXL**

Pos: Arts & Culture

**Gist: Culture**

**TF-IMP: The Banks Exhibit in Cambridge, NA is absolutely remarkable.

Neg: The Banks Exhibit in Cambridge, NA is remarkable.

**Mean Choice** (Pos vs dist): Pos Pos Pos

**ChatGPF** **chice** (Pos vs dist): Pos

**ChatGPF** **reason** (Pos vs dist): The news title is about an art exhibit, so Assistant B's response of 'Arts & Culture' is more accurate and specific than Assistant A's 'Culture'. (Assistant A: dist, Assistant B: Pos)

### Example 4

Instruction: You will be tested on your knowledge of classic uitticisms and aphorisms by completing the given aphorism. Write the original quote as the answer. <H>

Input: the things are infinite.

**Reference Output:** the universe and human stupidity; and I'm not sure about the universe.

(attributed to Albert Einstein)

**T-IMP word: A**

**LLaMa-7B**

Pos: The number of integers and the number of uitticisms.

**Gist:** "To things are infinite, the universe and human stupidity."

**- Albert Einstein**

**TF-IMP: The number of integers and the number of real numbers.

**Reg:** To things are infinite: the number of integers and the number of primes.

**Mean choice (Pos vs dist): Gist Gist Gist Gist**

**ChatGPF** **chice** (Pos vs dist): Assistant A correctly completes the quote with the original author, while Assistant B's response is not factually accurate. (Assistant A: dist, Assistant B: Pos)

**FLAN-T5-XXL**

Pos: "Time and space are infinite."

**Gist:** "Factions and consequences are infinite."

**TF-IMP: The first sentence is a statement, while the second one is an opinion.

**Reg:** The two things that are infinite are time and space.

**Mean choice (Pos vs dist): Pos Pos Pos

**ChatGPF** **chice** (Pos vs dist): The

**ChatGPF** **reason** (Pos vs dist): Both assistants did not provide the correct completion of the given aphorism. The original quote is'Two things are infinite: the universe and human stupidity; and I'm not sure about the universe.' Therefore, neither assistant is helpful. (Assistant A: dist, Assistant B: Pos)

### Example 5

Instruction: Write the HMR syntax for the required action. <H>

Input: adding an image to the page

Reference Output: Gimg src="url" alt="alternatetext">

TF-IMP Word: syntax

**LLaMa-7B**

Pos: Gimg src="image.jpg"

Gist: compounds

**TF-IMP:** cing src="image.jpg"

Neg: Gimg src="image.jpg"

Neg: Gimg src="image.jpg"

Neg: IRR Pos

**ChatGPF** **chice** (Pos vs dist): Pos

**ChatGPF** **reason** (Pos vs dist): Assistant A provides the correct HMR syntax for adding an image to the page, while Assistant B doesn't provide any output. (Assistant A: Pos, Assistant B: Gist)

**FLAN-T5-XXL**

Pos: Gimg src="image.jpg"

Gist: Gimg src="image.jpg"

Neg: Adding an image to the page.

**Mean choice** (Pos vs dist): Tie Gist

**ChatGPF** **chice** (Pos vs dist): The

**ChatGPF** **reason** (Pos vs dist): Both assistants provided the correct HMR syntax for adding an image to the page. (Assistant A: Pos, Assistant B: Gist)

## Example 6

Instructions: Write a to-do list based on the given information. GCD> Input: Daily lists in the pharmacy

Deference Output: - Police sidewalk and entrance for litter

- Assess window displays to make sure they are not outdated or faded

- Clean all doors and remove outdated and unnecessary signs

- Fill bags at each checkout area

- Clean and straighten checkout counters

- Tidy up seating/suiting area

- Face (pull product forward) in all departments

- Check one department a day for short and outdated product

- Empty all train receptacles

- Vacuum (when the store is closed or only if customer traffic allows for it during store hours)

TF-IDF Word: information

### LLaM-7b

Pos: - Check inventory of medications

- Reorder medications

- Prepare patient profiles

- Enter patient information into the computer system

- Verify patient information

- Monitor expiration dates of medications

- Clean and resducc tables

- Answer telephone and provide assistance as needed

- Process prescriptions

- Refill prescriptions

- Monitor inventory of supplies

- Stock shelves

- Clean and resducc tables

- Answer telephone and provide assistance as needed

dist. - Check inventory and order new supplies

- Refill prescriptions

- Answer customer questions

- Process insurance paperwork

- Clean the pharmacy

- Stock shelves

- Count pills

- Take customer phone calls

- Monitor expiration dates on medications

- Update patient profiles

- Perform administrative tasks

- Count caders about prescriptions

- Answer questions from customers

- Process payments

- Maintain inventory

- Monitor controlled substances

- Perform quality control tests

- Review patient profiles

- Contact doctors about prescriptions

- Answer questions from customers

- Process payments

- Maintain inventory

- Monitor expiration dates on medications

- Update patient profiles

- Perform administrative tasks

- Review patient charts

- Contact doctors about prescriptions

- Answer questions from customers

- Process payments

[MISSING_PAGE_POST]

Checking empty dates of medications

2.

No-stocking shelves

3.

Entering prescription information into the computer system

4.

Answering customer questions

5.

Proparing and labeling prescriptions

6.

Scanning barcodes to check for potential drug interactions

7.

Processing insurance claims

8.

Maintaining inventory levels of medications

9.

Cleaning and organizing the pharmacy

10.

Updating patient profiles with new information

Neg: The pharmacist's daily tasks include checking and filling prescriptions, maintaining inventory, and counseling patients on the proper use of medications.

Human choice (Pos vs dist): Pos Tie dist

ChatFT choice (Pos vs dist): Pos

ChatFT person (Pos vs dist): Assistant A provides a concise and comprehensive list of daily tasks in the pharmacy, while Assistant

B:'s list is repetitive and includes irrelevant tasks like counting pills and performing quality control tests. Assistant A is more helpful. (Assistant A: Pos, Assistant B: dist)

Flan-Tg-Xkl

Pos: - Check inventory - fill prescriptions - Prepare medication - Label and package medications - Check expiration dates - Ensure all medications are in stock - Check for any errors in the inventory - Check for any problems with the medication - Check for any problems with the packaging - Check for any problems with the labeling - Check for any problems with the expiration date backing - Check for any problems with the packaging - Check for any problems with the

dist: To-do list for pharmacy:

Check inventory and stock shelves 2.

Prepare prescriptions 3.

Check inventory and stock shelves 4.

Check inventory and stock shelves 5.

Check inventory and stock shelves 6.

Check inventory and stock shelves 7.

Check inventory and stock shelves 8.

Check inventory and stock shelves 10.

WT-IDE: 1.

Stocking shelves 2.

Filling paperty: 3.

Greeting customers 4.

Taking orders 5.

Performing administrative tasks

Neg: The daily tasks in the pharmacy include preparing medications, filling prescriptions, assisting customers, and maintaining inventory.

Human choice (Pos vs dist): Gist Pos Gist

ChatFT choice (Pos vs dist): Pos

ChatFT reason (Pos vs dist): Assistant A provides a comprehensive list of daily tasks in the pharmacy, including preparing and labeling medications, checking for errors and problems, and ensuring all medications are in stock. Assistant B only lists two tasks and repeats one of them multiple times. Assistant A is more helpful. (Assistant A: Pos, Assistant B: dist)

Example 7

Instruction: You are given a paper citation, convert it to the requested citation style. <ID>

Input: Chicago Vaxanni, Ashish, Shazeer, Noma, Parmar, Niki, Usikoreit, Jakob, Jones, Llion, Gomez, Aidan N., Kaiser, Lukasz, and

Illinois Pollishishish. "Attention Is All You Need."

N.A.

Reference Output: Vaxanni, Ashish, et al.

"Attention Is All You Need."

arXiv, 2017, [https://doi.org/10.48550/arXiv.1706.03762](https://doi.org/10.48550/arXiv.1706.03762).

WT-IDE word:

LLaMA-7B

Pos: Vaxanni, Ashish, Shazeer, Noma, Parmar, Niki, Usikoreit, Jakob, Jones, Llion, Gomez, Aidan N., Kaiser, Lukasz, and Illia

Polosukhin.

"Attention Is All You Need."

arXiv (2017). [https://doi.org/10.48550/arXiv.1706.03762](https://doi.org/10.48550/arXiv.1706.03762).

Misti: Vaxanni, Ashish, Shazeer, Noma, Parmar, Niki, Usikoreit, Jakob, Jones, Llion, Gomez, Aidan N., Kaiser, Lukasz, and Illia

Polosukhin.

"Attention Is All You Need."

arXiv, 2017, [https://doi.org/10.48550/arXiv.1706.03762](https://doi.org/10.48550/arXiv.1706.03762).

TF-IDE: Attention Is All You Need.

arXiv, 2017, [https://doi.org/10.48550/arXiv.1706.03762](https://doi.org/10.48550/arXiv.1706.03762).

Reg: Chicago Vaxanni, Ashish, Shazeer, Noma, Parmar, Niki, Usikoreit, Jakob, Jones, Llion, Gomez, Aidan N., Kaiser, Lukasz, and

Illinois Pollishishish. "Attention Is All You Need."

arXiv, (2017). [https://doi.org/10.48550/arXiv.1706.03762](https://doi.org/10.48550/arXiv.1706.03762).

Manna Choice (Pos vs dist): IME Tie Tie

ChatFT choice (Pos vs dist): GistChatGPF reason (Pos vs dist): Both assistants provide correct MLA citation style, but Assistant A includes a comma after 'arXiv' which is not necessary in MLA style. (Assistant A: dist, Assistant B: Pos)

**FLAN-TS-XL**

Pos: Vaswani, Ashish, Shazeer, Noam, Parmar, Niki, Uszkoreit, Jakob, Jones, Llion, Gomez, Aidan N., Kaiser, Lukasz, and Illia Polosukhin.

"Attention Is All You Need."

arXiv, (2017).

@ist: Chicago: Vaswani, Ashish, Shazeer, Noam, Parmar, Niki, Uszkoreit, Jakob, Jones, Llion, Gomez, Aidan N., Kaiser, Lukasz, and Illia Polosukhin.

"Attention Is All You Need."

arXiv, (2017).

[https://doi.org/10.48552/arXiv.1706.03762](https://doi.org/10.48552/arXiv.1706.03762).

TF-IDF: Attention is all you need.

Neg: Attention Is All You Need

Wanan Choice (Pos vs dist): IOX IOX IOX IOX IOX ChatGPE (Pos vs dist): Gist ChatGPF reason (Pos vs dist): The instruction asks for MLA citation style, but neither assistant provided the citation in MLA format. However, Assistant A provided more information such as the ROI, so Assistant A is slightly more helpful. (Assistant A: dist, Assistant B: Pos)

### Example 8

Instruction: Provide a pleasant compliment based on the quality you have been given. dfl>

Input: Truthfulness

Reference Output: Everything would be better if more people were like you.

TF-IDF word: pleasant

#### LLaMa-Tb

Pos: You are remarkably truthful and honest.

Gist: You have a gift for being truthful.

TF-IDF: Truthfulness is a pleasant trait because it leads to trust, which is essential for any healthy relationship.

It also helps people build trust in their relationships with others, which can lead to better communication and understanding.

Additionally, it can help people avoid getting caught up in lines and deception, which can be damaging to relationships.

Neg: Truthfulness is like a lion in the forest: Ioud, proud, and unfraful to ear.

Human Choice (Pos vs dist): Pos Pos Pos

ChatGPF reason (Pos vs dist): Pos

ChatGPF reason (Pos vs dist): Both responses are good, but Assistant B's response is more detailed and specific, making it a more pleasant compliment overall. (Assistant A: dist, Assistant B: Pos)

#### FLAN-TS-XL**

Pos: You are incredibly honest.

Gist: Thank you for being so honest.

TF-IDF: Truthfulness is like a light that guides us through the darkness.

Neg: Truthfulness is like a lighthouse, providing guidance and comfort in times of trouble.

Human Choice (Pos vs dist): IOX Pos

ChatGPF reason (Pos vs dist): Pos

ChatGPF reason (Pos vs dist): Pos

and personal, making it a more pleasant compliment. (Assistant A: dist, Assistant B: Pos)

### Example 9

Instruction: Make a list of adjectives that can be used to describe the given brand. dfl>

Input: a creative tech startup

Reference Output: 1.

Innovative

2.

Creative

3.

Disruptive

4. cutting-edge

5. revolutionary

6. game-changing

7. industry-leading

8. trailblazing

9. groundbreaking

10. world-changing

TF-IDF Word: brand

LLaMa-Tb

Pos: Innovative, creative, tech-savoy, forward-thinking, dynamic, gille, fast-paced, cutting-edge, inventive, pioneering, and disruptive.

Gist: - Innovative

- Creative

- Abbitious

- Incentive

- Passionate

- Innovative

- Imaginative

- Visionary

- Entrepreneurial

- Optimistic

- Determined

- Persistent

- Responsible

- Thoughful

- Respectful

- Enpathetic

- Generous

- Reliable
* UT-IDF: TechnicalYamp
* Img: A creative tech startup is like a race car: it needs a great idea, a well-tuned team and the right resources to reach the finish line.
* Human choice (Vis Vis Visit): Pos Tie Gist
* Chatgrf Cisco (Pos Vis Visit): Pos
* Chatgrf Cisco (Pos Vis Sit): Pos
* Chatgrf Cisco (Pos Vis Sit): Pos
* Chatgrf Cisco (Pos Vis Sit): Pos
* Chatgrf Cisco (Pos Vis Sit): Assistant B: Pos
* Chatgrf Cisco (Pos Vis Sit): Assistant B: Pos
* Chatgrf Cisco (Pos Vis Sit): Assistant B: Pos
* Chatgrf Cisco (Assistant A: Gist, Assistant B: Pos)
* Pos: - Innovative - Creative - Tech - Startup - Innovative - Tech - Startup - Creative - Tech - Startup - Tech - Startup - Creative - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Test - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Test - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Test - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Test - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Test - Startup - Tech - Startup - Tech - Startup - Test - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Test - Startup - Tech - Startup - Test - Startup - Tech - Startup - Test - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Test - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Test - Startup - Tech - Startup - Tech - Startup - Test - Startup - Tech - Startup - Tech - Startup - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Tech - Startup - Test - - Startup - Tech - Startup - Tech - Startup - Test - - Startup - Tech - Startup - - Test - - Startup - Tech - Startup - Tech - Startup - Tech - Startup - - Startup - Tech - Startup - Tech - Startup - - Startup - Tech - Startup - - Tech - Startup - - Test - - Startup - Tech - Startup - - Tech - Startup - - Tech - -

# The best popular social media platform in Europe is the Android-based Ovi from Nokia.

Nuran choice (Pos vs Gist): Gist Gist Gist

Chiqrtcp Choice (Pos vs Gist): Gist

Response:
## Review

### Summary

This paper introduces a novel framework for compressing instruction prompts into smaller sets of 'gist' tokens, aiming to enhance computational efficiency in language models. The proposed method, called 'gisting', modifies attention masks to allow these tokens to encode and compress prompts, achieving significant compression rates and reductions in computational load. The authors conducted extensive experiments using large models and various evaluation methods, demonstrating that while the method shows promise, there are concerns about the marginal performance loss and the reliability of human evaluation. Additionally, the paper addresses the implications of this method for real-world applications, particularly in reducing context window usage.

### Strengths

- Novelty in compressing instruction prompts to save context windows and computation resources.
- Well-designed experiments with comprehensive evaluations using ROUGE-L, ChatGPT, and human assessments.
- Clear writing with well-illustrated figures and tables.
- Gisting is straightforward to implement, requiring minimal modifications to existing models.
- The method shows promising results in efficiency and effectiveness across different models.

### Weaknesses

- Misleading descriptions about 'gist tokens' and their role, as they are not true compressed information but special tokens in vocabulary.
- Low agreement among human evaluators raises concerns about the reliability of the evaluation results.
- The claimed 4% reduction in wall-clock time is not significant compared to the accuracy compromise.
- Compression rates are modest and not sufficient relative to the model's full context length.
- The method's fixed number of gist tokens for all tasks may limit its adaptability.

### Questions

- How is the compression factor in Figure 3 estimated?
- Could you clarify the claim about having too many gist tokens hurting performance?
- What is the tie-breaking process referenced regarding human evaluations?
- How does the method generalize for unseen prompts and what are the implications for task-specific performance?

### Soundness

**Score:** 3

**Description:** Good: The methodology is sound but has some concerns regarding the evaluation and the presented results.

### Presentation

**Score:** 3

**Description:** Good: The writing is generally clear, though some technical descriptions could be misleading.

### Contribution

**Score:** 3

**Description:** Good: The paper makes a relevant contribution to the field, although more clarity on certain aspects is needed.

### Rating

**Score:** 6

**Description:** Weak Accept: The paper is technically solid and presents an interesting approach, but it requires further evaluation and clarification.

### Paper Decision

**Decision:** Accept (poster)

**Reasons:** The paper presents a novel idea in the compression of prompts for language models, with good experimental backing and potential real-world applications. Despite some weaknesses and concerns regarding evaluation reliability, the strengths and contributions outweigh the issues, making it suitable for acceptance with minor revisions.