# Approximating Human-Like Few-shot Learning
with GPT-based Compression

Cynthia Huang Yuqing Xie††footnotemark:  Zhiying Jiang††footnotemark:  Jimmy
Lin Ming Li
Cheriton School of Computer Science, University of Waterloo
{hhuang, yuqing.xie, zhiying.jiang, jimmylin<EMAIL_ADDRESS>equal
contribution

###### Abstract

In this work, we conceptualize the learning process as information
compression. We seek to equip generative pre-trained models with human-like
learning capabilities that enable data compression during inference. We
present a novel approach that utilizes the Generative Pre-trained Transformer
(GPT) to approximate Kolmogorov complexity, with the aim of estimating the
optimal Information Distance for few-shot learning. We first propose using GPT
as a prior for lossless text compression, achieving a noteworthy compression
ratio. Experiment with LLAMA2-7B backbone achieves a compression ratio of 15.5
on enwik9. We justify the pre-training objective of GPT models by
demonstrating its equivalence to the compression length, and, consequently,
its ability to approximate the information distance for texts. Leveraging the
approximated information distance, our method allows the direct application of
GPT models in quantitative text similarity measurements. Experiment results
show that our method overall achieves superior performance compared to
embedding and prompt baselines on challenging NLP tasks, including semantic
similarity, zero and one-shot text classification, and zero-shot text ranking.

##  1 Introduction

Large labeled datasets are often scarce in the real world where annotation is
expensive and time consuming. This has prompted the development of few-shot
learning, where the model is learned using only a few annotated samples [17].
One resort to the few-shot scenario is to utilize the pre-trained models like
Generative Pre-trained Transformers (GPTs) [45, 46, 5, 41] with in-context
learning [5, 75], fine-tuning [36] or the combination [2]. However, in-context
learning requires heavy engineering to achieve a high accuracy [37], and its
ability to generalize to different tasks is constrained by the input size and
the need for precise formatting. Fine-tuning also has limitations, notably its
inability to generalize to rare out-of-distribution datasets with limited
labeled samples [73, 40].

Contrary to the data-hungry nature of machine learning, humans demonstrate
exceptional ability in zero-shot and few-shot learning, where only a handful
of labeled examples are needed to grasp a concept. Inspired by this, [24]
proposed a human-like few-shot learning framework and boils it down to the
ability of compressing data at inference time, measured by the universal
information distance. Derived from a simple and accepted assumption in
thermodynamics [3], the universal information distance emerges as the key
component in taking the effective usage of few labeled data. Hence, accurate
approximations of the information distance can lead to improved learning,
though its incomputability has posed significant challenges. This information
distance, consisting of Kolmogorov complexity [34], is both data-type-agnostic
and distribution-agnostic. Additionally, its parameter-free usage enables the
metric’s applicability across diverse applications. Inspired by this theory,
we propose a novel method GPT-AC, that leverages the knowledge GPTs learned
during pre-training. Our method tackles tasks traditionally challenging for
prompt engineering or fine-tuning, including semantic similarity, zero-shot
text classification and ranking.

Kolmogorov complexity, also known as the length of the shortest computer
program for a target output, is often approximated by the compression length.
At the same time, entropy coding algorithms attempt to approach the code
length lower bound declared by Shannon’s source coding theorem: the entropy of
the source. By using rich priors, such as pre-trained language models, that
more accurately predict the source distribution, we can optimize the
compression ratio and more closely approximate Kolmogorov complexity. Despite
the potential of large language model-based compressors, their direct
application to downstream tasks is nearly infeasible due to speed constraints.
In addition to the inference speed required by the language model itself, the
overhead of the compressor is even more substantial. Fortunately, the
information distance only requires the compressed length instead of actual
compression of the text sequence. We demonstrate an equivalence of the
compression length under arithmetic coding to the negative log probability of
text tokens when using GPT as the prior. This easy-to-compute compression
length enables us to efficiently approximate the universal information
distance without the overheads of the actual compression. By approximating
normalized information distances [10, 32, 33] using GPT-based compression, we
significantly enhance GPT’s ability to quantitatively capture text
similarities, which forms the foundation for its application in downstream NLP
tasks.

Our contributions are as follows:

  1. 1.

We propose a novel method that utilizes generative pre-trained language models
to approximate information distance for few-shot learning without the need for
fine-tuning or prompt engineering.

  2. 2.

By connecting arithmetic coding’s compression length to the cumulative
negative log probabilities in GPT models, we efficiently compute and
approximate the information distance derived from Kolmogorov complexity.

  3. 3.

We validate the effectiveness of our method through experiments in semantic
textual similarity, text classification and re-ranking under zero-shot and
one-shot settings, exhibiting notable improvement over embedding and prompt
baselines.

  4. 4.

We also demonstrate that our lossless text compression method GPT-AC achieves
SOTA compression ratio with Llama2-7B backbone, highlighting the potential of
pre-trained large language models as powerful priors in compression.

##  2 Related Works

###  2.1 Few-shot Learning

Prior to the emergence of large pre-trained models, the majority of previous
works on few-shot learning can be divided into two streams: meta/transfer-
learning based methods [64, 15, 54, 57] and data augmentation based methods
[30, 44, 14, 18]. However, the former relies on constraining the hypothesis
space by prior knowledge from other tasks or support datasets while the latter
depends on the heuristics of the datasets, often accompanied with the
smoothness assumption [61] (i.e., close data points in the input space share
the same label). Pre-trained models, on the other hand, have incorporated
prior knowledge during the pre-training stage and are proved to be excellent
few-shot learners [5]. However, pre-trained models suffer from (1) high
computational cost and (2) unsatisfactory performance in out-of-distributed
datasets [73]. The problem of computational cost is especially prominent for
large language models like GPT-3 where it is infeasible to fine-tune locally.
We utilize pre-trained language model for one-shot and zero-shot
classification tasks with no fine-tuning required.

###  2.2 Kolmogorov Complexity and Compression Distance

Information distance was first proposed by [3] as a universal similarity
metric between two objects. It was proved to be universal in a way that it is
optimal and can discover every possible metric [3]. Due to the problem of
incomputability, [9, 33, 10] have derived computable version of information
distances for tasks like clustering and plagiarism detection, shedding light
on the possibility of using real-world compressors to approximate Kolmogorov
complexity. These prior works utilize traditional compressors, indicating that
the performance gain on downstream tasks mainly comes from the compressor-
based distance metric.

Recently, [23] propose non-parametric learning by compression with latent
variables (NPC-LV) where neural networks are incorporated into information
distance. They demonstrate that trained generative models like variational
autoencoders can be used directly with zero parameters for downstream few-shot
image classification. Also, [25] employ this method in text classification
using GZIP, achieving competitive results compared to several widely-used deep
learning approaches. However, it remains open in how to incorporate pre-
trained language models into this framework, which we aim to address in this
paper. A recent study [71] explores the use of compression length for in-
context example selection. They rely on a large candidate set and applied the
model under generation setting. In contrast, we focus on adapting generative
models to zero/one-shot learning for text similarity tasks.

###  2.3 Neural Compression

Our GPT-based compressor falls under the category of neural compression where
neural networks are used for data compression. Shannon’s source coding theorem
[51] establishes the lower bound of the lossless compression on random
variables with probability distribution. With near-optimal coding schemes, the
bottleneck is the entropy approximation. Fortunately, deep generative models
with explicit density estimation serve as the entropy model that can learn
adaptively. [59] propose Bits-Back with Asymmetric Numeral Systems (BB-ANS), a
lossless compression algorithm based on VAE. Bit-Swap [28] further improves
BB-ANS by incorporating multiple latent variables and hierarchical networks.
In addition to autoencoders, Flow [47, 69]-based lossless compression [22]
outperform Bit-Swap and achieve the state of the art compression ratio of
images. The development of deep neural networks also benefits lossless text
compression. [19] use recurrent neural networks [50] combining with arithmetic
coding [70] to achieve higher compression ratio than GZIP. Recent
advancements, such as the fast transformer-based general-purpose lossless
compressor TRACE [39], have demonstrated promising results in enhancing
compression performance with transformer architecture. NNCP [1] v3.1,
adaptively encodes the source message with Transformers, achieves state-of-
the-art performance on the Matt Mahoney’s Large Text Compression
Benchmark111http://mattmahoney.net/dc/text.html.

###  2.4 Pre-trained Models

Pre-training has been adopted in numerous deep learning models with the rise
of transformer [62] due to its ability of learning task-agnostic
representation. In NLP, encoder-only transformers like BERT [13] has achieved
impressive performance on GLUE benchmark [68] including tasks like natural
language inference and sentiment analysis with only MLP and fine-tuning.
Decoder-only transformers like GPT [46, 5] can treat downstream discriminative
tasks in a generative way. However, previous works on few-shot learning using
language models are either prompt-based [43, 42, 35] or fine-tuning-based [75,
72, 40] while in this work, we propose a new way to leverage pre-trained
language models for few-shot learning without fine-tuning or in-context
learning.

##  3 Method

###  3.1 Human-Like Few-Shot Learning and Universal Information Distance

We consider the following generalized human-like few-shot learning setting:
assume a universe of objects $\Omega$ comprising various concept classes.
Given an unlabelled subset $U\subset\Omega$, a hypothesis $\phi$ is
formulated. For any concept class $c$, we have information $D_{c}$
representing either a description or representations derived from a few
examples. The goal is to determine the concept class for any new instance
$x\in\Omega$ under a computable metric $\mathcal{M}$:

$argmin_{c\in\mathcal{C}}\mathcal{M}(x,D_{c}|\phi(U)).$ (1)

Here, $\phi$ can be a pre-trained model that learned the distribution from the
unlabeled data $U$. For instance, GPT can be seen as the hypothesis $\phi$
learned from a large unlabeled corpus.

This learning scenario differs from traditional machine learning settings as
it permits extensive pre-training using unlabeled data but provides very
limited labeled data for learning concept classes. Under this framework, the
optimal metric $\mathcal{M}$ for few-shot learning algorithms is proven to be
the universal information distance, defined by Kolmogorov complexity (details
are shown in Appendix), based on the von-Neuman-Landauer Principle. Formally,
the universal information distance $\mathcal{M}_{UID}$ between two objects $x$
and $y$ is defined as:

$\mathcal{M}_{UID}(x,y)=\max\\{K(x|y),K(y|x)\\},$ (2)

$K(x|y)$ is the Kolmogorov complexity of $x$ given $y$, or informally, the
length of the shortest program that outputs $x$ given input $y$. Since the
Kolmogorov complexity is not computable [78], it is often approximated via
compression in practice.

###  3.2 GPT-AC: Generative Pre-trained Transformer based Arithmetic Coding
for Compression

In this section, we propose GPT-based Arithmetic Coding (GPT-AC) where GPT is
integrated into adaptive arithmetic coding, an entropy-based compressor.

GPT as the Entropy Model
Consider a text $T=(t_{1},t_{2},...,t_{n})$ composed of a sequence of tokens.
Let $\phi$ represent a GPT model, where
$\phi(t_{1:(i-1)})=P_{i}(t_{i}|t_{1},t_{2},...,t_{i-1})$ models the
probability distribution $P_{i}$ of the next token $t_{i}$. The function
$\phi(T)$ outputs all next-token probability distributions
$(P_{2},\cdots,P_{n+1})$. To derive the distribution for $P_{1}$, an EOS (End
Of Sentence) token is added at the start of the text as $t_{0}$. For each
token $t_{i}$, the associated $P_{i}$ serves as the entropy model for both
encoding and decoding in the compressor.

Figure 1: Illustration of GPT-based Arithmetic Encoding

GPT-AC Encoding
In the encoding phase, under the scheme of adaptive arithmetic coding, we
start with an initial interval $I^{0}=[0,1)$. For each token $t_{i}$ in the
sequence, we calculate the cumulative distribution function $F_{i}(t_{i})$ and
$P_{i}(t_{i})$ based on $\phi(t_{1:(i-1)})$. Then, the interval
$I=[I_{\text{low}},I_{\text{high}})$ is updated according to the range
assigned to $t_{i}$:

$\begin{split}&I_{\text{low}}^{i}=I_{\text{low}}^{i-1}+(I_{\text{high}}^{i-1}-I_{\text{low}}^{i-1})\times
F_{i}(t_{i}),\\\
&I_{\text{high}}^{i}=I_{\text{low}}^{i-1}+(I_{\text{high}}^{i-1}-I_{\text{low}}^{i-1})\times(F_{i}(t_{i})+P_{i}(t_{i}))\end{split}$
(3)

After updating $I$ for each token in the sequence, we can pick any number $x$
within the final interval to represent the entire text sequence.

GPT-AC Decoding
When decoding the encoded message $x$, the token $t_{1}$ can be identified by
finding the range $[F_{1}(t_{1}),F_{1}(t_{1})+P_{1}(t_{1})]$ that includes
$x$. The value of $x$ is then updated by normalizing it within the range of
$t_{1}$, using the formula: $x\leftarrow\frac{x-F_{1}(t_{1})}{P_{1}(t_{1})}$.
With this updated $x$ and the next-token probability distribution
$\phi(t_{2})$, we can decode the next token $t_{2}$. This process is repeated
until an EOS token is encountered, indicating the end of the text. The text
can be losslessly decoded using $x$ and $\phi$ alone.

Negative Log Probability as Compression Length
During the arithmetic encoding, the length of the interval $I^{i}$ equals to
$I^{i-1}*P_{i}(t_{i})$. From an initial interval of length 1, the entire
message’s encoding results in a final interval with a length of
$\prod^{n}_{i=1}P_{i}(t_{i})$. The number of bits required to represent this
final interval, and thus the message $T$, is
$\sum_{i=1}^{n}-\log_{2}P_{i}(t_{i})$. This reveals a method to approximate
the compression length directly without exactly performing the compression.
With the triangular forward attention masking in GPT, we can pass the full
tokenized text sequence to the model and obtain probability distributions for
all tokens.

GPT Pre-training Optimizes for Compression Length
The optimization target during pre-training for auto-regressive models such as
GPT is defined as:

$\displaystyle L(T|p_{model})$ $\displaystyle=-\log p_{model}(T)=-\log
p_{model}(t_{1},t_{2},...,t_{n})=\sum_{i=1}^{n}-\log_{2}P_{i}(t_{i}).$

For entropy coding, $H(T)\triangleq\mathbb{E}(-\log p_{data}(T))$, defining
the optimal code length. While $p_{data}$ is often unknown, we thus use the
observation $p_{\bar{data}}$ to approximate $p_{data}$:

$\displaystyle H(T)$ $\displaystyle=\mathbb{E}_{p_{data}}[-\log
p_{model}(T)]\simeq\mathbb{E}_{p_{\bar{data}}}[-\log p_{model}(T)]$

According to The Shannon–Fano–Elias coding scheme [11], we can construct a
prefix-free code of length $-\log p_{\text{model}}(t_{1},t_{2},...,t_{n})+2$
bits. Consequently, the pre-training phase of GPT models is essentially
learning a compressor that optimizes the coding length.

The Rank Coding Variant
In the method outlined above, we primarily employ arithmetic coding for text
compression. An alternative variant of lossless coding uses rank instead of
probability [21]. After the GPT model predicts the distribution of the next
token, we can rank all tokens according to their probability. The target token
is assigned the corresponding rank index, with higher probability tokens
having smaller rank indices. In this variant, we approximate the compression
length as $\sum^{n}_{i=1}\log_{2}(rank_{i})$ where $rank_{i}$ denotes the rank
for token $i$.

Applicability of Our Method
When calculating compression length, the algorithm is most efficient with
transformers that use forward-only attention. This approach allows for a
single-pass processing of the entire text sequence, thus ensuring efficient
computation. However, through the use of a sliding window technique, the
method’s applicability can be extended to all generative language models
covering decoder-only and encoder-decoder architectures.

###  3.3 Universal Information Distance Approximation

Having computed the compression length using the GPT-AC method, we can now
utilize it to approximate the universal information distance. Let
$x=\\{x_{1},\cdots,x_{n}\\}$ and $y=\\{y_{1},\cdots,y_{m}\\}$ denote two
tokenized text sequences, where each $x_{i}\text{ or }y_{i}$ represents a
token in the sequence. We approximate $K(x)$ using the compression length
$C(x)=\sum_{i=1}^{n}-\log_{2}{P_{i}(x_{i})}$ where $P_{i}$ represents the
probability distribution for $x_{i}$ as predicted by the GPT model.

As in Equation 2, we also need $K(x|y)$, approximated as follows: let
$P_{i}=\phi(y,x_{1:{i-1}})$ denotes the probability distribution for token
$x_{i}$ output by the GPT model, given $y=(y_{1},\cdots,y_{m})$ and previous
tokens in $x$, $K(x|y)$ can be estimated as
$C(x|y)=\sum_{i=1}^{n}-\log_{2}{P_{i}(x_{i})}$. A similar approach can be used
to estimate $K(y)$ and $K(y|x)$. We denote all compressed-based approximations
in $C(\cdot)$.

However, compression lengths vary when the lengths of the input text differ.
We need a normalized version to enable comparison across diverse object pairs.
There are several normalized measures. [33] introduced a normalized version,
referred to as the Normalized Information Distance (NID):

$\mathcal{M}_{max}(x,y)=\frac{\max\\{K(x|y),K(y|x)\\}}{\max\\{K(x),K(y)\\}}$
(4)

To tackle challenges such as partial matching 222Partial matching means
situations where only portions of two objects match each other. , [31]
proposed the following variants of the universal distances suitable for
broader application scenarios:

$\mathcal{M}_{min}(x,y)=\frac{\min\\{K(x|y),K(y|x)\\}}{\min\\{K(x),K(y)\\}}.$
(5)

[27] proposed the Compression-Based Dissimilarity Measure (CDM) for data
mining applications, demonstrating its effectiveness in practice. This is
rescaled to fit the range $[0,1)$:

$\mathcal{M}_{mean}(x,y)=\frac{C(x|y)+C(y|x)}{C(x)+C(y)}=2*CDM-1\text{,
}CDM=\frac{C(xy)}{C(x)+C(y)}$ (6)

###  3.4 Applications of Universal Information Distance

We will now explain how the aforementioned distances can be applied to various
NLP tasks. To determine the text similarity score between two texts $x,y$, we
first compute their individual compression lengths $C(x),C(y)$. We also
calculate the joint and conditional compression lengths $C(xy),C(x|y),C(y|x)$.
Using these values, we compute the distance metrics defined in Section 3.3 as
$\mathcal{M}$. We can then apply these distance measures to specific tasks:

  * •

For semantic textual similarity, we treat the two sentences as $x$ and $y$,
and use $\mathcal{M}$ as predictions.

  * •

For zero-shot text classification, we treat the label descriptions as $x$ and
the multiple choice options as $y$. For one-shot text classification, we
consider the training sample as $x$ and the test sample as $y$. We classify
the text sample by comparing $\mathcal{M}$ values for different classes.

  * •

For text re-ranking, we treat the documents as $x$ and the queries as $y$,
ranking according to $\mathcal{M}$.

##  4 Experiments

Our experimental evaluation consists of four key components: lossless text
compression and three downstream tasks, namely semantic textual similarity,
text classification, and text re-ranking. For downstream applications, we
mainly conduct experiments with the GPT-2 small (124M), comparing to GPT-2
embedding or prompt tuning baselines and BERT-base-uncased (110M) models from
HuggingFace Hub333https://huggingface.co/. We take GPT-2 as an example due to
its light weights and availability. However, the proposed method is not
limited to GPT models. It can be readily applied to more advanced large
language models, such as LLAMA[58], provided that the output probabilities are
available.

###  4.1 Lossless Text Compression

In the Lossless Text Compression task, we assess our method on the Enwik9 [53]
and the first gigabyte of BookCorpus [77] datasets. In addition to GPT-2
models, we tested our method on LLAMA2-7B [58]. GPT-AC is benchmarked against
both traditional methods, such as GZIP [12], and contemporary neural network-
based techniques like DZIP [20] and TRACE [39]. In the actual implementation,
GPT-AC processes chunks of 2,500 characters for GPT-2 and 10,000 characters
for LLAMA2-7B independently. Although this would slightly compromise the
compression ratio, it enables parallel computing.

Table 1: Compression Ratio by Compression Method. Note the compression ratio
equals to Original text length / Compressed text length.

Model $\rightarrow$ | GPT-AC (Ours) |  GZIP |  7z |  DZIP |  TRACE |  NNCP
---|---|---|---|---|---|---
Dataset $\downarrow$ | Llama2-7B | GPT2-L | GPT2-M | GPT2-S |  |  |  |  |
Enwik9 | 15.56 | 8.05 | 7.71 | 6.53 |  3.09 |  4.35 |  4.47 |  5.29 |  9.33
BookCorpus | 10.55 | 8.34 | 7.89 | 7.22 |  2.77 |  3.80 |  3.95 |  4.58 |  -

As shown in Table 1, GPT-AC significantly outperforms conventional methods
like GZIP and 7z in compression ratio. Even with the GPT-2 small model, GPT-AC
achieves a more than 2-fold enhancement in compression ratio compared to the
widely-used GZIP, on both the Enwik9 and Book datasets. On Enwik9, GPT-AC with
Llama2-7B records a compression ratio of 15.56, a 67% enhancement over the
previous state of the art of 9.33, based on NNCP [1] 444Refer
to:http://mattmahoney.net/dc/text.html. As the language model increases, the
compression ratio consistently improves, suggesting that larger and better-
trained language models will further amplify these results.

Note that NNCP involves updating the parameters of a large transformer model
(v3.1, 199M) during encoding. We can also achieve an even higher compression
ratio with encode-time optimization. However, the encode-time optimization
process only enables an increase in compression ratio as the input message
length goes up, and would overfit to the specific message. With random
initialization, the compression ratio will be around 1.0 for the beginning of
the input message, offering no benefits to similarity measurement.

###  4.2 Semantic Textual Similarity

For the Text Semantic Modeling, we test the models on the Semantic Textual
Similarity benchmark (STS-b) [8]. The dataset consist of sentence pairs with
labels from 0 to 5 indicating the semantic relatedness. We compare GPT-AC
against GPT-2-emb [46], where we take the last token embedding vector, and
BERT-emb [13], where we take the averaged token embedding vector, which has
been proven to be effective in previous studies[52]. We then calculate the
cosine similarity between these vectors to serve as the distance measure. For
GZIP, we follow [23, 25] and use the normalized compression distance as the
metric.

Table 2: Semantic Textual Similarity Performance. Spearman Rank Correlation $\rho$ between the distance metrics and given labels for the STS datasets. $\rho*100$ is reported. Dataset | # Test | GPT-AC (Ours) | GZIP | GPT-emb | BERT-emb
---|---|---|---|---|---
STS-12 | 3,108 | 40.2 | 50.4 | 5.4 | 30.9
STS-13 | 1,500 | 66.0 | 48.4 | 14.6 | 59.9
STS-14 | 3,750 | 55.3 | 43.3 | 10.9 | 47.7
STS-15 | 3,000 | 70.3 | 59.1 | 9.6 | 60.3
STS-16 | 1,186 | 69.5 | 59.4 | 26.8 | 60.3
STS-b | 1,379 | 55.0 | 50.7 | 12.4 | 47.3

As shown in Table 2, our method substantially outperforms the cosine
similarity distance metrics derived from GPT-2 embeddings and shows moderate
enhancement over those utilizing BERT embeddings. These results demonstrate
the effectiveness of the approximated information distance in capturing
semantic similarities.

###  4.3 Text Classification

For Text Classification, we evaluate the models on PIQA (Physical Interaction:
Question Answering [4]) and CaseHOLD (Case Holdings On Legal Decisions [76])
for zero-shot classification, and SST-5(sentiment analysis) [56], Medical
abstracts [49], AG-News(news headlines) [74], and Banking77(banking and
financial) [7] for one-shot classification. We compare our method with two
main approaches: 1) fine-tuning GPT-2 or BERT with a classification layer, 2)
in-context learning with GPT-2 (Refer to Appendix for detailed settings) and
3) calculate cosine similarity of Sentence-BERT (all-MiniLM-L12-v2) embeddings
as a metric for classification.

Table 3: Text Classification Accuracy (100%). We report the averaged accuracy
across 5 runs with different random seeds, together for the standard
deviations. This does not apply to zero-shot experiments because the models do
not contain randomness.

|  | Model$\rightarrow$ | GPT-AC | GZIP | GPT-prompt | GPT | BERT | (SBERT)
---|---|---|---|---|---|---|---|---
Dataset$\downarrow$ | # C | Domain$\downarrow$ | (Ours) |  |  |  |  |
Zero-shot Multiple Choice |
PIQA | 2 | Reasoning | 61.5 | 53.4 | 50.5 | 49.2 | 50.1 | (56.5)
CaseHOLD | 5 | Legal | 58.3 | 52.4 | 20.3 | 19.9 | 35.0 | (50.6)
One-shot |
AGNews | 4 | News |  47.8$\pm$3.3 | 30.2$\pm$3.0 | 47.2$\pm$2.9 | 37.7$\pm$7.2 | 45.5$\pm$3.1 | (45.8$\pm$10.2)
Medical | 5 | Bio-Med |  27.9$\pm$3.2 | 25.6$\pm$2.8 | 22.1$\pm$1.2 | 23.7$\pm$3.5 | 23.8$\pm$4.8 | (39.7$\pm$9.1)
SST5 | 5 | Sentiment | 26.8$\pm$3.1 | 21.2$\pm$2.7 |  29.8$\pm$1.6 | 22.7$\pm$2.8 | 21.1$\pm$3.3 | (26.2$\pm$2.3)
Banking77 | 77 | Finance |  34.0$\pm$1.3 | 20.3$\pm$1.5 | - | 21.7$\pm$1.7 | 24.5$\pm$3.9 | (53.1$\pm$1.9)

As depicted in Table 3, in the zero-shot multiple-choice classification
context, the information distance approximated by GPT-AC delivers superior
results than cosine similarity distance metrics based on the embeddings from
GPT-2, BERT, and even SBERT. Note that SBERT, which is fine-tuned on 1 billion
high-quality labeled sentence pairs, does not fall under our category of
human-learning few-shot models; it is included to provide a point of
reference.

In one-shot text classification, our method surpasses both fine-tuned GPT and
BERT on all datasets. Additionally, we also outperform the GPT-prompt version
in all datasets except SST-5. Given that SST-5 is a widely used classification
benchmark, we hypothesize that the superior performance of the prompt approach
could be due to data leakage during GPT pre-training. Moreover, we did not
apply the GPT-prompt method to the banking77 dataset because accommodating
one-shot samples of 77 classes [7] within the GPT-2 prompt proves challenging,
and adjusting the prompt can be complex. This issue represents a significant
hurdle when applying GPT-2 with in-context learning.

###  4.4 Text Re-ranking

For Text Re-Ranking, we evaluate the models on various domain-specific zero-
shot text retrieval datasets, including Trec-Covid [65], Trec-News [55] ,
SciFact [67] , BioASQ [60] , FiQA-2018 [38] , and ArguAna [66]. Given a query,
we first retrieve the top relevant document with BM25 [48] with Elastic Search
API555https://github.com/elastic/elasticsearch. We then re-rank the documents
with the models. We compare our system with the original BM25 ranking, and the
Dense Passage Retrieval (DPR) [26] model, a BERT-based model already fine-
tuned on the MS MARCO [6] for ranking, and a text GZIP [25] compressor.

Table 4: Text Re-Ranking Effectiveness (NDCG@10). Retrieving top 100 relevant passages using BM25 and re-ranking using GPT-AC. We present the best result among different metric combinations.  |  |  Model $\rightarrow$ | GPT-AC(Ours) | BM25 | GZIP | (DPR)
---|---|---|---|---|---|---
Dataset $\downarrow$ | # Test | Domain |  |  |  |
TREC-COVID | 50 | COVID | 0.694 | 0.656 | 0.447 | (0.332)
TREC-NEWS | 57 | News | 0.225 | 0.398 | 0.142 | (0.161)
SciFact | 300 | Scientific | 0.648 | 0.665 | 0.053 | (0.318)
BioASQ | 500 | Bio-Med | 0.517 | 0.465 | 0.157 | (0.127)
FiQA-2018 | 648 | Finance | 0.239 | 0.236 | 0.032 | (0.112)
ArguAna | 1406 | Argument | 0.327 | 0.315 | 0.073 | (0.175)

As shown in Table 4, our proposed method outperforms the BERT-based DPR model
across all settings. Despite the DPR model being fine-tuned on the massive
labeled MS MARCO dataset, our performance remains superior. We also benchmark
our model against the GZIP compression method. The improvements observed
indicate that GPT-AC can provide significant semantic information, leading to
improved ranking results. Notably, BM25 is a strong baseline in that domain-
specific texts can contain many out-of-distribution terms, potentially
hampering the performance of the language model. Despite this, our method
demonstrates comparable performance across most of the datasets and surpasses
BM25 in certain domains, particularly in Bio-Medical and Finance, which
require much domain-specific understanding.

##  5 Analysis

###  5.1 Comparison of Distance Metrics

In Table 5, we compare the results of all distance metrics with two different
coding variants. As stated in Section 3.3, we always take the longer sequence
results with $\mathcal{M}_{max}$, the shorter sequence results with
$\mathcal{M}_{min}$, and the average over both of them with
$\mathcal{M}_{mean}$. We aim to avoid bias towards any single sequence when
both texts contain similar or equally important information, suggesting that
$\mathcal{M}_{mean}$ is the most suitable. This intuition is justified by the
results for STS and one-shot classification. Yet, the approach shifts for re-
ranking. Typically, queries contain fewer than 20 tokens, contrasting with
documents that often contain hundreds of tokens. We thus want to focus the
measurement on the query part, without being disturbed by the extra or non-
relevant information in the document. Therefore, $\mathcal{M}_{min}$ is the
preferred choice. ArguAna, however, is a unique case as both the query and the
document contain about 250 tokens with similar content, thus making
$\mathcal{M}_{mean}$ more suitable. For zero-shot classification, despite the
varying content and text length between the two sequences, both pieces of
information remain important, making $\mathcal{M}_{mean}$ the optimal choice.
Finally, we observe $\mathcal{M}_{mean}$ works better with Log-Rank, while
$\mathcal{M}_{min}$ works better with Log-Prob. However, no specific trend is
evident for a broader conclusion.

Table 5: Distance metric Analysis. We also list the token length of both $x$
and $y$

|  |  | Log-Prob | Log-Rank
---|---|---|---|---
Dataset | $len(x)$ | $len(y)$ | $\mathcal{M}_{max}$ | $\mathcal{M}_{min}$ | $\mathcal{M}_{mean}$ | $\mathcal{M}_{max}$ | $\mathcal{M}_{min}$ | $\mathcal{M}_{mean}$
Semantic Textual Similarity (Spearman Rank Correlation $\rho*100$) |  |  |
sts-b | 13 | 13 | 44.7 | 47.6 | 48.2 | 50.4 | 54.2 | 55.0
Zero-shot Classification (Accuracy %) |  |  |
PIQA | 24 | 10 | 60.0 | 56.4 | 62.0 | 59.5 | 55.0 | 61.5
CaseHold | 29 | 219 | 56.0 | 27.2 | 57.9 | 55.2 | 27.6 | 58.3
One-shot Classification (Accuracy %) |  |  |
AGNews | 53 | 53 | 41.3 | 41.1 | 47.8 | 40.5 | 39.9 | 43.3
Medical | 296 | 268 | 25.4 | 27.0 | 27.9 | 25.5 | 26.3 | 27.7
SST5 | 23 | 24 | 24.8 | 26.2 | 26.7 | 24.3 | 26.0 | 26.8
Banking77 | 15 | 14 | 30.6 | 25.2 | 33.9 | 29.8 | 24.7 | 34.0
Re-ranking (NDCG@10) |  |  |
TREC-COVID | 303 | 17 | 0.459 | 0.655 | 0.467 | 0.473 | 0.694 | 0.489
TREC-NEWS | 808 | 16 | 0.173 | 0.161 | 0.167 | 0.184 | 0.161 | 0.186
BioASQ | 304 | 14 | 0.306 | 0.517 | 0.364 | 0.267 | 0.507 | 0.330
FiQA-2018 | 247 | 15 | 0.128 | 0.239 | 0.154 | 0.118 | 0.222 | 0.143
ArguAna | 276 | 247 | 0.277 | 0.257 | 0.301 | 0.282 | 0.262 | 0.307
SciFact | 345 | 21 | 0.389 | 0.648 | 0.519 | 0.351 | 0.635 | 0.478

###  5.2 Information Distance and Classification Accuracy

Figure 2: Relation between Prediction Distance Ratio and One-shot
Classification Accuracy. Experiment result under $\mathcal{M}_{mean}$ with
Log-Prob.

In Figure 2, we aim to illustrate the performance variance when the test cases
have different distance scoring. For each test case, we compute the prediction
distance ratio
$R_{pred}(x)=\frac{\mathcal{M}(x,D_{c^{*}})}{\frac{1}{|C|}\sum_{c}\mathcal{M}(x,D_{c})}$.
Here, $D_{c}$ represents the one-shot example in class $c$, $C$ embodies all
the classes, and $c^{*}$ stands for the class predicted under metric
$\mathcal{M}$. A smaller $R_{pred}$ suggests that the predicted class is more
distant from the average distance. We then categorize all the test samples
according to their $R_{pred}$ value, with each group containing 10% of the
data. In Figure 2, the x-axis represents the average $R_{pred}$ within each
group, and the y-axis represents the group accuracy. The plot indicates that
the further the predicted class deviates from the average, the better
performance of our method.

##  6 Conclusion and Discussion

In this work, we introduce GPT-based Compression, a novel approach that
leverages GPT models to estimate the information distance for few-shot
learning. Our proposed method facilitates an out-of-the-box application of
generative language models under zero-shot and one-shot scenarios without
fine-tuning or prompting. This enhances the generalizability of pre-trained
generative models, demonstrated by our experiments across various downstream
NLP tasks. While this method can synergize with existing techniques such as
further fine-tuning and various prompting techniques, we leave these
combinations as potential areas for future research.

Tapping into the capabilities of pre-trained language models can bring about
significant improvements in lossless text compression. Our experiments
indicate a direct relationship between the scale of the language model and the
improvement in compression ratio. Performance can also be elevated with fine-
tuning during encoding. As pre-trained large language models become more
accessible, integrating LLM-driven compression into practical applications can
lead to significant advantages, including reduced storage expenses and
minimized transmission overheads in various contexts.

The universal information distance is foundational, unifying various popular
deep learning approaches for few-shot learning. For example, Siamese Network
[29] uses twin networks to extract features, where $\mathcal{M}$ refers to a
contrastive loss; Prototypical Network [54] optimizes $\mathcal{M}$ to learn a
better $\mathcal{D}_{c}$ in the embedding space; Bi-Encoder architecture used
in SBERT can also be unified where $\mathcal{M}$ can be cosine similarity.

Lastly, we wish to highlight the distinction between two learning paradigms:
large-data dependent learning and human-like few-shot learning. Despite the
impressive ability of recent GPT models to learn from vast data, we contend
that the creation of new concepts and ideas will predominantly occur in a few-
shot learning manner, regardless of future advancements in these models. In
the context of few-shot learning, where labeled data are scarce and cannot be
used to approximate the non-computable information distance, both humans and
machines are poised on an equal footing to unearth new regularities that
augment compression.

## Limitations

For downstream NLP tasks, our experiments use older versions of pre-trained
language models, constrained by computational constraints and limited access.

## References

  * [1] Fabrice Bellard.  Lossless data compression with transformer, 2021.
  * [2] Eyal Ben-David, Nadav Oved, and Roi Reichart.  Pada: Example-based prompt learning for on-the-fly adaptation to unseen domains.  Transactions of the Association for Computational Linguistics, 10:414–433, 2022.
  * [3] Charles H Bennett, Péter Gács, Ming Li, Paul MB Vitányi, and Wojciech H Zurek.  Information distance.  IEEE Transactions on Information Theory, 44(4):1407–1423, 1998\.
  * [4] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi.  Piqa: Reasoning about physical commonsense in natural language.  In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020\.
  * [5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [6] Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, and Bhaskar Mitra.  Ms marco: A human generated machine reading comprehension dataset.  ArXiv, abs/1611.09268, 2016.
  * [7] Iñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic.  Efficient intent detection with dual sentence encoders.  In Proceedings of the 2nd Workshop on NLP for ConvAI - ACL 2020, mar 2020.  Data available at https://github.com/PolyAI-LDN/task-specific-datasets.
  * [8] Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada, August 2017. Association for Computational Linguistics.
  * [9] Xin Chen, Brent Francia, Ming Li, Brian Mckinnon, and Amit Seker.  Shared information and program plagiarism detection.  IEEE Transactions on Information Theory, 50(7):1545–1551, 2004\.
  * [10] Rudi Cilibrasi and Paul M. B. Vitányi.  Clustering by compression.  CoRR, cs.CV/0312044, 2003.
  * [11] Thomas M Cover.  Elements of information theory.  John Wiley & Sons, 1999.
  * [12] L. Peter Deutsch.  GZIP file format specification version 4.3.  RFC 1952, May 1996.
  * [13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019.
  * [14] Matthijs Douze, Arthur Szlam, Bharath Hariharan, and Hervé Jégou.  Low-shot learning with large-scale diffusion.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pages 3349–3358, 2018.
  * [15] Harrison Edwards and Amos Storkey.  Towards a neural statistician.  International Conference on Learning Representations (ICLR), 2016\.
  * [16] R. Fagin and L. Stockmeyer.  Relaxing the triangle inequality in pattern matching.  In International Journal of Computer Vision, pages 219–231, 1998\.
  * [17] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In International conference on machine learning, pages 1126–1135. PMLR, 2017.
  * [18] Hang Gao, Zheng Shou, Alireza Zareian, Hanwang Zhang, and Shih-Fu Chang.  Low-shot learning via covariance-preserving adversarial augmentation networks.  In Advances in Neural Information Processing Systems (NeurIPS), 2018\.
  * [19] Mohit Goyal, Kedar Tatwawadi, Shubham Chandak, and Idoia Ochoa.  Deepzip: Lossless data compression using recurrent neural networks.  In 2019 Data Compression Conference (DCC), pages 575–575. IEEE, 2019.
  * [20] Mohit Goyal, Kedar Tatwawadi, Shubham Chandak, and Idoia Ochoa.  Dzip: improved general-purpose lossless compression based on novel neural network modeling.  CoRR, abs/1911.03572, 2019.
  * [21] Michael Herrera and Kasey Luo.  Lossless neural text compression, 2021.
  * [22] Emiel Hoogeboom, Jorn Peters, Rianne Van Den Berg, and Max Welling.  Integer discrete flows and lossless compression.  Advances in Neural Information Processing Systems, 32, 2019.
  * [23] Zhiying Jiang, Yiqin Dai, Ji Xin, Ming Li, and Jimmy Lin.  Few-shot non-parametric learning with deep latent variable model.  In Advances in Neural Information Processing Systems, 2022.
  * [24] Zhiying Jiang, Rui Wang, Dongbo Bu, and Ming Li.  A theory of human-like few-shot learning, 01 2023.
  * [25] Zhiying Jiang, Matthew Yang, Mikhail Tsirlin, Raphael Tang, Yiqin Dai, and Jimmy Lin.  “low-resource” text classification: A parameter-free classification method with compressors.  In Findings of the Association for Computational Linguistics: ACL 2023, pages 6810–6828, Toronto, Canada, July 2023. Association for Computational Linguistics.
  * [26] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.  Dense passage retrieval for open-domain question answering.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online, November 2020\. Association for Computational Linguistics.
  * [27] Eamonn Keogh, Stefano Lonardi, and Chotirat Ann Ratanamahatana.  Towards parameter-free data mining.  In ., KDD ’04, page 206–215, New York, NY, USA, 2004. Association for Computing Machinery.
  * [28] Friso H. Kingma, P. Abbeel, and Jonathan Ho.  Bit-swap: Recursive bits-back coding for lossless compression with hierarchical latent variables.  In International Conference on Machine Learning, 2019.
  * [29] Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al.  Siamese neural networks for one-shot image recognition.  ., 2(1), 2015.
  * [30] Roland Kwitt, Sebastian Hegenbart, and Marc Niethammer.  One-shot learning of scene locations via feature trajectory transfer.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pages 78–86, 2016.
  * [31] Ming Li.  Information distance and its applications.  In Oscar H. Ibarra and Hsu-Chun Yen, editors, Implementation and Application of Automata, pages 1–9, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
  * [32] Ming Li, Jonathan H. Badger, Xin Chen, Sam Kwong, Paul Kearney, and Haoyong Zhang.  An information-based sequence distance and its application to whole mitochondrial genome phylogeny .  Bioinformatics, 17(2):149–154, 02 2001.
  * [33] Ming Li, Xin Chen, Xin Li, Bin Ma, and P.M.B. Vitanyi.  The similarity metric.  IEEE Transactions on Information Theory, 50(12):3250–3264, 2004\.
  * [34] Ming Li and Paul M. B. Vitányi.  An Introduction to Kolmogorov Complexity and its Applications.  Springer, 2008.
  * [35] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, 2021.
  * [36] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  Advances in Neural Information Processing Systems, 35:1950–1965, 2022.
  * [37] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  ACM Computing Surveys, 55(9):1–35, 2023.
  * [38] Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur.  Www’18 open challenge: Financial opinion mining and question answering.  In Companion Proceedings of the The Web Conference 2018, WWW ’18, page 1941–1942, Republic and Canton of Geneva, CHE, 2018. International World Wide Web Conferences Steering Committee.
  * [39] Yu Mao, Yufei Cui, Tei-Wei Kuo, and Chun Jason Xue.  Trace: A fast transformer-based general-purpose lossless compressor.  In Proceedings of the ACM Web Conference 2022, WWW ’22, page 1829–1838, New York, NY, USA, 2022. Association for Computing Machinery.
  * [40] Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin.  Document ranking with a pretrained sequence-to-sequence model.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708–718, 2020.
  * [41] OpenAI.  Gpt-4 technical report, 2023.
  * [42] Ethan Perez, Douwe Kiela, and Kyunghyun Cho.  True few-shot learning with language models.  Advances in neural information processing systems, 34:11054–11070, 2021.
  * [43] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller.  Language models as knowledge bases?  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463–2473, 2019.
  * [44] Tomas Pfister, James Charles, and Andrew Zisserman.  Domain-adaptive discriminative one-shot learning of gestures.  In European Conference on Computer Vision (ECCV), pages 814–829. Springer, 2014.
  * [45] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training, 2018.
  * [46] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  OpenAI, 2019.
  * [47] Danilo Rezende and Shakir Mohamed.  Variational inference with normalizing flows.  In International conference on machine learning, pages 1530–1538. PMLR, 2015.
  * [48] Stephen Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford.  Okapi at trec-3.  In Overview of the Third Text REtrieval Conference (TREC-3), pages 109–126. Gaithersburg, MD: NIST, January 1995.
  * [49] Tim Schopf, Daniel Braun, and Florian Matthes.  Evaluating unsupervised text classification: Zero-shot and similarity-based approaches.  In 2022 6th International Conference on Natural Language Processing and Information Retrieval (NLPIR), NLPIR 2022, New York, NY, USA, 2023\. Association for Computing Machinery.
  * [50] Mike Schuster and Kuldip K Paliwal.  Bidirectional recurrent neural networks.  IEEE transactions on Signal Processing, 45(11):2673–2681, 1997\.
  * [51] Claude Elwood Shannon.  A mathematical theory of communication.  The Bell System Technical Journal, 27(3):379–423, 1948.
  * [52] Tianye Sheng, Lisong Wang, Zongfeng He, Mingjie Sun, and Guohua Jiang.  An unsupervised sentence embedding method by maximizing the mutual information of augmented text representations.  In Artificial Neural Networks and Machine Learning – ICANN 2022: 31st International Conference on Artificial Neural Networks, Bristol, UK, September 6–9, 2022, Proceedings, Part II, page 174–185, Berlin, Heidelberg, 2022. Springer-Verlag.
  * [53] Harshdeep Singh, Robert West, and Giovanni Colavizza.  Wikipedia citations: A comprehensive data set of citations with identifiers extracted from english wikipedia.  Quantitative Science Studies, pages 1–19, 2020.
  * [54] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  Advances in Neural Information Processing Systems (NeurIPS), 2017\.
  * [55] Ian Soboroff, Shudong Huang, and Donna Harman.  Trec 2019 news track overview.  In TREC, 2019.
  * [56] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
  * [57] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Learning to compare: Relation network for few-shot learning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pages 1199–1208, 2018.
  * [58] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * [59] James Townsend, Thomas Bird, and David Barber.  Practical lossless compression with latent variables using bits back coding.  In International Conference on Learning Representations, 2019.
  * [60] George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al.  An overview of the bioasq large-scale biomedical semantic indexing and question answering competition.  BMC bioinformatics, 16(1):138, 2015.
  * [61] Jesper E Van Engelen and Holger H Hoos.  A survey on semi-supervised learning.  Machine learning, 109(2):373–440, 2020.
  * [62] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [63] R.C. Veltkamp.  Shape matching: similarity measures and algorithms.  In Proceedings International Conference on Shape Modeling and Applications, pages 188–197, 2001.
  * [64] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  Advances in Neural Information Processing Systems (NeurIPS), 29:3630–3638, 2016.
  * [65] Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang.  Trec-covid: Constructing a pandemic information retrieval test collection.  SIGIR Forum, 54(1), February 2021.
  * [66] Henning Wachsmuth, Shahbaz Syed, and Benno Stein.  Retrieval of the best counterargument without prior topic knowledge.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 241–251. Association for Computational Linguistics, 2018.
  * [67] David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi.  Fact or fiction: Verifying scientific claims.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online, November 2020\. Association for Computational Linguistics.
  * [68] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, 2018.
  * [69] Siyu Wang, Jianfei Chen, Chongxuan Li, Jun Zhu, and Bo Zhang.  Fast lossless neural compression with integer-only discrete flows.  In International Conference on Machine Learning, pages 22562–22575. PMLR, 2022.
  * [70] Ian H. Witten, Radford M. Neal, and John G. Cleary.  Arithmetic coding for data compression.  Commun. ACM, 30(6):520–540, jun 1987.
  * [71] Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong.  Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering, 2023.
  * [72] Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto.  Luke: Deep contextualized entity representations with entity-aware self-attention.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6442–6454, 2020.
  * [73] Yaodong Yu, Heinrich Jiang, Dara Bahri, Hossein Mobahi, Seungyeon Kim, Ankit Singh Rawat, Andreas Veit, and Yi Ma.  An empirical study of pre-trained vision models on out-of-distribution generalization.  In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
  * [74] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun.  Character-level convolutional networks for text classification.  In NIPS, 2015.
  * [75] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh.  Calibrate before use: Improving few-shot performance of language models.  In International Conference on Machine Learning, pages 12697–12706. PMLR, 2021.
  * [76] Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho.  When does pretraining help?: assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings.  Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, 2021.
  * [77] Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  2015 IEEE International Conference on Computer Vision (ICCV), pages 19–27, 2015.
  * [78] A Zvonkin and L Levin.  The complexity of finite objects and the development of the concepts of information and randomness by means of the theory of algorithms.  Russian Mathematical Surveys, 25:83, 10 2007.

##  Appendix A A Theory of Information Distance

The fundamental component of the information distance is the conditional
Kolmogorov complexity. Formally, it’s defined as follows:

######  Definition 1 (Conditional Kolmogorov Complexity).

$K(x|y)=\min\\{|p|:U(p|y)=x\\}$,

where $p$ is the program that converts $y$ to $x$. Put simply, Definition 1
defines the size of the shortest program with respect to Turing Machine $U$
that generates $x$ given $y$.

######  Definition 2 (Information Distance).

$\mathcal{E}(x,y)=\max\\{K(x|y),K(y|x)\\}$.

Definition 2 is the formal definition of the information distance. It
describes the shortest program length that can convert from $x$ to $y$ or $y$
to $x$. To compare across different objects, we normalize the information
distance defined as $\mathcal{M}_{max}(x,y)$ in Equation(4).

######  Theorem 1.

$\mathcal{E}(x,y)$ is an admissible distance metric. It is minimal in the
sense that for every admissible distance $D$, we have $\mathcal{E}(x,y)\leq
D(x,y)+c$ where $c$ is a constant.

To better understand Theorem 1, we need two more definitions on metric and
admissible distance according to [3].

######  Definition 3 (Admissible Distance).

A function $D:\Omega\times\Omega\rightarrow\mathbb{R}^{+}$ is an admissible
distance if for every pair of objects $x,y\in\Omega$, the distance $D(x,y)$ is
computable, symmetric and satisfies the density condition
$\sum_{y}2^{-D(x,y)}\leq 1$.

######  Definition 4 (Metric).

A distance function $D:\Omega\times\Omega\rightarrow\mathbb{R}^{+}$ is a
metric if it satisfies the following three criteria for any $x,y,z\in\Omega$,
where $\Omega$ is a non-empty set, $\mathbb{R}^{+}$ represents the set of non-
negative real number:

  1. 1.

Identity: $D(x,y)=0$ iff $x=y$

  2. 2.

Symmetry: $D(x,y)=D(y,x)$

  3. 3.

Triangle Inequality: $D(x,y)\leq D(x,z)+D(z,y)$

We now prove Theorem 1 based on Lemma 1 following [3].

######  Lemma 1.

For every upper-semicomputable function $f(x,y)$, satisfying
$\sum_{y}2^{-f(x,y)}\leq 1$, we have $K(y|x)<f(x,y)$.

###### Proof.

To prove that $\mathcal{E}(x,y)$ is a metric, we show it satisfies metric
(in)equalities. We can infer the non-negativity and symmetry directly from the
definition $\mathcal{E}(x,y)=\max\\{K(x|y),K(y|x)\\}$. For triangle
inequality, given $x,y,z$, without loss of generality, let
$\mathcal{E}(x,z)=K(z|x)$. By the self-limiting property, we have

$\begin{split}\mathcal{E}(x,z)&=K(z|x)<K(y,z|x)<K(y|x)+K(z|x,y)\\\
&<K(y|x)+K(z|y)\leq\mathcal{E}(x,y)+\mathcal{E}(y,z).\end{split}$ (7)

To prove $\mathcal{E}(x,y)$ is admissible, we show it satisfies density
requirement:

$\sum\limits_{y:y\neq x}2^{-\mathcal{E}(x,y)}\leq\sum\limits_{y:y\neq
x}2^{-K(y|x)}\leq 1.$ (8)

The second inequality is due to Kraft’s inequality for prefix codes.
To prove the minimality, as for every admissible distance metric $D(x,y)$, it
satisfies $\sum\limits_{y:y\neq x}2^{D(x,y)}\leq 1$. According to Lemma 1, we
have $K(y|x)<D(x,y)$ and $K(x|y)<D(y,x)$. ∎

Despite of many successful applications, the $\mathcal{M}_{max}$ and
$\mathcal{M}_{mean}$ introduced in Equation(4) and Equation(6) face several
major practical problems. One of the major problems is the validity of
triangle inequality in the condition of metric. Fagin and Stockmeyer [16] give
an example of partial pattern matching where the triangle inequality does not
hold. Veltkamp [63] puts it vividly that under the partial matching, the
distance between a man and a horse is larger than the sum of the distances
between a man and a centaur and between a centaur and a horse, respectively.
The QA problems often depend on partial information and are in a similar
situation as partial pattern matching.

Li [31] defines the cost of conversion between $x$ and $y$ with respect to a
universal Turing machine U as:

######  Definition 5.

$\mathcal{E}_{min}(x,y)=min\\{|p|:U(x,p,r)=y,U(y,p,q)=x,|p|+|q|+|r|\leq\mathcal{E}(x,y)\\}$

Furthermore, the following theorem is proven which establishes a justifiable
alternative information distance measure.

######  Theorem 2.

$\mathcal{E}_{min}(x,y)=\min\\{K(x|y),K(y|x)\\}$.

While $\mathcal{E}_{min}(x,y)$ is symmetric and positive, it does not satisfy
the triangle inequality. But it is also universal as $\mathcal{E}(x,y)$ is
universal. Similar to normalizing $\mathcal{E}(x,y)$, we also normalize
$\mathcal{E}_{min}(x,y)$, which give us $\mathcal{M}_{min}(x,y)$ as shown in
Equation(5).

##  Appendix B A Theory of Human-Like Few-shot Learning

We follow the definition in [24] and consider the following formalization of
few-shot learning:

######  Definition 6 (Few-Shot Learning).

Consider a universe $\Omega$, partitioned into H disjoint concept classes:
$\mathcal{C}_{h},h=1,2,\dots,H$. Few-shot ($k$-shot including zero-shot)
learning can be described as follows:

  1. 1.

Given $n$ unlabelled samples $y_{1},\cdots,y_{n}$ in or outside of $\Omega$;

  2. 2.

Given $k$-labeled examples for each class $C_{h}$, for small $k$ ($k=0$ when
the setting is for zero-shot);

  3. 3.

The algorithm learns $\mathcal{C}_{h},h=1,2,\cdots,H$ using the metric
$\mathcal{M}$ with $n$ unlabeled samples and $k$ labeled samples, minimizing
the objective function:

$\sum^{H}_{h=1}\sum^{|\mathcal{C}_{h}|}_{i=1}\mathcal{M}(x_{i},core_{h})|y_{1},\cdots,y_{n},x_{i}\in\mathcal{C}_{h},$
(9)

where $core_{h}=\phi(k\text{ samples of }\mathcal{C}_{h})$ representing a
transformed representation of the k labeled samples from $\mathcal{C}_{h}$.

For zero-shot learning, $core_{h}$ is typically another sample (i.e. to
determine if they are from the same class) or description of the query. For
one-shot learning, it can be the single labeled dataset.

In order to incorporate the information of unlabeled data, we can extend the
definition by adding distribution learned from any data sources available
including $n$ unlabeled data and even external datasets, leading to:

$\sum^{H}_{h=1}\sum^{|\mathcal{C}_{h}|}_{i=1}\mathcal{M}(x_{i},x_{h}|\mathcal{H}(y_{1},\cdots,y_{n})),$
(10)

where $\mathcal{H}(y_{1},\cdots,y_{n})$ is a pre-trained model of
$y_{1},\cdots,y_{n}$, capturing the distribution.

We derive the optimal $\mathcal{M}$ from von-Neuman-Landauer Principle.

######  Definition 7 (von-Neuman-Landauer Principle).

Irreversibly processing 1 bit of information costs 1kT; reversible computation
is free.

Then for two objects $x$, $y$, the minimum energy needed to convert between x
and y is:

$\mathcal{E}_{U}(x,y)=min\\{|p|:U(x,p)=y,U(y,p)=x\\},$ (11)

where U is a universal Turing machine, assuming Church-Turing thesis. To
interpret, $\mathcal{E}(x,y)$ is the length of the shortest program that
reversibly converts between $x$ and $y$. These bits used in the shortest
program $p$ when they are erased will cost $|p|kT$ of energy, according to the
John von Neuman and Rolf Landuaer’s law. It’s also proved in [3] that:

$\mathcal{E}_{U}(x,y)=\mathcal{E}(x,y)+O(1).$ (12)

As we’ve proved in Appendix A that $\mathcal{E}(x,y)$ is the optimal
information distance, we can rely on the adoption of $\mathcal{E}(x,y)$ to,
for example, $\mathcal{M}_{max}$ to guarantee the optimality of finding
$\mathcal{M}$.

##  Appendix C Adaptive Arithmetic Coding using GPT

1:Input: $T=\\{t_{0},t_{1},t_{2},...,t_{n}\\}$ where $t_{0}$ is the EOS token,
GPT model $\phi$.

2:Initialize range $I_{low}=0,I_{high}=1$

3:for $t_{i}\in T,i=1,\cdots,(n+1)$ do

4: Obtain $F_{i}(t_{i})$ and $P_{i}(t_{i})$ based on $\phi(t_{1:(i-1)})$

5: $I_{low-previous}\leftarrow I_{low}$

6: $I_{high-previous}\leftarrow I_{high}$

7: $I_{low}\leftarrow I_{low-previous}+(I_{high-previous}-I_{low-
previous})*F_{i}(t_{i})$

8: $I_{high}\leftarrow I_{low-previous}+(I_{high-previous}-I_{low-
previous})*(F_{i}(t_{i})+P_{i}(t_{i}))$

9:end for

10:Output a single number $I_{low}\leq r<I_{high}$ as the encoded text.

Algorithm 1 GPT-AC Encoding

In practice, since we cannot have float number with infinite precision. We
always output the left several bits and shift the current interval by same
bits once we reach the float number overflow limit.

1:Input: encoded message $r$, GPT model $\phi$.

2:Initialize decoded text $D=\\{t_{0}\\}$ where $t_{0}$ is an EOS token. $i=0$

3:while $i=0$ or $t_{i}$ is not EOS token do

4: Calculate $P_{i+1}=\phi(D)$

5: Find index $j$ such that $F_{i+1}(v_{j})\leq
r<F_{i+1}(v_{j})+P_{i+1}(v_{j})$ where $v_{i}$ represents the $i$-th token in
GPT vocabulary

6: Update $t_{i+1}\leftarrow v_{j}$

7: Update $D=D+\\{t_{i+1}\\}$

8: Update $r\leftarrow\frac{r-F_{i+1}(v_{j})}{P_{i+1}(v_{j})}$

9: Update $i\leftarrow i+1$

10:end while

11:Output D as the decoded message

Algorithm 2 GPT-AC Decoding

##  Appendix D Experiment Details

We conduct our experiment on two 16G Tesla V100 GPUs, with framework and
implementation adopt from HuggingFace Transformers with PyTorch. Each
experiment costs from 20min to 3h GPU time depends on the dataset size. We
finish all the experiment with no more than 300 GPU hours(about 6 days).

For fine-tuning GPT and BERT with classification layers, we search within the
following hyper-parameters:

  * •

Learning Rate: 1e-5, we also tried 1e-4, 5e-5, 5e-6, 1e-6, 1e-5 gives the best
learning performance

  * •

Batch Size: since we are finetune with one-shot, the we select the batch size
within 2, 4, 8, and restricted by the class number.

  * •

Epochs: search between 500, 1000, 1500, 2000, and later the learning curve
converges.

  * •

Max Sequence Length: GPT2: 1024, BERT: 512

We also list all the models card we used:

  * •

gpt-2 https://huggingface.co/gpt2, gpt2-medium
https://huggingface.co/gpt2-medium, gpt2-large
https://huggingface.co/gpt2-large, gpt2-xl https://huggingface.co/gpt2-xl,

  * •

BERT https://huggingface.co/bert-base-uncased

  * •

Sentence BERT: https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2

For all the dataset, we take the pre-processed datasets from hugging face,
except for the Medical dataset. For the Medical dataset, we follow the
official repo: https://github.com/sebischair/Medical-Abstracts-TC-Corpus.

##  Appendix E GPT In-Context Learning

Table 6: Zero-Shot Prompts CaseHold Prompt |  Citing: {citing}. Holding 0: {holding_0}. Holding 1: {holding_1}. Holding 2: {holding_2}. Holding 3: {holding_3}. Holding 4: {holding_4}. Which holding is correct (0, 1, 2, 3, or 4)? Answer:
---|---
PIQA Prompt |  Goal: {goal}. Option 0: {option_0}. Option 1: {option_1}. Which option is correct (0 or 1)? Answer:
Table 7: One-Shot Prompt AG News Prompt |  Please classify text input into one of the following categories: World, Sports, Business, and Science/Technology. Here are some examples: Input: {Example 1 Text}, Label: {Example 1 Label} Input: {Example 2 Text}, Label: {Example 2 Label} Input: {Testing Text}, Label:
---|---
Label Token Mapping |  ’World’: 10603, ’Sports’: 18153, ’Business’: 24749, ’Science/Technology’: 26959
Medical Prompt |  Please classify text input into one of the following categories: World, Sports, Business, and Science/Technology. Here are some examples: Input: {Example 1 Text}, Label: {Example 1 Label} Input: {Example 2 Text}, Label: {Example 2 Label} Input: {Testing Text}, Label:
Label Token Mapping |  ’Neoplasms’: 10603, ’Digestive system diseases’: 18153, ’Nervous system diseases’: 24749, ’Cardiovascular diseases’: 26959, ’Pathological conditions’: 0
SST5 Prompt |  Please classify text input into one of the following categories: World, Sports, Business, and Science/Technology. Here are some examples: Input: {Example 1 Text}, Label: {Example 1 Label} Input: {Example 2 Text}, Label: {Example 2 Label} Input: {Testing Text}, Label:
Label Token Mapping |  ’very negative’: 10603, ’negative’: 18153, ’neutral’: 24749, ’positive’: 26959, ’very positive’: 0


* [Hir1976] Hirsch, M.; Differential topology; GTM 33, Springer (1976)
  * [HN1971] Hoghe-Nlend, H.; Théorie des bornologies et applications Lect. Notes in Math. 273 (1971)
  * [Ho1971] Hörmander,L.; Fourier integral operators. I; Acta Mathematica 127 79-189 (1971)
  * [HCR2015] Huber, M.; Campagnari, D.; Reinhardt, H.; Vertex functions of Coulomb gauge Yang–Mills theory Phys. Rev. D 91, 025014 (2015)
  * [Lim2012] Non-abelian gauge theory for Chern-Simons path integral on $\mathbb{R}^{3}$ , Journal of Knot Theory and its Ramifications 21 no. 4., articleID1250039 (24p) (2012)
  * [HM1983] Hughes, T.; Marsden, J.E.; Mathematical foundations of elasticity Prentice-Hall Civil Engineering and Engineering Mechanics Series. Englewood Cliffs, New Jersey: Prentice-Hall, Inc. XVIII, (1983).
  * [IZ1985] Iglesias-Zemmour, P.; Fibrations difféologiques et homotopie, PhD thesis, université de Provence (1985)
  * [IZ1987] Iglesias-Zemmour, P.; Connexions et difféologie Aspects dynamiques et topologiques des groupes infinis de transformation de la mécanique Travaux en cours 25, Hermann (1987), 61-78
  * [IZ2013] Iglesias-Zemmour, P.; Diffeology Mathematical Surveys and Monographs 185 (2013)
  * [McI2011] Mc Intosh, I.; The quaternionic KP hierarchy and conformally immersed 2-tori on the 4-sphere. Tohoku Math. J. 63 183-215 (2011)
  * [Ism1996] Ismaginov, O.V; Representations of infinite dimensional groups Translations of mathematical monographs; American mathematical society (1996)
  * [Ka1989] Kassel, Ch.; Le résidu non commutatif (d’après M. Wodzicki) Séminaire Bourbaki, Vol. 1988/89. Astérisque 177-178, Exp. No. 708, 199-229 (1989)
  * [KW2009] Khesin, B.A. and Wendt, R.; “The geometry of infinite-dimensional groups” (2009). Springer-Verlag, Berlin.
  * [KZ1995] Khesin, B.A. and Zakharevich, I.; Poisson-Lie groups of pseudodifferential symbols, Comm. Math. Phys. 171, no. 3 (1995), 475–530.
  * [KN63-69] Kobayashi, S.; Nomizu, K.; Fundations of differential geometry; 1,2 , Wiley interscience (1963-1969)
  * [Ko1993] Koczkodaj, W.W., A new definition of consistency of pairwise comparisons, Mathematical and Computer Modelling 7 79-84 (1993)
  * [KKL2014] Koczkodaj, W.W.; Kulakowski, K.; Ligeza, A., On the quality evaluation of scientific entities in Poland Supported by consistency-driven pairwise comparisons method, Scientometrics, 99 no3 911-926 (2014)
  * [KoSza2010] Koczkodaj, W.W.; Szarek, S.; On distance-based inconsistency reduction algorithms for pairwise comparisons Logic J. of the IGPL 18 no 6 859-869 (2010)
  * [KoSzw2014] Koczkodaj, W.W.; Szwarc, R.; Axiomatization of Inconsistency Indicators for Pairwise Comparisons, Fundamenta Informaticae, 132(4): 485-500, 2014.
  * [KoSzy2015] Koczkodaj, W.W.; Szybowski, J.; Axiomatization of Inconsistency Indicators for Pairwise Comparisons Matrices Revisited , arXiv:1509.03781
  * [KMS1993] Kolar, I.; Michor, P.W.; Slovak, J.; Natural operations in differential geometry; Springer (1993)
  * [KV1994] Kontsevich, M.; Vishik, S.; Determinants of elliptic pseudo-differential operators Max Plank Institut fur Mathematik, Bonn, Germany, preprint n. 94-30 (1994)
  * [KV1995] Kontsevich, M.; Vishik, S.; Geometry of determinants of elliptic operators. Functional analysis on the eve of the 21st century, Vol. 1 (New Brunswick, NJ, 1993), Progr. Math. 131,173-197 (1995)
  * [KK1991] Kravchenko, O.S.; Khesin, B.A.; A central extension of the algebra of pseudo-differential symbols Funct. Anal. Appl. 25 152-154 (1991)
  * [KM2000] Kriegl, A.; Michor, P.W.; The convenient setting for global analysis Math. surveys and monographs 53, American Mathematical society, Providence, USA. (2000)
  * [KMR2015] Kriegl, A.; Michor, P. W.; Rainer, A.; An exotic zoo of diffeomorphism groups on $\mathbb{R}^{n}.$ Ann. Glob. Anal. Geom. 47, 2 (2015), 179-222
  * [Ku1998] Kubo, F.; Non-commutative Poisson algebra structures on affine Kac-Moody algebras. J. Pure and Applied Algebra 126 (1998), 267–286.
  * [Kub2020] Kubrusly, C. S.; Spectral theory of bounded linear operators. Birkhauser 249 p. (2020).
  * [K1965] Kuiper, N.; The homotopy type of the unitary group of Hilbert spaces Topology 3 19-30 (1965)
  * [Ku2000] Kuperschmidt, B.A.; KP or mKP: Mathematics of Lagrangian, Hamiltonian and Integrable Systems. Math. Surv. Monographs 78 (2000)
  * [LLA2012] Lahby, M.; Leghris, C.; Adib, A.; Network Selection Decision based on handover history in Heterogeneous Wireless Networks; International Journal of Computer Science and Telecommunications 3 no 2, (2012) 21-25
  * [Lan1985] Lang, S.; Differentiable manifolds 2nd edition, Springer-Verlag, New-York (1985)
  * [Lax1968] Lax, P.; Integrals of nonlinear equations of evolution and solitary waves. Comm. Pure Appl. Math. 21 no5 467-490 (1968)
  * [dLS2009] C. De Lellis, L. Székelyhidi Jr.; The Euler equations as a differential inclusion. Ann. of Math. (2) 170 no. 3, 1417–1436 (2009)
  * [dLS2010] C. De Lellis, L. Székelyhidi Jr.; On admissibility criteria for weak solutions of the Euler equations. Arch. Ration. Mech. Anal. 195 no. 1, 225–260 (2010)
  * [dLS2012] C. De Lellis, L. Székelyhidi Jr.; The h-principle and the equations of fluid dynamics. Bull. Amer. Math. Soc. 49, 347-375 (2012)
  * [dLS2021] C. De Lellis, L. Székelyhidi Jr.; Weak stability and closure in turbulence arXiv:2108.01597
  * [Le1998] Lesch, M.; On the non commutative residue for pseudo-differential operators with log-polyhomogeneous symbol Ann. Glob. Anal. Geom. 17 151-187 (1998)
  * [LP2007] Lescure, Jean-Marie; Paycha, Sylvie Uniqueness of multiplicative determinants on elliptic pseudodifferential operators. Proc. Lond. Math. Soc. (3) 94 No. 3, 772-812 (2007)
  * [Lesl2003] Leslie, J.; On a Diffeological Group Realization of certain Generalized symmetrizable Kac-Moody Lie Algebras J. Lie Theory 13 (2003), 427-442
  * [Lich1956] Lichnerowicz, A. Théorie globale des connexions et des groupes d’holonomie ed. Cremonese, Roma (1956)
  * [Marsch1998] Marschall, J.; Pseudo-differential operators with coefficients in Sobolev spaces; Trans. AMS  307, no1 (1988) 335-361.
  * [MRT2014] Maeda, Y.; Rosenberg, S. Torres-Ardila, F. The geometry of loop spaces II: characteristic classes arXiv:1407.2491 (2015 for v3)
  * [Maj1995] Majid, S.; Foundations of quantum groups Cambridge University Press (1995)
  * [Mal1972] Malliavin, P.; Géométrie différentielle intrinsèque Hermann (1972)
  * [MaRob1995] Marion, J.; Robart, T.; Regular Fréchet Lie groups of invertible elements in some inverse limits of unital involutive Banach algebras Georgian Math. J. 2 no 4 425-444 (1995)
  * [MR1999] Marsden, J. and Ratiu, T.; “Introduction to mechanics and symmetry. A basic exposition of classical mechanical systems. Second edition” (1999). Texts in Applied Mathematics, 17. Springer-Verlag, New York.
  * [Mich1980] Michor, P.; Manifolds of smooth maps III: The principal bundle of embeddings of a non compact smooth manifold. Cahiers Topologie Geometrie Differentielle 21 325-337 (1980)
  * [Mick1994] Mickelsson, J.; Wodzicki residue and anomalies on current algebras  Integrable models and strings A. Alekseev and al. eds. Lecture notes in Physics 436, Springer (1994)
  * [Mick1989] Mickelsson, J.; Current algebras and groups Plenum monographs in non linear physics (1989)
  * [Mick1988] Mickelsson, J.; Current algebra representation for the 3+1 dimensional Dirac-Yang-Mills theory Comm. Math. Phys. 117 261 (1988)
  * [MP2007] Mickelsson, J.; Paycha, S.; Renormalized Chern-Weil forms associated with families of Dirac operators J. Geom. Phys. 57, no9, 1789-1814 (2007)
  * [MR1988] Mickelsson, J.; Rajeev, S.; Current algebras and Determinant bundles for infinite dimensional grassmannians Comm. Math. Phys. 116 365 (1988)
  * [Mil1984] Milnor, J.; Remarks on infinite dimensional Lie groups; Proc. Summer school on Quantum Gravity (Les Houches, 1983), ed. DeWitt, B., North-Holland, Amsterdam, (1984) 1008-1056
  * [Mil1963] Milnor, J.; Morse theory Based on lecture notes by M. Spivak and R. Wells.; Annals of Mathematics Studies. 51; Princeton University Press (1963)
  * [Mil1956-1] Milnor, J.; Construction of univeral bundles, I _Ann. of Math. (2)_ , 63 (1956), 272–284.
  * [Mil1956-2] John Milnor, “Construction of universal bundles, II”, _Ann. of Math. (2)_ , 63 (1956), 430–436.
  * [Mis1997] Misiołek, G.; The exponential map on the free loop space is Fredholm. Geom. Funct. Anal. 7, No. 5, 954-969 (1997).
  * [Mis1999] Misiolek, G.; Exponential maps of Sobolev metrics on loop groups. Proc. Am. Math. Soc. 127, No. 8, 2475-2482 (1999) .
  * [MP2009] Misiołek, G.; Preston, S. C.; Fredholm properties of Riemannian exponential maps on diffeomorphism groups. Invent. Math. 179, No. 1, 191-227 (2010).
  * [Mo2008] Molitor, M.; La Grassmannienne Non-linéaire comme Variété Fréchétique Homogène; J. Lie Theory 18 no. 3, 523–539 (2008)
  * [MS2016] Montaldi, J.; Smolyanov, O.G; Feynman path integrals and Lebesgue-Feynman measures Dokl. Math. 96, No. 1, 368-372 (2017); translation from Dokl. Akad. Nauk, Ross. Akad. Nauk 475, No. 5, 490-495 (2016).
  * [Mu1984] Mulase, M.; Complete integrability of the Kadomtsev-Petviashvili equation Adv. Math. 54 (1984) 57-66
  * [Mu1988] Mulase, M.; Solvability of the super KP equation and a generalization of the Birkhoff decomposition Invent. Math. 92 (1988), 1-46
  * [Mu1983] Mulase, M.; Geometry of soliton equations; MSRI preprint (1983)
  * [Nee2007] Neeb, K-H.; Towards a Lie theory of locally convex groups Japanes J. Math. 1 (2006), 291-468
  * [Olv1993] Olver, P.J.; Applications of Lie groups to differential equations (2nd edition) GTM 107, Springer (1993)
  * [Om1997] Omori, H.; Infinite dimensional Lie groups AMS translations of mathematical monographs 158 (1997)
  * [Om1970] Omori, H.; On the group of diffeomorphisms on a compact manifold; 1970 Global Analysis (Proc. Sympos. Pure Math., Vol. XV, Berkeley, Calif., 1968) 167-183 Amer. Math. Soc., Providence, R.I.
  * [Om1981] Omori, H.; A remark on non-enlargable Lie algebras J. Math. Soc. Japan 33 no4 707-710 (1981)
  * [Om2007] Omori, H.; Toward geometric Quantum theory Progress in Math 252 213-252 (2007)
  * [OmMMY2007] Omori, H.; Maeda, Y.; Myazaki, N. Yoshioka, A.; Geometric objects in an approach to quantum geometry Progress in Math 252 303-324 (2007)
  * [OMY1] Omori, H.; Maeda, Y.; Yoshioka, A.; On regular Fréchet Lie groups, I; Some differential geometric expressions of Fourier integral operators on a Riemannian manifold Tokyo J. Math. 3, no 2, 353-390 (1980)
  * [OMY1994] Omori, H; Maeda, Y; Yoshioka, A.; A Poincaré-birkhoff-Witt theorem for infinite dimensional Lie algebras J. Math. Soc. Japan 46 no1 25-50 (1994)
  * [OMY2] Omori, H; Maeda, Y; Yoshioka, A.; On regular Fréchet Lie groups II; Composition rules of Fourier integral operatorson a Riemannian manifold Tokyo J. Math. 4, no 2, 221-253 (1981)
  * [OMYK3] Omori, H; Maeda, Y; Yoshioka, A.; Kobayashi, O.; On regular Fréchet Lie groups III; A second cohomology class related to theLie algebra of pseudo-differential oprators of order 1 Tokyo J. Math. 4, no 2, 255-277 (1981)
  * [OMYK4] Omori, H; Maeda, Y; Yoshioka, A.; Kobayashi, O.; On regular Fréchet Lie groups IV; Definition and fundamentaltheorems Tokyo J. Math. 5, no 2, 365-398 (1981)
  * [OMYK5] Omori, H; Maeda, Y; Yoshioka, A.; Kobayashi, O.; On regular Fréchet Lie groups V; Several basic properties Tokyo J. Math. 6, no 1, 39-64 (1983)
  * [OMYK6] Omori, H; Maeda, Y; Yoshioka, A.; Kobayashi, O.; Infinite dimensional Lie groups that appear in general relativity Tokyo J. Math. 6, no 2, 217-246 (1983)
  * [OMYK7] Omori, H; Maeda, Y; Yoshioka, A.; Kobayashi, O.; On regular Fréchet Lie groups VII; The group generated by pseudo-differential operators of negative order Tokyo J. Math. 7, no 2, 315-336 (1984)
  * [OMYK8] Omori, H; Maeda, Y; Yoshioka, A.; Kobayashi, O.; Primordial operators and Fourier integral operators Tokyo J. Math. 8, no 1, 1-47 (1985)
  * [Pat1988] Paterson, A.T.; Amenability Math. surveys and Monographs 29, Amer. Math. Soc., Provodence, R.I. (1988)
  * [Pay2001] Paycha, S.; Renormalized traces as a looking glass into infinite dimensional geometry Infin. Dimens. Anal. Quantum Probab. Relat. Top. 4, no2, 221-266 (2001)
  * [PR2004] Paycha, S.; Rosenberg, S.; Traces and Characteristic classes on loop spaces in IRMA lectures in mathematics and theorical physics ed. T. Wurzbacher (2004) .
  * [Pay2012] Paycha, S; Regularised integrals, sums and traces. An analytic point of view. University Lecture Series 59, AMS (2012).
  * [Pay2013] Paycha, S.; Paths towards an extension of Chern-Weil calculus to a class of infinite dimensional vector bundles. Geometric and topological methods for quantum field theory, 81–143, Cambridge Univ. Press, Cambridge, (2013)
  * [Pes2006] Pestov, V.; Dynamics of Infinite-Dimensional Groups : the Ramsey-Dvoretzky-Milman phenomenon University Lecture Series 40,Amer. Math. Soc. (2006)
  * [Pen1970] Penot, J.-P.; Sur le théorème de Frobenius ; Bull. Soc. math. France  98 (1970), 47-80
  * [Per1990] Perelomov, A.M.; “Integrable systems of classical mechanics and Lie algebras” (1990) Birkhäuser Verlag, Berlin.
  * [PS2021] Pitcho, J.; Sorella, M.; Almost everywhere non-uniqueness of integral curves for divergence-free Sobolev vector fields. arXiv:2108.03194
  * [PS1988] Pressley, A.; Segal, G.; Loop Groups Oxford Univ. Press (1988)
  * [Prok1956] Prokhorov, Yu.V.; Convergence of random processes and limit theorems in probability theory. Theor. Prob. Appl. 1,ézéé 157-214 (1956)
  * [Rad1991] O.A.Radul; Lie albegras of differential operators, their central extensions, and W-algebras Funct. Anal. Appl. 25, 25-39 (1991)
  * [RatSc1981] Ratiu, T.; Schmid, R.; The differentiable structure of three remarkable diffeomorphism groups; Math Z. 177, 81-100 (1981)
  * [RS1981] Reyman, A.G.; Semenov-Tian-Shansky, M.A.; Reduction of Hamiltonian Systems, Affine Lie Algebras and Lax Equations II. Invent. math. 63 (1981), 423–432.
  * [Re1997] Reinhardt, H.; Yang-Mills in axial gauge Phys.Rev. D 55 2331-2346 (1997)
  * [RSF1985] Reyman, A.; Semenov-Tyan-Shanskii, M.A.; Faddeev, L.; Quantum anomalies and cocycles on gauge groups Funct. Anal. Appl.  18 319 (1985)
  * [Rob1997] Robart, T.; Sur l’intégrabilité des sous-algèbres de Lie en dimension infinie; Can. J. Math. 49 (4) (1997), 820-839
  * [RV2014] Rovelli, C. Vidotto, F. Covariant Quantum Loop Gravity Cambridge University Press (2014)
  * [Saa1977] Saaty, T.; A scaling methods for priorities in hierarchical structures; J. Math. Psychol. 15 (1977) 234-281
  * [SGO2021] di Salvo, R.; Gorgone, M.; Oliveri, F.; A consistent approach to approximate symmetries Nonlinear Dyn. 91, 371–386 (2018)
  * [SavSt2013] Savin, A.Yu.; Sternin, B.Yu. Uniformization of nonlocal elliptic operators and KK-theory; Russ. J. Math. Phys. 20, no. 3, 345-359 (2013).
  * [Sche1993] Scheffer, V.; An inviscid flow with compact support in space-time. J. Geom. Anal. 3 no 4 , 343–401 (1993)
  * [Shn1997] Shnilerman, A.; On the nonuniqueness of weak solution of the Euler equation. Comm. Pure Appl. Math. 50 no 12, 1261–1286 (1997)
  * [Shn2000] Shnilerman, A.; Weak solutions with decreasing energy of incompressible Euler equations. Comm. Math. Phys. 210 no 3, 541–603 (2000)
  * [Sch1959] Schwinger, J.; Field theory of commutators; Phys. Rev. Lett. 3, 296-297 (1959)
  * [Sco2010] Scott, S.; Traces and determinants of pseudodifferential operators; OUP (2010)
  * [SchW1999] Schaefer, H. H.; Wolff, M. P.; Topological vector spaces Second edition. Graduate Texts in Mathematics 3 Springer-Verlag, New York (1999)
  * [See1967] Seeley, R.T.; Complex powers of an elliptic operator; Proc. Symp. Pure Math. 10 (1967) 288-307
  * [SW1985] Segal, G.; Wilson, G.; Loop groups and equations of KdV type Publ. Math. IHES 61 5 (1985)
  * [SSSA2000] Sen, S.; Sen, S.; Sexton, J.C.; Adams, D.H.; A geometric discretisation scheme applied to the Abelian Chern-Simons theory Phys.Rev. E 61 3174-3185 (2000)
  * [Sou1985] Souriau, J.M.; Un algorithme générateur de structures quantiques; Astérisque, Hors Série, (1985) 341-399
  * [T2006] Terng, Chuu-Lian Applications of loop group factorization to geometric soliton equations. Sanz-Solé, Marta (ed.) et al., Proceedings of the international congress of mathematicians (ICM), Madrid, Spain, August 22–30, 2006. Volume II: Invited lectures. Zürich: European Mathematical Society (EMS) 927-950 (2006).
  * [Ue1985] Ueno, K.; Analytic and algebraic aspects of the Kadomtsev-Petviashvili hierarchy from the viewpoint of the universal Grassmann manifold;  Infinite Dimensiona groups with applications, MSRI publications 4, Springer (1985), 335-353
  * [Vas1978] Vassiliou, E, On the infinite dimensional holonomy theorem, Bull. Soc. Roy. Sc. Liège, 9-10, 223-228, (1978)
  * [Vil2006] Villani, C.; Optimal transport, old and new Springer (2006)
  * [Vil2008] Villani, C.; Paradoxe de Scheffer-Shnirelman revu sous l’angle de l’intégration convexe (d’après C. De Lellis, L. Székelyhidi). Séminaire Bourbaki Exp. 1001 (November 2008)
  * [Vin2013] Vinogradov, A.; What are symmetries of nonlinear PDEs and what are they themselves? arXiv:1308.5861
  * [Wan1983] Watanabe, Y.; Hamiltonian structure of Sato’s hierarchy of KP equations and a coadjoint orbit of a certain formal Lie group. Lett. Math. Phys. 7 (1983), 99–106.
  * [Wan1984] Watanabe, Y.; Hamiltonian structure of M. Sato’s hierarchy of Kadomtsev–Petviashvili equation. Ann. Mat. Pura Appl. (4) 136 (1984), 77–93.
  * [Wa2012] Watts, J.; Diffeologies, differentiable spaces and symplectic geometry PhD thesis, university of Toronto (2012) arXiv:1208.3634
  * [We2021] Welker, K.; Suitable Spaces for Shape Optimization. Appl. Math. Optim. 84, Suppl. 1, S869-S902 (2021)
  * [Wh1957] Whitney, H., Geometric Integration Theory, Princeton University Press, Princeton, NJ, 1957.
  * [Wid1980] Widom, H.; A complete symbolic calculus for pseudo-differential operators; Bull. Sc. Math. 2e serie 104 (1980) 19-63
  * [Wod1984] Wodzicki, M.; Local invariants in spectral asymmetry Inv. Math. 75, 143-178 (1984)
  * [Wur1995] Wurzbacher, T.; Symplectic geometry of the loop space of a Riemannian manifold Journal of Geometry and Physics 16, 345-384 (1995)
  * [Y1999] Young, K.; Foreign exchange markets as a lattice gauge theory, Am. J. Phys. 67 no 10 862–868 (1999)