# Transformer-based Models for Long-Form Document Matching: Challenges and
Empirical Analysis

Akshita Jha
Virginia Tech, Arlington, VA
<EMAIL_ADDRESS>
&Adithya Samavedhi
Virginia Tech, Arlington, VA
<EMAIL_ADDRESS>
&Vineeth Rakesh
InterDigital, CA
<EMAIL_ADDRESS>
Jaideep Chandrashekar
InterDigital, CA
<EMAIL_ADDRESS>
&Chandan K. Reddy
Virginia Tech, Arlington, VA
<EMAIL_ADDRESS>

###### Abstract

Recent advances in the area of long document matching have primarily focused
on using transformer-based models for long document encoding and matching.
There are two primary challenges associated with these models. Firstly, the
performance gain provided by transformer-based models comes at a steep cost –
both in terms of the required training time and the resource (memory and
energy) consumption. The second major limitation is their inability to handle
more than a pre-defined input token length at a time. In this work, we
empirically demonstrate the effectiveness of simple neural models (such as
feed-forward networks, and CNNs) and simple embeddings (like GloVe, and
Paragraph Vector) over transformer-based models on the task of document
matching. We show that simple models outperform the more complex BERT-based
models while taking significantly less training time, energy, and memory. The
simple models are also more robust to variations in document length and text
perturbations.

##  1 Introduction

Matching long documents (e.g.: research papers, Wikipedia articles, patents,
etc.) is an important task that can help understand the (dis)similarity
between documents for downstream tasks like long document search. The first
step towards better document matching is obtaining meaningful long document
representations. Recent advances in this area have primarily focused on using
transformer-based models for long document encoding and matching Beltagy et
al. (2020); Jha et al. (2022); Yang et al. (2020b); Zaheer et al. (2020). We
use the term transformers to mean pre-trained transformers. Despite promising
results, there are two primary challenges associated with such models. First,
the performance gain provided by the huge transformer-based language models
(LMs), like BERT Devlin et al. (2019), GPT-2 Radford et al. (2019), and
Longformer Beltagy et al. (2020) come at a steep cost – both in terms of the
required training time, and the resource (memory and energy) consumption. For
example, the smaller $BERT_{BASE}$ model has 110 million parameters, whereas
the bigger $BERT_{LARGE}$ model has a total of 340 million parameters and
fine-tuning a single $BERT_{BASE}$ model on GPU can take hours. The second
major limitation of transformer-based models is their inability to handle more
than a pre-defined input token length at a time (512 tokens for BERT, and 4096
tokens for Longformer). This is a big drawback as they cannot handle long
documents like research papers, patents, long articles, etc., without using
aggregation techniques Reimers and Gurevych (2019).

In this work, we empirically demonstrate that  embeddings obtained from GloVe
Pennington et al. (2014), and Paragraph Vectors Le and Mikolov (2014) along
with simple neural models, such as feed-forward networks, and CNNs, outperform
several transformer-based models for the document matching task. We define
these models as simple as they take significantly less time to train and
consume less memory and energy overall when compared to complex transformer-
based models. Our long document matching setting is fundamentally different
from long-form question answering and sentence similarity tasks. For the
latter tasks the query is ‘short’, unlike the long document matching task
where both the query and the target text are ‘long’. We experiment with three
different kinds of semi-structured long document datasets in English: (i) ACL
Anthology research papers, (ii) English Wikipedia articles, and (iii) Patents
from US Patent and Trademark Office (USPTO). Our primary contributions are
summarized as follows:

  * •

We provide insights into the challenges of using transformer-based models for
the task of long document matching. For this task, simple neural models are as
effective and take a fraction of the training time and resources to outperform
transformer-based models.

  * •

We provide insights into the best input embeddings for the simpler models in
this task.

  * •

We demonstrate that simple models are also more robust to changes in document
length and text perturbation.

  * •

We create benchmark long document datasets (by pre-processing ACL Anthology
2014 papers and Wikipedia articles) that will be made publicly available.

##  2 Related Work

Early work on long document matching focused on clustering techniques
Friburger et al. (2002); Huang et al. (2008); Strehl et al. (2000). Recently,
Guo et al. (2016) proposed a deep learning based architecture for ad-hoc
retrieval when comparing documents. Some works have also used convolutional
networks Hu et al. (2014); Pang et al. (2016); Yu et al. (2018), with
weighting mechanism Yang et al. (2016a) to generate a final query-document
score. Mitra et al. (2017) propose a combination model that uses weighted sum
representation-based and interaction-based results. Yang et al. (2016b)
propose HAN, a hierarchical attention network for document matching. Jiang et
al. (2019) propose SMASH, a multi-depth attention based hierarchical recurrent
neural network for long-document matching. However, Yang et al. (2020b) pre-
train SMITH, a transformer based hierarchical model for text matching that
outperforms SMASH across multiple datasets. Jha et al. (2022) use supervised
contrative learning for interpretable long document matching. We compare
several of these models on the required training time and resources.

A growing body of literature has used transformer based model for long
document encoding Child et al. (2019); Ho et al. (2019); Kitaev et al. (2019);
Liu and Lapata (2019); Qiu et al. (2020); Yang et al. (2020a). Longformer
Beltagy et al. (2020) adapts transformers to use an attention mechanism that
scales linearly with sequence length. Big bird Zaheer et al. (2020) uses a
sparse attention mechanism that reduces BERT’s quadratic dependency on the
sequence length to linear. CogLTX Ding et al. (2020) uses text blocks for
rehearsal and decay over key sentences to overcome the insufficient long-range
attentions in BERT. Transformer-XL Dai et al. (2019) and Compressive
Tranformers Rae et al. (2019) compress the transformers to use attentive
sequence over long text. Although promising, we demonstrate that transformer-
based models are not considerably better than simple neural models on the task
of long document matching.

##  3 Empirical Evaluation

Here we provide details of the simple and the transformer-based models and
present an empirical comparison between them based on their overall
performance, training time, resource consumption, and robustness on the
document matching task.

##### Task Formulation

We define the task of document matching as follows. Given a source document
$s$, and a set of target documents $D_{T}$, the goal is to estimate the
semantic match $\hat{y}=sim(s,t)$, where $t\in D_{T}$ for every document pair
$(s,t)$. Similar target documents will have a higher similarity score. The
document matching problem can be seen as a binary classification task, where
the model predicts $1$ for similar documents, and $0$ for dissimilar
documents. We use the term “matching" in the broad sense of document relevance
(see Appendix A.2). The models take as input a pair of documents (source and
target), and compute the cosine similarity between the encoded document
representations. If the cosine similarity is greater than a similarity
threshold $\theta$, they are considered similar; otherwise they are considered
dissimilar.

Figure 1: A schematic siamese comparison model | AAN | WIKI | PAT
---|---|---|---
Model | P | R | F1 | Acc | P | R | F1 | Acc | P | R | F1 | Acc
HAN-G | 0.504 | 0.881 | 0.641 | 0.607 | 0.566 | 0.584 | 0.575 | 0.775 | 0.609 | 0.848 | 0.709 | 0.522
DSSM-T | 0.768 | 0.809 | 0.787 | 0.780 | 0.823 | 0.939 | 0.877 | 0.869 | 0.869 | 0.957 | 0.911 | 0.905
DSSM-G | 0.550 | 0.541 | 0.545 | 0.549 | 0.966 | 0.986 | 0.975 | 0.975 | 0.992 | 0.998 | 0.995 | 0.995
DSSM-D | 0.852 | 0.763 | 0.805 | 0.815 | 0.933 | 0.984 | 0.958 | 0.957 | 0.841 | 0.959 | 0.896 | 0.949
ARC-I-G | 0.643 | 0.873 | 0.734 | 0.743 | 0.992 | 0.983 | 0.987 | 0.987 | 0.905 | 0.963 | 0.933 | 0.939
ARC-I-D | 0.841 | 0.763 | 0.800 | 0.809 | 0.987 | 0.985 | 0.986 | 0.986 | 0.967 | 0.958 | 0.962 | 0.983
BERT | 0.760 | 0.914 | 0.793 | 0.761 | 0.980 | 0.950 | 0.960 | 0.960 | 1.0 | 0.988 | 0.994 | 0.994
LONG | 0.681 | 0.833 | 0.749 | 0.773 | 0.974 | 0.960 | 0.967 | 0.967 | 1.0 | 0.984 | 0.992 | 0.992
SMITH | 0.726 | 0.565 | 0.635 | 0.676 | 0.949 | 0.982 | 0.965 | 0.963 | 0.892 | 0.939 | 0.865 | 0.939
Table 1: Performance on the document matching task up to the model’s maximum
allowed input token length ($512$ for BERT; $4096$ for Longformer, $>8000$ for
all other models). We experiment with Trigrams (T), GloVe (G), and Doc2Vec (D)
Embeddings as input for the simple neural models. The best performance is
highlighted in bold.

##### Models

We pick a representative set of models from different categories and compare
them by building their siamese versions (shown in Figure 1). The siamese
network has three primary components: (i) Input (Source and Target Document),
(ii) Document Encoder, and (iii) Loss Function. The source and target document
encoder networks share weights. We experiment with three simple neural models:
(i) DSSM: A simple feed-forward network Huang et al. (2013), (ii) ARC-I: A
CNN-based model Hu et al. (2014) , and (iii) HAN: An RNN-based Hierarchical
Attention Network Yang et al. (2016b) designed for long documents. Their
performance is compared with three state-of-the-art transformer-based models:
(i) BERT Devlin et al. (2019), (ii) LONG: Longformer Beltagy et al. (2020),
and (iii) SMITH: Siamese Multi-depth Transformer based Hierarchical Encoder
Yang et al. (2020b). We report the mean precision, recall, F1, and accuracy
over 5 folds for the best performing hyper-parameters. The code can be found
here: https://github.com/AkshitaJha/SimpleModelsforLongDocumentMatching.

##### Datasets

We follow the previous literature Yang et al. (2016b); Jiang et al. (2019);
Yang et al. (2020b) and experiment with the following three standard long
document datasets: (i) ACL Anthology Network Corpus
(AAN)111https://aan.how/download/#aanNetworkCorpus, (ii) Wikipedia Articles
(WIKI)222https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-
articles.xml.bz2, and (iii) USPTO Patents
(PAT)333https://github.com/google/patents-public-data. Each dataset consists
of balanced 15,000 pairs of documents with 50% of them being similar pairs,
and the remaining being dissimilar. The PAT dataset is an industry gold
standard but we will publicly release the pre-processed AAN and the WIKI
datasets (see Appendix A.2 for details). The dataset can be found here:
https://github.com/AkshitaJha/SimpleModelsforLongDocumentMatching

##### Performance on Document Matching Task

We experiment with three input representations for simple neural models: (i)
char-Trigram Hashing (T) Huang et al. (2013), (ii) GloVe Embeddings (G)
Pennington et al. (2014), and (iii) Paragraph Vector/Doc2Vec Embeddings (D) Le
and Mikolov (2014) (See Appendix 4). Unlike most transformer-based models that
take as input tokens up to a pre-defined length (512 for BERT, and 4096 for
Longformer), simple models and SMITH have the ability to take the entire long
document ($>8000$ tokens) as input. Table 1 demonstrates the performance of
different models on the task of document matching up to their maximum allowed
input document length (see Appendix A.8 for different document lengths.) We
observe that despite being relatively simple and not taking into account
contextual embeddings, DSSM and ARC-I outperform the transformer based models
using GloVe and Doc2Vec Embeddings on the AAN, WIKI, and the PAT dataset.

(a) Training time on the log scale

(b) Memory consumption

(c) Energy Consumption on log scale

Figure 2: Comparison of simple neural models with transformer-based models
based on (a) Training Time, (b) Memory Consumption, and (c) Energy
Consumption.

(a) Comparison with BERT [512 tokens]

(b) Comparison with LONG [4096 tokens]

(c) Original vs. Shuffled Documents

Figure 3: Comparison between the robustness of simple neural models and
transformer-based models w.r.t. document length and text perturbation on the
AAN dataset.

##### Training Time

Figure 2(a) shows the training time taken to reach the best performance for
every model for their maximum allowed input token lengths. We only report the
fine-tuning time after downloading the pre-trained models. All experiments
were done on a 16GiB Tesla V100. The simple models like DSSM, ARC-I, and HAN
take 1/12 to 1/15 of the training time taken by the transformer-based models
to outperform them on all three datasets (see Appendix A.4 for the training
time for different document lengths on all datasets.)

##### Memory and Energy consumption

Memory consumption on a 16GiB Tesla V100, for a batch size of 1, for different
models can be seen in Figure 2(b). Compared to transformer-based models simple
neural models consume significantly less memory for the same document length
(12 GiB for Longformer vs. a maximum of 8 GiB for DSSM for 4000 tokens). We
also compute the overall energy required for training the models to achieve
their best performance (Figure 2(c)) by measuring the power consumption of the
GPU over their training lifetime. Longformer consumes $>6\text{MJ}$ of energy
for fine-tuning on documents with 4096 tokens, BERT consumes $\approx
500\text{kJ}$ of energy for fine-tuning on documents with just 512 tokens, and
the SMITH model consumes $\approx 200\text{kJ}$ of energy for fine-tuning on
longer documents; whereas the simple models consume $<100\text{kJ}$ of energy
for training from scratch for documents with $>8000$ tokens.

##### Robustness to Document Length

We limit the maximum number of tokens in each document during training and
testing, and observe the final test accuracy on the document matching task. It
should be noted that documents of different lengths are actually ‘truncated
long documents’ without the complete contextual information needed to compute
the actual similarity between two long documents. Figure 3 compares the model
accuracy of simple models (with their default input embeddings) with
transformer-based models upto their maximum allowed token lengths for the AAN
dataset – 512 tokens for BERT (Figure 3(a)), and 4096 tokens for Longformer
(Figure 3(b)). DSSM outperforms the baseline models for all documents lengths.
BERT and Longformer have a consistent performance on AAN for different input
lengths, unlike HAN and SMITH that are not as robust to the variations in
document length, although they were designed specifically for long documents.
We found similar results for WIKI and PAT dataset (see Appendix A.5). We also
experiment with documents of length $>512$ tokens for BERT, and $>4096$ tokens
for Longformer by aggregating the chunk representations upto their maximum
allowed token length. We used the SUM and AVG aggregation techniques and
observed an overall performance drop (see Table 5).

##### Robustness to Text Perturbation

For text perturbation, we split documents into paragraphs of 512 tokens and
randomly shuffle the order of these paragraphs before training different
models to check for learned positional bias. We measure their test accuracy on
the original document matching task. Figure 3(c) shows the model performance
for all the baseline methods on AAN dataset. We observe a significant drop in
the model performance for transformer-based models (BERT, Longformer, and
SMITH). There is no significant change in the accuracy for the simple models –
DSSM, ARC-I, and HAN. The transformer based models are more sensitive to text
perturbation. The simple models, on the other hand, use non-contextual
embeddings, such as GloVE, and Doc2Vec and are more robust to text
perturbation (see Appendix A.6).

##  4 Conclusion

We empirically demonstrate the trade-off of using transformer-based models for
semi-structured long English documents like research papers, Wikipedia
articles, and patents. Transformer-based models have an overall strong
performance and smaller variability across datasets. However, we observe that
for the task of long document matching, using contextual embeddings do not
provide any added advantage. A simple feed-forward network or a CNN-based
model using GloVe or Doc2Vec embedding outperforms several state-of-the-art
pre-trained transformer-based models at a fraction of their overall training
time and resources (memory and energy). These simple neural models are also
more robust to changes in document length and text-perturbation.

##  5 Limitation

One of the limitations of our work is that we experimented only with long
documents in English. Comparing simple neural models and transformer-based
models in different languages would be an interesting study but is outside the
scope of this short paper. We would also like to highlight that we use
classification metrics instead of information-retrieval metrics due to the
limitations of the dataset which has very few positive samples (2-3) for every
document.

## References

  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Child et al. (2019) Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019.  Generating long sequences with sparse transformers.  _arXiv preprint arXiv:1904.10509_.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019.  Transformer-xl: Attentive language models beyond a fixed-length context.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2978–2988.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Ding et al. (2020) Ming Ding, Chang Zhou, Hongxia Yang, and Jie Tang. 2020.  Cogltx: Applying bert to long texts.  _Advances in Neural Information Processing Systems_ , 33.
  * Friburger et al. (2002) Nathalie Friburger, Denis Maurel, and Arnaud Giacometti. 2002.  Textual similarity based on proper names.  In _Proc. of the workshop Mathematical/Formal Methods in Information Retrieval_ , pages 155–167.
  * Guo et al. (2016) Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016.  A deep relevance matching model for ad-hoc retrieval.  In _Proceedings of the 25th ACM international on conference on information and knowledge management_ , pages 55–64.
  * Ho et al. (2019) Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. 2019.  Axial attention in multidimensional transformers.  _arXiv preprint arXiv:1912.12180_.
  * Hu et al. (2014) Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014.  Convolutional neural network architectures for matching natural language sentences.  _Advances in neural information processing systems_ , 27:2042–2050.
  * Huang et al. (2008) Anna Huang et al. 2008.  Similarity measures for text document clustering.  In _Proceedings of the Sixth New Zealand Computer Science Research Student Conference (NZCSRSC2008), Christchurch, New Zealand_ , volume 4, pages 9–56.
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013\.  Learning deep structured semantic models for web search using clickthrough data.  In _Proceedings of the 22nd ACM international conference on Information & Knowledge Management_, pages 2333–2338.
  * Jha et al. (2022) Akshita Jha, Vineeth Rakesh, Jaideep Chandrashekar, Adithya Samavedhi, and Chandan K. Reddy. 2022.  Supervised contrastive learning for interpretable long-form document matching.
  * Jiang et al. (2019) Jyun-Yu Jiang, Mingyang Zhang, Cheng Li, Michael Bendersky, Nadav Golbandi, and Marc Najork. 2019.  Semantic text matching for long-form documents.  In _The World Wide Web Conference_ , pages 795–806.
  * Kitaev et al. (2019) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2019.  Reformer: The efficient transformer.  In _International Conference on Learning Representations_.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed representations of sentences and documents.  In _International conference on machine learning_ , pages 1188–1196. PMLR.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Hierarchical transformers for multi-document summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5070–5081, Florence, Italy. Association for Computational Linguistics.
  * Mitra et al. (2017) Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017.  Learning to match using local and distributed representations of text for web search.  In _Proceedings of the 26th International Conference on World Wide Web_ , pages 1291–1299.
  * Pang et al. (2016) Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016\.  Text matching as image recognition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 30.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Qiu et al. (2020) Jiezhong Qiu, Hao Ma, Omer Levy, Wen-tau Yih, Sinong Wang, and Jie Tang. 2020.  Blockwise self-attention for long document understanding.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 2555–2565.
  * Radev et al. (2013) Dragomir R. Radev, Pradeep Muthukrishnan, Vahed Qazvinian, and Amjad Abu-Jbara. 2013\.  The acl anthology network corpus.  _Language Resources and Evaluation_ , pages 1–26.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rae et al. (2019) Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. 2019.  Compressive transformers for long-range sequence modelling.  In _International Conference on Learning Representations_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3973–3983.
  * Strehl et al. (2000) Alexander Strehl, Joydeep Ghosh, and Raymond Mooney. 2000.  Impact of similarity measures on web-page clustering.  In _Workshop on artificial intelligence for web search (AAAI 2000)_ , volume 58, page 64.
  * Yang et al. (2020a) Linyi Yang, Tin Lok James Ng, Barry Smyth, and Riuhai Dong. 2020a.  Html: Hierarchical transformer-based multi-task learning for volatility prediction.  In _Proceedings of The Web Conference 2020_ , pages 441–451.
  * Yang et al. (2016a) Liu Yang, Qingyao Ai, Jiafeng Guo, and W Bruce Croft. 2016a.  anmm: Ranking short answer texts with attention-based neural matching model.  In _Proceedings of the 25th ACM international on conference on information and knowledge management_ , pages 287–296.
  * Yang et al. (2020b) Liu Yang, Mingyang Zhang, Cheng Li, Michael Bendersky, and Marc Najork. 2020b.  Beyond 512 tokens: Siamese multi-depth transformer-based hierarchical encoder for long-form document matching.  In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, pages 1725–1734.
  * Yang et al. (2016b) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016b.  Hierarchical attention networks for document classification.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1480–1489, San Diego, California. Association for Computational Linguistics.
  * Yu et al. (2018) Jianfei Yu, Minghui Qiu, Jing Jiang, Jun Huang, Shuangyong Song, Wei Chu, and Haiqing Chen. 2018.  Modelling domain relationships for transfer learning on retrieval-based question answering systems in e-commerce.  In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_ , pages 682–690.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020\.  Big bird: Transformers for longer sequences.  _arXiv preprint arXiv:2007.14062_.

##  Appendix A Appendix

###  A.1 Model Description

The encoder networks and their default inputs have been described below:

####  A.1.1 Simple Models

  * •

DSSM Huang et al. (2013): A simple three-layered feed forward network that
takes as input the vectorized representation of a document.

  * •

ARC-I Hu et al. (2014): A CNN-based model that takes as input a 2D-matrix
representation of a document where words in the sentences are represented
using GloVe embeddings Pennington et al. (2014). These are then passed through
convolutional and max-pooling layers to finally obtain a document
representation for both source and target documents, independently. The
document representations are concatenated and passed through a multi-layer
perceptron to predict if the pair of documents are similar or not.

  * •

Hierarchical Attention Network (HAN) Yang et al. (2016b): A hierarchical GRU-
based model with attention mechanism that aggregates GloVe embeddings at word
level into sentence representations to arrive at the final document
representation.

####  A.1.2 Transformer-Based Models

  * •

BERT Devlin et al. (2019): A siamese matching model with BERT. For long
document inputs, BERT only uses the first 512 tokens of each document. We use
a pre-trained $BERT_{BASE}$ model which is fine-tuned during training.

  * •

Longformers (LONG) Beltagy et al. (2020) : A siamese model with transformer-
based Longformers for long sequences. It has an attention mechanism that
scales linearly with sequence length and takes as input a maximum of 4096
tokens. We consider an attention window of size 256.

  * •

Siamese Multi-depth Transformer based Hierarchical Encoder (SMITH) Yang et al.
(2020b): A transformer-based hierarchical encoder which is the current SOTA
model for long-form document matching task.

####  A.1.3 Implementation Details

For all the models presented in the paper, we use the same architecture as the
original papers. We tune the hyperparamters and report the best results. The
DSSM, ARC-I, HAN, and SMITH models were implemented in Tensorflow. BERT and
Longformer were implemented in PyTorch. DSSM has hidden units of dimension 300
for its hidden layers and an output dimension of 128. The learning rate was
0.0075. ARC-I takes as input a 2D matrix of the size [no. of sentences x
sentence length]. This is given as input to two 1D-convolutional (filter size
of 200, kernel size 3) and MaxPooling layers of size 2, in order to get the
final document representations. The representations of both the source and the
target documents are concatenated and passed through a two-layer multi-layer
perceptron. The first hidden layer of the MLP has a dimension of 64 with ReLU
activation. The second layer has 1 node and sigmoid activation which predicts
if the pair of documents are similar or not. The learning rate was set to
0.00075. HAN uses a bi-Directional GRU layer and applies attention mechanism
to arrive at final sentence representation for the source and the target
documents with a learning rate of 0.001. We use the pre-trained
$\text{BERT}_{\text{BASE}}$444https://huggingface.co/transformers/model_doc/bert.html
and the Longformer
555https://huggingface.co/transformers/model_doc/longformer.html models
provided by the Huggingface library. The SMITH code was publicly available.
The BERT, Longformer, and SMITH models are fine-tuned during training. All
other models are trained from scratch. The learning rate is set to 5e-5 for
the transformer based models. We use an Adam optimizer for all models with a
weight decay of 0.01. We use binary cross entropy as the loss function for the
simple models, pairwise loss for BERT and Longformer, and triplet loss for
SMITH. These loss functions resulted in the best performance for the model.
The three datasets are split into 80-10-10 for train, validation, and test
sets, respectively. We use cosine similarity and the similarity threshold
$\theta$ is set to 0.5. We perform 5 fold cross-validation and use early
stopping on the validation set to prevent over-fitting. The models were
trained on one 16GB Tesla V100 GPU.

###  A.2 Dataset

  * •

ACL Anthology Network Corpus
(AAN)666https://aan.how/download/#aanNetworkCorpus: The AAN corpus Radev et
al. (2013) consists of 23,766 papers written by 18,862 authors in 373 venues
related to NLP and forms a citation network. Each paper is represented by a
node with directed edges connecting a paper (the parent node) to all its cited
papers (children nodes). Papers that have been cited by the parent paper are
treated as similar samples Jiang et al. (2019). For every similar sample, an
irrelevant paper is randomly chosen to create a balanced dataset. Sets of
similar papers are given the same labels. To prevent leakage of information
and make the task more difficult, the references and the abstract sections are
removed. Papers without any content are also removed. We then randomly sample
15,000 research paper pairs for our experiment.

  * •

Wikipedia (WIKI)777https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-
pages-articles.xml.bz2: We use the Wikipedia dump, and adopt a similar
methodology proposed by Jiang et al. Jiang et al. (2019) to process this data.
From the Wikipedia dump containing 6 million articles, we randomly sampled
250,000 articles along with the articles present in their outlinks. We create
a dataset of similar Wikipedia articles by assuming that similar articles have
similar outgoing links. The Jaccard similarity between the outgoing links of
the source and the target articles is calculated. If the Jaccard similarity
$>0.5$, the documents are assumed to be similar, otherwise they are considered
dissimilar. Only articles with two or more similar articles are selected. We
then randomly sample 15,000 research paper pairs for our experiment.

  * •

Patent (PAT)888https://github.com/google/patents-public-data: The patent
dataset is an internally curated industry gold-standard. This dataset consists
of patents sampled from the publicly available USPTO patents belonging to four
different categories: video, wireless, image compression, and network
compression. A patent document is extremely long and primarily consists of (i)
Abstract, (ii) Claims, and (iii) Description sections. We only make use of the
Claims and the Description sections for our experiments to prevent leakage of
information from Abstracts. Three internal human annotators, with expert
domain knowledge, were given pairs of documents and were asked to label them
as similar or dissimilar based based on the technology presented in the
patents. They referred to the Abstract, Claims, and CPC
Codes999https://www.uspto.gov/web/patents/classification/cpc/html/cpc.html of
the patents to measure the similarity. The final document content similarity
label was based on majority vote.

The dataset statistics can be found in Table 2. We would like to note that
although considering only the cited papers and outgoing links for AAN and
Wikipedia articles, respectively is not the most optimal approach for creating
similar document pairs, we adopt it for the following two reasons: (i) We do
not have annotated fine-grained similarity scores for AAN and WIKI datasets,
and (ii) We follow an approach similar to the previously published work Jiang
et al. (2019); Yang et al. (2016b, 2020b). We use the term “document matching"
or “document similarity" in the broad sense of “citation matching" or
“document relevance" – Given a pair of documents, are the two documents
relevant to each other and should they be cited by each other.

Dataset | Avg #Tokens | Max #Tokens | Avg #Sentences | Max #Sentences | Vocabulary
---|---|---|---|---|---
AAN | 5,381.1 | 54,556 | 215.7 | 2,183 | 515,422
WIKI | 3,777.0 | 26,172 | 190.6 | 1,685 | 1,151,309
PAT | 8,177.4 | 50,322 | 214.1 | 2,709 | 220,023
Table 2: Dataset statistics

###  A.3 Performance on Document Matching Task

Table 3 shows the average performance of simple models when compared to
transformer based models on the document matching task for AAN, WIKI, and PAT
datasets. ARC-I with Doc2Vec embeddings has the best average precision and
accuracy. The average F1 score is comparable to BERT which re-emphasizes the
benefits of using simple models for document matching.

Model | P | R | F1 | Acc
---|---|---|---|---
HAN-G | 0.559 | 0.771 | 0.641 | 0.634
DSSM-T | 0.820 | 0.901 | 0.858 | 0.851
DSSM-G | 0.836 | 0.841 | 0.838 | 0.839
DSSM-D | 0.875 | 0.902 | 0.886 | 0.907
ARC-I-G | 0.846 | 0.939 | 0.884 | 0.889
ARC-I-D | 0.931 | 0.902 | 0.916 | 0.926
BERT | 0.913 | 0.950 | 0.915 | 0.905
LONG | 0.885 | 0.925 | 0.902 | 0.910
SMITH | 0.855 | 0.828 | 0.821 | 0.860
Table 3: Average performance across AAN, WIKI, and PAT datasets on the
document matching task (shown in Table 1). We experiment with Trigrams (T),
GloVe (G), and Doc2Vec (D) Embeddings as input for the simple neural models.
The best performance is highlighted in bold.

###  A.4 Training Time for Different Document Lengths

The training time for different document lengths for all three datasets can be
seen in Figure 4. BERT can only take up to 512 tokens. DSSM, ARC-1, and HAN
take considerably less time to train when compared to transformer-based.

(a) AAN

(b) WIKI

(c) PAT

Figure 4: Doc Length vs Training Time (in log scale) for different document
lengths.

###  A.5 Robustness to Document Length

We check the robustness of simple models and transformer-based models for
different document lengths on the task of document matching. From Figure 5 and
Figure 6, we observe that the simple models DSSM and ARC-I, and the
transformer-based models BERT and Longformer, though not specifically designed
for long documents are robust for different document lengths. BERT can only
handle up to 512 tokens at a time, and Longformer can only handle up to 4096
input tokens. HAN and SMITH, on the other hand, were specially designed for
long documents and have a high variance in their performance on the document
matching tasks for different document lengths.

We also experimented with longer documents ($>512$ tokens for BERT, and
$>4096$ tokens for Longformer). We obtained the final document representation
by dividing the document into chunks of their maximum allowed token length. We
then aggregated these chunk representations. We experimented with the SUM and
AVG aggregation techniques by taking the representations of the ‘[CLS]’ token
and ‘the pooler output’ for these models. We observed an overall performance
drop because of aggregation. The results were the same for both SUM and AVG
aggregation techniques (Table 5).

(a) AAN

(b) WIKI

(c) PAT

Figure 5: Document Length vs Accuracy upto 512 tokens

(a) AAN

(b) WIKI

(c) PAT

Figure 6: Document Length vs Accuracy upto 4096 tokens

###  A.6 Robustness to Text Perturbation

We randomly shuffle the documents before training different models and measure
their test accuracy on the original document matching task (Figure 7) for all
three datasets. The first few paragraphs in Wikipedia articles, research
papers, and patents are highly informative. We wanted to verify if the models
give too much importance to the position of the initial text. In the context
of long documents, just re-ordering the paragraphs of a document spanning
pages should not have an effect on the downstream tasks of document matching.
(Note: We use the term document matching broadly to refer to citation matching
or document relevance. Given a document pair, we would like to verify if the
two documents are relevant to each other.) In order to verify this assumption,
we shuffle the paragraphs to distribute the important texts randomly and check
the performance of all models on this downstream task. Although, we do observe
a small drop after paragraph shuffling because the simple models do take into
account a shallow context of the input text, the simple neural models overall
prove more robust to text perturbation when compared to transformer-based
models that take into account deep contextual information.

(a) AAN

(b) WIKI

(c) PAT

Figure 7: Original vs Shuffled Documents

###  A.7 Best Input Embeddings for Simple Models

| AAN | WIKI | PAT
---|---|---|---
Model | P | R | F1 | Acc | P | R | F1 | Acc | P | R | F1 | Acc
DSSM-T | 0.768 | 0.809 | 0.787 | 0.780 | 0.823 | 0.939 | 0.877 | 0.869 | 0.869 | 0.957 | 0.911 | 0.905
ARC-I-T | 0.641 | 0.606 | 0.622 | 0.634 | 0.969 | 0.944 | 0.956 | 0.957 | 0.536 | 0.754 | 0.626 | 0.793
HAN-T | 0.665 | 0.885 | 0.759 | 0.720 | 0.911 | 0.929 | 0.920 | 0.920 | 0.477 | 0.857 | 0.618 | 0.751
DSSM-G | 0.550 | 0.541 | 0.545 | 0.549 | 0.966 | 0.986 | 0.975 | 0.975 | 0.992 | 0.998 | 0.995 | 0.995
ARC-I-G | 0.643 | 0.872 | 0.734 | 0.676 | 0.992 | 0.983 | 0.987 | 0.987 | 0.905 | 0.963 | 0.933 | 0.929
HAN-G | 0.504 | 0.881 | 0.641 | 0.507 | 0.935 | 0.984 | 0.959 | 0.958 | 0.609 | 0.848 | 0.709 | 0.522
DSSM-D | 0.852 | 0.763 | 0.805 | 0.815 | 0.933 | 0.984 | 0.958 | 0.957 | 0.841 | 0.959 | 0.896 | 0.949
ARC-I-D | 0.841 | 0.763 | 0.800 | 0.809 | 0.987 | 0.985 | 0.986 | 0.986 | 0.967 | 0.958 | 0.962 | 0.983
HAN-D | 0.709 | 0.919 | 0.801 | 0.771 | 0.875 | 0.859 | 0.866 | 0.873 | 0.946 | 0.996 | 0.970 | 0.975
Table 4: Comparison of different input aggregation techniques: (i) charTrigrams (T), (ii) GloVe Embeddings (G), and (iii) Doc2Vec embeddings (D), for simple models. | AAN | WIKI | PAT
---|---|---|---
Model | P | R | F1 | Acc | P | R | F1 | Acc | P | R | F1 | Acc
BERT-CLS | 0.992 | 0.579 | 0.732 | 0.637 | 0.998 | 0.499 | 0.666 | 0.499 | 1 | 0.639 | 0.783 | 0.714
BERT-POOL | 0.572 | 0.992 | 0.726 | 0.625 | 1 | 0.500 | 0.667 | 0.501 | 1 | 0.637 | 0.778 | 0.711
LONG-CLS | 0.599 | 0.842 | 0.699 | 0.743 | 0.968 | 0.764 | 0.854 | 0.835 | 0.927 | 0.911 | 0.919 | 0.917
LONG-POOL | 0.727 | 0.819 | 0.770 | 0.783 | 0.994 | 0.812 | 0.894 | 0.883 | 0.996 | 0.918 | 0.955 | 0.953
Table 5: Aggregation using the [CLS] token, and the pooler output [POOL] from
BERT and Longformer for documents $>512$ and $>4096$ tokens for BERT and
Longformer, respectively. The results were the same for SUM and AVG
aggregation techniques.

For simple models, we evaluate if the input vector representations play a role
in the final results. We use the following input representations.

  * •

Tri-Gram Hashing (T): Bag-of-charTrigrams is a technique for word hashing
Huang et al. (2013) where each word is broken down into character trigrams
(charTrigrams). Since, the number of possible charTrigarms are fixed and
limited, this is a scalable solution for long documents. The charTrigrams are
obtained for every token in the input text after appending the symbol ‘#’
before and after every token. For example, the word ‘good’ [#good#] is split
into [#go, goo, ood, od#] and then mapped to a 30,621 dimensional hash table.
This vector representation for the document is then given as input to the
models. For DSSM, each document is represented as a bag-of-charTrigrams and
given as input to the model. For ARC-I and HAN, we split each document into
$n$ chunks which are represented in the form of a trigram hash. We construct a
matrix of size $n~{}\times~{}\text{trigram hash}$ for the entire document
which is given input to ARC-I and HAN.

  * •

GloVe Embeddings (G): GloVe Pennington et al. (2014) is an unsupervised
learning algorithm for obtaining vector representations for words. We download
the pre-trained GloVe embeddings and get the vector representations for words
in a long document. These vector representations are given as input to
different models. For DSSM, we divide the document into chunks of a specified
maximum length. We then take GloVe embedding representation of tokens for each
chunk upto a maximum length and average them to get a document representation.
For ARC-I and HAN, each document is represented as a matrix of size
$[\text{embedding dimension}~{}\times~{}\text{max length}]$ in each document.

  * •

Doc2Vec Embeddings (D): Doc2Vec Le and Mikolov (2014) embeddings can be used
to get vector representations for a document. We train Doc2Vec models from
scratch on different datasets to get relevant document representations. These
document representations are then given as input to different document
matching models.

Table 4 shows the model performance of the simple models for the above three
input representations. We observe that using GloVe (G) and Doc2Vec (D) input
embeddings improve the model performance of the simple models overall.

###  A.8 Different Aggregation Techniques for Transformer Based Models

We experiment with different aggregation techniques (SUM and AVG) for $>512$
tokens for BERT, and $>4096$ tokens for Longformer. We chunk the documents and
aggregate the representations from the ‘[CLS]’ token and ‘the pooler output’.
The results can be seen in Table 5 and were the same for SUM and AVG
aggregation techniques. Aggregation resulted in an overall performance drop
when compared to just truncating the documents up to 512 tokens and 4096
tokens for BERT and Longformers, respectively for the document matching.


# Field Calculus: quantum and statistical field theory without the Feynman
diagrams

John Gough
Department of Physics, Aberystwyth University,
SY23 3BZ, Wales, UK.
<EMAIL_ADDRESS>
###### Abstract

For a given base space $M$ (spacetime), we consider the Guichardet space over
the Guichardet space over $M$. Here we develop a “field calculus” based on the
Guichardet integral. This is the natural setting in which to describe Green
function relations for Boson systems. Here we can follow the suggestion of
Schwinger and develop a differential (local field) approach rather than the
integral one pioneered by Feynman. This is helped by a DEFG (Dyson-Einstein-
Feynman-Guichardet) shorthand which greatly simplifies expressions. This gives
a convenient framework for the formal approach of Schwinger and Tomonaga as
opposed to Feynman diagrams The Dyson-Schwinger is recast in this language
with the help of bosonic creation/annihilation operators. We also give the
combinatorial approach to tree-expansions.

##  1 On Three Levels

###  1.1 Introduction

Feynman introduced his diagrammatic approach as a computational technique for
quantum electrodynamics [1] which bypassed quantum field theory. The diagrams
are the terms in the perturbative expansion [2] and their importance is that
they allow a systematic approximation starting with the lowest order diagrams.
The diagrammatic approach has since found widespread application in the
development of quantum field theory, [3]. However, the diagrams are the trees
that make up the wood of quantum field theory, and their dominance means that
we often cannot see the underlying combinatorial structure. The Feynman
approach places emphasis on the particle description at the expense of the
local fields themselves. Schwinger considered his approach as the differential
version of Feynman’s integral approach, and that the former was mathematically
more flexible [5].

Here we build on ideas presented in [4] to develop a calculus built around the
Guichardet’s representation of Fock space [6] which is an alternate way to
describe quantum field theory which avoids having to draw diagrams and instead
uses a combinatorial identities.

###  1.2 Guichardet Spaces

Let $M$ be Minkowski spacetime with $dx$ denoting the 4-volume element. This
will be the level 1 description of fields. Specifically, a field $\varphi$
over $M$ is a complex-valued function on $M$ and its value at a point $x\in M$
is denote either as $\varphi(x)$ or $\varphi_{x}$.

The power set of $M$ is the set of all finite subsets and we will denote this
as $\mathscr{P}(M)$. Let $G=(G_{n})_{n}$ be a family of completely symmetric
functions of an indefinite number of spacetime variables: that is, for each
$n=0,1,2,\cdots$ we have $G_{n}:\times^{n}M\mapsto\mathbb{R}$ with
$G_{n}(x_{1},\cdots,x_{n})$ invariant under arbitrary permutation of the
arguments. The family may then be identified as a function
$G:\mathscr{P}(M)\mapsto\mathbb{C}$ by

$\displaystyle G(X)\equiv G_{n}(x_{1},\cdots,x_{n}),$ (1)

whenever $X=\\{x_{1},\cdots,x_{n}\\}$. This includes the value $G_{0}$ which
is assigned to the empty set: $G_{0}\equiv G\left(\emptyset\right)$. Note that
(1) actually suggests that we should use multisets rather than sets, but we
can avoid this since in standard configuration there will be no degeneracy for
the Lesbegue measure $dx$. The power set of $M$ will be our level 2
description.

The measure $dx$ at level 1 induces a measure $dX$, called the Guichardet
measure [6], at level 2. Here, $dX$ on $\mathscr{P}(M)$ is defined as follows:

$\displaystyle\int_{\mathscr{P}(M)}G(X)dX\triangleq\sum_{n\geq
0}\frac{1}{n!}\int_{\times^{n}M}G_{n}(x_{1},\cdots,x_{n})\,dx_{1}\cdots
dx_{n}.$ (2)

The Guichardet measure was extensively used in the kernel approach to quantum
stochastic processes by Hans Maassen [7] and Paul-Andre Meyer [8]. The
extension to space follows from an important remark due to Joachim Kupsch [8]
and, as Meyer remarks, the results may be extended to a Luzin space $X$ with a
non-atomic measure.

Starting from the base space $(M,dx)$, the Guichardet construction has allowed
us to go up one level to the measure space $\big{(}\mathscr{P}(M),dX\big{)}$.
We may then use the Guichardet construction again to go up another level
again! Here we would form the power set of $\mathscr{P}(M)$ with basic
elements now $\mathcal{X}=\\{X_{1},\cdots,X_{n}\\}$ which are sets of subsets:
functions $G(\mathcal{X})\equiv G(X_{1},\cdots,X_{n})$ are again completely
symmetric in there arguments. The induced measure $d\mathcal{X}$ is given by

$\displaystyle\int_{\mathscr{P}(\mathscr{P}(M))}G(\mathcal{X})d\mathcal{X}\triangleq\sum_{n\geq
0}\frac{1}{n!}\int_{\times^{n}\mathscr{P}(M)}G_{n}(X_{1},\cdots,X_{n})\,dX_{1}\cdots
dX_{n},$ (3)

so that we obtain the measure space
$\big{(}\mathscr{P}(\mathscr{P}(M)),d\mathcal{X}\big{)}$.

Level | Framework | Elements | Measure
---|---|---|---
1 | spacetime, $M$ | $x$ | $dx$
2 | power set of $M$, $\mathscr{P}(M)$ | $X=\\{x_{1},\cdots,x_{n}\\}$ | $dX$
3 | power set of $\mathscr{P}(M)$, $\mathscr{P}(\mathscr{P}(M))$ | $\mathcal{X}=\\{X_{1},\cdots,X_{n}\\}$ |  $d\mathcal{X}$ .

Why would we want to go up another level? It turns out that we naturally
encounter subsets of subsets of spacetime events when dealing with fields, and
this level will turn out to be highly convenient when we try to describe
combinatorial features.

For $X\in\mathscr{P}(M)$, we will denote the number of elements as $\\#X$. An
equation of the form $X+Y=Z$ signifies that $Z$ is the union of disjoint sets
$X$ and $Y$. A decomposition of $X$ is an ordered sequence
$(X_{1},\cdots,X_{n})$ of disjoint sets (some of which may be empty!) whose
union is $X$.

###### Definition 1

An apportionment of $X\in\mathscr{P}(M)$ is a set
$\mathcal{X}\in\mathscr{P}(\mathscr{P}(M))$ of (unordered) disjoint subsets
which are allowed to be empty and whose union is $X$. The number of subsets
making up $\mathcal{X}$ is denoted as $N(\mathcal{X})$. An apportionment is
said to be a partition if each of its elements (which we then refer to as
parts) are non-empty. We will write $\mathrm{App}(X)$ and $\mathrm{Part}(X)$
for the collection of all apportionments and partitions of $X$ respectively.
We denote by $\mathrm{Pair}(X)$ the collection of all pair partitions.

For definiteness, a apportionment of $X$ will be a set
$\mathcal{X}=\\{X_{1},\cdots,X_{n}\\}$ where $X_{j}\cap X_{k}=\emptyset$ for
all $j\neq k$, and $X_{1}+\cdots+X_{n}=X$. In this case $N(\mathcal{X})=n$. It
will be a partition if we furthermore have $\\#X_{k}\geq 1$ for each $k$. In
the case of pair partitions, we have the additional constraint that
$\\#X_{k}=2$ for each $k$ (so $\\#X$ must be even).

######  Notation 2 (Einstein-Guichardet Convention)

We use lowercase letters to denote functions on $M$: a repeated spacetime
index $x$ implies integration with respect to $dx$ over $M$:

$\displaystyle f^{x}g^{x}\triangleq\int_{M}f(x)g(x)\,dx.$ (4)

We use uppercase letters to denote functions on $\mathscr{P}(M)$: a repeated
subset index implies integration with respect to $dX$ over $\mathscr{P}(M)$:

$\displaystyle F^{X}G^{X}\triangleq\int_{\mathscr{P}(M)}F(X)G(X)\,dX.$ (5)

Finally, we use uppercase calligraphic letters to denote functions on
$\mathscr{P}(\mathscr{P}(M))$: a repeated subset index implies integration
with respect to $d\mathcal{X}$ over $\mathscr{P}(\mathscr{P}(M))$:

$\displaystyle
F^{\mathcal{X}}G^{\mathcal{X}}\triangleq\int_{\mathscr{P}(\mathscr{P}(M))}F(\mathcal{X})G(\mathcal{X})\,d\mathcal{X}.$
(6)

Note that we place arguments as either subscripts or superscripts. For
functions on $M$, the idea is that the fields are covariant and so carry a
subscript: $\varphi(x)\equiv\varphi_{x}$. Dual to these we will have sources
$j(x)\equiv j^{x}$ which carry a contravariant index. The duality is the
$\langle j,\varphi\rangle=j^{x}\varphi_{x}$. We will take $\Phi$ to be a
suitable class of fields, and $\mathfrak{J}$ to be a suitable class of
sources.

###### Notation 3

A further shorthand is the following: let $f$ be measurable on $M$ and
$X\in\mathscr{P}(M)$ then write

$\displaystyle f^{X}\triangleq\prod_{x\in X}f(x).$ (7)

Likewise, we use $F^{\mathcal{X}}=\prod_{X\in\mathcal{X}}F^{X}$.

As a first foray into the Einstein-Guichardet convention, let us show the
following result

###### Proposition 4

We have the following identities:

$\displaystyle f^{X}g^{X}\equiv e^{f^{x}g^{x}},\qquad
F^{\mathcal{X}}G^{\mathcal{X}}\equiv e^{F^{X}G^{X}}.$ (8)

Proof. The right hand side in the first identity reads as follows

$\displaystyle\int_{\mathscr{P}(M)}f^{X}g^{X}\,dX$ $\displaystyle=$
$\displaystyle\sum_{n\geq 0}\frac{1}{n!}\int_{\times^{n}M}f(x_{1})\cdots
f(x_{n})g(x_{1})\cdots g(x_{n})\,dx_{1}\cdots dx_{n}$ (9) $\displaystyle=$
$\displaystyle\sum_{n\geq
0}\frac{1}{n!}\bigg{[}\int_{M}f(x)g(x)\,dx\bigg{]}^{n}=\exp\int_{M}f(x)g(x)\,dx.$

The second identity is just the level 3 analogue.

With the convention $f^{\mathcal{X}}=\prod_{X\in\mathcal{X}}\prod_{x\in
X}f(x)$, we have

$\displaystyle f^{\mathcal{X}}g^{\mathcal{X}}\equiv e^{e^{f^{x}g^{x}}}.$ (10)

Note that the exponential of an exponential arises naturally as generating
functional associated with Poisson point process distributions.

The space $L^{2}(\mathscr{P}(M),dX)$ is, in fact, then the Fock space over
$L^{2}(M,dx)$. Given a wave-function $\Psi$ for an indefinite number of boson
particles, we may write it as function $\Psi:X\mapsto\Psi(X)$, and we have the
overlaps

$\displaystyle\langle\Phi|\Psi\rangle\triangleq\int\Phi(X)^{\ast}\Psi(X)dX\equiv(\Phi^{\ast})^{X}\Psi^{X}.$
(11)

In particular, the function $\exp^{f}$ defined by $\exp^{f}(X)=f^{X}$ is an
exponential vector and from Proposition 4 we have that
$\langle\exp^{f}|\exp^{g}\rangle=e^{\langle f|g\rangle}.$

Moving up a level, we see that
$L^{2}(\mathscr{P}(\mathscr{P}(M)),d\mathcal{X})$ is the Fock space over
$L^{2}(\mathscr{P}(M),dX)$\- that is, the Fock space over the Fock space over
$L^{2}(M,dx)$.

###  1.3 Generating Functionals and the Wick Product

###### Definition 5

Let $G$ be a measurable function on $\mathscr{P}(M)$ then we define its
associated generating functional, for test functions $j$ belonging to some
suitable class $\mathfrak{J}$, as $Z_{G}(j)=G_{X}\,j^{X}$ or, in longhand

$\displaystyle Z_{G}(j)\triangleq\sum_{n\geq 0}\frac{1}{n!}\int
G(x_{1},\cdots,x_{n})\,j(x_{1})\cdots j(x_{n})\,dx_{1}\cdots dx_{n}.$ (12)

###### Definition 6

The Wick product of functions on $\mathscr{P}(M)$ is defined to be

$\displaystyle F\diamond
G(X)\triangleq\sum_{X_{1}+X_{2}=X}F(X_{1})\,G(X_{2}).$ (13)

Here the sum is over all decompositions of the set $X$ into ordered pairs
$(X_{1},X_{2})$ whose union is $X$.

The importance of the Wick product is revealed in the next result.

###### Proposition 7

We have $Z_{F\diamond G}=Z_{F}Z_{G}$, where the $F\diamond G$ is the Wick
product.

Proof. In our Einstein-Guichardet notation, we have that
$Z_{F}(j)Z_{G}(j)=F_{X_{1}}j^{X_{1}}G_{X_{2}}j^{X_{2}}=F_{X_{1}}G_{X_{2}}j^{X_{1}+X_{2}}$
which we rewrite as $(F\diamond G)_{X}j^{X}$.

To appreciate the compactness of the notation, let us redo the proof longhand.
We have that $\int F(Y)G(Z)j^{Y+Z}dYdZ$ equals

$\displaystyle\sum_{n_{1},n_{2}\geq 0}\frac{1}{n_{1}!n_{2}!}\int
F(y_{1},\cdots,y_{n_{1}})G(z_{1},\cdots,z_{n_{2}})$
$\displaystyle\qquad\qquad\times j(y_{1})\cdots j(y_{n_{1}})j(z_{1})\cdots
j(z_{n_{2}})\,dy_{1}\cdots dy_{n_{1}}dz_{1}\cdots dz_{n_{2}}$
$\displaystyle\equiv$ $\displaystyle\sum_{n\geq 0}\frac{1}{n!}\int F\diamond
G(x_{1},\cdots,x_{n})\,j(x_{1})\cdots j(x_{n})\,dx_{1}\cdots dx_{n}$

and comparing powers of $j$ we get $F\diamond G(X)=\sum_{Y+Z=X}F(Y)G(Z)$.

The Wick product is a symmetric associative product with general form

$\displaystyle G_{1}\diamond\cdots\diamond G_{n}(X)=\sum_{X_{1}+\cdots
X_{n}=X}G_{1}(X_{1})\cdots G_{n}(X_{n}),$

where the sum is now over all decomposition of $X$ into $n$ ordered disjoint
subsets.

Some care is needed when the Green’s functions are the same. Denoting the
$n$th Wick power of $F$ by $F^{\diamond
n}=\mathrel{\mathop{\underbrace{F\diamond\cdots\diamond F}}\limits_{n\,\text{
times}}}$, we have

$\displaystyle F^{\diamond n}(X)=\sum_{X_{1}+\cdots+X_{n}=X}F(X_{1})\cdots
F(X_{n}).$

This leads to the following Wick product of exponential vectors:
$\exp(f)\diamond\exp(g)\equiv\exp(f+g)$. The next result is due to Hans
Maassen [7].

######  Lemma 8 (The $\Sigma\int$ Lemma)

For $F$ a measurable function on $\times^{p}\mathscr{P}(M)$ we have the
identity

$\displaystyle\int F\left(X_{1},\cdots,X_{p}\right)\,dX_{1}\cdots
dX_{p}\equiv\int\sum_{X_{1}+\cdots+X_{p}=X}F\left(X_{1},\cdots,X_{p}\right)\,dX.$
(14)

Proof. If we write both of these expressions out longhand, then the left hand
side picks up the factors $\frac{1}{n_{1}!\cdots n_{p}!}$ where
$\\#X_{k}=n_{k}$. On the right hand side we get $\frac{1}{n!}$ where $\\#X=n$.
Both expressions are multiple integrals with respect to either $dX_{1}\cdots
dX_{p}$ or $dX$ with $X_{1}+\cdots X_{p}=X$ however on the right hand side we
obtain an additional factor $\binom{n}{n_{1},\cdots,n_{p}}$ giving the number
of decompositions of $X$ with $n_{k}$ elements in the $k$th subset. This
accounts precisely the combinatorial factors so both sides are equal.

###  1.4 The Composition Formula

Suppose we are given a generating functional $K_{X}j^{X}$ then its exponential
will also take the form of a generating functional, say $G_{X}j^{X}$. We have

$\displaystyle G_{X}j^{X}=e^{K_{X}j^{X}}=K_{\mathcal{X}}j^{\mathcal{X}},$ (15)

where $j^{\mathcal{X}}=\prod_{X\in\mathcal{X}}\prod_{x\in X}j^{x}$, that is,
the product of all $j^{x}$ where $x$ is one of the spacetime events making up
$\mathcal{X}$. We see that

$\displaystyle
G_{X}=\sum_{\mathcal{X}\in\mathrm{App}(X)}K_{\mathcal{X}}\equiv\sum_{n}\frac{1}{n!}(K^{\diamond
n})_{X}.$ (16)

If we furthermore assume that $K_{\emptyset}\ =0$, then the nontrivial
apportionments come from partitions and so

$\displaystyle G_{X}=\sum_{\mathcal{X}\in\mathrm{Part}(X)}K_{\mathcal{X}}.$
(17)

For instance, $G_{\emptyset}=1$, $G_{\\{x_{1}\\}}=K_{\\{x_{1}\\}}$,
$G_{\\{x_{1},x_{2}\\}}=K_{\\{x_{1},x_{2}\\}}+K_{\\{x_{1}\\}}K_{\\{x_{2}\\}}$,
etc.

###### Theorem 9

Let $h$ be a function with Maclaurin series
$h(z)=\sum_{n}\frac{1}{n!}h_{n}z^{n}$ and $F_{X}j^{X}$ a generating
functional, then $h(Z_{F}(j))\equiv H_{X}j^{X}$ where

$\displaystyle
H_{X}=\sum_{\mathcal{X}\in\mathrm{App}(X)}h_{N(\mathcal{X})}F_{\mathcal{X}}.$
(18)

Here $H_{X}\equiv\sum_{n}\frac{1}{n!}h_{n}(F^{\diamond
n})_{X}=\sum_{n}h_{n}\sum_{X_{1}+\cdots+X_{n}=X}^{\text{unordered}}F(X_{1})\cdots
F(X_{n})$ where we have dropped the ordering of the elements of the
decomposition which absorbs the $n!$.

We list out some important examples.

  1. 1.

Exponentials $G_{X}j^{X}=e^{K_{X}j^{X}}$, then
$G_{X}=\sum_{\mathcal{X}\in\mathrm{App}(X)}K_{\mathcal{X}}$;

  2. 2.

Logarithms $K_{X}j^{X}=\ln G_{X}j^{X}$, then
$K_{X}=\sum_{\mathcal{X}\in\mathrm{App}(X)}\mu_{N(\mathcal{X})}G_{\mathcal{X}}$,
with $\mu_{n}=(-1)^{n}(n-1)!$;

  3. 3.

Inverse $F_{X}j^{X}=\frac{1}{G_{X}j^{X}}$, then
$F_{X}=\sum_{\mathcal{X}\in\mathrm{App}(X)}\nu_{N(\mathcal{X})}G_{\mathcal{X}}$,
with $\nu_{n}=(-1)^{(n+1)}n!$.

##  2 Random Fields

A field $\varphi$ is a complex-valued function on $M$. We will write
$\varphi(x)$ as $\varphi_{x}$ (covariant index) and denote the space of fields
as $\Phi$. We will dual space $\mathfrak{J}$ of source fields and write
$j(x)=j^{x}$ and the duality is written as $j^{x}\varphi_{x}$ (contravariant
index).

A multi-linear map $T:\times^{n}\mathfrak{J}\mapsto\mathbb{C}$ is called a
tensor of covariant rank $n$ and it will be determined by the components
$T_{x_{1}\cdots x_{n}}$ such that
$T\left(j_{\left(1\right)},\cdots,j_{\left(n\right)}\right)=T_{x_{1}\cdots
x_{n}}\,j_{\left(1\right)}^{x_{1}}\cdots j_{\left(n\right)}^{x_{n}}$. We take
the dual space to $\mathfrak{J}$ to be $\Phi$ and this is the space in which
the fields live. Likewise, we refer to a multilinear map from $\times^{n}\Phi$
to the complex numbers as a tensor of contravariant rank $n$.

###  2.1 Random Fields and Their Green’s Functions

A random field $\phi$ is a random variable $\phi_{x}$ to each point $x\in M$.
More exactly, it is determined by its characteristic functional

$\displaystyle
Z_{G}(j)\triangleq\mathbb{E}\left[e^{j^{x}\phi_{x}}\right]=\mathbb{E}[j^{X}\phi_{X}]=j^{X}G_{X}$
(19)

for suitable functions $j$ belonging to some class $\mathfrak{J}$. It’s
moments are the Green’s functions
$G\left(x_{1},\cdots,x_{n}\right)=\mathbb{E}\left[\phi_{x_{1}}\cdots\phi_{x_{n}}\right]$
and we may write this as

$\displaystyle G_{X}=\mathbb{E}\left[\phi_{X}\right]=\left.\frac{\delta
Z(j)}{\delta j^{X}}\right|_{j=0}$ (20)

where we introduce the shorthand $\phi_{X}=\prod_{x\in X}\phi_{x}$.

The cumulant Green’s functions $K_{X}$ are defined by $K_{X}jX=\ln
G_{X}j^{X}$. As $G_{\emptyset}=1$, we have $K_{\emptyset}=0$ and from the
composition formula the two types of moments are related by

$\displaystyle
G_{X}=\sum_{\mathcal{X}\in\mathrm{Part}(X)}K_{\mathcal{X}},\qquad
K_{X}=\sum_{\mathcal{X}\in\mathrm{Part}(X)}(-1)^{N(\mathcal{X})}N(\mathcal{X})!\,G_{\mathcal{X}}.$
(21)

###### Definition 10

A function $A$ on fields is said to be analytic if it takes the form
$A(\varphi)=A^{X}\varphi_{X}$, or in longhand

$\displaystyle A(\varphi)=\sum_{n\geq
0}\frac{1}{n!}\int_{\times^{n}M}A^{\\{x_{1},\cdots,x_{n}\\}}\varphi_{x_{1}}\cdots\varphi_{x_{n}}\,dx_{1}\cdots
dx_{n}.$ (22)

Let $A$ be an analytic function on $\Phi$, say $A(\varphi)=a^{X}\varphi_{X}$,
then the average of is $A(\phi)$ is

$\displaystyle\mathbb{E}[A(\phi)]=a^{X}\mathbb{E}[\phi_{X}]=a^{X}G_{X}.$ (23)

This suggests that a useful way to think of the averaging process for fields
is as a dual map acting on the sequences $(A^{X})_{X}$ of coefficients. The
moments $(G_{X})_{X}$ act as the dual coefficients and must satisfy
normalization, $G_{\emptyset}=1$, and positivity. We should have, for
instance, that for each integer $N$ and for each set of (complex) sources
$j_{1},\cdots,j_{N}\in\mathfrak{J}$, the property
$0\leq\mathbb{E}[|\sum_{n=1}^{N}e^{j^{x}_{n}\phi_{x}}|^{2}]$ which equates to
the following Bochner condition
$\sum_{n,m=1}^{N}G_{X}(j_{n}^{\ast}-j_{m})^{X}\geq 0$.

###### Lemma 11

Let $\mathbb{E}\left[\cdot\right]$ be an expectation with the corresponding
Green’s function $G_{X}=\mathbb{E}\left[\phi_{X}\right]$. For analytic
functionals $A(\phi)=\int A^{X}\phi_{X}dX$, $B(\phi)=\int B^{X}\phi_{X}dX$,
etc., we have the formula

$\displaystyle\mathbb{E}\left[A(\phi)B(\phi)\cdots
Z(\phi)\right]=\left(A\diamond B\diamond\cdots\diamond Z\right)^{X}G_{X}.$
(24)

Proof. The expectation in (24) reads as

$\displaystyle A^{X_{a}}B^{X_{b}}\cdots
Z^{X_{z}}\mathbb{E}\left[\phi_{X_{a}}\phi_{X_{b}}\cdots\phi_{X_{z}}\right]=A^{X_{a}}B^{X_{b}}\cdots
Z^{X_{z}}G_{X_{a}+X_{b}+\cdots+X_{z}}$ $\displaystyle\equiv$
$\displaystyle\left(\sum_{X_{a}+X_{b}+\cdots+X_{z}=X}A^{X_{a}}B^{X_{b}}\cdots
Z^{X_{z}}\right)G_{X}$

where we use the $\Sigma\int$ lemma.

###  2.2 Gaussian States

We now describe how to define Gaussian states on $\Phi$.

###### Definition 12

A metric $g:\times^{2}\mathfrak{J}\mapsto\mathbb{R}$ is an invertible
symmetric tensor. We write $g_{xy}$ for the components of the tensor: symmetry
means $g_{xy}=g_{yx}$ and invertibility means that there exists a
contravariant tensor $g^{-1}:\times^{2}\Phi\mapsto\mathbb{R}$ with components
$g^{xy}$ with $g_{xy}g^{yz}=\delta(x-z)$.

Let $\overline{\phi}\in\Phi$ and $g$ be a metric. The Gaussian state with mean
$\overline{\phi}$ and covariance $g$ is then determined by the characteristic
function

$\displaystyle\mathbb{E}_{\overline{\phi},g}[e^{j^{x}\varphi_{x}}]=e^{j^{x}\overline{\phi}_{x}+\frac{1}{2}j^{x}g_{xy}j^{y}}.$
(25)

If we set the mean field $\overline{\phi}=0$, then the Gaussian state is
completely characterized by the fact that the only non-vanishing cumulant is
now $K_{\left\\{x,y\right\\}}=g_{xy}$. We will denote this by
$\mathbb{E}_{g}$. If we now use the identity
$G_{X}=\sum_{\mathcal{X}\in\mathrm{Part}(X)}K_{\mathcal{X}}$ to obtain the
Green’s functions, then we see that all odd moments vanish, while

$\displaystyle\mathbb{E}_{g}\left[\phi_{x\left(1\right)}\cdots\phi_{x\left(2k\right)}\right]=\sum_{\mathrm{Pair}\left(2k\right)}g_{x\left(p_{1}\right)x\left(q_{1}\right)}\cdots
g_{x\left(p_{k}\right)x\left(q_{k}\right)}$ (26)

where the sum is over all pair partitions of $\left\\{1,\cdots,2k\right\\}$.
The right-hand side will of course consist of
$\frac{\left(2k\right)!}{2^{k}k!}$ terms. To this end we introduce some
notation. Let $\mathcal{P}\in\mathrm{Pair}\left(X\right)$ be a given pair
partition of a subset $X$, say
$\mathcal{P}=\left\\{x_{p(1)},x_{q(1)}),\cdots,\left(x_{p(k)},x_{q(k)}\right)\right\\}$,
then we write
$g_{\mathcal{P}}=g_{x_{p\left(1\right)}x_{q\left(1\right)}}\cdots
g_{x_{p\left(k\right)}x_{q\left(k\right)}}$ in which case the Gaussian moments
are

$\displaystyle G_{X}^{g}\equiv\mathrm{Pair}(g)_{X}.$ (27)

where we introduce

$\displaystyle\mathrm{Pair}(g)_{X}\triangleq\sum_{\mathcal{P}\in\mathrm{Pair}\left(X\right)}g_{\mathcal{P}}.$
(28)

###  2.3 General States

Suppose that we have an analytic functional on the fields, say
$V(\varphi)=V^{X}\varphi_{X}$. (If we wish, we can take $V^{x,y}\equiv 0$ as
otherwise it would be absorbed into the covariance $g$. We typically take
$V^{\emptyset}=0$ as this plays no role.) We additionally assume that

$\displaystyle\Xi\equiv\mathbb{E}_{g}\left[e^{V(\phi)}\right]<\infty.$ (29)

Given a reference Gaussian state with covariance $g$ (a metric!), we then
obtain new state according to

$\displaystyle\mathbb{E}[A(\phi)]=\frac{1}{\Xi}\mathbb{E}_{g}[A(\varphi)e^{V(\phi)}].$
(30)

A formal approach is to introduce functional integrals. Let $S$ be a
functional on $\Phi$, which we call the action and consider formal expressions
of the form

$\displaystyle\mathbb{E}[A(\phi)]=\frac{\int_{\Phi}A(\varphi)e^{S(\phi)}\mathscr{D}\varphi}{\int_{\Phi}e^{S(\phi)}\mathscr{D}\varphi}.$
(31)

For instance, let us then set

$\displaystyle S_{0}(\varphi)=-\frac{1}{2}\varphi_{x}g^{xy}\varphi_{y},$ (32)

and suppose (by analogy to standard Gaussian integrals) that

$\displaystyle\int_{\Phi}e^{j^{x}\varphi_{x}}e^{S_{0}(\varphi)}\mathscr{D}\varphi=e^{\frac{1}{2}j^{x}g_{xy}j^{y}}.$
(33)

To be clear, this is all formal! All we have done is to say that the
expression “$e^{S_{0}(\phi)}\mathscr{D}\varphi$” appearing in the functional
integral can be made rigorous by interpreting it as a Gaussian measure
$\mathbb{P}_{g}[d\varphi]$ which gives us the well defined Gaussian state
$\mathbb{E}_{g}$ introduced before. The general state is then
$\mathbb{P}[d\varphi]=e^{V(\varphi)}\mathbb{P}_{g}[d\varphi]$. Formally, this
corresponds to the action

$\displaystyle S(\varphi)=S_{0}(\varphi)+V(\varphi).$ (34)

##  3 Why We Need Level 3 for Random Fields!

In the average (30) we encounter the exponential $e^{V(\phi)}$. We can use
(24) to get $\mathbb{E}_{g}\left[V(\phi)^{n}\right]=\left(V^{\diamond
n}\right)^{X}G_{X}^{g}$, but there is a more natural way to calculate
$\mathbb{E}_{g}\left[e^{V(\phi)}\right]$. By Proposition 4, we have
$e^{V(\phi)}=e^{V^{X}\phi_{X}}\equiv V^{\mathcal{X}}\phi_{\mathcal{X}}$ and
the Gaussian expectations yields
$\Xi=V^{\mathcal{X}}\mathbb{E}_{g}[\phi_{\mathcal{X}}]=\sum_{n=0}^{\infty}\frac{1}{n!}V^{X_{1}}\cdots
V^{X_{n}}G^{g}_{X_{1}+\cdots+X_{n}}$. As we have $V^{\emptyset}=0$, the sum
may be considered as being over all partitions $\mathcal{X}$ and this will
further absorb the $n!$ factor since the partitions are unordered. We obtain

$\displaystyle\Xi=\mathrm{Part}\left(V\right)^{X}\,G^{g}_{X}$ (35)

where we introduce
$\mathrm{Part}\left(V\right)^{X}=\sum_{\mathcal{X}\in\mathrm{Part}\left(X\right)}V^{\mathcal{X}}$.

The expression (42) applies to a general state. If we wish to specify to a
Gaussian state $\mathbb{E}_{g}$ then the expression specializes further as

$\displaystyle\Xi=\mathrm{Part}\left(V\right)^{X}\,\mathrm{Pair}(g)_{X}.$ (36)

A similar argument gives the moments.

###### Theorem 13

Let $\mathbb{E}$ the probability state given by (30), then its moments are
given by

$\displaystyle
G_{X}=\frac{1}{\Xi}\mathrm{Part}\left(V\right)^{Y}\,\mathrm{Pair}(g)_{Y+X}.$
(37)

###  3.1 Calculus For Fields

If a generating function $Z_{G}(j)$ is (Fréchet) differentiable about $j=0$ to
all orders, then we may work out the components $G(X)$ according to

$\displaystyle G(X)=\left.\left\\{\prod_{x\in
X}\frac{\delta\;\;\;\;\;\;\;}{\delta
j\left(x\right)}\right\\}Z_{G}(j)\right|_{j=0}.$ (38)

###### Notation 14

A useful shorthand is to introduce the multiple Fréchet differential operator

$\displaystyle\frac{\delta\;\;\;}{\delta j^{X}}=\prod_{x\in
X}\frac{\delta\;\;\;\;\;\;\;}{\delta j\left(x\right)}$ (39)

along with the derivative $\dfrac{\delta Z_{G}}{\delta j}$ defined by
$\frac{\delta Z_{G}(j)}{\delta j}:X\mapsto\frac{\delta Z_{G}(j)}{\delta
j^{X}}$.

In particular, $\left.\dfrac{\delta Z_{G}}{\delta j}\right|_{j=0}=G$. We note
that

$\displaystyle\frac{\delta Z_{G}(j)}{\delta
j\left(x\right)}=\frac{\delta}{\delta
j\left(x\right)}\sum_{n=0}^{\infty}\frac{1}{n!}\int
G_{n}\left(x_{1},\cdots,x_{n}\right)\,j\left(x_{1}\right)\cdots
j\left(x_{n}\right)\,dx_{1}\cdots dx_{n}$ $\displaystyle=$
$\displaystyle\sum_{n=1}^{\infty}\frac{1}{(n-1)!}\int
G_{n}\left(x,x_{2},\cdots,x_{n}\right)\,j\left(x_{2}\right)\cdots
j\left(x_{n}\right)\,dx_{2}\cdots dx_{n}$ $\displaystyle=$ $\displaystyle\int
G\left(\\{x\\}+Y\right)\,j^{Y}\,dY.$

This may be written more succinctly as $\frac{\delta}{\delta
j\left(x\right)}G_{Y}j^{Y}=G_{Y+\\{x\\}}j^{Y}$. More generally we have

$\displaystyle\frac{\delta Z_{G}(j)}{\delta j^{X}}=\int
G\left(X+Y\right)j^{Y}dY,$ (40)

or $\dfrac{\delta Z_{G}(j)}{\delta j}=\int G(\cdot+Y)j^{Y}dY$.

######  Proposition 15 (The Leibniz Rule for fields)

For analytic functionals $U$ and $V$ we have that

$\displaystyle\dfrac{\delta}{\delta j}\big{(}UV\big{)}=\dfrac{\delta U}{\delta
j}\diamond\dfrac{\delta V}{\delta j}.$

To see this, set $U=Z_{F}$ and $V=Z_{G}$, then we are required to show that

$\displaystyle\dfrac{\delta}{\delta
j^{X}}Z_{F}(j)Z_{G}(j)=\sum_{X_{1}+X_{2}=X}\dfrac{\delta Z_{F}(j)}{\delta
j^{X_{1}}}\dfrac{\delta Z_{G}(j)}{\delta j^{X_{2}}}.$

The proof follows by elementary induction. For several terms we just find
multiple Wick products.

We can also investigate the form of the chain rule. For a given functional
$Z=Z(j)$,

$\displaystyle\frac{\delta h\left(Z\right)}{\delta j\left(x_{1}\right)}$
$\displaystyle=$ $\displaystyle h^{\prime}\left(Z\right)\frac{\delta Z}{\delta
j\left(x_{1}\right)},$ $\displaystyle\frac{\delta^{2}h\left(Z\right)}{\delta
j\left(x_{1}\right)\delta j\left(x_{2}\right)}$ $\displaystyle=$
$\displaystyle h^{\prime\prime}\left(Z\right)\frac{\delta Z}{\delta
j\left(x_{1}\right)}\frac{\delta Z}{\delta
j\left(x_{2}\right)}+h^{\prime}\left(Z\right)\frac{\delta^{2}Z}{\delta
j\left(x_{1}\right)\delta j\left(x_{2}\right)},$ $\displaystyle\vdots$

the pattern is easy to spot and we establish it in the next Lemma.

######  Lemma 16 (Chain Rule for fields)

Let $Z=Z(j)$ possess Fréchet derivatives to all orders and let $h$ be smooth
analytic, then

$\displaystyle\frac{\delta\;}{\delta
j^{X}}h\left(Z(j)\right)=\sum_{\mathcal{X}\in\mathrm{Part}\left(X\right)}h^{\left(N\left(\mathcal{X}\right)\right)}\left(Z(j)\right)\,\left(\frac{\delta
Z}{\delta j}\right)_{\mathcal{X}},$ (41)

where $h^{\left(n\right)}\left(\cdot\right)$ is the $n$th derivative of $h$.

Proof. This is easily seen by induction on. As $\dfrac{\delta
h\left(Z\right)}{\delta j^{x}}=h^{\prime}\left(Z\right)\dfrac{\delta Z}{\delta
j\left(x\right)}$, the identity is trivially true for $n=1$. Now assume that
it is true for $n$, and let $|X|=n$, then

$\displaystyle\dfrac{\delta\;}{\delta j\left(x\right)}\frac{\delta
h\left(Z\right)}{\delta j^{X}}$ $\displaystyle=$
$\displaystyle\dfrac{\delta}{\delta
j\left(x\right)}\sum_{\mathcal{X}\in\mathrm{Part}\left(X\right)}h^{\left(N\left(\mathcal{X}\right)\right)}\left(Z(j)\right)\,\left(\frac{\delta
Z}{\delta j}\right)_{\mathcal{X}}$ $\displaystyle=$
$\displaystyle\sum_{\mathcal{X}\in\mathrm{Part}\left(X\right)}h^{\left(N\left(\mathcal{X}\right)+1\right)}\left(Z(j)\right)\,\frac{\delta
Z}{\delta j\left(x\right)}\left(\frac{\delta Z}{\delta
j}\right)_{\mathcal{X}}$
$\displaystyle+\sum_{\mathcal{X}\in\mathrm{Part}\left(X\right)}h^{\left(N\left(\mathcal{X}\right)\right)}\left(Z(j)\right)\,\dfrac{\delta\;}{\delta
j\left(x\right)}\left(\frac{\delta Z}{\delta j}\right)_{\mathcal{X}}$

however, the first term on the right hand side is a sum over all parts of
$X+\left\\{x\right\\}$ having $x$ occurring as a singleton, while the second
term, when differentiated with respect to $j\left(x\right)$, will be a sum
over all parts of $X+\left\\{x\right\\}$ having $x$ in some part containing at
least one element of $X$. Thus we may write the above as

$\displaystyle\frac{\delta h\left(Z\right)\;}{\delta
j^{X+\left\\{x\right\\}}}=\sum_{\mathcal{X}\in\mathrm{Part}\left(X+\left\\{x\right\\}\right)}h^{\left(N\left(\mathcal{X}\right)\right)}\left(Z(j)\right)\,\left(\frac{\delta
Z}{\delta j}\right)_{\mathcal{X}}.$

The identity then follows by induction.

The chain rule is in fact a generalization of the Faà di Bruno formula.

###  3.2 Feynman Diagrams

###### Corollary 17

Given an analytic functional $V\left[\phi\right]=\int v^{X}\phi_{X}dX$, where
we assume that $V\left[0\right]=0$, then

$\displaystyle\mathbb{E}\left[e^{V(\phi)]}\right]=\left(\sum_{\mathcal{X}\in\mathrm{Part}\left(X\right)}v_{\mathcal{X}}\right)G_{X}$
(42)

where $v_{\mathcal{X}}\triangleq\prod_{A\in\mathcal{X}}v^{A}$.

Proof. We have $v^{\emptyset}=0$, and so we use (24) to get

$\displaystyle\mathbb{E}\left[V(\phi)^{n}\right]$ $\displaystyle=$
$\displaystyle\left(v^{\diamond n}\right)^{X}G_{X}\equiv
n!\left(\sum_{\mathcal{X}\in\mathrm{Part}_{n}\left(X\right)}v_{\mathcal{X}}\right)G_{X}.$

The relation (42) then follows by summing the exponential series.

The expression (42) applies to a general state. If we wish to specify to a
Gaussian state $\mathbb{E}_{g}$ then the expression specializes further.

###### Theorem 18

Let $\Xi=\mathbb{E}_{g}\left[e^{V(\phi)}\right]$, then

$\displaystyle\Xi\equiv\int\sum_{\mathcal{X}\in\mathrm{Part}\left(X\right)}\sum_{\mathcal{P}\in\mathrm{Pair}\left(X\right)}v_{\mathcal{X}}g_{\mathcal{P}}\,dX.$
(43)

The proof is then just a simple substitution of the explicit form (27) for the
Gaussian moments into (42). To understand this expression, let us look at a
typical term appearing on the right hand side.

Let us fix a set $X$, say $X=\left\\{x_{1},\cdots,x_{10}\right\\}$ \- there
must be an even number of elements otherwise the contribution vanishes! We fix
a partition $\mathcal{X}=\left\\{A,B,C\right\\}$ of $X$, say
$A=\left\\{x_{1},x_{2},x_{3}\right\\}$, $B=\left\\{x_{4},x_{5},x_{6}\right\\}$
and $C=\left\\{x_{7}x_{8},x_{9},x_{10}\right\\}$, and a pair partition
$\mathcal{P}$ consisting of the pairs
$\left(x_{1},x_{2}\right),\left(x_{3},x_{8}\right),\left(x_{7},x_{10}\right),\left(x_{9},x_{5}\right),\left(x_{4},x_{6}\right)$.
The contribution is the

$\displaystyle
v^{x_{1}x_{2}x_{3}}v^{x_{4},x_{5}x_{6}}v^{x_{7}x_{8}x_{9}x_{10}}g_{x_{1}x_{2}}g_{x_{3}x_{8}}g_{x_{7}x_{10}}g_{x_{9}x_{5}}g_{x_{4}x_{6}}$

where we have an implied integration over repeated dummy indices. With this
contribution we associate the diagram in Figure 1.

Figure 1: With each of the three parts $A,B,C$ in the partition, we associate
a vertex with one line for each of the elements for each part. We then connect
the lines to form edges of a graph.

More generally we have the following [4].

###### Theorem 19

The moments of the state (30) are given by

$\displaystyle
G_{X}\equiv\frac{1}{\Xi}\int\sum_{\mathcal{X}\in\mathrm{Part}\left(Y\right)}\sum_{\mathcal{P}\in\mathrm{Pair}\left(X+Y\right)}v_{\mathcal{X}}g_{\mathcal{P}}\,dY.$
(44)

Here the rules are as follows: choose all possible subsets $Y$, all possible
partitions $\mathcal{X}$ of $Y$ and all possible pair partitions $\mathcal{P}$
of $X+Y$; draw an $m$-vertex for each part of $\mathcal{X}$ of size $m$, label
all the edges at each vertex by the corresponding elements of $Y$, connect up
all elements of $X+Y$ according the pair partition. We integrate over all
$Y$’s, and sum over all $\mathcal{X}$’s and $\mathcal{P}$’s.

##  4 The Dyson-Schwinger Equation

###### Definition 20

Let $g$ be a metric. We define the following operations on function over
$\mathscr{P}(M)$ by

$\displaystyle(b_{x}G)_{X}$ $\displaystyle\triangleq$ $\displaystyle
G_{X+\\{x\\}},$ $\displaystyle(b_{x}^{\ast}G)_{X}$ $\displaystyle\triangleq$
$\displaystyle\sum_{x^{\prime}\in X}g_{xx^{\prime}}G_{X-\\{x\\}},$
$\displaystyle b^{x}$ $\displaystyle\triangleq$ $\displaystyle g^{xy}b_{y}.$
(45)

We also use the notation $b_{X}=\prod_{x\in X}b_{x}$, etc.

The operators satisfy the canonical commutation relations

$\displaystyle[b_{x},b_{y}^{\ast}]=g_{xy},\qquad[b^{x},b_{y}^{\ast}]=\delta^{x}_{y}.$
(46)

We also note the identity

$\displaystyle(b_{x}G)_{X}j^{X}\equiv\frac{\delta}{\delta j^{x}}(G_{X}j^{X}).$
(47)

The next result is very easily established.

###### Proposition 21

The Gaussian functional
$Z_{g}(j)=G_{X}^{g}j^{X}=e^{\frac{1}{2}j^{x}g_{xy}j^{y}}$ satisfies

$\displaystyle(g^{xy}\frac{\delta}{\delta j^{y}}-j^{x})Z_{g}=0.$ (48)

Alternatively we may look at what this means for the Gaussian moments. We have
$(b_{x}G^{g})_{X}=G^{g}_{X+\\{x\\}}$. This will clearly vanish for $X$ even,
while for $X$ odd we have that $G^{g}_{X+\\{x\\}}$ will have one factor of the
form $g_{xx^{\prime}}$ for some $x^{\prime}\in X$ with the other factors being
metric terms corresponding to pairs from $X-\\{x^{\prime}\\}$. Therefore,
$(b_{x}G^{g})_{X}=\sum_{x^{\prime}\in
X}g_{xx^{\prime}}G^{g}_{X-\\{x^{\prime}\\}}\equiv(b^{\ast}_{x}G^{g})_{X}$.
This leads to the striking formula for Gaussian moments

$\displaystyle(b_{x}-b^{\ast}_{x})G^{g}\equiv 0.$ (49)

######  Theorem 22 (The Dyson-Schwinger Equation)

Let $\mathbb{E}_{G}$ be the state, then

$\displaystyle\bigg{(}g^{xy}\frac{\delta}{\delta
j^{y}}+F_{I}^{x}(\frac{\delta}{\delta j})-j^{x}\bigg{)}Z_{G}=0,$ (50)

where $F^{x}_{I}(\varphi)=-\frac{\delta V}{\delta\varphi_{x}}$. In terms of
the moments, we have

$\displaystyle\bigg{(}b_{x}-b_{x}^{\ast}+g_{xy}V^{Y+\\{y\\}}b_{Y}\bigg{)}G\equiv
0.$ (51)

Proof. The moment generating function $Z_{G}(j)$ may be written as

$\displaystyle
Z_{G}(j)=\frac{1}{\Xi}\mathbb{E}_{g}\left[e^{\varphi_{x}j^{x}+V\left[\varphi\right]}\right]=\frac{1}{\Xi}\exp\left\\{V(\frac{\delta}{\delta
j})\right\\}Z_{g}(j).$ (52)

We observe that
$j^{x}Z_{G}(j)=\frac{1}{\Xi}j^{x}\exp\left\\{V(\frac{\delta}{\delta
j})\right\\}\,Z_{g}(j)$, and using the commutation identity

$\displaystyle\left[j^{x},e^{V(\frac{\delta}{\delta
j})}\right]=F^{x}_{I}\big{(}\frac{\delta}{\delta
j}\big{)}e^{V(\frac{\delta}{\delta j})}$

we find

$\displaystyle j^{x}Z_{G}(j)$ $\displaystyle=$
$\displaystyle\frac{1}{\Xi}e^{V(\frac{\delta}{\delta
j})}\,j^{x}Z_{g}(j)+\frac{1}{\Xi}F^{x}_{I}(\frac{\delta}{\delta
j})e^{V(\frac{\delta}{\delta j})}\,Z_{g}(j)$ $\displaystyle=$ $\displaystyle
g^{xy}\frac{\delta}{\delta j^{y}}Z_{G}(j)+F_{I}^{x}(\frac{\delta}{\delta
j})Z_{G}(j)$

which gives the result.

An alternative form of the Dyson-Schwinger Equation is

$\displaystyle\left\\{F^{x}\left(\frac{\delta}{\delta
j}\right)+j^{x}\right\\}Z(j)=0,$ (53)

where $F^{x}(\varphi)=\frac{\delta
S(\varphi)}{\delta\varphi_{x}}=-\frac{1}{2}g^{xy}\varphi_{y}+F_{I}^{x}(\varphi)$.

The algebraic equations (51) may be written as

$\displaystyle G_{X+x}=\sum_{x^{\prime}\in
X}g_{xx^{\prime}}G_{X-x^{\prime}}+g_{xy}\int_{\mathscr{P}(M)}v^{y+Y}\,G_{X+Y}dY.$
(54)

We remark that the first term on the right hand side of (54) contains the
moments $\mathbb{E}\left[\phi_{X-x^{\prime}}\right]$ which are of order two
smaller than the left hand side $\mathbb{E}\left[\phi_{X+x}\right]$. The
second term on the right hand side of (54) contains the moments of higher
order, and so we generally cannot use this equation recursively. In the
Gaussian case we have
$\mathbb{E}_{g}\left[\phi_{X+x}\right]=\sum_{x^{\prime}\in
X}g_{xx^{\prime}}\mathbb{E}_{g}\left[\phi_{X-x^{\prime}}\right]\mathbb{E}_{g}\left[\phi_{X-x^{\prime}}\right]$
from which we can deduce (26) by just knowing the first and second moments,
$\mathbb{E}_{g}\left[\phi_{x}\right]=0$ and
$\mathbb{E}_{g}\left[\phi_{x}\phi_{y}\right]=g_{xy}$.

##  5 Tree Expansions

The generating functions
$Z(j)=\frac{1}{\Xi}\int_{\Phi}e^{\left\langle\varphi,j\right\rangle+S\left[\varphi\right]}\,\mathcal{D}\varphi$
may be given a stationary phase approximation

$\displaystyle Z(j)\simeq e^{\psi_{x}j^{x}+S(\psi)}.$ (55)

Assuming that the stationary solution $\psi=\psi(j)$ exists and is unique for
each fixed $j$, we have the identity

$\displaystyle j^{x}+\left.\frac{\delta
S(\varphi)}{\delta\varphi_{x}}\right|_{\varphi=\psi(j)}=0,$ (56)

that is, $j^{x}-g^{xy}\psi_{y}+\int v^{x+X}\psi_{X}\,dX=0$, or rearranging
gives (lowering indices using the metric $g$)

$\displaystyle\psi_{x}=j_{x}+v_{x}^{\;X}\psi_{X}.$ (57)

We may rewrite (57) as $\psi_{x}=j_{x}+\mathcal{V}_{x}(\psi)$ where we have
the linear operator $\mathcal{V}_{x}(\psi)=v_{x}^{\;X}\psi_{X}$. Therefore,
$(I-\mathcal{V})\psi=j$ and formally this may be solved as the geometric
series $\psi=\sum_{n=0}^{\infty}\mathcal{V}^{n}j$. Here, for instance, the
second iterate would be
$(\mathcal{V}^{2}j)_{x}=v_{x}^{\;X}\prod_{x^{\prime}\in
X}\big{(}v_{x^{\prime}}^{\;X(x^{\prime})}j_{X(x^{\prime})}\big{)}$. This last
expression has an implied Guichardet integration over $X$ and also the $\\#X$
variables $X(x^{\prime})$ labelled by $x^{\prime}\in X$.

In general, we encounter coefficients of the form
$v_{\mathsf{X}_{0}}^{\;\mathsf{X}_{1}}v_{\mathsf{X}_{1}}^{\;\mathsf{X}_{2}}\cdots
v_{\mathsf{X}_{n-1}}^{\;\mathsf{X}_{n}}$ in the $n$th iterate where
$(\mathsf{X}_{0},\mathsf{X}_{1},\cdots,\mathsf{X}_{n})$ is a sequence with
$\mathsf{X}_{m}\in\mathscr{P}^{m}(M)$.

We may write the expansion in terms of hierarchies. A hierarchy over a finite
set $X$ is a directed tree having subsets of $X$ assigned to each node with
the property that if $B\subset X$ is assigned to a particular node then
subsets assigned to the immediate daughter nodes form a partition of $B$. The
hierarchies $\mathrm{Hier}(X)$ are equivalent to the rooted phylogenetic trees
whose leaves are the elements of $X$, as well as the total partitions of $X$:
these are enumerated as sequence A000311 on the Online Encyclopedia of Integer
Sequences [9]. In fact we can give the analytic expansion of $\psi_{x}$ in
terms of the current $j$ which in this case is a tree-expansion [4]

$\displaystyle\psi_{x}=\sum_{H\in\text{Hier}\left(X\right)}v_{x}\left(H\right)\,j^{X}$
(58)

where the weight $v_{x}\left(H\right)$ is calculated by drawing out the tree
with $x$ as root and having a factor $v_{x_{B}}^{\;x_{A_{1}}\cdots x_{A_{m}}}$
for each node where the labels $x_{B},x_{A_{1}},\cdots,x_{A_{m}}$ are dummy
variables in $M$ for the node $B$ and the daughters $A_{1},\cdots,A_{m}$.
Apart from the root ($x_{B}=x$), and the leaves, all these variables are
contracted over in the product of such factors (the contractions corresponding
to the branches between nodes!).

## References

  * [1] S.S. Schweber, QED and the men who made it: Dyson, Feynman, Schwinger, and Tomonaga, Princeton University Press (1994).
  * [2] Feynman, Richard, The Theory of Positrons, Physical Review. 76 (6): 749–759, (1949).
  * [3] R.J. Rivers, Path integral methods in quantum field theory, Cambridge Monographs on Mathematical Physics, (1987); P. Cvitanović, Field Theory, ChaosBook.org/FieldTheory, Niels Bohr Institute (Copenhagen 2004); M. Veltman, Diagrammatica: The Path to Feynman Diagrams, Cambridge Lecture Notes in Physics, (1994).
  * [4] J. Gough, J. Kupsch, Quantum Fields and Processes: A Combinatorial Approach Cambridge studies in advanced mathematics, 171, Cambridge, 2018.
  * [5] J. Schwinger, A report on quantum electrodynamics. In J. Mehra (ed.), The Physicist’s Conception of Nature. Dordrecht: Reidel (1973).
  * [6] A. Guichardet, Symmetric Hilbert Spaces. Lecture Notes in Mathematics, 261, 1970.
  * [7] H. Maassen, Quantum Markov Processes on Fock space described by integral kernels, in Lecture Notes in Mathematics 1136, Eds. L. Accardi and W. von Waldenfels, pp. 361-374, 1985.
  * [8] P.-A. Meyer, Quantum Probability for Probabilists, Lecture Notes in Mathematics, 1538, 1993.
  * [9] The On-line Encyclopedia of Integer Sequences, www.oeis.org