# Likelihood-Based Text-to-Image Evaluation with
Patch-Level Perceptual and Semantic Credit Assignment

Qi Chen1,∗, Chaorui Deng1,∗, Zixiong Huang2, Bowen Zhang1, Mingkui Tan2, Qi
Wu1,†
1Australia Institute of Machine Learning, University of Adelaide
2South China University of Technology

###### Abstract

Text-to-image synthesis has made encouraging progress and attracted lots of
public attention recently. However, popular evaluation metrics in this area,
like the Inception Score and Fréchet Inception Distance, incur several issues.
First of all, they cannot explicitly assess the perceptual quality of
generated images and poorly reflect the semantic alignment of each text-image
pair. Also, they are inefficient and need to sample thousands of images to
stabilise their evaluation results. In this paper, we propose to evaluate
text-to-image generation performance by directly estimating the likelihood of
the generated images using a pre-trained likelihood-based text-to-image
generative model, i.e., a higher likelihood indicates better perceptual
quality and better text-image alignment. To prevent the likelihood of being
dominated by the non-crucial part of the generated image, we propose several
new designs to develop a credit assignment strategy based on the semantic and
perceptual significance of the image patches. In the experiments, we evaluate
the proposed metric on multiple popular text-to-image generation models and
datasets in accessing both the perceptual quality and the text-image
alignment. Moreover, it can successfully assess the generation ability of
these models with as few as a hundred samples, making it very efficient in
practice. 00footnotetext: ∗Co-first author. †Corresponds to
<EMAIL_ADDRESS>Code is available at
https://github.com/chenqi008/LEICA.

##  1 Introduction

Recent works [6, 7, 13, 17, 16, 22, 33] have made promising achievements on
the text-to-image generation task, which can produce photo-realistic images or
impressive artworks according to the text inputs. However, automatic
evaluation of the text-to-image generation performance has always been a
challenging problem, especially in complex, open-vocabulary scenarios. Most
existing text-to-image generation models, including those state-of-the-art
models, still rely on the popular evaluation metrics in general image
generation, such as the Inception Score (IS) [23] and Fréchet Inception
Distance (FID) [9].

The IS leverages a pre-trained image recognition model (Inception-V3 [24]) to
provide the recognition confidence of the generated images to assess the
generative performance. A higher confidence score may indicate that the
generated images are more similar to those used for training the Inception-V3
(i.e., ImageNet [4]). However, the critical issue is that the similarity with
the pre-training images does not necessarily reflect the perceptual quality or
semantic alignment of the generated images, since there exists a significant
domain shift between the pre-training data (which has relatively limited
semantic diversity) and the open-vocabulary natural languages. Besides, the
pre-trained model can also be over/under-fitted. Thus, the confidence scores
may be unreliable. Likewise, the followers of IS, such as Classification
Accuracy Score (CAS) [18] and Semantic Object Accuracy (SOA) [10], could also
suffer from these problems.

The FID, on the other hand, computes the distribution distance between the
real images and the generated images in a latent space, where the backbone of
the Inception-V3 is used to map those images into the latent space, and a
multivariate Gaussian assumption is imposed on the latent space to estimate
the distribution statistics. Although it addresses the domain shift problem of
IS (to some extent) by considering real images during evaluation, FID still
faces several issues. First, the assumption of the multivariate Gaussian
distribution is groundless, making it a poor indicator of the perceptual
quality of generated images. Second, FID is designed only to consider the
overall similarity of two sets of images, which cannot measure the semantic
alignment between each input text and the corresponding generated image.
Finally, both IS and FID is inefficient, as they have to process lots of image
samples using the Inception V3 model to stabilize their evaluation results.
Most of these drawbacks can also be found in Kernel Inception Distance (KID)
[2], a kernelized variant of FID.

Tackle these problems are challenging. In this paper, we propose a new metric
for text-to-image evaluation based on likelihood estimation. Specifically,
given $n$ image descriptions $X=\\{x_{i}\\}_{i=1}^{n}$ and the corresponding
generated images $Y=\\{y_{i}\\}_{i=1}^{n}$, we leverage a pre-trained
likelihood-based text-to-image model $\mathcal{G}$ to estimate the likelihood
of each $y_{i}$ given $x_{i}$ as condition, i.e.,
$P_{\mathcal{G}}(y_{i}|x_{i})$. Ideally, when the model-estimated density is
closely aligned with the real-world data density, then
$P_{\mathcal{G}}(y_{i}|x_{i})$ would effectively indicate how likely the image
$y_{i}$ exists in real-world when the text $x_{i}$ is given [25]. In this
sense, to achieve a high $P_{\mathcal{G}}(y_{i}|x_{i})$, $y_{i}$ should not
only be semantically aligned with $x_{i}$, but also be perceptually good
enough as the real-world images. Otherwise, we will get a low likelihood score
for $y_{i}$. In fact, average log-likelihood has been the de facto standard
and widely considered as the default measure for detecting out-of-distribution
data [20] or quantifying generative modeling performance [25, 26]. However,
existing text-to-image evaluation methods have to resort to the aforementioned
alternatives due to the lack of a well-trained likelihood estimator
$\mathcal{G}$.

We address this problem using the recently emerged likelihood-based
autoregressive image generation models [17, 30], which have shown impressive
text-to-image generation performance. Concretely, they first rely on a
discrete variational autoencoder (dVAE) [27, 7] to learn a codebook so that an
image can be converted into a sequence of visual codes and vice versa. The
generative modeling is then accomplished by a transformer model [28], which
takes the text $x_{i}$ as input and predicts the likelihood of the visual
codes autoregressively for the image $y_{i}$. In this way, we can obtain
$P_{\mathcal{G}}(y_{i}|x_{i})$ as the product of the likelihood of the visual
codes. In the evaluation setting, $y_{i}$ is given, and its visual codes can
be directly obtained from the dVAE. Thus, their likelihoods can be efficiently
predicted in parallel with just one forward pass of the transformer, instead
of the step-by-step autoregressive manner.

Nevertheless, we observe that even with such a good estimator,
$P_{\mathcal{G}}(y_{i}|x_{i})$ can still be dominated by: 1) The visual codes
whose corresponding patterns occasionally appear in the generated images but
rarely exist in real-world scenes. We refer to them as visual codes with low
perceptual significance. Although they may not affect the overall perceptual
quality of $y_{i}$ as they generally account for only a small ratio in the
whole image, they tend to have extremely small likelihoods and drag down the
overall likelihood $P_{\mathcal{G}}(y_{i}|x_{i})$; 2) The visual codes that
are semantically irrelevant to the text, i.e., those corresponding to the
background. which exhibits low semantic significance, but usually accounts for
the largest portion of the image and thus affects
$P_{\mathcal{G}}(y_{i}|x_{i})$ mostly.

To tackle these issues, we explicitly introduce a credit assignment strategy
based on the perceptual and semantic significance of each visual code.
Specifically, we first design an indicator function to eliminate the factors
of the rare visual codes from $P_{\mathcal{G}}(y_{i}|x_{i})$, so that only the
perceptually significant codes are considered. Then, we design a semantic
scoring function using a pre-trained image-text matching model CLIP [15] to
down-weight the visual codes with low semantic significance. Since the
original CLIP can only predict an alignment score for the whole image and the
text, we make non-trivial modifications to make it suitable for patch-level
image-text matching, including a new inference-time model architecture and a
new scoring function. We denote our proposed metric as Likelihood-based tExt-
to-Image evaluation with Credit Assignment (LEICA).

We verify the effectiveness of the proposed LEICA score on 11 state-of-the-art
text-to-image generation models, including three Generative Adversarial
Network (GAN) based models [37, 10, 32], four autoregressive models [17, 16,
5, 6], and four diffusion models [21, 8, 22, 13]. We also evaluate LEICA on
multiple text-to-image datasets, including one open-domain dataset COCO [12],
and two specific-domain datasets CUB [29] and Oxford-Flower [14]. We provide
extensive analysis, including human studies, to show the superiority of our
evaluation metric.

##  2 Related Works

#### Text-to-Image Generation

models can be roughly categorized into three types: Generative Adversarial
Networks (GAN), likelihood-based models, and score-based models. Early works
in this area are primarily GAN-based, such as StackGAN [36], AttnGAN [31], DM-
GAN [38], and OP-GAN [10]. However, GAN is essentially to model the sampling
process of a distribution, which means we cannot directly obtain the
likelihoods of the generated samples or anything that is related to the
generation performance, and thus cannot be used for text-to-image evaluation.
For score-based models, a representative line of works is the recently popular
diffusion models [21, 8, 22, 13], which achieve excellent performance in
generating photo-realistic images and artworks. Although diffusion models are
able to obtain the likelihood of the generated image, it requires multiple
forward steps of the model using inputs with different noise levels, which
cannot be parallelized when estimating the likelihood of a generated image.
The dominant likelihood-based models, such as DALL-E [17], OFA [30], CogView
[5, 6], and Parti [33], are all autoregressive models. They not only perform
as well as diffusion models, but more importantly, given a generated image,
they can efficiently obtain its likelihood within one forward step, making
them very suitable for evaluating text-to-image generation.

#### Evaluation Metrics for Text-to-Image Generation.

As we have discussed in Section 1, the widely used metrics for generative
image modeling, such as IS [23], CAS [18], FID [9], and KID [2], tend to be
problematic in text-to-image evaluation. SOA [10] is specifically designed for
text-to-image generation on the MSCOCO [12] dataset. It first filters all
captions in the validation set for the keywords that are related to the object
categories in MSCOCO. For each filtered caption, several images are generated
and fed into an object detector (YOLOv3 [19] trained on MSCOCO) to see if any
keywords-related object can be detected. The average recall of these keywords
over all categories or images is reported as the SOA-C and SOA-I scores,
respectively. Although it can evaluate the semantic alignment between the
captions and the generated images, it is inconvenient to use (due to caption
pre-processing) and not generalizable (captions and detectors are all bound to
MSCOCO). Besides, it has to face all the problems that IS encounters, as
mentioned in Section 1. Some retrieval metrics, like R-precision, are used
only to evaluate the semantic alignment of the generated images [31] while
ignoring the perceptual quality. In contrast, average log-likelihood avoids
the above phenomena, which is also widely considered as the default measure
for evaluating generative models [25, 26].

##  3 Method

#### Overview

The overall pipeline of our evaluation metric LEICA consists of a likelihood
estimation step and two credit assignment steps on the perceptual and semantic
significance, respectively. First of all, the likelihood of the generated
image $y_{i}$ is estimated by

$P_{\mathcal{G}}(y_{i}|x_{i})=\prod_{t=1}^{m}P_{\mathcal{G}}(c_{i,t}|c_{i,<t},x_{i}),$
(1)

where $\\{c_{i,t}\\}_{t=1}^{m}$ are the visual codes obtained from the dVAE
given $y_{i}$ as input. Each $c_{i,t}$ is mainly responsible for representing
the information nearby a specific spatial (patch) location in $y_{i}$, and $t$
is the flattened index of the patch. $m$ is the total number of patches, and
$c_{i,<t}$ indicates the codes before $t$, i.e., $c_{i,1},...,c_{i,t-1}$. Note
that, Eq. (1) serves as the basis of the proposed evaluation metric, where
this vanilla likelihood estimation can provide a generally unbiased but
unstable evaluation due to the existence of perceptually & semantically
insignificant visual codes.

Then, we introduce an indicator function $H$ to explicitly eliminate the
visual codes with low perceptual significance from Eq. (1), i.e.,

$\frac{1}{m}\sum_{t=1}^{m}H(P_{\mathcal{G}}(c_{i,t}|c_{i,<t},x_{i})).$ (2)

We convert the likelihoods into log-likelihoods inside $H$ so the summation
sign is used here, and the scores are averaged among all codes. Lastly, we
leverage the CLIP [15], a well-trained image-text matching model (denoted by
$\mathcal{M}$) to focus on the locations with high semantic significance. As
the CLIP can only provide the global matching score between an image-text
pair, we modify its inference-time architecture to get $\mathcal{M}^{\prime}$,
and further devise a scoring function $S$, so as to assign a semantic
alignment score for each location $t$. The final evaluation score of LEICA on
a sample ($x_{i}$, $y_{i}$) is then computed by

$\text{LEICA}(x_{i},y_{i})=\frac{1}{m}\sum_{t=1}^{m}S(x_{i},y_{i},t)H(P_{\mathcal{G}}(c_{i,t}|c_{i,<t},x_{i})).$
(3)

More details are in the following.

(a) Stable Diffusion

(b) OPGAN

Figure 1: Generated images for the caption “A bathroom with a blue shower
curtain and walls” with OPGAN and stable diffusion (the first column) and
their visualized likelihood maps (the second column). Although stable
diffusion produces a much better result, its average likelihood is
$1.401\times 10^{-4}$, less than OPGAN ($8.020\times 10^{-4}$). After removing
the rare codes (the last column), their likelihoods increase to $3.297\times
10^{-3}$ and $1.670\times 10^{-3}$, respectively, where stable diffusion is
considered to be better.

###  3.1 Evaluation via Likelihood Estimation

To directly estimate the likelihood of a generated image $y_{i}$ conditioned
on the image description $x_{i}$, we leverage a well-trained likelihood-based
text-to-image generator $\mathcal{G}$, which consists of a dVAE model and a
transformer model.

Specifically, the dVAE is originally learned to convert an image into a
sequence of visual codes and vice versa, so that the generation of images can
be reformulated as the generation of their code sequences. It is a
convolutional encoder-decoder architecture accompanied by a learnable codebook
that contains $K$ visual codes. In our evaluation setting, we can directly use
the trained dVAE to tokenize the generated images. Suppose
$y_{i}\in\mathbb{R}^{3\times H\times W}$, the encoder of dVAE first maps it
into a feature map $\bm{f}_{i}\in\mathbb{R}^{d\times h\times w}$ where
$h,w<H,W$. Then, each spatial location in $\bm{f}_{i}$ will be represented by
its nearest code retrieved from the codebook, and $y_{i}$ is thus represented
by $m=h\times w$ visual codes, i.e., $\\{c_{i,t}\\}_{t=1}^{m}$. The decoder of
dVAE is used for reconstructing $y_{i}$ from $\\{c_{i,t}\\}_{t=1}^{m}$, which
is not required in our setting.

The transformer is trained to model the joint distribution between image
descriptions and the code sequences of the corresponding images. The joint
distribution is explicitly optimized to have a causal chain structure, where
each visual code is predicted conditioning only on previous visual codes and
the text. Thus, during the generation procedure, the visual codes are
predicted in an autoregressive manner. When it comes to our evaluation
setting, both $x_{i}$ and $\\{c_{i,t}\\}_{t=1}^{m}$ are given, we thus can
obtain the conditional likelihood
$P_{\mathcal{G}}(\\{c_{i,t}\\}_{t=1}^{m}|x_{i})$ by one forward pass of the
transformer, which serves as a lossless approximate to
$P_{\mathcal{G}}(y_{i}|x_{i})$. Due to its causal chain structure,
$P_{\mathcal{G}}(\\{c_{i,t}\\}_{t=1}^{m}|x_{i})$ can be factorized as the
product of the likelihood of each $c_{i,t}$, leading to the form in Eq. (1).
We further consider using log-likelihood instead of likelihood to avoid too
small values, i.e., $\ln P_{\mathcal{G}}(y_{i}|x_{i})=\sum_{t=1}^{m}\ln
P_{\mathcal{G}}(c_{i,t}|c_{i,<t},x_{i})$.

###  3.2 Credit Assignment of Perceptual Significance

While it is feasible to directly assess the text-to-image generation
performance using the likelihood estimated above, we observed that there
usually exists a small portion of codes in $\\{c_{i,t}\\}_{t=1}^{m}$ that have
extremely small likelihoods (typically less than $10^{-9}$) and severely drag
down the overall likelihood of $y_{i}$. As shown in Figure 1, this phenomenon
occurs no matter what the actual qualitative performance is. As a result, a
well-synthesized image may be rated lower than a poorly-synthesized image just
because it contains more rarely-used visual codes.

To handle this problem, we introduce an indicator function $H$ on the
likelihood of each visual code into Eq. (1) to explicitly focus on those with
high perceptual significance:

$\begin{split}H(P_{\mathcal{G}}(c_{i,t}|c_{i,<t},x_{i}))=\mathbbm{1}(\ln
P_{\text{data}}(c_{i,t})-\lambda)&\\\ ~{}*~{}\max(\ln
P_{\mathcal{G}}(c_{i,t}|c_{i,<t},x_{i})-\lambda,0)&,\end{split}$ (4)

Here, $\mathbbm{1}(\cdot)$ equals 1 for positive inputs and 0 otherwise.
$P_{\text{data}}(c_{i,t})$ is the prior probability of $c_{i,t}$ estimated
from real images. $\lambda<0$ is a threshold that controls whether the effect
of $c_{i,t}$ should be considered or not according to both its prior
probability $P_{\text{data}}(c_{i,t})$, i.e., the perceptual significance of
$c_{i,t}$ in real images, and the predicted likelihood
$P_{\mathcal{G}}(c_{i,t}|c_{i,<t},x_{i})$, which indicates its perceptual
significance in the generated image. In this way, only the visual codes whose
predicted likelihoods are larger than $e^{\lambda}$ are rewarded with positive
values, and those values will only take effect when the visual codes have
higher prior probability than $e^{\lambda}$. Otherwise,
$H(P_{\mathcal{G}}(c_{i,t}|c_{i,<t},x_{i}))$ will be zeroed out from the
summation in Eq. (2). By default, $\lambda$ is set to a small value, e.g.,
$\ln 10^{-9}$. Thus, $H$ will only affect a few codes in
$\\{c_{i,t}\\}_{t=1}^{m}$ that have extremely small likelihoods.

###  3.3 Credit Assignment of Semantic Significance

Since the background information (that is irrelevant to the input description
$x_{i}$) usually constitutes a large part of the generated image $y_{i}$, it
can easily dominate the overall likelihood, as shown in Figure 2. Therefore,
it would be beneficial to focus on the semantically significant part of
$y_{i}$, which is also very close to human behavior. To achieve this, an
intuitive idea is to use a pre-trained image-text matching model with zero-
shot recognition abilities to estimate the semantic significance. We choose
the CLIP [15], a multi-modality model trained on web-scale image-text pairs.

Figure 2: (a) are the images generated by the stable diffusion model. (b) and
(c) are the likelihood maps before and after the semantic credit assignment,
respectively. For these two images, their overall likelihoods are $2.422\times
10^{-4}$ and $4.123\times 10^{-4}$, respectively, where the second image which
misses the foreground object “male surfer” is rated higher than the first
image. This can be corrected after the semantic credit assignment, where the
overall likelihoods become $1.049\times 10^{-4}$ and $4.704\times 10^{-5}$ in
(c).

#### Adapt CLIP for Patch-Level Scoring

The CLIP model we adopted (denoted by $\mathcal{M}$) has two transformer
encoders to process images and texts, respectively. A feature vector is
extracted from each modality with the corresponding encoder and is linearly
projected into the same dimensional space and $L2$-normalized. The
optimization of $\mathcal{M}$ is achieved via contrastive learning, where the
feature vectors that belong to the same image-text pair are pulled close, and
the others are pushed away. However, when used for our evaluation setting,
although we can assess how the generated $y_{i}$ and the text $x_{i}$ are
semantically close using the cosine similarity of their feature vectors, we
are unfortunately unable to assess the alignment at a more detailed level,
i.e., how each spatial location in $y_{i}$ aligns with $x_{i}$. This is caused
by the feature extraction mechanism of the image encoder. It takes $s\times s$
non-overlapping patches divided from the image and an extra [CLS] token as
input, and the final embedding of the [CLS] token is adopted as the output
image feature vector, while the final patch embeddings are all ignored.
Consequently, the contrastive loss is only applied on the [CLS] token, making
the similarity between the text feature vector and the final patch embeddings
uncertain during inference. We address this problem by proposing a new
inference-time architecture for $\mathcal{M}$, termed as
$\mathcal{M}^{\prime}$.

Our basic idea is to look for the patch representations in the computation
graph of $\mathcal{M}$ that is closest to the loss computation node, which
will also be the most relevant patch representations to the text feature
vector. As shown in Figure 3, the last interaction between the patch
embeddings and the [CLS] token embedding occurs within the last self-attention
operation of the image encoder. Thus, all the operations afterward, e.g., the
last feed-forward network, the last layer normalization, etc, are only used
for the [CLS] token and have no benefit to the modeling of patch-level image-
text relation. We thus remove them from $\mathcal{M}$. Moreover, inside the
last self-attention operation, the [CLS] token is updated with the patch
representations output by the value projection layer, denoted by
$\\{\bm{v}_{i,t}\\}_{t=1}^{s^{2}}$, while the query and key projection layers
are only used for computing the self-attention map. Therefore, we leverage
$\\{\bm{v}_{i,t}\\}_{t=1}^{s^{2}}$ to compute the patch-level alignment based
on their cosine similarities with the feature vector of $x_{i}$, denoted by
$\bm{x}_{i}$.

$\phi_{i}(t)=\frac{(\bm{W}^{\top}\bm{v}_{i,t})^{\top}\bm{x}_{i}}{\|\bm{W}_{1}^{\top}\bm{v}_{i,t}\|_{2}^{2}\cdot\|\bm{x}_{i}\|_{2}^{2}},~{}t={1,...,s^{2}}$
(5)

where $\bm{W}$ is the final linear project layer of the image encoder that
transforms $\bm{v}_{i,t}$ into the same dimensional space as $\bm{x}_{i}$. As
shown in Figure 7, with our modified CLIP model $\mathcal{M}^{\prime}$, the
obtained patch-level similarity scores are closely aligned with the foreground
part of the images.

(a) CLIP (b) Ours

Figure 3: The computation graph of the last layer in the original CLIP and our
modified CLIP.

#### Scoring Function for Semantic Significance

To obtain the final semantic significance scores for the predicted visual
codes $\\{c_{i,t}\\}_{t=1}^{m}$, we first apply a bilinear interpolation on
$\phi_{i}$ to resize its sequence length from $s^{2}$ to $m$. We then clip the
negative values in $\phi_{i}$ to zeros so that the unrelated parts in $y_{i}$
will not affect the final evaluation score. Finally, we can further consider
the global image-text alignment score with the original CLIP model
$\mathcal{M}$, to enhance the ability of our proposed LEICA metric on the
semantic alignment evaluation. Our scoring function for semantic significance
is defined as follows:

$S(x_{i},y_{i},t)=e^{\psi_{i}/\tau}\max(\phi_{i}(t),0),$ (6)

where $\psi_{i}$ is the cosine similarity between the feature vectors of
$y_{i}$ and $x_{i}$. As the cosine similarities obtain from CLIP are generally
within $(0.2,0.4)$, we use the exponential function to scale the values to a
broader range to distinguish the semantically-aligned/unaligned results
better. $\tau$ is a temperature hyper-parameter that controls the scaling
effect, which is set to $0.07$ following [3]. The final LEICA score is
calculated via Eq. (3).

##  4 Experiments

#### Meta Evaluation

To assess the quality of the proposed evaluation metrics, following [34], we
use four measures for the meta-evaluation: 1) Accuracy, in our experiments,
measures the percentage of correct judgment made by the metrics on whether a
text is matched with an image or is mismatched/altered. 2) Kendall’s Tau
Correlation measures the ordinal association between two sets of data. 3)
Pearson Correlation measures the linear correlation between two variables. 4)
Spearman Correlation assesses the consistency between the ranking of two
variables.

#### Implementation Details

For the pre-trained likelihood-based text-to-image generation model $G$, we
use OFA [30], which is the best-performed and open-sourced autoregressive
text-to-image generation model on MSCOCO [12] dataset. Still, we can always
update it to the most powerful model in the future for better evaluation
capability. For the images used for evaluation, we generate them using the
captions from the validation set of Karpathy’s split on MSCOCO, where the
images and captions are semantically diverse enough and are also commonly used
to evaluate the performance of text-to-image generation models in previous
works [37, 31, 35, 8]. Besides, we consider two datasets with specific
domains: CUB [29] and Oxford-Flower [14] to test the effectiveness of our
score. We set the default resolution for the generated images to $256\times
256$, following the common practice in text-to-image generation.

(a) Gaussian Noise (b) Gaussian Blur

(c) Salt & Pepper Noise (d) Word Replacement

Figure 4: (a)-(c): The trend of the proposed LEICA score (Score Value) and the
corresponding values of Kendall’ Tau (K-Tau) when adding different noises with
different disturbance degrees to the images. (d): The trend of the LEICA score
and Accuracy (Acc) of its judgments w.r.t.  semantic alignment when gradually
replacing the keywords in texts.

###  4.1 Results on Manually Synthesized Data

We first manually introduce artifacts into the real-world image-text pairs
from the MSCOCO dataset, so that we have oracle judgments for those data that
can be used to assess the performance of LEICA. Specifically, for the
effectiveness of LEICA on perceptual image quality, we add noises with
multiple degrees into the images while keeping the paired texts unchanged. We
consider several types of noises, including Gaussian Noise (GN), Gaussian Blur
(GB), and Salt & Pepper Noise (SPN). For more complicated distortions, we
combine a “funny mirror” distortion [1] with these three noises, named GN+,
GB+, and SPN+. Besides, we reconstruct images using the dVAE model of DALL-E
[17] to obtain a type of model-based distortion. For semantic text-image
alignment, we gradually replace the keywords (i.e., those with strong semantic
meanings) in the text with random words, or directly use mismatched text for
the image, where the paired image stays unaltered. To ensure the reliability
of our results, we run each experiment five times and calculate their mean
values and standard deviations.

#### Perceptual Quality

As shown in Figure 4(a)-(c), as the disturbance degree gradually increases,
the LEICA score of these images decreases accordingly, showing that the
proposed LEICA score can reflect the perceptual quality of the given images.
Furthermore, for each text $x$, we take its paired clean image
$y_{\text{clean}}$ as well as the corresponding noised one $y_{\text{noise}}$
as a triplet, i.e., $(x,y_{\text{noise}},y_{\text{clean}})$. For each triplet,
if the LEICA score of $y_{\text{clean}}$ is higher than that of
$y_{\text{noise}}$, it is concordant with the oracle judgment. Otherwise, the
triplet is discordant. Based on this, we calculate Kendall’s Tau coefficient
for the triplets. From Figure 4(a)-(c), Kendall’s Tau coefficient becomes
larger as the disturbance becomes heavier, which further shows that LEICA can
reliably measure the perceptual quality of the image.

#### Semantic Alignment

See the result in Figure 4(d), as the number of replaced words increases, the
LEICA scores on the paired images drop clearly, which demonstrates that our
proposed metric is sensitive to the semantic alignment between text and image.
Moreover, the accuracy of making a correct judgment, i.e., predicting a larger
score for the paired image with clean text than that with noised text,
increases steadily from $70\%$ to more than $90\%$. This indicates that LEICA
is able to perceive the mild changes in the semantic meaning of the text, and
can also correctly assess the alignment degree between the image-text pairs.

Table 1: Ablation studies on the components of LEICA. The first row of results
indicates the performance of LEICA. The last row is the performance of
averaged log-likelihood. “+” denotes combining the “funny mirror” distortion
with the current distortion. dVAE: distortion using the dVAE model of DALL-E.
Best results are highlighted with bold.

$H$ | $S$ | Acc | Kendall’s Tau ($\uparrow$)
---|---|---|---
GN | GB | SPN | GN+ | GB+ | SPN+ | dVAE
$\surd$ | $\surd$ | 1.00 | 0.86 | 0.66 | 0.73 | 0.70 | 0.60 | 0.72 | 0.44
| $\surd$ | 0.96 | 0.60 | 0.40 | 0.60 | 0.66 | 0.32 | 0.72 | 0.20
$\surd$ |  | 0.86 | 0.73 | 0.66 | 0.60 | 0.54 | 0.56 | 0.64 | 0.42
|  | 0.54 | 0.33 | -0.20 | 0.40 | 0.43 | 0.20 | 0.40 | -0.04

#### Performance Analysis

We analyze the effect of components of LEICA, i.e., the indicator function $H$
for perceptual significance and the scoring function $S$ for semantic
significance. Notably, when all these components are removed, LEICA is
annealed to the log version of Eq. (1). We calculate the accuracy of LEICA on
discriminating matched or mismatched text-image pairs with a $1:1$ ratio. We
set GN & GN+ with a variance of $0.2$, GB & GB+ with a standard deviation of
$8$, and SPN & SPN+ with a noise level of $0.4$.

As shown in Table 1, when discarding the perceptual significance indicator $H$
from LEICA, the results reduce clearly in all the meta-evaluation metrics,
e.g., the accuracy drops from $1.00$ to $0.96$ and the Kendall’s Tau
coefficient drops from $0.86$ to $0.60$ when adding Gaussian noise (GN). Note
that the performance drops more severely in the settings related to the image
disturbances, indicating that $H$ mainly focuses on the perceptual image
quality. Besides, when removing the patch-level scoring function $S$, the
performance of LEICA drops dramatically in terms of accuracy ($1.00$ to
$0.86$), demonstrating its importance in assessing the semantic text-image
alignment. Lastly, we show the performance of a naive likelihood-based metric,
i.e., the averaged log-likelihood, in the last row of the table. We find that
even with such a naive strategy, it can also achieve the positive Kendall’s
Tau correlations with human judgment in most settings. Overall, the above
results have shown the effectiveness of the basic likelihood-based evaluation
paradigm and the proposed credit assignment strategies w.r.t. perceptual and
semantic significance.

Table 2: Pearson, Spearman and Kendall’s Tau correlations between different
evaluation metrics and human judgment. The highest correlation is bold.

Methods | Pearson ($\uparrow$) | Spearman ($\uparrow$) | K-Tau ($\uparrow$)
---|---|---|---
IS [23] | 0.7175 | 0.4636 | 0.3091
FID [9] | 0.5932 | 0.2363 | 0.1636
KID [2] | 0.3039 | 0.2181 | 0.2000
SOA-C [10] | 0.8449 | 0.7363 | 0.5636
SOA-I [10] | 0.8301 | 0.7909 | 0.6000
CLIP-FID [11] | 0.6419 | 0.5636 | 0.4545
LEICA w/o $H$ | 0.8640 | 0.8727 | 0.7454
LEICA w/o $S$ | 0.8918 | 0.8636 | 0.7091
LEICA w/o $e^{\psi/\tau}$ | 0.9101 | 0.9091 | 0.7818
LEICA (base) | 0.8979 | 0.9181 | 0.7818
LEICA (large) | 0.9365 | 0.9181 | 0.7818

Figure 5: The ranks of each generative model derived by human raters and the
proposed evaluation metric. Rank-1 means the best model while rank-11 is the
worst one.

###  4.2 Results on Model Generated Data

We investigate the effectiveness of the proposed metric on generated images.
We consider 11 well-known generative models of various types, including
Generative Adversarial Networks, Autoregressive models and Diffusion models:
CogView [5], CogView2 [6], DALL-E [17], DALL-E2 [16], GLIDE [13], Imagen [22],
LAFITE [37], OP-GAN [10], RAT-GAN [32], Stable Diffusion [21], VQ-Diffusion
[8]. Each model synthesizes $5,000$ images, the same as the size of the
validation set of MSCOCO.

(a) IS (b) FID

(c) SOA-C (d) Ours

Figure 6: Impacts of the number of evaluated samples. We test three commonly
used evaluation metrics (i.e., IS, FID and SOA-C) as well as the proposed
LEICA for the generated images derived from four different generative models.

#### Alignment with Human Judgment

We conduct a human study to test the alignment between our evaluation metric
and human judgment. Specifically, we randomly choose $100$ texts from the
Karpathy’s validation split, and obtain corresponding generated images from
all $11$ models, so that each testing sample for our human study contains one
text and $11$ images. We use a Mean Opinion Score (MOS) evaluation approach to
score the test samples, where the score ranges from 1 to 10 in 1-point
increments. For each testing sample, we ask the raters to score the generated
image according to the perceptual image quality and semantic text-image
alignment, and emphasize that these two aspects should be equally considered.
Finally, we collect $8,479$ scores for all the $1,100$ testing samples, where
each sample has $7.7$ human judgments on average.

After we obtain the human study results, we conduct a meta-evaluation that
compares our metric with six commonly used evaluation metrics in the field of
text-to-image generation, i.e., Inception Score (IS) [23], Fréchet Inception
Distance (FID) [9], CLIP-FID [11], Kernel Inception Distance (KID) [2] and
Semantic Object Accuracy (SOA) [10] for class average (SOA-C) and image
average (SOA-I). For each metric, we compute the Pearson, Spearman, and
Kendall’s Tau correlation coefficient between the model rankings under its
judgment and the human judgment.

As shown in Table 2, our evaluation metric consistently outperforms the
baseline metrics on all meta-evaluations by a large margin. Besides, we also
investigate the impact of each component in LEICA as in Section 4.1. From this
table, we can draw a similar conclusion as in Table 1, where discarding any of
these components leads to a decrease in performance. Nevertheless, all the
ablation results of our method are still superior to the compared metrics in
terms of the correlation with human judgment. Besides, in Table 2, we also
show the impact of the term $e^{\psi/\tau}$ in Eq. (6): remove it decreases
the performance.

Moreover, in Figure 5, we plot the model ranking derived from human raters and
the ranking based on the proposed LEICA. We observe that the two rankings
yield very similar trends, which means our metric is well aligned with human
judgment and would not bias to any specific model. Lastly, comparing the
performance of LEICA (OFA-base) and LEICA (OFA-large), in Table 2, we find
that a better likelihood estimator will improve the reliability of LEICA,
suggesting its scalability to future generative foundation models.

Table 3: Pearson, Spearman and Kendall’s Tau correlations between different
evaluation metrics and human judgment on CUB & Oxford-Flower. The highest
correlation is bold.

Methods | Pearson ($\uparrow$) | Spearman ($\uparrow$) | K-Tau ($\uparrow$)
---|---|---|---
CUB
IS [23] | 0.2636 | 0.3 | 0.2
FID [9] | 0.7033 | 0.7 | 0.6
KID [2] | 0.6806 | 0.6 | 0.4
LEICA (Ours) | 0.8189 | 0.7 | 0.6
Oxford-Flower
IS [23] | 0.3166 | 0.2 | 0.2
FID [9] | 0.4719 | 0.5 | 0.4
KID [2] | -0.3113 | -0.1 | 0.0
LEICA (Ours) | 0.4880 | 0.6 | 0.4

#### Impacts of Number of Evaluated Samples

In this part, we investigate how the number of evaluation samples influences
the evaluation results for our LEICA metric and three baseline metrics,
including IS, FID and SOA-C. We calculate the scores of these metrics using
different numbers of generated samples ranging from $100$ to $5,000$ for four
different generative models. In Figure 6, the values of IS and FID are
severely affected by the number of samples. SOA-C performs more stable than IS
and FID, but still suffers from fluctuation when the number of samples is
changed. Therefore, these metrics have to use a very large number of samples
to make their evaluation results stable. In contrast, the scores from the
proposed LEICA are very stable, which means that our metric is able to
evaluate the performance of the generative models using only a few (one
hundred) generated samples, making it very efficient in practice.

#### Evaluating Samples from the Same Model & Text

Besides accessing different generative models, we further investigate LEICA in
another useful scenario, i.e., assessing different samples generated from the
same model using the same text. Specifically, we randomly choose $20$ texts
from COCO and generate $5$ images for each text with Stable Diffusion (SD) and
DALL-E (DE), respectively. Two raters are asked to score these images, and the
Pearson correlations between their judgments are $0.203$ (SD) and $0.298$
(DE), showing that this is a very challenging setting even for humans.
Nevertheless, LEICA still achieves positive correlations with these raters
(Pearson: $0.105$ for SD and $0.257$ for DE), indicating that our proposed
metric has the ability to evaluate individual samples in this setting.

Figure 7: The semantic significance maps obtained from the original CLIP and
our modified CLIP on generated images.

#### Evaluation Performance on Specific Domains

In addition to COCO, in Table 3, we also provide results on the CUB dataset,
where LEICA achieves a 0.819 Pearson correlation with the human judgment,
clearly better than IS (0.264), FID (0.703) and KID (0.681). SOA is not
compared as it is designed only for COCO. We further evaluate LEICA on Oxford-
Flower using the same setting, and it still yields the best performance (e.g.,
Pearson: 0.488), while IS, FID and KID achieve 0.317, 0.472, and -0.311,
respectively. This demonstrates the generalization ability of LEICA,
especially in class-specific domains.

#### Visualization Results

We visualize the patch-level alignment performance of the original CLIP and
our modified model. Figure 7 shows that our modified CLIP is able to capture
the patches semantically aligned with the given texts in a well-synthesized
image (row 1), while suppressing all the regions for a low-quality image (row
2). However, the original CLIP fails to capture the semantically significant
regions of images described in the given texts. These results verify the
effectiveness of our modified CLIP.

##  5 Conclusion

In this paper, we propose a likelihood-based metric for text-to-image
evaluation based on the intuition that a good likelihood estimator will
reflect how likely a generated image is to exist in the real world given the
text. Besides, we propose two credit assignment strategies w.r.t. perceptual
significance and semantic significance. The former eliminates the effect of
the perceptually-insignificant image regions, which may not affect the overall
perceptual quality of a generated image but severely drag down its overall
likelihood. The latter down-weights the effect of the background regions with
low semantic significance, which are usually not related to the text but
account for the largest portion of the image. We evaluate our metric on 11
models and 3 datasets, showing that it is effective, efficient, and stable.

## References

  * [1] https://github.com/kaustubh-sadekar/FunMirrors.
  * [2] Mikołaj Bińkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton.  Demystifying mmd gans.  ICLR, 2018.
  * [3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In ICML, pages 1597–1607, 2020.
  * [4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, pages 248–255, 2009.
  * [5] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al.  Cogview: Mastering text-to-image generation via transformers.  NeurIPS, pages 19822–19835, 2021.
  * [6] Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang.  Cogview2: Faster and better text-to-image generation via hierarchical transformers.  NeurIPS, 2022.
  * [7] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In CVPR, pages 12873–12883, 2021.
  * [8] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo.  Vector quantized diffusion model for text-to-image synthesis.  In CVPR, pages 10696–10706, 2022.
  * [9] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  NeurIPS, 2017.
  * [10] Tobias Hinz, Stefan Heinrich, and Stefan Wermter.  Semantic object accuracy for generative text-to-image synthesis.  IEEE TPAMI, 2020.
  * [11] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen.  The role of imagenet classes in fr$\backslash$’echet inception distance.  ICLR, 2023.
  * [12] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [13] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  ICML, 2022.
  * [14] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729. IEEE, 2008.
  * [15] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, pages 8748–8763, 2021.
  * [16] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [17] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In ICML, pages 8821–8831, 2021.
  * [18] Suman Ravuri and Oriol Vinyals.  Classification accuracy score for conditional generative models.  NeurIPS, 32, 2019.
  * [19] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [20] Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan.  Likelihood ratios for out-of-distribution detection.  Advances in neural information processing systems, 32, 2019.
  * [21] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, pages 10684–10695, 2022.
  * [22] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  NeurIPS, 2022.
  * [23] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  NeurIPS, 2016.
  * [24] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In CVPR, pages 2818–2826, 2016.
  * [25] Lucas Theis, Aäron van den Oord, and Matthias Bethge.  A note on the evaluation of generative models.  ICLR, 2016.
  * [26] Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu.  Pixel recurrent neural networks.  In ICML, pages 1747–1756, 2016.
  * [27] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  NeurIPS, 2017.
  * [28] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  NeurIPS, 30, 2017.
  * [29] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * [30] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  In ICML, pages 23318–23340, 2022.
  * [31] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In CVPR, pages 1316–1324, 2018.
  * [32] Senmao Ye, Fei Liu, and Minkui Tan.  Recurrent affine transformation for text-to-image synthesis.  arXiv preprint arXiv:2204.10482, 2022.
  * [33] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  arXiv preprint arXiv:2206.10789, 2022.
  * [34] Weizhe Yuan, Graham Neubig, and Pengfei Liu.  Bartscore: Evaluating generated text as text generation.  NeurIPS, pages 27263–27277, 2021.
  * [35] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang.  Cross-modal contrastive learning for text-to-image generation.  In CVPR, pages 833–842, 2021.
  * [36] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas.  Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks.  In ICCV, pages 5907–5915, 2017.
  * [37] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun.  Towards language-free training for text-to-image generation.  In CVPR, pages 17907–17917, 2022.
  * [38] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang.  Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis.  In CVPR, pages 5802–5810, 2019.


# Role of IR-Improvement in Precision LHC/FCC Physics and in Quantum Gravity

Baylor Un iversity, Waco, TX, USA
E-mail  S. Jadach
Institute of Nuclear Physics, Krakow, Poland
E-mail<EMAIL_ADDRESS>W. Placzek
Institute of Physics, Jagiellonian University, Krakow, Poland
E-mail<EMAIL_ADDRESS>M. Skrzypek
Institute of Nuclear Physics, Krakow, Poland
E-mail<EMAIL_ADDRESS>Z. A. Was
Institute of Nuclear Physics, Krakow, Poland
E-mail<EMAIL_ADDRESS>S.A. Yost
The Citadel, Charleston, SC, USA
E-mail<EMAIL_ADDRESS>
###### Abstract:

IR-improvement based on amplitude-level resummation allows one to control
unintegrable results in quantum field theory with arbitrary precision in
principle. We illustrate such resummation in specific examples in precision
LHC and FCC physics and in quantum gravity.

BU-HEPP-19-08, Dec. 2019

##  1 Introduction

With the era of precision LHC physics squarely upon us, the relevance of
resummation in quantum field theory is large. This relevance is amplified by
the need to plan for future colliders in view of the current lack of
experimental evidence for new degrees of freedom beyond those in the Standard
Theory111We follow Prof. D.J. Gross [1] and henceforth refer to the Standard
Model as the Standard Theory of elementary particles. in the LHC data at this
writing. Indeed, several proposals for the next big colliding beam device
involve a precision physics $e^{+}e^{-}$ colliding beams phase in the regime
of $\sqrt{s}$ from $91$ GeV to 390 GeV as a first step: FCC, ILC, CEPC, and
CLIC. We mention that, in the problem of the union of quantum mechanics with
gravity as given by the Einstein-Hilbert theory, precision theory methods seem
also to be relevant. In what follows, we discuss the role of IR-improvement in
the particular approach amplitude-based resummation in quantum field theory as
it is applied to the aforementioned precision physics considerations.

In discussing the role of IR-improvement in quantum field theory in precision
physics considerations, we are called to address the general idea of
resummation in quantum field theory for IR, UV or CL (collinear) effects. The
matter can be put into focus by comparing with the simple example of summation
for a geometric series, such as

$\frac{1}{1-x}=\sum_{n=0}^{\infty}x^{n}.$ (1)

Here the result of the summation is an analytic result that is well defined
except for a simple pole at $x=1$. While the mathematical tests for
convergence on the simple series would only guarantee convergence for $|x|<1$,
the result of the summation yields a result that is well-behaved in the entire
complex plane except for a simple pole at $x=1$. The message is that infinite
order summation can yield results that are much better behaved than what one
can deduce order-by-order in the respective series. One is accordingly
invited, as a next step, to resum series that are already being summed to try
to improve ones knowledge of the represented function.

Continuing in this way, we write for the Feynman series for a process the
following:

$\sum_{n=0}^{\infty}C_{n}\alpha_{s}^{n}\begin{cases}&=F_{\rm
RES}(\alpha_{s})\sum_{n=0}^{\infty}B_{n}\alpha_{s}^{n},\;\text{\rm EXACT}\\\
&\cong G_{\rm
RES}(\alpha_{s})\sum_{n=0}^{\infty}\tilde{B}_{n}\alpha_{s}^{n},\;\text{\rm
APPROX.}\end{cases}$ (2)

Two versions of resumming the original Feynman series for the quantity on the
LHS are given on the RHS. One is an exact re-arrangement of the original
series. It is labeled ‘exact’.The other only agrees with the LHS to some fixed
order N in the expansion parameter $\alpha_{s}$. It is labeled ‘approx’. Which
version of resummation is to be preferred has been discussed for some time
[2]. At this writing, the discussion continues to new paradigms.

In QCD$\otimes$ QED the discussion continues with added dimensions. We have
the ongoing comparison, by two of us (BFLW and SAY), of the hard IR cut-offs
in MC’s such as Herwig [3], Herwig++ [4], Pythia [5] and Sherpa [6] versus the
resummed integrability of the IR in Herwiri1.031 [7]. In addition, we have the
approaches to ISR radiation from quarks in hard hadron - hadron scattering
processes, wherein we have the approach with QED PDF’s [8] versus exact
Feynman diagrams with short-distance quark masses as pursued by four of us
(SJ, BFLW, ZAW, and SAY) in Ref. [9]. In the FCC and other future collider
frameworks, the attendant precision physics programs necessitate IR-improved
quantum field theory results to exploit properly the respective new physics
windows. In quantum gravity, it has been found that resummed IR leads to UV
finiteness [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25].
In what follows, we expand on these developments in IR-improvement of quantum
field theory from the standpoint of applications to precision LHC/FCC physics
and applications of one of us (BFLW) of quantum gravity.

We close this introductory section by calling attention to the 50-year
celebrations of the Standard Model of Elementary Particles [26, 27, 28, 29,
30, 31, 32, 33]. At the same time, plans are underway for the future
explorations in our field as we have noted. We call attention here to the FCC
Study [34] which we show in Fig. 1.

Figure 1: Tentative schedule and physical layout in FCC Study as portrayed in
Ref. [34].

The plans call for a precision studies $e^{+}e^{-}$ phase featuring more than
5 Tera Z’s, for example, followed by a 100 TeV hadron collider, reminiscent of
the canceled SSC in Waxahachie, TX, USA. Only now do we really see how big of
a mistake the latter cancellation was. It is important to keep this historical
perspective in mind.

The discussion proceeds as follows. In the next section, we review briefly
exact amplitude-based resummation theory. This is followed by a discussion of
applications in precision LHC physics in Section 3. Applications in Precision
FCC physics are discussed in Section 4. Section 5 present applications in
quantum gravity. Section 6 contains our summary remarks.

##  2 Review of Exact Amplitude-Based Resummation Theory

This approach to higher order corrections is founded on the master formula

$\displaystyle d\bar{\sigma}_{\rm res}=e^{\rm
SUM_{IR}(QCED)}\sum_{{n,m}=0}^{\infty}\frac{1}{n!m!}\int\prod_{j_{1}=1}^{n}\frac{d^{3}k_{j_{1}}}{k_{j_{1}}}$
(3)
$\displaystyle\prod_{j_{2}=1}^{m}\frac{d^{3}{k^{\prime}}_{j_{2}}}{{k^{\prime}}_{j_{2}}}\int\frac{d^{4}y}{(2\pi)^{4}}e^{iy\cdot(p_{1}+q_{1}-p_{2}-q_{2}-\sum
k_{j_{1}}-\sum{k^{\prime}}_{j_{2}})+D_{\rm QCED}}$ (4)
$\displaystyle{\tilde{\bar{\beta}}_{n,m}(k_{1},\ldots,k_{n};k^{\prime}_{1},\ldots,k^{\prime}_{m})}\frac{d^{3}p_{2}}{p_{2}^{\,0}}\frac{d^{3}q_{2}}{q_{2}^{\,0}},$
(5)

where new (YFS-style) non-Abelian residuals
$\tilde{\bar{\beta}}_{n,m}(k_{1},\ldots,k_{n};k^{\prime}_{1},\ldots,k^{\prime}_{m})$
have $n$ hard gluons and $m$ hard photons. The infrared functions ${\rm
SUM_{IR}(QCED)}$ and ${D_{\rm QCED}}$ and the residuals are defined in Ref.
[35, 36]. In the implementation of parton shower/ME matching, we have the
replacements
$\tilde{\bar{\beta}}_{n,m}\rightarrow\hat{\tilde{\bar{\beta}}}_{n,m}$. These
replacements allow us, via the basic formula

${d\sigma}=\sum_{i,j}\int
dx_{1}dx_{2}{F_{i}(x_{1})F_{j}(x_{2})}d\hat{\sigma}_{\rm res}(x_{1}x_{2}s),$
(6)

to connect with MC@NLO [37, 38] as explained in Ref. [35, 36].

We have made various applications of Eq.(5) to precision LHC and FCC physics
and we have made applications of its extension to general relativity as an
approach to quantum gravity. We discuss such applications in precision LHC
physics in the next Section.

##  3 Applications in Precision LHC Physics

In Refs. [35, 36, 39], two of us (BFLW and SAY) have interfaced the
realization of IR-improved DGLAP-CS [40, 41, 42, 43, 44, 45, 46, 47, 48]
theory as realized in Herwiri1.031 [7] with MC@NLO and with MG5_aMC@NLO [49]
for precision studies of Z and W+jets production at the LHC. In Refs. [9, 50,
51] four of us (SJ, BFLW, ZAW, and SAY) have realized exact ${\cal
O}(\alpha^{2}L)$ CEEX EW corrections for hadron-hadron scattering processes in
the MC ${\cal KK}$MC-hh which features built-in parton showers from Herwig6.5
and Herwiri1.031 as well as an LHE [52] format to facilitate the interface
with other parton shower MC’s. In Z and W+ jets productions, IR-improvement
gives a comparable or better data fit without ad hoc parameters for the
intrinsic transverse momentum spectrum of partons inside the proton. In ${\cal
KK}$MC-hh, IR-improvement allows to quantify role of ISR in precision
predictions for Z production observables, as we now illustrate.

Specifically, our implementation for DIZET [53, 54] in ${\cal KK}$MC-hh uses a
modified $G_{\mu}$ scheme with Standard Theory222We follow D.J. Gross [1] and
henceforth refer to the Standard Model [26, 27, 28, 29, 30, 31] as the
Standard Theory. input parameters as given in Ref. [55] with the exception
that we use the light quark masses $m_{u}=6.0$ MeV,$\;m_{d}=10.0$ MeV
($mu=2.2$ MeV,$\;m_{d}=4.7$MeV) where the latter set is used to estimate the
uncertainty of our results due to the uncertainty in the precise values of the
respective short-distance quark masses. For a systematic discussion of the
effects of ISR and IFI as predicted by ${\cal KK}$MC-hh on the angular
observables in Z production and decay to lepton pairs at the LHC see Ref.
[56]. Here, we focus on the use of the Z production observables to quantify
the systematics in the world-leading ATLAS measurement [57] of $M_{W}$:

$M_{W}=80370\pm 7(\text{stat.})\pm 11(\text{exp. syst.})\pm 14(\text{mod.
syst.})\text{MeV}\\\ =80370\pm 19\text{MeV},$

where the first uncertainty is statistical, the second is the experimental
systematic uncertainty, and the third is the physics-modeling systematic
uncertainty. Z/$\gamma^{*}$ production with decay to lepton pairs are used in
determining the latter uncertainty. Specifically, the distributions for the
lepton $p_{T}$ spectra in single Z/$\gamma^{*}$ in Ref. [57], as reproduced
here in Fig. 2, are used to analyze expectations for the corresponding lepton
spectra in the single W production.

Figure 2: ATLAS electron and muon $p_{T}$ spectra from Fig. 15 in Ref. [57].

In both spectra featured in Fig. 2, we see that, at the lower end, the theory
used by ATLAS is about 2% below the data and, at the high end, it is a similar
amount above the data. We address these spectra in Ref. [50] as shown here in
Fig. 3.

Figure 3: Muon transverse momentum distributions and their ratios for ${\cal
KK}$MC-hh with the cuts specified in the text for the EW-CORR (electroweak-
correction) labels “CEEX2” (red – medium dark shade), “CEEX2 (no IFI)” (violet
– light dark shade), “EEX1” (blue – dark shade), and “EEX1 (no IFI)” (green –
light shade), showered by HERWIG 6.5. The labels are explained in the text.
The ratio plot features “CEEX2” as the reference distribution as noted in the
respective title.

In Fig. 3, the best prediction of the exact ${\cal O}(\alpha^{2}L)$ CEEX
calculation (labeled CEEX2) is compared with less precise predictions denoted
as follows: ${\cal O}(\alpha^{2}L)$ CEEX with ISR+FSR+IFI – labeled as
“CEEX2”; ${\cal O}(\alpha^{2}L)$ CEEX without IFI (initial state final state
interference) – labeled “ CEEX2 (no IFI)”;${\cal O}(\alpha)$ EEX – labeled
“EEX1”; ${\cal O}(\alpha)$ EEX without ISR (initial state radiation) – labeled
“EEX1 (no ISR)”. Here, EEX denotes YFS exponentiation at the level of squared
amplitudes as originally formulated in Ref. [58] in contrast to CEEX, which
represents the coherent realization of YFS exponentiation at the level of the
amplitudes, as explained in Ref. [59]. We see that the EEX1 result with no ISR
is about 2% below the best calculation on the low end of the spectrum and is a
similar amount above the best calculation on the high end, just as we see in
the ATLAS data, where the theory prediction also has no QED ISR other than
that in its Pythia [60] shower, which is essentially collinear by
construction. We conclude that the corrections in ${\cal KK}$MC-hh should be
implemented into the ATLAS analysis and other similar analyses.

##  4 Applications in Precision FCC Physics

The FCC project involves a 100 km tunnel in which one first intalls an
$e^{+}e^{-}$ collider that operates for precision Z, WW, Higgs and top
physics,to be followed by the installation of a 100 TeV proton-proton
collider. The first phase is FCC-ee, the second phase is FCC-hh. As three of
us (BFLW, ZAW and SAY) have argued in Ref. [51], IR-improvement of even the
FCC-hh discovery spectra is needed. In the FCC-ee, more than 5 Tera Z’s will
be observed333For this scenario, recently, thanks to the modular program
design improvements for the electroweak sector calculations became available
in ${\cal KK}$MC [59]. The comparison of numerical results is assured. This is
of importance for future long-term projects as discussed in [61]. In
particular the up-to-date parametrization of $\alpha_{QED,eff}$[62] (section
B.1) is available as well as the electroweak library DIZET version 6.42 [63]
and the recent version 6.45. The same is true and is of importance for
precision measurements of electroweak effects with $\tau$ lepton polarization
[64] where benefits from future Belle experimental results on $\tau$ lepton
decays are of a sizable potential. In this context technical tests embedded in
the older version of TAUOLA for the $\tau$ lepton decay [65] may need to be
revisited to assure technical correctness at better than the precision
standard of today., so that a key issue is the theoretical precision of the
luminosity as discussed by five of us (SJ, WP, MS, BFLW, SAY) in Ref. [66], a
discussion to which we now turn.

To set the perspective, we show in Tab. 1 the state of the art at the time of
LEP, where the precision tag on the theoretical prediction of the luminosity
was 0.061%(0.054%),

| LEP1 | LEP2
---|---|---
Type of correction/error | 1996 | 1999 | 1996 | 1999
(a) Missing photonic ${\cal O}(\alpha^{2})$ [67, 68] | 0.10% | 0.027% | 0.20% | 0.04%
(b) Missing photonic ${\cal O}(\alpha^{3}L_{e}^{3})$ [69] | 0.015% | 0.015% | 0.03% | 0.03%
(c) Vacuum polarization [70, 71] | 0.04% | 0.04% | 0.10% | 0.10%
(d) Light pairs [72, 73] | 0.03% | 0.03% | 0.05% | 0.05%
(e) $Z$ and $s$-channel $\gamma$ [74, 75] | 0.015% | 0.015% | 0.0% | 0.0%
Total | 0.11% [75] | 0.061% [76] | 0.25% [75] | 0.12% [76]

Table 1: Summary of the total (physical+technical) theoretical uncertainty for
a typical calorimetric detector. For LEP1, the above estimate is valid for a
generic angular range within $1^{\circ}$–$3^{\circ}$ ($18$–$52$ mrads), and
for LEP2 energies up to $176$ GeV and an angular range within
$3^{\circ}$–$6^{\circ}$. Total uncertainty is taken in quadrature. Technical
precision included in (a).

according to whether one does not(does) treat the effects of soft pairs [77,
78]. See Ref. [76] for a discussion of the various error estimates as they
relate to the cited published works in the table. The current state of the
art, 0.038%, is shown in Table 2, where we note that recently this precision
tag has been effected

Type of correction / Error | 1999 | Update 2018
---|---|---
(a) Photonic ${\cal O}(L_{e}\alpha^{2})$ | 0.027% [68] | 0.027%
(b) Photonic ${\cal O}(L_{e}^{3}\alpha^{3})$ | 0.015% [69] | 0.015%
(c) Vacuum polariz. | 0.040% [70, 71] | 0.013% [79]
(d) Light pairs | 0.030% [73] | 0.010% [77, 78]
(e) $Z$ and $s$-channel $\gamma$ exchange | 0.015% [80, 81] | 0.015%
(f) Up-down interference | 0.0014% [82] | 0.0014%
(f) Technical Precision | – | (0.027)%
Total | 0.061% [76] | 0.038%

Table 2: Summary of the total (physical+technical) theoretical uncertainty for
a typical calorimetric LEP luminosity detector within the generic angular
range of $18$–$52$ mrad. Total error is summed in quadrature.

in Ref. [83] in reducing the error on the number, $N_{\nu}$, of light
neutrinos so that it is now $N_{\nu}=2.9975\pm 0.00742.$ The FCC-ee
requirement [84] on the theoretical precision of the luminosity is 0.01%. The
steps involved in realizing this requirement are discussed in Ref. [66]. The
conclusion of Ref. [66] is that, with sufficient resources to make the
required improvements to the MC BHLUMI [85], the 0.01% precision tag can be
realized, as we show in Table 3. Again, we cite the published works associated
to the error estimates in the table as discussed in Ref. [66].

Type of correction / Error | Update 2018 | FCC-ee forecast
---|---|---
(a) Photonic $[{\cal O}(L_{e}\alpha^{2})]\;{\cal O}(L_{e}^{2}\alpha^{3})$ | 0.027% | $0.1\times 10^{-4}$
(b) Photonic $[{\cal O}(L_{e}^{3}\alpha^{3})]\;{\cal O}(L_{e}^{4}\alpha^{4})$ | 0.015% | $0.6\times 10^{-5}$
(c) Vacuum polariz. | 0.014% [79] | $0.6\times 10^{-4}$
(d) Light pairs | 0.010% [77, 78] | $0.5\times 10^{-4}$
(e) $Z$ and $s$-channel $\gamma$ exchange | 0.090% [74] | $0.1\times 10^{-4}$
(f) Up-down interference | 0.009% [82] | $0.1\times 10^{-4}$
(f) Technical Precision | (0.027)% | $0.1\times 10^{-4}$
Total | 0.097% | $1.0\times 10^{-4}$

Table 3: Anticipated total (physical+technical) theoretical uncertainty for a
FCC-ee luminosity calorimetric detector with the angular range being
$64$–$86\,$mrad (narrow), near the $Z$ peak. Description of photonic
corrections in square brackets is related to the 2nd column. The total error
is summed in quadrature.

The current planning for the FCC [84, 86] includes a provision for such
resources as those called for in Ref. [66].

##  5 Applications in Quantum Gravity

We turn now to the role of IR-improvement in quantum gravity. The question of
whither or not quantum gravity is calculable in relativistic quantum field
theory is still open in the literature, as one of us (BFLW) has discussed in
Refs. [87, 88]. Here, one of us argues that it is if he extends the YFS [58,
89, 59] version444YFS-type soft resummation and its extension to quantum
gravity was also worked-out by Weinberg in Ref. [90]. of the exact resummation
example to resum the Feynman series for the Einstein-Hilbert Lagrangian for
quantum gravity. The resultant resummed theory, resummed quantum gravity
(RQG), in analogy with what we see in the example in Eq.(1), is very much
better behaved in the UV compared to what one would estimate from that Feynman
series.

One of us (BFLW) has shown [22, 23, 24, 25] that the RQG realization of
quantum gravity leads to Weinberg’s [91] UV-fixed-point behavior for the
dimensionless gravitational and cosmological constants and that the resummed
theory is actually UV finite. He has shown further that the RQG theory, taken
together with the Planck scale inflationary [92, 93] cosmology formulation in
Refs. [94, 95]555The authors in Ref. [96] also proposed the attendant choice
of the scale $k\sim 1/t$ used in Refs. [94, 95]. from the asymptotic safety
approach to quantum gravity in Refs. [10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
20, 21], allows us to predict the cosmological constant $\Lambda$.
Specifically, he obtains, employing the arguments in Refs. [97] ($t_{eq}$ is
the time of matter-radiation equality), the result [98]

$\begin{split}\rho_{\Lambda}(t_{0})&\cong\frac{-M_{Pl}^{4}(1+c_{2,eff}k_{tr}^{2}/(360\pi
M_{Pl}^{2}))^{2}}{64}\sum_{j}\frac{(-1)^{F}n_{j}}{\rho_{j}^{2}}\times\frac{t_{tr}^{2}}{t_{eq}^{2}}\times(\frac{t_{eq}^{2/3}}{t_{0}^{2/3}})^{3}\cr&\cong\frac{-M_{Pl}^{2}(1.0362)^{2}(-9.194\times
10^{-3})}{64}\frac{(25)^{2}}{t_{0}^{2}}\cong(2.4\times
10^{-3}eV)^{4}.\cr\end{split}$ (7)

$t_{0}\cong 13.7\times 10^{9}$ yrs is the age of the universe, $t_{tr}\sim
25t_{Pl}$ is the transition time between the Planck regime and the classical
Friedmann-Robertson-Walker(FRW) regime in the Planck scale cosmology
description of inflation in Ref. [95], $c_{2,eff}\cong 2.56\times 10^{4}$ is
defined in Refs. [22, 23, 24], $t_{eq}$ is the time of matter-radiation
equality, and $M_{Pl}$ is the Planck mass.

Due to the prediction’s closeness to the observed value [99, 100,
101],$\rho_{\Lambda}(t_{0})|_{\text{expt}}\cong((2.37\pm 0.05)\times
10^{-3}eV)^{4}$, one of us (BFLW) discusses in Ref. [102] its reliability and
argues that its uncertainty is at the level of a factor of ${\cal O}(10)$.
There follow constraints on susy GUT’s as well as discussed in Refs. [98].
Various consistency checks on our result for $\rho_{\Lambda}(t_{0})$ are also
considered in Ref. [98].

##  6 Summary

What we can see is that the resummation of the IR regime of quantum field
theory, coupled with exact results to a given order, when done at the level of
the amplitude in an exact re-arrangement of the original Feynman series, has
wide applicability for precision phenomenology. We have illustrated the range
of this applicability from the current precision LHC physics to the futuristic
FCC precision physics program and reliable estimates for quantum gravity by
one of us (BFLW). The future of particle physics is intimately interwoven with
continued progress on IR-improved quantum field theoretic predictions, when
taken together with control of the attendant collinear and UV limits.

Acknowledgments

This work was supported in part by the Programme of the French–Polish
Cooperation between IN2P3 and COPIN within the Collaborations Nos. 10-138 and
11-142 and by a grant from the Citadel Foundation. The authors also thank
Prof. G. Giudice for the support and kind hospitality of the CERN TH
Department.

## References

  * [1] D. J. Gross, in Proc. SM@50, Cambridge University Press, 2019, in press.
  * [2] F. A. Berends, private communication, 1988.
  * [3] G. Corcella, I. G. Knowles, G. Marchesini, S. Moretti, K. Odagiri, P. Richardson, M. H. Seymour, and B. R. Webber, JHEP 0101 (2001) 010, hep-ph/0011363.
  * [4] M. Bahr et al., Eur. Phys. J. C58 (2008) 639–707, 0803.0883.
  * [5] T. Sjostrand et al., Comput. Phys. Commun. 135 (2001) 238.
  * [6] T. Gleisberg et al., J. High Energy Phys. 02 (2009) 007, hep-ph/0811.4622.
  * [7] S. Joseph et al., Phys. Rev. D 81 (2010) 076008, hep-ph/1001.1434.
  * [8] A. Manohar et al., Phys. Rev. Lett. 117 (2016) 242002.
  * [9] S. Jadach, B. F. L. Ward, Z. Wa̧s, and S. Yost, Phys. Rev. D 94 (2016) 074006, hep-ph/1608.01260.
  * [10] M. Reuter, Phys. Rev. D 57 (1998) 971.
  * [11] O. Lauscher and M. Reuter, Phys. Rev. D 66 (2002) 025026.
  * [12] E. Manrique, M. Reuter, and Saueressig, Ann. Phys. 326 (2011) 44.
  * [13] A. Bonanno and M. Reuter, Phys. Rev. D 62 (2000) 043008.
  * [14] D. F. Litim, Phys. Rev. Lett. 92 (2004) 201301.
  * [15] D. F. Litim, Phys. Rev. D 64 (2001) 105007.
  * [16] P. Fischer and D. F. Litim, Phys. Lett. B 638 (2006) 497.
  * [17] D. Don and R. Percacci, Class. Quant. Grav. 15 (1998) 3449, Class. Quant. Grav. 15 (1998) 3449.
  * [18] R. Percacci and D. Perini, Phys. Rev. D 67 (2003) 081503.
  * [19] R. Percacci, Phys. Rev. D 73 (2006) 041501.
  * [20] R. Percacci and D. Perini, Phys. Rev. D 68 (2003) 044018.
  * [21] A. Codello and R. Percacci and C. Rahmede, Int.J. Mod. Phys. A 23 (2008) 143.
  * [22] B.F.L. Ward, Open Nucl.Part.Phys.Jour. 2 (2009) 1.
  * [23] B.F.L. Ward, Mod. Phys. Lett. A 17 (2002) 237.
  * [24] B.F.L. Ward, Mod. Phys. Lett. 19 (2004) 143.
  * [25] B.F.L. Ward, Mod. Phys. Lett. A 23 (2008) 3299.
  * [26] S. L. Glashow, Nucl. Phys. 22 (1961) 579–588.
  * [27] A. Salam, Elementary Particle Theory.  N. Svartholm (Almqvist and Wiksell), Stockholm, 1968.
  * [28] S. Weinberg, Phys. Rev. Lett. 19 (1967) 1264–1266.
  * [29] G. ’t Hooft and M. Veltman, Nucl. Phys. B 44 (1972) 189.
  * [30] D. ’Gross and F. Wilczek, Phys. Rev. Lett. 30 (1973) 1343.
  * [31] H. D. ’Politzer, Phys. Rev. Lett. 30 (1973) 1346.
  * [32] B. W. Lynn et al., Proc. SM@50, Cambridge University Press, 2019, in press.
  * [33] T. Rizzo et al., Proc. 46th SLAC Summer Institute, C18-07-30, 2018\.
  * [34] F. Gianotti, in New Year presentation, CERN, Geneva, Switzerland, Jan. 15, 2019.
  * [35] S.K. Majhi et al., Phys. Lett. B 719 (2013) 367, hep-ph/1208.4750.
  * [36] A. Mukhopadhyay and B.F.L. Ward, Mod. Phys. Lett. A 31 (2016) 1650063, hep-ph/1412.8717.
  * [37] S. Frixione and B.Webber, J. High Energy Phys. 0206 (2002) 029.
  * [38] S. Frixione et al., J. High Energy Phys. 1101 (2011) 053, hep-ph/1010.0568.
  * [39] B. Shakerin and B.F.L. Ward, Phys. Rev. D 100 (2019) 034026, hep-ph/1809.01492.
  * [40] G. Altarelli and G. Parisi, Nucl. Phys. B 126 (1977) 298.
  * [41] Y. L. Dokshitzer, Sov. Phys. JETP 46 (1977) 641.
  * [42] L. Lipatov, Yad. Fiz. 20 (1974) 181.
  * [43] V. Gribov and L. Lipatov, Sov. J. Nucl. Phys. 15 (1972) 675, 938.
  * [44] J. Collins and J. Qiu, Phys. Rev. D 39 (1989) 1398.
  * [45] C. G. Callan Jr., Phys. Rev. D 2 (1970) 1541.
  * [46] K. Symanzik, Commun. Math. Phys. 18 (1970) 227.
  * [47] K. Symanzik, Springer Tracts. Mod. Phys. 57 (1971) 222.
  * [48] S. Weinberg, Phys. Rev. D 8 (1973) 3497.
  * [49] J. Alwall et al., J. High Energy Phys. 07 (2014) 079, hep-ph/1405.0301.
  * [50] S. Jadach, B. F. L. Ward, Z. Wa̧s, and S. Yost, Phys. Rev. D 99 (2019) 076016, hep-ph/1707.06502.
  * [51] B.F.L. Ward et al., PoS RADCOR2017 (2018) 083\.
  * [52] E. Boos et al., hep-ph/0109068.
  * [53] D. Bardin et al., EW Library.
  * [54] D. Bardin et al., e-print: hep-ph/9908433.
  * [55] S. Alioli et al., CERN-TH-2016-137; CERN-LPCC-2016-002 (2016) hep-ph/1606.02330.
  * [56] S. A. Yost et al., in Proc. RADCOR2019, PoS, 2019, in press.
  * [57] ATLAS Collaboration, M. Aaboud et al., Eur. Phys. J. C 78 (2018) 110, hep-ex/1701.07240.
  * [58] D. R. Yennie, S. Frautschi, and H. Suura, Ann. Phys. (NY) 13 (1961) 379 – 452.
  * [59] S. Jadach, B. F. L. Ward, and Z. Wa̧s, Phys. Rev. D 63 (2001) 113009, preprint UTHEP-99-09-01.
  * [60] T. Sjostrand, S. Mrenna, and P. Skands, Comput. Phys. Commun. 178 (2008) 852–867, 0710.3820.
  * [61] S. Banerjee, M. Chrzaszcz, Z. Was, and J. Zaremba, “Heritage projects, archivization and re-usability concerns”, in Theory report on the 11th FCC-ee workshop, pp. 137–140, 2019\.
  * [62] Theory report on the 11th FCC-ee workshop, 2019\.
  * [63] A. B. Arbuzov, M. Awramik, M. Czakon, A. Freitas, M. W. Grunewald, K. Monig, S. Riemann, and T. Riemann, Comput. Phys. Commun. 174 (2006) 728–758, hep-ph/0507146.
  * [64] S. Banerjee and Z. Was, “FCC Tau Polarization”, in Theory report on the 11th FCC-ee workshop, pp. 215–216, 2019\.
  * [65] S. Jadach, Z. Wa̧s, R. Decker, and J. H. Kühn, Comput. Phys. Commun. 76 (1993) 361\.
  * [66] S. Jadach, W. Placzek, M. Skrzypek, B. F. L. Ward, and S. A. Yost, Phys. Lett. B 790 (2018) 314.
  * [67] S. Jadach, M. M. Melles, B. F. L. Ward, and S. A. Yost, Phys. Lett. B377 (1996) 168–176.
  * [68] S. Jadach, M. M. Melles, B. F. L. Ward, and S. A. Yost, Acta Phys. Polon. B30 (1999) 1745 – 1750.
  * [69] S. Jadach and B. F. L. Ward, Phys. Lett. B389 (1996) 129–136.
  * [70] H. Burkhardt and B. Pietrzyk, Phys. Lett. B 356 (1995) 398 – 403.
  * [71] S. Eidelman and F. F. Jegerlehner, Z. Phys. C 67 (1995) 585 – 602.
  * [72] S. Jadach, M. Skrzypek, and B. Ward, Phys. Rev. D 47 (1993) 3733 – 3741.
  * [73] S. Jadach, M. Skrzypek, and B. Ward, Phys. Rev. D 55 (1997) 1206 – 1215.
  * [74] S. Jadach, W. Placzek, and B. Ward, Phys. Lett. B 353 (1995) 349 – 361.
  * [75] A. Arbuzov et al., Phys. Lett. B 383 (1996) 238 – 242, hep-ph/9605239.
  * [76] S. Jadach, M. M. Melles, B. F. L. Ward, and S. A. Yost, Phys. Lett. B450 (1999) 262–266.
  * [77] G. Montagna et al., Nucl. Phys. B 547 (1999) 39 – 59, hep-ph/9811436.
  * [78] G. Montagna et al., Phys. Lett. B459 (1999) 649 – 652, hep-ph/9905235.
  * [79] F. Jegerlehner, talk in FCC-ee Mini-Workshop, Physics Behind Precision, https://indico.cern.ch/event/469561/contributions/1977974/attachments/1221704/1786449/SMalphaFCCee16.pdf.
  * [80] S. Jadach, W. Placzek, and B. F. L. Ward, Phys. Lett. B353 (1995) 349–361.
  * [81] A. Arbuzov et al., Phys. Lett. B383 (1996) 238–242, hep-ph/9605239.
  * [82] S. Jadach, E. E. Richter-Wa̧s, B. F. L. Ward, and Z. Wa̧s, Phys. Lett. B253 (1991) 469 – 477.
  * [83] P. Janot and S. Jadach, hep-ph/1912.02067.
  * [84] A. Blondel et al., hep-ph/1905.05078.
  * [85] S. Jadach, W. Placzek, E. Richter-Wa̧s, B. F. L. Ward, and Z. Wa̧s, Comput. Phys. Commun. 102 (1997) 229\.
  * [86] A. Blondel et al., hep-ph/1809.01830.
  * [87] B.F.L. Ward, PoS ICHEP2018 (2018) 620\.
  * [88] B.F.L. Ward, Int. J. Mod. Phys. A 33 (2018) 1830028.
  * [89] S. Jadach and B. F. L. Ward, Phys. Rev. D38 (1988) 2897.
  * [90] S. Weinberg, Phys. Rev. 140 (1965) B516.
  * [91] S. Weinberg, in General Relativity, an Einstein Centenary Survey, eds. S. W. Hawking and W. Israel, (Cambridge Univ. Press, Cambridge, 1979).
  * [92] A. H. Guth, Phys. Rev.D 23 (1981) 347.
  * [93] A. Linde, Lecture Notes in Phys. 738 (2008) 1.
  * [94] A. Bonanno and M. Reuter, Phys. Rev. D 65 (2002) 043508.
  * [95] A. Bonanno and M. Reuter, Jour. Phys. Conf. Ser. 140 (2008) 012008\.
  * [96] I. L. Shapiro and J. Sola, Phys. Lett. B 475 (2000) 236.
  * [97] V. Branchina and D. Zappala, G. R. Gravit. 42 (2010) 141.
  * [98] B.F.L. Ward, Mod. Phys. Lett. A 30 (2015) 1550206.
  * [99] A. G. Riess et al., Astron. Jour. 116 (1998) 1009.
  * [100] S. Perlmutter et al., Astrophys. Jour. 517 (1999) 565.
  * [101] C. Amsler et al., Phys. Lett. B 667 (2008) 1.
  * [102] B.F.L. Ward, Phys. Rev. D 57 (1998) 971.