# Data Curation for Image Captioning with Text-to-Image Generative Models

Wenyan Li1, Jonas F. Lotz1,2, Chen Qiu3, Desmond Elliott1
1Department of Computer Science, University of Copenhagen
2ROCKWOOL Foundation Research Unit
3School of Computer Science and Technology, Wuhan University of Science and
Technology
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent advances in image captioning are mainly driven by large-scale vision-
language pretraining, relying heavily on computational resources and
increasingly large multimodal datasets. Instead of scaling up pretraining
data, we ask whether it is possible to improve performance by improving the
quality of the samples in existing datasets. We pursue this question through
two approaches to data curation: one that assumes that some examples should be
avoided due to mismatches between the image and caption, and one that assumes
that the mismatch can be addressed by replacing the image, for which we use
the state-of-the-art Stable Diffusion model. These approaches are evaluated
using the BLIP model on MS COCO and Flickr30K in both finetuning and few-shot
learning settings. Our simple yet effective approaches consistently outperform
baselines, indicating that better image captioning models can be trained by
curating existing resources. Finally, we conduct a human study to understand
the errors made by the Stable Diffusion model and highlight directions for
future work in text-to-image generation.

##  1 Introduction

Large-scale vision-language pretraining has been the driving force for recent
advances in image captioning. Most recent approaches have improved downstream
performance by increasing the amount of image–text pairs used to pretrain
generative language models Li et al. (2022a); Hu et al. (2022); Wang et al.
(2021), as opposed to earlier work that trained masked language models (Tan
and Bansal, 2019; Chen et al., 2019, etc.). However, it is expensive to
collect human-authored captions, such as those in the MS COCO dataset Lin et
al. (2014), which has resulted in the exploitation of “noisy” examples
harvested from the web during pretraining, e.g., using the Conceptual Caption
Sharma et al. (2018) or LAION datasets (Schuhmann et al., 2021). The use of
these noisy sources often involve extra alignment and filtering steps, in
addition to much more pretraining, which leads to greatly increased resource
requirements.

Figure 1: Overview of the data curation approaches. For dynamic removal or
replacement of captions, the high loss image-text pairs are either removed or
the image is paired with an alternative caption in the following training
epoch. For image replacement, captions of original images are used as prompts
for text-to-image generation to synthesize new image-text pairs. We experiment
with both options of replacing the image only, or pair another relevant
caption to the synthesized image.

As an alternative, we propose simple data curation methods to improve image
captioning without additional alignment between synthetic texts and images,
scaling up or changes to pretraining models. We introduce two approaches based
on dynamically updating the training dataset, with the goal of improving the
consistency of the existing training data, based on the notion that some
samples are difficult to learn from:

  1. 1.

Removal or replacement of captions. Inconsistent data may arise from
mismatched image–caption pairs. It is also possible that some captions are too
long and detailed. The original dataset is updated based on sample-wise
losses, in order to remove the samples entirely or to replace the captions.

  2. 2.

Image replacement with a text-to-image generation model. Instead of removing
difficult examples, we hypothesise that the captions may better describe
different images. To this end, we use a publicly-available text-to-image
generation model (Rombach et al., 2022, Stable Diffusion), to create
replacement images for difficult samples.

Experiments on the Flickr30K and MS COCO dataset shows that these approaches
to data curation can improve the performance of the pretrained BLIP model on
the image captioning task. This shows that better models can be trained by
curating an existing dataset, and, more broadly, that text-to-image generation
is a promising direction for model-in-the-loop multimodal learning.

##  2 Motivation and Problem Formulation

Our goal is to improve image captioning models by increasing the consistency
of existing datasets through data curation. In the most commonly used
datasets, e.g. MS COCO Lin et al. (2014) and Flickr30K Young et al. (2014),
each image is paired with five human-annotated reference captions. Previous
research has highlighted some problems with many-to-one annotation, mainly
concerning a lack of diversity in the data Atliha and Šešok (2020), and that
some captions may include verbose details or contain inconsistent
descriptions.

Given an image captioning training dataset $\mathcal{D}$, let Ii be the $i$th
image, and C${}^{j}_{i}$ be $j$th caption of the image, and let (Ii,
C${}_{i}^{j}$) be a sample in dataset. Assume a model $\mathcal{M}$, which is
being trained on the dataset, calculates the loss of each sample in the
dataset at each epoch $t$: $\mathcal{L}_{\mathcal{M}}^{t}$(Ii, C${}^{j}_{i}$).
We argue that the model $\mathcal{M}$ can be improved if it does not continue
to process difficult samples with unusually high losses. Therefore, our data
curation methods are designed to remove those difficult examples from the
original training data, thus creating an updated datasets throughout the
training process $\mathcal{D}_{1},\ldots,\mathcal{D}_{T}$. See Figure 1 for an
overview of the proposed data curation method. With our experiments, we aim to
answer these questions:

  1. 1.

Can image captioning models benefit from increasing consistency within a
dataset by simply removing difficult samples or replacing the captions for
image-text pairs that are likely to contain mismatches?

  2. 2.

Is it useful to synthesize new training images with a state-of-the-art text-
to-image generation model in order to enrich datasets with more unique
training samples?

To answer the above questions, we use two methods to curate the samples in
existing datasets during finetuning. Given the most recent loss calculated for
a sample, we can either remove training samples or replace image captions
based on the sample loss (Section 3.2). Or, based on existing captions for
each image, we can use Stable Diffusion to synthesize new images in order to
introduce better learning examples to replace the difficult ones in the
original training set (Section 3.3).

##  3 Data Curation for Captioning

###  3.1 Identifying the difficult samples

Difficult training samples may contain mismatches or inconsistencies between
the image and the caption. We use the model itself to automatically identify
such samples. After each epoch, we compute the loss incurred for each training
sample. The highest loss samples are targets for our data curation methods. In
this way, the training dataset is dynamically updated according to the model’s
captioning capability.

###  3.2 Removal/Caption Replacement

The simplest approach to data curation is to remove or replace the difficult
samples from the training process. As shown in Figure 1, given a set of high
loss samples, we experiment with two simple options, Remove and ReplaceCap. In
Remove, these samples are completely removed from the remainder of the
training process, effectively reducing the total number of image-caption
pairs. In ReplaceCap, we simply replace the caption in the image-text pair
with another more reasonable caption. What does it mean to be reasonable? A
reasonable caption is either short or it already achieves a low loss for this
image. Longer captions are typically much more difficult to learn from,
yielding undesired gradient updates. With the caption replacement method, the
total number of the samples seen by the model remain the same, as well as the
total number of the unique images.

###  3.3 Image Generation-based Replacement

An alternative to removing difficult samples is to pair the caption with a new
image. This has the benefit of training the model on the same total number of
samples while increasing the consistency of the dataset. In ReplaceImg, we
experiment with using a text-to-image generation model to synthesize images
based on the other sentences that describe an image for a difficult sample.

Our methodology is based on the open source Stable Diffusion model Rombach et
al. (2022), which can generate images given a textual prompt.111It is also
possible to use API-based models but we chose Stable Diffusion for two
reasons: integration and cost. Stable Diffusion integrated directly into our
training pipeline, while we estimate the cost of a single experiment on the
Flickr30K dataset to be $7,424 through the DALLE-2 API. We experiment with two
options for integrating image generation-based replacement into image
captioning based on either standard fully-supervised finetuning or a few-shot
learning setting:

##### Standard Finetuning:

Given an image $I_{i}$ in the training data and its captions
$\\{(I_{i},C^{1}_{i}),\dots,(I_{i},C_{i}^{J})\\}$, we can synthesize a new
image $I^{{}^{\prime}}_{i}$ without increasing the total number of samples in
the original dataset. Instead, we replace the original image in the sample
with the generated image. Specifically, for image $I_{i}$, we replace the
sample $(I_{i},C^{1}_{i})$ with the synthesized image-text pair
$(I^{{}^{\prime}}_{i},C^{1}_{i})$.

##### Few-shot Learning:

Image Generation-based curation can also be applied in a lower-resource
setting, where the model is only able to learn from a very limited number of
unique image-text pairs. Reference captions of the original shots are reused
as prompts for text-to-image generation and paired to the synthesized images.

####  3.3.1 Image Generation Methodology

We now explain our image generation methodology, which includes the importance
of using good prompts, of fine-tuning the image generation model, and we
present a round-trip image captioning evaluation that can be used to
quantitatively measure the quality of the generated images.

#### Prompt engineering matters

Figure 2: Different prompting strategies for synthetic image generation with
Stable Diffusion and representative examples. Based on our Round-trip
Captioning Evaluation, prompting with the concatenated captions and the styler
generates the best images for the task.

Text-to-image diffusion models generate images based on the text prompts. As
shown in Figure 2, given a set of five captions $[C^{1},\dots,C^{5}]$ that
describe an image, there are multiple options for how to prompt the image
generation model. We experiment with three options:

  * •

Concatenation: All five captions are concatenated as the text prompt for
generation.

  * •

Sentence-BERT selection: There is a lot of variety in how different captions
describe the same image. Instead of using all captions, we can use a
representative caption from the set. This is achieved using the Sentence-BERT
(Reimers and Gurevych, 2019, SBERT) model to find the caption that is closest
to the average embedding of all captions.

  * •

Single caption: Each caption can be used as an individual prompt to generate a
synthesized image for augmentation.

  * •

+Styler: For all three approaches mentioned above, we can append an additional
string to the prompt as a styler to force a specific style in the generated
image. The styler used here is: "national geographic, high quality
photography, Canon EOS R3, Flickr".222The styler was chosen by manually
evaluating the quality of the images with a preference against “artistic”
outputs.

#### Finetuning improves image relevance

Through preliminary experiments, we find that the Stable Diffusion model
benefits from specific style guidance and details regarding the objects in the
image for disambiguation purpose. To fill the gap between the text prompts and
original image captions, we also finetune Stable Diffusion on the MS COCO Lin
et al. (2014) dataset, in order to generate images that better reflect the
distribution of image captioning datasets. We finetune the Stable Diffusion on
the MS COCO training split using a concatenation of all 5 captions as the
prompt. The finetuning is done for 15,000 steps with a constant learning rate
of $110-5$ and a batch size of 32. We find that version 1.5 achieves the
highest performance in downstream evaluation.

#### Round-trip captioning evaluation

Figure 3: Illustration of the round-trip captioning process. Captions of
original images are used as ground truth. If the synthesize images are of high
quality, the predicted captions of the synthesized images are expected to be
similar to the ground truth captions.

Model | FT | Prompt | B | C | M | R | S
---|---|---|---|---|---|---|---
Upper-bound | 37.6 | 27.2 | 57.1 | 92.8 | 20.1
SD 1.5 | - | concat | 31.0 | 24.7 | 52.5 | 77.9 | 18.4
SD 1.5 | - | \+ styler | 30.8 | 24.2 | 52.5 | 75.6 | 19.4
SD 1.5 | F | \+ styler | 33.5 | 25.0 | 53.5 | 81.3 | 20.5
SD 1.5 | F | SBERT + styler | 30.6 | 24.1 | 52.0 | 76.2 | 17.7
SD 2.0 | - | concat + styler | 31.2 | 24.8 | 52.0 | 77.6 | 19.9

Table 1: Round-trip captioning evaluation with different Stable Diffusion
models, prompts, and fine-tuning of the diffusion model. Compared to the
Upper-bound, which is evaluated using the original images, the best image
generation method is Stable Diffusion v1.5, fine-tuned on the COCO dataset,
with a concat + styler prompt.

Most work on text-to-image generation evaluate image quality using image-
oriented measures like FID Heusel et al. (2017) or image-text alignment using
CLIPScore Hessel et al. (2021a). Both measures have their limitations, as
discussed in Saharia et al. (2022c). An alternative evaluation that is
directly related to our task is to determine the quality of the caption
generated by an image-captioning model using the generated image compared to
the original image (Figure 3). The assumption is that if the synthesized
images are of a similar quality to the original ones, the predicted captions
should also contain same information that describe them.

There are three steps involved in the round-trip evaluation. First, we use the
captions in the validation set to generate synthetic images. Second, we use an
image-captioning model to predict captions on the synthetic images. Third, we
compare the predicted captions for the synthetic images to the original. To
make sure that the image-captioning model is reliable to produce high-quality
captions, we used a captioning model which was finetuned on the original
human-annotated data. We use this evaluation to determine the best performing
image generation method for our data curation method. Table 1 shows the
results, where the Upper-bound is the caption generation model using the
original images. The best performance is found by fine-tuning the diffusion
model on the MS COCO dataset and using a prompt with a concatenation of the
captions and the styler text. We use this configuration in the remainder of
the paper.

##  4 Experiments

We evaluate our proposed data curation methods on the MS COCO and Flickr30K
datasets in finetuning and few-shot learning based on the state-of-the-art
pretrained V&L model, BLIP Li et al. (2022a). We evaluate the captions using
BLEU Papineni et al. (2002), METEOR Denkowski and Lavie (2014), ROUGE Lin
(2004), CIDEr Vedantam et al. (2015), SPICE Anderson et al. (2016), CLIPScore,
and RefCLIPScore Hessel et al. (2021b).

###  4.1 BLIP model

We use the ViT-based BLIP model Li et al. (2022a) as the V&L backbone.
Compared to other pretrained V&L models, BLIP has the advantage of requiring
no visual feature extraction and being relatively computation-friendly Chen et
al. (2019). BLIP includes a captioning and filtering (CapFilt) data
augmentation process during its pretraining, and the captioner and filter were
both finetuned on the COCO dataset. Therefore we use pretrained checkpoint
BLIPCapFilt for Flickr30k and BLIPbase for COCO in our experiment, excluding
effects from the CapFilt process. We finetune on both Flickr30K and MS COCO
with our proposed curation methods using a batch size of $128$ and train for
$5$ epochs on $4\times$ A100 GPUs. Following Li et al. (2022a), captions are
generated using the prompt: "a picture of ".

###  4.2 Data Curation improves Finetuning

#### Removal/Caption Replacement

As shown in Table 2, dynamically removing mismatched image-text pairs or
replacing captions can effectively improve performance on both datasets over
baselines on all metrics. The dynamic updates work best when apply to the top
1% samples for Remove, and to samples whose loss are two standard deviations
away from the mean for ReplaceCap. During the updating process, no additional
data samples or computation cost is introduced. We further study the effect of
the amount of curation involved in Section 5.

#### Image Generation-based Replacement

We evaluate Image Generation-based Replacement on both the Flickr30K and COCO
dataset. During finetuning, we replace images in the original text-image pairs
with Stable Diffusion-synthesized images (ReplaceImg in Table 2). The results
show improvements compared to the baseline in every evaluation measure with
best performance obtained at replacement ratio of 40% for Flickr30K and at 10%
for COCO. We show qualitative examples in Figure 4, where models finetuned
with our proposed curation method can generate better captions for some scenes
that may confuse the standard finetuned model. In Section 5.1, we analyze the
effects of varying the amount of synthetic images replaced, and in Section
5.2, we conduct a human study of the types of errors found in the generated
images.

Figure 4: Qualitative examples from the COCO dataset of captions generated by
the BLIP model (top), and the same model trained using our ReplaceImg data
curation (bottom). The errors made by the BLIP model (shown in red) are
avoided by ReplaceImg curation (shown in blue).

Method B M R C S CS RCS Flickr30K BLIPCapFilt 37.6 27.2 57.1 92.8 20.1 78.6
81.1 +Remove 38.6 27.4 57.5 95.8 21.0 79.2 81.9 +ReplaceCap 37.9 27.4 57.4
94.5 21.1 78.9 81.5 +ReplaceImg 39.0 27.3 57.4 95.7 20.7 79.1 82.0 COCO
BLIPbase 39.9 30.8 59.9 132.0 23.8 77.3 82.8 +Remove 40.1 30.9 60.0 132.5 23.6
77.3 82.8 +ReplaceCap 40.2 30.9 60.1 132.7 23.9 77.3 82.8 +ReplaceImg 40.2
31.0 60.1 133.1 23.9 77.3 82.8

Table 2: Results for standard finetuning with data curation. We find
improvements for all methods compared to the baseline of training on the
original datasets. Best scores are in bold, second-best are in italic.

###  4.3 Image Generation for Few-shot Learning

We perform 16-shot learning on both datasets and train the captioning model
for 20 iterations. Instead of replacing examples in the dataset, we augment
the small dataset with additional image-caption samples where the images are
generated from the Stable Diffusion image generation model. The Stable
Diffusion generated images contribute to better few-shot learning on both
datasets (Table 3).

Shots | B | M | R | C | S | CS | RCS
---|---|---|---|---|---|---|---
Flickr30K
0 | 25.6 | 20.6 | 49.4 | 61.0 | 14.5 | 75.3 | 79.5
4 | 26.9 | 21.0 | 50.0 | 62.7 | 15.1 | 75.2 | 79.3
COCO
0 | 35.7 | 27.2 | 56.5 | 113.4 | 20.5 | 76.5 | 82.0
16 | 36.1 | 27.0 | 56.6 | 113.6 | 20.0 | 75.6 | 81.4

Table 3: Results for 16-shot image captioning with $n$=0, 4, or 16 additional
shots using Image Generation-based Augmentation. Results are averaged across
three runs for Flickr30K and two runs for MS COCO. Image Generated-based
Augmentation improves performance comapred to using the limited few-shot
examples.

##  5 Analysis and Discussion

To better understand how our data curation methods can improve the model, we
study different factors in the process, such as amount of data curated and
static versus dynamic image replacement. We also present the findings of a
human study of the errors made by the Stable Diffusion model to highlight how
it could further improve image captioning.

###  5.1 Amount of curation matters

We analyze how the amount of curation affects image captioning performance for
standard finetuning. We examine different ratios of training samples that are
removed, replaced with an alternative caption, or replaced with a synthesized
image.

Figure 5: The effect of removing or replacing different amounts of the data.
Dashed lines are baseline scores for the metrics. Flickr30K benefits more from
removing high-loss training samples, indicating the original dataset may be
noisier than MS COCO.

Figure 6: Effects of curation ratio of ReplaceImg in Flickr30K and COCO
finetuning, where we can observe that Flickr30K needs more curation (40%) than
COCO (10%). Blue and red dashed lines indicate baseline BLEU4 and CIDEr
scores. After each epoch we replace images based on their loss. Our approach
show consistent improvements over both baselines.

##### Flickr30K needs more curation than COCO.

As shown in Figure 5 and Figure 6, the highest performance for Flickr30K is
achieved either through removing high loss samples that are two standard
deviations away, or replacing images for 40% of the high loss samples.
However, for COCO finetuning, replacing images for 10% of the high loss
samples outperforms other options. The second best performing method for COCO
is removing or replacing captions of only 1% of the high loss samples. This
indicates that Flickr30K may contain more noisy samples than the MS COCO
dataset. Compared to MS COCO, Flickr30K contains more samples with long
captions (See Figure 11 in the Appendix), which may include overly-specific
details that are inconsistent with other captions and are hard for the model
to learn. See Figure 14 for more examples. This may indicate that our method
is efficient when training with noisy datasets.

##### Static image replacement versus dynamic replacement

Figure 7: Dynamic image replacement against static replacement, as a function
of the number of samples replaced. It is clear that dynamically replacing
images during training is preferred over a static replacement.

In ReplaceImg (Section 3.3), we dynamically replace images for the difficult
training samples. Another static approach is to replace the identical images
with unique SD-synthesized images before training, instead of updating the
training samples online. We compare the efficacy of these two approaches in
Figure 7. With static image replacement, for each of the reference captions,
we replace their original image with a SD-synthesized image. Static
replacement with 20%–80% curation ratio corresponds to replacing images for
one–four captions of the original five. The 50% replacement ratio mimics a
fair coin-flip, where for each of the text-image samples, there is 50%
probability for the image to be replaced by a synthesized image. When
evaluating on the original 1k validation set, we see that for both approaches,
incorporating synthesized images of 20% or 40% can assist finetuning and
achieves higher BLEU4 and CIDEr scores. Nevertheless, dynamic image
replacement consistently performs better than the static method, showing
focusing on the hard samples is effective. For both replacement methods,
performance starts to decrease when the curation ratio is too high. This may
indicate that when incorporating too many images from the synthetic
distribution, the gap increases between the training and evaluation sets.

Figure 8: Loss distribution of training samples across epochs with different
curation methods. For the removal approach, training samples with loss that
are 2std away are dynamically removed during finetuning, leading to the
shrinking tail of the loss distribution.

We can also observe from Figure 8 that, while dynamic removal gradually
eliminates high-loss samples from the tail of the distribution during
training, SD-based image replacement gradually reducing losses through
learning from a mixture of Gaussian distribution from original image-text
pairs and the ones contain synthesized images.

(a) Distribution of text-to-image generation errors.

(b) Human evaluation versus CLIPScore.

Figure 9: Results of the human study of the errors made by the Stable
Diffusion model in 100 images. The images used in the study were chosen to
represent either low or high model loss. (a) Histogram of the number of errors
annotated in each category. The most frequently occurring annotations concern
weird deformations in the expected objects or humans. (b) Relationship between
average number of identified errors by human annotations for each synthesized
image and its captioning loss with regard to original captions. More errors
are identified in images of higher loss. However, CLIPScore appears to fail in
validating qualities of the synthesized images, as the score ranges are almost
identical for samples that contain more errors.

###  5.2 Human Study: Errors made by SD models

Finally, we conduct a human study of the errors present in the SD-synthesized
images. This will serve to better understand any shortcomings with this
approach that is not captured by automatic evaluation measures.

We first ranked SD-synthesized images by model loss from the 1K images in the
validation set. This validation set of synthesized images was generated using
the best performing configuration of the Stable Diffusion model (see Section
3.3). We then sampled a subset for human annotation using the top and bottom
$50$ images based on their loss using our fine-tuned captioning model. These
images are uniformly divided into $5$ sets, each containing $20$ images with
equal number of the high loss ones and the low loss ones. The data was
annotated by 12 people, members of a university research lab with a basic
understanding of Stable Diffusion but no knowledge of the bi-modal
distribution of images. The annotators were asked to categorize the errors
they observed in the synthesized images, given both the image and the
reference sentences that were used to generate the images. Each participant
annotated one set of 20 images.

Starting from the categories defined by van Miltenburg and Elliott (2017), we
predefined 25 categories including general errors such as color, or number
mismatches, and errors related to people and objects in the images (Figure
10). We analyze the human judgements for the images that have at least three
annotations, yielding 74 unique images.

As shown in Figure 9(a), the most common problem of SD-synthesized images are
that they often generate weird face or body parts, which makes the images less
natural or pleasant. The Stable Diffusion model is also weak at generating the
correct number of people or objects. From Figure 9(b) we confirm the quality
of our collected annotations that high loss figures often contain more errors
on average. Furthermore, we note that CLIPScore does not appear to align with
human judgements, indicating its weak capability of evaluating quality of
generated images. Please see more concrete examples in Figure 12.

##  6 Related work

##### Image Captioning

Image Captioning is the task of describing images with syntactically and
semantically sentences. Current deep learning-based image captioning models
have evolved as the encode-decoder frameworks with multi-modal connection Chen
et al. (2017, 2018), attentive Huang et al. (2019); Guo et al. (2020) and
fusion strategies Zhou et al. (2020). Standard captioning datasets contain
Flickr30K Young et al. (2014) and the commonly used MS COCO Lin et al. (2014),
which consisting of images with events, objects and scenes. Each image is
paired with five captions. Some works have demonstrated the benefits of
training on synthetic captions Li et al. (2022b); Atliha and Šešok (2020) or
datasets collected from other vision-and-language learning tasks Radford et
al. (2021); Changpinyo et al. (2022).

##### Data Augmentation

Data augmentation Feng et al. (2021) has achieved increasing attention in both
the natural language processing Liu et al. (2020) and vision-and-language
learning Li et al. (2021) fields. Early methods generate augmented examples in
the model’s feature space Wei and Zou (2019) or interpolate the inputs and
labels of few examples Zhang et al. (2018). For downstream tasks in the text
domain, Yang et al. (2020) and Anaby-Tavor et al. (2020) generate synthetic
text examples through state-of-the-art pretrained language models and show
improved performance on common-sense reasoning and text-classification. For
image captioning, Atliha and Šešok (2020) used BERT Kenton and Toutanova
(2019) to generate additional captions to improve the diversity of the
dataset.

##### Diffusion Models and Application

Diffusion models Song et al. (2021); Nichol and Dhariwal (2021) have grown
rapidly and become the powerful deep generative models. They have shown
potential in a variety of applications, including text-to-image generation
Nichol et al. (2022b); Gu et al. (2022), image-to-image translation Saharia et
al. (2022a), as well as semantic segmentation Jiang et al. (2018); Baranchuk
et al. (2022) and video generation Ho et al. (2022); Singer et al. (2022);
Villegas et al. (2022). While recent large scale latent diffusion models have
shown strong capability in generating both artistic and photo-realistic high-
resolution images (Rombach et al., 2022; Nichol et al., 2022a; Ramesh et al.,
2022; Saharia et al., 2022b), applying large-scale stable diffusion models in
vision-language downstream tasks remains under-explored.

Concurrently, Azizi et al. (2023) and Jain et al. (2023) show that image
classifiers can be improved by learning from augmentation images generated by
finetuned stable-diffusion models. To the best of our knowledge, we are the
first to explore how image captioning models can benefit from simple data
curation without scaling up existing datasets, and how stable-diffusion text-
to-image models can be applied and contribute in the process.

##  7 Conclusion

In this paper, we have shown a simple, yet effective, data curation framework
that can improve the performance of image captioning models. We investigated
two approaches to data curation that dynamically update the training dataset
based on difficult image-caption samples. The first method involved removing
or replacing the caption in difficult samples, while the second used Stable
Diffusion as a model-in-the-loop to synthesize new images for difficult
samples. Experimental results on the Flickr30K and MS COCO datasets show the
effectiveness of both approaches to data curation. A deeper analysis of the
images synthesized by Stable Diffusion shows frequent errors on generating
objects of a certain amount or color, and struggles with human body features.
A human evaluation of the errors in those images shows a clear difference in
images that prove easy/difficult to model.

In the future, we expect that better text-to-image generation models will lead
to further improvements from using synthesized images for difficult captions
in existing training datasets. We plan on verifying whether these findings
extend to other image captioning models, which was not possible here due to
computational issues. Finally, we are interested in applying the same
framework to other multimodal tasks, especially those with undercomplete
datasets that cannot comprehensively cover the distributional space due to the
cost of crowdsourcing enough data, e.g. visual question answering, or
visually-grounded dialog.

## Limitations

While our curation methods being effective on image-captioning in the
finetuning and fewshot-learning settings, it is not clear if the same strategy
can be scaled and adapted also to vision-language pretraining. Currently our
data curation methods also rely on state-of-the art pretrained models for both
image understanding and text-to-image generation. In pretraining, models will
often be trained from scratch and pretraining data are often collected from
multiple datasets and resources.

Moreover, while we take an online approach to data curation, our current
approach is upper bounded in speed and performance of the text-to-image
generation model. This might be a large bottle neck for adapting the strategy
for more complicated vision-and-language tasks.

## Ethics Statement

Text-to-image generation with Stable Diffusion is controversial in the broader
AI and ethics community. For example, it can generate images according to
gender of racial stereotypes, which may prove harmful to members of those
communities Li et al. (2022c). In this paper, we use Stable Diffusion to
improve the quality of an image captioning model, given a specific set of
crowdsourced captions. Those captions may themselves contain harmful
stereotypes that would become more prevalent in our dynamically updated
training datasets. As we dynamically update the model with new images based on
loss values, we remove the water-marker in our generated images to prevent
information leak to the model. Use of the synthesized images will strictly
follow community guidelines.

## References

  * Anaby-Tavor et al. (2020) Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020.  Do not have enough data? deep learning to the rescue!  In _AAAI_ , pages 7383–7390. AAAI Press.
  * Anderson et al. (2016) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016.  Spice: Semantic propositional image caption evaluation.  In _European conference on computer vision_ , pages 382–398. Springer.
  * Atliha and Šešok (2020) Viktar Atliha and Dmitrij Šešok. 2020.  Text augmentation using bert for image captioning.  _Applied Sciences_.
  * Azizi et al. (2023) Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. 2023.  Synthetic data from diffusion models improves imagenet classification.
  * Baranchuk et al. (2022) Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 2022.  Label-efficient semantic segmentation with diffusion models.  In _International Conference on Learning Representations_.
  * Changpinyo et al. (2022) Soravit Changpinyo, Doron Kukliansy, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. 2022.  All you may need for VQA are image captions.  In _NAACL_ , pages 1947–1963. Association for Computational Linguistics.
  * Chen et al. (2017) Fuhai Chen, Rongrong Ji, Jinsong Su, Yongjian Wu, and Yunsheng Wu. 2017.  Structcap: Structured semantic embedding for image captioning.  In _MM_ , pages 46–54. ACM.
  * Chen et al. (2018) Fuhai Chen, Rongrong Ji, Xiaoshuai Sun, Yongjian Wu, and Jinsong Su. 2018.  Groupcap: Group-based image captioning with structured relevance and diversity constraints.  In _CVPR_ , pages 1345–1353. Computer Vision Foundation / IEEE Computer Society.
  * Chen et al. (2019) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019.  Uniter: Universal image-text representation learning.  _european conference on computer vision_.
  * Denkowski and Lavie (2014) Michael Denkowski and Alon Lavie. 2014.  Meteor universal: Language specific translation evaluation for any target language.  In _Proceedings of the ninth workshop on statistical machine translation_ , pages 376–380.
  * Feng et al. (2021) Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard H. Hovy. 2021.  A survey of data augmentation approaches for NLP.  In _ACL_ , volume ACL/IJCNLP 2021 of _Findings of ACL_ , pages 968–988. Association for Computational Linguistics.
  * Gu et al. (2022) Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. 2022.  Vector quantized diffusion model for text-to-image synthesis.  In _CVPR_ , pages 10686–10696. IEEE.
  * Guo et al. (2020) Longteng Guo, Jing Liu, Xinxin Zhu, Peng Yao, Shichen Lu, and Hanqing Lu. 2020.  Normalized and geometry-aware self-attention network for image captioning.  In _CVPR_ , pages 10324–10333. Computer Vision Foundation / IEEE.
  * Hessel et al. (2021a) Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021a.  CLIPScore: a reference-free evaluation metric for image captioning.  In _EMNLP_.
  * Hessel et al. (2021b) Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021b.  CLIPScore: A reference-free evaluation metric for image captioning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7514–7528, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In _NIPS_.
  * Ho et al. (2022) Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. 2022.  Video diffusion models.  _arXiv:2204.03458_.
  * Hu et al. (2022) Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2022.  Scaling up vision-language pre-training for image captioning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17980–17989.
  * Huang et al. (2019) Lun Huang, Wenmin Wang, Jie Chen, and Xiaoyong Wei. 2019.  Attention on attention for image captioning.  In _ICCV_ , pages 4633–4642. IEEE.
  * Jain et al. (2023) Saachi Jain, Hannah Lawrence, Ankur Moitra, and Aleksander Madry. 2023.  Distilling model failures as directions in latent space.  In _The Eleventh International Conference on Learning Representations_.
  * Jiang et al. (2018) Peng Jiang, Fanglin Gu, Yunhai Wang, Changhe Tu, and Baoquan Chen. 2018.  Difnet: Semantic segmentation by diffusion networks.  In _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of naacL-HLT_ , volume 1, page 2.
  * Li et al. (2021) Guodun Li, Yuchen Zhai, Zehao Lin, and Yin Zhang. 2021.  Similar scenes arouse similar emotions: Parallel data augmentation for stylized image captioning.  In _MM_ , pages 5363–5372. ACM.
  * Li et al. (2022a) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022a.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _ICML_.
  * Li et al. (2022b) Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022b.  BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _ICML_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 12888–12900. PMLR.
  * Li et al. (2022c) Minghui Li, Yan Wan, and Jinping Gao. 2022c.  What drives the ethical acceptance of deep synthesis applications? a fuzzy set qualitative comparative analysis.  _Computers in Human Behavior_ , 133:107286.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll’a r, and C. Lawrence Zitnick. 2014.  Microsoft COCO: common objects in context.  _CoRR_ , abs/1405.0312.
  * Liu et al. (2020) Ruibo Liu, Guangxuan Xu, Chenyan Jia, Weicheng Ma, Lili Wang, and Soroush Vosoughi. 2020.  Data boost: Text data augmentation through reinforcement learning guided conditional generation.  In _EMNLP_ , pages 9031–9041. Association for Computational Linguistics.
  * Nichol et al. (2022a) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2022a.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.
  * Nichol and Dhariwal (2021) Alexander Quinn Nichol and Prafulla Dhariwal. 2021.  Improved denoising diffusion probabilistic models.  In _ICML_ , volume 139, pages 8162–8171. PMLR.
  * Nichol et al. (2022b) Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2022b.  GLIDE: towards photorealistic image generation and editing with text-guided diffusion models.  In _ICML_ , volume 162, pages 16784–16804.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _ICML_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical text-conditional image generation with clip latents.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Rombach et al. (2022) Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10674–10685.
  * Saharia et al. (2022a) Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. 2022a.  Palette: Image-to-image diffusion models.  In _SIGGRAPH_ , pages 15:1–15:10. ACM.
  * Saharia et al. (2022b) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. 2022b.  Photorealistic text-to-image diffusion models with deep language understanding.
  * Saharia et al. (2022c) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. 2022c.  Photorealistic text-to-image diffusion models with deep language understanding.  _arXiv preprint_.
  * Schuhmann et al. (2021) Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint_.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2556–2565, Melbourne, Australia. Association for Computational Linguistics.
  * Singer et al. (2022) Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. 2022.  Make-a-video: Text-to-video generation without text-video data.
  * Song et al. (2021) Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2021.  Score-based generative modeling through stochastic differential equations.  In _ICLR_.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  LXMERT: Learning cross-modality encoder representations from transformers.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5100–5111, Hong Kong, China. Association for Computational Linguistics.
  * van Miltenburg and Elliott (2017) Emiel van Miltenburg and Desmond Elliott. 2017.  Room for improvement in automatic image description: an error analysis.  _CoRR_ , abs/1704.04198.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4566–4575.
  * Villegas et al. (2022) Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. 2022.  Phenaki: Variable length video generation from open domain textual description.
  * Wang et al. (2021) Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2021\.  Simvlm: Simple visual language model pretraining with weak supervision.  _Learning_.
  * Wei and Zou (2019) Jason W. Wei and Kai Zou. 2019.  EDA: easy data augmentation techniques for boosting performance on text classification tasks.  In _EMNLP-IJCNLP_ , pages 6381–6387. Association for Computational Linguistics.
  * Yang et al. (2020) Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020.  G-daug: Generative data augmentation for commonsense reasoning.  In _EMNLP_ , volume EMNLP 2020 of _Findings of ACL_ , pages 1008–1025. Association for Computational Linguistics.
  * Young et al. (2014) Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  _TACL_ , 2:67–78.
  * Zhang et al. (2018) Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. 2018\.  mixup: Beyond empirical risk minimization.  In _ICLR_. OpenReview.net.
  * Zhou et al. (2020) Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. 2020.  Unified vision-language pre-training for image captioning and VQA.  In _AAAI_ , pages 13041–13049. AAAI Press.

##  Appendix A Example Appendix

###  A.1 User interface for human study on categorizing text-to-image
generation errors

Figure 10: Annotation interface for categorizing SD errors. Figure 11:
Distribution of number of words in the captions. Some of the captions in the
Flickr30K dataset are extremely long and contain unnecessary details, which
makes it hard for the model to learn. Figure 12: Examples of synthesized
images that are of high losses. Human annotations show that consistent error
types have been recognized while CLIPScore fails to align with human
judgement.

###  A.2 Examples of synthesized images and high/low loss training samples

We visualize the top five images that get contradictory ranks in loss
validation and CLIPScore (Figure 4.) From the human annotations, we can see
human judgements correlate better with the image loss instead of the
CLIPScore. In Figure 13, we also show that some of the synthesized images that
are of low losses are visually less complicated than the higher loss ones,
however can still contain errors or look unnatural.

Figure 13: Examples of synthesized images that are of low losses. Human
annotations show that these images are visually less complicated than the
higher loss ones, but can still often look weird and contain errors in color
or objects Figure 14: High loss training samples in COCO after the first
epoch, ranked by loss in descending order. The top samples all have very long
captions over 30 words, compared to the mean of 11 words.


# Propagation of equilibrium states in stable families of endomorphisms of
$\mathbb{P}^{k}(\mathbb{C})$.

Maxence Brevard  and  Karim Rakhimov  Université de Toulouse - IMT, UMR CNRS
5219, 31062 Toulouse Cedex, France<EMAIL_ADDRESS><EMAIL_ADDRESS>V.I. Romanovskiy Institute of Mathematics of Uzbek Academy
of Sciences, Tashkent, Uzbekistan<EMAIL_ADDRESS>
###### Abstract.

We prove that, within any holomorphic family of endomorphisms of
$\mathbb{P}^{k}(\mathbb{C})$ in any dimension $k\geq 1$ and algebraic degree
$d\geq 2$, the measurable holomorphic motion associated to dynamical stability
in the sense of Berteloot-Bianchi-Dupont preserves the class of equilibrium
states associated with weight functions $\psi$ satisfying
$\sup\psi-\inf\psi<\log d$.

##  1\. Introduction

Let $f:X\to X$ be a continuous map on a compact metric space $X$. Given a
Borel bounded function $\psi:X\to\mathbb{R}$, an _equilibrium state_ with
respect to the _weight_ $\psi$ is an $f$-invariant probability measure $\nu$
on $X$ maximizing the pressure functional
$P_{\nu}(\psi):=h_{\nu}(f)+\int\psi\nu$, where $h:\nu\mapsto h_{\nu}(f)$ is
the measure-theoretic entropy. The existence of equilibrium states follows
from a compactness argument, whereas their uniqueness is a strong statistical
property. In this paper, we are interested in the persistence of equilibrium
states and their uniqueness in _stable families of endomorphisms of
$\mathbb{P}^{k}=\mathbb{P}^{k}(\mathbb{C})$_, which we now define.

Given a (connected) complex manifold $M$, a holomorphic family
$\\{f_{\lambda}:\mathbb{P}^{k}\to\mathbb{P}^{k}\\}_{\lambda\in M}$ of
endomorphisms of $\mathbb{P}^{k}$ parametrized by $M$ is a holomorphic map
$F:M\times\mathbb{P}^{k}\to M\times\mathbb{P}^{k}$ of the form
$F(\lambda,z)=(\lambda,f_{\lambda}(z))$ such that all the maps $f_{\lambda}$
are holomorphic endomorphisms of $\mathbb{P}^{k}$ of the same algebraic degree
$d\geq 2$. In dimension $k=1$, the notion of stability within such families
was introduced by Mañé-Sad-Sullivan [33] and Lyubich [31]. A very fruitful
characterization based on potential theory was later introduced by DeMarco
[21], see also [35, 37]. The family
$\\{f_{\lambda}:\mathbb{P}^{1}\to\mathbb{P}^{1}\\}_{\lambda\in M}$ is said to
be _stable_ if there exists a _holomorphic motion_ of the Julia sets (i.e.,
the supports of the measures of maximal entropy of $f_{\lambda}$ [32]) near
every $\lambda_{0}\in M$, namely there exists a neighbourhood $U$ of
$\lambda_{0}$ and, for every $\lambda\in U$, an injective map
$\Phi_{\lambda_{0},\lambda}:J_{\lambda_{0}}\to J_{\lambda}$ depending
holomorphically on $\lambda$. By Hurwitz theorem, these maps are automatically
continuous in $z\in J_{\lambda_{0}}$, hence they define a topological
conjugacy between the systems $(J_{\lambda_{0}},f_{\lambda_{0}})$ and
$(J_{\lambda},f_{\lambda})$. In this context, the action of the holomorphic
motion over equilibrium states is straightforward. If $\nu$ is a
$f_{\lambda_{0}}$-invariant measure with support in $J_{\lambda_{0}}$, the
measure $\nu_{\lambda}:=(\Phi_{\lambda_{0},\lambda})_{*}\nu$ inherits all the
statistical properties from $\nu$. In particular, if $\nu$ is an equilibrium
state with respect to a Hölder continuous weight, since the map
$\Phi_{\lambda_{0},\lambda}$ itself is a Hölder continuous conjugacy (see for
instance [24]), the measure $\nu_{\lambda}$ is still an equilibrium state with
respect to a Hölder continuous weight. We refer to [36] as a general reference
for the thermodynamical formalism in dimension $1$.

The definition and characterization of stability for families in any dimension
$k\geq 1$ are due to Berteloot-Bianchi-Dupont [6, 9], see also [5] and [3, 4,
7, 34] for further characterizations. As Hurwitz Theorem does not hold in
higher dimension, the approach relies on ergodic and pluripotential
techniques. In particular, the holomorphic motion as above is replaced by a
more adapted measure-theoretic notion. Consider the space $\mathcal{J}$ of all
holomorphic maps $\gamma:M\to\mathbb{P}^{k}$ such that $\gamma(\lambda)\in
J_{\lambda}:=\operatorname{Supp}\mu_{\lambda}$ for any parameter $\lambda\in
M$, where $\mu_{\lambda}$ is the unique ergodic measure of maximal entropy of
$f_{\lambda}$, see [17, 23, 28]. Define the dynamical system
$\mathcal{F}:\mathcal{J}\to\mathcal{J}$ induced on $\mathcal{J}$ by
$\mathcal{F}(\gamma)(\lambda):=f_{\lambda}(\gamma(\lambda))$, for all
$\gamma\in\mathcal{J}$ and $\lambda\in M$. For each parameter $\lambda\in M$,
we also consider the evaluation map $p_{\lambda}:\mathcal{J}\to J_{\lambda}$
defined by $p_{\lambda}(\gamma):=\gamma(\lambda)$, where
$\gamma\in\mathcal{J}$.

######  Definition 1.1.

Fix $\lambda_{0}\in M$. Let $\nu$ be an $f_{\lambda_{0}}$-invariant
probability measure supported on $J_{\lambda_{0}}$. A _$(\lambda_{0},\nu)$
-web_ (or _$\nu$ -web_ for brevity) is an $\mathcal{F}$-invariant compactly
supported Borel probability measure $\mathcal{M}$ on $\mathcal{J}$ such that
$(p_{\lambda_{0}})_{*}\mathcal{M}=\nu$.

Dynamical stability in the sense of Berteloot-Bianchi-Dupont is equivalent to
the existence of a special (_acritical_ , see Remark 2.4) _equilibrium web_ ,
namely a probability measure $\mathcal{M}$ which is a
$(\lambda,\mu_{\lambda})$-web for every parameter $\lambda\in M$. Such a
probability measure is also called a _measurable holomorphic motion_. The
construction was generalized by Bianchi and the second author [13] to a larger
class of measures including, among others, all measures whose measure-
theoretic entropy is larger than $(k-1)\log d$. This class includes, for
instance, all equilibrium states with respect to any Hölder continuous
function $\psi$ with $\max\psi-\min\psi<\log d$, whose existence, uniqueness,
and properties were recently proved in [10, 11, 38, 40], see also [27].

In this context, it is a natural question whether the class of equilibrium
states is preserved by the measurable holomorphic motions, i.e., if, given an
equilibrium state $\nu$ for $f_{\lambda_{0}}$, the measures
$\nu_{\lambda}:=(p_{\lambda})_{*}\mathcal{M}$ also satisfy the same property,
where $\mathcal{M}$ is a $(\lambda_{0},\nu)$-web. Observe that, if
$f_{\lambda_{0}}$ is hyperbolic (i.e., uniformly expanding on
$J_{\lambda_{0}}$), the measurable holomorphic motion is indeed a holomorphic
motion in a neighborhood of $\lambda_{0}$, hence gives a topological conjugacy
on the Julia sets (see for instance [29]). On the other hand, unlike in
dimension 1 or in specific families [1], we do not know whether the presence
of a hyperbolic parameter in a stable family implies that all parameter are
hyperbolic, hence the propagation of the equilibrium states in a stable family
is not clear even in the presence of hyperbolic parameters. More generally,
the theory of stability in higher dimensions is known to have a number of
different features with respect to the one-dimensional counterpart, see for
instance [2, 14, 15, 25, 39].

In this note, we give the following answer to the above problem.

######  Main Theorem.

Let $\\{f_{\lambda}:\mathbb{P}^{k}\to\mathbb{P}^{k}\\}_{\lambda\in M}$ be a
stable family of endomorphisms of $\mathbb{P}^{k}$ of algebraic degree $d\geq
2$. Consider a parameter $\lambda_{0}\in M$ and a Borel function
$\psi:\mathbb{P}^{k}\to\mathbb{R}$ satisfying $\sup\psi-\inf\psi<\log d$.
Assume there exists an (respectively a unique) equilibrium state $\nu$ for the
system $(J_{\lambda_{0}},f_{\lambda_{0}})$ with respect to $\psi$ and let
$\mathcal{M}$ be a $(\lambda_{0},\nu)$-web. Then

  1. (i)

the $(\lambda_{0},\nu)$-web $\mathcal{M}$ is unique;

  2. (ii)

for every parameter $\lambda\in M$, there exists a Borel function
$\psi_{\lambda}:\mathbb{P}^{k}\to\mathbb{R}$ such that
$\nu_{\lambda}:=(p_{\lambda})_{*}\mathcal{M}$ is a (resp. the unique)
equilibrium state for the system $(J_{\lambda},f_{\lambda})$ with respect to
$\psi_{\lambda}$.

The condition $\sup\psi-\inf\psi<\log d$ implies that the measure-theoretic
entropy of $\nu$ satisfies $h_{\nu}(f_{\lambda_{0}})>(k-1)\log d$ (hence, in
particular, it has strictly positive Lyapunov exponents [20, 26], and falls
within the scope of [13]). The existence of a $(\lambda_{0},\nu)$-web then
follows from [13, Corollary 1.5]. The proof of the above result relies on the
existence of a $\mathcal{M}$-full-measure subset of $\mathcal{J}$ of elements
whose graphs do not intersect and is given in Section 2. In Section 3 we also
give an interpretation of $\mathcal{M}$ as an equilibrum state for the
dynamical system $(\mathcal{J},\mathcal{F})$, see Proposition 3.1. Following
this interpretation, it is also natural to study the distribution of repelling
graphs with respect to $\mathcal{M}$, which corresponds to the distribution of
repelling points for the systems $(\mathcal{J},\mathcal{F})$. This is done in
[12].

### Acknowledgments

The authors would like to thank Fabrizio Bianchi for helpful discussions
during the preparation of this paper, and Yûsuke Okuyama and Gabriel Vigny for
their very useful comments.

##  2\. Proof of the Main Theorem

Given a holomorphic family
$\\{f_{\lambda}:\mathbb{P}^{k}\to\mathbb{P}^{k}\\}_{\lambda\in M}$ of
endomorphisms of $\mathbb{P}^{k}$, observe that for every parameter
$\lambda\in M$, the evaluation map $p_{\lambda}:\mathcal{J}\to J_{\lambda}$ is
continuous, and that $p_{\lambda}\circ\mathcal{F}=f_{\lambda}\circ
p_{\lambda}$. The map $p_{\lambda}$ may turn out not to be injective. This is
due to the possible existence of some crossings between graphs in
$\mathcal{J}$. The notion of lamination, introduced in [6], will be central in
our approach.

######  Definition 2.1.

A _lamination_ is a Borel subset $\mathcal{L}\subset\mathcal{J}$ such that for
any maps $\gamma_{1},\gamma_{2}\in\mathcal{L}$, we have

$\Gamma_{\gamma_{1}}\cap\Gamma_{\gamma_{2}}\neq\emptyset\implies\gamma_{1}=\gamma_{2},$

where $\Gamma_{\gamma}\subset M\times\mathbb{P}^{k}$ denotes the graph of an
element $\gamma\in\mathcal{J}$. If $\mathcal{M}$ is a Borel probability
measure on $\mathcal{J}$, we say that $\mathcal{L}$ is an _$\mathcal{M}$
-lamination_ if, furthermore, $\mathcal{M}(\mathcal{L})=1$.

Given an $\mathcal{F}$-invariant lamination $\mathcal{L}\subset\mathcal{J}$,
one can consider the restriction
$p_{\lambda}|_{\mathcal{L}}\>:\mathcal{L}\to\mathcal{L}_{\lambda}$ of the
application $p_{\lambda}$, where
$\mathcal{L}_{\lambda}:=p_{\lambda}(\mathcal{L})$. The map
$p_{\lambda}|_{\mathcal{L}}$ is automatically $1$-to-$1$. Yet, observe that
$p_{\lambda}|_{\mathcal{L}}^{-1}$ may not be continuous.

######  Lemma 2.2.

Let $\mathcal{L}\subset\mathcal{J}$ be a lamination, and fix a parameter
$\lambda\in M$. Then, for any Borel subset $\mathcal{A}\subset\mathcal{J}$, we
have

$\mathcal{L}\cap\mathcal{A}=\mathcal{L}\cap
p_{\lambda}^{-1}(p_{\lambda}(\mathcal{L}\cap\mathcal{A})).$

In particular, for any probability measure $\nu$ on $\mathbb{P}^{k}$, there
exists at most one $(\lambda,\nu)$-web $\mathcal{M}$ such that
$\mathcal{M}(\mathcal{L})=1$.

###### Proof.

Let $\mathcal{A}\subset\mathcal{J}$ be a Borel set. The direct inclusion is
immediate. For the converse one, take $\gamma\in\mathcal{L}$ such that
$\gamma(\lambda)\in p_{\lambda}(\mathcal{L}\cap\mathcal{A})$. Take
$\gamma^{\prime}\in\mathcal{L}\cap\mathcal{A}$ such that
$\gamma(\lambda)=\gamma^{\prime}(\lambda)$. Since $\mathcal{L}$ is a
lamination, we have $\gamma=\gamma^{\prime}$. In particular, we have
$\gamma\in\mathcal{L}\cap\mathcal{A}$, which proves the desired equality.

Let now $\mathcal{M}$ be as in the statement. We have

(2.1)
$\mathcal{M}(\mathcal{A})=\mathcal{M}(\mathcal{L}\cap\mathcal{A})=\mathcal{M}(\mathcal{L}\cap
p_{\lambda}^{-1}(p_{\lambda}(\mathcal{L}\cap\mathcal{A})))=\nu(p_{\lambda}(\mathcal{L}\cap\mathcal{A})).$

The last expression does not depend on the choice of the $(\lambda,\nu)$-web
$\mathcal{M}$. The proof is complete. ∎

From now on, we will assume that the family
$\\{f_{\lambda}:\mathbb{P}^{k}\to\mathbb{P}^{k}\\}_{\lambda\in M}$ is stable.
Our goal is to propagate the class of equilibrium states. To do this, we will
make use of the $\mathcal{M}$-laminations. Given a lamination $\mathcal{L}$,
for every $\lambda_{0},\lambda\in M$ we can consider the bijection
$\Phi_{\lambda_{0},\lambda}^{\mathcal{L}}:=p_{\lambda}\circ
p_{\lambda_{0}}|_{\mathcal{L}}^{-1}:\mathcal{L}_{\lambda_{0}}\to\mathcal{L}_{\lambda}$.

######  Proposition 2.3.

Let $\\{f_{\lambda}:\mathbb{P}^{k}\to\mathbb{P}^{k}\\}_{\lambda\in M}$ be a
stable family of holomorphic endomorphisms of $\mathbb{P}^{k}$ of algebraic
degree $d\geq 2$. Let $\lambda_{0}\in M$ and $\nu$ be an ergodic
$f_{\lambda_{0}}$-invariant measure with $h_{\nu}(f_{\lambda_{0}})>(k-1)\log
d$. Then

  1. (i)

there exists a unique $(\lambda_{0},\nu)$-web $\mathcal{M}$. Moreover,
$\mathcal{M}$ is ergodic and for any parameter $\lambda\in M$, we have
$h_{\mathcal{M}}(\mathcal{F})=$ $h_{\nu_{\lambda}}(f_{\lambda})$, where
$\nu_{\lambda}:=(p_{\lambda})_{*}\mathcal{M}$;

  2. (ii)

if $\mathcal{L}$ is an $\mathcal{F}$-invariant $\mathcal{M}$-lamination and
$\psi:J_{\lambda_{0}}\to\mathbb{R}$ is a Borel measurable function, then for
any parameter $\lambda\in M$, we have
$P_{\nu_{\lambda}}(\psi_{\lambda})=P_{\nu}(\psi)$, where
$\psi_{\lambda}:=\psi\circ(\Phi_{\lambda_{0},\lambda}^{\mathcal{L}})^{-1}\cdot\mathds{1}_{\mathcal{L}_{\lambda}}$.

######  Remark 2.4.

The web $\mathcal{M}$ in Proposition 2.3 is actually _acritical_ (see [13,
Definition 1.3] and [13, Theorem 1.4]). We do not stress the acriticality in
this paper, since the existence of an $\mathcal{M}$-lamination guarantees this
condition, and we only work with such laminations.

######  Remark 2.5.

If we are allowed to shrink $M$ to some neighbourhood of $\lambda_{0}$
(depending on $\nu$), the assumption on $\nu$ can be weakened as in [13],
namely it is enough that $\nu$ is a measure on the Julia set with strictly
positive Lyapunov exponents and not giving mass to the post-critical set of
$f_{\lambda_{0}}$, see [13, Theorem 1.4]. The fact that any $\nu$ satisfies
these conditions is a consequence of [20, 26] and [19, 22].

###### Proof of Proposition 2.3.

We first prove (i). By [13, Corollary 1.5] there exists an ergodic
$(\lambda_{0},\nu)$-web $\mathcal{M}$ and an $\mathcal{F}$-invariant
$\mathcal{M}$-lamination $\mathcal{L}$. To show the uniqueness property,
consider another $(\lambda_{0},\nu)$-web $\mathcal{M}^{\prime}$. We have
$\mathcal{M}^{\prime}(\mathcal{L})=\nu(\mathcal{L}_{\lambda_{0}})=\mathcal{M}(\mathcal{L})=1$.
Lemma 2.2 then yields $\mathcal{M}=\mathcal{M}^{\prime}$. Take $\lambda\in M$.
The application
$p_{\lambda}|_{\mathcal{L}}:\mathcal{L}\to\mathcal{L}_{\lambda}$ is
continuous, hence Borel measurable. The space $\mathcal{J}$ being Polish, as a
closed subspace of the space $\mathcal{O}(M,\mathbb{P}^{k})$ of all
holomorphic functions from $M$ to $\mathbb{P}^{k}$, since
$\mathcal{L}\subset\mathcal{J}$ is a Borel subset, the inverse application
$p_{\lambda}|_{\mathcal{L}}^{-1}$ is automatically Borel measurable (see for
example [30, Theorem 3 §39.V]). Since
$\nu_{\lambda}=(p_{\lambda})_{*}\mathcal{M}$, the two measurable systems
$(\mathcal{L},\mathcal{F},\mathcal{M})$ and
$(\mathcal{L}_{\lambda},f_{\lambda},\nu_{\lambda})$ are then conjugated by
$p_{\lambda}$. The conclusion follows from the fact that the measure-theoretic
entropy is invariant under measurable conjugacy between subsets of full
measure.

We now prove (ii). Fix $\lambda\in M$. The above shows that the measurable
isomorphism $\Phi_{\lambda_{0},\lambda}^{\mathcal{L}}$ conjugates the two
systems $(\mathcal{L}_{\lambda_{0}},f_{\lambda_{0}},\nu)$ and
$(\mathcal{L}_{\lambda},f_{\lambda},\nu_{\lambda})$, and that
$h_{\nu}(f_{\lambda_{0}})=h_{\nu_{\lambda}}(f_{\lambda})$. It remains to show
that $\int\psi\nu=\int\psi_{\lambda}\nu_{\lambda}$. Since
$(\Phi_{\lambda_{0},\lambda}^{\mathcal{L}})_{*}\nu_{\lambda}=\nu$ and
$(\Phi_{\lambda_{0},\lambda}^{\mathcal{L}})^{*}\psi_{\lambda}=\psi$ on
$\mathcal{L}_{\lambda_{0}}$, we have

$\displaystyle\int\psi_{\lambda}\nu_{\lambda}$
$\displaystyle=\int_{\mathcal{L}_{\lambda}}\psi_{\lambda}\nu_{\lambda}=\int_{\Phi_{\lambda_{0},\lambda}^{\mathcal{L}}(\mathcal{L}_{\lambda_{0}})}\psi_{\lambda}\nu_{\lambda}$
$\displaystyle=\int_{\mathcal{L}_{\lambda_{0}}}(\Phi_{\lambda_{0},\lambda}^{\mathcal{L}})^{*}\psi_{\lambda}\cdot(\Phi_{\lambda_{0},\lambda}^{\mathcal{L}})^{*}\nu_{\lambda}=\int_{\mathcal{L}_{\lambda_{0}}}\psi\nu=\int\psi\nu.$

The proof is complete. ∎

######  Remark 2.6.

Since the mapping $\Phi_{\lambda_{0},\lambda}^{\mathcal{L}}$ is a measurable
conjugacy, the proof of Proposition 2.3 shows that the measure $\nu_{\lambda}$
inherits the statistical properties that $\nu$ may have, such as (exponential)
mixing, Central Limit Theorem, Almost Sure Invariant Principle or Large
Deviation Principle (see [10, 11, 27, 38, 40] for large classes of examples of
measures satisfying these properties). On the other hand, as the space of
Hölder functions is in general not preserved by the measurable conjugacy, the
space of observables associated with each $\nu_{\lambda}$ will in general
depend on $\lambda$.

###### Proof of the Main Theorem.

Recall that $\mu_{\lambda}$ denotes the measure of maximal entropy of the
endomorphism $f_{\lambda}$. Up to adding to $\psi$ a constant, for simplicity,
we will assume that $\inf\psi=0$. Then, by the definition of the equilibrium
state $\nu$, we have

(2.2) $h_{\nu}(f_{\lambda_{0}})+\int\psi\nu=P_{\nu}(\psi)\geq
P_{\mu_{\lambda_{0}}}(\psi)=k\log d+\int\psi\mu_{\lambda_{0}}\geq k\log d.$

By assumption, we have $\int\psi\nu<\log d$. Hence, (2.2) yields
$h_{\nu}(f_{\lambda_{0}})>(k-1)\log d$. Proposition 2.3 then yields a unique
$(\lambda_{0},\nu)$-web $\mathcal{M}$, which proves assertion (i).

Denote $\nu_{\lambda}=(p_{\lambda})_{*}\mathcal{M}$. Observe that Proposition
2.3 also gives that $h_{\nu_{\lambda}}(f_{\lambda})>(k-1)\log d$ for any
parameter $\lambda\in M$. For every parameter $\lambda\in M$, consider the
Borel function $\psi_{\lambda}:\mathbb{P}^{k}\to\mathbb{R}$ as defined in
Proposition 2.3. To prove assertion (ii), consider a parameter $\lambda\in M$
and assume that there exists an ergodic $f_{\lambda}$-invariant measure
$\omega_{\lambda}$ such that

(2.3) $P_{\omega_{\lambda}}(\psi_{\lambda})\geq
P_{\nu_{\lambda}}(\psi_{\lambda}).$

Up to replacing $\omega_{\lambda}$ by $\mu_{\lambda}$ if
$P_{\omega_{\lambda}}(\psi_{\lambda})<P_{\mu_{\lambda}}(\psi_{\lambda})$, we
can also assume that $h_{\omega_{\lambda}}(f_{\lambda})>(k-1)\log d$.

Proposition 2.3 then yields a $(\lambda,\omega_{\lambda})$-web
$\mathcal{M}^{\prime}$ and we have

(2.4) $P_{\omega}(\psi)=P_{\omega_{\lambda}}(\psi_{\lambda})\geq
P_{\nu_{\lambda}}(\psi_{\lambda})=P_{\nu}(\psi),$

where $\omega:=(p_{\lambda_{0}})_{*}\mathcal{M}^{\prime}$. As $\nu$ is an
equilibrium state with respect to $\psi$, the inequality in (2.4) is actually
an equality. Thus, $\nu_{\lambda}$ is an equilibrium state with respect to
$\psi_{\lambda}$.

Assume now that $\nu$ is unique. Then, (2.4) shows that $\omega$ as above is
an equilibrium state with respect to $\psi$ as well. By the uniqueness
assumption, we have $\omega=\nu$ and both $\mathcal{M}$ satisfying
$\mathcal{M}^{\prime}$ are $(\lambda_{0},\nu)$-webs and
$\mathcal{M}^{\prime}(\mathcal{L})=\mathcal{M}^{\prime}(p_{\lambda_{0}}^{-1}(\mathcal{L}_{\lambda_{0}})=\omega(\mathcal{L}_{\lambda_{0}})=\nu(\mathcal{L}_{\lambda_{0}})=\mathcal{M}(\mathcal{L})=1$.
Lemma 2.2 then yields $\mathcal{M}=\mathcal{M}^{\prime}$. In particular, one
has $\omega_{\lambda}=\nu_{\lambda}$, which ends the proof. ∎

######  Remark 2.7.

As we have seen above, the condition $\sup\psi-\inf\psi<\log d$ guarantees
that we have $h_{\nu}(f_{\lambda_{0}})>(k-1)\log d$. This, thanks to
Proposition 2.3, is one of the main ingredients in the proof of the Main
Theorem. Let us assume that we are just given an equilibrium state $\nu$ with
respect to some bounded Borel function $\psi$ (with no further conditions)
satisfying $h_{\nu}(f_{\lambda_{0}})>(k-1)\log d$. Then, with the notations as
in the proof of the Main Theorem, Proposition 2.3 still implies that
$P_{\nu_{\lambda}}(\psi_{\lambda})=P_{\nu}(\psi)$. On the other hand, in this
case, a priori $\nu_{\lambda}$ may not be an equilibrium state of
$\psi_{\lambda}$. Indeed, following the proof of the Main Theorem, let
$\omega_{\lambda}$ be an $f_{\lambda}$-invariant measure satisfying (2.3).
Even if $P_{\omega_{\lambda}}(\psi_{\lambda})\geq k\log d$, since we do not
have any condition on $\psi$, the integral
$\int\psi_{\lambda}\omega_{\lambda}$ can be large enough so that
$h_{\omega_{\lambda}}(f_{\lambda})\leq(k-1)\log d$. Hence, in this case, we
cannot apply Proposition 2.3 to obtain (2.4).

##  3\. The web $\mathcal{M}$ is an equilibrium state

In this section, we observe that the system $(\mathcal{J},\mathcal{F})$ itself
is well-suited for the thermodynamical formalism. To this purpose, an
additional compactness assumption is necessary. Define an _equilibrium state_
for the system $(\mathcal{J},\mathcal{F})$ with respect to a Borel bounded
function $\Psi:\mathcal{J}\to\mathbb{R}$ as an ergodic $\mathcal{F}$-invariant
compactly supported probability measure $\mathcal{M}$ on $\mathcal{J}$
maximizing the functional
$P_{\mathcal{M}}(\Psi)=h_{\mathcal{M}}(\mathcal{F})+\int\Psi\mathcal{M}$. We
have to ensure that the set of ergodic $\mathcal{F}$-invariant compactly
supported measures on $\mathcal{J}$ is not empty. In general, one may need to
shrink $M$ so that $\mathcal{J}$ becomes compact. Assuming stability, that is
simply guaranteed by the existence of an equilibrium web as in [6].

In this framework, given an equilibrium state $\nu$ with respect to a weight
function $\psi$ at a parameter $\lambda_{0}$, we construct a naturally defined
weight function $\Psi$ on $\mathcal{J}$, compatible with the measurable
holomorphic motion, with respect to which the $(\lambda_{0},\nu)$-web
$\mathcal{M}$ from the Main Theorem is an equilibrium state. We show as well
that the uniqueness property holds. In particular, the equilibrium web of
Berteloot-Bianchi-Dupont is the unique measure of maximal entropy for the
dynamical system $(\mathcal{J},\mathcal{F})$.

######  Proposition 3.1.

Under the hypothesis of the Main Theorem, there exists a Borel bounded
function $\Psi:\mathcal{J}\to\mathbb{R}$ such that $\Psi=\psi\circ
p_{\lambda_{0}}$ $\mathcal{M}$-a.e. and $P_{\mathcal{M}}(\Psi)=P_{\nu}(\psi)$.
Moreover, the $(\lambda_{0},\nu)$-web $\mathcal{M}$ is an (resp. the unique)
equilibrium state for the system $(\mathcal{J},\mathcal{F})$ with respect to
$\Psi$.

###### Proof.

As in the proof of the Main Theorem, up to adding to $\psi$ a constant, for
simplicity, we will assume that $\inf\psi=0$. As in the proof of Proposition
2.3, let $\mathcal{L}$ be a $\mathcal{M}$-lamination (provided by [13]) and
define $\Psi:=\psi\circ
p_{\lambda_{0}}|_{\mathcal{L}}\cdot\mathds{1}_{\mathcal{L}}$. We first prove
the equality $P_{\mathcal{M}}(\Psi)=P_{\nu}(\psi_{\lambda_{0}})$. By
Proposition 2.3 we have
$h_{\mathcal{M}}(\mathcal{F})=h_{\nu}(f_{\lambda_{0}}).$ So, it is enough to
prove that $\int\Psi\mathcal{M}=\int\psi\nu$. Since
$\mathcal{M}(\mathcal{L})=1$, we have

$\int\Psi\mathcal{M}=\int_{\mathcal{L}}\psi\circ
p_{\lambda_{0}}\mathcal{M}=\int_{\mathcal{L}}(p_{\lambda_{0}})^{*}(\psi\cdot(p_{\lambda_{0}})_{*}\mathcal{M})=\int_{\mathcal{L}_{\lambda_{0}}}\psi\nu=h_{\nu}(f_{\lambda_{0}}),$

which gives the desired equality.

We now prove the last assertion. Assume there exists another ergodic
$\mathcal{F}$-invariant Borel probability measure $\mathcal{M}^{\prime}$ with
compact support on $\mathcal{J}$ such that

$P_{\mathcal{M}^{\prime}}(\Psi)\geq P_{\mathcal{M}}(\Psi).$

Define $\omega:=(p_{\lambda_{0}})_{*}\mathcal{M}^{\prime}$, so that the
measure $\mathcal{M}^{\prime}$ is a $(\lambda_{0},\omega)$-web. Observe that
Proposition 2.3 yields

(3.1) $P_{\omega}(\psi)=P_{\mathcal{M}^{\prime}}(\Psi)\geq
P_{\mathcal{M}}(\Psi)=P_{\nu}(\psi).$

As $\nu$ is an equilibrium state with respect to $\psi$, we have that
$P_{\nu}(\psi)\geq P_{\omega}(\psi)$. Hence (3.1) yields
$P_{\mathcal{M}^{\prime}}(\Psi)=P_{\mathcal{M}}(\Psi)$. This shows that
$\mathcal{M}$ is an equilibrium state for $\Psi$. Moreover, if $\nu$ is the
unique equilibrium state, then $\omega=\nu$. Then Proposition 2.3 (i) implies
that $\mathcal{M}^{\prime}=\mathcal{M}$. The proof is complete. ∎

## References

  * [1] Matthieu Astorg and Fabrizio Bianchi, _Hyperbolicity and Bifurcations in holomorphic families of polynomial skew products_ , American Journal of Mathematics, 145.3 (2023), 861-898.
  * [2] Matthieu Astorg and Fabrizio Bianchi, _Higher bifurcations for polynomial skew products_ , Journal of Modern Dynamics 18 (2022), 69-99.
  * [3] Giovanni Bassanelli and François Berteloot, Bifurcation currents in holomorphic dynamics on $\mathbb{P}^{k}$, Journal für die reine und angewandte Mathematik (Crelle’s Journal) 606 (2007), 201-235.
  * [4] François Berteloot, _Les cycles répulsifs bifurquent en chaîne,_ Annali di Matematica Pura ed Applicata, 197.2 (2018), 517-520.
  * [5] François Berteloot and Fabrizio Bianchi, _Stability and bifurcations in projective holomorphic dynamics_ , Dynamical systems, Banach Center Publications, Polish Academy of Sciences Institute of Mathematics, Warsaw, 115 (2018), 37–71.
  * [6] François Berteloot, Fabrizio Bianchi, and Christophe Dupont, _Dynamical stability and Lyapunov exponents for holomorphic endomorphisms of $\mathbb{P}^{k}$,_ Annales Scientifiques de l’ENS, 51.1 (2018), 215-262.
  * [7] François Berteloot and Maxence Brévard, _Ramification current, post-critical normality and stability of holomorphic endomorphisms of $\mathbb{P}^{k}$_, Fundamenta Mathematicae, 263 (2023), 253-270.
  * [8] Fabrizio Bianchi, _Motions of Julia sets and dynamical stability in several complex variables_ , PhD Thesis, Université Toulouse III Paul Sabatier and Università di Pisa (2016).
  * [9] Fabrizio Bianchi, Misiurewicz parameters and dynamical stability of polynomial-like maps of large topological degree, Mathematische Annalen, 373 (2019), 901–928.
  * [10] Fabrizio Bianchi and Tien-Cuong Dinh, Equilibrium states of endomorphisms of $\mathbb{P}^{k}$ I: existence and properties, _Journal de mathématiques pures et appliquées_ , 172 (2023), 164-201.
  * [11] Fabrizio Bianchi and Tien-Cuong Dinh, _Equilibrium states of endomorphisms of $\mathbb{P}^{k}$: spectral gap and limit theorems_, Geometric and Functional Analysis (GAFA), to appear (2024).
  * [12] Fabrizio Bianchi and Maxence Brévard, _Holomorphic motions of weighted periodic points_ , arXiv preprint arXiv:2307.12734 (2023).
  * [13] Fabrizio Bianchi and Karim Rakhimov, _Strong probabilistic stability in holomorphic families of endomorphisms of $\mathbb{P}^{k}$ and polynomial-like maps_, International Mathematics Research Notices, to appear (2024).
  * [14] Fabrizio Bianchi and Johan Taflin, _Bifurcations in the elementary Desboves family_ , 145.10 (2017), 4337-4343.
  * [15] Sébastien Biebler, _Lattès maps and the interior of the bifurcation locus_ , Journal of Modern Dynamics, 15 (2019), 95-130.
  * [16] Maxence Brévard, _Étude des composantes de stabilité pour les familles d’endomorphismes holomorphes des espaces projectifs complexes_ , PhD thesis, Université Toulouse III Paul Sabatier (2023).
  * [17] Jean-Yves Briend and Julien Duval, _Deux caractérisations de la mesure d’équilibre d’un endomorphisme de $\mathbb{C}\mathbb{P}^{k}$_, Publications Mathématiques de l’IHÉS, 93 (2001), 145-159, and _erratum_ in 109 (2009), 295-296.
  * [18] Henry De Thélin, _Un phénomène de concentration de genre_ , Mathematische Annalen 332 (2005), 483-498.
  * [19] Henry De Thélin, Sur la construction de mesures selles, _Annales de l’Institut Fourier_ , 56.2 (2006), 337-372.
  * [20] Henry De Thélin, _Sur les exposants de Lyapounov des applications méromorphes_ , Inventiones Mathematicae, 172 (2008), 89–116.
  * [21] Laura DeMarco, _Dynamics of rational maps: Lyapunov exponents, bifurcations, and capacity_ , Mathematische Annalen, 326.1 (2003), 43–73.
  * [22] Tien-Cuong Dinh, _Attracting current and equilibrium measure for attractors on $\mathbb{P}^{k}$_, Journal of Geometric Analysis, 17.2 (2007), 227-244.
  * [23] Tien-Cuong Dinh and Nessim Sibony, _Dynamics in several complex variables: endomorphisms of projective spaces and polynomial-like mappings_ , Holomorphic dynamical systems, Lecture Notes in Mathematics 1998, Springer, Berlin (2010), 165–294.
  * [24] Romain Dujardin, _Approximation des fonctions lisses sur certaines laminations_ , Indiana University Mathematics Journal, 55 (2006), 579-592.
  * [25] Romain Dujardin, _Non density of stability for holomorphic mappings on $\mathbb{P}^{k}$_, Journal de l’École Polytechnique, 4 (2017), 813-843.
  * [26] Christophe Dupont, _Large entropy measures for endomorphisms of $\mathbb{CP}^{k}$_, Israel Journal of Mathematics, 192 (2012), 505–533.
  * [27] Christophe Dupont, _Bernoulli coding map and almost sure invariance principle for endomorphisms of $\mathbb{P}^{k}$_, Probability theory and related fields 146.3-4 (2010), 337-359.
  * [28] John Erik Fornaess and Nessim Sibony, Complex dynamics in higher dimensions, _Complex potential theory (Montreal, PQ, 1993)_ , NATO Advanced Science Institute Series C: Mathematical and Physical Sciences, Kluwer Academic Publichers, Dordrecht, 439 (1994), 131–186.
  * [29] Mattias Jonsson, _Holomorphic motions of hyperbolic sets_ , Michigan Mathematical Journal, 45 (1998), 409-415.
  * [30] Kazimierz Kuratowski, _Topology_ , Volume I, Academic Press, New York (1966).
  * [31] Mikhail Lyubich, _Some typical properties of the dynamics of rational mappings_ , Russian Mathematical Surveys, 38.5 (1983), 154–155.
  * [32] Mikhail Lyubich, _Entropy properties of rational endomorphisms of the Riemann sphere_ , Ergodic Theory and Dynamical Systems, 3 (1983), 351-385.
  * [33] Ricardo Mañé, Paulo Sad, and Dennis Sullivan, _On the dynamics of rational maps_ , Annales Scientifiques de l’ENS (4), 16.2 (1983), 193–217.
  * [34] Ngoc-mai Pham, _Lyapunov exponents and bifurcation current for polynomial-like maps_ , Preprint arXiv:0512557 (2005).
  * [35] Feliks Przytycki, _Hausdorff dimension of harmonic measure on the boundary of an attractive basin for a holomorphic map_ , Inventiones mathematicae, 80.1 (1985), 161–179.
  * [36] Feliks Przytycki and Mariusz Urbański _Conformal fractals: ergodic theory methods_ , Volume 371, Cambridge University Press (2010).
  * [37] Nessim Sibony, Seminar talk at Orsay, October (1981).
  * [38] Michał Szostakiewicz, Mariusz Urbański, and Anna Zdunik, _Stochastics and thermodynamics for equilibrium measures of holomorphic endomorphisms on complex projective spaces_ , Monatshefte für Mathematik, 174.1 (2014), 141-162.
  * [39] Johan Taflin, _Blenders near polynomial product maps of $\mathbb{C}^{2}$_, Journal of the European Mathematical Society, 23 (2021), 3555-3589.
  * [40] Mariusz Urbański and Anna Zdunik, _Equilibrium measures for holomorphic endomorphisms of complex projective spaces_ , Fundamenta Mathematicae 220 (2013), 23-69.