Diffusion-based image editing techniques frequently require the user to
provide an input map in order to determine which parts of the image are to be
edited and which are to be left untouched. (Hertz et al., 2023) provide a
method to perform localized edits based on text only, without the need for
input maps. The architecture itself follows the standard text-to-image
diffusion template, but it operates on pixels rather than in latent space.
Also, rather than one diffusion process, it employs two: one for generating an
image from the original text prompt and one for generating an edited image
from an edited text prompt. In addition, cross-attention is used to relate the
text input with specific pixels in the image. Edits are made by fusing the
attention maps from the original and the edited diffusion process. Three edit
functions are considered for this, based on the kind of edit that the user
wants to make: a word swap edit, a prompt refinement edit and an emphasis
edit. The system also has a process to obtain an edit mask without user input
when localized edits are specifically requested. The method, in its main
version, uses a pre-trained Imagen (Saharia et al., 2022) model and does not
require additional model re-training. Evaluation data is produced based on
pre-determined text edit templates and the used pre-trained model. Comparisons
are run against: the VQGAN (Esser et al., 2021a) with CLIP, Text2Live (Bar-Tal
et al., 2022) \- both text-guided, as well as Blended Diffusion (Avrahami et
al., 2022) and GLIDE (Nichol et al., 2022) \- both inpainting-mask-guided.
Qualitative evaluations suggest that the proposed method is able to edit
images based on text edits alone, and that the results are visually appealing
and more consistent than in the compared approaches. A user study is run to
assess the generated samples on three axes: (1) background and structure
preservation, (2) alignment with text, and (3) realism. The study compares the
proposed method against VQGAN with CLIP and Text2Live and concludes that users
prefer the proposed system to the other methods for all three aspects.

Within the diffusion-based image editing realm, (Brooks et al., 2023) take an
approach bootstrapping from the results of prior models. They use the Prompt-
to-Prompt model (Hertz et al., 2023) and GPT-3 (Brown et al., 2020) to
generate a dataset of over 450,000 training examples, where each example
consists of the original image, an image modified by Prompt-to-Prompt, and the
corresponding text edit prompt. The unique characteristic of this dataset is
that the conditioning text is now an edit instruction, not a description of
the desired edited image. An architecture largely adhering to the text-to-
image diffusion template is proposed, with Stable Diffusion being the core
model (Rombach et al., 2022), and a modification in the form of the inclusion
of not only text but also image conditioning information in the reverse
diffusion process. This extension is necessary to handle the setup where an
image is provided together with an edit text. Apart from that, the method
includes classifier-free guidance (Ho and Salimans, 2021). The results of
training the model are qualitatively compared with several baselines: SDEdit
(Meng et al., 2022), Text2Live (Bar-Tal et al., 2022) and Prompt-to-Prompt
(Hertz et al., 2023). These comparisons suggest that the proposed method has
significant advantages over the other ones regarding its ability to generate
images consistent with the edit instructions and consistent in their internal
structure. Apart from that, it is the only method which is able to operate
directly on edit prompts. Quantitative comparisons with SDEdit and Prompt-to-
Prompt on metrics measuring image consistency and edit quality show that for
the same level of image consistency, the proposed method achieves higher
levels of edit quality than the other approaches. In particular, it
outperforms Prompt-to-Prompt, which has been used in its training procedure.

The possibilities of text-only image editing models are investigated further
by (Mokady et al., 2023). In particular, their focus is on applying editing
models to real-world images rather than to ones generated by a diffusion
model. To this end they propose an inversion procedure where the forward
diffusion process is reversed by DDIM inversion (Song et al., 2021), (Dhariwal
and Nichol, 2021), however, this DDIM inversion is treated as a pivot (Roich
et al., 2021) around which the actual optimization procedure operates. In
particular, the reverse process utilizes classifier-free guidance (Ho and
Salimans, 2021) where the unconditional null text embedding is optimized and
the final predicted latent code for a given step is pushed toward the pivot,
i.e. the representation from the DDIM inversion process. Even though the
approach significantly modifies the text-to-image diffusion template, the core
of the vanilla method is still present in the form of a Stable Diffusion model
(Rombach et al., 2022) which is used as the backbone of the proposed method.
The actual editing process uses Prompt-to-Prompt, without retraining this
model. Experimental results on a subset of the MS COCO evaluation set show
that the model is able to successfully reconstruct images beating a number of
baselines, such as the DDIM inversion process itself and the encoder-decoder
part of Stable Diffusion. For the image editing task itself, qualitative
assessment on images used by (Bar-Tal et al., 2022) shows that the method
generates appealing and consistent imagery relative to other text-editing
approaches: VQGAN (Esser et al., 2021a) with CLIP (Radford et al., 2021),
SDEdit (Meng et al., 2022) and Text2Live (Bar-Tal et al., 2022). A user study
finds the proposed method outperforming these approaches in terms of fidelity
to both the original image and the text edit. An additional user study on
ImageNet suggests that the proposed model outperforms SDEdit, Text2Live and
FlexIT (Couairon et al., 2022) in terms of the accuracy of edits, but not
necessarily in terms of similarity to the original image. Qualitative tests
also show similar results for a comparison with inpainting-mask-based methods:
Stable Diffusion (Rombach et al., 2022) itself, Blended Diffusion (Avrahami et
al., 2022) and GLIDE (Nichol et al., 2022).

A slightly different take on diffusion-based image editing formulates it as a
task of editing real images but without the need for a priori specification of
editing types and without the need for synthetically generated images. At its
core, the method uses the vanilla text-to-image diffusion template, albeit it
does not specifically require it to be a latent diffusion model. Apart from
that, the optimization and inference procedures introduce significant changes.
Two specific stages are introduced for the traing procedure. In the first one,
the diffusion model is treated as pre-trained and not optimized. An input
image is treated with the diffusion model together with the desired text
description of the image after the edit. This is in contrast to methods such
as (Brooks et al., 2023) where the text is the edit. The text embeddings are
optimized to allow for the reconstruction of the input image. In the second
stage, the text embeddings are frozen, and the pre-trained diffusion model is
fine-tuned with the same reconstruction objective. Inference is performed
based on the input random noise as in standard diffusion approaches, however,
it also includes a linear interpolation between the initial unoptimized text
embedding and the final optimized text embedding. Two specific models are used
as the core diffusion components in experiments: Stable Diffusion (Rombach et
al., 2022) and Imagen (Saharia et al., 2022). The method is applied on a
variety of images collected by the authors from Unsplash and Pixabay - two
stock photography services. Importantly, the whole training procedure involves
orders of magnitude less computing resources than other text-to-image methods.
Qualitative results show that the trained architecture is able to perform a
variety of involved edits, while preserving the internal consistency of the
images. Comparisons with Text2Live (Bar-Tal et al., 2022), SDEdit (Meng et
al., 2022) and DDIB (Su et al., 2023) point to the possibility that the
designed model preserves more consistency in the image than the other methods,
while successfully applying the edit. The Text Editing Benchmark (TEdBench)
dataset is introduced for evaluation purposes. A user study shows that a
significant majority of users prefer images generated by the proposed approach
to images generated by any of the compared methods.

###  2.7. Graphs

Text-to-image generation methods tend to rely on generating the whole image
from the provided text or on manipulating the image based on the text. An
extension in which the input text is not presented in a raw form but rather
structured in a graph representation can also be considered. In this problem
setup, additional information about the relationships between various parts of
the text input can be utilized to obtain more relevant representations and
images of higher quality.

Since graph approaches to text-to-image generation are an active area of
research, there is yet to emerge one template method with which all the
ongoing work could be contrasted. It should be noted, however, that there are
specific components, which emerge within the graph-to-image line of research.
Most notably, the input to the model is a graph, which necessitates the use of
graph convolutional networks (Duvenaud et al., 2015), (Kipf and Welling, 2017)
to obtain a graph representation. In current methods, this representation is
then used to obtain the crops of objects represented in the graph structure.
These crops are processed by a CNN image encoder and fed into a TCNN image
decoder. Depending on the specific approach, the whole system may be VAE- or
GAN-based.

(Li et al., 2019a) construct a GAN-inspired architecture which is able to
generate images from scene graphs in order to enable more control in the
generation process, such as explicit manipulation of specific objects in the
image and not the image as a whole. Based on a scene graph and object crops
delivered as input, the model is able to generate whole images incorporating
the information from both the graph and the crops. Experiments suggest that in
quantitative evaluations this method is able to perform considerably better
than very strong baselines: sg2im (Johnson et al., 2018) and layout2im (Zhao
et al., 2019).

A GAN-based architecture to parse a textual layout and generate an image with
a scene corresponding to this layout is proposed in (Ashual and Wolf, 2019)
and (Ashual and Wolf, 2020). In these works, the scene graph is transformed
into embeddings of each object in the graph, bounding boxes for each object
are created and appearance information is added. Based on this information,
the final image is generated. In qualitative evaluations, the technique
demonstrates the ability to parse and successfully generate relatively
coherent images for progressively complicated layouts. The quantitative
results show that the architecture has the potential to improve upon the
results of sg2im (Johnson et al., 2018) and layout2im (Zhao et al., 2019) \-
previous baselines.

###  2.8. Other

While significant swaths of the text-to-image research area fall into one of
the above-mentioned sub-fields, the presented categories are by no means
exhaustive. There are other lines of research which do not neatly fit into any
one of these categories. Nonetheless, these research directions constitute
interesting avenues, and so, even a glimpse at the possible other approaches
to text-to-image generation could be worthwhile, even if, by definition, there
are no common standard templates for those methods.

An interesting sub-field within text-to-image generation is the task of
generating realistic handwriting from input text. (Kang et al., 2020) approach
the problem of generating handwriting images from text and images of
handwriting. In a GAN-based architecture, three learning objectives are
introduced: to produce realistic images, to imitate handwriting styles
presented in the input images and to be consistent with the provided text
input. The generator is conditioned on the text input and the images of
written words. The produced handwriting is then passed to a discriminator, to
a classifier for writer recognition and to an encoder-decoder RNN for word
recognition. Experiments show that the proposed architecture is able to
generate handwriting which is indistinguishable for humans from real
handwriting.

(Li et al., 2021) address the problem of generating talking heads tailored to
a specific user, based on text input, by designing a two-stage GAN-inspired
architecture. In the first stage, time-aligned text is used to produce
parameters relating to the head pose, upper face and mouth shape
characteristics. This speaker-independent information is then fed into the
second-stage speaker-specific model. This model incorporates an attention
network to generate face landmarks based on the received characteristics. The
obtained landmarks are then used by yet another network to generate videos.
Quantitative results for the sub-modules relating to the upper head and to the
mouth suggest that these components achieve favorable scores on evaluation
metrics relative to strong baselines, for instance, pix2pixHD (Wang et al.,
2018a) and CSG (Sadoughi and Busso, 2021). Qualitative results relative to
approaches including, i.a., Wav2Lip (Prajwal et al., 2020) and NVP (Thies et
al., 2020) also suggest that the proposed method is able to generate talking
head of relatively high fidelity. Additionally, in human evaluations, the
videos generated from this system are evaluated as real in more than 50% of
the cases, which outperforms all the analyzed baselines, relative to over 90%
for real videos.

Rather than simply generate frames from text, (Liu et al., 2022b) have tackled
the related task of generating human gestures from multi-modal input data.
Their architecture has similarities with the GAN video-from-text template, but
it accepts multiple inputs: text, speaker ID, emotion labels, audio data and
images. Each handled input modality has its specialized encoder. On the text
input side, the architecture uses a temporal convolutional encoder rather than
a classic RNN one. On the output side, rather than using the RNN-TCNN decoder,
the model applies separate body and hand decoders and one discriminator. The
system is trained adversarially, with additional loss components for the
reconstruction of body and hand gestures. A new dataset is specifically
constructed for the experiments – the Body-Expression-Audio-Text (BEAT)
dataset. The proposed network outperforms a series of strong baselines such as
MultiContext (Yoon et al., 2020) in quantitative comparisons.

While generating video from text and image editing have been popular research
lines, the related task of video editing is only starting to enjoy increased
attention. (Fu et al., 2022) tackle precisely the problem of editing videos
based on text input. An overview of this task is presented in Figure 10. The
proposed approach shares similarities with both the video-from-text GAN
template and the image editing GAN template. The text input is processed by an
RNN encoder as in the former and the individual input video frames are fed
into a CNN encoder as in the latter. There are significant departures from the
two standard templates, though. The RNN encoder models both the word- and
sentence-level representations. Those representations are handled by a
Transformer network together with the video frame features and spatial
coordinates. This network produces features to be decoded by a frame generator
network, which is similar to the generator in the standard templates. The
generated video frames are submitted to two discriminator networks in a
further departure from the standard methods. One of those discriminators
operates on the frame level and the other one on the level of sequences of
consecutive frames. Three new datasets are built for evaluation purposes:
E-MNIST, E-CLEVR and E-JESTER. Results on these datasets show that, in
quantitative terms, the proposed architecture performs favorably to strong
baselines such as E3D-LSTM (Wang et al., 2019).

Figure 10. Overview of the video editing task. Video frames come from the
FERV39k dataset (Wang et al., 2022).

##  3\. Directions for future research

Cross-modal generation is at a point where it has seen a significant increase
in the interest of the research community, while at the same time a number of
its sub-fields remain relatively weakly covered, particularly when compared
with the most popular one - image captioning. Such a combination of factors
makes cross-modal generation a potentially fruitful area for future research.
In particular, there is significant potential in the less developed areas
which are rapidly getting more attention. This includes text-to-image.

Paired datasets remain the dominant for most cross-modal generation problems.
For tasks linking the text and visual domains, they usually come in the form
of images or videos paired with text descriptions. Both text-to-image and
image-to-text methods rely on such data. The problem with using such datasets
is that there is an inherent limit to how much this approach can scale. In
particular, the construction of large, high-quality datasets of images with
their descriptions requires some form of manual annotation. The problem is
more significant than that, though. For problems of high dimensionality, e.g.
understood as the size of the output space, it is not feasible to construct
large enough datasets in any reasonable time frame. For instance, the output
space for the problem of generating images from text encompasses multiple
possible values for each considered pixel and there are potentially infinitely
many valid pictures for a specific input text. The lack of sufficiently large
aligned datasets puts a natural cap on classic approaches. One remedy might be
the development of methods which do not explicitly rely on paired data.
Examples of this include models based on domain adaptation (Chen et al., 2017)
or weakly-supervised learning (Laina et al., 2019). Such approaches may help
alleviate the problem, however, significantly more work needs to be done in
order to bring the results of methods such as weakly-supervised ones in line
with what is currently possible for fully-supervised models. Also, the
research into such methods tends to focus on the image-to-text area, which
makes it even more important to further work on such methods for text-to-image
as well.

The introduction of additional structure to the latent space of text/visual
architectures has been one of the most fruitful research directions in terms
of the improvement of evaluation metrics. This trend has been visible for
models working specifically with the text/visual data, but also for more
general architectures, in particular, ones which do not necessarily assume
specific modalities, but rather are constructed in ways which allow the
modalities to be swapped without modifying the core architecture. The latter
general approaches proposed in the field of cross-modal generation tend to
rely on latent space decomposition or on a specific learning objective (Shi et
al., 2019), (Sutter et al., 2020), (Mahajan et al., 2020). These approaches
show some promise as far as the quality of the generated samples is concerned.
This notwithstanding, the design of general cross-modal architectures is still
very much an open problem. One possible way forward is to consider further
approaches to latent space decomposition. This in itself is a challenging
problem, as any such design cannot rely on the structure of any specific
modalities, as the general nature of the architecture requires it to be
possible to change the modalities without modifying the architecture. The
methods relying on general latent space decomposition tend to be VAE-based.
This opens yet another possibility - one of proposing GAN- or diffusion-based
architectures geared toward the same general task. This could potentially
result in an improvement in the quality of the generated data, but requires
significant research into training techniques.

An interesting focal point is the possibility of using text data from
different languages. Multilingual cross-modal architectures have enjoyed some
interest (Wang and Xiong, 2021), with attempts to utilize unpaired samples
(Chen et al., 2019). Given the relatively limited nature of hand-made datasets
and the problems with scaling supervised approaches on such data, there is
significant potential in the area focusing on the move from classic paired
supervised training examples to more loosely coupled data points. The
multilingual problem is a natural way to increase the amount of available
data. For instance, for the problem of generating images from a text
description, the inclusion of descriptions in languages other than the target
one could significantly increase the size of the available datasets. The main
problem here is the design of architectures able to handle different
languages. The extent to which such an architecture would have to change
between languages, and how much of it could actually be general, are
interesting research questions.

There has been some effort to incorporate knowledge broader than what can be
learned from the training dataset itself. One particular area of application
is visual dialogue, where external knowledge bases have been proposed (Yang et
al., 2021b), showing that it is possible to rely on such sources of
information. This has, however, been limited to specific cases. An attempt to
apply similar techniques for text-to-image generation is one potential avenue
of research, with the particular challenge being that while in VQA (Visual
Question Answering) the external knowledge might come in the form of text, for
text-to-image other knowledge sources, in particular visual ones, could be
required. The application of text-to-image methods need not be limited to VQA.
Other areas of Visual Reasoning might also benefit from the adoption of
approaches generating images from text, a prominent example being the field of
Abstract Visual Reasoning (Małkiński and Mańdziuk, 2022), (Małkiǹski and
Maǹdziuk, 2023), where the ability to generate visual data might find its use,
for instance, in completing ambiguous sequences.

Text-to-image generation does not have to be the end task. Cross-modal
generation admits various kinds of inter-connectivity between its sub-areas.
For instance, it is possible to train a text-to-image model to solve an
upstream task and use the generated data for a downstream VQA task to improve
results of a VQA model (Yang et al., 2021a), as VQA systems are still
relatively far from human-level performance (Talmor et al., 2021). The
challenge for this specific setup relates to the fact that the images obtained
from a text-to-image architecture are still likely to lack the quality and
diversity of the samples from the actual image dataset. There is a gap as far
as research on improving quality and diversity of images for downstream tasks
is concerned. Text-to-image models could also be on the receiving end of data
augmentation procedures. One example could be a system for the handling of
text data, which is trained to replace parts of a description with their
synonyms. Such a text model could be used as a preprocessing pipeline for a
text-to-image model.

A prominent trend is the idea of iterative improvements to the generated
images (Zhang et al., 2017), (Bodla et al., 2018), (Xu et al., 2018), (El-
Nouby et al., 2019), (Zhu et al., 2019), (Liang et al., 2020), (Cheng et al.,
2020). In such methods, hierarchical output generation pipelines are used. It
would potentially be of interest to utilize a hierarchical input
representation as well and verify whether enforcing input representations at
different levels of the hierarchy to correspond to the improvements introduced
by the subsequent decoders could be beneficial for the system. As an example,
it would be interesting to see whether it is possible to construct a system in
which the first level of input representation is related to specific parts of
the text and higher levels are more abstract, and make these layers of
abstraction inform the generative part of the system, where the first
generated image is conditioned on the abstract representation and the
subsequent ones introduce more specific input representations.

One of the most important developments in text-to-image generation in recent
years is the rapid shift from VAEs and GANs to diffusion. The initial
introduction of diffusion for image generation (Sohl-Dickstein et al., 2015),
along with improvements to the techniques (Ho et al., 2020), (Dhariwal and
Nichol, 2021) and the advent of latent diffusion architectures (Rombach et
al., 2022) has seen a tectonic shift in terms of a transition towards
diffusion. The field of text-to-image generation is no different with
significant progress being made. There is still a lot of potential in
exploring the scaling properties of diffusion models and, in particular, in
investigating the potential to combine existing models in zero-shot settings.

The propagation of diffusion has significantly impacted image editing research
where the standard stable diffusion model (Rombach et al., 2022) has been
shown to be easily adaptable to image-editing tasks. Consequently, diffusion
methods in various shapes and forms (Hertz et al., 2023), (Brooks et al.,
2023), (Mokady et al., 2023), (Kawar et al., 2023) have become the most
prominent line of investigation. The main aspects that would benefit from
being addressed are the consistency of the generated images and the broadening
of the range of potential edits.

Text-to-video generation has seen significant improvements (Marwah et al.,
2017), (Balaji et al., 2019), (Deng et al., 2019), although one specific
aspect of video generation seems to be underrepresented in this research area.
Namely, given a text description and the video stream up to a given moment, it
is possible to generate multiple plausible completions of the video. Further
research in the text-to-video space could explicitly try to model such a range
of possible outcomes. It would also be interesting to see methods to construct
datasets useful for such architectures explicitly handling ambiguity in the
output. Exploring the applicability of diffusion models to this domain seems
like a particularly promising endeavor.

Another avenue of research within the text-to-video domain could involve text-
based video editing. Such methods have already been shown effective for text-
to-image (Nam et al., 2018), (Li et al., 2020b), (Jiang et al., 2021), (Xia et
al., 2021). An extension of this research to handle video streams would
involve changing the input video based on the provided description. Going from
images to videos would require the systems to not only be able to generate
individual modified frames, but also to carry over the information from the
input description and apply it in a temporal manner where not all the required
modifications/instructions would be applied at each time step. This could
constitute a fruitful line of investigation. The same goes for the
incorporation of diffusion methods, especially image-editing models. The
enforcement of temporal consistency for diffusion might be a potential
highlight.

The limited amount of labeled data and the need to extend the learning
algorithms to data without explicit labels has led to a significant amount of
work being done on self-supervised methods (Zolfaghari et al., 2021), (Shi et
al., 2021a), (Zhang et al., 2021a), largely in the contrastive learning
paradigm (Chopra et al., 2005). These methods have shown the applicability of
self-supervised approach in cross-modal learning, but they suffer from the
drawbacks of contrastive approaches, namely the need for hard negative
samples, which limits the scalability of such methods. Because of that, it
might be of interest to investigate non-contrastive self-supervised
approaches, which have already shown some degree of applicability to single
modalities (Zbontar et al., 2021), (Bardes et al., 2022). To the best of our
knowledge, there is no direct research on text-to-image non-contrastive self-
supervised methods. In our opinion, this constitutes a research opportunity
with much potential.

It should be noted that text-to-image generation, in contrast to image-to-
text, faces the significant challenge of modeling images, which currently
requires the modeling of raw pixel values on the output end. A lot of the
research effort relies on including additional structure in the generative
process and there is still a lot of room for the continuation of this kind of
work. Significant research opportunities might also lie in the addition of
structure on the output end of the generative pipeline, namely for the
specific step of image production. Rather than rely on pixels themselves,
intermediate representations tied to graph structures or semantic meaning
could be used before offloading the final work of producing the actual pixels.
The extent to which this is a viable option for image generation remains an
area of potential research.

##  4\. Conclusions

In this work, we have reviewed the state of research in the cross-modal text-
to-image generation domain. To this end, we have analyzed papers published at
8 leading machine learning conferences in the 2016-2022 time frame, along with
additional relevant papers. We have found that the field is still
underdeveloped in terms of publication output relative to the more popular
image-to-text area, yet it has seen both a significant increase in interest
and considerable progress. This combination of factors makes it a particularly
interesting area of future research.

We have proposed to partition the field of text-to-image generation into
several relevant sub-fields. We have discussed: the vanilla text-to-image
area, its iterative and Transformer-based extensions, self-supervised methods,
the text-to-video area, image-editing, graph methods, and the remaining
idiosyncratic approaches. The iterative and Transformer-based methods are the
core of the current advances in text-to-image cross-modal generation. Actual
generative models in the like of VAEs, GANs and diffusion are dominant within
all the analyzed fields. In particular, diffusion methods have recently seen a
substantial increase in the quality of produced samples and in the interest of
the research community.

Text-to-image generation remains less covered than image-to-text methods,
potentially due to the fact that the output space is significantly more
involved than for image-to-text approaches, as the space of potential valid
images given a description is, in principle, significantly larger than the
space of valid descriptions given an image. This complexity of the output
space also results in the application of generative models to this problem, as
non-generative approaches are relatively limited in their ability to produce
convincing visual data. Frequently, these methods are based on diffusion, GANs
or VAEs to produce the actual visual data. Research generally focuses on
topics such as generating images or video from text, with specific problems
including iterative and non-iterative generation, Transformer-based models,
self-supervised methods, image edition and graph inclusion. The fact that
text-to-image generation has received increased attention and shown successful
approaches, but the number of published papers is still relatively limited but
increasing suggests that this is an interesting area of research.

The text-to-image domain has shown significant promise, yet considerable
research gaps have been identified as well. It is possible to pursue
incremental lines of research, as well as to investigate the possibility of
designing novel architectures, the applicability of self-supervised methods,
zero-shot approaches and other avenues of inquiry. We are of the opinion that
text-to-image cross-modal generation may receive further elevated levels of
interest, similarly to what has been the case for image-to-text methods. Work
on general architectures and data efficiency improvements might take center
stage.

The factors considered in this review point to the text-to-image area as a
relatively young one, with increasing numbers of successful approaches, many
existing research gaps, significant practical impact and a potential to bring
together methods designed for single-modality settings. With interesting lines
of research, and the potential for new ones to crop up, text-to-image cross
modal generation is one of the most interesting areas in contemporary machine
learning research.

## References

  * (1)
  * Ackley et al. (1985) David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. 1985.  A learning algorithm for boltzmann machines.  _Cognitive Science_ 9, 1 (1985), 147–169.
  * Aifanti et al. (2010) Niki Aifanti, Christos Papachristou, and Anastasios Delopoulos. 2010.  The MUG facial expression database. In _11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10_. 1–4.
  * Andonian et al. (2021) Alex Andonian, Sabrina Osmany, Audrey Cui, YeonHwan Park, Ali Jahanian, Antonio Torralba, and David Bau. 2021.  Paint by Word.  arXiv:2103.10951
  * Ashual and Wolf (2019) Oron Ashual and Lior Wolf. 2019.  Specifying Object Attributes and Relations in Interactive Scene Generation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_.
  * Ashual and Wolf (2020) Oron Ashual and Lior Wolf. 2020.  Interactive Scene Generation via Scene Graphs with Attributes. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 13651–13654.  https://doi.org/10.1609/aaai.v34i09.7112
  * Avrahami et al. (2022) Omri Avrahami, Dani Lischinski, and Ohad Fried. 2022.  Blended Diffusion for Text-Driven Editing of Natural Images. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 18208–18218.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural Machine Translation by Jointly Learning to Align and Translate. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Yoshua Bengio and Yann LeCun (Eds.).
  * Balaji et al. (2019) Yogesh Balaji, Martin Renqiang Min, Bing Bai, Rama Chellappa, and Hans Peter Graf. 2019.  Conditional GAN with Discriminative Filter Generation for Text-to-Video Synthesis. In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_. International Joint Conferences on Artificial Intelligence Organization, 1995–2001.  https://doi.org/10.24963/ijcai.2019/276
  * Bar-Tal et al. (2022) Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. 2022.  Text2live: Text-driven layered image and video editing. In _European Conference on Computer Vision_. Springer, 707–723.
  * Bardes et al. (2022) Adrien Bardes, Jean Ponce, and Yann LeCun. 2022.  VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning. In _International Conference on Learning Representations_.
  * Bengio et al. (2013) Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. 2013.  Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation.  arXiv:1308.3432 [cs.LG]
  * Blattmann et al. (2022) Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. 2022.  Retrieval-Augmented Diffusion Models. In _Advances in Neural Information Processing Systems_ , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 15309–15324.  https://proceedings.neurips.cc/paper_files/paper/2022/file/62868cc2fc1eb5cdf321d05b4b88510c-Paper-Conference.pdf
  * Bodla et al. (2018) Navaneeth Bodla, Gang Hua, and Rama Chellappa. 2018.  Semi-supervised FusedGAN for Conditional Image Generation. In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Bromley et al. (1993) Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. 1993.  Signature Verification using a ”Siamese” Time Delay Neural Network. In _Advances in Neural Information Processing Systems_ , J. Cowan, G. Tesauro, and J. Alspector (Eds.), Vol. 6. Morgan-Kaufmann.
  * Brooks et al. (2023) Tim Brooks, Aleksander Holynski, and Alexei A. Efros. 2023.  InstructPix2Pix: Learning To Follow Image Editing Instructions. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 18392–18402.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.
  * Chen et al. (2019) Shizhe Chen, Qin Jin, and Alexander Hauptmann. 2019.  Unsupervised Bilingual Lexicon Induction from Mono-Lingual Multimodal Data.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 33, 01, 8207–8214.  https://doi.org/10.1609/aaai.v33i01.33018207
  * Chen et al. (2017) Tseng-Hung Chen, Yuan-Hong Liao, Ching-Yao Chuang, Wan-Ting Hsu, Jianlong Fu, and Min Sun. 2017.  Show, Adapt and Tell: Adversarial Training of Cross-Domain Image Captioner. In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Chen and Zhu (2022) Wenxue Chen and Jianke Zhu. 2022.  Multimodal Adversarially Learned Inference with Factorized Discriminators. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 36. 6304–6312.
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. 2015.  Microsoft COCO Captions: Data Collection and Evaluation Server.  arXiv:1504.00325 [cs.CV]
  * Cheng et al. (2020) Jun Cheng, Fuxiang Wu, Yanling Tian, Lei Wang, and Dapeng Tao. 2020.  RiFeGAN: Rich Feature Generation for Text-to-Image Synthesis From Prior Knowledge. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Child et al. (2019) Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019.  Generating Long Sequences with Sparse Transformers.  arXiv:1904.10509 [sc.LG]
  * Choi et al. (2018) Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. 2018.  StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Chopra et al. (2005) S. Chopra, R. Hadsell, and Y. LeCun. 2005.  Learning a similarity metric discriminatively, with application to face verification. In _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)_ , Vol. 1. 539–546.  https://doi.org/10.1109/CVPR.2005.202
  * Couairon et al. (2022) Guillaume Couairon, Asya Grechka, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. 2022.  FlexIT: Towards Flexible Semantic Image Translation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Daunhawer et al. (2022) Imant Daunhawer, Thomas M. Sutter, Kieran Chin-Cheong, Emanuele Palumbo, and Julia E Vogt. 2022.  On the Limitations of Multimodal VAEs. In _International Conference on Learning Representations_.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database. In _2009 IEEE conference on Computer Vision and Pattern Recognition_. 248–255.
  * Deng et al. (2019) Kangle Deng, Tianyi Fei, Xin Huang, and Yuxin Peng. 2019.  IRC-GAN: Introspective Recurrent Convolutional GAN for Text-to-video Generation. In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_. International Joint Conferences on Artificial Intelligence Organization, 2216–2222.  https://doi.org/10.24963/ijcai.2019/307
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Nichol. 2021.  Diffusion Models Beat GANs on Image Synthesis. In _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 8780–8794.
  * Ding et al. (2021) Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. 2021.  CogView: Mastering Text-to-Image Generation via Transformers. In _Advances in Neural Information Processing Systems_ , A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (Eds.).
  * Dong et al. (2017) Hao Dong, Simiao Yu, Chao Wu, and Yike Guo. 2017.  Semantic Image Synthesis via Adversarial Learning. In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In _International Conference on Learning Representations_.
  * Du et al. (2020) Yilun Du, Shuang Li, and Igor Mordatch. 2020.  Compositional Visual Generation with Energy Based Models. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 6637–6647.
  * Dumoulin and Visin (2016) Vincent Dumoulin and Francesco Visin. 2016.  A guide to convolution arithmetic for deep learning.  arXiv:1603.07285 [stat.ML]
  * Duvenaud et al. (2015) David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alan Aspuru-Guzik, and Ryan P Adams. 2015.  Convolutional Networks on Graphs for Learning Molecular Fingerprints. In _Advances in Neural Information Processing Systems_ , C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc.
  * El-Nouby et al. (2019) Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W. Taylor. 2019.  Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_.
  * Esser et al. (2021b) Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer. 2021b.  ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis. In _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 3518–3532.
  * Esser et al. (2021a) Patrick Esser, Robin Rombach, and Bjorn Ommer. 2021a.  Taming Transformers for High-Resolution Image Synthesis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 12873–12883.
  * Fu et al. (2022) Tsu-Jui Fu, Xin Eric Wang, Scott T. Grafton, Miguel P. Eckstein, and William Yang Wang. 2022.  M3L: Language-Based Video Editing via Multi-Modal Multi-Level Transformers. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 10513–10522.
  * Gal et al. (2022) Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. 2022.  StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators. In _SIGGRAPH_.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014.  Generative Adversarial Nets. In _Advances in Neural Information Processing Systems_ , Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (Eds.), Vol. 27. Curran Associates, Inc.
  * Gu et al. (2022) Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. 2022.  Vector Quantized Diffusion Model for Text-to-Image Synthesis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 10696–10706.
  * Han et al. (2022) Ligong Han, Jian Ren, Hsin-Ying Lee, Francesco Barbieri, Kyle Olszewski, Shervin Minaee, Dimitris Metaxas, and Sergey Tulyakov. 2022.  Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 3615–3625.
  * Hayes et al. (2022) Thomas Hayes, Songyang Zhang, Xi Yin, Guan Pang, Sasha Sheng, Harry Yang, Songwei Ge, Qiyuan Hu, and Devi Parikh. 2022.  MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration. In _Computer Vision - ECCV 2022_ , Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (Eds.). Springer Nature Switzerland, Cham, 431–449.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep Residual Learning for Image Recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Hertz et al. (2023) Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. 2023.  Prompt-to-Prompt Image Editing with Cross-Attention Control. In _The Eleventh International Conference on Learning Representations_.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ (Long Beach, California, USA) _(NIPS’17)_. Curran Associates Inc., Red Hook, NY, USA, 6629–6640.
  * Hinton (2002) Geoffrey E. Hinton. 2002.  Training Products of Experts by Minimizing Contrastive Divergence.  _Neural Computation_ 14, 8 (2002), 1771–1800.  https://doi.org/10.1162/089976602760128018
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising Diffusion Probabilistic Models. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 6840–6851.
  * Ho and Salimans (2021) Jonathan Ho and Tim Salimans. 2021.  Classifier-Free Diffusion Guidance. In _NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications_.
  * Ho et al. (2022) Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. 2022.  Video Diffusion Models. In _Advances in Neural Information Processing Systems_ , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 8633–8646.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Computation_ 9, 8 (1997), 1735–1780.
  * Huang et al. (2008) Gary B. Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. 2008.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments. In _Workshop on Faces in ’Real-Life’ Images: Detection, Alignment, and Recognition_. Erik Learned-Miller and Andras Ferencz and Frédéric Jurie, Marseille, France.
  * Huang et al. (2022) Xun Huang, Arun Mallya, Ting-Chun Wang, and Ming-Yu Liu. 2022.  Multimodal Conditional Image Synthesis with Product-of-Experts GANs. In _Computer Vision - ECCV 2022_ , Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (Eds.). Springer Nature Switzerland, Cham, 91–109.
  * Jiang et al. (2021) Wentao Jiang, Ning Xu, Jiayun Wang, Chen Gao, Jing Shi, Zhe Lin, and Si Liu. 2021.  Language-Guided Global Image Editing via Cross-Modal Cyclic Mechanism. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 2115–2124.
  * Johnson et al. (2018) Justin Johnson, Agrim Gupta, and Li Fei-Fei. 2018.  Image Generation from Scene Graphs. In _CVPR_.
  * Johnson et al. (2017) Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. 2017.  CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Joy et al. (2022) Tom Joy, Yuge Shi, Philip Torr, Tom Rainforth, Sebastian M Schmon, and Siddharth N. 2022.  Learning Multimodal VAEs through Mutual Supervision. In _International Conference on Learning Representations_.
  * Kang et al. (2020) Lei Kang, Pau Riba, Yaxing Wang, Marçal Rusiñol, Fornésm Alicia, and Mauricio Villegas. 2020.  GANwriting: Content-Conditioned Generation of Styled Handwritten Word Images. In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A Style-Based Generator Architecture for Generative Adversarial Networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Karras et al. (2020) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020.  Analyzing and Improving the Image Quality of StyleGAN. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Kawar et al. (2023) Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. 2023.  Imagic: Text-Based Real Image Editing With Diffusion Models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 6007–6017.
  * Kim et al. (2022b) Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. 2022b.  DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 2426–2435.
  * Kim et al. (2022a) Jin-Hwa Kim, Yunji Kim, Jiyoung Lee, Kang Min Yoo, and Sang-Woo Lee. 2022a.  Mutual Information Divergence: A Unified Metric for Multimodal Generative Models. In _Advances in Neural Information Processing Systems_ , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 35072–35086.
  * Kim et al. (2019) Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, and Devi Parikh. 2019.  CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 6495–6513.  https://doi.org/10.18653/v1/P19-1651
  * Kingma and Welling (2014) Diederik P. Kingma and Max Welling. 2014.  Auto-Encoding Variational Bayes. In _2nd International Conference on Learning Representations, ICLR 2014_.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-Supervised Classification with Graph Convolutional Networks. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_.
  * Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017.  Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations.  _Int. J. Comput. Vision_ 123, 1 (May 2017), 32–73.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012.  ImageNet Classification with Deep Convolutional Neural Networks. In _Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1_ (Lake Tahoe, Nevada) _(NIPS’12)_. Curran Associates Inc., Red Hook, NY, USA, 1097–1105.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. Association for Computational Linguistics, Brussels, Belgium, 66–71.  https://doi.org/10.18653/v1/D18-2012
  * Kuznetsova et al. (2020) Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. 2020.  The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale.  _IJCV_ 128, 7 (2020), 1956–1981.
  * Laina et al. (2019) Iro Laina, Christian Rupprecht, and Nassir Navab. 2019.  Towards Unsupervised Image Captioning With Shared Multimodal Embeddings. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_.
  * LeCun et al. (1998) Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. 1998.  Gradient-based learning applied to document recognition.  _Proc. IEEE_ 86, 11 (1998), 2278–2324.  https://doi.org/10.1109/5.726791
  * Lee et al. (2020) Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. 2020.  MaskGAN: Towards Diverse and Interactive Facial Image Manipulation. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Li et al. (2019b) Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. 2019b.  Controllable Text-to-Image Generation. In _Advances in Neural Information Processing Systems 32_.
  * Li et al. (2020a) Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr. 2020a.  ManiGAN: Text-guided image manipulation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7880–7889.
  * Li et al. (2020b) Bowen Li, Xiaojuan Qi, Philip H. S. Torr, and Thomas Lukasiewicz. 2020b.  Lightweight Generative Adversarial Networks for Text-Guided Image Manipulation. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.).
  * Li et al. (2022) Lei Li, Kai Fan, and Chun Yuan. 2022.  Cross-modal Representation Learning and Relation Reasoning for Bidirectional Adaptive Manipulation. In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_ , Lud De Raedt (Ed.). International Joint Conferences on Artificial Intelligence Organization, 3222–3228.
  * Li et al. (2021) Lincheng Li, Suzhen Wang, Zhimeng Zhang, Yu Ding, Yixing Zheng, Xin Yu, and Changjie Fan. 2021.  Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 35, 3, 1911–1920.
  * Li et al. (2019a) Yikang Li, Tao Ma, Yeqi Bai, Nan Duan, Sining Wei, and Xiaogang Wang. 2019a.  PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph. In _Advances in Neural Information Processing Systems 32_.
  * Li et al. (2018) Yitong Li, Martin Min, Dinghan Shen, David Carlson, and Lawrence Carin. 2018.  Video Generation From Text.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 32, 1 (Apr. 2018).  https://doi.org/10.1609/aaai.v32i1.12233
  * Liang et al. (2020) Jiadong Liang, Wenjie Pei, and Feng Lu. 2020.  CPGAN: Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis. In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014.  Microsoft COCO: Common Objects in Context. In _Computer Vision – ECCV 2014_ , David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, 740–755.
  * Liu et al. (2022b) Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. 2022b.  BEAT: A Large-Scale Semantic and Emotional Multi-modal Dataset for Conversational Gestures Synthesis. In _Computer Vision - ECCV 2022_ , Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (Eds.). Springer Nature Switzerland, Cham, 612–630.
  * Liu et al. (2021) Nan Liu, Shuang Li, Yilun Du, Josh Tenenbaum, and Antonio Torralba. 2021.  Learning to Compose Visual Relations.  _Advances in Neural Information Processing Systems_ 34\.
  * Liu et al. (2022a) Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. 2022a.  Compositional Visual Generation with Composable Diffusion Models. In _Computer Vision - ECCV 2022_ , Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (Eds.). Springer Nature Switzerland, Cham, 423–439.
  * Liu et al. (2019) Wen Liu, Zhixin Piao, Min Jie, Wenhan Luo, Lin Ma, and Shenghua Gao. 2019.  Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis. In _The IEEE International Conference on Computer Vision (ICCV)_.
  * Liu et al. (2015) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015.  Deep Learning Face Attributes in the Wild. In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Ma et al. (2018) Shuang Ma, Jianlong Fu, Chang Wen Chen, and Tao Mei. 2018.  DA-GAN: Instance-Level Image Translation by Deep Attention Generative Adversarial Networks. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Mahajan et al. (2020) Shweta Mahajan, Iryna Gurevych, and Stefan Roth. 2020.  Latent Normalizing Flows for Many-to-Many Cross-Domain Mappings. In _International Conference on Learning Representations_.
  * Małkiński and Mańdziuk (2022) Mikołaj Małkiński and Jacek Mańdziuk. 2022.  Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven’s Progressive Matrices.  arXiv:2201.12382 [cs.AI]
  * Małkiǹski and Maǹdziuk (2023) Mikołaj Małkiǹski and Jacek Maǹdziuk. 2023.  A review of emerging research directions in Abstract Visual Reasoning.  _Information Fusion_ 91 (2023), 713–736.  https://doi.org/10.1016/j.inffus.2022.11.011
  * Manning et al. (2008) Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008.  _Introduction to Information Retrieval_.  Cambridge University Press.
  * Mansimov et al. (2016) Elman Mansimov, Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. 2016.  Generating Images from Captions with Attention. In _ICLR_.
  * Marwah et al. (2017) Tanya Marwah, Gaurav Mittal, and Vineeth N. Balasubramanian. 2017.  Attentive Semantic Video Generation Using Captions. In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Meng et al. (2022) Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. 2022.  SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. In _International Conference on Learning Representations_.
  * Mittal et al. (2017) Gaurav Mittal, Tanya Marwah, and Vineeth N. Balasubramanian. 2017.  Sync-DRAW: Automatic Video Generation Using Deep Recurrent Attentive Architectures. In _Proceedings of the 25th ACM International Conference on Multimedia_ (Mountain View, California, USA) _(MM ’17)_. Association for Computing Machinery, New York, NY, USA, 1096–1104.  https://doi.org/10.1145/3123266.3123309
  * Mokady et al. (2023) Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2023.  NULL-Text Inversion for Editing Real Images Using Guided Diffusion Models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 6038–6047.
  * Nagrani et al. (2020) Arsha Nagrani, Joon Son Chung, Weidi Xie, and Andrew Zisserman. 2020.  VoxCeleb: Large-scale speaker verification in the wild.  _Computer Speech & Language_ 60 (2020), 101027.  https://doi.org/10.1016/j.csl.2019.101027
  * Nam et al. (2018) Seonghyeon Nam, Yunji Kim, and Seon Joo Kim. 2018.  Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language. In _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc.
  * Netzer et al. (2011) Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. 2011.  Reading Digits in Natural Images with Unsupervised Feature Learning. In _NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011_.
  * Nguyen et al. (2017) Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. 2017.  Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Nichol and Dhariwal (2021) Alexander Quinn Nichol and Prafulla Dhariwal. 2021.  Improved Denoising Diffusion Probabilistic Models. In _Proceedings of the 38th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 139)_ , Marina Meila and Tong Zhang (Eds.). PMLR, 8162–8171.
  * Nichol et al. (2022) Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. 2022.  GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In _Proceedings of the 39th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 162)_ , Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 16784–16804.
  * Nie et al. (2021) Weili Nie, Arash Vahdat, and Anima Anandkumar. 2021.  Controllable and compositional generation with latent-space energy-based models. In _Thirty-Fifth Conference on Neural Information Processing Systems_.
  * Nilsback and Zisserman (2008) Maria-Elena Nilsback and Andrew Zisserman. 2008.  Automated Flower Classification over a Large Number of Classes. In _Indian Conference on Computer Vision, Graphics and Image Processing_.
  * Patashnik et al. (2021) Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. 2021.  StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 2085–2094.
  * Pont-Tuset et al. (2020) Jordi Pont-Tuset, Jasper Uijlings, Beer Changpinyo, Radu Soricut, and Vittorio Ferrari. 2020.  Connecting Vision and Language with Localized Narratives. In _ECCV_.
  * Prajwal et al. (2020) K R Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C.V. Jawahar. 2020.  A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild. In _Proceedings of the 28th ACM International Conference on Multimedia_ (Seattle, WA, USA) _(MM ’20)_. Association for Computing Machinery, New York, NY, USA, 484–492.  https://doi.org/10.1145/3394171.3413532
  * Qiao et al. (2019) Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. 2019.  MirrorGAN: Learning Text-To-Image Generation by Redescription. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning Transferable Visual Models From Natural Language Supervision. In _Proceedings of the 38th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 139)_ , Marina Meila and Tong Zhang (Eds.). PMLR, 8748–8763.
  * Radford et al. (2016) Alec Radford, Luke Metz, and Soumith Chintala. 2016.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. In _4th International Conference on Learning Representations, ICLR 2016_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _J. Mach. Learn. Res._ 21, 1, Article 140 (Jan 2020).
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical Text-Conditional Image Generation with CLIP Latents.  arXiv:2204.06125 [cs.CV]
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021.  Zero-Shot Text-to-Image Generation. In _Proceedings of the 38th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 139)_ , Marina Meila and Tong Zhang (Eds.). PMLR, 8821–8831.
  * Razavi et al. (2019) Ali Razavi, Aaron van den Oord, and Oriol Vinyals. 2019.  Generating Diverse High-Fidelity Images with VQ-VAE-2. In _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc.
  * Reed et al. (2016a) Scott Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee. 2016a.  Learning What and Where to Draw. In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ (Barcelona, Spain) _(NIPS’16)_. Curran Associates Inc., Red Hook, NY, USA, 217–225.
  * Reed et al. (2016b) Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. 2016b.  Generative Adversarial Text to Image Synthesis. In _Proceedings of The 33rd International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 48)_ , Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, New York, USA, 1060–1069.
  * Roich et al. (2021) Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. 2021.  Pivotal Tuning for Latent-based Editing of Real Images.  _ACM Trans. Graph._ (2021).
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-Resolution Image Synthesis With Latent Diffusion Models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 10684–10695.
  * Sadoughi and Busso (2021) Najmeh Sadoughi and Carlos Busso. 2021.  Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networks.  _IEEE Trans. Affect. Comput._ 12, 4 (Nov 2021), 1031–1044.  https://doi.org/10.1109/TAFFC.2019.2916031
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. 2022.  Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. In _Advances in Neural Information Processing Systems_ , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 36479–36494.
  * Salimans et al. (2016) Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. 2016.  Improved Techniques for Training GANs. In _Advances in Neural Information Processing Systems_ , D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Vol. 29. Curran Associates, Inc.
  * Schuhmann et al. (2021) Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021.  LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs. In _Advances in Neural Information Processing Systems_.
  * Schuldt et al. (2004) C. Schuldt, I. Laptev, and B. Caputo. 2004.  Recognizing human actions: a local SVM approach. In _Proceedings of the 17th International Conference on Pattern Recognition (ICPR)_ , Vol. 3. 32–36.  https://doi.org/10.1109/ICPR.2004.1334462
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural Machine Translation of Rare Words with Subword Units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, 1715–1725.  https://doi.org/10.18653/v1/P16-1162
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018.  Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. In _ACL_. 2556–2565.
  * Shi et al. (2020) Jing Shi, Ning Xu, Trung Bui, Franck Dernoncourt, Zheng Wen, and Chenliang Xu. 2020.  A Benchmark and Baseline for Language-Driven Image Editing. In _Computer Vision – ACCV 2020: 15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 – December 4, 2020, Revised Selected Papers, Part VI_ (Kyoto, Japan). Springer-Verlag, Berlin, Heidelberg, 636–651.  https://doi.org/10.1007/978-3-030-69544-6_38
  * Shi et al. (2021b) Jing Shi, Ning Xu, Yihang Xu, Trung Bui, Franck Dernoncourt, and Chenliang Xu. 2021b.  Learning by Planning: Language-Guided Global Image Editing. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 13590–13599.
  * Shi et al. (2019) Yuge Shi, Siddharth N, Brooks Paige, and Philip Torr. 2019.  Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models. In _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc.
  * Shi et al. (2021a) Yuge Shi, Brooks Paige, Philip Torr, and Siddharth N. 2021a.  Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models. In _International Conference on Learning Representations_.
  * Shi et al. (2022) Yichun Shi, Xiao Yang, Yangyue Wan, and Xiaohui Shen. 2022.  SemanticStyleGAN: Learning Compositional Generative Priors for Controllable Image Synthesis and Editing. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 11254–11264.
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015.  Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In _Proceedings of the 32nd International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 37)_ , Francis Bach and David Blei (Eds.). PMLR, Lille, France, 2256–2265.
  * Song et al. (2021) Jiaming Song, Chenlin Meng, and Stefano Ermon. 2021.  Denoising Diffusion Implicit Models. In _International Conference on Learning Representations_.
  * Srivastava et al. (2015) Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. 2015.  Unsupervised Learning of Video Representations Using LSTMs. In _Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37_ (Lille, France) _(ICML’15)_. JMLR.org, 843–852.
  * Su et al. (2023) Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. 2023.  Dual Diffusion Implicit Bridges for Image-to-Image Translation. In _International Conference on Learning Representations_.
  * Sutter et al. (2020) Thomas M. Sutter, Imant Daunhawer, and Julia E. Vogt. 2020.  Multimodal Generative Learning Utilizing Jensen-Shannon-Divergence. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.).
  * Sutter et al. (2021) Thomas M. Sutter, Imant Daunhawer, and Julia E Vogt. 2021.  Generalized Multimodal ELBO. In _International Conference on Learning Representations_.
  * Suzuki et al. (2017) Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. 2017.  Joint Multimodal Learning with Deep Generative Models. In _ICLR_.
  * Talmor et al. (2021) Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. 2021.  MultiModalQA: complex question answering over text, tables and images. In _International Conference on Learning Representations_.
  * Tan et al. (2019) Fuwen Tan, Song Feng, and Vicente Ordonez. 2019.  Text2Scene: Generating Compositional Scenes From Textual Descriptions. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Tao et al. (2022) Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu. 2022.  DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 16515–16525.
  * Thies et al. (2020) Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. 2020.  Neural Voice Puppetry: Audio-driven Facial Reenactment.  _ECCV 2020_ (2020).
  * Tulyakov et al. (2018) Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. 2018.  MoCoGAN: Decomposing motion and content for video generation. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 1526–1535.
  * van den Oord et al. (2016) Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. 2016.  Pixel Recurrent Neural Networks. In _Proceedings of The 33rd International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 48)_ , Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, New York, USA, 1747–1756.
  * van den Oord et al. (2017) Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017.  Neural Discrete Representation Learning. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ (Long Beach, California, USA) _(NIPS’17)_. Curran Associates Inc., Red Hook, NY, USA, 6309–6318.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All you Need. In _Advances in Neural Information Processing Systems_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc.
  * Wah et al. (2011) C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. 2011.  _Technical Report CNS-TR-2011-001_.  Technical Report. California Institute of Technology.
  * Wang and Xiong (2021) Dexin Wang and Deyi Xiong. 2021.  Efficient Object-Level Visual Context Modeling for Multimodal Machine Translation: Masking Irrelevant Objects Helps Grounding. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 2720–2728.
  * Wang et al. (2018b) Hai Wang, Jason D. Williams, and SingBing Kang. 2018b.  Learning to Globally Edit Images with Textual Description.  arXiv:1810.05786 [cs.CV]
  * Wang et al. (2018a) Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. 2018a.  High-Resolution Image Synthesis and Semantic Manipulation With Conditional GANs. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Wang et al. (2019) Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei. 2019.  Eidetic 3D LSTM: A Model for Video Prediction and Beyond. In _International Conference on Learning Representations_.
  * Wang et al. (2022) Yan Wang, Yixuan Sun, Yiwen Huang, Zhongying Liu, Shuyong Gao, Wei Zhang, Weifeng Ge, and Wenqiang Zhang. 2022.  FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videos. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Wu et al. (2022) Chen Henry Wu, Saman Motamed, Shaunak Srivastava, and Fernando D De la Torre. 2022.  Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models. In _Advances in Neural Information Processing Systems_ , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 22422–22437.
  * Wu and Goodman (2018) Mike Wu and Noah Goodman. 2018.  Multimodal Generative Models for Scalable Weakly-Supervised Learning. In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ (Montréal, Canada) _(NIPS’18)_. Curran Associates Inc., Red Hook, NY, USA, 5580–5590.
  * Xia et al. (2021) Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu. 2021.  TediGAN: Text-Guided Diverse Face Image Generation and Manipulation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 2256–2265.
  * Xu et al. (2018) Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. 2018.  AttnGAN: Fine-Grained Text to Image Generation With Attentional Generative Adversarial Networks. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Yan et al. (2016) Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. 2016.  Attribute2Image: Conditional Image Generation from Visual Attributes. In _Computer Vision – ECCV 2016_ , Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, Cham, 776–791.
  * Yang et al. (2021b) Shiquan Yang, Rui Zhang, Sarah M. Erfani, and Jey Han Lau. 2021b.  UniMF: A Unified Framework to Incorporate Multimodal Knowledge Bases into End-to-End Task-Oriented Dialogue Systems. In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21_ , Zhi-Hua Zhou (Ed.). International Joint Conferences on Artificial Intelligence Organization, 3978–3984.  https://doi.org/10.24963/ijcai.2021/548 Main Track.
  * Yang et al. (2021a) Ze Yang, Wei Wu, Huang Hu, Can Xu, Wei Wang, and Zhoujun Li. 2021a.  Open Domain Dialogue Generation with Latent Images. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 14239–14247.
  * Yin et al. (2019) Guojun Yin, Bin Liu, Lu Sheng, Nenghai Yu, Xiaogang Wang, and Jing Shao. 2019.  Semantics Disentangling for Text-To-Image Generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Yoon et al. (2020) Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee. 2020.  Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity.  _ACM Trans. Graph._ 39, 6 (Nov. 2020).  https://doi.org/10.1145/3414685.3417838
  * Yu et al. (2016) Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. 2016.  LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop.  arXiv:1506.03365 [cs.CV]
  * Zbontar et al. (2021) Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. 2021.  Barlow Twins: Self-Supervised Learning via Redundancy Reduction. In _Proceedings of the 38th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 139)_ , Marina Meila and Tong Zhang (Eds.). PMLR, 12310–12320.
  * Żelaszczyk and Mańdziuk (2023) Maciej Żelaszczyk and Jacek Mańdziuk. 2023.  Cross-modal text and visual generation: A systematic review. Part 1: Image to text.  _Information Fusion_ 93 (2023), 302–329.  https://doi.org/10.1016/j.inffus.2023.01.008
  * Zhang et al. (2021a) Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. 2021a.  Cross-Modal Contrastive Learning for Text-to-Image Generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 833–842.
  * Zhang et al. (2017) Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Metaxas. 2017.  StackGAN: Text to Photo-Realistic Image Synthesis With Stacked Generative Adversarial Networks. In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Zhang et al. (2019) Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Metaxas. 2019.  StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 41, 8 (2019), 1947–1962.  https://doi.org/10.1109/TPAMI.2018.2856256
  * Zhang et al. (2021b) Zhu Zhang, Jianxin Ma, Chang Zhou, Rui Men, Zhikang Li, Ming Ding, Jie Tang, Jingren Zhou, and Hongxia Yang. 2021b.  UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis. In _Advances in Neural Information Processing Systems_ , A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (Eds.).
  * Zhao et al. (2019) Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal. 2019.  Image Generation from Layout. In _CVPR_.
  * Zhou et al. (2022) Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. 2022.  Towards Language-Free Training for Text-to-Image Generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 17907–17917.
  * Zhu et al. (2019) Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. 2019.  DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zitnick and Parikh (2013) C. Lawrence Zitnick and Devi Parikh. 2013.  Bringing Semantics into Focus Using Visual Abstraction. In _2013 IEEE Conference on Computer Vision and Pattern Recognition_. 3009–3016.  https://doi.org/10.1109/CVPR.2013.387
  * Zolfaghari et al. (2021) Mohammadreza Zolfaghari, Yi Zhu, Peter Gehler, and Thomas Brox. 2021.  CrossCLR: Cross-Modal Contrastive Learning for Multi-Modal Video Representations. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 1450–1459.