#  From Images to Textual Prompts: Zero-shot Visual Question Answering with
Frozen Large Language Models

Jiaxian Guo1 Junnan Li2 Dongxu Li2 Anthony Meng Huat Tiong2,3
Boyang Li3 Dacheng Tao1 Steven Hoi2
1 The University of Sydney 2 Salesforce Research 3 Nanyang Technological
University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Work done while Jiaxian Guo was an
intern at Salesforce Research.

###### Abstract

Large language models (LLMs) have demonstrated excellent zero-shot
generalization to new language tasks. However, effective utilization of LLMs
for zero-shot visual question-answering (VQA) remains challenging, primarily
due to the modality disconnect and task disconnect between the LLM and VQA
tasks. End-to-end training on multimodal data may bridge the disconnects, but
is inflexible and computationally expensive. To address this issue, we propose
_Img2LLM_ , a plug-and-play module that provides LLM prompts to enable LLMs to
perform zero-shot VQA tasks without end-to-end training. We develop LLM-
agnostic models describe image content as exemplar question-answer pairs,
which prove to be effective LLM prompts. Img2LLM offers the following
benefits: 1) It achieves comparable or better performance than methods relying
on end-to-end training. For example, we outperform Flamingo [3] by 5.6% on
VQAv2. On the challenging A-OKVQA dataset, our method outperforms few-shot
methods by as much as 20%. 2) It flexibly interfaces with a wide range of LLMs
to perform VQA. 3) It eliminates the need to specialize LLMs using end-to-end
finetuning and serve highly specialized LLMs to end users, thereby reducing
cost. Code is available via the LAVIS [31] framework at
https://github.com/salesforce/LAVIS/tree/main/projects/img2llm-vqa.

##  1 Introduction

Visual question answering (VQA) [5] is a prominent vision-language task that
finds a broad range of real-world applications, such as assisting blind
individuals in understanding their environments. A diverse set of VQA datasets
have been proposed, some focusing on image recognition [18, 5] and others on
logical reasoning [42]. However, human annotations are expensive to obtain and
may introduce a variety of human biases [11, 6, 67], making the VQA system
brittle towards new answer styles and question types[1, 23]. This has led
researchers to zero-shot VQA methods [11, 6, 23] that do not require ground-
truth question-answer annotations, thereby facilitating more generalizable VQA
systems.

Recently, large language models (LLMs) (e.g., [9, 70]) have demonstrated
excellent capabilities to perform tasks with zero in-domain data, conduct
logical reasoning, and apply commonsense knowledge in NLP tasks [29, 59, 61].
As a result, recent approaches [3, 65, 56] have resorted to leverage LLMs in
zero-shot VQA.

However, applying LLMs to VQA tasks is less than straightforward, due to (1)
the modality disconnect between vision and language and (2) the task
disconnect between language modeling and question answering. A common
technique is to finetune a vision encoder jointly with the LLM [56, 3, 22] to
align the vision and language representation spaces, but this can incur
prohibitive computational and data cost. For example, Flamingo [3] finetunes
on billions of image-text pairs with thousands of TPUs. Further, the
finetuning specializes and introduces strong interdependence between the
vision encoder and the LLM. If we need to upgrade the LLM as new versions
emerge, the entire model needs to undergo expensive re-training.

In contrast to the end-to-end integration of LLM into a VQA system, this paper
proposes a modular VQA system built on top of frozen off-the-shelf LLMs. This
brings two benefits. First, it can reduce the deployment cost and simplify the
deployment. Second, upgrading the LLM is straightforward. However, it is
challenging to bridge the modality disconnect and task disconnect without end-
to-end training. PICa [65] converts images into captions, and provides
exemplar QA pairs from training data as prompt to the LLM. However, doing so
assumes the existence of annotated training data and the performance is
sensitive to the selection of few-shot exemplars.

We propose _Img2LLM_ , a plug-and-play module that enables off-the-shelf LLMs
to perform zero-shot VQA. The central insight of Img2LLM is that we can
utilize a vision-language model (_e.g._ BLIP [33]) and a question-generation
model to translate the image content into synthetic question-answer (QA)
pairs, which are fed to the LLM as part of the prompt. These exemplar QA pairs
tackle the modality disconnect by describing the image content verbally, and
tackle the task disconnect by demonstrating the QA task to the LLM. Notably,
the exemplar QA pairs are constructed entirely based on the test image and
question, obviating the need for similar few-shot examples as required by PICa
[65], which are not always available in practical zero-shot scenarios. When
applied to the open-source OPT language models [70], Img2LLM achieves
comparable or superior zero-shot VQA performance to methods that perform
costly end-to-end training.

With this paper, we make the following contributions.

  * •

We propose Img2LLM, a plug-and-play module that converts an image into
synthetic question-answer pairs based solely on the current image of the
question. Img2LLM bridges the modality disconnect between language and vision
as well as the task disconnect between language modeling and visual question-
answering.

  * •

Img2LLM enables off-the-shelf LLMs to perform zero-shot VQA without costly
end-to-end training or specialized textual QA networks [43], thereby allowing
low-cost and flexible model deployment and painless LLM upgrades (Table 3).

  * •

Our experimental results show that the OPT models equipped with Img2LLM
achieve zero-shot VQA performance that is competitive or superior to the end-
to-end trained models. For example, we outperform Flamingo [3] by 5.6% on
VQAv2. We even outperform many few-shot VQA methods.

##  2 Related Work

###  2.1 Recent Advances in VQA Methods

As a multi-modal evaluation benchmark, Visual Question Answering (VQA) that
requires the model to answer a natural language question according to the
image, has been the focus of active research [66, 4, 5, 50, 2]. The past few
years witnessed rapid performance advances with large-scale image-text
pretraining [20, 68, 37, 34, 35, 69, 58, 52, 33, 22, 14] followed byfine-
tuning on VQA datasets. To tackle knowledge-based VQA [50, 42], recent works
[19, 36, 63, 40, 39, 41, 17, 32] incorporate external knowledge, such as
ConceptNet [53] or Wikipedia, but experimental results in [50] show that these
methods still struggle to answer questions requiring complex reasoning.

###  2.2 LLM for Zero/Few-Shot VQA Tasks

Large language models (LLMs) [10, 70, 13] trained on web-scale corpus are
powerful in natural language understanding and reasoning [71, 9]. To infer on
task data, LLMs typically generate target tokens autoregressively. In
specific, given prompt $C$ and task input $x$, an LLM generates target tokens
$Y=\\{y_{i}\\}_{i=1}^{n}$, with $y_{i}$ =
$\displaystyle\operatorname*{arg\,max}p_{\theta}(y_{i}|y_{<i},C,x)$ and
$\theta$ the model parameters. Prior VQA methods using LLMs mainly fall into
two categories: multi-modal pretraining and language-mediated VQA.

Multi-modal pretraining. These approaches align vision and language embeddings
by training additional alignment modules, as shown in Figure 1(a). Considering
that LLMs are too large to finetune efficiently, [56] opt to fine-tune only
the visual encoder while Flamingo [3] trains extra cross-attention layers to
model cross-modality interactions. However, this paradigm suffers from two
drawbacks: 1) Highly compute-inefficient. Jointly aligning vision backbones
and LLMs requires large compute resources. For example, training Flamingo
requires 1536 TPUv4 over two weeks. Hence, it becomes prohibitively expensive
to switch to a different LLM. 2) Catastrophic forgetting. The alignment step
may be detrimental to LLMs’ reasoning ability, if the LLMs are jointly trained
with the visual model [3].

Language-mediated VQA. Instead of vectorized representations, this VQA
paradigm directly resorts to natural language as the intermediate
representation of the image and no longer requires expensive pretraining. As
depicted by Figure 1(b), it first converts the current image to language
descriptions and feeds the descriptions, possibly accompanied by in-context
exemplars, to a frozen LLM. In a few-shot setting, PICa [65] generates
captions for the image and selects training data samples as in-context
exemplars, but its performance degrades substantially when the exemplars are
omitted. As a concurrent zero-shot approach, [43] generates question-relevant
captions. Due to the zero-shot requirement, it is unable to provide in-context
exemplars and does not reap the benefits of in-context learning. As a result,
it has to rely on a QA-specific LLM, UnifiedQAv2 [27], to achieve high
performance.

##  3 Method

Figure 1: The illustrative comparison of three tyepes of methods that enable
LLM to perform VQA tasks, where blue block denotes that the the inner
parameters are frozen while pink block indicates the inner parameters are
trainable.

Difficulties in utilizing LLMs effectively in zero-shot VQA stem mainly from
two obstacles: (i) _The modality disconnection_ : LLMs do not natively process
images and encoding visual information into a format that LLMs can process can
be a challenge. (ii) _The task disconnection_ : LLMs are usually pretrained
using generative [9] or denoising objectives [15] on language modeling tasks.
As the LLMs are unaware of the tasks of question answering or VQA, they often
fail to fully utilize contextual information in generating the answers.

In language-mediated VQA [65, 43], the modality disconnection is addressed by
converting the image to intermediate language descriptions instead of dense
vectors (§2.2). The task disconnection must be addressed using either few-shot
in-context exemplars [65] or an LLM directly finetuned on textual QA [43]. It
is not clear how to tackle the task disconnection on generic LLMs under zero-
shot settings.

We propose a new zero-shot technique to address the task disconnection on
generic LLMs, Img2LLM (Figure 1c), which generates image-relevant exemplar
prompts for the LLM. Given a question $Q$ and an image, our key insight is
that we can generate synthetic question-answer pairs as in-context exemplars
from the _current_ image. The exemplars not only demonstrate the QA task but
also communicate the content of the image to the LLM for answering the
question $Q$, thereby hitting two birds with one stone. Img2LLM is LLM-
agnostic; it unlocks the knowledge and the reasoning capacity of off-the-shelf
LLMs, offering a powerful yet flexible solution for zero-shot VQA.

###  3.1 Answer Extraction

In order to incorporate the image content into the exemplars for in-context
learning, from the current VQA image, we first seek words that could serve as
answers to synthetic questions. We generate a number of captions using an off-
the-shelf question-relevant caption generation module (§3.3). Following recent
papers [11, 30], we extract noun phrases (including named entities), verb
phrases, adjective phrases, numbers, and boolean-typed words like “yes” and
“no” as potential answers111We use the spaCy parser at https://spacy.io/,
though are not tied to the parser in any way.. We show some extracted answer
candidates in Figure 2 and Appendix A.3.

Figure 2: The overall pipeline of Img2LLM, including Caption Prompt and Exemplar Prompt generation.  Table 1: Results from mixing captions and exemplar prompts on 30B OPT [70]. Prompt Template | Caption Prompt | Exemplar Prompt | VQAv2 val | OK-VQA
---|---|---|---|---
Instruction | ✗ | ✗ | 18.1 | 3.3
Instruction + Captions | ✓ | ✗ | 46.1 | 23.5
Instruction + Question-Answer Pairs | ✗ | ✓ | 57.9 | 41.1
Instruction + Captions + Question-Answer Pairs | ✓ | ✓ | 59.5 | 41.8

###  3.2 Question Generation

With the extracted answer candidate set $\\{\hat{a}_{j}\\}_{j=1}^{U}$, we can
directly use any question generation network [24, 38, 64, 28, 2] to generate
specific questions for each answer candidate. In this paper, we experiment
with both template-based and neural question-generation methods. Note that to
avoid violating the zero-shot requirements, our method is purely textual-based
without access to any VQA data.

Template-based Question Generation. Using an off-the-shelf parser, we obtain
the part-of-speech for each answer, and design specific question templates for
each POS type. For example, for answers that are nouns, we use the question
“What object is in this image?” For verb answers, we use the question “What
action is being taken in this image?” Due to space constraints, we put the
complete list of templates in Appendix A.5.

Neural Question Generation. Inspired by [11], we train a neural question
generation model on textual QA datasets. Specifically, we finetune a
pretrained T5-large model [46] to generate questions from answers. The input
to the model contains the prompt “Answer: [answer]. Context: [context]”, where
[answer] denotes the answer text and [context] denotes the context text from
textual QA datasets. During inference, we replace [answer] with an extracted
answer candidate and [context] with the generated caption from which the
answer was extracted. The model is finetuned on five textual QA datasets
including SQuAD2.0 [47], MultiRC [26], BookQA [44], CommonsenseQA [54] and
Social IQA[48].

With the above question generation methods, we acquire a set of synthetic
question-answer pairs $\\{\hat{q}_{j},\hat{a}_{j}\\}_{j=1}^{U}$. We use these
question-answer pairs as exemplars of LLM in-context learning [9], which
guides the LLM to perform QA task given the image content and bridges the task
disconnect between language modelling and VQA.

As a sneak preview, we show effects of exemplar QA pairs in Table 1. The
details of the instructions are explained in §3.4. We observe that exemplar QA
prompts perform considerably better than caption prompts (detailed in §3.3)
only, demonstrating their efficacy in bridging the task disconnection between
LLM pre-training and VQA tasks. Moreover, since the exemplar prompts already
describe much content of the image, which helps to bridge the modality
disconnection, adding captions on top does not provide much new information
and brings only limited performance gains.

###  3.3 Question-relevant Caption Prompt

In addition to the synthetic exemplar QA pairs, we also supply question-
relevant image captions to the LLM. We observe that the question may ask about
specific objects or regions in the image [62] but generic captions generated
by existing networks may not contain relevant information. In Figure 2, the
question _“What items are spinning in the background which can be used to
control electricity?”_ is relevant only to the wind turbines. However,
captions generated from the whole image are likely to focus on the salient
orange boat, leaving LLM with no information to answer the question. To
address this issue, we generate captions about the question-relevant portion
of the image and include them in the prompt to the LLM.

To achieve this, we first determine the regions of the image that are relevant
to the question, by using the Image-grounded Text Encoder (ITE) in BLIP [33],
as which assigns a similarity score $\text{sim}(v,q)$ to any pair of image $v$
and textual question $q$. With ITE, we use GradCAM [51], a feature-attribution
interpretability technique, to generate a coarse localisation map highlighting
matching image regions given a question [33]. Briefly, GradCam qualifies the
cross-attention scores from the Transformer network by the gradient of ITE
simlarity function $\text{sim}(v,q)$ with respect to the cross-attention
scores. As this technique was proposed in [43], we leave the details to
Appendix A.1.

Having obtained the patch relevance $r$, we sample a subset of image patches
with probability proportional to patch relevance $r$. After that, we generate
captions from the sampled image patches using top-k sampling [16]. To generate
semantically meaningful captions, a short prompt, “a picture of,” is also fed
into the text decoder. We repeat this $M$ times for each image to generate $M$
diverse captions, and keep only captions that are not exact substrings of
others.

However, due to the non-deterministic nature of top-k sampling, the caption
model may generate noisy captions that have a negative impact on performance.
To remove noisy captions, we use ITE to calculate the similarity score between
the generated caption and sampled question-relevant image patches, and filter
captions with less than 0.5 matching scores. Overall, this process yields
synthetic captions that are question-relevant, diverse, and clean, providing a
bridge between visual and language information.

###  3.4 Prompt Design

With synthetic question-relevant captions and question-answer pairs, we
construct complete prompts for LLM by concantenating the instruction,
captions, and QA exemplars. The instruction text is “Please reason the answers
of question according to the contexts.” The caption prompt is formatted as
“Contexts: [all captions]”. Individual QA exemplars are formatted as
“Question: [question] Answer: [answer]” and concatenated. We position the
current question as the last portion of the prompt, formatted as “Question:
[question]. Answer: ”. Finally, to get the answer, we perform greedy decoding
on the LLM and remove meaningless tokens as in Flamingo.

Furthermore, as the input to LLMs has maximum lengths, _e.g._ 2048 in OPT and
GPT3, it is necessary to select a subset of question-relevant captions and
question-answer pairs to construct the prompt. To select the most informative
prompt, we first count the frequency of the synthetic answer candidates in 100
generated captions. We then select 30 answer candidates with highest
frequencies and generate one question for each. Also, we include 30 answers
with the lowest frequency and one caption containing each answer. See §4.5 for
analysis of caption selection strategies.

##  4 Experiment

In this section, we first validate the efficacy of Img2LLM by comparing it
with other zero-shot and few-shot VQA methods. Then, we perform ablation
studies on important design choices, such as prompt patterns and caption
selection strategies, to understand their effect. We also show qualitative
examples and include discussion on observed failure cases.

###  4.1 Environment Setup

Datasets. We validate our method on VQAv2 [18], OK-VQA [42] and A-OKVQA [50]
datasets, which contain questions requiring perception, reasoning and
commonsense to answer. Specifically, VQAv2 [18] contains 214,354 questions in
the validation set and 107,394 in the test-dev dataset. OK-VQA [42] and A-OK-
VQA [50] emphasize on commonsense reasoning, among which OK-VQA contains 5,046
test questions and A-OKVQA [50] contains 1,100 validation questions and 6,700
test questions.

Implementation details. To obtain question-relevant caption prompt, we use
BLIP [33] to generate captions and perform image-question matching. To
localize the image regions relevant to the question, we generate GradCam from
the cross-attention layer of BLIP image-grounded text encoder. We then sample
$K^{\prime}=20$ image patches based on GradCam, and use them to obtain 100
question-relevant captions. For the LLMs, our main result uses the open-source
OPT model with multiple different sizes. Our ablation study also experiments
with various other LLMs to show the generalization ability of our method. We
use LLMs to generate answers auto-regressively, without access to either
answer list or training samples, thereby facilitating zero-shot VQA. We follow
official evaluation protocols and report VQA scores on each dataset.

Competing methods. We compare with prior VQA methods, which rougly fall into
three categories: (i) _Zero-shot methods with frozen LLMs_ , such as PICa
[65]. Our method also belongs to this category, yet unlike PICa, Img2LLM
requires no training samples to compose the prompts. (ii) _Zero-shot methods
with extra multi-modal pre-training_ , such as Flamingo [3], Frozen [56],
VL-T5 [12], FewVLM [22] and VLKD [14]. These methods require large-scale
vision-language datasets and are costly to update. We also include results
from VQ2A [11] and WeaQA [6] in this category, with _caveats_ that they assume
access to answer candidates which may not be available in practice. Therefore,
their results should be interpreted with caution. (iii) For reference
purposes, we also include available results from _few-shot methods_. These
include few-shot results of PICa [65], FewVLM [22] and ClipCap [45].

###  4.2 Main Results

Table 2: Performance on VQAv2, OK-VQA, and A-OKVQA. A few methods do not strictly satisfy the zero/few-shot requirements: methods without end-to-end training but assumes access to training samples are labeled with $\dagger$; methods that answer from a predefined list of candidates are in grey. Further, ✗ annotates methods requiring no end-to-end training, which is desirable, and ✓ otherwise.  Methods | End-to-End | Shot | VQAv2 | OK-VQA | A-OKVQA
---|---|---|---|---|---
| Training? | Number | val | test | test | val | test
_Zero-Shot Evaluation with Frozen Large Language Model_
PICa175B† | ✗ | 0 | - | - | 17.7 | - | -
Img2LLM6.7B | ✗ | 0 | 57.6 | 57.0 | 38.2 | 33.3 | 32.2
Img2LLM13B | ✗ | 0 | 57.1 | 57.3 | 39.9 | 33.3 | 33.0
Img2LLM30B | ✗ | 0 | 59.5 | 60.4 | 41.8 | 36.9 | 36.0
Img2LLM66B | ✗ | 0 | 59.9 | 60.3 | 43.2 | 38.7 | 38.2
Img2LLM175B | ✗ | 0 | 60.6 | 61.9 | 45.6 | 42.9 | 40.7
_Zero-Shot Evaluation with Extra End-to-End Training_
VL-T5no-vqa | ✓ | 0 | 13.5 | - | 5.8 | - | -
FewVLMbase | ✓ | 0 | 43.4 | - | 11.6 | - | -
FewVLMlarge | ✓ | 0 | 47.7 | - | 16.5 | - | -
VLKD ViT-B/16 | ✓ | 0 | 38.6 | 39.7 | 10.5 | - | -
VLKD ViT-L/14 | ✓ | 0 | 42.6 | 44.5 | 13.3 | - | -
Frozen7B | ✓ | 0 | 29.5 | - | 5.9 | - |
Flamingo3B | ✓ | 0 | - | 49.2 | 41.2 | - | -
Flamingo9B | ✓ | 0 | - | 51.8 | 44.7 | - | -
Flamingo80B | ✓ | 0 | - | 56.3 | 50.6 | - | -
_Zero-shot Evaluation with Access to Answer Candidates_
WeaQA ZSL | ✓ | 0 | 46.8 | - | - | - | -
VQ2A | ✓ | 0 | 61.1 | - | 19.8 | - | -
_Few-Shot Evaluation_
ClipCap$\rightarrow$Cap$\rightarrow$GPT175B | ✗ | 10 | - | - | - | 16.6 | 15.8
ClipCap$\rightarrow$Rel$\rightarrow$GPT175B | ✗ | 10 | - | - | - | 18.1 | 15.8
FewVLMbase | ✓ | 16 | 48.2 | - | 15.0 | - |
FewVLMlarge | ✓ | 16 | 51.1 | - | 23.1 | - | -
PICa175B† | ✗ | 1 | - | - | 36.4 | - | -
PICa175B† | ✗ | 4 | - | - | 43.3 | - | -
PICa175B† | ✗ | 16 | 54.3 | - | 46.5 | - | -
PICa175B-Ensemble | ✗ | 80 | 56.1 | - | 48.0 | - | -

Main quantitative results are shown in Table 2. We summarize our findings as
follows.

State-of-the-art results on zero-shot evaluation with plug-in frozen LLMs.
Img2LLM surpasses PICa, the best prior zero-shot model with frozen LLMs, by a
significant margin (45.6 _versus_ 17.7 on OK-VQA), thereby establishing a new
state-of-the-art. In addition, we remark that despite PICa uses frozen LLMs,
it requires training samples to build prompts. In contrast, our method
generates question-answers with no access to VQA samples, thus fully
fulfilling the zero-shot requirements.

Scaling effect of LLMs and their emergent capabilities on VQA. When increasing
the number of parameters of LLMs from 6.7B to 175B, we see a 3-10 points
improvement in VQA across datasets. This shows that stronger language
modelling capabilities help better comprehend the question, thus giving more
accurate answers. Such a trend is more clear and consistent on OK-VQA and
A-OKVQA, whose questions demand commonsense reasoning and external knowledge
that LLMs excel at providing. This corroborates our belief that LLMs are
beneficial to VQA.

Another intriguing phenomenon we observe is that the effect of scaling LLMs
becomes obvious only when the model size becomes sufficiently large, for
example, when using 30B or larger models, while not entirely predictable on
smaller ones (6.7B and 13B). This echoes with the recent finding on the
emergent abilities when using LLMs off-the-shelf [60] for language tasks,
while confirming the same trend for the first time when using frozen LLMs for
vision(-language) tasks.

Competitive performance with end-to-end pretraining and few-shot models.
Img2LLM obtains superior performance to most models with end-to-end
pretraining, as well as those evaluated in few-shot setups. For example, on
VQAv2 our method surpasses Flamingo${}_{80\textnormal{B}}$, which cost over
500K TPU hours and billion-scale datasets to train, by a margin of 5.6 points.
On A-OKVQA, Img2LLM more than doubles the best reported results so far, from
ClipClap. The only a few exceptions are on OK-VQA, where our method obtains
better results than Flamingo${}_{9\textnormal{B}}$, yet is not able to stay on
par with Flamingo${}_{80\textnormal{B}}$. Considering that Img2LLM is flexible
to adapt to updated and stronger LLMs with zero extra training cost, we
consider it a more approachable solution to practical adoption of VQA systems,
than those trained end-to-end. We also include comparisons with supervised
models in Appendix A.4. Img2LLM achieves better performance than most
supervised models, despite the fact that it uses zero training data and is
evaluated in a zero-shot setup. These results once again validates its
effectiveness.

Table 3: Zero-shot VQA performance with different LLMs. Methods | VQAv2 val | OK-VQA
---|---|---
PICa GPT-3 175B | - | 17.7
Frozen7B | 29.5 | 5.9
Ours GPT-Neo 2.7B | 50.1 | 31.5
Ours BLOOM 7.1B | 52.4 | 32.4
Ours GPT-J 6B | 56.4 | 37.4
Ours OPT 6.7B | 57.6 | 38.2
Ours OPT 175B | 60.6 | 45.6

###  4.3 Experimental Results of Different LLMs

In Table 3, we evaluate the performance of Img2LLM on various open-sourced
LLMs other than OPT, including GPT-J [57], GPT-Neo [8] and BLOOM [49]. The
experimental results show that Img2LLM enables various LLMs to perform zero-
shot VQA tasks, and that all of them achieve superior performance to zero-shot
PICa [65] and Frozen [56]. This is a strong evidence for showing our method’s
generalization ability with different LLMs.

###  4.4 Analysis on Question Generation Methods

Table 4 shows the performance of different question selection strategies
described in Section 3.2. We compare three question generation techniques,
include image-agnostic, which uses questions sampled from other images;
template-based, which uses template questions, and neural-based, which uses
neural generated questions. Further, we compare two synthetic QA selection
strategies. The random strategy, which selects QA pairs for prompt randomly;
the max freq. approach, which selects answer candidates that are most frequent
in the captions, and also retrieve the associated synthetic questions to build
the prompt.

Among the three question generation techniques, Agnostic perform the worst
whereas Neural performs the best. We attribute the differences to the quality
of QA pairs. Agnostic QA pairs contain information irrelevant to the current
image and may mislead the LLM. Template questions feature little linguistic
variation and hence cannot demonstrate different QA strategies. Neural has the
most relevant information and the most linguistic diversity. QA pair with
maximum answer frequency outperform random questions. We hypothesize that the
most frequent answers describe the most salient or important aspects of the
image, thereby providing more information than random questions.

Table 4: Effect of question selection strategies. |  | OK-VQA | VQAv2
---|---|---|---
PICa175B | 17.7 | -
Agnostic | Random | 35.9 | 52.9
Template | Random | 40.2 | 53.0
Max Freq. | 41.5 | 55.8
Neural | Random | 40.5 | 57.0
Max Freq. | 41.8 | 59.5

In addition, we evaluate visual information quality encoded in the exemplar
prompts using the answer hit rate and the answer noise rate. Answer hit rate
(AHR) is defined as the proportion of QA pairs containing the ground-truth
answer. Answer noise rate (ANR) is defined as the ratio of ground-truth
answers to the total number tokens in the exemplar prompts. Table 7 indicates
that exemplar prompts generated from question-relevant captions have a higher
AHR, hence enhancing the VQA performance. In addition, the caption filter
procedure can remove some noisy captions, allowing it to achieve a higher ANR
than its competitors. The experimental results demonstrate that improving both
the AHR and the ANR can improve the quality of prompts and VQA performance.

Table 5: Ablations on prompts designs. Methods | OK-VQA | VQAv2 val
---|---|---
CQA-CQA-CQA | 37.8 | 52.1
CCC-QAQAQA | 41.8 | 59.5
Table 6: Ablation on caption selection methods. Caption | Random | Max | Min
---|---|---|---
Selection | Frequency | Frequency
OK-VQA Acc | 41.3 | 41.1 | 41.8
Table 7: The experimental results on QA pairs generated from different captions. The results are run with OPT 30B. Exemplar Prompts Generation Source | OK-VQA | VQAv2 val
---|---|---
VQA | Answer | Answer | VQA | Answer | Answer
Score | Noise Rate | Hit Rate | Score | Noise Rate | Hit Rate
Caption from Complete Image | 39.8 | 0.018 | 0.480 | 57.1 | 0.0290 | 0.725
Question-relevant Caption | 40.6 | 0.022 | 0.581 | 58.1 | 0.0303 | 0.821
Question-relevant Caption with Filter | 41.8 | 0.025 | 0.566 | 59.5 | 0.0313 | 0.804

###  4.5 Ablation on Caption Selection

As Table 6 shows, we evaluate the performance different caption selection
strategies, where Max Frequency selects captions containing 30 answers with
highest frequencies and Min Frequency selects answers with the lowest
frequencies. As the exemplar prompts are produced with answers with the
highest frequencies, the Max Frequency strategy does not provide more
information than exemplar prompts. In contrast, the Min Frequency strategy
chooses captions that can provide some information not in the QA pairs,
providing a performance boost.

###  4.6 Ablation Study on Prompt Design

We have two options to construct LLM’s prompt. The first option is to append a
syntheic QA pair after the caption that the QA pair is generated from. This
can be described as CQA-CQA-CQA, where C, Q, A stand for caption, synthetic
question, and synthetic answer respectively. Alternatively, we can present all
captions at once, followed by all question-answer pairs, which we denote as
CCC-QAQAQA. Experimentally (Table 5), the second design performs significantly
better than the first. We hypothesize that the first design may induce the LLM
to read only one caption before answering, since in the prompt this caption
contains all the information needed for the question. While it is hard to
pinpoint the actual mechanism, the results highlight the importance of QA
prompts and their positions.

Figure 3: Example predictions made by Img2LLM. Specifically, (a) and (b) are
successful cases, while (c) and (d) are failure cases. See more examples at
Appendix A.5.

###  4.7 Examples and Failure Case Analysis

In Figure 3, we show four examples of caption and exemplar prompts and the
predictions, including cases of success and failure. In Figure 3(a), the
captions and the synthetic QA pairs provide the information that a man is
making drinks at a bar. The LLM draws on background knowledge and correctly
infers that his job is bartender. In Figure 3(c), while the prediction is
understandable (even if not strictly grammatical), the LLM is unable to make
inferences based on qualitative physics and predict the right answer. These
results highlight the importance to apply appropriate commonsense knowledge in
open-ended VQA.

##  5 Limitation

One limitation of the proposed approach is that generating image captions and
question-answer pairs incurs extra inference overhead. On an 8$\times$A100
machine, our current implementation brings about 24.4% additional
computational time on top of the inference time of 175B OPT. We note that
further reduction of the overhead can be obtained by shortening the prompt,
trading accuracy for speed. Notably, our method avoids expensive end-to-end
multimodal representation alignment, which took more than 500K TPU hours in
the case of Flamingo.

##  6 Conclusion

In this paper, we propose Img2LLM, a plug-and-play module designed to exploit
the knowledge and reasoning power of large language models (LLMs) off-the-
shelf for zero-shot VQA tasks. Concretely, Img2LLM provides visual information
and task guidance to LLMs in the format of easily-digestible prompts. This
eliminates the requirement for the expensive end-to-end vision-language
alignment, increasing model deployment flexibility while decreasing model
deployment cost. The experiments show that Img2LLM enables different LLMs to
achieve comparable or even superior zero-shot VQA performance to other methods
that require costly end-to-end training.

##  7 Acknowledgments

Jiaxian Guo was supported in part by Australian Research Council Projects
FL170100117 and IH180100002, the University of Sydney Completion Stipend
Scholarship, and Faculty of Engineering PhD Completion Award. Boyang Li was
supported by the Nanyang Associate Professorship and the National Research
Foundation Fellowship (NRF-NRFF13-2021-0006), Singapore.

## References

  * [1] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi.  Don’t just assume; look and answer: Overcoming priors for visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4971–4980, 2018.
  * [2] Arjun Akula, Soravit Changpinyo, Boqing Gong, Piyush Sharma, Song-Chun Zhu, and Radu Soricut.  Crossvqa: Scalably generating benchmarks for systematically testing vqa generalization.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2148–2166, 2021.
  * [3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.  Flamingo: a visual language model for few-shot learning.  arXiv Preprint 2204.14198, 2022.
  * [4] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077–6086, 2018.
  * [5] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015.
  * [6] Pratyay Banerjee, Tejas Gokhale, Yezhou Yang, and Chitta Baral.  WeaQA: Weak supervision via captions for visual question answering.  In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3420–3435, Online, Aug. 2021. Association for Computational Linguistics.
  * [7] Yuchen Bian, Jiaji Huang, Xingyu Cai, Jiahong Yuan, and Kenneth Church.  On attention redundancy: A comprehensive study.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 930–945, Online, June 2021. Association for Computational Linguistics.
  * [8] Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021.  If you use this software, please cite it using these metadata.
  * [9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * [10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [11] Soravit Changpinyo, Doron Kukliansky, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut.  All you may need for vqa are image captions.  In North American Chapter of the Association for Computational Linguistics, 2022.
  * [12] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal.  Unifying vision-and-language tasks via text generation.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1931–1942. PMLR, 18–24 Jul 2021.
  * [13] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311, 2022.
  * [14] Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung.  Enabling multimodal generation on CLIP via vision-language knowledge distillation.  In Findings of the Association for Computational Linguistics: ACL 2022, pages 2383–2395, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [16] Angela Fan, Mike Lewis, and Yann Dauphin.  Hierarchical neural story generation.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia, July 2018. Association for Computational Linguistics.
  * [17] François Gardères, Maryam Ziaeefard, Baptiste Abeloos, and Freddy Lecue.  Conceptbert: Concept-aware representation for visual question answering.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 489–498, 2020.
  * [18] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 6904–6913, 2017.
  * [19] Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao.  Kat: A knowledge augmented transformer for vision-and-language.  arXiv preprint arXiv:2112.08614, 2021.
  * [20] Haojun Jiang, Yuanze Lin, Dongchen Han, Shiji Song, and Gao Huang.  Pseudo-q: Generating pseudo language queries for visual grounding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15513–15523, 2022.
  * [21] Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh.  Pythia v0. 1: the winning entry to the vqa challenge 2018.  arXiv preprint arXiv:1807.09956, 2018.
  * [22] Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren.  A good prompt is worth millions of parameters: Low-resource prompt-based learning for vision-language models.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2763–2775, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [23] Kushal Kafle and Christopher Kanan.  An analysis of visual question answering algorithms.  In Proceedings of the IEEE international conference on computer vision, pages 1965–1973, 2017.
  * [24] Kushal Kafle, Mohammed Yousefhussien, and Christopher Kanan.  Data augmentation for visual question answering.  In Proceedings of the 10th International Conference on Natural Language Generation, pages 198–202, 2017.
  * [25] Amita Kamath, Christopher Clark, Tanmay Gupta, Eric Kolve, Derek Hoiem, and Aniruddha Kembhavi.  Webly supervised concept expansion for general purpose vision models.  arXiv preprint arXiv:2202.02317, 2022.
  * [26] Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth.  Looking beyond the surface: A challenge set for reading comprehension over multiple sentences.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252–262, 2018.
  * [27] Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi.  Unifiedqa: Crossing format boundaries with a single qa system.  arXiv preprint arXiv:2005.00700, 2020.
  * [28] Jihyung Kil, Cheng Zhang, Dong Xuan, and Wei-Lun Chao.  Discovering the unknown knowns: Turning implicit knowledge in the dataset into explicit training examples for visual question answering.  arXiv preprint arXiv:2109.06122, 2021.
  * [29] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  arXiv preprint arXiv:2205.11916, 2022.
  * [30] Hwanhee Lee, Thomas Scialom, Seunghyun Yoon, Franck Dernoncourt, and Kyomin Jung.  Qace: Asking questions to evaluate an image caption.  arXiv preprint arXiv:2108.12560, 2021.
  * [31] Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven CH Hoi.  Lavis: A library for language-vision intelligence.  arXiv preprint arXiv:2209.09019, 2022.
  * [32] Guohao Li, Xin Wang, and Wenwu Zhu.  Boosting visual question answering with context-aware knowledge aggregation.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 1227–1235, 2020.
  * [33] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International Conference on Machine Learning, 2022.
  * [34] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 9694–9705. Curran Associates, Inc., 2021.
  * [35] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, volume 12375 of Lecture Notes in Computer Science, pages 121–137. Springer, 2020\.
  * [36] Yuanze Lin, Yujia Xie, Dongdong Chen, Yichong Xu, Chenguang Zhu, and Lu Yuan.  Revive: Regional visual representation matters in knowledge-based visual question answering.  arXiv preprint arXiv:2206.01201, 2022.
  * [37] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [38] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee.  12-in-1: Multi-task vision and language representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10437–10446, 2020.
  * [39] Man Luo, Yankai Zeng, Pratyay Banerjee, and Chitta Baral.  Weakly-supervised visual-retriever-reader for knowledge-based question answering.  arXiv preprint arXiv:2109.04014, 2021.
  * [40] Ziyang Luo, Yadong Xi, Rongsheng Zhang, and Jing Ma.  Vc-gpt: Visual conditioned gpt for end-to-end generative vision-and-language pre-training.  arXiv preprint arXiv:2201.12723, 2022.
  * [41] Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach.  Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14111–14121, 2021.
  * [42] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019.
  * [43] Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven CH Hoi.  Plug-and-play vqa: Zero-shot vqa by conjoining large pretrained models with zero training.  In EMNLP Findings, 2022.
  * [44] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  arXiv preprint arXiv:1809.02789, 2018.
  * [45] Ron Mokady, Amir Hertz, and Amit H Bermano.  Clipcap: Clip prefix for image captioning.  arXiv preprint arXiv:2111.09734, 2021.
  * [46] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of Machine Learning Research, 21(140):1–67, 2020.
  * [47] Pranav Rajpurkar, Robin Jia, and Percy Liang.  Know what you don’t know: Unanswerable questions for squad.  arXiv preprint arXiv:1806.03822, 2018.
  * [48] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi.  Socialiqa: Commonsense reasoning about social interactions.  arXiv preprint arXiv:1904.09728, 2019.
  * [49] Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Bideman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et al.  What language model to train if you have one million gpu hours?  arXiv preprint arXiv:2210.15424, 2022.
  * [50] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi.  A-okvqa: A benchmark for visual question answering using world knowledge.  arXiv preprint arXiv:2206.01718, 2022.
  * [51] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-CAM: Visual explanations from deep networks via gradient-based localization.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 618–626, 2017.
  * [52] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.  Flava: A foundational language and vision alignment model.  arXiv preprint arXiv:2112.04482, 2021.
  * [53] Robyn Speer, Joshua Chin, and Catherine Havasi.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In Thirty-first AAAI conference on artificial intelligence, 2017\.
  * [54] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  arXiv preprint arXiv:1811.00937, 2018.
  * [55] Hao Tan and Mohit Bansal.  LXMERT: Learning cross-modality encoder representations from transformers.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100–5111, Hong Kong, China, Nov. 2019. Association for Computational Linguistics.
  * [56] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill.  Multimodal few-shot learning with frozen language models.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 200–212. Curran Associates, Inc., 2021.
  * [57] Ben Wang and Aran Komatsuzaki.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax, May 2021.
  * [58] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao.  SimVLM: Simple visual language model pretraining with weak supervision.  In International Conference on Learning Representations, ICLR 2022. OpenReview.net, 2022.
  * [59] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  arXiv preprint arXiv:2206.07682, 2022.
  * [60] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  arXiv preprint arXiv:2206.07682, 2022.
  * [61] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  arXiv preprint arXiv:2201.11903, 2022.
  * [62] Jialin Wu, Zeyuan Hu, and Raymond J Mooney.  Generating question relevant captions to aid visual question answering.  arXiv preprint arXiv:1906.00513, 2019.
  * [63] Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi.  Multi-modal answer validation for knowledge-based vqa.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2712–2721, 2022.
  * [64] Xing Xu, Tan Wang, Yang Yang, Alan Hanjalic, and Heng Tao Shen.  Radial graph convolutional network for visual question generation.  IEEE transactions on neural networks and learning systems, 32(4):1654–1667, 2020.
  * [65] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang.  An empirical study of GPT-3 for few-shot knowledge-based VQA.  In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
  * [66] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola.  Stacked attention networks for image question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 21–29, 2016.
  * [67] Desen Yuan.  Language bias in visual question answering: A survey and taxonomy.  arXiv preprint arXiv:2111.08531, 2021.
  * [68] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  arXiv preprint arXiv:2111.11432, 2021.
  * [69] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  Vinvl: Revisiting visual representations in vision-language models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579–5588, 2021.
  * [70] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  arXiv preprint arXiv:2205.01068, 2022.
  * [71] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi.  Least-to-most prompting enables complex reasoning in large language models.  arXiv Preprint 2205.10625, 2022.

##  Appendix A Appendix

###  A.1 Reproducibility Statement

We acknowledge the importance of reproducibility for research work and try
whatever we can to ensure the reproducibility of our work. As for the
implementation of our method, details such as hyperparameters are provided in
Section 4.1 in the main paper. We will publicly release all codes after the
acceptance of this paper.

###  A.2 Broader Impact Statement

We acknowledge that while the Img2LLM achieves comparable or superior
performance to other zero-shot VQA methods, it has not reduced the inherent
bias of these systems. Social-economic biases based on gender, age, race, and
ethnicity exist in the datasets, LLMs, and VQA systems presented in this
paper, including Img2LLM. Future work could assess the magnitude of this bias
and mitigate its impact.

###  A.3 Details about Question-Relevant Caption Generation

Concretely, we denote features of image patches extracted by ITE as
$f_{v}^{i}\in\displaystyle\mathbb{R}^{K\times D^{i}_{v}}$ and question
features as $f_{q}^{i}\in\displaystyle\mathbb{R}^{L\times D_{q}^{i}}$, where
$i$ is the number of the layer of ITE, $K$ is the number of images patches,
$L$ is the number of token in the given question, $D^{i}_{v}$ is the dimension
of patch feature in the $i$-th layer of ITE network and $D^{i}_{q}$ is the
dimension of textual feature in the $i$-th layer of ITE network. For cross-
attention head in $i$-th layer, the cross-attention scores $W^{i}$ between
each image patch and each token in question can be calculated directly as

$\vspace{-0.1em}\footnotesize
W^{i}=\operatorname{softmax}\left(\frac{f_{q}^{i}W_{Q}^{i}{W_{K}^{i}}^{\top}{f_{v}^{i}}^{\top}}{\sqrt{D_{q}^{i}}}\right).\vspace{-0.1em}$
(1)

where $W_{Q}^{i}\in\displaystyle\mathbb{R}^{D_{q}^{i}\times D_{q}^{i}}$ is the
query head and $W_{K}^{i}\in\displaystyle\mathbb{R}^{D_{v}^{i}\times
D_{q}^{i}}$ is the key head in the $i$-th layer of ITE network. With Equation
1, we obtain a cross-attention matrix
$W^{i}\in\displaystyle\mathbb{R}^{L\times K}$, where each row is the cross-
attention scores of each token in the question over all image patches.
Specifically, the attention matrix $W^{i}$ can be regarded as the patch
importance for ITE to calculate the similarity of whole image and question,
but it still contains redundancy that contributes only a minor performance
loss [7], indicating that some patches are uninformative. In order to find
these less relevant image patches, we follwing GradCAM and compute the
derivative of the cross-attention score from ITE function $\text{ sim}(v,q)$,
_i.e._ , $\partial\text{ sim}(v,q)/\partial W$, and multiplying its gradient
matrix with the cross-attention scores element-wisely. The relevance of the
$k$th image patch with the question, $r_{k}^{i}$, can be computed as the
average over $H$ attention heads and the sum over $L$ textual tokens:

$\vspace{-0.1em}\footnotesize
r_{k}^{i}=\frac{1}{H}\sum^{L}_{l=1}\sum^{H}_{h=1}\min\left(0,\frac{\partial\text{
sim}(v,q)}{\partial W_{lk}^{ih}}\right)W_{lk}^{ih},\vspace{-0.1em}$ (2)

where $h$ is the index of attention heads and $i$ is the layer index of ITE.

###  A.4 Experimental Results of Supervised Learning Methods in A-OKVQA

We show the experimental comparisons between our method and supervised model
on A-OKVQA dataset [50] as Table 10 shows. We can observe that our method
outperform almost all supervised model with smaller size language model. This
strongly support our method’s effectiveness in leveraging reasoning power of
large language models.

Table 8: The experimental comparisons with models trained in A-OKVQA training dataset. Methods | A-OKVQA
---|---
| Val | Test
_Models Fine-Tuned in A-OKVQA Training Set_
Pythia [21] | 25.2 | 21.9
ViLBERT [37] | 30.6 | 25.9
LXMERT [55] | 30.7 | 25.9
KRISP [41] | 33.7 | 27.1
GPV-2 [25] | 48.6 | 40.7
_Zero-Shot Evaluation with Plug-in Frozen Large Language Model_
Ours6.7B | 33.3 | 32.2
Ours13B | 33.3 | 33.0
Ours30B | 36.9 | 36.0
Ours66B | 38.7 | 38.2
Ours175B | 42.9 | 40.7

###  A.5 Template-Based Question Design

We design question templates for each part of speech type of answers as Table
9 shows.

Table 9: The question templates for answers with different part of speech. Part of Speech of Answer | Question Templates
---|---
Noun |  | What item is this in this picture?
---
What item is that in this picture?
Verb |  | What action is being done in this picture?
---
Why is this item doing in this picture?
Which action is being taken in this picture?
What action is item doing in this picture?
What action is item performing in this picture?
Adjective |  | How to describe one item in this picture?
---
What is item’s ADJ TYPE in this picture?
What is the ADJ TYPE in this picture?
Num | How many things in this picture?
Table 10: The experimental results of using different number of captions and QA pairs as prompts. The experiments are run on OK-VQA with OPT 30B.  QA Pairs Caption | 0 | 10 | 20 | 30 | 40 | 50
---|---|---|---|---|---|---
0 | 3.3 | 19.6 | 22.7 | 23.4 | 24.0 | 24.8
10 | 40.9 | 41.6 | 42.1 | 42.1 | 41.9 | 42.2
20 | 41.2 | 41.3 | 41.3 | 41.7 | 42.2 | 42.0
30 | 41.0 | 41.0 | 41.7 | 41.8 | 41.6 | 41.5
40 | 40.3 | 40.7 | 40.6 | 40.3 | 40.3 | 41.1
50 | 40.6 | 40.6 | 40.7 | 40.9 | 40.6 | 41.1

###  A.6 Sensitive Analysis

We evaluate the sensitive analysis about the QA pairs and number of captions
in prompt for LLM as Table 10 shows. We can observe that the differences in QA
scores on OK-VQA dataset are not higher than 1 as long as QA pairs in prompts.
The results demonstrate the performance of our method is robust with different
numbers of QA pairs and captions.

Table 11: The experimental results of using different number of patches to generate question-relevant captions. The experiments are run on OK-VQA with OPT 30B.  Patch_num | 10 | 20 | 40 | Full
---|---|---|---|---
| 41.2 | 41.8 | 41.6 | 39.8
Table 12: The experimental results of generating different number of question-relevant captions. The experiments are run on OK-VQA with OPT 30B.  Caption_num | PICa | 10 | 30 | 50 | 100
---|---|---|---|---|---
| 17.7 | 38.3 | 40.9 | 41.4 | 41.8

###  A.7 Examples

(a)

(b)

(c)

(d)

(e)

Figure 4: Success case analysis for OK-VQA. Green color indicates answer cues
and correct prediction.

(a)

(b)

(c)

(d)

(e)

Figure 5: Failure case analysis for OK-VQA. Red color indicates incorrect
prediction.

(a)

(b)

(c)

(d)

(e)

Figure 6: Success case analysis for A-OKVQA. Green color indicates answer cues
and correct prediction.

(a)

(b)

(c)

(d)

(e)

Figure 7: Failure case analysis for A-OKVQA. Red color indicates incorrect
prediction.

(a)

(b)

(c)

(d)

(e)

Figure 8: Success case analysis for VQAv2. Green color indicates answer cues
and correct prediction.

(a)

(b)

(c)

(d)

(e)

Figure 9: Failure case analysis for VQAv2. Red color indicates incorrect
prediction.


# On the ray-wavefront duality

Antonio Díaz-Cano  Departamento de Algebra, Geometría y Topología, Universidad
Complutense, 28040 Madrid, Spain  Instituto de Matemática Interdisciplinar,
Universidad Complutense, 28040 Madrid, Spain  Francisco González-Gascón
Departamento de Física Teórica, Universidad Complutense, 28040 Madrid, Spain
Luis L. Sánchez-Soto  Departamento de Óptica, Universidad Complutense, 28040
Madrid, Spain  Max-Planck-Institut für die Physik des Lichts, 91058 Erlangen,
Germany

###### Abstract

We investigate the behavior of the solutions of the ray equation in isotropic
media. We discuss local and global transversals and wavefronts and give
examples of rays without wavefronts.

##  I Introduction

Geometrical optics is a peculiar science. Its fundamental ingredients are
rays, which do not exist (except as a mathematical idealization), and
wavefronts, which indeed do exist, but are not directly observable [1]. Yet it
works: even with such an unsophisticated background, it maintains a unique
position in modern technology [2].

Superficially, geometrical optics might appear as a naive picture of light
propagation. However, the seminal work of Luneburg [3] and Kline and Kay [4]
laid the solid foundations of this discipline: the wavefronts come associated
with the eikonal equation, which is a short-wavelength approximation to
Maxwell’s equations. With a small amount of calculation, one can also show
that the rays are normal to the wavefronts. It is thus not surprising that the
authoritative textbook by Born and Wolf [5] states that “only normal
congruences are of interest for geometrical optics”. Consequently, non-normal
congruences are safely ignored, with a few exceptions [3, 6] that look at them
more as an exotic curiosity than as a feasible possibility.

Put in a slightly different manner, a perfect duality between rays and
wavefronts is tacitly assumed, so both can be used interchangeably. Indeed,
given the shape of a wavefront, the direction of any ray crossing the
wavefront can be immediately calculated via the eikonal equation. Conversely,
one might hope that the shape of a wavefront can be calculated by tracing
enough rays.

However, a closer examination reveals that this latter belief is not fully
justified. We revisit here that problem: taking the ray equation as our
starting point, we address the simple and unexplored question of whether given
a family of rays one can always find the associated wavefronts. The unforeseen
answer we find is that while in a two-dimensional world, the wavefronts always
exist (so the ray-wavefront duality is correct), it is not generally the case
for three-dimensional rays.

##  II The Ray Equation

To be as self-contained as possible, we first briefly summarize the essential
ingredients that we shall need for our purposes. In geometrical optics, light
propagates along rays, which are taken as oriented curves whose direction
coincides everywhere with the direction of the propagation of the energy
(i.e., the average Poynting vector).

Let $\mathbf{x}(t)\in\mathbb{R}^{m}$ (with $m=1,2,$ or 3 being the space
dimensionality) denote the position vector of a point on a ray, considered as
a function of an arbitrary parameter $t$, which can be thought of as time.
These curves can be obtained via Fermat’s principle [7]; that is, as the
variational problem $\delta\mathcal{A}=0$ [8], where

$\mathcal{A}=\int_{t_{1}}^{t_{2}}L(\mathbf{x},\dot{\mathbf{x}})\;dt$ (1)

and the optical Lagrangian is [9]

$L(\mathbf{x},\dot{\mathbf{x}})=n(\mathbf{x})\,\lVert\dot{\mathbf{x}}\rVert\,.$
(2)

Here, $n(\mathbf{x})$ is the refractive index, the dot indicates the
derivative with respect to $t$, and $\lVert\cdot\rVert=$ stands for the
Euclidean norm. Notice that we are assuming that light propagates in an
isotropic nondispersive medium.

This is a standard problem in the calculus of variations and the time-honored
Euler–Lagrange equations (which give a sufficient condition of extremality)
reduce in this case to

$n\,\ddot{\mathbf{x}}=\dot{\mathbf{x}}^{2}\,{\nabla}n-2\dot{\mathbf{x}}\,{\nabla}n\cdot\dot{\mathbf{x}}\,,$
(3)

which is called the ray equation. Quite often, this equation is rewritten in
the form

$\frac{d}{ds}\left(n\frac{d\mathbf{x}}{ds}\right)=\nabla n\,,$ (4)

in terms of the Euclidean arc-length parameter $s$, such that
$ds=\lVert\dot{\mathbf{x}}\rVert\,dt$.

The function

$I(\mathbf{x},\dot{\mathbf{x}})=n^{2}(\mathbf{x})\,\dot{\mathbf{x}}^{2}$ (5)

is a first integral of (3), as can be checked by observing that

$\dot{I}(\mathbf{x},\dot{\mathbf{x}})=\frac{\partial
I}{\partial\mathbf{x}}\,\dot{\mathbf{x}}+\frac{\partial
I}{\partial\dot{\mathbf{x}}}\,\ddot{\mathbf{x}}=0\,,$ (6)

where $\partial/\partial\mathbf{x}$ denotes here the gradient with respect to
$\mathbf{x}$ and analogously for $\partial/\partial\dot{\mathbf{x}}$.

Since for light in isotropic media
$n(\mathbf{x})=1/\lVert\dot{\mathbf{x}}\rVert$ (taking, for definiteness, the
speed of light in vacuum as 1), only the level set of $I=1$ is of optical
interest.

On the other hand, the Hamilton–Jacobi equation [10] associated to the
extremal problem (1) is the eikonal equation:

$(\nabla\mathcal{S})^{2}=n^{2}(\mathbf{x})\,,$ (7)

a term coined by Bruns as early as 1895 [11]. This equation can alternatively
be obtained as an asymptotic limit (for short wavelengths) of Maxwell’s
equations [5]: the real scalar function $\mathcal{S}(\mathbf{x})$ represents
the optical path for a locally plane wave.

The characteristics associated to the first-order partial differential
equation (7) are [12, 13, 14]

$\dot{\mathbf{x}}=\frac{\nabla\mathcal{S}}{(\nabla\mathcal{S})^{{}^{2}}}\,.$
(8)

This shows that the rays are orthogonal to the level sets of
$\mathcal{S}(\mathbf{x})$, which are called wavefronts. The derivative along
the streamlines of the vector field
$\frac{\nabla\mathcal{S}}{(\nabla\mathcal{S})^{{}^{2}}}$ is

$\dot{\mathcal{S}}=\nabla\mathcal{S}\cdot\dot{\mathbf{x}}=1\,,$ (9)

which ensures that the level sets $\mathcal{S}(\mathbf{x})=C$ are transported
by the rays $\mathbf{x}$.

The vector field $\frac{\nabla\mathcal{S}}{(\nabla\mathcal{S})^{{}^{2}}}$ is
complete (i.e., its flow is defined $\forall t\in\mathbb{R}$) since

$\left\lVert\frac{\nabla\mathcal{S}}{(\nabla\mathcal{S})^{{}^{2}}}\right\rVert=\frac{1}{n}\leq
1\,.$ (10)

This vector field defines the symmetry group of the level sets
$\mathcal{S}(\mathbf{x})=C$ and this, in turn, implies that the level curves
of $\mathcal{S}(\mathbf{x})$ are topological straight lines (or planes). This
is confirmed by a direct integration of (9):
$\mathcal{S}(t)=\mathcal{S}_{0}+t$. These conclusions are no longer valid,
however, when (3) is only satisfied locally in a region
$\Omega\subset\mathbb{R}^{m}$; for example, when $\mathcal{S}$ vanishes at
some points. In that case, the level sets of $\mathcal{S}$ can have many
interesting forms.

It is straightforward to check that all the solutions of (8) do satisfy the
ray equation (3). Therefore, once we know the wavefronts, the rays can be
always directly determined: this is the backbone of the ray–wavefront duality.

If instead of the vector field
$\frac{\nabla\mathcal{S}}{(\nabla\mathcal{S})^{{}^{2}}}$ we consider a smooth
field $\mathbf{X}$, the question arises of whether or not functions
$n(\mathbf{x})$ and $\lambda(\mathbf{x})$ can be found such that all the
solutions of

$\dot{\mathbf{x}}=\lambda(\mathbf{x})\,\mathbf{X}(\mathbf{x})$ (11)

satisfy the ray equation (3). A sufficient condition can be directly found by
introducing $\dot{\mathbf{x}}=\mathbf{X}/(n\lVert\mathbf{X}\rVert)$ into (3),
getting

$\frac{d}{dt}\left(\frac{\mathbf{X}}{n\,\lVert\mathbf{X}\rVert}\right)+\frac{2\mathbf{X}(\nabla
n\cdot\mathbf{X})}{n^{3}\,\lVert X\rVert^{2}}-\frac{\nabla n}{n^{3}}=0\,.$
(12)

This constitutes a first-order set of partial differential equations for
$n(\mathbf{x})$ that, in general, has no solutions for $n(\mathbf{x})$ when
$\mathbf{X}$ is given.

##  III Rays in $\mathbb{R}^{2}$

In this section, we restrict our attention to the case of two-dimensional
vector fields $\mathbf{X}$. Then, we can show that a function
$\lambda(\mathbf{x})$ can be always found such that locally we have

$\lambda(\mathbf{x})\,\mathbf{X}(\mathbf{x})=\frac{\nabla\mathcal{S}}{(\nabla\mathcal{S})^{2}},$
(13)

inside a Euclidean ball $B_{0}$ of center $P_{0}$ [with $\mathbf{X}(P_{0})\neq
0$]. The proof is simple. Let $\mathbf{X}=(X,Y)^{\top}$ (the superscript
$\top$ being the transpose) be a smooth field and
$\mathbf{X}_{\perp}=(-Y,X)^{\top}$. Let $\mathcal{S}(x,y)$ be a local first
integral of $\mathbf{X}_{\perp}$ in a neighborhood of the point
$P_{0}\in\mathbb{R}^{2}$ such that $\mathbf{X}_{\perp}(P_{0})\neq 0$ and
$\nabla\mathcal{S}(x,y)|_{P_{0}}\neq 0$. Then, (13) defines $\lambda$, since
by construction $\mathbf{X}$ and $\nabla\mathcal{S}$ are parallel in the ball
$B_{0}$.

Some remarks seem in order here. Assume that the vector field
$\lambda\mathbf{X}$ has orbits near $P_{0}$ that are bounded for $t>0$. This
is fulfilled when the orbits are compact (case of cyclic orbits) or tend to a
compact set $K$ for $t>0$. Then, the solutions $\lambda\mathbf{X}$ satisfy the
ray equation on $B_{0}$, and since they are analytic functions of $t$ for any
$t\geq 0$, by prolongation, the ray equation will also be satisfied for any
$t$. Therefore, orbits of plane vector fields tending to an equilibrium point
[15] or a limit cycle can be considered, conveniently parametrized, as
solutions of the ray equation.

We recall that Thom’s theorem [16] implies that all the orbits of a
$\mathbb{R}^{2}$ analytic vector field $\mathbf{X}$ of type gradient cannot
spiralize around an equilibrium point of $\mathbf{X}$. This means that near a
focus of $\mathbf{X}$, (13) does not hold, but it holds near a node. Note that
the equilibrium point itself is not a solution of the ray equation, as the
constraint $n^{2}(\mathbf{x})\,\dot{\mathbf{x}}^{2}=1$ is not satisfied when
$n(\mathbf{x})$ is a global function of $\mathbf{x}$.

We conclude this section with two _global_ results on wavefronts for
nonvanishing $\mathbb{R}^{2}$ vector fields. Assume first that a conformal
(i.e., angle-preserving) diffeomorphism $D$ can be found such that
$D(\mathbf{X})=\frac{\partial}{\partial x}$. Then $D^{-1}(x=C)$ and
$D^{-1}(y=C)$ are first integrals of $\mathbf{X}_{\perp}$ and $\mathbf{X}$,
respectively.

Let $\mathbf{X}_{\perp}$ be a nonvanishing divergence-free vector field. Then,
the global function $\mathcal{S}$ satisfying
$\mathbf{X}_{\perp}=(\mathcal{S}_{x},-\mathcal{S}_{y})^{\top}$ is computable
by quadratures and is a global first integral of $\mathbf{X}_{\perp}$.
Therefore, $\mathcal{S}(x,y)=c$ are wavefronts of
$\mathbf{X}=\frac{\nabla\mathcal{S}}{(\nabla\mathcal{S})^{2}}$ since
$\dot{\mathcal{S}}=1$. The divergence-free hypothesis is essential; actually,
Wazewski [17] constructed examples of nonvanishing smooth vector fields
$\mathbf{X}_{\perp}$ free from nontrivial global first integrals.

##  IV Rays in $\mathbb{R}^{3}$

We turn our attention to the three-dimensional vector fields reducible to the
type (8). In general, the eikonal defining the wavefronts does not exist even
locally. To guarantee the existence of local orthogonal surface
$\mathcal{S}(x,y,z)=C$ to the vector field $\mathbf{X}(x,y,z)$, we require

$Xdx+Ydy+Zdz=0\,,$ (14)

which can be recast as

$dz=U\,dx+V\,dy\,,$ (15)

with $U=-X/Z$ and $V=-Y/Z$. The integrability condition of (15) reads

$\frac{\partial U}{\partial y}+\frac{\partial U}{\partial z}V=\frac{\partial
V}{\partial x}+\frac{\partial V}{\partial z}U\,.$ (16)

When (16) is satisfied, we get the function $\mathcal{S}(x,y,z)$ from (15),
such that $\nabla\mathcal{S}$ is parallel to $\mathbf{X}$. Note that equations
(15) and (16) are invariant under the replacement
$\mathbf{X}\mapsto\lambda(x,y,z)\mathbf{X}$.

The vector fields of the type $\mathbf{X}=\nabla\Phi$, where $\Phi$ is a
smooth scalar function, satisfy these equations automatically, and the same
happens for two-dimensional vector fields, writing $dx=-X/Ydy$ instead of
(15).

The vector fields orthogonal to $\mathbf{X}$ lie on the level sets of
$\mathcal{S}$. According to (16), these vector fields form an integrable
distribution of dimension 2 [18], $\mathcal{S}$ being a first integral of it.

Let us explore the situation with some examples. Let $\mathbf{X}$ be

$\mathbf{X}=\begin{pmatrix}y^{2}\\\ z\\\ -y\end{pmatrix}\,.$ (17)

One can immediately check that this field satisfies the integrability
constraint (16). Equation (15) takes now the form (for $z$ constant)

$y\,dx+\frac{z}{y}\,dy=0\,,$ (18)

whose solution, apart from an additive constant, reads

$x=\frac{z}{y}\,.$ (19)

Therefore, we conclude that the family of surfaces $x-z/y=C$ are locally
orthogonal to $\mathbf{X}$. In consequence, the rays are determined by
$\nabla\mathcal{S}=(1,z/y^{2},-1/y)^{\top}$, and the resulting refractive
index is

$n(x,y,z)=\sqrt{1+\frac{1}{y^{2}}+\frac{z^{2}}{y^{4}}}\,,$ (20)

which is not defined on $y=0$. Nonetheless, the vector field (17) on $y=0$ is
orthogonal to the singular plane $y=0$. Note that
$\dot{\mathcal{S}}=\nabla\mathcal{S}\cdot\mathbf{X}\neq F(\mathcal{S})$ so
_global_ wavefronts of $\mathbf{X}$ cannot be obtained.

Figure 1:  Plots of the vector fields (24) (left) and (28) (right).

As a second example, let $\mathbf{X}$ be

$\mathbf{X}=\begin{pmatrix}-y\\\ x\\\ -\lambda(x^{2}+y^{2})\end{pmatrix}\,.$
(21)

The integrability condition (16) holds when $\lambda(u)=Cu$, where
$C\in\mathbb{R}$. If, for simplicity, we fix $C=1$, (14) becomes

$-y\,dx+x\,dy+(x^{2}+y^{2})dz=0\,,$ (22)

with solution give a local family of transversals to $\mathbf{X}$:

$\mathcal{S}(x,y,z)=z-\arctan\left(\frac{x}{y}\right)\,.$ (23)

The wavefront condition $\dot{\mathcal{S}}=1$ is not satisfied because
$\dot{\mathcal{S}}=\nabla\mathcal{S}\cdot\mathbf{X}=1+x^{2}+y^{2}$, and
accordingly the vector field does not admit wavefronts.

Let now $\mathbf{X}$ be the vector field

$\mathbf{X}=\begin{pmatrix}-y\\\ x\\\ 1\end{pmatrix}\,,$ (24)

whose orbits are the helices

$\displaystyle x(t)=a\,\cos(t+\varphi)\,,\quad y(t)=a\,\sin(t+\varphi)\,,\quad
z(t)=t\,,$ (25)

with $a,\varphi\in\mathbb{R}$, $a\geq 0$. In Fig. 1, we plot the integral
curves of this vector field, showing an intriguing chiral behavior. One can
check that, in this case, the equation

$dz=-y\,dx+x\,dy$ (26)

does not satisfy the integrability condition (16). Nevertheless, the rays (25)
do satisfy the ray equation when

$n(x,y,z)=\frac{1}{\sqrt{x^{2}+y^{2}}}\,,$ (27)

where we take the coordinates normalized in such a way that $x^{2}+y^{2}<1$.
We thus conclude that rays without orthogonal wavefronts exist in this case.

One could think that this strange behavior occurs only in very special
inhomogeneous media that impart exotic rotational behaviors. This is not the
case, as the following example clearly demonstrates: the vector field

$\mathbf{X}=\begin{pmatrix}\displaystyle\frac{x}{y-1}\\\ 1\\\
\displaystyle\frac{z}{y}\end{pmatrix}\,,$ (28)

originates the congruence

$\frac{x-a}{a}=\frac{y}{-1}=\frac{z}{-b}\,,$ (29)

where $a,b\in\mathbb{R}$. Here, the rays are straight lines, as we can
appreciate in Figure 1, and therefore light is propagating in a homogeneous
medium, with a constant refractive index. These rays pass simultaneously by
the axis $X$ and the axis $y=1$. The integrability (16) does not hold either
here, so we are dealing with a rectilinear non-normal congruence.

To conclude this section, we show that for any solution $\mathbf{x}(t)$ of the
ray equation, we can locally construct a vector field $\mathbf{X}(\mathbf{x})$
with local orthogonal surfaces and having $\mathbf{x}(t)$ as one of its
solutions.

In fact, let $\mathcal{C}$ be a local graph of $\mathbf{x}(t)$ ($t\in[a,b]$).
$\mathcal{C}$ is a simple smooth curve in $\mathbb{R}^{3}$ (no self-
intersections are allowed). Consider now the infinite family of normal planes
$\pi_{\alpha}$ ($\alpha\in\mathbb{R}$) to $\mathcal{C}$ at $P\in\mathcal{C}$
and small bits $B_{\alpha}$ of them near $P=P(\alpha)$. By making these pieces
sufficiently small, we can make them disjoint. Define now $\mathbf{X}$ on
$B_{\alpha}$ in this way:

$\mathbf{X}(P)=\left.\dot{\mathbf{x}}(t)\right|_{P}\,,\qquad\mathbf{X}(B_{\alpha}\backslash
P)=X_{0}(\alpha)\,,$ (30)

with $\mathbf{X}_{0}(\alpha)$ being any vector field extending
$\left.\dot{\mathbf{x}}(t)\right|_{P}$ smoothly and orthogonal to $B_{\alpha}$
at each of its points. This local vector field $\mathbf{X}$ obviously has
$B_{\alpha}$ as local orthogonal transversals near $\mathcal{C}$.

##  V Rays on Level Sets of the Refractive Index

We start by considering rays lying on the level set $n=n_{0}$, assuming that
this is a surface, so that $\nabla n|_{n_{0}}\neq 0$. Writing
$\ddot{\mathbf{x}}(t)=\dot{v}\bm{\tau}+v^{2}/\varrho\,\bm{\nu}$, with
$v=\lVert\dot{\mathbf{x}}\rVert$, $\bm{\tau}$ and $\bm{\nu}$ the unitary
vector tangent and normal to $\mathbf{x}(t)$ and $\varrho$ is the radius of
curvature of $\mathbf{x}(t)$. Since we are in a surface, and
$v^{2}=1/n_{0}^{2}$, the ray equation gives in this case that $v$ is constant
and determined by

$\frac{v^{2}}{\varrho}\bm{\nu}=\frac{\nabla n}{n^{3}}\,,$ (31)

and, consequently, the graph of $\mathbf{x}(t)$ is a geodesic on $n=n_{0}$.

On the other hand, if $\lVert\nabla n\rVert$ has a constant value on
$n=n_{0}$, it follows from (31) that $\varrho=n_{0}/\lVert\nabla n\rVert$ has
a constant value along $\mathbf{x}(t)$. This assumption holds when
$n(\mathbf{x})$ is of one of the following forms:

  1. 1.

$n(x^{2}+y^{2}+z^{2})$

  2. 2.

$n(x^{2}+y^{2})$

  3. 3.

$n(z)$ .

Observe that $\lVert\nabla n\rVert$ is also constant when $n$ satisfies the
eikonal equation

$\lVert\nabla n\rVert^{2}=f(n)$ (32)

for some smooth non-negative function $f(n)>0$.

For the three aforementioned cases, the graph of $\mathbf{x}(t)$ is as
follows:

  1. 1.

A maximum circle on the sphere $n=n_{0}$ (which we parametrize as
$x^{2}+y^{2}+z^{2}=R^{2}$) with radius

$R=\frac{n_{0}(R^{2})}{\lvert 2n^{\prime}_{0}(R^{2})R\rvert}\,.$ (33)

  2. 2.

A helix on the cylinder $x^{2}+y^{2}=R^{2}$ and $\varrho$ constant. The value
of $\nabla n|_{x^{2}+y^{2}=R^{2}}$ implies that straight lines parallel to the
$z$-axis cannot be light rays.

  3. 3.

A straight line $L$ on the plane $n(z)=n_{0}$, say $z=0$.

The infinity of solutions obtained in this way can be understood by noticing
that the ray equation is symmetrical under rotations when
$n(x^{2}+y^{2}+z^{2})$, under rotations around the $z$ axis and translations
along the $z$ axis when $n(x^{2}+y^{2})$ and under translations along the $x$
and $y$ axes when $n(z)$.

Consider now the case in which the level set $n=n_{0}$ is not a surface but a
curve $\mathcal{C}$. In this case, it is trivial to show that $\nabla
n|_{n=n_{0}}=0$. Assume in addition that $\mathcal{C}$ is a straight line $L$
(say, the $z$ axis). Then, it is easy to check that
$\mathbf{x}(t)=(0,0,t/n_{0})^{\top}$ is a solution of the ray equation.

Other straight-line solutions that are global in $t$ are obtained when $\nabla
n|_{L}$ is parallel to $L$. In this case, the ray equation has a solution of
the form $\mathbf{x}(t)=(0,0,z(t))^{\top}$, when $z(t)$ satisfies

$\ddot{z}+A(z)\dot{z}^{2}=B(z)\,,\qquad
A(z)=\left.\frac{\partial_{z}n}{n}\right|_{z}\,,\quad
B(z)=\left.\frac{\partial_{z}n}{n^{3}}\right|_{z}\,.$ (34)

This equation is integrable and reduces to the linear equation

$\frac{1}{2}\frac{du}{dz}+Au=B\,,$ (35)

with $u=\dot{z}^{2}$. Since $n^{2}\dot{\mathbf{x}}^{2}=1$,
$0<\dot{z}^{2}(t)=1/n^{2}<1$ and so

$\frac{dz}{dt}=\pm\sqrt{u(z,C)}\,,$ (36)

where $C$ is an integration constant. We finally get

$t-t_{0}=\int_{z_{0}}^{z}\frac{dz}{+\sqrt{u(z,C)}}\geq\int_{z_{0}}^{z}dz$ (37)

and therefore the solution (36) is defined for every $t\in\mathbb{R}$.

##  VI Concluding Remarks

In summary, we have challenged a traditional conviction by showing that non-
normal congruences are allowed by the laws of geometrical optics. Even if
wavefront detection is becoming a crucial tool for many modern optical
technologies [19], our findings seem to suggest that the role of rays, as the
carriers of the observable energy, should not be overlooked.

This research was funded by Ministerio de Ciencia, Innovación y Universidades
Grant number PGC2018-099183-B-I00. We thank Juan J. Monzón. José L. Romero and
Aaron Z. Goldberg for useful discussions.

## References

  * Stavroudis [2006] Stavroudis, O.N.  The Mathematics of Geometrical and Physical Optics; Wiley-VCH: Weinheim, 2006.
  * Iizuka [2008] Iizuka, K.  Engineering Optics; Springer: New York, 2008.
  * Luneburg [1966] Luneburg, R.K.  Mathematical Theory of Optics; University of California Press: Berkeley, 1966.
  * Kline and Kay [1965] Kline, M.; Kay, I.W.  Electromagnetic Theory and Geometrical Optics; Wiley: New York, 1965.
  * Born and Wolf [1999] Born, M.; Wolf, E.  Principles of Optics; Cambridge University Press: Cambridge, 1999.
  * Synge [1937] Synge, J.L.  Geometrical Optics; Cambridge University Press: Cambridge, 1937.
  * Carathéodory [1937] Carathéodory, C.  Geometrische Optik; Springer: Berlin, 1937.
  * Elsgolts [2003] Elsgolts, L.  Differential Equations and the Calculus of Variations; University Press of the Pacific: Honolulu, 2003.
  * Lakshminarayanan et al. [2002] Lakshminarayanan, V.; Ghatak, A.; Thyagarajan, K.  Lagrangian Optics; Springer: Boston, 2002.
  * Arnold [1989] Arnold, V.  Mathematical Methods of Classical Mechanics; Springer: Berlin, 1989.
  * Bruns [1895] Bruns, H.  Das Eikonal.  Abh. Math.-Phys. Kl. Sachs. Akad. Wiss. 1895, XXI, 325–435.
  * Kravtsov and Orlov [1990] Kravtsov, Y.A.; Orlov, Y.I.  Geometrical Optics of Inhomogeneous Media; Springer: Berlin, 1990.
  * Siddiqi et al. [1999] Siddiqi, K.; Tannenbaum, A.; Zucker, S.W.  A Hamiltonian approach to the eikonal equation. In Lecture Notes in Computer Science; Springer: Berlin, 1999; Volume 1654, pp. 1–13.
  * Slawinski and Bóna [2011] Slawinski, M.; Bóna, A.  Ray Theory: Characteristics and Asymptotics; World Scientific: Singapore, 2011.
  * Guckenheimer and Holmes [1983] Guckenheimer, J.; Holmes, P.  Nonlinear Oscillations, Dynamical Systems, and Bifurcations of Vector Fields; Springer: New York, 1983.
  * Kurdyka et al. [2000] Kurdyka, K.; Mostowski, T.; Parusinski, A.  Proof of the Gradient Conjecture of R. Thom.  Ann. Math. 2000, 152, 763–792.
  * Wazewski [1934] Wazewski, T.  Sur un problème de caractère intégral rélatif à l’équation ${\partial z\over\partial x}+Q(x,y){\partial z\over\partial y}=0$.  Mathematica 1934, 8, 103–116.
  * Boothby [2003] Boothby, W.M.  An Introduction to Differentiable Manifolds and Riemannian Geometry; Academic: London, 2003.
  * Geary [1995] Geary, J.M.  Introduction to Wavefront Sensors; SPIE Press: Bellingham, 1995.