<p>Generally, you fuse image features (CNN) and question features (RNN), pass these to another network with softmax output that corresponds to a one-word answer. See here: <a href="https://arxiv.org/pdf/1505.00468v6.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1505.00468v6.pdf</a></p>

<p>I can imagine trying a decoder network to produce sentences for answers, but those will be harder to evaluate.</p>
What is image captioning and how is it used in Visual Question Answer?
<p>I am trying to get me head around this but having difficulty.</p>

<p>As far as I understand: </p>

<blockquote>
  <p><a href="https://towardsdatascience.com/image-captioning-in-deep-learning-9cd23fb4d8d2" rel="nofollow noreferrer">Image Captioning</a> is the process of generating textual description of an image. It uses both Natural Language Processing and Computer Vision to generate the captions.</p>
</blockquote>

<hr>

<p>And from <a href="https://arxiv.org/abs/1412.6632" rel="nofollow noreferrer">this</a> paper:</p>

<blockquote>
  <p>It directly models the probability distribution of generating a word given previous words and an image.</p>
</blockquote>

<hr>

<p>So if I understand correctly, using some model which takes image and previous text as input, it generates probabilities for the next word. </p>

<p>Taking an example from "Deep Visual-Semantic Alignments for Generating Image Descriptions" <a href="https://cs.stanford.edu/people/karpathy/cvpr2015.pdf" rel="nofollow noreferrer">paper</a></p>

<p><a href="https://i.stack.imgur.com/RlRLo.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/RlRLo.png" alt="enter image description here"></a></p>

<hr>

<p>But how is that used in VQA (Visual Question Answering is a research area about building a computer system to answer questions presented in an image and a natural language. )? </p>

<p>Is the answer to a question taken from the caption generated from the image? </p>