Embodiments of a multimodal question answering (mQA) system are presented to answer a question about the content of an image. In embodiments, the model comprises four components: a Long Short-Term Memory (LSTM) component to extract the question representation; a Convolutional Neural Network (CNN) component to extract the visual representation; an LSTM component for storing the linguistic context in an answer, and a fusing component to combine the information from the first three components and generate the answer. A Freestyle Multilingual Image Question Answering (FM-IQA) dataset was constructed to train and evaluate embodiments of the mQA model. The quality of the generated answers of the mQA model on this dataset is evaluated by human judges through a Turing Test.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A computer-implemented method that improves computer-user interaction by generating an answer to a question input related to an image input, the method comprising: receiving a question input in a natural language form; receiving an image input related to the question input; and inputting the question input and the image input into a multimodal question answering (mQA) model to generate an answer comprising multiple words generated sequentially, the mQA model comprising: a first component that encodes the question input into a dense vector representation; a second component to extract a visual representation of the image input; a third component to extract representation of a current word in the answer and its linguistic context; and a fourth component to generate a next word after the current word in the answer using a fusion comprising the dense vector representation, the visual representation, and the representation of the current word. 2. The computer-implemented method of claim 1 wherein the first component is a first long short term memory (LSTM) network comprising a first word embedding layer and a first LSTM layer. 3. The computer-implemented method of claim 2 wherein the third component is a second LSTM network comprising a second word embedding layer and a second LSTM layer. 4. The computer-implemented method of claim 3 wherein the first word-embedding layer shares a weight matrix with the second word-embedding layer. 5. The computer-implemented method of claim 3 wherein the first LSTM layer does not share a weight matrix with the second LSTM layer. 6. The computer-implemented method of claim 1 wherein the second component is a deep Convolutional Neural network (CNN). 7. The computer-implemented method of claim 1 wherein the CNN is pre-trained and is fixed during question answering training. 8. The computer-implemented method of claim 1 wherein the first, the third, and the fourth components are jointly trained together. 9. The computer-implemented method of claim 3 wherein the fourth component is a fusing component comprising: a fusing layer that fuses information from the first LSTM layer, the second LSTM layer, and the second component to generate a dense multimodal representation for the current word in the answer; an intermediate layer that maps the dense multimodal representation in the fusing layer to a dense word representation; and a softmax layer that predicts a probability distribution of the next word in the answer. 10. A computer-implemented method that improves computer-user interaction by generating an answer to a question input related to an image input, the method comprising: extracting a semantic meaning of a question input using a first long short term memory (LSTM) component comprising a first word-embedding layer and a first LSTM layer; generating a representation of an image input related to the question input using a deep Convolutional Neural network (CNN) component; extracting a representation of a current word of the answer using a second LSTM component comprising a second word-embedding layer and a second LSTM layer; and fusing the semantic meaning, the representation of the image input, and a representation of the current word of the answer to predict a next word of the answer. 11. The computer-implemented method of claim 10 wherein the first word-embedding layer shares a weight matrix with the second word-embedding layer. 12. The computer-implemented method of claim 10 wherein the first LSTM layer does not share a weight matrix with the second LSTM layer. 13. The computer-implemented method of claim 10 wherein the deep CNN is pre-trained and is fixed during question answering training. 14. The computer-implemented method of claim 11 wherein predicting the next word in the answer further comprises: fusing information from the first LSTM layer, the second LSTM layer, and the CNN in a fusion layer to generate a dense multimodal representation for the current answer word; mapping in an intermediate layer the dense multimodal representation to a dense word representation; and predicting in a softmax layer a probability distribution of the next word in the answer. 15. The computer-implemented method of claim 14 wherein the dense multimodal representation in the fusion layer is a non-linear activation function. 16. The computer-implemented method of claim 15 wherein the non-linear activation function is a scaled hyperbolic tangent function. 17. The computer-implemented method of claim 15 wherein the first word-embedding layer, the second word-embedding layer, and the softmax layer share a weight matrix. 18. A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by one or more processors, causes the steps to be performed comprising: responsive to receiving from a user a question input, extracting a semantic meaning of the question input; responsive to receiving an image input related to the question input, generating a representation of the image input; starting with a start sign as a current word in an answer to the question input based upon the image input, generating a next answer word based on a fusion of the semantic meaning, the representation of the image input, and a semantic current answer word and adding it to the answer; repeating the next answer word generating step until an end sign of the answer is generated; and responsive to obtaining an end sign, outputting the answer. 19. The non-transitory computer-readable medium or media of claim 18 wherein generating the next answer word comprises: fusing information from the first LSTM layer, the second LSTM layer, and the CNN in a fusion layer for a dense multimodal representation for the current answer word; mapping in an intermediate layer the dense multimodal representation to a dense word representation; and predicting in a softmax layer a probability distribution of the next word in the answer. 20. The non-transitory computer-readable medium or media of claim 19 wherein the softmax layer has a weight matrix to decode the dense word representation into a pseudo one-word representation.