Visual Question Answering ( VQA ) is a joint task that aims to answer questions based on the given images . The correct analysis of multiple album aggregate issues to remain a key issue in the VQA case , especially when answering question from multiple albums , how to correctly understand album images and corresponding question is an urgent problem . Furthermore , ten reference videos and 150 distorted videos in the LIVE video database were used to test the metric . Moreover , based on the results of evaluating the videos in LIVE , VQEG , IRCCyN , EPFL-PoliMI , IVP , CSIQ , and Lisbon databases , the performance of BRVPVC is respectively compared with that of six full-reference ( FR ) metrics and ten NR VQA metrics . The results show that our VQA metric has a higher accuracy than six common FR VQA metrics and eight NR VQA metrics , and it is close to other two NR VQA metrics in accuracy . Further , a feature fusion strategy is proposed to guide the training of the navigator without much additional computational cost . The ablation experiments show that our method boosts the performance of VQA module by 4.2 % ( 68.99 % vs 64.73 % ) and leads to 3.6 % ( 48.59 % vs 44.98 % ) overall improvement in EQA accuracy . C2VQA , which expands on Visual Query Answering ( VQA ) and the Visual Turing Test ( VTT ) , engages deep semantic alignment and joint embedding using deep learning ( DL ) for image analysis , vector space as skip-grams and long-term dependencies as gated recurrent networks for context prediction , and multi-strategy learning including conformal prediction for control and meta-reasoning . C2VQA would engage in purposeful dialog to address and correct for misinformation and uncertainty and considers behavior to model realistic VQA problems characteristic of open rather than closed set VQA . Additionally , the absence of a video reference in streaming systems oblige the creation of No-Reference quality metrics , because the content is not available to compare the video received with the original source . For that reason , this paper proposes a novel algorithm based on the detection transmission artefacts using visual VQA techniques in order to improve users ' Quality of Experience ( QoE ) . The algorithm is based on the detection of main streaming visual errors and artefacts , such as color degradation , frozen frames , complete absence of video or packet losses . We present a technique to annotate multiple organs shown in 2-D abdominal/pelvic CT images using CBIR . This annotation task is motivated by our research interests in visual question-answering ( VQA ) . We aim to apply results from this effort in Open-iSM , a multimodal biomedical search engine developed by the National Library of Medicine ( NLM ) . Then we propose saliency-based methods for both image QA ( IQA ) and video QA ( VQA ) by adding weights related to saliency features to these original IQA or VQA criteria . Experimental results show that our saliency-based strategy can approach more closely to human subjective assessment compared with these original IQA or VQA methods and does not take more time because of the fast PQFT algorithm . In this case , VQA must work in real-time to assess the video clarity level . VQA assessment results are valuable information for the decision-making process . Thus , the antenna can rotate automatically looking for the ideal direction without user ’s control . We also conducted a subjective video quality assessment study using this data , wherein each video was assessed by 36 unique subjects . We evaluated several top-performing No-Reference IQA and VQA algorithms on the new database and find insights on how real-world in-capture distortions challenge both human subjects as well as automatic perceptual quality prediction models . Predicting If a Crowd Will Agree on the Answer . Visual question answering ( VQA ) systems are emerging from a desire to empower users to ask any natural language question about visual content and receive a valid answer in response . However , close examination of the VQA problem reveals an unavoidable , entangled problem that multiple humans may or may not always agree on a single answer to a visual question . ‘ encoder ’ to produce a fixed - length vector representation , which is then fed into the ‘ decoder ’ RNN to generate a caption . Visual Question Answering ( VQA ) is a more recent challenge than image captioning . It is distinct from many problems in Computer Vision because the question to be answered is not determined until run time . In addition , it achieves an accuracy of 70.98 % on the Toronto COCO - QA , while the latest state of the art is 61.60 % . On the VQA evaluation server ( which does not publish ground truth answers for its test set ) , we also produce the state - of - the - art result , which is 59.50 % . A preliminary version of this work was published at CVPR 2016 . More results can be found in the supplementary material . section : A VQA Model with External Knowledge The key differentiator of our VQA model is that it is able to usefully combine image information with that extracted from a Knowledge Base , within the LSTM framework . Indeed , at the time of submitting this paper , our image captioning model outperforms the state - of - the - art on several captioning datasets . Secondly , in this paper we have shown that it is possible to extend the state - of - the - art RNN - based VQA approach so as to incorporate the large volumes of information required to answer general , open - ended , questions about images . The knowledge bases which are currently available do not contain much of the information which would be beneficial to this process , but nonetheless can still be used to significantly improve performance on questions requiring external knowledge ( such as ’ Early works on visual question answering ( VQA ) involved encoding the question using an RNN , encoding the image using a CNN and combining them to answer the question antol2015vqa , Malinowski2015AskYN . Attention mechanisms have also been successfully employed for the VQA task and can be broadly clustered based on the granularity of their attention and the approach to construct the attention matrix . At the coarse level of granularity , the question attends to different patches in the image Zhu2015Visual7WGQ , xiong2016dynamic . Figure [ reference ] shows a snapshot of the demo . People could play with the demo to see the strength and weakness of VQA model . section : Concluding Remarks The motivation for Pythia comes from the following observation – a majority of today ’s Visual Question Answering ( VQA ) models fit a particular design paradigm , with modules for question encoding , image feature extraction , fusion of the two ( typically with attention ) , and classification over the space of answers . The long - term goal of Pythia is to serve as a platform for easy and modular research & development in VQA and related directions like visual dialog . For the VQA challenge ( see section 6 ) , test - dev is used for debugging and validation experiments and allows for unlimited submission to the evaluation server . Test - standard is the ' default ' test data for the VQA competition . When comparing to the state of the art ( e.g. , in papers ) , results should be reported on test - standard . Abstract Scenes section : VQA DATASET ANALYSIS In this section , we provide an analysis of the questions and answers in the VQA train dataset . , we can first find the basket and the cat objects , and then compare their locations . We propose a new deep learning approach to VQA that incorporates explicit spatial attention , which we call the Spatial Memory Network VQA ( SMem - VQA ) . Our approach is based on memory networks , which have recently been proposed for text Question Answering ( QA ) . Visual question answering is fundamentally compositional in nature-a question like where is the dog ? shares substructure with questions like what color is the dog ? This new real-world task provides a new perspective for studying scene text related problems . We also demonstrate the uniqueness of our task via a comparison between our problem and a typical Visual Question Answering problem . Recent progress in big data and computer vision with deep learning models has gained a lot of attention . Deep learning has been performed on tasks such as image classification , object detection , image segmentation , image captioning , visual question and answering , using large collections of annotated images . This calls for more curated large image datasets with clearer descriptions , cleaner contents , and diversified usability . Nevertheless lacking of global consciousness in image caption has not been completely solved . Separation from the bottom-up visual attention mechanism and the top-down visual attention mechanism has been widely used in image description and visual question and answers . In this article , we put forward image description based on a global attention mechanism research methods . Humans can easily perform tasks that use vision and language jointly , such as describing a scene and answering questions about objects in the scene and how they are related . Image captioning and visual question & answer are two popular research tasks that have emerged from advances in deep learning and the availability of datasets that specifically address these problems . However recent work has shown that deep learning based solutions to these tasks are just as brittle as solutions for only vision or only natural language tasks . CognitiveCam is a Visual Question Answering application designed for helping the visually impaired . A user can ask any question pointing towards the object , the app will then attempt to provide the relevant answer . These are merged with the word embedding or with a sentence embedding of the question to predict the answer . This work was successfully submitted to the Visual Question Answering Challenge 2016 , where it achieved a 53,62 % of accuracy in the test dataset . The developed software has followed the best programming practices and Python code style , providing a consistent baseline in Keras for different configurations . On the other hand , a direct input of 3D structure is of low efficiency due to the sparse 3D matrix , and is also prevented by limited number of co-crystal structures available for training . In this work , we propose an end-to-end deep learning framework to predict the interactions by representing proteins with 2D distance map from monomer structures ( Image ) , and drugs with molecular linear notation ( String ) , following the Visual Question Answering mode . For an efficient training of the system , we introduced a dynamic attentive convolutional neural network to learn fixed-size representations from the variable-length distance maps and a self-attentional sequential model to automatically extract semantic features from the linear notations . Current visual question answering datasets do not consider the rich semantic information conveyed by text within an image . In this work , we present a new dataset , ST-VQA , that aims to highlight the importance of exploiting high-level semantic information present in images as textual cues in the Visual Question Answering process . We use this dataset to define a series of tasks of increasing difficulty for which reading the scene text in the context provided by the visual information is necessary to reason and generate an appropriate answer . We further show that the same mechanism can be used to incorporate external knowledge , which is critically important for answering high level visual questions . Specifically , we design a visual question answering model that combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image - based questions . It particularly allows questions to be asked where the image alone does not contain the the information required to select the appropriate answer . paragraph : V - LSTM baseline The LSTM model described above is augmented with a visual component following a standard Visual Question Answering baseline model . Following initial representation of P and H in 512D vectors through an LSTM , a fully - connected layer projects the L2 - normalized 4096D image vector coming from the penultimate layer of a VGGnet16 Convolutional Neural Network to a reduced 512D vector . section : Introduction We present Pythia v0.1 , a modular framework for Visual Question Answering research , which formed the basis for the winning entry to the VQA Challenge 2018 from Facebook AI Research ( FAIR ) ’s A - STAR team . A major drawback of existing models is that they do not have any explicit notion of object position , and do not support the computation of intermediate results based on spatial attention . Our intuition is that answering visual questions often involves looking at different spatial regions and comparing their contents and / or locations . For example , to answer the questions in Fig . Eight papers were accepted for this issue . They can be roughly divided into three categories : ( 1 ) image question answering , ( 2 ) video question answering , and ( 3 ) other applications related to visual question answering . The first category of this special issue contains three papers that focus on generating answers for the questions over a single image . Visual question answering ( VQA ) aims at answering questions about the visual content of an image or a video . Currently , most work on VQA is focused on image-based question answering , and less attention has been paid into answering questions about videos . However , VQA in video presents some unique challenges that are worth studying : it not only requires to model a sequence of visual features over time , but often it also needs to reason about associated subtitles . Progress in language and image understanding by machines has sparkled the interest of the research community in more open-ended , holistic tasks , and refueled an old AI dream of building intelligent machines . We discuss a few prominent challenges that characterize such holistic tasks and argue for " question answering about images " as a particular appealing instance of such a holistic task . In particular , we point out that it is a version of a Turing Test that is likely to be more robust to over-interpretations and contrast it with tasks like grounding and generation of descriptions . Free-Form Visual Question Answering Based on Knowledge from External Sources . We propose a method for visual question answering which combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image-based questions . This allows more complex questions to be answered using the predominant neural network-based approach than has previously been possible . In this paper , we propose to employ the convolutional neural network ( CNN ) for the image question answering ( QA ) . Our proposed CNN provides an end-to-end framework with convolutional architectures for learning not only the image and question representations , but also their inter-modal interactions to produce the answer . An image related question defines a specific visual task that is required in order to produce an appropriate answer . The answer may depend on a minor detail in the image and require complex reasoning and use of prior knowledge . This work aims to address the problem of imagebased question-answering ( QA ) with new models and datasets . In our work , we propose to use recurrent neural networks and visual semantic embeddings without intermediate stages such as object detection and image segmentation . document : Stacked Attention Networks for Image Question Answering This paper presents stacked attention networks ( SANs ) that learn to answer natural language questions from images . We further show that the same mechanism can be used to incorporate external knowledge , which is critically important for answering high level visual questions . Specifically , we design a visual question answering model that combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image - based questions . It particularly allows questions to be asked where the image alone does not contain the the information required to select the appropriate answer . Answering visual questions need acquire daily common knowledge and model the semantic connection among different parts in images , which is too difficult for VQA systems to learn from images with the only supervision from answers . Meanwhile , image captioning systems with beam search strategy tend to generate similar captions and fail to diversely describe images . In this paper we first propose a method of incorporating high-level concepts into the successful CNN-RNN approach , and show that it achieves a significant improvement on the state-of-the-art in both image captioning and visual question answering . We further show that the same mechanism can be used to incorporate external knowledge , which is critically important for answering high level visual questions . Specifically , we design a visual question answering model that combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image-based questions . In this work , we propose the design of an explainable , human-in-the-loop , and interactive personal assistant system . The assistant will help a Systems Engineer in their daily work environment through complex information management and high-level questionanswering to augment their problem-solving abilities . We describe the fundamental characteristics of the assistant by understanding operational , functional , and system requirements from Systems Engineers and NASA ’s Systems Engineering Handbook . In this paper we first propose a method of incorporating high - level concepts into the successful CNN - RNN approach , and show that it achieves a significant improvement on the state - of - the - art in both image captioning and visual question answering . We further show that the same mechanism can be used to incorporate external knowledge , which is critically important for answering high level visual questions . Specifically , we design a visual question answering model that combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image - based questions . Antol et al . proposed a large - scale open - ended VQA dataset based on COCO , which is called VQA . Inspired by Xu et al . who encode visual attention in the Image Captioning , propose to use the spatial attention to help answering visual questions . formulate the VQA as a classification problem and restrict the answer only can be drawn from a fixed answer space .
Visual Question Answering ( VQA ) is a joint task that aims to answer questions based on the given images . The correct analysis of multiple album aggregate issues to remain a key issue in the VQA case , especially when answering question from multiple albums , how to correctly understand album images and corresponding question is an urgent problem .
Furthermore , ten reference videos and 150 distorted videos in the LIVE video database were used to test the metric . Moreover , based on the results of evaluating the videos in LIVE , VQEG , IRCCyN , EPFL-PoliMI , IVP , CSIQ , and Lisbon databases , the performance of BRVPVC is respectively compared with that of six full-reference ( FR ) metrics and ten NR VQA metrics . The results show that our VQA metric has a higher accuracy than six common FR VQA metrics and eight NR VQA metrics , and it is close to other two NR VQA metrics in accuracy .
Further , a feature fusion strategy is proposed to guide the training of the navigator without much additional computational cost . The ablation experiments show that our method boosts the performance of VQA module by 4.2 % ( 68.99 % vs 64.73 % ) and leads to 3.6 % ( 48.59 % vs 44.98 % ) overall improvement in EQA accuracy .
C2VQA , which expands on Visual Query Answering ( VQA ) and the Visual Turing Test ( VTT ) , engages deep semantic alignment and joint embedding using deep learning ( DL ) for image analysis , vector space as skip-grams and long-term dependencies as gated recurrent networks for context prediction , and multi-strategy learning including conformal prediction for control and meta-reasoning . C2VQA would engage in purposeful dialog to address and correct for misinformation and uncertainty and considers behavior to model realistic VQA problems characteristic of open rather than closed set VQA .
Additionally , the absence of a video reference in streaming systems oblige the creation of No-Reference quality metrics , because the content is not available to compare the video received with the original source . For that reason , this paper proposes a novel algorithm based on the detection transmission artefacts using visual VQA techniques in order to improve users ' Quality of Experience ( QoE ) . The algorithm is based on the detection of main streaming visual errors and artefacts , such as color degradation , frozen frames , complete absence of video or packet losses .
We present a technique to annotate multiple organs shown in 2-D abdominal/pelvic CT images using CBIR . This annotation task is motivated by our research interests in visual question-answering ( VQA ) . We aim to apply results from this effort in Open-iSM , a multimodal biomedical search engine developed by the National Library of Medicine ( NLM ) .
Then we propose saliency-based methods for both image QA ( IQA ) and video QA ( VQA ) by adding weights related to saliency features to these original IQA or VQA criteria . Experimental results show that our saliency-based strategy can approach more closely to human subjective assessment compared with these original IQA or VQA methods and does not take more time because of the fast PQFT algorithm .
In this case , VQA must work in real-time to assess the video clarity level . VQA assessment results are valuable information for the decision-making process . Thus , the antenna can rotate automatically looking for the ideal direction without user ’s control .
We also conducted a subjective video quality assessment study using this data , wherein each video was assessed by 36 unique subjects . We evaluated several top-performing No-Reference IQA and VQA algorithms on the new database and find insights on how real-world in-capture distortions challenge both human subjects as well as automatic perceptual quality prediction models .
Predicting If a Crowd Will Agree on the Answer . Visual question answering ( VQA ) systems are emerging from a desire to empower users to ask any natural language question about visual content and receive a valid answer in response . However , close examination of the VQA problem reveals an unavoidable , entangled problem that multiple humans may or may not always agree on a single answer to a visual question .
‘ encoder ’ to produce a fixed - length vector representation , which is then fed into the ‘ decoder ’ RNN to generate a caption . Visual Question Answering ( VQA ) is a more recent challenge than image captioning . It is distinct from many problems in Computer Vision because the question to be answered is not determined until run time .
In addition , it achieves an accuracy of 70.98 % on the Toronto COCO - QA , while the latest state of the art is 61.60 % . On the VQA evaluation server ( which does not publish ground truth answers for its test set ) , we also produce the state - of - the - art result , which is 59.50 % . A preliminary version of this work was published at CVPR 2016 .
More results can be found in the supplementary material . section : A VQA Model with External Knowledge The key differentiator of our VQA model is that it is able to usefully combine image information with that extracted from a Knowledge Base , within the LSTM framework .
Indeed , at the time of submitting this paper , our image captioning model outperforms the state - of - the - art on several captioning datasets . Secondly , in this paper we have shown that it is possible to extend the state - of - the - art RNN - based VQA approach so as to incorporate the large volumes of information required to answer general , open - ended , questions about images . The knowledge bases which are currently available do not contain much of the information which would be beneficial to this process , but nonetheless can still be used to significantly improve performance on questions requiring external knowledge ( such as ’
Early works on visual question answering ( VQA ) involved encoding the question using an RNN , encoding the image using a CNN and combining them to answer the question antol2015vqa , Malinowski2015AskYN . Attention mechanisms have also been successfully employed for the VQA task and can be broadly clustered based on the granularity of their attention and the approach to construct the attention matrix . At the coarse level of granularity , the question attends to different patches in the image Zhu2015Visual7WGQ , xiong2016dynamic .
Figure [ reference ] shows a snapshot of the demo . People could play with the demo to see the strength and weakness of VQA model . section : Concluding Remarks
The motivation for Pythia comes from the following observation – a majority of today ’s Visual Question Answering ( VQA ) models fit a particular design paradigm , with modules for question encoding , image feature extraction , fusion of the two ( typically with attention ) , and classification over the space of answers . The long - term goal of Pythia is to serve as a platform for easy and modular research & development in VQA and related directions like visual dialog .
For the VQA challenge ( see section 6 ) , test - dev is used for debugging and validation experiments and allows for unlimited submission to the evaluation server . Test - standard is the ' default ' test data for the VQA competition . When comparing to the state of the art ( e.g. , in papers ) , results should be reported on test - standard .
Abstract Scenes section : VQA DATASET ANALYSIS In this section , we provide an analysis of the questions and answers in the VQA train dataset .
, we can first find the basket and the cat objects , and then compare their locations . We propose a new deep learning approach to VQA that incorporates explicit spatial attention , which we call the Spatial Memory Network VQA ( SMem - VQA ) . Our approach is based on memory networks , which have recently been proposed for text Question Answering ( QA ) .
Visual question answering is fundamentally compositional in nature-a question like where is the dog ? shares substructure with questions like what color is the dog ?
This new real-world task provides a new perspective for studying scene text related problems . We also demonstrate the uniqueness of our task via a comparison between our problem and a typical Visual Question Answering problem .
Recent progress in big data and computer vision with deep learning models has gained a lot of attention . Deep learning has been performed on tasks such as image classification , object detection , image segmentation , image captioning , visual question and answering , using large collections of annotated images . This calls for more curated large image datasets with clearer descriptions , cleaner contents , and diversified usability .
Nevertheless lacking of global consciousness in image caption has not been completely solved . Separation from the bottom-up visual attention mechanism and the top-down visual attention mechanism has been widely used in image description and visual question and answers . In this article , we put forward image description based on a global attention mechanism research methods .
Humans can easily perform tasks that use vision and language jointly , such as describing a scene and answering questions about objects in the scene and how they are related . Image captioning and visual question & answer are two popular research tasks that have emerged from advances in deep learning and the availability of datasets that specifically address these problems . However recent work has shown that deep learning based solutions to these tasks are just as brittle as solutions for only vision or only natural language tasks .
CognitiveCam is a Visual Question Answering application designed for helping the visually impaired . A user can ask any question pointing towards the object , the app will then attempt to provide the relevant answer .
These are merged with the word embedding or with a sentence embedding of the question to predict the answer . This work was successfully submitted to the Visual Question Answering Challenge 2016 , where it achieved a 53,62 % of accuracy in the test dataset . The developed software has followed the best programming practices and Python code style , providing a consistent baseline in Keras for different configurations .
On the other hand , a direct input of 3D structure is of low efficiency due to the sparse 3D matrix , and is also prevented by limited number of co-crystal structures available for training . In this work , we propose an end-to-end deep learning framework to predict the interactions by representing proteins with 2D distance map from monomer structures ( Image ) , and drugs with molecular linear notation ( String ) , following the Visual Question Answering mode . For an efficient training of the system , we introduced a dynamic attentive convolutional neural network to learn fixed-size representations from the variable-length distance maps and a self-attentional sequential model to automatically extract semantic features from the linear notations .
Current visual question answering datasets do not consider the rich semantic information conveyed by text within an image . In this work , we present a new dataset , ST-VQA , that aims to highlight the importance of exploiting high-level semantic information present in images as textual cues in the Visual Question Answering process . We use this dataset to define a series of tasks of increasing difficulty for which reading the scene text in the context provided by the visual information is necessary to reason and generate an appropriate answer .
We further show that the same mechanism can be used to incorporate external knowledge , which is critically important for answering high level visual questions . Specifically , we design a visual question answering model that combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image - based questions . It particularly allows questions to be asked where the image alone does not contain the the information required to select the appropriate answer .
paragraph : V - LSTM baseline The LSTM model described above is augmented with a visual component following a standard Visual Question Answering baseline model . Following initial representation of P and H in 512D vectors through an LSTM , a fully - connected layer projects the L2 - normalized 4096D image vector coming from the penultimate layer of a VGGnet16 Convolutional Neural Network to a reduced 512D vector .
section : Introduction We present Pythia v0.1 , a modular framework for Visual Question Answering research , which formed the basis for the winning entry to the VQA Challenge 2018 from Facebook AI Research ( FAIR ) ’s A - STAR team .
A major drawback of existing models is that they do not have any explicit notion of object position , and do not support the computation of intermediate results based on spatial attention . Our intuition is that answering visual questions often involves looking at different spatial regions and comparing their contents and / or locations . For example , to answer the questions in Fig .
Eight papers were accepted for this issue . They can be roughly divided into three categories : ( 1 ) image question answering , ( 2 ) video question answering , and ( 3 ) other applications related to visual question answering . The first category of this special issue contains three papers that focus on generating answers for the questions over a single image .
Visual question answering ( VQA ) aims at answering questions about the visual content of an image or a video . Currently , most work on VQA is focused on image-based question answering , and less attention has been paid into answering questions about videos . However , VQA in video presents some unique challenges that are worth studying : it not only requires to model a sequence of visual features over time , but often it also needs to reason about associated subtitles .
Progress in language and image understanding by machines has sparkled the interest of the research community in more open-ended , holistic tasks , and refueled an old AI dream of building intelligent machines . We discuss a few prominent challenges that characterize such holistic tasks and argue for " question answering about images " as a particular appealing instance of such a holistic task . In particular , we point out that it is a version of a Turing Test that is likely to be more robust to over-interpretations and contrast it with tasks like grounding and generation of descriptions .
Free-Form Visual Question Answering Based on Knowledge from External Sources . We propose a method for visual question answering which combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image-based questions . This allows more complex questions to be answered using the predominant neural network-based approach than has previously been possible .
In this paper , we propose to employ the convolutional neural network ( CNN ) for the image question answering ( QA ) . Our proposed CNN provides an end-to-end framework with convolutional architectures for learning not only the image and question representations , but also their inter-modal interactions to produce the answer .
An image related question defines a specific visual task that is required in order to produce an appropriate answer . The answer may depend on a minor detail in the image and require complex reasoning and use of prior knowledge .
This work aims to address the problem of imagebased question-answering ( QA ) with new models and datasets . In our work , we propose to use recurrent neural networks and visual semantic embeddings without intermediate stages such as object detection and image segmentation .
document : Stacked Attention Networks for Image Question Answering This paper presents stacked attention networks ( SANs ) that learn to answer natural language questions from images .
We further show that the same mechanism can be used to incorporate external knowledge , which is critically important for answering high level visual questions . Specifically , we design a visual question answering model that combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image - based questions . It particularly allows questions to be asked where the image alone does not contain the the information required to select the appropriate answer .
Answering visual questions need acquire daily common knowledge and model the semantic connection among different parts in images , which is too difficult for VQA systems to learn from images with the only supervision from answers . Meanwhile , image captioning systems with beam search strategy tend to generate similar captions and fail to diversely describe images .
In this paper we first propose a method of incorporating high-level concepts into the successful CNN-RNN approach , and show that it achieves a significant improvement on the state-of-the-art in both image captioning and visual question answering . We further show that the same mechanism can be used to incorporate external knowledge , which is critically important for answering high level visual questions . Specifically , we design a visual question answering model that combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image-based questions .
In this work , we propose the design of an explainable , human-in-the-loop , and interactive personal assistant system . The assistant will help a Systems Engineer in their daily work environment through complex information management and high-level questionanswering to augment their problem-solving abilities . We describe the fundamental characteristics of the assistant by understanding operational , functional , and system requirements from Systems Engineers and NASA ’s Systems Engineering Handbook .
In this paper we first propose a method of incorporating high - level concepts into the successful CNN - RNN approach , and show that it achieves a significant improvement on the state - of - the - art in both image captioning and visual question answering . We further show that the same mechanism can be used to incorporate external knowledge , which is critically important for answering high level visual questions . Specifically , we design a visual question answering model that combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image - based questions .
Antol et al . proposed a large - scale open - ended VQA dataset based on COCO , which is called VQA . Inspired by Xu et al . who encode visual attention in the Image Captioning , propose to use the spatial attention to help answering visual questions . formulate the VQA as a classification problem and restrict the answer only can be drawn from a fixed answer space .
Semantic Reanalysis of Scene Words in Visual Question Answering Bitrate-Based No-Reference Video Quality Assessment Combining the Visual Perception of Video Contents SegEQA: Video Segmentation Based Visual Attention for Embodied Question Answering Biometrics and forensics integration using deep multi-modal semantic alignment and joint embedding Blind quality algorithm to analyze streaming video contents in 5g networks Automatic multi-label annotation of abdominal CT images using CBIR New strategy for image and video quality assessment Real-time Video Quality Assessment for Analog Television Based on Adaptive Fuzzy Membership Function Tuning Subjective and objective quality assessment of Mobile Videos with In-Capture distortions Visual Question: Predicting If a Crowd Will Agree on the Answer Image Captioning and Visual Question Answering Based on Attributes and External Knowledge Bidirectional Attention Flow for Machine Comprehension Simple Baseline for Visual Question Answering Pythia v0.1: the Winning Entry to the VQA Challenge 2018 VQA: Visual Question Answering Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering Neural Module Networks Guided Attention for Large Scale Scene Text Verification PinterNet: A thematic label curation tool for large image datasets Research for image caption based on global attention mechanism Application of deep learning and geo-knowledge bases to scene understanding CognitiveCam: A Visual Question Answering Application Open-Ended Visual Question-Answering Predicting Drug Protein Interaction using Quasi-Visual Question Answering System Scene Text Visual Question Answering Image Captioning and Visual Question Answering Based on Attributes and External Knowledge Grounded Textual Entailment Pythia v0.1: the Winning Entry to the VQA Challenge 2018 Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering Introduction to the Special Issue on the Cross-Media Analysis for Visual Question Answering BERT Representations for Video Question Answering Hard to Cheat: A Turing Test based on Answering Questions about Images Ask Me Anything: Free-Form Visual Question Answering Based on Knowledge from External Sources Learning to Answer Questions from Image Using Convolutional Neural Network Understand, Compose and Respond - Answering Visual Questions by a Composition of Abstract Procedures Image Question Answering: A Visual Semantic Embedding Model and a New Dataset Stacked Attention Networks for Image Question Answering Image Captioning and Visual Question Answering Based on Attributes and External Knowledge Joint Image Captioning and Question Answering Image Captioning and Visual Question Answering Based on Attributes and External Knowledge SEVA: A Systems Engineer's Virtual Assistant Image Captioning and Visual Question Answering Based on Attributes and External Knowledge
87045ac2a065b14a3f6608ac9147fa81b89aa4fb 8ef36282c2e70b9569217efeba85ac427dfa688c 5ec461c01c6724aa0f40422429b0550ebfed7ea1 504cdf63f70ba22c683deb34b7481aa074ee126d de8d03c51a64d906c4ba0a023b3650f637f285d6 8c7c00adb4a8fe7622f476c5ec3ee2e0227b2640 7d254ab49fe7a085205e0a30532936b7d8ee1b6b 26700885fd98532638947be20fb7b4dca2ccb327 976d6ef37a4534be0e678d47bea93e183782f53e ceca60c4bf1a5c4e5893ae6685e7a9f80ca47f27 21c99706bb26e9012bfb4d8d48009a3d45af59b2 b9953824b3d4cd2be77ecbc5db3f7dec3dfa031e 7f201b4226d62bf449a68ebcc159acf8b95289be b37b3cdcd6d839aaee661324c9d4a5411de20a11 7a84b972b385109fc9eb76522de8344a5d495763 125172e566293a80869a7b4ed6c5f6c7d7b85568 f01ad151706d8fb2d2c3e5e134e609e8664f30be d4a78af211768d1513d66ec28fe0596218300f22 0033346700dc450ac22c9b704eab0e906d868662 ebefd4e048195b22a0f0856b8ee439d5b3b396be 39b2d8b8233a53dc7eadb819c52213369dff8648 f4a397577302ad1f270c2541e6230064882921cc 20dbdf02497aa84510970d0f5e8b599073bca1bc 98bd5dd1740f585bf25320ba504e2c1ae57f2e5f 86553974fabf38bbe022dc44794f345339b45c0b df4f851e3c37017822a683b1356c6c390b5b5487 91956c41190231eefd2186f21b79d1ca1495a68e 6acd385b2742f65359efb99543ebfb9a0d1b850f cbcadeadf83f29c773726f324c34e4782337e29a
["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science","Engineering"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science","Engineering"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Biology"] ["Computer Science"] ["Computer Science","Sociology"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science","Medicine"] ["Engineering","Computer Science"]
2019 2019 2019 2017 2017 2017 2010 2018 2017 2016 2018 2017 2015 2018 2015 2016 2015 2018 2016 2020 2018 2017 2016 2019 2019 2018 2018 2018 2016 2019 2020 2015 2015 2015 2018 2015 2016 2018 2018 2018 2019 2018
PRCV (1) ICCV IEEE Transactions on Pattern Analysis and Machine Intelligence ICLR ArXiv ArXiv 2015 IEEE International Conference on Computer Vision (ICCV) ECCV BigData MEDES IEEE Transactions on Pattern Analysis and Machine Intelligence COLING ArXiv ECCV WACV 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) IEEE Transactions on Pattern Analysis and Machine Intelligence AAAI Spring Symposium: Combining Machine Learning with Knowledge Engineering IEEE Transactions on Pattern Analysis and Machine Intelligence
https://api.semanticscholar.org/87045ac2a065b14a3f6608ac9147fa81b89aa4fb https://api.semanticscholar.org/8ef36282c2e70b9569217efeba85ac427dfa688c https://api.semanticscholar.org/5ec461c01c6724aa0f40422429b0550ebfed7ea1 https://api.semanticscholar.org/504cdf63f70ba22c683deb34b7481aa074ee126d https://api.semanticscholar.org/de8d03c51a64d906c4ba0a023b3650f637f285d6 https://api.semanticscholar.org/8c7c00adb4a8fe7622f476c5ec3ee2e0227b2640 https://api.semanticscholar.org/7d254ab49fe7a085205e0a30532936b7d8ee1b6b https://api.semanticscholar.org/26700885fd98532638947be20fb7b4dca2ccb327 https://api.semanticscholar.org/976d6ef37a4534be0e678d47bea93e183782f53e https://api.semanticscholar.org/ceca60c4bf1a5c4e5893ae6685e7a9f80ca47f27 https://www.semanticscholar.org/paper/6acd385b2742f65359efb99543ebfb9a0d1b850f https://www.semanticscholar.org/paper/3a7b63b50c64f4ec3358477790e84cbd6be2a0b4 https://www.semanticscholar.org/paper/267fb4ac632449dbe84f7acf17c8c7527cb25b0d https://www.semanticscholar.org/paper/36c3972569a6949ecca90bfa6f8e99883e092845 https://www.semanticscholar.org/paper/97ad70a9fa3f99adf18030e5e38ebe3d90daa2db https://www.semanticscholar.org/paper/1cf6bc0866226c1f8e282463adc8b75d92fba9bb https://api.semanticscholar.org/21c99706bb26e9012bfb4d8d48009a3d45af59b2 https://api.semanticscholar.org/b9953824b3d4cd2be77ecbc5db3f7dec3dfa031e https://api.semanticscholar.org/7f201b4226d62bf449a68ebcc159acf8b95289be https://api.semanticscholar.org/b37b3cdcd6d839aaee661324c9d4a5411de20a11 https://api.semanticscholar.org/7a84b972b385109fc9eb76522de8344a5d495763 https://api.semanticscholar.org/125172e566293a80869a7b4ed6c5f6c7d7b85568 https://api.semanticscholar.org/f01ad151706d8fb2d2c3e5e134e609e8664f30be https://api.semanticscholar.org/d4a78af211768d1513d66ec28fe0596218300f22 https://api.semanticscholar.org/0033346700dc450ac22c9b704eab0e906d868662 https://www.semanticscholar.org/paper/6acd385b2742f65359efb99543ebfb9a0d1b850f https://www.semanticscholar.org/paper/1235dd37312cb20aced0e97d953f6379d8a0c7d4 https://www.semanticscholar.org/paper/36c3972569a6949ecca90bfa6f8e99883e092845 https://www.semanticscholar.org/paper/1cf6bc0866226c1f8e282463adc8b75d92fba9bb https://api.semanticscholar.org/ebefd4e048195b22a0f0856b8ee439d5b3b396be https://api.semanticscholar.org/39b2d8b8233a53dc7eadb819c52213369dff8648 https://api.semanticscholar.org/f4a397577302ad1f270c2541e6230064882921cc https://api.semanticscholar.org/20dbdf02497aa84510970d0f5e8b599073bca1bc https://api.semanticscholar.org/98bd5dd1740f585bf25320ba504e2c1ae57f2e5f https://api.semanticscholar.org/86553974fabf38bbe022dc44794f345339b45c0b https://api.semanticscholar.org/df4f851e3c37017822a683b1356c6c390b5b5487 https://www.semanticscholar.org/paper/2c1890864c1c2b750f48316dc8b650ba4772adc5 https://www.semanticscholar.org/paper/6acd385b2742f65359efb99543ebfb9a0d1b850f https://api.semanticscholar.org/91956c41190231eefd2186f21b79d1ca1495a68e https://api.semanticscholar.org/6acd385b2742f65359efb99543ebfb9a0d1b850f https://api.semanticscholar.org/cbcadeadf83f29c773726f324c34e4782337e29a https://www.semanticscholar.org/paper/6acd385b2742f65359efb99543ebfb9a0d1b850f