By the aid of attention mechanisms to weight the image features adaptively , recent advanced deep learning-based salient object detection models encourage the predicted results to approximate the ground-truth masks with as large predictable areas as possible . However , these methods do not pay enough attention to small areas prone to misprediction . This article presents a computational model of motivation for learning agents to achieve adaptive , multitask learning in complex , dynamic environments . Motivation is modeled as an attention focus mechanism to extend existing learning algorithms to environments in which tasks can not be completely predicted prior to learning . Two agent models are presented for motivated reinforcement learning and motivated supervised learning , which incorporate this model of motivation . The modeling is done using a recurrent neural network or a factorization machine . Items ’ aspects that interest users are extracted from review text and leveraged using an attention-like mechanism . Since the path between a user and an item can be easily interpreted , and the important aspects between a user and an item can be interpreted by observing the attention weight , the proposed model can generate a reasonable recommendation explanation . To address it , we proposed a traffic-oriented model to classify congestion from large dataset of ultra-low frame rate video captured from traffic surveillance system . The proposed deeply supervised traffic congestion detector has two modules : attention proposal module and deeply supervised inception network . Specifically , within the shallow layers , the binary edge/corner density features are used in attention proposal module to generate the rang of interest ( ROI ) mask automatically . This strategy keeps the training process focusing on the congestion features without disturbances . Following the attention proposal module , a very deep structure based on the inception network was used together to effectively extract rich and discriminative features then detect traffic congestion . The approach was tested on a self-established dataset based on empirical data , which contains images captured from 14470 surveillance cameras for monitoring 5,215 km of freeway in Shaanxi province , China . The ROIs in the facial image are marked before the image is input into the neural network . In particular , the attention concept is adopted in the first layer of the proposed neural network to perform ROIs-related convolution calculation , and ROIs-related convolution calculation results of the specific fields in the ROIs are increased by extracting more robust features . Next , the idea of features ’ vectors inspired by CapsNet is used in the following layer of the proposed neural network . The rules of the inferenceengine are learned from audio-visual observation of the world , to form an in-teractive perceptual agent that can respond suitably to its environment . Fromthe video and audio input streams interesting objects are identied using anattention mechanism . Unsupervised clustering of the objects into perceptualcategories generates a symbolic data stream . We propose an attention-attracting method for wall advertisements by using a virtual agent whose walk precedes passersby from the viewpoint of conforming behaviors . Conforming behaviors such as adapting opinions and actions to surrounding people are often seen in human communications . Therefore , they are usually insufficient for dealing with multi-aspect sentences and the syntactically complex sentence structures . To solve this problem , we propose a novel content attention based aspect based sentiment classification model , with two attention enhancing mechanisms : sentence-level content attention mechanism is capable of capturing the important information about given aspects from a global perspective , whiles the context attention mechanism is responsible for simultaneously taking the order of the words and their correlations into account , by embedding them into a series of customized memories . Experimental results demonstrate that our model outperforms the state-of-the-art , in which the proposed mechanisms play a key role . Besides , we present independent metric learning to combine the multi-part appearance and spatial-temporal features , which obtains several metric kernels by feeding these features into distance metric learning respectively . These kernels are fused with the weights learned by the attention measure . The novel way of features fusion can achieve better functional complementarity of these features . [ reference ][ reference ] . One of the benefits of attention mechanisms is that they allow for dealing with variable sized inputs , focusing on the most relevant parts of the input to make decisions . When an attention mechanism is used to compute a representation of a single sequence , it is commonly referred to as self - attention or intra - attention . Together with Recurrent Neural Networks ( RNNs ) or convolutions , self - attention has proven to be useful for tasks such as machine reading [ reference ] and learning sentence representations [ reference ] . However , [ reference ] showed that not only self - attention can improve a method based on RNNs or convolutions , but also that it is sufficient for constructing a powerful model obtaining state - of - the - art performance on the machine translation task . The idea is to compute the hidden representations of each node in the graph , by attending over its neighbors , following a self - attention strategy . The attention architecture has several interesting properties : ( 1 ) the operation is efficient , since it is parallelizable across nodeneighbor pairs ; ( 2 ) it can be applied to graph nodes having different degrees by specifying arbitrary weights to the neighbors ; and ( 3 ) the model is directly applicable to inductive learning problems , including tasks where the model has to generalize to completely unseen graphs . We validate the proposed approach on four challenging benchmarks : Cora , Citeseer and Pubmed citation networks as well as an inductive protein - protein interaction dataset , achieving or matching state - of - the - art results that highlight the potential of attention - based models when dealing with arbitrarily structured graphs . However , we argued that simple recurrent models such as the LSTM probably have insufficient expressive power for solving tasks that require complex inference . We expect that the attention - based models would therefore outperform the pure LSTM - based approaches . Considering the second dimension of our investigation , the comparison of traditional versus neural approaches to NLP , we do not have a strong prior favouring one approach over the other . Many attentional models utilize the residual learning to deal with various tasks , including textual reasoning and visual question - answering . They use an attentional mechanism to handle two different information sources , a query and the context of the query ( e.g . contextual sentences or an image ) . The query is added to the output of the attentional module , that makes the attentional module learn the residual of query mapping as in deep residual learning . In this paper , we demonstrated that the alignDRAW model , a combination of a recurrent variational autoencoder with an alignment model over words , succeeded in generating images that correspond to a given input caption . By extensively using attentional mechanisms , our model gained several advantages . Namely , the use of the visual attention mechanism allowed us to decompose the problem of image generation into a series of steps instead of a single forward pass , while the attention over words provided us an insight whenever our model failed to generate a relevant image . The entire model is trained end - to - end and the evidence for the computed answer can be visualized using the attention weights . To summarize our contributions , in this paper we propose a novel multi - hop memory network with spatial attention for the VQA task which allows one to visualize the spatial inference process used by the deep network ( a CAFFE implementation will be made available ) , design an attention architecture in the first hop which uses each word embedding to capture fine - grained alignment between the image and question , create a series of synthetic questions that explicitly require spatial inference to analyze the working principles of the network , and show that it learns logical inference rules by visualizing the attention weights , provide an extensive evaluation of several existing models and our own model on the same publicly available datasets . Sec . [ reference ] . However , it is worth noting that while they use Recurrent Neural Networks ( RNN ) to handle sequential inputs / outputs , there is no RNN structure in our method . We only borrow their differentiable memory addressing / attention scheme for our feature aggregation . section : Neural Aggregation Network Deep neural network with value - shared weights : We introduce a novel value - shared weighting scheme in deep neural networks as a counterpart of the position - shared weighting scheme in CNNs , based on the idea that semantic matching between a question and answer is mainly about the ( semantic similarity ) value regularities rather than spatial regularities . Incorporate attention scheme over question terms : We incorporate the attention scheme over the question terms using a gating function , so that we can explicitly discriminate the question term importance . Extensive experimental evaluation and promising results . We also include an additional positional attention module in each decoder layer , which is a multi - head attention module with the same general attention mechanism used in other parts of the Transformer network , i.e. where is the model hidden size , but with the positional encoding as both query and key and the decoder states as the value . This incorporates positional information directly into the attention process and provides a stronger positional signal than the embedding layer alone . We also hypothesize that this additional information improves the decoder For additional experimental results on the impact of hop size for graphs of different sizes , please refer to the Table 4 in Appendix C. Impact of Attention Mechanism . To investigate the impact of attention mechanism to the Graph2Seq model , we still evaluate our model on SDP , SDP and SDP datasets but without considering the attention strategy . As shown in Table 4 , we find that the attention strategy significantly improves the performance of all variants of Graph2Seq by at least 14.9 % . The y - axis denotes the “ normal ” perplexity ( not calibrated by baselines ) . appendix : Attention Visualization In this section , we provide some visualization of the attention learned by the SoTA model on the WikiText - 103 validation set . The entire system is a single , unified network for object detection ( Figure [ reference ] ) . Using the recently popular terminology of neural networks with ‘ attention ’ mechanisms , the RPN module tells the Fast R - CNN module where to look . In Section [ reference ] we introduce the designs and properties of the network for region proposal . Importantly , the shape and the size of an area are dynamically determined via learning , which enables a model to attend to information with varying granularity . Area attention can easily work with existing model architectures such as multi-head attention for simultaneously attending to multiple areas in the memory . We evaluate area attention on two tasks : neural machine translation ( both character and token-level ) and image captioning , and improve upon strong ( state-of-the-art ) baselines in all the cases . In this work we propose the use of an attention mechanism to obtain a discriminative speaker embedding given non fixed length speech utterances . Our system is based on a Convolutional Neural Network ( CNN ) that encodes short-term speaker features from the spectrogram and a self multi-head attention model that maps these representations into a long-term speaker embedding . The attention model that we propose produces multiple alignments from different subsegments of the CNN encoded states over the sequence . Hence this mechanism works as a pooling layer which decides the most discriminative features over the sequence to obtain an utterance level representation . We have tested this approach for the verification task for the VoxCeleb1 dataset . The results show that self multi-head attention outperforms both temporal and statistical pooling methods with a 18\% of relative EER . Obtained results show a 58\% relative improvement in EER compared to i-vector+PLDA . In the field of deep learning , response selection is the key to retrieval-based chatbots . Faced with the challenge of contextual meaning comprehension and semantic matching , we propose matching a response through diverse information by stacked multi-head attention . First , we construct multi-granular representations on embedded input sentences by stacked multi-head attention . Based on these representations , we build two different matching matrices for context-response segment pairs with another multi-head attention stack . However , spoken language and environmental sound , which contain rich communication and contextual information necessary for trauma team cooperation , are still largely ignored . In this paper , we propose a multimodal attention network ( MAN ) that uses both verbal transcripts and environmental audio stream as input ; the model extracts textual and acoustic features using a multi-level multi-head attention module , and forms a final shared representation for trauma activity classification . We evaluated the proposed architecture on 75 actual trauma resuscitation cases collected from a hospital . In today 's world , classifying text in a binary way is no longer sufficient and thus a flexible solution able to generalize well on multilabel text is the need of the hour . In this paper , we propose a multihead attention-based approach for detection of profane text . We couple our model with power weighted average ensembling techniques to further improve the performance . In this paper , we propose a mean-max attention autoencoder ( mean-max AAE ) within the encoder-decoder framework . Our autoencoder rely entirely on the MultiHead self-attention mechanism to reconstruct the input sequence . In the encoding we propose a mean-max strategy that applies both mean and max pooling operations over the hidden vectors to capture diverse information of the input . Attention mechanisms have been at the forefront in order to learn contextual sentence representations . Current state-of-art approaches in representation learning use single-head and multi-head attention mechanisms to learn context-aware representations . However , these approaches can be largely parameter intensive resulting in low-resource bottlenecks . In vaswani2017attention , the authors introduce the Transformer network , a novel architecture that avoids the recurrence equation and maps the input sequences into hidden states solely using attention . Specifically , the authors use positional encodings in conjunction with a multi - head attention mechanism . This allows for increased parallel computation and reduces time to convergence . The Transformer network vaswani2017attention avoids the recurrence completely and uses only self - attention . We propose a modified Transformer network wherein the multi - head attention layer is replaced by a branched self - attention layer . The contributions of the various branches is learned as part of the training procedure . This is identical to dot - product attention except for the scaling factor , which improves numerical stability . Multi - head attention mechanisms obtain different representations of ( , , ) , compute scaled dot - product attention for each representation , concatenate the results , and project the concatenation with a feed - forward layer . This can be expressed in the same notation as Equation ( [ reference ] ) : where the and are parameter projection matrices that are learned . For regularization and ease of training , the network uses layer normalization ba2016layer after each sub - layer and a residual connection around each full layer he2016deep . Analogously , each layer of the decoder contains the two sub - layers mentioned above as well as an additional multi - head attention sub - layer that receives as inputs from the output of the corresponding encoding layer . In the case of the decoder multi - head attention sub - layers , the scaled dot - product attention is masked to prevent future positions from being attended to , or in other words , to prevent illegal leftward - ward information flow . Each layer has two sub - layers . The first is a multi - head self - attention mechanism , and the second is a simple , position - wise fully connected feed - forward network . We employ a residual connection For each of these we use . Due to the reduced dimension of each head , the total computational cost is similar to that of single - head attention with full dimensionality . subsubsection : Applications of Attention in our Model However , prior work has shown that gold syntax trees can dramatically improve SRL decoding , suggesting the possibility of increased accuracy from explicit modeling of syntax . In this work , we present linguistically - informed self - attention ( LISA ) : a neural network model that combines multi - head self - attention with multi - task learning across dependency parsing , part - of - speech tagging , predicate detection and SRL . Unlike previous models which require significant pre - processing to prepare linguistic features , LISA can incorporate syntax using merely raw tokens as input , encoding the sequence only once to simultaneously perform parsing , predicate detection and role labeling for all predicates . These works suggest that though syntax has the potential to improve neural network SRL models , we have not yet designed an architecture which maximizes the benefits of auxiliary syntactic information . In response , we propose linguistically - informed self - attention ( LISA ) : a model that combines multi - task learning caruana1993multitask with stacked layers of multi - head self - attention vaswani2017attention ; the model is trained to : ( 1 ) jointly predict parts of speech and predicates ; ( 2 ) perform parsing ; and ( 3 ) attend to syntactic parse parents , while ( 4 ) assigning semantic role labels . Whereas prior work typically requires separate models to provide linguistic analysis , including most syntax - free neural models which still rely on external predicate detection , our model is truly end - to - end : earlier layers are trained to predict prerequisite parts - of - speech and predicates , the latter of which are supplied to later layers for scoring . Our model has very simple structure yet achieves good results on SQuAD 2.0 test . Our future work is to reconstruct the structure of U - Net by replacing the current multi - level attention block with a simpler self - attention mechanism , which we believe can capture the question and passage information , and intuitively is also coherent with the rest of our U - Net model . In addition , we will improve the answer boundary detection performance based on some of the previous successful models . We model this blur using a weighted set of camera poses , which induce homographies on the image being captured . The blur in a particular image is parameterised by the set of weights , which provides a compact global descriptor for the blur , analogous to a convolution kernel . This descriptor fully captures the spatially-variant blur at all pixels , and is able to model camera shake more accurately than previous methods . The core module in our proposed approach is called Double Re-localization Decision Process ( DRDP ) by performing two selection of relocation decisions before and after the pose update process with two different sensor sources . A compact global descriptor is used for fast image association and a scan matching using vanilla ICP ( Iterative Closest Point ) of Point-to-line metric is applied to obtain further pose of the proposal candidate . Several experiments are designed to verify the effectiveness of the our approach in indoor environment . The invention provides a method for obtaining compact global feature descriptors of an image and an image retrieval method . The method for obtaining the compact global feature descriptors of the image comprises the following steps : obtaining at least a local feature descriptor of the image ; selecting one or more local feature descriptors from all the local feature descriptors ; carrying on dimensionality reduction to the selected local feature descriptors ; obtaining the local feature descriptors after dimensionality reduction ; converting the local feature descriptors after dimensionality reduction into global feature descriptors for expressing visual feature of the image according to a first rule ; carrying on data compression to the global feature descriptors ; and obtaining the compact global feature descriptors of the image , wherein byte size of the global feature descriptors can be varied according to the variation of parameter values in the first rule . Through the adoption of the method , the obtained global feature descriptors are more compact and have scalability , and the defect that a mobile terminal with lower internal memory has insufficient space in the prior art can be solved . Bilateral convolutional layers project the input point cloud onto an even tessellation of a hyperplane in the ( d + 1)-dimensional space known as the permutohedral lattice and perform convolutions over this representation . In contrast to existing point cloud based learning approaches , this allows us to learn over the underlying geometry of the object to create a robust global descriptor . We demonstrate its accuracy by evaluating across the shapenet and modelnet datasets , in order to illustrate 2 main scenarios , known and unknown object reconstruction . We propose an approach for occluded pedestrian detection with the following contributions . First , we introduce a novel mask-guided attention network that fits naturally into popular pedestrian detection pipelines . Our attention network emphasizes on visible pedestrian regions while suppressing the occluded ones by modulating full body features . How to extract discriminative and robust features invariant to background clutters is the core problem . In this paper , we first introduce the binary segmentation masks to construct synthetic RGB-Mask pairs as inputs , then we design a mask-guided contrastive attention model ( MGCAM ) to learn features separately from the body and background regions . Moreover , we propose a novel region-level triplet loss to restrain the features learnt from different regions , i.e. , pulling the features from the full image and body region close , whereas pushing the features from backgrounds away . In this paper , we present a novel framework called Mask-Guided Region Attention Network ( MGRAN ) for person ReID . MGRAN consists of two major components : Mask-guided Region Attention ( MRA ) and Multi-feature Alignment ( MA ) . MRA aims to generate spatial attention masks and meanwhile mask out the background clutters and occlusions . The proposed framework lends itself to one-pass inference that effectively and efficiently performs video object segmentation . Moreover , we propose a maskguided attention module for modeling the multi-scale object boundary and multi-level feature fusion . Experiments on the challenging DAVIS 2017 demonstrate the effectiveness of the proposed framework with comparable performance to the state-of-the-art , and the great efficiency about 11.5 FPS towards pioneering real-time work to our knowledge , more than 5 times faster than other state-of-the-art methods . To learn a superior saliency detection model , we introduce a neural network architecture emphasizing multi-level integration and attention application . We design a spatial attention mask generator cooperating with channel attention to produce representative features . With a Inference method with refinement , we can get results with respectable performance .
By the aid of attention mechanisms to weight the image features adaptively , recent advanced deep learning-based salient object detection models encourage the predicted results to approximate the ground-truth masks with as large predictable areas as possible . However , these methods do not pay enough attention to small areas prone to misprediction .
This article presents a computational model of motivation for learning agents to achieve adaptive , multitask learning in complex , dynamic environments . Motivation is modeled as an attention focus mechanism to extend existing learning algorithms to environments in which tasks can not be completely predicted prior to learning . Two agent models are presented for motivated reinforcement learning and motivated supervised learning , which incorporate this model of motivation .
The modeling is done using a recurrent neural network or a factorization machine . Items ’ aspects that interest users are extracted from review text and leveraged using an attention-like mechanism . Since the path between a user and an item can be easily interpreted , and the important aspects between a user and an item can be interpreted by observing the attention weight , the proposed model can generate a reasonable recommendation explanation .
To address it , we proposed a traffic-oriented model to classify congestion from large dataset of ultra-low frame rate video captured from traffic surveillance system . The proposed deeply supervised traffic congestion detector has two modules : attention proposal module and deeply supervised inception network . Specifically , within the shallow layers , the binary edge/corner density features are used in attention proposal module to generate the rang of interest ( ROI ) mask automatically . This strategy keeps the training process focusing on the congestion features without disturbances . Following the attention proposal module , a very deep structure based on the inception network was used together to effectively extract rich and discriminative features then detect traffic congestion . The approach was tested on a self-established dataset based on empirical data , which contains images captured from 14470 surveillance cameras for monitoring 5,215 km of freeway in Shaanxi province , China .
The ROIs in the facial image are marked before the image is input into the neural network . In particular , the attention concept is adopted in the first layer of the proposed neural network to perform ROIs-related convolution calculation , and ROIs-related convolution calculation results of the specific fields in the ROIs are increased by extracting more robust features . Next , the idea of features ’ vectors inspired by CapsNet is used in the following layer of the proposed neural network .
The rules of the inferenceengine are learned from audio-visual observation of the world , to form an in-teractive perceptual agent that can respond suitably to its environment . Fromthe video and audio input streams interesting objects are identied using anattention mechanism . Unsupervised clustering of the objects into perceptualcategories generates a symbolic data stream .
We propose an attention-attracting method for wall advertisements by using a virtual agent whose walk precedes passersby from the viewpoint of conforming behaviors . Conforming behaviors such as adapting opinions and actions to surrounding people are often seen in human communications .
Therefore , they are usually insufficient for dealing with multi-aspect sentences and the syntactically complex sentence structures . To solve this problem , we propose a novel content attention based aspect based sentiment classification model , with two attention enhancing mechanisms : sentence-level content attention mechanism is capable of capturing the important information about given aspects from a global perspective , whiles the context attention mechanism is responsible for simultaneously taking the order of the words and their correlations into account , by embedding them into a series of customized memories . Experimental results demonstrate that our model outperforms the state-of-the-art , in which the proposed mechanisms play a key role .
Besides , we present independent metric learning to combine the multi-part appearance and spatial-temporal features , which obtains several metric kernels by feeding these features into distance metric learning respectively . These kernels are fused with the weights learned by the attention measure . The novel way of features fusion can achieve better functional complementarity of these features .
[ reference ][ reference ] . One of the benefits of attention mechanisms is that they allow for dealing with variable sized inputs , focusing on the most relevant parts of the input to make decisions . When an attention mechanism is used to compute a representation of a single sequence , it is commonly referred to as self - attention or intra - attention . Together with Recurrent Neural Networks ( RNNs ) or convolutions , self - attention has proven to be useful for tasks such as machine reading [ reference ] and learning sentence representations [ reference ] . However , [ reference ] showed that not only self - attention can improve a method based on RNNs or convolutions , but also that it is sufficient for constructing a powerful model obtaining state - of - the - art performance on the machine translation task .
The idea is to compute the hidden representations of each node in the graph , by attending over its neighbors , following a self - attention strategy . The attention architecture has several interesting properties : ( 1 ) the operation is efficient , since it is parallelizable across nodeneighbor pairs ; ( 2 ) it can be applied to graph nodes having different degrees by specifying arbitrary weights to the neighbors ; and ( 3 ) the model is directly applicable to inductive learning problems , including tasks where the model has to generalize to completely unseen graphs . We validate the proposed approach on four challenging benchmarks : Cora , Citeseer and Pubmed citation networks as well as an inductive protein - protein interaction dataset , achieving or matching state - of - the - art results that highlight the potential of attention - based models when dealing with arbitrarily structured graphs .
However , we argued that simple recurrent models such as the LSTM probably have insufficient expressive power for solving tasks that require complex inference . We expect that the attention - based models would therefore outperform the pure LSTM - based approaches . Considering the second dimension of our investigation , the comparison of traditional versus neural approaches to NLP , we do not have a strong prior favouring one approach over the other .
Many attentional models utilize the residual learning to deal with various tasks , including textual reasoning and visual question - answering . They use an attentional mechanism to handle two different information sources , a query and the context of the query ( e.g . contextual sentences or an image ) . The query is added to the output of the attentional module , that makes the attentional module learn the residual of query mapping as in deep residual learning .
In this paper , we demonstrated that the alignDRAW model , a combination of a recurrent variational autoencoder with an alignment model over words , succeeded in generating images that correspond to a given input caption . By extensively using attentional mechanisms , our model gained several advantages . Namely , the use of the visual attention mechanism allowed us to decompose the problem of image generation into a series of steps instead of a single forward pass , while the attention over words provided us an insight whenever our model failed to generate a relevant image .
The entire model is trained end - to - end and the evidence for the computed answer can be visualized using the attention weights . To summarize our contributions , in this paper we propose a novel multi - hop memory network with spatial attention for the VQA task which allows one to visualize the spatial inference process used by the deep network ( a CAFFE implementation will be made available ) , design an attention architecture in the first hop which uses each word embedding to capture fine - grained alignment between the image and question , create a series of synthetic questions that explicitly require spatial inference to analyze the working principles of the network , and show that it learns logical inference rules by visualizing the attention weights , provide an extensive evaluation of several existing models and our own model on the same publicly available datasets . Sec .
[ reference ] . However , it is worth noting that while they use Recurrent Neural Networks ( RNN ) to handle sequential inputs / outputs , there is no RNN structure in our method . We only borrow their differentiable memory addressing / attention scheme for our feature aggregation . section : Neural Aggregation Network
Deep neural network with value - shared weights : We introduce a novel value - shared weighting scheme in deep neural networks as a counterpart of the position - shared weighting scheme in CNNs , based on the idea that semantic matching between a question and answer is mainly about the ( semantic similarity ) value regularities rather than spatial regularities . Incorporate attention scheme over question terms : We incorporate the attention scheme over the question terms using a gating function , so that we can explicitly discriminate the question term importance . Extensive experimental evaluation and promising results .
We also include an additional positional attention module in each decoder layer , which is a multi - head attention module with the same general attention mechanism used in other parts of the Transformer network , i.e. where is the model hidden size , but with the positional encoding as both query and key and the decoder states as the value . This incorporates positional information directly into the attention process and provides a stronger positional signal than the embedding layer alone . We also hypothesize that this additional information improves the decoder
For additional experimental results on the impact of hop size for graphs of different sizes , please refer to the Table 4 in Appendix C. Impact of Attention Mechanism . To investigate the impact of attention mechanism to the Graph2Seq model , we still evaluate our model on SDP , SDP and SDP datasets but without considering the attention strategy . As shown in Table 4 , we find that the attention strategy significantly improves the performance of all variants of Graph2Seq by at least 14.9 % .
The y - axis denotes the “ normal ” perplexity ( not calibrated by baselines ) . appendix : Attention Visualization In this section , we provide some visualization of the attention learned by the SoTA model on the WikiText - 103 validation set .
The entire system is a single , unified network for object detection ( Figure [ reference ] ) . Using the recently popular terminology of neural networks with ‘ attention ’ mechanisms , the RPN module tells the Fast R - CNN module where to look . In Section [ reference ] we introduce the designs and properties of the network for region proposal .
Importantly , the shape and the size of an area are dynamically determined via learning , which enables a model to attend to information with varying granularity . Area attention can easily work with existing model architectures such as multi-head attention for simultaneously attending to multiple areas in the memory . We evaluate area attention on two tasks : neural machine translation ( both character and token-level ) and image captioning , and improve upon strong ( state-of-the-art ) baselines in all the cases .
In this work we propose the use of an attention mechanism to obtain a discriminative speaker embedding given non fixed length speech utterances . Our system is based on a Convolutional Neural Network ( CNN ) that encodes short-term speaker features from the spectrogram and a self multi-head attention model that maps these representations into a long-term speaker embedding . The attention model that we propose produces multiple alignments from different subsegments of the CNN encoded states over the sequence . Hence this mechanism works as a pooling layer which decides the most discriminative features over the sequence to obtain an utterance level representation . We have tested this approach for the verification task for the VoxCeleb1 dataset . The results show that self multi-head attention outperforms both temporal and statistical pooling methods with a 18\% of relative EER . Obtained results show a 58\% relative improvement in EER compared to i-vector+PLDA .
In the field of deep learning , response selection is the key to retrieval-based chatbots . Faced with the challenge of contextual meaning comprehension and semantic matching , we propose matching a response through diverse information by stacked multi-head attention . First , we construct multi-granular representations on embedded input sentences by stacked multi-head attention . Based on these representations , we build two different matching matrices for context-response segment pairs with another multi-head attention stack .
However , spoken language and environmental sound , which contain rich communication and contextual information necessary for trauma team cooperation , are still largely ignored . In this paper , we propose a multimodal attention network ( MAN ) that uses both verbal transcripts and environmental audio stream as input ; the model extracts textual and acoustic features using a multi-level multi-head attention module , and forms a final shared representation for trauma activity classification . We evaluated the proposed architecture on 75 actual trauma resuscitation cases collected from a hospital .
In today 's world , classifying text in a binary way is no longer sufficient and thus a flexible solution able to generalize well on multilabel text is the need of the hour . In this paper , we propose a multihead attention-based approach for detection of profane text . We couple our model with power weighted average ensembling techniques to further improve the performance .
In this paper , we propose a mean-max attention autoencoder ( mean-max AAE ) within the encoder-decoder framework . Our autoencoder rely entirely on the MultiHead self-attention mechanism to reconstruct the input sequence . In the encoding we propose a mean-max strategy that applies both mean and max pooling operations over the hidden vectors to capture diverse information of the input .
Attention mechanisms have been at the forefront in order to learn contextual sentence representations . Current state-of-art approaches in representation learning use single-head and multi-head attention mechanisms to learn context-aware representations . However , these approaches can be largely parameter intensive resulting in low-resource bottlenecks .
In vaswani2017attention , the authors introduce the Transformer network , a novel architecture that avoids the recurrence equation and maps the input sequences into hidden states solely using attention . Specifically , the authors use positional encodings in conjunction with a multi - head attention mechanism . This allows for increased parallel computation and reduces time to convergence .
The Transformer network vaswani2017attention avoids the recurrence completely and uses only self - attention . We propose a modified Transformer network wherein the multi - head attention layer is replaced by a branched self - attention layer . The contributions of the various branches is learned as part of the training procedure .
This is identical to dot - product attention except for the scaling factor , which improves numerical stability . Multi - head attention mechanisms obtain different representations of ( , , ) , compute scaled dot - product attention for each representation , concatenate the results , and project the concatenation with a feed - forward layer . This can be expressed in the same notation as Equation ( [ reference ] ) : where the and are parameter projection matrices that are learned .
For regularization and ease of training , the network uses layer normalization ba2016layer after each sub - layer and a residual connection around each full layer he2016deep . Analogously , each layer of the decoder contains the two sub - layers mentioned above as well as an additional multi - head attention sub - layer that receives as inputs from the output of the corresponding encoding layer . In the case of the decoder multi - head attention sub - layers , the scaled dot - product attention is masked to prevent future positions from being attended to , or in other words , to prevent illegal leftward - ward information flow .
Each layer has two sub - layers . The first is a multi - head self - attention mechanism , and the second is a simple , position - wise fully connected feed - forward network . We employ a residual connection
For each of these we use . Due to the reduced dimension of each head , the total computational cost is similar to that of single - head attention with full dimensionality . subsubsection : Applications of Attention in our Model
However , prior work has shown that gold syntax trees can dramatically improve SRL decoding , suggesting the possibility of increased accuracy from explicit modeling of syntax . In this work , we present linguistically - informed self - attention ( LISA ) : a neural network model that combines multi - head self - attention with multi - task learning across dependency parsing , part - of - speech tagging , predicate detection and SRL . Unlike previous models which require significant pre - processing to prepare linguistic features , LISA can incorporate syntax using merely raw tokens as input , encoding the sequence only once to simultaneously perform parsing , predicate detection and role labeling for all predicates .
These works suggest that though syntax has the potential to improve neural network SRL models , we have not yet designed an architecture which maximizes the benefits of auxiliary syntactic information . In response , we propose linguistically - informed self - attention ( LISA ) : a model that combines multi - task learning caruana1993multitask with stacked layers of multi - head self - attention vaswani2017attention ; the model is trained to : ( 1 ) jointly predict parts of speech and predicates ; ( 2 ) perform parsing ; and ( 3 ) attend to syntactic parse parents , while ( 4 ) assigning semantic role labels . Whereas prior work typically requires separate models to provide linguistic analysis , including most syntax - free neural models which still rely on external predicate detection , our model is truly end - to - end : earlier layers are trained to predict prerequisite parts - of - speech and predicates , the latter of which are supplied to later layers for scoring .
Our model has very simple structure yet achieves good results on SQuAD 2.0 test . Our future work is to reconstruct the structure of U - Net by replacing the current multi - level attention block with a simpler self - attention mechanism , which we believe can capture the question and passage information , and intuitively is also coherent with the rest of our U - Net model . In addition , we will improve the answer boundary detection performance based on some of the previous successful models .
We model this blur using a weighted set of camera poses , which induce homographies on the image being captured . The blur in a particular image is parameterised by the set of weights , which provides a compact global descriptor for the blur , analogous to a convolution kernel . This descriptor fully captures the spatially-variant blur at all pixels , and is able to model camera shake more accurately than previous methods .
The core module in our proposed approach is called Double Re-localization Decision Process ( DRDP ) by performing two selection of relocation decisions before and after the pose update process with two different sensor sources . A compact global descriptor is used for fast image association and a scan matching using vanilla ICP ( Iterative Closest Point ) of Point-to-line metric is applied to obtain further pose of the proposal candidate . Several experiments are designed to verify the effectiveness of the our approach in indoor environment .
The invention provides a method for obtaining compact global feature descriptors of an image and an image retrieval method . The method for obtaining the compact global feature descriptors of the image comprises the following steps : obtaining at least a local feature descriptor of the image ; selecting one or more local feature descriptors from all the local feature descriptors ; carrying on dimensionality reduction to the selected local feature descriptors ; obtaining the local feature descriptors after dimensionality reduction ; converting the local feature descriptors after dimensionality reduction into global feature descriptors for expressing visual feature of the image according to a first rule ; carrying on data compression to the global feature descriptors ; and obtaining the compact global feature descriptors of the image , wherein byte size of the global feature descriptors can be varied according to the variation of parameter values in the first rule . Through the adoption of the method , the obtained global feature descriptors are more compact and have scalability , and the defect that a mobile terminal with lower internal memory has insufficient space in the prior art can be solved .
Bilateral convolutional layers project the input point cloud onto an even tessellation of a hyperplane in the ( d + 1)-dimensional space known as the permutohedral lattice and perform convolutions over this representation . In contrast to existing point cloud based learning approaches , this allows us to learn over the underlying geometry of the object to create a robust global descriptor . We demonstrate its accuracy by evaluating across the shapenet and modelnet datasets , in order to illustrate 2 main scenarios , known and unknown object reconstruction .
We propose an approach for occluded pedestrian detection with the following contributions . First , we introduce a novel mask-guided attention network that fits naturally into popular pedestrian detection pipelines . Our attention network emphasizes on visible pedestrian regions while suppressing the occluded ones by modulating full body features .
How to extract discriminative and robust features invariant to background clutters is the core problem . In this paper , we first introduce the binary segmentation masks to construct synthetic RGB-Mask pairs as inputs , then we design a mask-guided contrastive attention model ( MGCAM ) to learn features separately from the body and background regions . Moreover , we propose a novel region-level triplet loss to restrain the features learnt from different regions , i.e. , pulling the features from the full image and body region close , whereas pushing the features from backgrounds away .
In this paper , we present a novel framework called Mask-Guided Region Attention Network ( MGRAN ) for person ReID . MGRAN consists of two major components : Mask-guided Region Attention ( MRA ) and Multi-feature Alignment ( MA ) . MRA aims to generate spatial attention masks and meanwhile mask out the background clutters and occlusions .
The proposed framework lends itself to one-pass inference that effectively and efficiently performs video object segmentation . Moreover , we propose a maskguided attention module for modeling the multi-scale object boundary and multi-level feature fusion . Experiments on the challenging DAVIS 2017 demonstrate the effectiveness of the proposed framework with comparable performance to the state-of-the-art , and the great efficiency about 11.5 FPS towards pioneering real-time work to our knowledge , more than 5 times faster than other state-of-the-art methods .
To learn a superior saliency detection model , we introduce a neural network architecture emphasizing multi-level integration and attention application . We design a spatial attention mask generator cooperating with channel attention to produce representative features . With a Inference method with refinement , we can get results with respectable performance .
Salient Object Detection with Purificatory Mechanism and Structural Similarity Loss Motivated Learning from Interesting Events: Adaptive, Multitask Learning Agents for Complex Environments Explainable Recommendation Using Review Text and a Knowledge Graph Exploiting Deeply Supervised Inception Networks for Automatically Detecting Traffic Congestion on Freeway in China Using Ultra-Low Frame Rate Videos ROI-Attention Vectorized CNN Model for Static Facial Expression Recognition Autonomous learning of perceptual categories and symbolic protocols from audio-visual input Attracting Attention and Changing Behavior toward Wall Advertisements with a Walking Virtual Agent Content Attention Model for Aspect Based Sentiment Analysis Independent metric learning with aligned multi-part features for video-based person re-identification Graph Attention Networks Teaching Machines to Read and Comprehend Multimodal Residual Learning for Visual QA Generating Images from Captions with Attention Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering Neural Aggregation Network for Video Face Recognition aNMM: Ranking Short Answer Texts with Attention-Based Neural Matching Model Non-Autoregressive Neural Machine Translation Graph2Seq: Graph to Sequence Learning with Attention-based Neural Networks Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks Area Attention Self Multi-Head Attention for Speaker Recognition Stacked Multi-head Attention for Multi-turn Response Selection in Retrieval-based Chatbots Multimodal Attention Network for Trauma Activity Recognition from Spoken Language and Environmental Sound An Attention Ensemble Based Approach for Multilabel Profanity Detection Learning Universal Sentence Representations with Mean-Max Attention Autoencoder Low Rank Factorization for Compact Multi-Head Self-Attention Weighted Transformer Network for Machine Translation Attention is All you Need Linguistically-Informed Self-Attention for Semantic Role Labeling U-Net: Machine Reading Comprehension with Unanswerable Questions Removing camera shake blur and unwanted occluders from photographs. (Restauration des images par l'élimination du flou et des occlusions) Robust Global Localization by Using Global Visual Features and Range Finders Data Method for obtaining compact global feature descriptors of image and image retrieval method MouldingNet: Deep-learning for 3D Object Reconstruction Mask-Guided Attention Network for Occluded Pedestrian Detection Mask-Guided Contrastive Attention Model for Person Re-identification Mask-Guided Region Attention Network for Person Re-Identification OVSNet : Towards One-Pass Real-Time Video Object Segmentation Learning Essential Region Mask In Spatial Dimension For Saliency Detection
7f9743749e677c146a67f6eb770115af56d053e4 f9524a01a9dd87f3ebe09de4e4a39b80a404d633 49eff07ac5a7f249bc2ef4c1b0781fd9f47caf13 6a1ae33cdbbf53dd4907a2d119091e82f856b345 2c440e21e3459a9626afd2ffb7da1a94d84e778a effee697f092dfbc16795e2e6477bcbe03285f33 c4583ae81b9e26e68a42cca9039f9c31c099ee8e a88c05fa0458dca07eb371ef7e4b8922702c484a 38dd4782fabeb13bf6ec57a0fa5749d701d74152 31b3b0a526683048f69e703d5f098aea0e8a0ce0 ff6107328ea0b890efd8c4a10b16fd40e2511e8d 5228bb6e8e7e77a7dda9ce62b3c8306196c4e7ee df88420f4d7b76db5afca4cb9893a03d3a5e4190 bb8d47da5e74fcb86135b98ad64dbfeb82177370 1fed705d42846c005b68207c2f94cb0842d062ac be7378bfae209e91689e707814568da1d9c190a9 16fe919816982c390b0ff81af793a6a7628a0b18 f284e17ffc0c93850a0fafe86d623a90844f7927 f12253010a09625652cfdccf6efb11f5a651a776 0df15b62f2d3a4bdd8544890d3fe073ea1727233 53e059a06d738815ee74f4cad153625de5dcb221 17092ce59ed56eb5fcbf467a889c7ec3e930f4c2 56e8d2e6170d90046b98f8d80694e5f3a2749346 3831902399e9004082b10b232209cc78b3b81f5d 458f53aca4343a422568b951aa0ad59186eeab7e
["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Mathematics","Computer Science"] ["Computer Science","Mathematics"] ["Computer Science"] ["Computer Science","Medicine"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science","Mathematics"] ["Engineering","Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"]
2019 2009 2019 2020 2020 2004 2018 2018 2019 2018 2015 2016 2016 2016 2017 2016 2018 2018 2019 2015 2019 2019 2019 2019 2019 2018 2019 2017 2017 2018 2018 2012 2018 2013 2019 2019 2018 2020 2019 2019
BigData HAI WWW ICLR NIPS NIPS ICLR ECCV 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) CIKM ICLR ArXiv ACL IEEE Transactions on Pattern Analysis and Machine Intelligence INTERSPEECH ICHI ICDM Workshops EMNLP ArXiv NIPS EMNLP ArXiv ROBIO ICCV CVPR PAKDD (2)
https://api.semanticscholar.org/7f9743749e677c146a67f6eb770115af56d053e4 https://api.semanticscholar.org/f9524a01a9dd87f3ebe09de4e4a39b80a404d633 https://api.semanticscholar.org/49eff07ac5a7f249bc2ef4c1b0781fd9f47caf13 https://api.semanticscholar.org/6a1ae33cdbbf53dd4907a2d119091e82f856b345 https://api.semanticscholar.org/2c440e21e3459a9626afd2ffb7da1a94d84e778a https://api.semanticscholar.org/effee697f092dfbc16795e2e6477bcbe03285f33 https://api.semanticscholar.org/c4583ae81b9e26e68a42cca9039f9c31c099ee8e https://api.semanticscholar.org/a88c05fa0458dca07eb371ef7e4b8922702c484a https://api.semanticscholar.org/38dd4782fabeb13bf6ec57a0fa5749d701d74152 https://www.semanticscholar.org/paper/33998aff64ce51df8dee45989cdca4b6b1329ec4 https://www.semanticscholar.org/paper/d1505c6123c102e53eb19dff312cb25cea840b72 https://www.semanticscholar.org/paper/1afb710a5b35a2352a6495e4bf6eef66808daf1b https://www.semanticscholar.org/paper/0875fc92cce33df5cf7df169590dbf0ca00d2652 https://www.semanticscholar.org/paper/1cf6bc0866226c1f8e282463adc8b75d92fba9bb https://www.semanticscholar.org/paper/59fb2a2898decc1d2f8f6259af8036500c95c422 https://www.semanticscholar.org/paper/364c1a3df58d87cb40ab33fdf3831cf2862f3570 https://www.semanticscholar.org/paper/15e81c8d1c21f9e928c72721ac46d458f3341454 https://www.semanticscholar.org/paper/94eb48c1878efbe2ccff121bd600dd0fd8a75650 https://www.semanticscholar.org/paper/c4744a7c2bb298e4a52289a1e085c71cc3d37bc6 https://www.semanticscholar.org/paper/424561d8585ff8ebce7d5d07de8dbf7aae5e7270 https://api.semanticscholar.org/31b3b0a526683048f69e703d5f098aea0e8a0ce0 https://api.semanticscholar.org/ff6107328ea0b890efd8c4a10b16fd40e2511e8d https://api.semanticscholar.org/5228bb6e8e7e77a7dda9ce62b3c8306196c4e7ee https://api.semanticscholar.org/df88420f4d7b76db5afca4cb9893a03d3a5e4190 https://api.semanticscholar.org/bb8d47da5e74fcb86135b98ad64dbfeb82177370 https://api.semanticscholar.org/1fed705d42846c005b68207c2f94cb0842d062ac https://api.semanticscholar.org/be7378bfae209e91689e707814568da1d9c190a9 https://www.semanticscholar.org/paper/3861ae2a6bdd2a759c2d901a6583e63a216bc2fc https://www.semanticscholar.org/paper/204e3073870fae3d05bcbc2f6a8e263d9b72e776 https://www.semanticscholar.org/paper/060ff1aad5619a7d6d6cdfaf8be5da29bff3808c https://www.semanticscholar.org/paper/27e98e09cf09bc13c913d01676e5f32624011050 https://api.semanticscholar.org/16fe919816982c390b0ff81af793a6a7628a0b18 https://api.semanticscholar.org/f284e17ffc0c93850a0fafe86d623a90844f7927 https://api.semanticscholar.org/f12253010a09625652cfdccf6efb11f5a651a776 https://api.semanticscholar.org/0df15b62f2d3a4bdd8544890d3fe073ea1727233 https://api.semanticscholar.org/53e059a06d738815ee74f4cad153625de5dcb221 https://api.semanticscholar.org/17092ce59ed56eb5fcbf467a889c7ec3e930f4c2 https://api.semanticscholar.org/56e8d2e6170d90046b98f8d80694e5f3a2749346 https://api.semanticscholar.org/3831902399e9004082b10b232209cc78b3b81f5d https://api.semanticscholar.org/458f53aca4343a422568b951aa0ad59186eeab7e