Multimodal Adaptive Fusion of Face and Gait Features using Keyless
  attention based Deep Neural Networks for Human Identification
Biometrics plays a significant role in vision-based surveillance
applications. Soft biometrics such as gait is widely used with face in
surveillance tasks like person recognition and re-identification. Nevertheless,
in practical scenarios, classical fusion techniques respond poorly to changes
in individual users and in the external environment. To this end, we propose a
novel adaptive multi-biometric fusion strategy for the dynamic incorporation of
gait and face biometric cues by leveraging keyless attention deep neural
networks. Various external factors such as viewpoint and distance to the
camera, are investigated in this study. Extensive experiments have shown
superior performanceof the proposed model compared with the state-of-the-art
model.
Ashwin Prakash, Thejaswin S, Athira Nambiar, Alexandre Bernardino
Multimodal Adaptive Fusion of Face and Gait Features using Keyless attention based Deep Neural Networks for Human Identification

###### Abstract

Biometrics plays a significant role in vision-based surveillance applications. Soft biometrics such as gait is widely used with face in surveillance tasks like person recognition and re-identification. Nevertheless, in practical scenarios, classical fusion techniques respond poorly to changes in individual users and in the external environment. To this end, we propose a novel adaptive multi-biometric fusion strategy for the dynamic incorporation of gait and face biometric cues by leveraging keyless attention deep neural networks. Various external factors such as viewpoint and distance to the camera, are investigated in this study. Extensive experiments have shown superior performance of the proposed model compared with the state-of-the-art model.

 Soft-biometrics, surveillance, Gait, Face, Adaptive fusion, person identification, Deep Learning, attention models, multimodal fusion.

## I Introduction

Human biometrics refers to the unique intrinsic physical or behavioural traits that allow distinguishing between different individuals, e.g., face, fingerprint, hand geometry, iris, and gait. The use of biometrics helps in various surveillance applications such as access control, human recognition, and re-identification. Single biometric modalities are often affected by practical challenges such as noisy data, lack of distinctiveness, intra/ inter-class variability, error rate, and spoof attacks. A common method to overcome this issue is to combine multiple biometric modalities, known as multimodal biometric fusion.

A critical constraint that any biometric system confronts is the variation in the environment owing to external conditions. This includes user-induced variability, i.e., inherent distinctiveness, pose, distance, and expression, or environment-induced variability, i.e., lighting condition, background noise, and weather conditions [1]. These constraints have not been adequately addressed in the literature on multimodal fusion. For instance, most of the existing works are based on static fusion strategies, wherein the fusion rules are fixed for certain external conditions such as pose/ lighting/ distance or based on manual computations. As a result, when the environment changes, the biometric system performs sub-optimally. To overcome this issue, a novel context-aware adaptive multi-biometric fusion strategy, which can dynamically adapt the fusion rules to external conditions, is proposed in this paper. In particular, the adaptive fusion of gait and face at different viewpoints was investigated using an attention-based deep learning technique.

Face is one of the predominant biometric traits commonly employed in human recognition. Similarly, gait is an important soft biometric commonly used in surveillance applications, because it is unobtrusive and perceivable from a distance [2]. While fusing gait and face, the most influential factors may be the view angle and distance from the subject to the camera. Notably, gait can be clearly captured in the lateral view, whereas the face can be well captured in the frontal view. Based on this rationale, a novel context-aware adaptive fusion mechanism was designed to assign weights to gait and face biometric cues based on the context. The key notion of the proposed model is that when the person is in far/ lateral view, gait features should be gaining more priority than the less visible facial cues, whereas when the person is in near/ frontal view, the face should be getting more importance than the partially occluded gait features.

To facilitate the aforesaid context-aware adaptive fusion strategy, a keyless attention-based deep learning fusion is leveraged in the multimodal biometric fusion framework. As mentioned in [3], keyless attention is a sophisticated and effective technique for better accounting for the sequential character of data without the need for supplementary input, thereby excelling in identifying relationships across modalities. Extensive experiments are conducted via individual biometric-based identification, naive bilinear pooling [4] based multimodal fusion and keyless attention-based adaptive fusion mechanism. Results clearly highlight the superior performance of the proposed model.

The remainder of this paper is organized as follows. Related works on face and gait-based human recognition are detailed in Section 2. Section 3 presents the framework of the proposed context-aware adaptive multibiometric fusion method. The experiments and results are presented in Sections 4 and 5, respectively. Finally, conclusions and future directions are presented in Section 6.
## II Related Work

One of the earliest face recognition systems was discovered in [5] using the manual marking of various facial landmarks. Recognition of faces in images with objects has gained popularity with [6], which introduced the eigenface method. Since then, various other similar techniques, e.g., Linear Discriminant Analysis, to produce Fisherfaces, Gabor, LBP, and PCANet were reported in [7]. Recently, deep learning-based techniques have also gained popularity, e.g. DeepFaces, Facenet, and Blazeface approach human-level performance under unconstrained conditions [7] (DeepFace: 97.35% vs. Human: 97.53%).

Classical gait-based identification approaches use either model-based or appearance-based approaches[2]. The former detects joints/body parts using 2D cameras or depth cameras. For example, [8] applied Hough transform to detect legs in each frame, whereas [9] leveraged Procrustes shape analysis to calculate joint angles of body parts. Gait recognition/re-identification using a Kinect camera has also been proposed in some works [10]. In contrast to model-based approaches, appearance-based approaches use richer information, such as silhouettes of the human body in gait frames, to recognise gaits, e.g., gait energy image (GEI) [11] and GEI-based local multi-scale feature descriptors [12]. Recent deep learning approaches presented advanced techniques, e.g., view-invariant gait recognition using a convolutional neural network GEINet [13], a comprehensive model with both LSTM and residual attention components for cross-view gait recognition [14].

On the fusion of gait and face for human identification, one of the early works [15] proposed a fusion strategy by combining the results of gait and face recognition algorithms based on sequential importance sampling. A probabilistic combination of facial and gait cues was studied in [16]. Yet another work on the adaptive fusion of gait and face is [17] via score-level fusion. All the aforementioned studies leverage either classical machine learning techniques using handcrafted features, static fusion rules, or manual computations. On the contrary, in this work, we present a deep learning technique based on a keyless attention-based adaptive fusion mechanism for human identification, one of its first kind to the best of our knowledge.

## III Multimodal Adaptive Fusion Methodology

The proposed keyless attention-based adaptive fusion of face and gait towards human identification is shown in Fig.1, in which all the symbols are introduced in the following subsections. The proposed framework maps spatio-temporal feature sequences corresponding to gait and face to a single label. First, the video sequence's descriptors of gait and face are extracted from each frame via a _Feature extractor_ module. Further, the _Attention & Fusion_ block is employed to compute the feature importance and adaptively amalgamate them. Finally, the class probabilities are generated by a _classifier_ module using a fully connected (FC) layer, followed by a softmax layer.

### _Gait feature extractor_

Gait recognition involves recognizing a person based on their gait features, i.e, movement patterns [18]. The temporal variation in human silhouettes is considered by calculating the cyclic pattern of movement, commonly referred to as the _gait cycle_. It can be observed that the size of the closed area between the legs and the aspect ratio in the human silhouettes are alternating periodically in a gait sequence (Refer Fig. 2(a)& 2(b)). Based on this notion, a complete gait cycle is determined by the number of frames between three consecutive local minima (two red points in Fig. 2(b)). The corresponding frames are extracted from the RGB images. This technique of gait cycle computation is applied to every person. Accordingly, the video is divided into an adequate number of frames required for gait feature computation.

The images are preprocessed and converted from RGB to grayscale to facilitate computational efficiency. Further, the extracted frames of gait silhouette images of height \(H\) and width \(W\) are fed onto a Convolutional LSTM [19] architecture as depicted in gait feature extractor network \(\mathcal{G}\) in Fig. 1 and obtains a gait feature descriptor \(G\). Formally, the gait feature sequence of a video can be represented as \(G=\{g_{1},\cdots,g_{L}\},g_{i}\in\mathbb{R}^{C}\) where \(g_{i}\) denotes the gait feature of frame \(i\), \(C\) denotes the feature dimension, and \(L\) denotes the number of frames.

### _Face feature extractor_

Face recognition involves recognizing a person by his facial features [5]. In our case, since the viewpoint and distance of the person vary significantly across the frames, traditional face detection algorithms that rely on the frontal view do not work well. Hence, facial bounding boxes are initially cropped out of the video frames leveraging Google Mediapipe human pose detection framework [20]. The framework employs a two-step detector-tracker setup where the detector locates the pose region-of-interest (ROI) within the frame and the tracker predicts all 33 keypoints from this ROI. In the case of videos, the detector is run only on the first frame and the ROI of the subsequent images is derived from the pose keypoints of the previous frame. As shown in Fig. 3, from the estimated Mediapipe keypoints, the human face is manually cropped out by fixed measurements, with respect to the facial coordinates.

The cropped face images are preprocessed, converted from RGB to grayscale, and resized to the dimension of _H_\(\times\)_W_. Further, the images are fed into ConvolutionallLSTM [19] architecture to extract facial feature descriptor \(F\) per person, as depicted in the face feature extractor network \(\mathcal{F}\) in Fig.1. Formally, the face feature descriptor corresponding to an _L_-frame video is represented as \(F=\{f_{1},\cdots,f_{L}\},f_{i}\in\mathbb{R}^{C}\), where \(f_{i}\) represents the facial feature of frame \(i\).

### _Naive fusion of Face and Gait via Bilinear pooling_

As an initial fusion technique, we propose the naive bilinear pooling (BLP) method [4] to fuse features. The method takes in the 3D tensor outputs from the final max-pooling
layers of the face (\(\mathcal{F}\)) and gait (\(\mathcal{G}\)) feature extraction networks. The outputs are further reshaped into the matrix of dimensions \(p\times d\) and are combined via the bilinear pooling method to obtain the fusion result \(Z\), as follows:

\[Z=FG^{T},F\in\mathbb{R}^{p\times d},G\in\mathbb{R}^{p\times d} \tag{1}\]

The matrix \(Z\) is then flattened into a vector and then passed onto the softmax activation function, where it computes the probability for class \(k\) out of \(K\) classes.

### _Keyless attention based Adaptive fusion of Face and Gait_

Attention mechanisms are widely used in sequence models, enabling the modelling of dependencies regardless of their location in the input or output sequences [21]. In our case, not every frame in a video helps identify a subject in the same way. In order to estimate the importance weights for each frame, we adapt the attention mechanism. An attention function is a process that takes a query vector and a set of key-value pairs and produces an output. In existing soft attention mechanisms [21], the weight computation is not limited to the feature vectors but also incorporates an additional input, such as the previously hidden state vector of the LSTM or a vector representing a target entity as in [22]. These additional inputs along with feature vectors referred to as _key vectors_, help to find the most related weighted average of feature vectors. However, the weights in our work depend only on the feature vectors and do not require any additional input, thus named as _keyless attention_, synonymous to the work [3]. In our case, referring to Fig. 1, the gait feature descriptor \(G\) and face feature descriptor \(F\) are further fed onto two attention modules _viz_. Gait attention block and Face attention block respectively. Further, multimodal adaptive weights are computed via the fusion mechanism. Detailed explanations are given below.

#### Iii-D1 **Face Attention**

The facial feature is updated by incorporating the attention mechanism to assign weighted visual elements. Formally, face attention is computed as follows:

\[\tilde{f}_{i}=\mathbf{W_{f}}f_{i}+\mathbf{b_{f}} \tag{2}\]

Fig. 1: Overall architecture of the proposed keyless attention-based adaptive fusion of face and gait for person recognition. The gait and face features are encoded by the gait and face feature extraction networks \(\mathcal{F}\) and \(\mathcal{G}\), respectively. The outputs are subsequently weighted using keyless attention. Context-aware adaptive multimodal fusion is then employed to fuse global gait and facial features. Finally, the outputs are passed through the classifier to determine the class (Person ID) of the person.

Fig. 3: Process of obtaining face images from Mediapipe pose estimation.

Fig. 2: (a) Human silhouette taken from a gait video sequence of CASIA-A. (b) Representation of silhouette aspect ratio over the whole video. The marked points in red represent the starting and ending of one gait cycle. (c),(d) & (e) Glimpses from the CASIA-A dataset at angles 0\({}^{\circ}\), 45\({}^{\circ}\), 90\({}^{\circ}\) respectively.
\[\bar{e_{i}}=\bar{u}^{T}\tanh\bigl{(}\bar{f_{i}}\bigr{)} \tag{3}\]

\[\bar{\alpha_{i}}=\frac{\exp\{(\lambda\bar{e_{i}})\}}{\sum_{k=1}^{L}\exp\{( \lambda\bar{e_{k}})\}} \tag{4}\]

Here, \(\bar{f_{i}}\) is the low-dimension representation of frame \(i\) and \(\mathbf{W_{f}}\) & \(\mathbf{b_{f}}\) are the learnable parameters. The importance weight, \(\bar{e_{i}}\), of the element \(f_{i}\) is computed by the inner product between the new representation of \(\bar{f_{i}}\) and a learnable vector \(\bar{u}\). The normalized importance weight of facial feature \(\bar{\alpha_{i}}\) is calculated using the softmax function, as shown in Eq. (5). \(\lambda\) is a scale factor that ensures that the importance weights are evenly distributed, which ranges between 0 and 1. Nevertheless, it is observed in Eq. (4) that \(\tanh\left(\cdot\right)\) non-linearity may not be effective for learning complicated linkages, since \(\tanh\left(x\right)\) is roughly linear for x \(\in\) [-1, 1]. Therefore, inspired by the method as in [23], we leverage an effective gated mechanism as shown in Eq. (6) to formulate a better normalized facial importance weight \(\bar{\alpha_{i}}\).

\[\bar{\alpha_{i}}=\frac{\exp\bigl{\{}\{\lambda\bar{u}^{T}(\tanh\bigl{(}\bar{f_ {i}}\bigr{)}\odot sigm(\bar{f_{i}})\}\bigr{\}}\bigr{\}}{\sum_{k=1}^{L}\exp\bigl{\{} \{\lambda\bar{u}^{T}(\tanh\bigl{(}\bar{f_{k}}\bigr{)}\odot sigm(\bar{f_{k}}) \}\bigr{\}}\bigr{\}}} \tag{5}\]

\[\bar{\alpha}=\sum_{i=1}^{L}\bar{\alpha_{i}} \tag{6}\]

where _sigm(\(\cdot\))_ is the sigmoid non-linearity and \(\odot\) is an element-wise multiplication. This new \(\bar{\alpha_{i}}\) is further used to compute global facial attention weight \(\bar{\alpha}\) by combining facial importance weight across all \(L\) frames(Refer Eq. (7)).

#### Iii-D2 **Gait Attention**

Analogous to the face modality, the attention mechanism is incorporated in the gait counterpart as well. The global gait attention weight \(\bar{\beta}\) by leveraging the weighted visual elements in the gait stream is computed as follows.

\[\bar{g_{i}}=\mathbf{W_{g}}g_{i}+\mathbf{b_{g}} \tag{7}\]

\[\bar{\beta_{i}}=\frac{\exp\bigl{\{}\{\lambda\bar{u}^{T}(\tanh(\bar{g_{i}}) \odot sigm(\bar{g_{i}})\}\bigr{\}}\bigr{\}}{\sum_{k=1}^{L}\exp\{\{\lambda\bar{u }^{T}(\tanh(\bar{g_{k}})\odot sigm(\bar{g_{k}})\}\}\bigr{\}}} \tag{8}\]

\[\bar{\beta}=\sum_{i=1}^{L}\bar{\beta_{i}} \tag{9}\]

#### Iii-D3 **Context-aware Adaptive Fusion**

From Eq.(6) & Eq.((9), we obtain the value of \(\bar{\alpha}\) and \(\bar{\beta}\), which are the global individual attention weights of the face and gait features, respectively. The weighted average of the face and gait feature is computed using the adaptive weights:

\[\alpha=\frac{\left\|\bar{\alpha}\right\|}{\left\|\bar{\alpha}\right\|+\left\| \bar{\beta}\right\|} \tag{10}\]

\[\beta=\frac{\left\|\bar{\beta}\right\|}{\left\|\bar{\alpha}\right\|+\left\| \bar{\beta}\right\|} \tag{11}\]

The adaptive fusion is performed by combining the two features multiplied individually by their weighted global attention weights, as follows:

\[\mathbb{Z}=\alpha F+\beta G \tag{12}\]

where \(\mathbb{Z}\) refers to the context-aware adaptively fused feature. \(\mathbb{Z}\) is further passed onto a fully-connected (FC) layer, followed by a softmax function that classifies the feature according to the \(K\) classes provided. The resultant column vector, \(R\) is then used to determine the class identifier (_Person ID_) of the subject in consideration for the fused feature \(\mathbb{Z}\) by

\[ID(\mathbb{Z})=argmax(R) \tag{13}\]

### _Objective functions_

The model classifier employs categorical cross-entropy loss, also known as Softmax loss. This supervised loss calculates the classification error among \(K\) classes. The number of nodes in the softmax layer depends on the number of identities in the training set. Considering \(t\) and \(w_{k}\) as the target vector and learnable vector respectively, the loss is computed as:

\[Loss=-\sum_{i}^{K}t_{i}\log\bigl{(}softmax(w_{k}^{T}\mathbb{Z})_{i}\bigr{)} \tag{14}\]

## IV Experimental Setup

**Dataset:** In this work, we use CASIA Gait Dataset A [24], which includes 19139 images of 20 subjects. Each person has 12 image sequences, 4 sequences for each of the three directions, i.e. 0\({}^{\circ}\), 45\({}^{\circ}\), 90\({}^{\circ}\)(Refer Fig. 2(c), 2(d) and 2(e)). Among the 4 sequences per angle, 2 sequences are used for training, and the remaining 2 sequences are used for testing.

**Evaluation protocols:** Standard evaluation metrics like _accuracy_ and _log-loss_ are employed to validate the performance of our model. _Accuracy_ is used to evaluate how well the algorithm is performing for all classes by giving them equal importance, whereas _log loss_ is considered to be a crucial metric that is based on probabilities. Mathematically, log-loss is computed by:

\[log\_loss=\frac{-1}{N\sum_{i=1}^{N}[y_{i}\ln p_{i}+(1-y_{i})\ln(1-p_{i})]} \tag{15}\]

where \(N\) is number of person and \(y_{i}\) is the observed value and \(p_{i}\) is predicted probability.

**Implementation details:** The proposed method is implemented using the TensorFlow framework. During training, video frames correspond to one gait cycle across three orientations 0\({}^{\circ}\), 45\({}^{\circ}\), and 90\({}^{\circ}\)are considered. In this work, the gait cycle corresponds to \(L\) = 24 frames, each with height _H_= 128, and width \(W\) = 128 is used. The images are normalized using the RGB mean and standard deviation of ImageNet before passing them to the network. After dimension reduction, the resulting dimension of the gait and face feature descriptor is 588 each. In the experiments, we use Optuna [25], a hyperparameter optimization framework, to obtain the best hyperparameter for our models. We train the network for approximately 1000 iterations. The implementations are done in a machine with Tesla V100 GPU with 12GB RAM and took around 1 hour to train the model.
## V Experimental Results

To verify the effectiveness of our proposed approach, various experiments using single feature-based and multimodal fusion-based human identification are carried out. The result summary is shown in Tables I and II. Referring to Table I, the first two rows are single-modality based results, whereas the remaining are multi-modality results.

**(i) Face feature based human recognition:** Training of facial features separately on each angle 0\({}^{\circ}\), 45\({}^{\circ}\)and 90\({}^{\circ}\) with custom parameters and hyperparameter tuning using Optuna [25] produce accuracies up to 65%, 80%, and 85%, respectively (Refer Table II). The overall accuracy of the face model incorporating all orientations is 80% (Refer Table I).

**(ii) Gait feature based human recognition:** Training of the gait features across three view-points produces accuracies 75%, 60%, and 55%, respectively (Refer Table II). Referring to Table I, the overall accuracy of the gait model across viewpoints is observed to be 70%.

One noteworthy observation from the aforesaid single-modality based results, is the outperformance of face and gait models in 90\({}^{\circ}\)and in 0\({}^{\circ}\)viewpoints, respectively. This accentuates our initial intuition of the influence viewpoint on feature modalities. To incorporate the best of both modalities in different viewpoints, we facilitate fusion techniques. In particular, four fusion approaches are carried out.

**(iii) Average based fusion:** Average weight-based fusion incorporates a manual weight input to the face and gait model. For this technique, a weightage of 0.5 is devised on the individual face and gait features, achieving an accuracy of \(75\%\) on the test dataset.

**(iv) Naive fusion via BLP:** Bilinear Pooling incorporates the fusion of both gait and face models, as explained in Section III (C). The model achieves 85%, 75%, and 85% viewpoint-wise accuracies, as shown in Table II. The overall fused model results in an accuracy of 80%. It was observed that compared to the _Average based fusion_ model, _Naive fusion with BLP_ improves the accuracy by \(5\%\).

**(v) Attention Fusion:** In this model, the keyless attention mechanism (discussed in Sec III-D) is implemented to obtain global face and gait attention weights \(\bar{\alpha}\) and \(\bar{\beta}\). It is further multiplied with the respective features \(F\) and \(G\) and is concatenated to obtain a single feature vector. Note that no adaptive fusion strategy is employed in this scheme. This model is able to achieve an overall accuracy of \(85\%\) incorporating features over all the viewpoints.

**(vi): Context-aware Adaptive Fusion with attention:** This strategy incorporates the proposed context-aware adaptive fusion strategy to the attention module, discussed in Sec.III-D. The viewpoint-wise accuracy attained by this method are 90%, 80%, and 90%, respectively. Referring to Table I, this model outperforms all other models by achieving an overall accuracy of 90%, highlighting the importance of the context-aware fusion of modalities across the view-points. In terms of the log loss metric, the adaptive fusion strategy has achieved the least value with 0.389 compared to all other models.

From the view-point-wise performance of the models in Table II, some key interpretations also can be drawn. For 0\({}^{\circ}\), the gait model surpasses the face model performance, which aligns with our intuition that the model learns gait features better when the subject walks laterally. Similarly, works well when the subject walks towards the camera at 90\({}^{\circ}\). However, while incorporating adaptive fusion strategy, the best of both modalities are incorporated adaptively based on the context, resulting in high performance irrespective of the view-point.

A visual interpretation of the attention based adaptive fusion model is depicted in Fig. 4 in terms of a confusion matrix. A confusion matrix visualizes and summarizes the performance of a classification algorithm. All the three viewpoints together on the test dataset of 20 subjects is depicted in Fig. 4. We can observe that 18 out of 20 subjects are correctly classified, except subject IDs '1' and '13'. The subject ID '1' is most often classified as 14 and the subject ID '13' as 5 and 20 in most models. Further, we observe that subject 13 being identified correctly by the face model but not in the gait model, which might be ascribable to the suboptimal learning of the model from the grayscale features of the image.

\begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline \multirow{2}{*}{**Angle(\({}^{\circ}\))**} & \multicolumn{4}{c|}{**Accuracy(\(\%\))**} \\ \cline{2-5}  & **Face** & **Gait** & **Naïve Fusion** & **Adaptive Fusion** \\ \hline
0\({}^{\circ}\) & 65 & 75 & 85 & 90 \\ \hline
45\({}^{\circ}\) & 75 & 60 & 75 & 80 \\ \hline
90\({}^{\circ}\) & 85 & 55 & 85 & 90 \\ \hline \hline \end{tabular}
\end{table} TABLE II: Result of all models conducted angle-wise \(0^{\circ}\), \(45^{\circ}\), and \(90^{\circ}\) with respect to camera

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Index** & **Model** & **Accuracy(\%)** & **Log Loss** \\ \hline (i) & Face feature model & 80 & 0.436 \\ (ii) & Gait feature model & 70 & 1.641 \\ \hline (iii) & Average based fusion & 75 & 0.779 \\ (iv) & Naïve Fusion via BLP & 80 & 0.519 \\ (v) & Attention Fusion & 85 & 1.619 \\ (vi) & **Adaptive Fusion Attention** & **90** & **0.389** \\ \hline (vii) & **Geng, Wang et. al. [17]** & 86.67 & - \\ \hline \end{tabular}
\end{table} TABLE I: Overall result summary of the models
## VI Conclusion and Future Works

In this work, we proposed a multimodal adaptive fusion of the face and gait toward human identification. In particular, a keyless attention-based deep neural network for learning the attention in the gait and face videos and a context-aware adaptive fusion strategy to efficiently extract and fuse the features is presented. Based on the observation that single biometric modality results in suboptimal results, various studies leveraging average based, naive fusion, attention fusion and context-aware adaptive fusion were investigated. Results of the proposed attention-based adaptive fusion strategy show superior performance compared to all the other models as well as the state-of-the-art result. Future Improvements can be made by introducing better attention mechanisms such as dense co-attention and spatial-channel attention, as well as advanced fusion mechanisms like tucker fusion, block fusion etc.
