title: Attentive Attention Mechanism for Answer Representation - Keras Layer
body: <p>With reference to the paper "A Stacked BiLSTM Neural Network Based on Coattention Mechanism for Question Answering" (<a href="https://www.hindawi.com/journals/cin/2019/9543490/" rel="nofollow noreferrer">https://www.hindawi.com/journals/cin/2019/9543490/</a>), I am trying to write a custom keras layer to implement the below set of equations.</p>
  
  <p>My context vector shapes for question and answer are as below.</p>
  
  <p>Shape of Context vector of Question (CQ):  (512, 256)</p>
  
  <p>Shape of Context vector of Answer   (CA):  (512, 256)</p>
  
  <p><strong>Expected output:</strong> Cosine score given the above two context vectors as inputs.</p>
  
  <p><a href="https://i.stack.imgur.com/EQU1U.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/EQU1U.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.stack.imgur.com/EAJQ5.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/EAJQ5.png" alt="enter image description here"></a></p>
  
  <p>This is my code for the same.</p>
  
  <pre><code>class score(tf.keras.layers.Layer):
  
      def __init__(self, units):
          super(score, self).__init__()
          # Initializing the weights
          self.Wam = tf.keras.layers.Dense(units) # Attention matrix of CA at time t.
          self.Wqm = tf.keras.layers.Dense(units) # Attention matrix of OQ
          self.Wms = tf.keras.layers.Dense(units) # Attention weight vector
  
      def call (self, CQ, CA):
  
          OQ = tf.keras.layers.GlobalMaxPool1D()(CQ)
          print(OQ.shape)
          CA = tf.expand_dims(CA,1)
          #Maq = tf.nn.tanh(tf.matmul(self.Wam, CA) + tf.matmul(self.Wqm, OQ))
          Maq = tf.nn.tanh(self.Wam(CA) + self.Wqm(OQ))
          Saq  = tf.exp(tf.linalg.matmul(Wms, Maq, transpose_a = True, transpose_b = False))
          OA = CA * Saq
          OA = tf.reduce_sum(OA, axis = 1)
  
          # Calcutate Cosine similarity of vectors OQ and OA
          dot_product = tf.tensordot(OQ, OA, axes=0)
          # Normalize input
          norm_OQ = tf.nn.l2_normalize(OQ)
          norm_OA = tf.nn.l2_normalize(OA)
          scoreCosine = dot_product / (norm_OQ * norm_OA)
  
          #Normalize the cosine similarity to the [0, 1] interval 
          scoreCosine = 0.5 * (scoreCosine) + 0.5
  
          return scoreCosine
  
  score_layer = score(10)
  scoreCosine = score_layer(CQ, CA)
  </code></pre>
  
  <p>I am facing issues with matrix shapes in each step. I tried to break the steps and execute each separately.</p>
  
  <p><strong>Issues I am facing:</strong> </p>
  
  <p>1) Input matrices are of shape 2D but GlobalMaxPool1D layer (1st equation) expects 3D input and produces 2D output.</p>
  
  <p>2) I initialized attention matrices (Wam, Wqm) and attention weight vector (Wms) as dense layer with 10 units (Reference Bahdanau Attention layer here -> <a href="https://www.tensorflow.org/tutorials/text/nmt_with_attention" rel="nofollow noreferrer">https://www.tensorflow.org/tutorials/text/nmt_with_attention</a>). Is it correct? This is not working for equation 3 (Wms transpose) as it is not a Tensor but a dense layer.</p>
  
  <p>Please advise if this was correct approach or where am I doing incorrect. 
  Any help is greatly appreciated.</p>