title: LSTM Model in Keras with Auxiliary Inputs
body: <p>I have a dataset with 2 columns - Each column contains a set of documents. I have to match the document in Col A with documents provided in Col B. This is a supervised classification problem. So my training data contains a label column indicating whether the documents match or not.</p>
  
  <p>To solve the problem, I have a created a set of features, say f1-f25 (by comparing the 2 documents) and then trained a binary classifier on these features. This approach works reasonably well, but now I would like to evaluate Deep Learning models on this problem (specifically, LSTM models).</p>
  
  <p>I am using the <code>keras</code> library in Python. After going through the keras documentation and other tutorials available online, I have managed to do the following:</p>
  
  <pre><code>from keras.layers import Input, Embedding, LSTM, Dense
  from keras.models import Model
  
  # Each document contains a series of 200 words 
  # The necessary text pre-processing steps have been completed to transform  
    each doc to a fixed length seq
  main_input1 = Input(shape=(200,), dtype='int32', name='main_input1')
  main_input2 = Input(shape=(200,), dtype='int32', name='main_input2')
  
  # Next I add a word embedding layer (embed_matrix is separately created    
  for each word in my vocabulary by reading from a pre-trained embedding model)
  x = Embedding(output_dim=300, input_dim=20000, 
  input_length=200, weights = [embed_matrix])(main_input1)
  y = Embedding(output_dim=300, input_dim=20000, 
  input_length=200, weights = [embed_matrix])(main_input2)
  
  # Next separately pass each layer thru a lstm layer to transform seq of   
  vectors into a single sequence
  lstm_out_x1 = LSTM(32)(x)
  lstm_out_x2 = LSTM(32)(y)
  
  # concatenate the 2 layers and stack a dense layer on top
  x = keras.layers.concatenate([lstm_out_x1, lstm_out_x2])
  x = Dense(64, activation='relu')(x)
  # generate intermediate output
  auxiliary_output = Dense(1, activation='sigmoid', name='aux_output')(x)
  
  # add auxiliary input - auxiliary inputs contains 25 features for each document pair
  auxiliary_input = Input(shape=(25,), name='aux_input')
  
  # merge aux output with aux input and stack dense layer on top
  main_input = keras.layers.concatenate([auxiliary_output, auxiliary_input])
  x = Dense(64, activation='relu')(main_input)
  x = Dense(64, activation='relu')(x)
  
  # finally add the main output layer
  main_output = Dense(1, activation='sigmoid', name='main_output')(x)
  
  model = Model(inputs=[main_input1, main_input2, auxiliary_input], outputs= main_output)
  model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
  
  model.fit([x1, x2,aux_input], y,
        epochs=3, batch_size=32)
  </code></pre>
  
  <p>However, when I score this on the training data, I get the same prob. score for all cases. The issue seems to be with the way auxiliary input is fed in (because it generates meaningful output when I remove the aux. input).
  I also tried inserting the auxiliary input at different places in the network. But somehow I couldnt get this to work.</p>
  
  <p>Any pointers?</p>