title: Keras TimeDistributed Not Masking CNN Model
body: <p>For the sake of example, I have an input consisting of 2 images,of total shape (2,299,299,3). I'm trying to apply inceptionv3 on each image, and then subsequently process the output with an LSTM. I'm using a masking layer to exclude a blank image from being processed (specified below).</p>
  
  <p>The code is:</p>
  
  <pre class="lang-py prettyprint-override"><code>import numpy as np
  from keras import backend as K
  from keras.models import Sequential,Model
  from keras.layers import Convolution2D, MaxPooling2D, ZeroPadding2D, BatchNormalization, \
  Input, GlobalAveragePooling2D, Masking,TimeDistributed, LSTM,Dense,Flatten,Reshape,Lambda, Concatenate
  from keras.layers import Activation, Dropout, Flatten, Dense
  from keras.applications import inception_v3
  
  IMG_SIZE=(299,299,3)
  def create_base():
      base_model = inception_v3.InceptionV3(weights='imagenet', include_top=False)
      x = GlobalAveragePooling2D()(base_model.output)
      base_model=Model(base_model.input,x)
      return base_model
  
  
  base_model=create_base()
  
  #Image mask to ignore images with pixel values of -1
  IMAGE_MASK = -2*np.expand_dims(np.ones(IMG_SIZE),0)
  
  final_input=Input((2,IMG_SIZE[0],IMG_SIZE[1],IMG_SIZE[2]))
  
  final_model = Masking(mask_value = -2.)(final_input)
  final_model = TimeDistributed(base_model)(final_model)
  final_model = Lambda(lambda x: x, output_shape=lambda s:s)(final_model)
  #final_model = Reshape(target_shape=(2, 2048))(final_model)
  #final_model = Masking(mask_value = 0.)(final_model)
  final_model = LSTM(5,return_sequences=False)(final_model)
  final_model = Model(final_input,final_model)
  
  
  #Create a sample test image
  TEST_IMAGE = np.ones(IMG_SIZE)
  
  #Create a test sample input, consisting of a normal image and a masked image
  TEST_SAMPLE = np.concatenate((np.expand_dims(TEST_IMAGE,axis=0),IMAGE_MASK))
  
  
  
  inp = final_model.input                                           # input placeholder
  outputs = [layer.output for layer in final_model.layers]          # all layer outputs
  functors = [K.function([inp]+ [K.learning_phase()], [out]) for out in outputs]
  layer_outs = [func([np.expand_dims(TEST_SAMPLE,0), 1.]) for func in functors]
  </code></pre>
  
  <p>This does not work correctly. Specifically, the model should mask the IMAGE_MASK part of the input, but it instead processes it with inception (giving a nonzero output). here are the details:</p>
  
  <p>layer_out[-1] , the LSTM output is fine:</p>
  
  <p><code>[array([[-0.15324114, -0.09620268, -0.01668587,  0.07938149, -0.00757846]], dtype=float32)]</code></p>
  
  <p>layer_out[-2] and layer_out[-3] , the LSTM input is <em>wrong</em>, it should have all zeros in the second array:</p>
  
  <p><code>[array([[[ 0.37713543,  0.36381325,  0.36197218, ...,  0.23298527,
             0.43247852,  0.34844452],
           [ 0.24972123,  0.2378867 ,  0.11810347, ...,  0.51930511,
             0.33289322,  0.33403745]]], dtype=float32)]</code></p>
  
  <p>layer_out[-4], the input to the CNN is correctly masked:</p>
  
  <pre><code>[[ 1.,  1.,  1.],
             [ 1.,  1.,  1.],
             [ 1.,  1.,  1.],
             ..., 
             [ 1.,  1.,  1.],
             [ 1.,  1.,  1.],
             [ 1.,  1.,  1.]]],
  
  
           [[[-0., -0., -0.],
             [-0., -0., -0.],
             [-0., -0., -0.],
             ..., 
             [-0., -0., -0.],
             [-0., -0., -0.],
             [-0., -0., -0.]],
  </code></pre>
  
  <p>Note that the code seems to work <strong>correctly</strong> with a simpler base_model such as:</p>
  
  <pre class="lang-py prettyprint-override"><code>def create_base():
      input_layer=Input(IMG_SIZE)
      base_model=Flatten()(input_layer)
      base_model=Dense(2048)(base_model)
      base_model=Model(input_layer,base_model)
      return base_model
  </code></pre>
  
  <p>I have exhausted most online resources on this. Permutations of this question have been asked on Keras's github, such as <a href="https://github.com/keras-team/keras/issues/5533" rel="noreferrer">here</a>, <a href="https://github.com/keras-team/keras/issues/9788" rel="noreferrer">here</a> and <a href="https://github.com/keras-team/keras/issues/5934#issuecomment-288811716" rel="noreferrer">here</a>, but I can't seem to find any concrete resolution.</p>
  
  <p>The links suggest that the issues seem to be stemming from a combination of TimeDistributed being applied to BatchNormalization, and the hacky fixes of either the Lambda identity layer, or Reshape layers remove errors but don't seem to output the correct model. </p>
  
  <p>I've tried to force the base model to support masking via:</p>
  
  <pre class="lang-py prettyprint-override"><code>base_model.__setattr__('supports_masking',True)
  </code></pre>
  
  <p>and I've also tried applying an identity layer via:</p>
  
  <pre class="lang-py prettyprint-override"><code>TimeDistributed(Lambda(lambda x: base_model(x), output_shape=lambda s:s))(final_model)
  </code></pre>
  
  <p>but none of these seem to work. Note that I would like the final model to be trainable, in particular the CNN part of it should remain trainable. </p>