title: Evaluate a function in a sliding window with Keras
body: <p>I'm trying to extend a matching matching algorithm across a sequence. My matches are 20 units long and have 4 channels at each timepoint. I have built a model that encapsulates the matching, I just can't figure out how to use that in a sliding window to apply it across a longer sequence to find the matches within the sequence.</p>
  
  <p>I have 2 <code>(20, 4)</code> input tensors (<code>query</code> and <code>target</code>) that I concatenate, add, flatten, and then apply a simple dense layer. I have data at this stage to train with 100K query, target pairs.</p>
  
  <pre><code>def sum_seqs(seqs):
      return K.sum(seqs, axis=3)
  
  def pad_dims(seq):
      return K.expand_dims(seq, axis=3)
  
  def pad_outshape(in_shape):
      return (in_shape[0], in_shape[1], in_shape[2], 1)
  
  
  query = Input((20, 4))
  query_pad = Lambda(pad_dims, output_shape=pad_outshape, name='gpad')(query)
  
  target = Input((20,4))
  target_pad = Lambda(pad_dims, output_shape=pad_outshape)(target)
  
  matching = Concatenate(axis = 3)([query_pad, target_pad])
  matching = Lambda(sum_seqs)(matching)
  
  matching = Flatten()(matching)
  matching = Dropout(0.1)(matching)
  matching = Dense(1, activation = 'sigmoid')(matching)
  
  match_model = Model([query, target], matching)
  </code></pre>
  
  <p>This works perfectly. Now I want to use this pre-trained model to search a longer <code>target</code> sequence with varying <code>query</code> sequences.</p>
  
  <p>It seems it should be something like:</p>
  
  <pre><code>long_target = Input((100, 4))
  
  short_target = Input((20, 4))
  choose_query = Input((20, 4))
  
  spec_match = match_model([choose_query, short_target])
  
  mdl = TimeDistributed(spec_match)(long_target)
  </code></pre>
  
  <p>But <code>TimeDistributed</code> takes a <code>Layer</code> not a <code>Tensor</code>. Is there a wrapper I'm missing? Am I going about this the wrong way? Do I need to reformulate this as a convolution problem somehow?</p>
  
  <p>Continued experimentation:
  After a day of beating my head against the keyboard it is clear that both <code>TimeDistributed</code> and <code>backend.rnn</code> only allow you to apply a model/layer to a single time-slice of the data. It doesn't seem like there is a way to do this. It looks like the only thing that can "walk" across multiple slices of the time dimension is a <code>Conv1D</code>.</p>
  
  <p>So, I reframed my problem as a convolution but that doesn't work well either. I was able to building a <code>Conv1D</code> filter that it would match a specific <code>query</code>. This worked reasonably well and it did allow me to scan longer sequences and get matches. BUT each filter is unique to each <code>query</code> tensor and there doesn't seem to be a way to go from a novel <code>query</code> to the appropriate filter weights without training a whole new <code>Conv1D</code> layer. Since my goal is to find new <code>query</code>s which match the most targets this doesn't help much.</p>
  
  <p>Since my "matching" requires the interaction of the target AND the query at each window there doesn't seem to be a way I can get an interaction of a 20-length <code>query</code> tensor at each window across a 100-length <code>target</code> tensor through <code>Conv1D</code>. </p>
  
  <p>Is there any way to do this sliding window type evaluation in Keras/tensorflow? It seems like something so simple yet so far away. Is there a way I can do this that I'm not finding? </p>
  
  <p>Responses and further experimentation.</p>
  
  <p>The solutions from @today and @nuric work but they end up replicating the input <code>target</code> data in a tiling type fashion. So, for a query of length <code>m</code> there will be a little under <code>m</code> copies of the input data in the graph. I was hoping to find a solution that would actually "slide" the evaluation across the <code>target</code> without the duplication.</p>
  
  <p>Here's a version of the <code>Conv1D</code> almost solution I came up with.</p>
  
  <pre><code>query_weights = []
  
  for query, (targets, scores) in query_target_gen():
      single_query_model = Sequential()
      single_query_model.add(Conv1D(1, 20, input_shape = (20, 4)))
      single_query_model.add(Flatten())
  
      single_query_model.fit(targets, scores)
  
      query_weights.append(single_query_model.layers[0].get_weights())
  
  multi_query_model_long_targets = Sequential()
  multi_query_model_long_targets.add(Conv1D(len(query_weights), 20, input_shape = (100, 4)))
  
  multi_query_model_long_targets.layers[0].set_weights(combine_weights(query_weights))
  
  multi_query_model_long_targets.summary()
  </code></pre>
  
  <p>The <code>combine_weights</code> function just does some unpacking and matrix rearrangement to stack the filters in the way <code>Conv1D</code> wants. </p>
  
  <p>This solution fixes the data duplication issue but it screws me in other ways. One is data based ... my data contains many <code>query</code>, <code>target</code> pairs but it tends to be the same <code>target</code> many <code>query</code>s, since it is easier to generate the real-world data in that orientation. So, doing it this way makes the training difficult. Second, this assumes that each <code>query</code> works in an independent way, when in reality, I know that the <code>query</code>, <code>target</code> pairing is what is actually important. So it makes sense to use a model that can look at many examples of the pairs, and not individuals.</p>
  
  <p>Is there a way to combine both methods? Is there a way to make it so <code>Conv1D</code> takes both the long <code>target</code> tensor combine it with the constant <code>query</code> as it walks along the sequence?</p>