title: How to arrange the features extracted from a CNN in a grid to pass in an RNN Encoder in Keras or tf 2.0?
body: <p>I am trying to produce the code used in paper <a href="https://arxiv.org/pdf/1609.04938v1.pdf" rel="nofollow noreferrer">What You Get Is What You See:
  A Visual Markup Decompiler</a> which is basically implemented in <code>Lua</code> by Harvard NLP by the name of <a href="https://github.com/harvardnlp/im2markup/" rel="nofollow noreferrer">im2latex</a>. The problem is that all of the codes available in either in Torch or tensorflow 1. I want to implement he same code in <code>Keras</code> or tensorflow 2.0.</p>
  <p>There is a part in the paper where it is written as:</p>
  <p><strong>The model first extracts image features using a convolutional neural network (CNN) and arranges the features in a grid. Each row is then encoded using a recurrent neural network (RNN)</strong></p>
  <p><a href="https://i.stack.imgur.com/nX6jy.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/nX6jy.png" alt="enter image description here" /></a></p>
  <p>So it is basically a three part process for the encodr part.
  Here is what I have done is that I have VGG19 model for transfer learning like. In my model, I'll be using the output of the VGG19 model to get the features as</p>
  <p>Part-1 : Extract the Features from the VGG-19</p>
  <pre><code>extracted_feat = tf.keras.applications.VGG19(include_top=False,weights=&quot;imagenet&quot;,
  input_shape=(224,224,3),pooling='avg',)
  </code></pre>
  <p>Part-2: Change the extracted features in the Grid</p>
  <pre><code>grid_output = change_to_grid(extracted_feat)
  </code></pre>
  <p>what and how Should the method <code>change_to_grid</code> be doing here?? How am I supposed to make a grid out of the tensors? By transposing or what?</p>
  <p>Part-3 : Pass Extracted Features to an LSTM or RNN to act as encoder</p>
  <pre><code>encoder_output = tf.keras.layers.LSTM(254,return_sequence=True,return_state=true)(grid_output)
  </code></pre>
  <p><strong>Please suggest what should be the code for the second part</strong>. Also please suggest if I am doing something wrong here.</p>