title: How to combine FCNN and RNN in Tensorflow?
body: <p>I want to make a Neural Network, which would have recurrency (for example, LSTM) at some layers and normal connections (FC) at others.
  I cannot find a way to do it in Tensorflow.
  It works, if I have only FC layers, but I don't see how to add just one recurrent layer properly.</p>
  
  <p>I create a network in a following way : </p>
  
  <pre><code>with tf.variable_scope("autoencoder_variables", reuse=None) as scope:
  
    for i in xrange(self.__num_hidden_layers + 1):
      # Train weights
      name_w = self._weights_str.format(i + 1)
      w_shape = (self.__shape[i], self.__shape[i + 1])
      a = tf.multiply(4.0, tf.sqrt(6.0 / (w_shape[0] + w_shape[1])))
      w_init = tf.random_uniform(w_shape, -1 * a, a)
      self[name_w] = tf.Variable(w_init,
                                 name=name_w,
                                 trainable=True)
      # Train biases
      name_b = self._biases_str.format(i + 1)
      b_shape = (self.__shape[i + 1],)
      b_init = tf.zeros(b_shape)
      self[name_b] = tf.Variable(b_init, trainable=True, name=name_b)
  
      if i+1 == self.__recurrent_layer:
        # Create an LSTM cell
        lstm_size = self.__shape[self.__recurrent_layer]
        self['lstm'] = tf.contrib.rnn.BasicLSTMCell(lstm_size)
  </code></pre>
  
  <p>It should process the batches in a sequential order. I have a function for processing just one time-step, which will be called later, by a function, which process the whole sequence :</p>
  
  <pre><code>def single_run(self, input_pl, state, just_middle = False):
    """Get the output of the autoencoder for a single batch
  
    Args:
      input_pl:     tf placeholder for ae input data of size [batch_size, DoF]
      state:        current state of LSTM memory units
      just_middle : will indicate if we want to extract only the middle layer of the network
    Returns:
      Tensor of output
    """
  
    last_output = input_pl
  
    # Pass through the network
    for i in xrange(self.num_hidden_layers+1):
  
      if(i!=self.__recurrent_layer):
        w = self._w(i + 1)
        b = self._b(i + 1)
        last_output = self._activate(last_output, w, b)
  
      else:
        last_output, state = self['lstm'](last_output,state)
  
    return last_output
  </code></pre>
  
  <p>The following function should take sequence of batches as input and produce sequence of batches as an output:</p>
  
  <pre><code>def process_sequences(self, input_seq_pl, dropout, just_middle = False):
    """Get the output of the autoencoder
  
    Args:
      input_seq_pl:     input data of size [batch_size, sequence_length, DoF]
      dropout:          dropout rate
      just_middle :     indicate if we want to extract only the middle layer of the network
    Returns:
      Tensor of output
    """
  
    if(~just_middle): # if not middle layer
      numb_layers = self.__num_hidden_layers+1
    else:
      numb_layers = FLAGS.middle_layer
  
    with tf.variable_scope("process_sequence", reuse=None) as scope:
  
      # Initial state of the LSTM memory.
      state = initial_state = self['lstm'].zero_state(FLAGS.batch_size, tf.float32)
  
      tf.get_variable_scope().reuse_variables() # THIS IS IMPORTANT LINE
  
      # First - Apply Dropout
      the_whole_sequences = tf.nn.dropout(input_seq_pl, dropout)
  
      # Take batches for every time step and run them through the network
      # Stack all their outputs
      with tf.control_dependencies([tf.convert_to_tensor(state, name='state') ]): # do not let paralelize the loop
        stacked_outputs = tf.stack( [ self.single_run(the_whole_sequences[:,time_st,:], state, just_middle) for time_st in range(self.sequence_length) ])
  
      # Transpose output from the shape [sequence_length, batch_size, DoF] into [batch_size, sequence_length, DoF]
  
      output = tf.transpose(stacked_outputs , perm=[1, 0, 2])
  
    return output
  </code></pre>
  
  <p>The issue is with a variable scopes and their property "reuse".</p>
  
  <p>If I run this code as it is I am getting the following error:
  ' Variable Train/process_sequence/basic_lstm_cell/weights does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope? '</p>
  
  <p>If I comment out the line, which tell it to reuse variables ( tf.get_variable_scope().reuse_variables() ) I am getting the following error:
  'Variable Train/process_sequence/basic_lstm_cell/weights already exists, disallowed. Did you mean to set reuse=True in VarScope?'</p>
  
  <p>It seems, that we need "reuse=None" for the weights of the LSTM cell to be initialized and we need "reuse=True" in order to call the LSTM cell. </p>
  
  <p>Please, help me to figure out the way to do it properly.</p>