title: Restore variables that are a subset of new model in Tensorflow?
body: <p>I am doing an example for boosting(4 layers DNN to 5 layers DNN) via Tensorflow. I am making it with save session and restore in TF because there is a brief paragraph in TF tute:
  'For example, you may have trained a neural net with 4 layers, and you now want to train a new model with 5 layers, restoring the parameters from the 4 layers of the previously trained model into the first 4 layers of the new model.', where tensorflow tute inspires in <a href="https://www.tensorflow.org/how_tos/variables/" rel="noreferrer">https://www.tensorflow.org/how_tos/variables/</a>.</p>
  
  <p>However, I found that nobody has asked how to use 'restore' when the checkpoint saves parameters of 4 layers but we need to put that into 5 layers, raising a red flag.</p>
  
  <p>Making this in real code, I made</p>
  
  <pre><code>with tf.name_scope('fcl1'):
      hidden_1 = fully_connected_layer(inputs, train_data.inputs.shape[1], num_hidden)            
  with tf.name_scope('fcl2'):
      hidden_2 = fully_connected_layer(hidden_1, num_hidden, num_hidden)                
  with tf.name_scope('fclf'):
      hidden_final = fully_connected_layer(hidden_2, num_hidden, num_hidden)    
  with tf.name_scope('outputl'):
      outputs = fully_connected_layer(hidden_final, num_hidden, train_data.num_classes, tf.identity)
      outputs = tf.nn.softmax(outputs)
  with tf.name_scope('boosting'):
      boosts = fully_connected_layer(outputs, train_data.num_classes, train_data.num_classes, tf.identity)
  </code></pre>
  
  <p>where variables inside(or called from) 'fcl1' - so that I have 'fcl1/Variable' and 'fcl1/Variable_1' for weight and bias - 'fcl2', 'fclf', and 'outputl' are stored by saver.save() in the script without 'boosting' layer. However, as we now have 'boosting' layer, saver.restore(sess, "saved_models/model_list.ckpt") does not work as</p>
  
  <pre><code>NotFoundError: Key boosting/Variable_1 not found in checkpoint
  </code></pre>
  
  <p>I really hope to hear about this problem. Thank you.
  Below code is the main part of the code I am in trouble.</p>
  
  <pre><code>def fully_connected_layer(inputs, input_dim, output_dim, nonlinearity=tf.nn.relu):
      weights = tf.Variable(
          tf.truncated_normal(
              [input_dim, output_dim], stddev=2. / (input_dim + output_dim)**0.5), 
          'weights')
      biases = tf.Variable(tf.zeros([output_dim]), 'biases')
      outputs = nonlinearity(tf.matmul(inputs, weights) + biases)    
  
      return outputs
  
  inputs = tf.placeholder(tf.float32, [None, train_data.inputs.shape[1]], 'inputs')
  targets = tf.placeholder(tf.float32, [None, train_data.num_classes], 'targets')
  
  with tf.name_scope('fcl1'):
      hidden_1 = fully_connected_layer(inputs, train_data.inputs.shape[1], num_hidden)            
  with tf.name_scope('fcl2'):
      hidden_2 = fully_connected_layer(hidden_1, num_hidden, num_hidden)                
  with tf.name_scope('fclf'):
      hidden_final = fully_connected_layer(hidden_2, num_hidden, num_hidden)    
  with tf.name_scope('outputl'):
      outputs = fully_connected_layer(hidden_final, num_hidden, train_data.num_classes, tf.identity)
  
  with tf.name_scope('error'):    
      error = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(outputs, targets))
  with tf.name_scope('accuracy'):
      accuracy = tf.reduce_mean(tf.cast(
          tf.equal(tf.argmax(outputs, 1), tf.argmax(targets, 1)), 
          tf.float32))
  with tf.name_scope('train'):
      train_step = tf.train.AdamOptimizer().minimize(error)
  
  init = tf.global_variables_initializer()  
  saver = tf.train.Saver()
  
  with tf.Session() as sess:
      sess.run(init)
      saver.restore(sess, "saved_models/model.ckpt")
      print("Model restored")
  
      print("Optimization Starts!")
      for e in range(training_epochs):
          ...
  
     #Save model - save session        
      save_path = saver.save(sess, "saved_models/model.ckpt")
      ### I once saved the variables using var_list, but didn't work as well...
      print("Model saved in file: %s" % save_path)
  </code></pre>
  
  <p>For clarity, the checkpoint file has </p>
  
  <pre><code>fcl1/Variable:0
  
  fcl1/Variable_1:0
  
  fcl2/Variable:0
  
  fcl2/Variable_1:0
  
  fclf/Variable:0
  
  fclf/Variable_1:0
  
  outputl/Variable:0
  
  outputl/Variable_1:0
  </code></pre>
  
  <p>As the original 4 layers model does not have 'boosting' layer.</p>