title: tensorflow training slow down
body: <p>It seems that tensorflow slow down when training. Some people said that this is because adding some operation into the graph as the training proceed. However, I can't find anywhere. Here's my code.</p>
  
  <pre><code>with tf.Session() as sess:
      tf.global_variables_initializer().run()
      tf.local_variables_initializer().run()
      loader.restore(sess, checkpoint_file)
      summaries_loger_train.add_graph(sess.graph)
  
      coord = tf.train.Coordinator()
      threads = tf.train.start_queue_runners(coord=coord)
      tf.get_default_graph().finalize()
      for i in range(10000000):
          imgs_r, labels_r= sess.run([imgs, labels])
          _, loss_r, cross_entropy_loss_r, center_loss_r, acc_r, lr_r, s_r, gs_r= sess.run([opt, loss, cross_entropy_mean, center_loss_without_decay, accuracy, lr, summaries, global_step],
                                          feed_dict={imgs_p: imgs_r, labels_p: labels_r})
          print(gs_r, loss_r, cross_entropy_loss_r, center_loss_r, acc_r, lr_r)
          summaries_loger_train.add_summary(s_r, gs_r)
          if (gs_r) % 100 == 1:
              imgs_r, labels_r= sess.run([imgs_test, labels_test])
              cross_entropy_loss_r, acc_r, s_r= sess.run([cross_entropy_mean, accuracy, summaries],
                                          feed_dict={imgs_p: imgs_r, labels_p: labels_r})
              summaries_loger_test.add_summary(s_r, gs_r)
              print('test', cross_entropy_loss_r, acc_r)
          if (gs_r) % 1000 == 0:
              saver.save(sess, 'models2/model.ckpt', global_step=gs_r)
  
      coord.request_stop()
      coord.join(threads)
  </code></pre>