title: How to calculate perplexity of RNN in tensorflow
body: <p>I'm running the word RNN implmentation of tensor flow of <a href="https://github.com/hunkim/word-rnn-tensorflow">Word RNN</a></p>
  
  <p>How to calculate the perplexity of RNN.</p>
  
  <p>Following is the code in training that shows training loss and other things in each epoch:</p>
  
  <pre><code>for e in range(model.epoch_pointer.eval(), args.num_epochs):
          sess.run(tf.assign(model.lr, args.learning_rate * (args.decay_rate ** e)))
          data_loader.reset_batch_pointer()
          state = sess.run(model.initial_state)
          speed = 0
          if args.init_from is None:
              assign_op = model.batch_pointer.assign(0)
              sess.run(assign_op)
              assign_op = model.epoch_pointer.assign(e)
              sess.run(assign_op)
          if args.init_from is not None:
              data_loader.pointer = model.batch_pointer.eval()
              args.init_from = None
          for b in range(data_loader.pointer, data_loader.num_batches):
              start = time.time()
              x, y = data_loader.next_batch()
              feed = {model.input_data: x, model.targets: y, model.initial_state: state,
                      model.batch_time: speed}
              summary, train_loss, state, _, _ = sess.run([merged, model.cost, model.final_state,
                                                           model.train_op, model.inc_batch_pointer_op], feed)
              train_writer.add_summary(summary, e * data_loader.num_batches + b)
              speed = time.time() - start
              if (e * data_loader.num_batches + b) % args.batch_size == 0:
                  print("{}/{} (epoch {}), train_loss = {:.3f}, time/batch = {:.3f}" \
                      .format(e * data_loader.num_batches + b,
                              args.num_epochs * data_loader.num_batches,
                              e, train_loss, speed))
              if (e * data_loader.num_batches + b) % args.save_every == 0 \
                      or (e==args.num_epochs-1 and b == data_loader.num_batches-1): # save for the last result
                  checkpoint_path = os.path.join(args.save_dir, 'model.ckpt')
                  saver.save(sess, checkpoint_path, global_step = e * data_loader.num_batches + b)
                  print("model saved to {}".format(checkpoint_path))
      train_writer.close()
  </code></pre>