{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/1733","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/1733/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/1733/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/1733/events","html_url":"https://github.com/tensorflow/tensorflow/issues/1733","id":145028886,"number":1733,"title":"rnn.dynamic_rnn() causes gradients graph building error","user":{"login":"yaoli","id":1051367,"avatar_url":"https://avatars.githubusercontent.com/u/1051367?v=3","gravatar_id":"","url":"https://api.github.com/users/yaoli","html_url":"https://github.com/yaoli","followers_url":"https://api.github.com/users/yaoli/followers","following_url":"https://api.github.com/users/yaoli/following{/other_user}","gists_url":"https://api.github.com/users/yaoli/gists{/gist_id}","starred_url":"https://api.github.com/users/yaoli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yaoli/subscriptions","organizations_url":"https://api.github.com/users/yaoli/orgs","repos_url":"https://api.github.com/users/yaoli/repos","events_url":"https://api.github.com/users/yaoli/events{/privacy}","received_events_url":"https://api.github.com/users/yaoli/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"ebrevdo","id":1794715,"avatar_url":"https://avatars.githubusercontent.com/u/1794715?v=3","gravatar_id":"","url":"https://api.github.com/users/ebrevdo","html_url":"https://github.com/ebrevdo","followers_url":"https://api.github.com/users/ebrevdo/followers","following_url":"https://api.github.com/users/ebrevdo/following{/other_user}","gists_url":"https://api.github.com/users/ebrevdo/gists{/gist_id}","starred_url":"https://api.github.com/users/ebrevdo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ebrevdo/subscriptions","organizations_url":"https://api.github.com/users/ebrevdo/orgs","repos_url":"https://api.github.com/users/ebrevdo/repos","events_url":"https://api.github.com/users/ebrevdo/events{/privacy}","received_events_url":"https://api.github.com/users/ebrevdo/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2016-03-31T21:52:27Z","updated_at":"2016-03-31T23:20:21Z","closed_at":null,"body":"I found rnn.dynamic_rnn() that seems to do what I want, but when I modified the following lines \r\nhttps://github.com/tensorflow/tensorflow/blob/master/tensorflow/models/rnn/ptb/ptb_word_lm.py#L114-L116 into \r\n```\r\nfrom tensorflow.models.rnn import rnn\r\noutputs, state = rnn.dynamic_rnn(cell, inputs, initial_state=self._initial_state)\r\noutput = tf.reshape(outputs, [-1, size])\r\n```\r\nIt throws out error at the line containing `tf.gradients(cost, tvars)`\r\n\r\n```\r\n  File \"/data/lisatmp3/yaoli/anaconda/lib/python2.7/site-packages/tensorflow/python/platform/default/_app.py\", line 30, in run\r\n    sys.exit(main(sys.argv))\r\n  File \"ptb_word_lm.py\", line 304, in main\r\n    m = PTBModel(is_training=True, config=config)\r\n  File \"ptb_word_lm.py\", line 148, in __init__\r\n    grads, _ = tf.clip_by_global_norm(tf.gradients(cost, tvars),\r\n  File \"/data/lisatmp3/yaoli/anaconda/lib/python2.7/site-packages/tensorflow/python/ops/gradients.py\", line 477, in gradients\r\n    in_grads = _AsList(grad_fn(op, *out_grads))\r\n  File \"/data/lisatmp3/yaoli/anaconda/lib/python2.7/site-packages/tensorflow/python/ops/tensor_array_grad.py\", line 137, in _TensorArrayPackGrad\r\n    grad_source = _GetGradSource(grad)\r\n  File \"/data/lisatmp3/yaoli/anaconda/lib/python2.7/site-packages/tensorflow/python/ops/tensor_array_grad.py\", line 62, in _GetGradSource\r\n    % op_or_tensor.name)\r\nValueError: Expected op/tensor name to start with gradients, got: model/gradients/model/RNN/transpose_grad/transpose:0\r\nUncaught exception. Entering post mortem debugging\r\nRunning 'cont' or 'step' will restart the program\r\n```\r\nI couldn't find any examples in the repo of using dynamic_rnn(), and hope someone could point out where it went wrong. The complete file is attached below (adapted based on `tensorflow/tensorflow/models/rnn/ptb/ptb_word_lm.py`). LOOP_VERSION=0 and 1 run all right, 2 raises the issue.  \r\n\r\n```\r\n# Copyright 2015 Google Inc. All Rights Reserved.\r\n#\r\n# Licensed under the Apache License, Version 2.0 (the \"License\");\r\n# you may not use this file except in compliance with the License.\r\n# You may obtain a copy of the License at\r\n#\r\n#     http://www.apache.org/licenses/LICENSE-2.0\r\n#\r\n# Unless required by applicable law or agreed to in writing, software\r\n# distributed under the License is distributed on an \"AS IS\" BASIS,\r\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\r\n# See the License for the specific language governing permissions and\r\n# limitations under the License.\r\n# ==============================================================================\r\n\r\n\"\"\"Example / benchmark for building a PTB LSTM model.\r\n\r\nTrains the model described in:\r\n(Zaremba, et. al.) Recurrent Neural Network Regularization\r\nhttp://arxiv.org/abs/1409.2329\r\n\r\nThere are 3 supported model configurations:\r\n===========================================\r\n| config | epochs | train | valid  | test\r\n===========================================\r\n| small  | 13     | 37.99 | 121.39 | 115.91\r\n| medium | 39     | 48.45 |  86.16 |  82.07\r\n| large  | 55     | 37.87 |  82.62 |  78.29\r\nThe exact results may vary depending on the random initialization.\r\n\r\nThe hyperparameters used in the model:\r\n- init_scale - the initial scale of the weights\r\n- learning_rate - the initial value of the learning rate\r\n- max_grad_norm - the maximum permissible norm of the gradient\r\n- num_layers - the number of LSTM layers\r\n- num_steps - the number of unrolled steps of LSTM\r\n- hidden_size - the number of LSTM units\r\n- max_epoch - the number of epochs trained with the initial learning rate\r\n- max_max_epoch - the total number of epochs for training\r\n- keep_prob - the probability of keeping weights in the dropout layer\r\n- lr_decay - the decay of the learning rate for each epoch after \"max_epoch\"\r\n- batch_size - the batch size\r\n\r\nThe data required for this example is in the data/ dir of the\r\nPTB dataset from Tomas Mikolov's webpage:\r\n\r\n$ wget http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz\r\n$ tar xvf simple-examples.tgz\r\n\r\nTo run:\r\n\r\n$ python ptb_word_lm.py --data_path=simple-examples/data/\r\n\r\n\"\"\"\r\nfrom __future__ import absolute_import\r\nfrom __future__ import division\r\nfrom __future__ import print_function\r\n\r\nimport time\r\n\r\nimport numpy as np\r\nimport tensorflow as tf\r\n\r\nfrom tensorflow.models.rnn.ptb import reader\r\n\r\nflags = tf.flags\r\nlogging = tf.logging\r\n\r\nflags.DEFINE_string(\r\n    \"model\", \"small\",\r\n    \"A type of model. Possible options are: small, medium, large.\")\r\nflags.DEFINE_string(\"data_path\", None, \"data_path\")\r\n\r\nFLAGS = flags.FLAGS\r\n\r\nLOOP_VERSION = 2\r\n\r\nclass PTBModel(object):\r\n  \"\"\"The PTB model.\"\"\"\r\n\r\n  def __init__(self, is_training, config):\r\n    self.batch_size = batch_size = config.batch_size\r\n    self.num_steps = num_steps = config.num_steps\r\n    size = config.hidden_size\r\n    vocab_size = config.vocab_size\r\n\r\n    self._input_data = tf.placeholder(tf.int32, [batch_size, num_steps])\r\n    self._targets = tf.placeholder(tf.int32, [batch_size, num_steps])\r\n\r\n    # Slightly better results can be obtained with forget gate biases\r\n    # initialized to 1 but the hyperparameters of the model would need to be\r\n    # different than reported in the paper.\r\n    lstm_cell = tf.nn.rnn_cell.BasicLSTMCell(size, forget_bias=0.0)\r\n    if is_training and config.keep_prob < 1:\r\n      lstm_cell = tf.nn.rnn_cell.DropoutWrapper(\r\n          lstm_cell, output_keep_prob=config.keep_prob)\r\n    cell = tf.nn.rnn_cell.MultiRNNCell([lstm_cell] * config.num_layers)\r\n    self._initial_state = cell.zero_state(batch_size, tf.float32)\r\n\r\n    with tf.device(\"/cpu:0\"):\r\n      embedding = tf.get_variable(\"embedding\", [vocab_size, size])\r\n      inputs = tf.nn.embedding_lookup(embedding, self._input_data) # (b, t, 200)\r\n\r\n    if is_training and config.keep_prob < 1:\r\n      inputs = tf.nn.dropout(inputs, config.keep_prob)\r\n\r\n    # Simplified version of tensorflow.models.rnn.rnn.py's rnn().\r\n    # This builds an unrolled LSTM for tutorial purposes only.\r\n    # In general, use the rnn() or state_saving_rnn() from rnn.py.\r\n    #\r\n    # The alternative version of the code below is:\r\n    if LOOP_VERSION == 0:\r\n      from tensorflow.models.rnn import rnn\r\n      inputs = [tf.squeeze(input_, [1])\r\n               for input_ in tf.split(1, num_steps, inputs)]\r\n      outputs, state = rnn.rnn(cell, inputs, initial_state=self._initial_state)\r\n      output = tf.reshape(tf.concat(1, outputs), [-1, size])\r\n    if LOOP_VERSION == 1:\r\n      outputs = []\r\n      state = self._initial_state\r\n      with tf.variable_scope(\"RNN\"):\r\n        for time_step in range(num_steps):\r\n          if time_step > 0: tf.get_variable_scope().reuse_variables()\r\n          (cell_output, state) = cell(inputs[:, time_step, :], state)\r\n          outputs.append(cell_output)\r\n      output = tf.reshape(tf.concat(1, outputs), [-1, size])\r\n    if LOOP_VERSION == 2:\r\n      from tensorflow.models.rnn import rnn\r\n      # inputs: (b,t,d)\r\n      outputs, state = rnn.dynamic_rnn(cell, inputs, initial_state=self._initial_state)\r\n      output = tf.reshape(outputs, [-1, size])\r\n\r\n    softmax_w = tf.get_variable(\"softmax_w\", [size, vocab_size])\r\n    softmax_b = tf.get_variable(\"softmax_b\", [vocab_size])\r\n    logits = tf.matmul(output, softmax_w) + softmax_b\r\n    loss = tf.nn.seq2seq.sequence_loss_by_example(\r\n        [logits],\r\n        [tf.reshape(self._targets, [-1])],\r\n        [tf.ones([batch_size * num_steps])])\r\n    self._cost = cost = tf.reduce_sum(loss) / batch_size\r\n    self._final_state = state\r\n\r\n    if not is_training:\r\n      return\r\n\r\n    self._lr = tf.Variable(0.0, trainable=False)\r\n    tvars = tf.trainable_variables()\r\n    grads, _ = tf.clip_by_global_norm(tf.gradients(cost, tvars),\r\n                                      config.max_grad_norm)\r\n\r\n    optimizer = tf.train.GradientDescentOptimizer(self.lr)\r\n    self._train_op = optimizer.apply_gradients(zip(grads, tvars))\r\n\r\n  def assign_lr(self, session, lr_value):\r\n    session.run(tf.assign(self.lr, lr_value))\r\n\r\n  @property\r\n  def input_data(self):\r\n    return self._input_data\r\n\r\n  @property\r\n  def targets(self):\r\n    return self._targets\r\n\r\n  @property\r\n  def initial_state(self):\r\n    return self._initial_state\r\n\r\n  @property\r\n  def cost(self):\r\n    return self._cost\r\n\r\n  @property\r\n  def final_state(self):\r\n    return self._final_state\r\n\r\n  @property\r\n  def lr(self):\r\n    return self._lr\r\n\r\n  @property\r\n  def train_op(self):\r\n    return self._train_op\r\n\r\n\r\nclass SmallConfig(object):\r\n  \"\"\"Small config.\"\"\"\r\n  init_scale = 0.1\r\n  learning_rate = 1.0\r\n  max_grad_norm = 5\r\n  num_layers = 2\r\n  num_steps = 20\r\n  hidden_size = 200\r\n  max_epoch = 4\r\n  max_max_epoch = 13\r\n  keep_prob = 1.0\r\n  lr_decay = 0.5\r\n  batch_size = 20\r\n  vocab_size = 10000\r\n\r\n\r\nclass MediumConfig(object):\r\n  \"\"\"Medium config.\"\"\"\r\n  init_scale = 0.05\r\n  learning_rate = 1.0\r\n  max_grad_norm = 5\r\n  num_layers = 2\r\n  num_steps = 35\r\n  hidden_size = 650\r\n  max_epoch = 6\r\n  max_max_epoch = 39\r\n  keep_prob = 0.5\r\n  lr_decay = 0.8\r\n  batch_size = 20\r\n  vocab_size = 10000\r\n\r\n\r\nclass LargeConfig(object):\r\n  \"\"\"Large config.\"\"\"\r\n  init_scale = 0.04\r\n  learning_rate = 1.0\r\n  max_grad_norm = 10\r\n  num_layers = 2\r\n  num_steps = 35\r\n  hidden_size = 1500\r\n  max_epoch = 14\r\n  max_max_epoch = 55\r\n  keep_prob = 0.35\r\n  lr_decay = 1 / 1.15\r\n  batch_size = 20\r\n  vocab_size = 10000\r\n\r\n\r\nclass TestConfig(object):\r\n  \"\"\"Tiny config, for testing.\"\"\"\r\n  init_scale = 0.1\r\n  learning_rate = 1.0\r\n  max_grad_norm = 1\r\n  num_layers = 1\r\n  num_steps = 2\r\n  hidden_size = 2\r\n  max_epoch = 1\r\n  max_max_epoch = 1\r\n  keep_prob = 1.0\r\n  lr_decay = 0.5\r\n  batch_size = 20\r\n  vocab_size = 10000\r\n\r\n\r\ndef run_epoch(session, m, data, eval_op, verbose=False):\r\n  \"\"\"Runs the model on the given data.\"\"\"\r\n  epoch_size = ((len(data) // m.batch_size) - 1) // m.num_steps\r\n  start_time = time.time()\r\n  costs = 0.0\r\n  iters = 0\r\n  state = m.initial_state.eval()\r\n  for step, (x, y) in enumerate(reader.ptb_iterator(data, m.batch_size,\r\n                                                    m.num_steps)):\r\n    cost, state, _ = session.run([m.cost, m.final_state, eval_op],\r\n                                 {m.input_data: x,\r\n                                  m.targets: y,\r\n                                  m.initial_state: state})\r\n    costs += cost\r\n    iters += m.num_steps\r\n\r\n    if verbose and step % (epoch_size // 10) == 10:\r\n      print(\"%.3f perplexity: %.3f speed: %.0f wps\" %\r\n            (step * 1.0 / epoch_size, np.exp(costs / iters),\r\n             iters * m.batch_size / (time.time() - start_time)))\r\n\r\n  return np.exp(costs / iters)\r\n\r\n\r\ndef get_config():\r\n  if FLAGS.model == \"small\":\r\n    return SmallConfig()\r\n  elif FLAGS.model == \"medium\":\r\n    return MediumConfig()\r\n  elif FLAGS.model == \"large\":\r\n    return LargeConfig()\r\n  elif FLAGS.model == \"test\":\r\n    return TestConfig()\r\n  else:\r\n    raise ValueError(\"Invalid model: %s\", FLAGS.model)\r\n\r\n\r\ndef main(_):\r\n  if not FLAGS.data_path:\r\n    raise ValueError(\"Must set --data_path to PTB data directory\")\r\n\r\n  raw_data = reader.ptb_raw_data(FLAGS.data_path)\r\n  # train: 929589, valid: 73760, test: 82430\r\n  train_data, valid_data, test_data, _ = raw_data\r\n\r\n  config = get_config()\r\n  eval_config = get_config()\r\n  eval_config.batch_size = 1\r\n  eval_config.num_steps = 1\r\n\r\n  with tf.Graph().as_default(), tf.Session() as session:\r\n    initializer = tf.random_uniform_initializer(-config.init_scale,\r\n                                                config.init_scale)\r\n    with tf.variable_scope(\"model\", reuse=None, initializer=initializer):\r\n      m = PTBModel(is_training=True, config=config)\r\n    with tf.variable_scope(\"model\", reuse=True, initializer=initializer):\r\n      mvalid = PTBModel(is_training=False, config=config)\r\n      mtest = PTBModel(is_training=False, config=eval_config)\r\n\r\n    tf.initialize_all_variables().run()\r\n\r\n    for i in range(config.max_max_epoch):\r\n      lr_decay = config.lr_decay ** max(i - config.max_epoch, 0.0)\r\n      m.assign_lr(session, config.learning_rate * lr_decay)\r\n\r\n      print(\"Epoch: %d Learning rate: %.3f\" % (i + 1, session.run(m.lr)))\r\n      train_perplexity = run_epoch(session, m, train_data, m.train_op,\r\n                                   verbose=True)\r\n      print(\"Epoch: %d Train Perplexity: %.3f\" % (i + 1, train_perplexity))\r\n      valid_perplexity = run_epoch(session, mvalid, valid_data, tf.no_op())\r\n      print(\"Epoch: %d Valid Perplexity: %.3f\" % (i + 1, valid_perplexity))\r\n\r\n    test_perplexity = run_epoch(session, mtest, test_data, tf.no_op())\r\n    print(\"Test Perplexity: %.3f\" % test_perplexity)\r\n\r\n\r\nif __name__ == \"__main__\":\r\n  tf.app.run()\r\n\r\n\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/204170366","html_url":"https://github.com/tensorflow/tensorflow/issues/1733#issuecomment-204170366","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/1733","id":204170366,"user":{"login":"ebrevdo","id":1794715,"avatar_url":"https://avatars.githubusercontent.com/u/1794715?v=3","gravatar_id":"","url":"https://api.github.com/users/ebrevdo","html_url":"https://github.com/ebrevdo","followers_url":"https://api.github.com/users/ebrevdo/followers","following_url":"https://api.github.com/users/ebrevdo/following{/other_user}","gists_url":"https://api.github.com/users/ebrevdo/gists{/gist_id}","starred_url":"https://api.github.com/users/ebrevdo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ebrevdo/subscriptions","organizations_url":"https://api.github.com/users/ebrevdo/orgs","repos_url":"https://api.github.com/users/ebrevdo/repos","events_url":"https://api.github.com/users/ebrevdo/events{/privacy}","received_events_url":"https://api.github.com/users/ebrevdo/received_events","type":"User","site_admin":false},"created_at":"2016-03-31T23:20:21Z","updated_at":"2016-03-31T23:20:21Z","body":"This has been fixed internally and is waiting for the next push to nightly."}}