{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3738","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3738/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3738/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3738/events","html_url":"https://github.com/tensorflow/tensorflow/issues/3738","id":170529634,"number":3738,"title":"Very different sequence lengths in the same minibatch make LSTM inefficient","user":{"login":"msevrens","id":1441846,"avatar_url":"https://avatars.githubusercontent.com/u/1441846?v=3","gravatar_id":"","url":"https://api.github.com/users/msevrens","html_url":"https://github.com/msevrens","followers_url":"https://api.github.com/users/msevrens/followers","following_url":"https://api.github.com/users/msevrens/following{/other_user}","gists_url":"https://api.github.com/users/msevrens/gists{/gist_id}","starred_url":"https://api.github.com/users/msevrens/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/msevrens/subscriptions","organizations_url":"https://api.github.com/users/msevrens/orgs","repos_url":"https://api.github.com/users/msevrens/repos","events_url":"https://api.github.com/users/msevrens/events{/privacy}","received_events_url":"https://api.github.com/users/msevrens/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"yuanbyu","id":2342391,"avatar_url":"https://avatars.githubusercontent.com/u/2342391?v=3","gravatar_id":"","url":"https://api.github.com/users/yuanbyu","html_url":"https://github.com/yuanbyu","followers_url":"https://api.github.com/users/yuanbyu/followers","following_url":"https://api.github.com/users/yuanbyu/following{/other_user}","gists_url":"https://api.github.com/users/yuanbyu/gists{/gist_id}","starred_url":"https://api.github.com/users/yuanbyu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yuanbyu/subscriptions","organizations_url":"https://api.github.com/users/yuanbyu/orgs","repos_url":"https://api.github.com/users/yuanbyu/repos","events_url":"https://api.github.com/users/yuanbyu/events{/privacy}","received_events_url":"https://api.github.com/users/yuanbyu/received_events","type":"User","site_admin":false},"assignees":[{"login":"yuanbyu","id":2342391,"avatar_url":"https://avatars.githubusercontent.com/u/2342391?v=3","gravatar_id":"","url":"https://api.github.com/users/yuanbyu","html_url":"https://github.com/yuanbyu","followers_url":"https://api.github.com/users/yuanbyu/followers","following_url":"https://api.github.com/users/yuanbyu/following{/other_user}","gists_url":"https://api.github.com/users/yuanbyu/gists{/gist_id}","starred_url":"https://api.github.com/users/yuanbyu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yuanbyu/subscriptions","organizations_url":"https://api.github.com/users/yuanbyu/orgs","repos_url":"https://api.github.com/users/yuanbyu/repos","events_url":"https://api.github.com/users/yuanbyu/events{/privacy}","received_events_url":"https://api.github.com/users/yuanbyu/received_events","type":"User","site_admin":false}],"milestone":null,"comments":34,"created_at":"2016-08-10T22:02:41Z","updated_at":"2016-08-19T21:03:01Z","closed_at":null,"body":"I'm attempting to recreate this [model](https://github.com/bplank/bilstm-aux) in TensorFlow but I can not seem to get anywhere close to the speed of the original code. On average the TensorFlow implementation is taking 60 ms per document on a GPU, while the pycnn is taking 12 ms per document on a CPU.\r\n\r\nWhat appears to be the issue is that _rnn_step will call the LSTM cell on every time step regardless of whether the input at that position is padding or not and then simply fill those values with zeros after the fact using sequence_length. This seems like a lot of unnecessary calculations. \r\n\r\nProcessing the tokens sequentially in multiple session.run calls with one token per batch in a similar performance to the original code but the embeddings do not update as the flow is broken.\r\n\r\nCurrent working (but slow) code is below.\r\n\r\n```\r\ndef char_encoding(config, graph, doc_len):\r\n\t\"\"\"Create graph nodes for character encoding\"\"\"\r\n\r\n\tc2i = config[\"c2i\"]\r\n\tmax_tokens = config[\"max_tokens\"]\r\n\r\n\twith graph.as_default():\r\n\r\n\t\t# Character Embedding\r\n\t\tword_lengths = tf.placeholder(tf.int64, [None], name=\"word_lengths\")\r\n\t\tword_lengths = tf.gather(word_lengths, tf.range(tf.to_int32(doc_len)))\r\n\t\tchar_inputs = tf.placeholder(tf.int32, [None, max_tokens], name=\"char_inputs\")\r\n\t\tcembed_matrix = tf.Variable(tf.random_uniform([len(c2i.keys()), config[\"ce_dim\"]], -0.25, 0.25), name=\"cembeds\")\r\n\t\t\r\n\t\tchar_inputs = tf.transpose(char_inputs, perm=[1,0])\r\n\t\tcembeds = tf.nn.embedding_lookup(cembed_matrix, char_inputs, name=\"ce_lookup\")\r\n\t\tcembeds = tf.gather(cembeds, tf.range(tf.to_int32(doc_len)))\r\n\t\tcembeds = tf.transpose(cembeds, perm=[1,0,2])\r\n\r\n\t\t# Create LSTM for Character Encoding\r\n\t\tfw_lstm = tf.nn.rnn_cell.BasicLSTMCell(config[\"ce_dim\"], state_is_tuple=True)\r\n\t\tbw_lstm = tf.nn.rnn_cell.BasicLSTMCell(config[\"ce_dim\"], state_is_tuple=True)\r\n\r\n\t\t# Encode Characters with LSTM\r\n\t\toptions = {\r\n\t\t\t\"dtype\": tf.float32,\r\n\t\t\t\"sequence_length\": word_lengths,\r\n\t\t\t\"time_major\": True\r\n\t\t}\r\n\r\n\t\t(output_fw, output_bw), output_states = tf.nn.bidirectional_dynamic_rnn(fw_lstm, bw_lstm, cembeds, **options)\r\n\t\toutput_fw = tf.transpose(output_fw, perm=[1,0,2])\r\n\t\toutput_bw = tf.transpose(output_bw, perm=[1,0,2])\r\n\r\n\t\treturn output_fw, output_bw, word_lengths\r\n\r\n```\r\n\r\nAn alternate approach where I loop through the tokens in the graph itself seems like it should be faster but is throwing an odd error: \r\n\r\nCode:\r\n\r\n```\r\ndef char_encoding(config, graph, doc_len):\r\n\t\"\"\"Create graph nodes for character encoding\"\"\"\r\n\r\n\tc2i = config[\"c2i\"]\r\n\tmax_tokens = config[\"max_tokens\"]\r\n\r\n\twith graph.as_default():\r\n\r\n\t\t# Character Embedding\r\n\t\tword_lengths = tf.placeholder(tf.int64, [None], name=\"word_lengths\")\r\n\t\tword_lengths = tf.gather(word_lengths, tf.range(tf.to_int32(doc_len)))\r\n\t\tchar_inputs = tf.placeholder(tf.int32, [None, max_tokens], name=\"char_inputs\")\r\n\t\tchar_inputs = tf.transpose(char_inputs, perm=[1, 0])\r\n\t\tcembed_matrix = tf.Variable(tf.random_uniform([len(c2i.keys()), config[\"ce_dim\"]], -0.25, 0.25), name=\"cembeds\")\r\n\r\n\t\t# Create LSTM for Character Encoding\r\n\t\tfw_lstm = tf.nn.rnn_cell.BasicLSTMCell(config[\"ce_dim\"], state_is_tuple=True)\r\n\t\tbw_lstm = tf.nn.rnn_cell.BasicLSTMCell(config[\"ce_dim\"], state_is_tuple=True)\r\n\r\n\t\tdef one_pass(i, o_fw, o_bw):\r\n\r\n\t\t\tint_i = tf.to_int32(i)\r\n\r\n\t\t\toptions = {\r\n\t\t\t\t\"dtype\": tf.float32,\r\n\t\t\t\t\"sequence_length\": tf.expand_dims(tf.gather(word_lengths, int_i), 0),\r\n\t\t\t\t\"time_major\": True\r\n\t\t\t}\r\n\r\n\t\t\tcembeds_invert = tf.nn.embedding_lookup(cembed_matrix, tf.gather(char_inputs, int_i))\r\n\t\t\tcembeds_invert = tf.transpose(tf.expand_dims(cembeds_invert, 0), perm=[1,0,2])\r\n\r\n\t\t\t(output_fw, output_bw), output_states = tf.nn.bidirectional_dynamic_rnn(fw_lstm, bw_lstm, cembeds_invert, **options)\r\n\r\n\t\t\t# Get Last Relevant\r\n\t\t\toutput_fw = tf.gather(output_fw, tf.gather(word_lengths, int_i) - 1)\r\n\t\t\toutput_bw = tf.gather(output_bw, tf.gather(word_lengths, int_i) - 1)\r\n\r\n\t\t\t# Append to Previous Token Encodings\r\n\t\t\to_fw = o_fw.write(int_i, tf.squeeze(output_fw))\r\n\t\t\to_bw = o_bw.write(int_i, tf.squeeze(output_bw))\r\n\r\n\t\t\treturn tf.add(i, 1), o_fw, o_bw\r\n\r\n\t\t# Build Loop in Graph\r\n\t\ti = tf.constant(0.0)\r\n\t\tfloat_doc_len = tf.to_float(doc_len)\r\n\t\to_fw = tensor_array_ops.TensorArray(dtype=tf.float32, size=tf.to_int32(doc_len))\r\n\t\to_bw = tensor_array_ops.TensorArray(dtype=tf.float32, size=tf.to_int32(doc_len))\r\n\t\tcond = lambda i, *_: tf.less(i, float_doc_len)\r\n\t\ti, char_embeds, rev_char_embeds = tf.while_loop(cond, one_pass, [i, o_fw, o_bw])\r\n\r\n\t\treturn char_embeds.pack(), rev_char_embeds.pack()\r\n```\r\n\r\nError:\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python3.4/runpy.py\", line 170, in _run_module_as_main\r\n    \"__main__\", mod_spec)\r\n  File \"/usr/lib/python3.4/runpy.py\", line 85, in _run_code\r\n    exec(code, run_globals)\r\n  File \"/home/ubuntu/git/Meerkat/meerkat/longtail/bilstm_tagger.py\", line 445, in <module>\r\n    run_from_command_line()\r\n  File \"/home/ubuntu/git/Meerkat/meerkat/longtail/bilstm_tagger.py\", line 441, in run_from_command_line\r\n    graph, saver = build_graph(config)\r\n  File \"/home/ubuntu/git/Meerkat/meerkat/longtail/bilstm_tagger.py\", line 311, in build_graph\r\n    optimizer = tf.train.GradientDescentOptimizer(config[\"learning_rate\"]).minimize(loss, name=\"optimizer\")\r\n  File \"/usr/local/lib/python3.4/dist-packages/tensorflow/python/training/optimizer.py\", line 196, in minimize\r\n    grad_loss=grad_loss)\r\n  File \"/usr/local/lib/python3.4/dist-packages/tensorflow/python/training/optimizer.py\", line 253, in compute_gradients\r\n    colocate_gradients_with_ops=colocate_gradients_with_ops)\r\n  File \"/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/gradients.py\", line 478, in gradients\r\n    in_grads = _AsList(grad_fn(op, *out_grads))\r\n  File \"/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/control_flow_grad.py\", line 202, in _EnterGrad\r\n    result = grad_ctxt.AddBackPropAccumulator(op, grad)\r\n  File \"/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/control_flow_ops.py\", line 1646, in AddBackPropAccumulator\r\n    forward_ctxt = self.grad_state.forward_ctxt\r\nAttributeError: 'GradLoopState' object has no attribute 'forward_ctxt'\r\n```\r\n\r\nLet me know if seeing the full graph will help at all. Right now that higher level LSTM isn't causing issues because I can just run one document at a time."},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/241134499","html_url":"https://github.com/tensorflow/tensorflow/issues/3738#issuecomment-241134499","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3738","id":241134499,"user":{"login":"msevrens","id":1441846,"avatar_url":"https://avatars.githubusercontent.com/u/1441846?v=3","gravatar_id":"","url":"https://api.github.com/users/msevrens","html_url":"https://github.com/msevrens","followers_url":"https://api.github.com/users/msevrens/followers","following_url":"https://api.github.com/users/msevrens/following{/other_user}","gists_url":"https://api.github.com/users/msevrens/gists{/gist_id}","starred_url":"https://api.github.com/users/msevrens/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/msevrens/subscriptions","organizations_url":"https://api.github.com/users/msevrens/orgs","repos_url":"https://api.github.com/users/msevrens/repos","events_url":"https://api.github.com/users/msevrens/events{/privacy}","received_events_url":"https://api.github.com/users/msevrens/received_events","type":"User","site_admin":false},"created_at":"2016-08-19T21:03:01Z","updated_at":"2016-08-19T21:03:01Z","body":"@ebrevdo \r\nThe original paper is not using truncated back propagation through time. The final outputs from the forward and backward passes of the inner LSTM on each token are concatenated to a word embedding and fed into the higher level LSTM. There is no loss at every step as the loss comes from the higher level LSTM. \r\n\r\nWe also discussed bucketing above and that will not work either as the mini batch itself is the sentence input. Both the inner and outer LSTM are optimized together using the loss from the higher level LSTM. So the batch for the inner LSTM will always be the tokens from the sentence being tagged. \r\n\r\nThis architecture seems pretty tied to what is causing the performance issue. At this point, barring a major change to how LSTM's are handled by TensorFlow I believe the best solution is this while loop approach. \r\n\r\n@girving \r\nI agree, I'm going to change the title\r\n\r\n"}}
{"actor":{"display_login":"msevrens"}}