{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/511","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/511/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/511/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/511/events","html_url":"https://github.com/tensorflow/tensorflow/issues/511","id":122146339,"number":511,"title":"Large RNN graphs, incredibly slow graph creation and step time...","user":{"login":"wchan","id":1131892,"avatar_url":"https://avatars.githubusercontent.com/u/1131892?v=3","gravatar_id":"","url":"https://api.github.com/users/wchan","html_url":"https://github.com/wchan","followers_url":"https://api.github.com/users/wchan/followers","following_url":"https://api.github.com/users/wchan/following{/other_user}","gists_url":"https://api.github.com/users/wchan/gists{/gist_id}","starred_url":"https://api.github.com/users/wchan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wchan/subscriptions","organizations_url":"https://api.github.com/users/wchan/orgs","repos_url":"https://api.github.com/users/wchan/repos","events_url":"https://api.github.com/users/wchan/events{/privacy}","received_events_url":"https://api.github.com/users/wchan/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"closed","locked":false,"assignee":{"login":"ebrevdo","id":1794715,"avatar_url":"https://avatars.githubusercontent.com/u/1794715?v=3","gravatar_id":"","url":"https://api.github.com/users/ebrevdo","html_url":"https://github.com/ebrevdo","followers_url":"https://api.github.com/users/ebrevdo/followers","following_url":"https://api.github.com/users/ebrevdo/following{/other_user}","gists_url":"https://api.github.com/users/ebrevdo/gists{/gist_id}","starred_url":"https://api.github.com/users/ebrevdo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ebrevdo/subscriptions","organizations_url":"https://api.github.com/users/ebrevdo/orgs","repos_url":"https://api.github.com/users/ebrevdo/repos","events_url":"https://api.github.com/users/ebrevdo/events{/privacy}","received_events_url":"https://api.github.com/users/ebrevdo/received_events","type":"User","site_admin":false},"assignees":[{"login":"ebrevdo","id":1794715,"avatar_url":"https://avatars.githubusercontent.com/u/1794715?v=3","gravatar_id":"","url":"https://api.github.com/users/ebrevdo","html_url":"https://github.com/ebrevdo","followers_url":"https://api.github.com/users/ebrevdo/followers","following_url":"https://api.github.com/users/ebrevdo/following{/other_user}","gists_url":"https://api.github.com/users/ebrevdo/gists{/gist_id}","starred_url":"https://api.github.com/users/ebrevdo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ebrevdo/subscriptions","organizations_url":"https://api.github.com/users/ebrevdo/orgs","repos_url":"https://api.github.com/users/ebrevdo/repos","events_url":"https://api.github.com/users/ebrevdo/events{/privacy}","received_events_url":"https://api.github.com/users/ebrevdo/received_events","type":"User","site_admin":false}],"milestone":null,"comments":44,"created_at":"2015-12-14T22:28:03Z","updated_at":"2016-07-01T15:08:43Z","closed_at":"2016-06-08T06:13:22Z","body":"When I build a large seq2seq RNN graph (i.e., 3000 timesteps encoder, and 100 time steps decoder), both the graph construction and step time is insanely slow.\r\n\r\nFor example, using approx 3000 timesteps of stacked 4 layers of GRUs (in a seq2seq model w/ attention), we get something like this:\r\n\r\n*** graph creation time ***\r\ncreate_encoder graph time 77.442740\r\ncreate_decoder graph time 7.298214\r\ncreate_loss graph time 0.934293\r\ncreate_optimizer graph time 349.426908\r\ncreate_model graph time 489.119399\r\n\r\ncreate_optimizer is the part where i call something like this (pulled from the tutorial):\r\n  def create_optimizer(self):\r\n    start_time = time.time()\r\n\r\n    params = tf.trainable_variables()\r\n\r\n    self.gradient_norms = []\r\n    self.updates = []\r\n    opt = tf.train.GradientDescentOptimizer(0.001)\r\n\r\n    gradients = tf.gradients(self.losses, params)\r\n    clipped_gradients, norm = tf.clip_by_global_norm(gradients, self.max_gradient_norm)\r\n\r\n    self.gradient_norms.append(norm)\r\n    self.updates.append(opt.apply_gradients(\r\n        zip(clipped_gradients, params), global_step=self.global_step))\r\n\r\n    print('create_optimizer graph time %f' % (time.time() - start_time))\r\n\r\n*** step time ***\r\nstep_time: 142.356251001\r\n\r\nWhen I run nvidia-smi while the graph is computing , the utilization is something like 12% ... definitely something wrong...\r\n\r\nIn my personal framework (not TF), creating a graph of equal size takes approx 2 secs (as opposed to TF almost 8 mins) and the step time is less 4 seconds (as opposed to TF 2 mins).... something really weird is going on.\r\n\r\nI wasn't able to profile / debug it too much (are there public tools for that? i cant find anything on tensorboard). also, is there a command i can run to confirm how many weights i have (and the dimensions and the device location of the variables).\r\n\r\nI've confirmed with the tensorboard and log_device_placement=True, and seems like most of the graph is in the GPU (except for embedding layers).\r\n\r\nIf you want a link to my graph, send me an email.\r\n\r\nThanks All!"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/229971665","html_url":"https://github.com/tensorflow/tensorflow/issues/511#issuecomment-229971665","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/511","id":229971665,"user":{"login":"LeavesBreathe","id":14891677,"avatar_url":"https://avatars.githubusercontent.com/u/14891677?v=3","gravatar_id":"","url":"https://api.github.com/users/LeavesBreathe","html_url":"https://github.com/LeavesBreathe","followers_url":"https://api.github.com/users/LeavesBreathe/followers","following_url":"https://api.github.com/users/LeavesBreathe/following{/other_user}","gists_url":"https://api.github.com/users/LeavesBreathe/gists{/gist_id}","starred_url":"https://api.github.com/users/LeavesBreathe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LeavesBreathe/subscriptions","organizations_url":"https://api.github.com/users/LeavesBreathe/orgs","repos_url":"https://api.github.com/users/LeavesBreathe/repos","events_url":"https://api.github.com/users/LeavesBreathe/events{/privacy}","received_events_url":"https://api.github.com/users/LeavesBreathe/received_events","type":"User","site_admin":false},"created_at":"2016-07-01T15:08:43Z","updated_at":"2016-07-01T15:08:43Z","body":"@wchan  thanks for your response -- Also big thanks for your attempts to make the LSTM faster. Really looking forward to that!\r\n\r\n@jakob-grabner  thanks for getting back but I'm pretty sure as @wchan stated its slower on comp time."}}
{"actor":{"display_login":"LeavesBreathe"}}