{"action":"created","issue":{"url":"https://api.github.com/repos/GPflow/GPflow/issues/181","repository_url":"https://api.github.com/repos/GPflow/GPflow","labels_url":"https://api.github.com/repos/GPflow/GPflow/issues/181/labels{/name}","comments_url":"https://api.github.com/repos/GPflow/GPflow/issues/181/comments","events_url":"https://api.github.com/repos/GPflow/GPflow/issues/181/events","html_url":"https://github.com/GPflow/GPflow/issues/181","id":173174666,"number":181,"title":"GPflow has become considerably slower on GPU with some recent changes.","user":{"login":"alexggmatthews","id":6596998,"avatar_url":"https://avatars.githubusercontent.com/u/6596998?v=3","gravatar_id":"","url":"https://api.github.com/users/alexggmatthews","html_url":"https://github.com/alexggmatthews","followers_url":"https://api.github.com/users/alexggmatthews/followers","following_url":"https://api.github.com/users/alexggmatthews/following{/other_user}","gists_url":"https://api.github.com/users/alexggmatthews/gists{/gist_id}","starred_url":"https://api.github.com/users/alexggmatthews/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexggmatthews/subscriptions","organizations_url":"https://api.github.com/users/alexggmatthews/orgs","repos_url":"https://api.github.com/users/alexggmatthews/repos","events_url":"https://api.github.com/users/alexggmatthews/events{/privacy}","received_events_url":"https://api.github.com/users/alexggmatthews/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GPflow/GPflow/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"jameshensman","id":154383,"avatar_url":"https://avatars.githubusercontent.com/u/154383?v=3","gravatar_id":"","url":"https://api.github.com/users/jameshensman","html_url":"https://github.com/jameshensman","followers_url":"https://api.github.com/users/jameshensman/followers","following_url":"https://api.github.com/users/jameshensman/following{/other_user}","gists_url":"https://api.github.com/users/jameshensman/gists{/gist_id}","starred_url":"https://api.github.com/users/jameshensman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jameshensman/subscriptions","organizations_url":"https://api.github.com/users/jameshensman/orgs","repos_url":"https://api.github.com/users/jameshensman/repos","events_url":"https://api.github.com/users/jameshensman/events{/privacy}","received_events_url":"https://api.github.com/users/jameshensman/received_events","type":"User","site_admin":false},"assignees":[{"login":"jameshensman","id":154383,"avatar_url":"https://avatars.githubusercontent.com/u/154383?v=3","gravatar_id":"","url":"https://api.github.com/users/jameshensman","html_url":"https://github.com/jameshensman","followers_url":"https://api.github.com/users/jameshensman/followers","following_url":"https://api.github.com/users/jameshensman/following{/other_user}","gists_url":"https://api.github.com/users/jameshensman/gists{/gist_id}","starred_url":"https://api.github.com/users/jameshensman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jameshensman/subscriptions","organizations_url":"https://api.github.com/users/jameshensman/orgs","repos_url":"https://api.github.com/users/jameshensman/repos","events_url":"https://api.github.com/users/jameshensman/events{/privacy}","received_events_url":"https://api.github.com/users/jameshensman/received_events","type":"User","site_admin":false}],"milestone":null,"comments":18,"created_at":"2016-08-25T11:05:18Z","updated_at":"2016-08-25T16:15:53Z","closed_at":null,"body":"I have raised this as a bug because as you will see the effect is large.\r\n\r\nThe main changes I can see are:\r\n\r\n1) The addition of user_ops for VecToTri and TriToVec\r\n2) The use of batch matrix triangular solve.\r\n\r\nThe git hashes I'm comparing are:\r\n\r\nnew with changes 23b47f590806f0da3cb71916d68674f8bee4d875\r\nold without changes 9c2faf62d04f8ec2ab35396553b5115d92d3c84b\r\n\r\nI'm running MNIST training with a minibatch size of 1000.\r\n\r\nGPU operation is 70%  slower between these two hashes.\r\nCPU operation is about the same. Did we observe a speed up when we merged these?\r\n\r\nThe new profiling code #174 allows us to give a detailed breakdown of the slow down. \r\n[user_ops_comparison.tar.gz](https://github.com/GPflow/GPflow/files/436776/user_ops_comparison.tar.gz)\r\n\r\nThe can be viewed in chrome using the GUI found at chrome://tracing .\r\n\r\nWe find that the batch matrix triangular solve does not parallelize as well as the triangular_solve which is arguably a flaw in TensorFlow itself.\r\n\r\nWe find that the VecToTri and TriToVec ops are currently very slow relatively considering they just reshape something.\r\n\r\n**What should we do about this?**\r\n\r\nI would argue we should find out exactly which recent changes are causing this and revert them as soon as possible. The new code looks really great and requires considerably skill but this is showing us that it isn't ready for master yet IMHO. The new user ops also make installation more difficult. We want our users at any moment to have the best available experience and we have to be dedicated to this if we are going to achieve our ambitions for the software. \r\n\r\nLonger term I argue that we should move the code to pull requests and expedite #174 #171 and #170\r\n\r\n@markvdw @jameshensman thoughts? \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/GPflow/GPflow/issues/comments/242447456","html_url":"https://github.com/GPflow/GPflow/issues/181#issuecomment-242447456","issue_url":"https://api.github.com/repos/GPflow/GPflow/issues/181","id":242447456,"user":{"login":"alexggmatthews","id":6596998,"avatar_url":"https://avatars.githubusercontent.com/u/6596998?v=3","gravatar_id":"","url":"https://api.github.com/users/alexggmatthews","html_url":"https://github.com/alexggmatthews","followers_url":"https://api.github.com/users/alexggmatthews/followers","following_url":"https://api.github.com/users/alexggmatthews/following{/other_user}","gists_url":"https://api.github.com/users/alexggmatthews/gists{/gist_id}","starred_url":"https://api.github.com/users/alexggmatthews/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexggmatthews/subscriptions","organizations_url":"https://api.github.com/users/alexggmatthews/orgs","repos_url":"https://api.github.com/users/alexggmatthews/repos","events_url":"https://api.github.com/users/alexggmatthews/events{/privacy}","received_events_url":"https://api.github.com/users/alexggmatthews/received_events","type":"User","site_admin":false},"created_at":"2016-08-25T16:15:53Z","updated_at":"2016-08-25T16:15:53Z","body":"@jameshensman a revert of #159 together with #182 restores the speed I saw in my original experiments. \r\n\r\nI understand the code should run better using the batch ops. This seems like a flaw with TensorFlow or something subtle with how we are using it.\r\n\r\nThe bottom line is the GPU code for this experiment on master is currently 45% slower than it was without the batch ops.\r\n\r\nCode, which is not to be merged, here https://github.com/GPflow/GPflow/tree/master_no_batch\r\n\r\nAs in the previous case I vote revert then re-introduce as a PR whilst we work out what is going wrong. What are your thoughts? "}}
{"actor":{"display_login":"alexggmatthews"}}