{"action":"created","issue":{"url":"https://api.github.com/repos/HIPERFIT/futhark-benchmarks/issues/4","repository_url":"https://api.github.com/repos/HIPERFIT/futhark-benchmarks","labels_url":"https://api.github.com/repos/HIPERFIT/futhark-benchmarks/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/HIPERFIT/futhark-benchmarks/issues/4/comments","events_url":"https://api.github.com/repos/HIPERFIT/futhark-benchmarks/issues/4/events","html_url":"https://github.com/HIPERFIT/futhark-benchmarks/issues/4","id":153668701,"number":4,"title":"Matrix multiply benchmark request","user":{"login":"mrakgr","id":6266635,"avatar_url":"https://avatars.githubusercontent.com/u/6266635?v=3","gravatar_id":"","url":"https://api.github.com/users/mrakgr","html_url":"https://github.com/mrakgr","followers_url":"https://api.github.com/users/mrakgr/followers","following_url":"https://api.github.com/users/mrakgr/following{/other_user}","gists_url":"https://api.github.com/users/mrakgr/gists{/gist_id}","starred_url":"https://api.github.com/users/mrakgr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrakgr/subscriptions","organizations_url":"https://api.github.com/users/mrakgr/orgs","repos_url":"https://api.github.com/users/mrakgr/repos","events_url":"https://api.github.com/users/mrakgr/events{/privacy}","received_events_url":"https://api.github.com/users/mrakgr/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-05-08T17:55:35Z","updated_at":"2016-05-10T13:29:40Z","closed_at":null,"body":"Besides the #3 , I have a question on matrix multiply. What is its performance compared to the hardcoded library functions? If it is something like a third, then it might be possible to speed it up by more than double.\r\n\r\nI had wanted to suggest a few days ago, that the most ideal solution to make Futhark great for machine learning would be to have an assembly optimized matrix multiply as a part of its core library. Really, as I mulled it over, Futhark does not need a FFI, but only this single piece and maybe a fast convolution to meet this goal, but when I did some research I realized that unlike in standard C, OpenCL does not allow complete access to hardware. More specifically, Cuda devices compile to an intermediate language PTX and there is no way to go lower than that. And going lower to SASS would be necessary to [draw out the 100%](https://github.com/NervanaSystems/maxas/wiki/SGEMM)...though reading that link I doubt Futhark doing optimizations at the assembly level would ever be feasible even with access to SASS. What I wanted to suggest was just a fancy idea.\r\n\r\nI think doing global optimizations for RNN architectures could go a long way on its own.\r\n\r\nFor example, two of the big symbolic differentiation machine learning libraries Theano and Tensorflow, are in fact doing code generation internally and large parts of them could be superseded by Futhark's functionality. Theano in particular is infamous for having ridiculous compile times for recurrent nets.\r\n\r\nFor the sake of potential future machine learning endeavors, it might be of interest to track the matrix multiply performance over time. Furthermore, for neural nets it would also be of interest not just how fast Futhark does `Ax`, but also `Ax+b` and `o(Ax+b)` where `o()` is a non-linear activation function and `+b` is a broadcast addition.\r\n\r\nFurthermore recurrent nets also have layers of the form `o(Ax+Bh+b)`.\r\n\r\nLSTM cells in particular have gates that all receive similar inputs and previous states like so:\r\n`block_input=o(A_1x+B_1h+b_1)`\r\n`input_gate=o(A_2x+B_2h+b_2)`\r\n`forget_gate=o(A_3x+B_3h+b_3)`\r\n`output_gate=o(A_4x+B_4h+b_4)`\r\n\r\nA common optimization is to concatenate the `1..4` matrices above into one and do the matrix multiplies as one operation. It would be interesting to know whether Futhark does that optimization automatically.\r\n\r\nAs an aside, the above actually made me throw the towel for my own machine learning library, because I realized that writing things like concatenation kernels, splitting kernels, fusing them manually with broadcast addition and non-linear activation was something that would be far too much of a bother. It is not that I could not do it, it is just that it would take too much time. Let the paid Google engineers do those things in Tensorflow or alternatively, Futhark could do it. Futhark could do even more and actually fuse all the matrix multiplies, bias additions and activations into a single function.\r\n\r\nIf you are interested, I could do a LSTM cell forward pass as a Futhark benchmark, though I am not familiar enough with OpenCL do get a matrix multiply right now.\r\n\r\nWould that be fine? \r\n\r\n...Also, the first question was whether a simple matrix multiply benchmark had been done. :)"},"comment":{"url":"https://api.github.com/repos/HIPERFIT/futhark-benchmarks/issues/comments/218157198","html_url":"https://github.com/HIPERFIT/futhark-benchmarks/issues/4#issuecomment-218157198","issue_url":"https://api.github.com/repos/HIPERFIT/futhark-benchmarks/issues/4","id":218157198,"user":{"login":"mrakgr","id":6266635,"avatar_url":"https://avatars.githubusercontent.com/u/6266635?v=3","gravatar_id":"","url":"https://api.github.com/users/mrakgr","html_url":"https://github.com/mrakgr","followers_url":"https://api.github.com/users/mrakgr/followers","following_url":"https://api.github.com/users/mrakgr/following{/other_user}","gists_url":"https://api.github.com/users/mrakgr/gists{/gist_id}","starred_url":"https://api.github.com/users/mrakgr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrakgr/subscriptions","organizations_url":"https://api.github.com/users/mrakgr/orgs","repos_url":"https://api.github.com/users/mrakgr/repos","events_url":"https://api.github.com/users/mrakgr/events{/privacy}","received_events_url":"https://api.github.com/users/mrakgr/received_events","type":"User","site_admin":false},"created_at":"2016-05-10T13:29:40Z","updated_at":"2016-05-10T13:29:40Z","body":"You are right `[[f32,n],n] B, [[f32,m],n] h` is pretty much the same thing that dumbfounded me yesterday and I see I got it wrong. Damn.\r\n\r\nI have no idea what is going on in the above. Basically `matmult(B,h`) has to have the same dimensions as `matmult(A,x)`. B has to be a square matrix and h itself also has to have the same dimensions as `matmult(A,x)` as it is the output from the previous timestep.\r\n\r\nSo all the matrix multiplications have to be of `[[f32,m],n]`.\r\n\r\nIf I make B of type `[[f32,m],m]` which would align the first dimension correctly, then h can only be of `[[f32,n],m]` which is wrong. It is also impossible that the prev_output needs to be transposed or anything like that.\r\n\r\nIt would be nice if I could insert some print statements into the program. This is such a strange logic error on my side. I have no idea how to debug this."}}