{"action":"created","issue":{"url":"https://api.github.com/repos/HIPERFIT/futhark/issues/152","repository_url":"https://api.github.com/repos/HIPERFIT/futhark","labels_url":"https://api.github.com/repos/HIPERFIT/futhark/issues/152/labels{/name}","comments_url":"https://api.github.com/repos/HIPERFIT/futhark/issues/152/comments","events_url":"https://api.github.com/repos/HIPERFIT/futhark/issues/152/events","html_url":"https://github.com/HIPERFIT/futhark/issues/152","id":151061529,"number":152,"title":"Fix lexer performance","user":{"login":"Athas","id":55833,"avatar_url":"https://avatars.githubusercontent.com/u/55833?v=3","gravatar_id":"","url":"https://api.github.com/users/Athas","html_url":"https://github.com/Athas","followers_url":"https://api.github.com/users/Athas/followers","following_url":"https://api.github.com/users/Athas/following{/other_user}","gists_url":"https://api.github.com/users/Athas/gists{/gist_id}","starred_url":"https://api.github.com/users/Athas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Athas/subscriptions","organizations_url":"https://api.github.com/users/Athas/orgs","repos_url":"https://api.github.com/users/Athas/repos","events_url":"https://api.github.com/users/Athas/events{/privacy}","received_events_url":"https://api.github.com/users/Athas/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/HIPERFIT/futhark/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/HIPERFIT/futhark/labels/newbie-friendly","name":"newbie-friendly","color":"bfdadc"}],"state":"open","locked":false,"assignee":{"login":"nqpz","id":1893523,"avatar_url":"https://avatars.githubusercontent.com/u/1893523?v=3","gravatar_id":"","url":"https://api.github.com/users/nqpz","html_url":"https://github.com/nqpz","followers_url":"https://api.github.com/users/nqpz/followers","following_url":"https://api.github.com/users/nqpz/following{/other_user}","gists_url":"https://api.github.com/users/nqpz/gists{/gist_id}","starred_url":"https://api.github.com/users/nqpz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nqpz/subscriptions","organizations_url":"https://api.github.com/users/nqpz/orgs","repos_url":"https://api.github.com/users/nqpz/repos","events_url":"https://api.github.com/users/nqpz/events{/privacy}","received_events_url":"https://api.github.com/users/nqpz/received_events","type":"User","site_admin":false},"assignees":[{"login":"nqpz","id":1893523,"avatar_url":"https://avatars.githubusercontent.com/u/1893523?v=3","gravatar_id":"","url":"https://api.github.com/users/nqpz","html_url":"https://github.com/nqpz","followers_url":"https://api.github.com/users/nqpz/followers","following_url":"https://api.github.com/users/nqpz/following{/other_user}","gists_url":"https://api.github.com/users/nqpz/gists{/gist_id}","starred_url":"https://api.github.com/users/nqpz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nqpz/subscriptions","organizations_url":"https://api.github.com/users/nqpz/orgs","repos_url":"https://api.github.com/users/nqpz/repos","events_url":"https://api.github.com/users/nqpz/events{/privacy}","received_events_url":"https://api.github.com/users/nqpz/received_events","type":"User","site_admin":false}],"milestone":null,"comments":3,"created_at":"2016-04-26T07:20:26Z","updated_at":"2016-08-01T12:32:32Z","closed_at":null,"body":"It turns out that the reason `futhark-test` has problems with large datasets (like the 50MiB kdd_cup.in file for kmeans) is that the *lexer* of all things consumes too much memory.  There is one spot where we convert a `Text` to a `String` in order to call `read` that appears to be the major cost centre, but I'm not sure why.  I've been looking for a space leak, but I can't find any..."},"comment":{"url":"https://api.github.com/repos/HIPERFIT/futhark/issues/comments/236567979","html_url":"https://github.com/HIPERFIT/futhark/issues/152#issuecomment-236567979","issue_url":"https://api.github.com/repos/HIPERFIT/futhark/issues/152","id":236567979,"user":{"login":"Athas","id":55833,"avatar_url":"https://avatars.githubusercontent.com/u/55833?v=3","gravatar_id":"","url":"https://api.github.com/users/Athas","html_url":"https://github.com/Athas","followers_url":"https://api.github.com/users/Athas/followers","following_url":"https://api.github.com/users/Athas/following{/other_user}","gists_url":"https://api.github.com/users/Athas/gists{/gist_id}","starred_url":"https://api.github.com/users/Athas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Athas/subscriptions","organizations_url":"https://api.github.com/users/Athas/orgs","repos_url":"https://api.github.com/users/Athas/repos","events_url":"https://api.github.com/users/Athas/events{/privacy}","received_events_url":"https://api.github.com/users/Athas/received_events","type":"User","site_admin":false},"created_at":"2016-08-01T12:32:32Z","updated_at":"2016-08-01T12:32:32Z","body":"I'll let @nqpz answer for himself, but you should be aware that the kind\nof parser we need is not the one you find in a Haskell tutorial.  We do\nnot need a new parser for the full language, but just a parser that can\nread (a subset of) the Futhark value syntax (no tuples).  This parser\nwill need to handle quite large inputs (dozens of MiB) and produce\nfairly large arrays.  I think the most reasonable way to do this is to\nwrite an imperative parser using the ST monad and an STArray (which is\nthen turned into a Futhark array at the end).  I doubt any parser\ncombinator library could deal with values of the size we need.  It will\nlook more like the parser we use in generated C code, which is easily\nfast enough:\nhttps://github.com/HIPERFIT/futhark/blob/master/rts/c/reader.h\n\nI'm not saying it's necessarily hard, I'm just saying that it's not just\n\"Haskell\", it's \"high-performance Haskell\", which is a different style\nthan you might have seen before.\n\nWe did however implement loop tiling, yes.  It's still a fairly rushed\nimplementation (paper deadline today!), probably brittle, and doesn't\nyet give as much speedup as we would like.  It is, however, a start, and\nwe have a decent idea of where to go from here.\n\n-- \n\\  Troels\n/\\ Henriksen\n"}}
{"actor":{"display_login":"Athas"}}