{"action":"created","issue":{"url":"https://api.github.com/repos/ocramz/xeno/issues/18","repository_url":"https://api.github.com/repos/ocramz/xeno","labels_url":"https://api.github.com/repos/ocramz/xeno/issues/18/labels{/name}","comments_url":"https://api.github.com/repos/ocramz/xeno/issues/18/comments","events_url":"https://api.github.com/repos/ocramz/xeno/issues/18/events","html_url":"https://github.com/ocramz/xeno/issues/18","id":321862261,"node_id":"MDU6SXNzdWUzMjE4NjIyNjE=","number":18,"title":"Use on files too large to fit in memory","user":{"login":"unhammer","id":56868,"node_id":"MDQ6VXNlcjU2ODY4","avatar_url":"https://avatars1.githubusercontent.com/u/56868?v=4","gravatar_id":"","url":"https://api.github.com/users/unhammer","html_url":"https://github.com/unhammer","followers_url":"https://api.github.com/users/unhammer/followers","following_url":"https://api.github.com/users/unhammer/following{/other_user}","gists_url":"https://api.github.com/users/unhammer/gists{/gist_id}","starred_url":"https://api.github.com/users/unhammer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/unhammer/subscriptions","organizations_url":"https://api.github.com/users/unhammer/orgs","repos_url":"https://api.github.com/users/unhammer/repos","events_url":"https://api.github.com/users/unhammer/events{/privacy}","received_events_url":"https://api.github.com/users/unhammer/received_events","type":"User","site_admin":false},"labels":[{"id":505482141,"node_id":"MDU6TGFiZWw1MDU0ODIxNDE=","url":"https://api.github.com/repos/ocramz/xeno/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2018-05-10T09:10:25Z","updated_at":"2018-07-06T14:36:01Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"From what I can tell from reading the (short!) source, there doesn't seem to be a way to use this on files too large to fit in memory. In particular, the ByteString's are strict, and if one simply tries splitting and sending in the parts, there's a possibility of getting errors like\r\n\r\n`Left (XenoParseError \"Couldn't find the matching quote character.\")`\r\n\r\nI'd rather not have Lazy ByteStrings because [reasons](https://stackoverflow.com/questions/5892653/whats-so-bad-about-lazy-i-o), though that would probably be the simplest solution.\r\n\r\nAn alternative would be to add an `eofF :: (Int -> m ())` to `process`, and pass it the index of the last fully processed element, and then call that wherever we now call `throw`, which would allow simply sending in arbitrarily sized ByteString's and stitch them using the eoF function. But most of the throw's happen in `s_index` – making that return a Maybe seems like it would hurt performance. \r\n\r\nOr is it possible to make `indexEndOfLastMatch` an argument of `XenoException`'s, and `catch` the throw's without losing the monad return value?"},"comment":{"url":"https://api.github.com/repos/ocramz/xeno/issues/comments/403052861","html_url":"https://github.com/ocramz/xeno/issues/18#issuecomment-403052861","issue_url":"https://api.github.com/repos/ocramz/xeno/issues/18","id":403052861,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMzA1Mjg2MQ==","user":{"login":"chrisdone","id":11019,"node_id":"MDQ6VXNlcjExMDE5","avatar_url":"https://avatars3.githubusercontent.com/u/11019?v=4","gravatar_id":"","url":"https://api.github.com/users/chrisdone","html_url":"https://github.com/chrisdone","followers_url":"https://api.github.com/users/chrisdone/followers","following_url":"https://api.github.com/users/chrisdone/following{/other_user}","gists_url":"https://api.github.com/users/chrisdone/gists{/gist_id}","starred_url":"https://api.github.com/users/chrisdone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrisdone/subscriptions","organizations_url":"https://api.github.com/users/chrisdone/orgs","repos_url":"https://api.github.com/users/chrisdone/repos","events_url":"https://api.github.com/users/chrisdone/events{/privacy}","received_events_url":"https://api.github.com/users/chrisdone/received_events","type":"User","site_admin":false},"created_at":"2018-07-06T14:36:01Z","updated_at":"2018-07-06T14:36:01Z","author_association":"COLLABORATOR","body":"@ocramz I think that's already available from https://hackage.haskell.org/package/mmap-0.5.9/docs/System-IO-MMap.html#v:mmapFileByteString "}}
{"actor":{"display_login":"chrisdone"}}