{"action":"created","issue":{"url":"https://api.github.com/repos/plusvic/yara/issues/33","repository_url":"https://api.github.com/repos/plusvic/yara","labels_url":"https://api.github.com/repos/plusvic/yara/issues/33/labels{/name}","comments_url":"https://api.github.com/repos/plusvic/yara/issues/33/comments","events_url":"https://api.github.com/repos/plusvic/yara/issues/33/events","html_url":"https://github.com/plusvic/yara/issues/33","id":23205700,"number":33,"title":"Optimize memory usage when scanning processes and files","user":{"login":"plusvic","id":182937,"avatar_url":"https://avatars.githubusercontent.com/u/182937?v=3","gravatar_id":"","url":"https://api.github.com/users/plusvic","html_url":"https://github.com/plusvic","followers_url":"https://api.github.com/users/plusvic/followers","following_url":"https://api.github.com/users/plusvic/following{/other_user}","gists_url":"https://api.github.com/users/plusvic/gists{/gist_id}","starred_url":"https://api.github.com/users/plusvic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/plusvic/subscriptions","organizations_url":"https://api.github.com/users/plusvic/orgs","repos_url":"https://api.github.com/users/plusvic/repos","events_url":"https://api.github.com/users/plusvic/events{/privacy}","received_events_url":"https://api.github.com/users/plusvic/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/plusvic/yara/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/plusvic/yara/labels/imported","name":"imported","color":"FFFFFF"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2013-11-24T14:04:35Z","updated_at":"2016-02-22T20:21:39Z","closed_at":null,"body":"_From [zeroStei...@gmail.com](https://code.google.com/u/111030040457214474971/) on December 20, 2011 19:01:35_\n\nThis is a feature request not a bug.\r\n\r\nIt would be helpful if Yara would not load entire files into memory to scan them but step through them with an sizeable overlap to avoid false negatives. It would also be very helpful if Yara would use a similar chunking method to scan processes because files can at least be scanned in as chunks of data by the controlling process.\r\n\r\nBasically Yara's memory usage needs to have some limitation.\n\n_Original issue: http://code.google.com/p/yara-project/issues/detail?id=33_"},"comment":{"url":"https://api.github.com/repos/plusvic/yara/issues/comments/187367817","html_url":"https://github.com/plusvic/yara/issues/33#issuecomment-187367817","issue_url":"https://api.github.com/repos/plusvic/yara/issues/33","id":187367817,"user":{"login":"plusvic","id":182937,"avatar_url":"https://avatars.githubusercontent.com/u/182937?v=3","gravatar_id":"","url":"https://api.github.com/users/plusvic","html_url":"https://github.com/plusvic","followers_url":"https://api.github.com/users/plusvic/followers","following_url":"https://api.github.com/users/plusvic/following{/other_user}","gists_url":"https://api.github.com/users/plusvic/gists{/gist_id}","starred_url":"https://api.github.com/users/plusvic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/plusvic/subscriptions","organizations_url":"https://api.github.com/users/plusvic/orgs","repos_url":"https://api.github.com/users/plusvic/repos","events_url":"https://api.github.com/users/plusvic/events{/privacy}","received_events_url":"https://api.github.com/users/plusvic/received_events","type":"User","site_admin":false},"created_at":"2016-02-22T20:21:38Z","updated_at":"2016-02-22T20:21:38Z","body":"3) I didn't realize you wanted to remove the data member from\nYR_MEMORY_BLOCK. That type is used everywhere and I think that would be an\ninvasive change. Also, in order for that to work, everywhere we would\neither need to keep the context in the block as well as the read function\nor update every place a block is used to take an ITERATOR instead.\n\nYes, block->data in some modules, and in exec.c, but in all cases can be\neasily changed to use the iterator and fetch_block. I prefer keeping\neverything coherent and simple even if we have to refactor a few things.\n\n\n\n\n\nOn Mon, Feb 22, 2016 at 8:43 PM, Kyle Reed <notifications@github.com> wrote:\n\n> I started a the impl of the design\n> <https://github.com/kallanreed/yara/commit/4b2a8cb36fb8388d4615360939deafb8a8851591?diff=split>\n> and ran into some issues. Some of which you addressed above, I added some\n> comments in the commit.\n>\n> 1) re: keeping the process open while iterating - I just wanted to make\n> sure we were on the same page.\n>\n> 2) re: prefetching removes the need for error checking on the block\n> first/next interface because all of the operations that could fail happen\n> when the iterator is opened and when it's closed. Fetch could still fail.\n>\n> 3) I didn't realize you wanted to remove the data member from\n> YR_MEMORY_BLOCK. That type is used everywhere and I think that would be\n> an invasive change. Also, in order for that to work, everywhere we would\n> either need to keep the context in the block as well as the read function\n> or update every place a block is used to take an ITERATOR instead.\n>\n> I was considering a second type like YR_DEFERRED_BLOCK which is like the\n> normal block without the data field. This way the existing code that\n> expects blocks could be left alone for now.\n>\n> 4) I see what you mean about the LIST_ITERATOR_CONTEXT I didn't notice\n> that the reason I added it at all was to support the thing I'm removing\n> (yr_process_get_memory). I'll still need implementations for\n> first/next/fetch for a single block but the type can go away.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/plusvic/yara/issues/33#issuecomment-187344694>.\n>\n"}}