{"action":"created","issue":{"url":"https://api.github.com/repos/nexB/scancode-toolkit/issues/369","repository_url":"https://api.github.com/repos/nexB/scancode-toolkit","labels_url":"https://api.github.com/repos/nexB/scancode-toolkit/issues/369/labels{/name}","comments_url":"https://api.github.com/repos/nexB/scancode-toolkit/issues/369/comments","events_url":"https://api.github.com/repos/nexB/scancode-toolkit/issues/369/events","html_url":"https://github.com/nexB/scancode-toolkit/pull/369","id":189221309,"number":369,"title":"Long scan timeout for #267","user":{"login":"pombredanne","id":675997,"avatar_url":"https://avatars.githubusercontent.com/u/675997?v=3","gravatar_id":"","url":"https://api.github.com/users/pombredanne","html_url":"https://github.com/pombredanne","followers_url":"https://api.github.com/users/pombredanne/followers","following_url":"https://api.github.com/users/pombredanne/following{/other_user}","gists_url":"https://api.github.com/users/pombredanne/gists{/gist_id}","starred_url":"https://api.github.com/users/pombredanne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pombredanne/subscriptions","organizations_url":"https://api.github.com/users/pombredanne/orgs","repos_url":"https://api.github.com/users/pombredanne/repos","events_url":"https://api.github.com/users/pombredanne/events{/privacy}","received_events_url":"https://api.github.com/users/pombredanne/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-11-14T20:45:42Z","updated_at":"2016-11-15T16:34:00Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/nexB/scancode-toolkit/pulls/369","html_url":"https://github.com/nexB/scancode-toolkit/pull/369","diff_url":"https://github.com/nexB/scancode-toolkit/pull/369.diff","patch_url":"https://github.com/nexB/scancode-toolkit/pull/369.patch"},"body":"In #267 the key issue is some pathological, hard to diagnose scan that takes a long time and eventually uses a lot of RAM to get prockilled  (at least on Linux).\r\n\r\nThis should both help handle the issue and help diagnose it:\r\n\r\n* a scan will be interrupted if it takes too long and exceeds a timeout: so if there is a problem somewhere, it should timeout rather than suck all the memory and get proc killed. The approach is to run each scan in a thread with a timeout using code from @dano\r\nThe timeout is computed for each file being scanned. It has a minimum of 60 seconds and a maximum of 600 seconds.  Each file bigger than 1MB gets an extra 30 of timeout for each extra MB up to the max  600s.\r\n\r\n* a scan can run on multiple processes with the `-n or --processes` option (this is a nice side effect of handling timeouts and can also speed up scanning of course, but would use more RAM). When running on multiple process the memory used by each process may vary but would be about 600MB + size of the scanned file in memory + some extras.\r\n\r\n* -`-verbose` output is always send to stdout, as you requested by @sschuberth which will help to diagnose issues for background, CI-like scan jobs. So running a --verbose scan on this would likely tell exactly which is the file that is causing an error.\r\n\r\n* a summary of the scan statistics is displayed at the end of a scan: this helps diagnose errors as it does list all the paths that failed to scan alright or did timeout. It also provide some timing and scanning speed details.\r\n"},"comment":{"url":"https://api.github.com/repos/nexB/scancode-toolkit/issues/comments/260692357","html_url":"https://github.com/nexB/scancode-toolkit/pull/369#issuecomment-260692357","issue_url":"https://api.github.com/repos/nexB/scancode-toolkit/issues/369","id":260692357,"user":{"login":"pombredanne","id":675997,"avatar_url":"https://avatars.githubusercontent.com/u/675997?v=3","gravatar_id":"","url":"https://api.github.com/users/pombredanne","html_url":"https://github.com/pombredanne","followers_url":"https://api.github.com/users/pombredanne/followers","following_url":"https://api.github.com/users/pombredanne/following{/other_user}","gists_url":"https://api.github.com/users/pombredanne/gists{/gist_id}","starred_url":"https://api.github.com/users/pombredanne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pombredanne/subscriptions","organizations_url":"https://api.github.com/users/pombredanne/orgs","repos_url":"https://api.github.com/users/pombredanne/repos","events_url":"https://api.github.com/users/pombredanne/events{/privacy}","received_events_url":"https://api.github.com/users/pombredanne/received_events","type":"User","site_admin":false},"created_at":"2016-11-15T16:34:00Z","updated_at":"2016-11-15T16:34:00Z","body":"@sschuberth I fixed the typos and a few more. Thanks ++ for this review! I need to fix the CI failures now."}}
{"actor":{"display_login":"pombredanne"}}