{"action":"created","issue":{"url":"https://api.github.com/repos/Arachni/arachni/issues/697","repository_url":"https://api.github.com/repos/Arachni/arachni","labels_url":"https://api.github.com/repos/Arachni/arachni/issues/697/labels{/name}","comments_url":"https://api.github.com/repos/Arachni/arachni/issues/697/comments","events_url":"https://api.github.com/repos/Arachni/arachni/issues/697/events","html_url":"https://github.com/Arachni/arachni/issues/697","id":148358484,"number":697,"title":"Overhaul the multi-process scan architecture","user":{"login":"Zapotek","id":375743,"avatar_url":"https://avatars.githubusercontent.com/u/375743?v=3","gravatar_id":"","url":"https://api.github.com/users/Zapotek","html_url":"https://github.com/Zapotek","followers_url":"https://api.github.com/users/Zapotek/followers","following_url":"https://api.github.com/users/Zapotek/following{/other_user}","gists_url":"https://api.github.com/users/Zapotek/gists{/gist_id}","starred_url":"https://api.github.com/users/Zapotek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zapotek/subscriptions","organizations_url":"https://api.github.com/users/Zapotek/orgs","repos_url":"https://api.github.com/users/Zapotek/repos","events_url":"https://api.github.com/users/Zapotek/events{/privacy}","received_events_url":"https://api.github.com/users/Zapotek/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Arachni/arachni/labels/Feature%20request","name":"Feature request","color":"444444"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/Arachni/arachni/milestones/14","html_url":"https://github.com/Arachni/arachni/milestones/1.6","labels_url":"https://api.github.com/repos/Arachni/arachni/milestones/14/labels","id":1700970,"number":14,"title":"1.6","description":"","creator":{"login":"Zapotek","id":375743,"avatar_url":"https://avatars.githubusercontent.com/u/375743?v=3","gravatar_id":"","url":"https://api.github.com/users/Zapotek","html_url":"https://github.com/Zapotek","followers_url":"https://api.github.com/users/Zapotek/followers","following_url":"https://api.github.com/users/Zapotek/following{/other_user}","gists_url":"https://api.github.com/users/Zapotek/gists{/gist_id}","starred_url":"https://api.github.com/users/Zapotek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zapotek/subscriptions","organizations_url":"https://api.github.com/users/Zapotek/orgs","repos_url":"https://api.github.com/users/Zapotek/repos","events_url":"https://api.github.com/users/Zapotek/events{/privacy}","received_events_url":"https://api.github.com/users/Zapotek/received_events","type":"User","site_admin":false},"open_issues":4,"closed_issues":0,"state":"open","created_at":"2016-04-12T10:42:52Z","updated_at":"2016-05-04T08:45:19Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2016-04-14T13:16:33Z","updated_at":"2016-08-07T14:10:57Z","closed_at":null,"body":"Remove the current multi-process scan code (that nobody uses anyway) and replace it with generic, all-in-one worker processes.\r\n\r\n## Architecture\r\n\r\nArchitecture should be similar to the `BrowserCluster` but with processes instead of threads.\r\n\r\n* [ ] Use `method(:my_handler)` callbacks rather than `proc{}`s to help out the GC.\r\n  * `proc` closures retain their env and we'll need to store a lot of callbacks.\r\n* [ ] Take advantage of copy-on-write so preload as much data as possible prior to forking.\r\n* [ ] Use `Arachni::RPC` for communication.\r\n  * Use UNIX sockets when available, otherwise TCP/IP.\r\n  * Disable SSL.\r\n  * Disable compression.\r\n* [ ] Should auto-scale by using #695.\r\n* [ ] Maybe have Dispatchers expose workers.\r\n  * This will allow multiple machines to share one scan's workload when setup in Grid mode.\r\n  * Similar to the existing multi-process system but much more efficient.\r\n\r\n## Responsibilities\r\n\r\nThe workers should perform actions like:\r\n\r\n* [ ] HTML/XML parsing.\r\n  * Can cause 100% usage when parsing very large documents, thus blocking the scan.\r\n  * The `Trainer` will massively benefit from this, since it does a lot of parsing during page audits.\r\n  * Should also perform the subsequent handling of the parsed document and send back the result instead of sending back the parsed document, otherwise there's no point to it.\r\n* [ ] `Arachni::Support::Signature` processing.\r\n  * Signature generation, refinement and matching can cause 100% CPU usage when dealing with very large data sets, thus blocking the scan.\r\n* [ ] Manage browser processes.\r\n  * The system is already launching Ruby life-line processes to ensure that PhantomJS processes don't zombie-out if the parent process disappears for whatever reason.\r\n  * Since we're gonna have the workers, let them deal with that as well to keep the amount of overall processes to a minimum.\r\n"},"comment":{"url":"https://api.github.com/repos/Arachni/arachni/issues/comments/238084612","html_url":"https://github.com/Arachni/arachni/issues/697#issuecomment-238084612","issue_url":"https://api.github.com/repos/Arachni/arachni/issues/697","id":238084612,"user":{"login":"Zapotek","id":375743,"avatar_url":"https://avatars.githubusercontent.com/u/375743?v=3","gravatar_id":"","url":"https://api.github.com/users/Zapotek","html_url":"https://github.com/Zapotek","followers_url":"https://api.github.com/users/Zapotek/followers","following_url":"https://api.github.com/users/Zapotek/following{/other_user}","gists_url":"https://api.github.com/users/Zapotek/gists{/gist_id}","starred_url":"https://api.github.com/users/Zapotek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zapotek/subscriptions","organizations_url":"https://api.github.com/users/Zapotek/orgs","repos_url":"https://api.github.com/users/Zapotek/repos","events_url":"https://api.github.com/users/Zapotek/events{/privacy}","received_events_url":"https://api.github.com/users/Zapotek/received_events","type":"User","site_admin":false},"created_at":"2016-08-07T14:10:57Z","updated_at":"2016-08-07T14:10:57Z","body":"Another, less time consuming, approach, would be to re-purpose the `multi` code and change it so that:\r\n\r\n* The _master_ instance will only:\r\n  1. Crawl.\r\n  2. Distribute the workload (i.e. discovered pages, deduplicate elements etc.).\r\n  3. Store issues and other report data as reported by _slave_ instances.\r\n* The _slave_  instances will:\r\n  1. Be completely dispensable -- not used to persist any data of any value.\r\n  2. Have a TTL counted in page audits.\r\n    1. This will put an end to memory leak issues once and for all.\r\n    2. Also take care of non-leak high RAM usage when processing large pages, since memory will never grow too large due to the GC not free-ing as an optimization.\r\n\r\nWorst case scenario, have a separate and dispensable TTL'd _crawler_ instance, because DOM crawls can have quite a RAM impact just by themselves."}}
{"actor":{"display_login":"Zapotek"}}