{"action":"created","issue":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/206","repository_url":"https://api.github.com/repos/cgiffard/node-simplecrawler","labels_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/206/labels{/name}","comments_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/206/comments","events_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/206/events","html_url":"https://github.com/cgiffard/node-simplecrawler/issues/206","id":127996014,"number":206,"title":"Calling crawler.stop() does not stop the crawling process ","user":{"login":"andreivisan","id":1696589,"avatar_url":"https://avatars.githubusercontent.com/u/1696589?v=3","gravatar_id":"","url":"https://api.github.com/users/andreivisan","html_url":"https://github.com/andreivisan","followers_url":"https://api.github.com/users/andreivisan/followers","following_url":"https://api.github.com/users/andreivisan/following{/other_user}","gists_url":"https://api.github.com/users/andreivisan/gists{/gist_id}","starred_url":"https://api.github.com/users/andreivisan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andreivisan/subscriptions","organizations_url":"https://api.github.com/users/andreivisan/orgs","repos_url":"https://api.github.com/users/andreivisan/repos","events_url":"https://api.github.com/users/andreivisan/events{/privacy}","received_events_url":"https://api.github.com/users/andreivisan/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/labels/New%20Feature","name":"New Feature","color":"02e10c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2016-01-21T19:01:18Z","updated_at":"2016-02-18T12:08:34Z","closed_at":null,"body":"I would like to be able to stop the crawler manually. I tried by calling crawler.stop() but nothing happens. Am I doing it wrong?"},"comment":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/comments/185688572","html_url":"https://github.com/cgiffard/node-simplecrawler/issues/206#issuecomment-185688572","issue_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/206","id":185688572,"user":{"login":"fredrikekelund","id":1101677,"avatar_url":"https://avatars.githubusercontent.com/u/1101677?v=3","gravatar_id":"","url":"https://api.github.com/users/fredrikekelund","html_url":"https://github.com/fredrikekelund","followers_url":"https://api.github.com/users/fredrikekelund/followers","following_url":"https://api.github.com/users/fredrikekelund/following{/other_user}","gists_url":"https://api.github.com/users/fredrikekelund/gists{/gist_id}","starred_url":"https://api.github.com/users/fredrikekelund/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fredrikekelund/subscriptions","organizations_url":"https://api.github.com/users/fredrikekelund/orgs","repos_url":"https://api.github.com/users/fredrikekelund/repos","events_url":"https://api.github.com/users/fredrikekelund/events{/privacy}","received_events_url":"https://api.github.com/users/fredrikekelund/received_events","type":"User","site_admin":false},"created_at":"2016-02-18T12:08:34Z","updated_at":"2016-02-18T12:08:34Z","body":"@tmpfs a \"crawlstop\" event sounds like a good idea, makes a lot of sense to me!\r\n\r\nI'm not sure I find it beyond the scope of simplecrawler to stop requests mid-flight. I understand the need for it, and it should be fairly simple to implement. I do however think that there might be a need for the current behavior as well: the developer might want the open requests to be completed before exiting the process or whatever.\r\n\r\nSo a suggestion would be to let the developer pass a single boolean parameter to `crawler.stop` that determines whether to abort all requests at once or not (default would be not to). Moreover, we would add a \"crawlstop\" event that would fire when all requests have ended, so either instantaneously if I pass `true` to `crawler.stop`, or after a little while if I pass nothing or a faulty value.\r\n\r\nThoughts?"}}