{"action":"created","issue":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/214","repository_url":"https://api.github.com/repos/cgiffard/node-simplecrawler","labels_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/214/labels{/name}","comments_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/214/comments","events_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/214/events","html_url":"https://github.com/cgiffard/node-simplecrawler/issues/214","id":132851806,"number":214,"title":"Crawler hangs after several rounds of fetchdataerror","user":{"login":"tibetty","id":5327503,"avatar_url":"https://avatars.githubusercontent.com/u/5327503?v=3","gravatar_id":"","url":"https://api.github.com/users/tibetty","html_url":"https://github.com/tibetty","followers_url":"https://api.github.com/users/tibetty/followers","following_url":"https://api.github.com/users/tibetty/following{/other_user}","gists_url":"https://api.github.com/users/tibetty/gists{/gist_id}","starred_url":"https://api.github.com/users/tibetty/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tibetty/subscriptions","organizations_url":"https://api.github.com/users/tibetty/orgs","repos_url":"https://api.github.com/users/tibetty/repos","events_url":"https://api.github.com/users/tibetty/events{/privacy}","received_events_url":"https://api.github.com/users/tibetty/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-02-10T23:50:08Z","updated_at":"2016-02-12T00:52:31Z","closed_at":null,"body":"```js\r\nvar Crawler = require('simplecrawler');\r\n\r\nconst webSite = 'www.jnto.go.jp';\r\nconst protocol = 'http';\r\nconst port = 80;\r\nconst initialPath = '/eng';\r\n\r\nvar crawler = new Crawler(webSite);\r\ncrawler.initialProtocol = protocol;\r\ncrawler.initialPort = port;\r\ncrawler.initialPath = initialPath;\r\ncrawler.maxDepth = 3;\r\n\r\ncrawler\r\n.on('fetchcomplete', function(queueItem, responseBuffer, response) {\r\n    console.log('Received %s (%d bytes)', queueItem.url, responseBuffer.length);\r\n})\r\n.on('fetcherror', function(queueItem, response) {\r\n    console.log('Error #%d (%s) happened when retrieve %s', response.statusCode, response.statusMessage, queueItem.url);\r\n})\r\non('fetchdataerror', function(queueItem, response) {\r\n    console.log('Data error happened when retrieve %s', queueItem.url);\r\n})\r\n.start();\r\n```\r\n\r\nAgain, I run above code to test simplecrawler, and it got frozen after reporting several errors consecutively and was never restored.\r\n\r\nError #301 (Moved Permanently) happened when retrieve http://www.jnto.go.jp/weather/eng/\r\nData error happened when retrieve http://www.jnto.go.jp/eng/pdf/regional/tohoku/fukushima.pdf\r\nData error happened when retrieve http://www.jnto.go.jp/eng/pdf/regional/chubu/takayama.pdf\r\nData error happened when retrieve http://www.jnto.go.jp/philippines/pdf/BROCHURE%20Final_philippines_20151217.pdf\r\nError #403 (Forbidden) happened when retrieve http://www.jnto.go.jp/jpn/member_logins/members_service/\r\nData error happened when retrieve http://www.jnto.go.jp/jpn/pdf/jnto_50aniv_pamphlet.pdf\r\n\r\nIt might imply some design/implementation flaws (in our code or dependencies) worth analyzing 'coz **js-crawler** could complete crawling the website (depth=2) (but unluckily w/ wrong data for binary resources). I tried to do a little CSI work by adding exception handling but nothing was reported out, so all up to you, simplecrawler staffs!\r\n\r\nRegards,\r\ntibetty\r\n"},"comment":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/comments/183133029","html_url":"https://github.com/cgiffard/node-simplecrawler/issues/214#issuecomment-183133029","issue_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/214","id":183133029,"user":{"login":"tibetty","id":5327503,"avatar_url":"https://avatars.githubusercontent.com/u/5327503?v=3","gravatar_id":"","url":"https://api.github.com/users/tibetty","html_url":"https://github.com/tibetty","followers_url":"https://api.github.com/users/tibetty/followers","following_url":"https://api.github.com/users/tibetty/following{/other_user}","gists_url":"https://api.github.com/users/tibetty/gists{/gist_id}","starred_url":"https://api.github.com/users/tibetty/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tibetty/subscriptions","organizations_url":"https://api.github.com/users/tibetty/orgs","repos_url":"https://api.github.com/users/tibetty/repos","events_url":"https://api.github.com/users/tibetty/events{/privacy}","received_events_url":"https://api.github.com/users/tibetty/received_events","type":"User","site_admin":false},"created_at":"2016-02-12T00:52:31Z","updated_at":"2016-02-12T00:52:31Z","body":"You are right, @fredrikekelund! I intecepted 'fetchstart' last night, and the log showed that simplecrawler worked pretty well, so there might be something wrong with either upstream dependencies (http/request) or the website itself.  Kudos to you!"}}