{"action":"created","issue":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/47","repository_url":"https://api.github.com/repos/cgiffard/node-simplecrawler","labels_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/47/labels{/name}","comments_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/47/comments","events_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/47/events","html_url":"https://github.com/cgiffard/node-simplecrawler/issues/47","id":17734322,"number":47,"title":"SimpleCrawler finds no links","user":{"login":"bjornwang","id":5180175,"avatar_url":"https://avatars.githubusercontent.com/u/5180175?v=3","gravatar_id":"","url":"https://api.github.com/users/bjornwang","html_url":"https://github.com/bjornwang","followers_url":"https://api.github.com/users/bjornwang/followers","following_url":"https://api.github.com/users/bjornwang/following{/other_user}","gists_url":"https://api.github.com/users/bjornwang/gists{/gist_id}","starred_url":"https://api.github.com/users/bjornwang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bjornwang/subscriptions","organizations_url":"https://api.github.com/users/bjornwang/orgs","repos_url":"https://api.github.com/users/bjornwang/repos","events_url":"https://api.github.com/users/bjornwang/events{/privacy}","received_events_url":"https://api.github.com/users/bjornwang/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":19,"created_at":"2013-08-07T07:54:47Z","updated_at":"2016-04-01T05:32:00Z","closed_at":"2013-09-26T05:40:05Z","body":"I have an issue crawling http://kulturradet.no.\r\n\r\nEvery now and then, the crawler doesn't move past the front. Hooking into the discoverycomplete event, I can see that no links are found when this happens. I was baffled by this behavior, as oftentimes it finds 80 links. The HTML-code itself has not changed.\r\n\r\nI spread some console.log(..) statemens in the crawler.js code, and found that the downloaded buffer seems garbled when this happens. When it works well, I get normal HTML from the page when i output the buffer. When the error hist, and no links are found, I get completely garbled output. This is both before and after the .toString(UTF8) conversion has been performed by the discover links-function.\r\n\r\nA strange thing concerning this error is that there seems to be a timeout involved. The crawler stays in it's erroneous state for some 10-15 minutes, and then when I run it again, it finds links just fine again. Very strange. Downloading the page in a browser or using curl displays the HTML-code just fine without being garbled.\r\n\r\nAny ideas on what can be causing this? What baffles me is the irratic behavior. Is there some caching involved, can a network error cause this so it's cached afterwards for example?"},"comment":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/comments/204254246","html_url":"https://github.com/cgiffard/node-simplecrawler/issues/47#issuecomment-204254246","issue_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/47","id":204254246,"user":{"login":"fredrikekelund","id":1101677,"avatar_url":"https://avatars.githubusercontent.com/u/1101677?v=3","gravatar_id":"","url":"https://api.github.com/users/fredrikekelund","html_url":"https://github.com/fredrikekelund","followers_url":"https://api.github.com/users/fredrikekelund/followers","following_url":"https://api.github.com/users/fredrikekelund/following{/other_user}","gists_url":"https://api.github.com/users/fredrikekelund/gists{/gist_id}","starred_url":"https://api.github.com/users/fredrikekelund/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fredrikekelund/subscriptions","organizations_url":"https://api.github.com/users/fredrikekelund/orgs","repos_url":"https://api.github.com/users/fredrikekelund/repos","events_url":"https://api.github.com/users/fredrikekelund/events{/privacy}","received_events_url":"https://api.github.com/users/fredrikekelund/received_events","type":"User","site_admin":false},"created_at":"2016-04-01T05:32:00Z","updated_at":"2016-04-01T05:32:00Z","body":"Responses that are compressed are supposed to be extracted automatically. Keep in mind that if you get strange results from calling `toString` on the response buffer, that could also be due to the response encoding. Try setting `crawler.decodeResponses` to true to get back a string that's decoded based on the response's charset"}}