{"action":"created","issue":{"url":"https://api.github.com/repos/lapwinglabs/x-ray/issues/211","repository_url":"https://api.github.com/repos/lapwinglabs/x-ray","labels_url":"https://api.github.com/repos/lapwinglabs/x-ray/issues/211/labels{/name}","comments_url":"https://api.github.com/repos/lapwinglabs/x-ray/issues/211/comments","events_url":"https://api.github.com/repos/lapwinglabs/x-ray/issues/211/events","html_url":"https://github.com/lapwinglabs/x-ray/issues/211","id":175379040,"number":211,"title":"Duplicates crawled data when limit is set","user":{"login":"schiz","id":10157876,"avatar_url":"https://avatars.githubusercontent.com/u/10157876?v=3","gravatar_id":"","url":"https://api.github.com/users/schiz","html_url":"https://github.com/schiz","followers_url":"https://api.github.com/users/schiz/followers","following_url":"https://api.github.com/users/schiz/following{/other_user}","gists_url":"https://api.github.com/users/schiz/gists{/gist_id}","starred_url":"https://api.github.com/users/schiz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/schiz/subscriptions","organizations_url":"https://api.github.com/users/schiz/orgs","repos_url":"https://api.github.com/users/schiz/repos","events_url":"https://api.github.com/users/schiz/events{/privacy}","received_events_url":"https://api.github.com/users/schiz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-09-06T23:48:10Z","updated_at":"2016-09-07T09:55:43Z","closed_at":null,"body":"### Subject of the issue\r\n\r\nThere is a fixed number of pages which I crawl, e.g. 67. Each page contains 25 articles, which I crawl, but the last page (67-th) contains 3 articles. The total number of articles is 1653. I want x-ray to crawl  articles from all pages by itself, without setting the limit equal to number of pages by myself, so I set the limit to 'n' : \r\n\r\n```\r\nfunction generateURLs(){\r\n  x(url, '.h3 a', [{\r\n    link: '@href'\r\n  }])\r\n    .paginate('.page a@href')\r\n    .limit('n')\r\n    .write('results.json');\r\n}\r\n```\r\n\r\nAnd in that case x-ray keeps crawling articles from the pages infinitely, producing duplicate data in results.json. Otherwise, if I set limit to 67: \r\n\r\n```\r\nfunction generateURLs(){\r\n  x(url, '.h3 a', [{\r\n    link: '@href'\r\n  }])\r\n    .paginate('.page a@href')\r\n    .limit(67)\r\n    .write('results.json');\r\n}\r\n```\r\n\r\nit generates 1675 articles instead of 1653. Is there any way to crawl all pages without setting limit? And if limit is set, is there any way to get crawled data without duplicates?\r\n\r\n### Your environment\r\n\r\n* version of node: v4.4.7\r\n* version of npm: 3.3.9"},"comment":{"url":"https://api.github.com/repos/lapwinglabs/x-ray/issues/comments/245232548","html_url":"https://github.com/lapwinglabs/x-ray/issues/211#issuecomment-245232548","issue_url":"https://api.github.com/repos/lapwinglabs/x-ray/issues/211","id":245232548,"user":{"login":"Crazometer","id":5444465,"avatar_url":"https://avatars.githubusercontent.com/u/5444465?v=3","gravatar_id":"","url":"https://api.github.com/users/Crazometer","html_url":"https://github.com/Crazometer","followers_url":"https://api.github.com/users/Crazometer/followers","following_url":"https://api.github.com/users/Crazometer/following{/other_user}","gists_url":"https://api.github.com/users/Crazometer/gists{/gist_id}","starred_url":"https://api.github.com/users/Crazometer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Crazometer/subscriptions","organizations_url":"https://api.github.com/users/Crazometer/orgs","repos_url":"https://api.github.com/users/Crazometer/repos","events_url":"https://api.github.com/users/Crazometer/events{/privacy}","received_events_url":"https://api.github.com/users/Crazometer/received_events","type":"User","site_admin":false},"created_at":"2016-09-07T09:55:43Z","updated_at":"2016-09-07T09:55:43Z","body":"Have you checked that your paginate selector is working like you think it is? If you are on the first page it may match the next button but on the other pages it may match the previous button. One way to check this is by enabling the output of `debug` and looking at the urls it is crawling to."}}
{"actor":{"display_login":"Crazometer"}}