{"action":"created","comment":{"url":"https://api.github.com/repos/isidzukuri/e_crawler/pulls/comments/83025023","pull_request_review_id":3892017,"id":83025023,"diff_hunk":"@@ -0,0 +1,15 @@\n+class CategoryCopier\n+  def self.copy(type, url, category_id)\n+    new(type, category_id).copy(url)\n+  end\n+\n+  def initialize(type, category_id)\n+    @crawler = type.to_s.classify.constantize","path":"app/repos/category_copier.rb","position":7,"original_position":7,"commit_id":"b4c6866f39db415505691dcdaa988d388f49bbfa","original_commit_id":"b4c6866f39db415505691dcdaa988d388f49bbfa","user":{"login":"AKovtunov","id":1730823,"avatar_url":"https://avatars.githubusercontent.com/u/1730823?v=3","gravatar_id":"","url":"https://api.github.com/users/AKovtunov","html_url":"https://github.com/AKovtunov","followers_url":"https://api.github.com/users/AKovtunov/followers","following_url":"https://api.github.com/users/AKovtunov/following{/other_user}","gists_url":"https://api.github.com/users/AKovtunov/gists{/gist_id}","starred_url":"https://api.github.com/users/AKovtunov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AKovtunov/subscriptions","organizations_url":"https://api.github.com/users/AKovtunov/orgs","repos_url":"https://api.github.com/users/AKovtunov/repos","events_url":"https://api.github.com/users/AKovtunov/events{/privacy}","received_events_url":"https://api.github.com/users/AKovtunov/received_events","type":"User","site_admin":false},"body":"won't it be the same crawler?","created_at":"2016-10-12T15:00:46Z","updated_at":"2016-10-12T15:00:46Z","html_url":"https://github.com/isidzukuri/e_crawler/pull/6#discussion_r83025023","pull_request_url":"https://api.github.com/repos/isidzukuri/e_crawler/pulls/6","_links":{"self":{"href":"https://api.github.com/repos/isidzukuri/e_crawler/pulls/comments/83025023"},"html":{"href":"https://github.com/isidzukuri/e_crawler/pull/6#discussion_r83025023"},"pull_request":{"href":"https://api.github.com/repos/isidzukuri/e_crawler/pulls/6"}}},"pull_request":{"url":"https://api.github.com/repos/isidzukuri/e_crawler/pulls/6","id":88980015,"html_url":"https://github.com/isidzukuri/e_crawler/pull/6","diff_url":"https://github.com/isidzukuri/e_crawler/pull/6.diff","patch_url":"https://github.com/isidzukuri/e_crawler/pull/6.patch","issue_url":"https://api.github.com/repos/isidzukuri/e_crawler/issues/6","number":6,"state":"open","locked":false,"title":"Crawler","user":{"login":"isidzukuri","id":5416083,"avatar_url":"https://avatars.githubusercontent.com/u/5416083?v=3","gravatar_id":"","url":"https://api.github.com/users/isidzukuri","html_url":"https://github.com/isidzukuri","followers_url":"https://api.github.com/users/isidzukuri/followers","following_url":"https://api.github.com/users/isidzukuri/following{/other_user}","gists_url":"https://api.github.com/users/isidzukuri/gists{/gist_id}","starred_url":"https://api.github.com/users/isidzukuri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/isidzukuri/subscriptions","organizations_url":"https://api.github.com/users/isidzukuri/orgs","repos_url":"https://api.github.com/users/isidzukuri/repos","events_url":"https://api.github.com/users/isidzukuri/events{/privacy}","received_events_url":"https://api.github.com/users/isidzukuri/received_events","type":"User","site_admin":false},"body":"Crawler copies items from given url from http://www.1a.lv/.\r\nYou can enter in url input smtn like http://www.1a.lv/datortehnika_preces_birojam/portativie_datori_un_aksesuari/sledzenes_portativajiem_datoriem and parser will do the job.\r\nCrawler takes content of this page and build list of good's urls from it by seted css selector (library can be easy extended to work with urls and regex patterns for example).\r\nif paginator selector specified crawler checks is there paginator. If it is crawler create set of pages with items and searchs for good's on each of them.\r\nAfter creating list of all products urls crawler begin to scrape data from every page. When data are collected rails controller and model handle them.\r\n\r\nUsage? \r\nThere are class Parser. For usage u should create your own for urs website, in this case site 1a.lv. So it looks like this: \"LaLv << Crawler::Parser\". In created class must be implemented extract_data method.  Here u describe how to scrape data from specific product's page. Run method parse. Thats it, nothing more.\r\n\r\nAlso you can easy to plugin classes for other websites. So module Crawler is reusable.\r\n\r\nPerformance tweaks:\r\n- parser can cashe visited pages. So there are no need to load them again \r\n- parser can work in many threads (25 by defaults).\r\n- reusage mechanize's created instances. Сreating new instances is costly and inflects performance.\r\n- packages storing data into db (100 items per query by defaults). There are no transaction for each item. For example category \"laptops\" has 523 goods. So its 523 transactions - quite long. With package saving only 6.\r\n\r\nModule Crawler is far not perfect. But it works and quite stable. ","created_at":"2016-10-12T09:16:15Z","updated_at":"2016-10-12T15:02:00Z","closed_at":null,"merged_at":null,"merge_commit_sha":"25892580e6265221dc1d2d6b9b92b628e2d092a1","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/isidzukuri/e_crawler/pulls/6/commits","review_comments_url":"https://api.github.com/repos/isidzukuri/e_crawler/pulls/6/comments","review_comment_url":"https://api.github.com/repos/isidzukuri/e_crawler/pulls/comments{/number}","comments_url":"https://api.github.com/repos/isidzukuri/e_crawler/issues/6/comments","statuses_url":"https://api.github.com/repos/isidzukuri/e_crawler/statuses/b4c6866f39db415505691dcdaa988d388f49bbfa","head":{"label":"isidzukuri:crawler","ref":"crawler","sha":"b4c6866f39db415505691dcdaa988d388f49bbfa","user":{"login":"isidzukuri","id":5416083,"avatar_url":"https://avatars.githubusercontent.com/u/5416083?v=3","gravatar_id":"","url":"https://api.github.com/users/isidzukuri","html_url":"https://github.com/isidzukuri","followers_url":"https://api.github.com/users/isidzukuri/followers","following_url":"https://api.github.com/users/isidzukuri/following{/other_user}","gists_url":"https://api.github.com/users/isidzukuri/gists{/gist_id}","starred_url":"https://api.github.com/users/isidzukuri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/isidzukuri/subscriptions","organizations_url":"https://api.github.com/users/isidzukuri/orgs","repos_url":"https://api.github.com/users/isidzukuri/repos","events_url":"https://api.github.com/users/isidzukuri/events{/privacy}","received_events_url":"https://api.github.com/users/isidzukuri/received_events","type":"User","site_admin":false},"repo":{"id":70460811,"name":"e_crawler","full_name":"isidzukuri/e_crawler","owner":{"login":"isidzukuri","id":5416083,"avatar_url":"https://avatars.githubusercontent.com/u/5416083?v=3","gravatar_id":"","url":"https://api.github.com/users/isidzukuri","html_url":"https://github.com/isidzukuri","followers_url":"https://api.github.com/users/isidzukuri/followers","following_url":"https://api.github.com/users/isidzukuri/following{/other_user}","gists_url":"https://api.github.com/users/isidzukuri/gists{/gist_id}","starred_url":"https://api.github.com/users/isidzukuri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/isidzukuri/subscriptions","organizations_url":"https://api.github.com/users/isidzukuri/orgs","repos_url":"https://api.github.com/users/isidzukuri/repos","events_url":"https://api.github.com/users/isidzukuri/events{/privacy}","received_events_url":"https://api.github.com/users/isidzukuri/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/isidzukuri/e_crawler","description":null,"fork":false,"url":"https://api.github.com/repos/isidzukuri/e_crawler","forks_url":"https://api.github.com/repos/isidzukuri/e_crawler/forks","keys_url":"https://api.github.com/repos/isidzukuri/e_crawler/keys{/key_id}","collaborators_url":"https://api.github.com/repos/isidzukuri/e_crawler/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/isidzukuri/e_crawler/teams","hooks_url":"https://api.github.com/repos/isidzukuri/e_crawler/hooks","issue_events_url":"https://api.github.com/repos/isidzukuri/e_crawler/issues/events{/number}","events_url":"https://api.github.com/repos/isidzukuri/e_crawler/events","assignees_url":"https://api.github.com/repos/isidzukuri/e_crawler/assignees{/user}","branches_url":"https://api.github.com/repos/isidzukuri/e_crawler/branches{/branch}","tags_url":"https://api.github.com/repos/isidzukuri/e_crawler/tags","blobs_url":"https://api.github.com/repos/isidzukuri/e_crawler/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/isidzukuri/e_crawler/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/isidzukuri/e_crawler/git/refs{/sha}","trees_url":"https://api.github.com/repos/isidzukuri/e_crawler/git/trees{/sha}","statuses_url":"https://api.github.com/repos/isidzukuri/e_crawler/statuses/{sha}","languages_url":"https://api.github.com/repos/isidzukuri/e_crawler/languages","stargazers_url":"https://api.github.com/repos/isidzukuri/e_crawler/stargazers","contributors_url":"https://api.github.com/repos/isidzukuri/e_crawler/contributors","subscribers_url":"https://api.github.com/repos/isidzukuri/e_crawler/subscribers","subscription_url":"https://api.github.com/repos/isidzukuri/e_crawler/subscription","commits_url":"https://api.github.com/repos/isidzukuri/e_crawler/commits{/sha}","git_commits_url":"https://api.github.com/repos/isidzukuri/e_crawler/git/commits{/sha}","comments_url":"https://api.github.com/repos/isidzukuri/e_crawler/comments{/number}","issue_comment_url":"https://api.github.com/repos/isidzukuri/e_crawler/issues/comments{/number}","contents_url":"https://api.github.com/repos/isidzukuri/e_crawler/contents/{+path}","compare_url":"https://api.github.com/repos/isidzukuri/e_crawler/compare/{base}...{head}","merges_url":"https://api.github.com/repos/isidzukuri/e_crawler/merges","archive_url":"https://api.github.com/repos/isidzukuri/e_crawler/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/isidzukuri/e_crawler/downloads","issues_url":"https://api.github.com/repos/isidzukuri/e_crawler/issues{/number}","pulls_url":"https://api.github.com/repos/isidzukuri/e_crawler/pulls{/number}","milestones_url":"https://api.github.com/repos/isidzukuri/e_crawler/milestones{/number}","notifications_url":"https://api.github.com/repos/isidzukuri/e_crawler/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/isidzukuri/e_crawler/labels{/name}","releases_url":"https://api.github.com/repos/isidzukuri/e_crawler/releases{/id}","deployments_url":"https://api.github.com/repos/isidzukuri/e_crawler/deployments","created_at":"2016-10-10T07:01:43Z","updated_at":"2016-10-10T07:19:18Z","pushed_at":"2016-10-12T14:42:31Z","git_url":"git://github.com/isidzukuri/e_crawler.git","ssh_url":"git@github.com:isidzukuri/e_crawler.git","clone_url":"https://github.com/isidzukuri/e_crawler.git","svn_url":"https://github.com/isidzukuri/e_crawler","homepage":null,"size":129,"stargazers_count":0,"watchers_count":0,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"isidzukuri:master","ref":"master","sha":"faff02607572446788e9f4c576ed5bc68c993c2e","user":{"login":"isidzukuri","id":5416083,"avatar_url":"https://avatars.githubusercontent.com/u/5416083?v=3","gravatar_id":"","url":"https://api.github.com/users/isidzukuri","html_url":"https://github.com/isidzukuri","followers_url":"https://api.github.com/users/isidzukuri/followers","following_url":"https://api.github.com/users/isidzukuri/following{/other_user}","gists_url":"https://api.github.com/users/isidzukuri/gists{/gist_id}","starred_url":"https://api.github.com/users/isidzukuri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/isidzukuri/subscriptions","organizations_url":"https://api.github.com/users/isidzukuri/orgs","repos_url":"https://api.github.com/users/isidzukuri/repos","events_url":"https://api.github.com/users/isidzukuri/events{/privacy}","received_events_url":"https://api.github.com/users/isidzukuri/received_events","type":"User","site_admin":false},"repo":{"id":70460811,"name":"e_crawler","full_name":"isidzukuri/e_crawler","owner":{"login":"isidzukuri","id":5416083,"avatar_url":"https://avatars.githubusercontent.com/u/5416083?v=3","gravatar_id":"","url":"https://api.github.com/users/isidzukuri","html_url":"https://github.com/isidzukuri","followers_url":"https://api.github.com/users/isidzukuri/followers","following_url":"https://api.github.com/users/isidzukuri/following{/other_user}","gists_url":"https://api.github.com/users/isidzukuri/gists{/gist_id}","starred_url":"https://api.github.com/users/isidzukuri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/isidzukuri/subscriptions","organizations_url":"https://api.github.com/users/isidzukuri/orgs","repos_url":"https://api.github.com/users/isidzukuri/repos","events_url":"https://api.github.com/users/isidzukuri/events{/privacy}","received_events_url":"https://api.github.com/users/isidzukuri/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/isidzukuri/e_crawler","description":null,"fork":false,"url":"https://api.github.com/repos/isidzukuri/e_crawler","forks_url":"https://api.github.com/repos/isidzukuri/e_crawler/forks","keys_url":"https://api.github.com/repos/isidzukuri/e_crawler/keys{/key_id}","collaborators_url":"https://api.github.com/repos/isidzukuri/e_crawler/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/isidzukuri/e_crawler/teams","hooks_url":"https://api.github.com/repos/isidzukuri/e_crawler/hooks","issue_events_url":"https://api.github.com/repos/isidzukuri/e_crawler/issues/events{/number}","events_url":"https://api.github.com/repos/isidzukuri/e_crawler/events","assignees_url":"https://api.github.com/repos/isidzukuri/e_crawler/assignees{/user}","branches_url":"https://api.github.com/repos/isidzukuri/e_crawler/branches{/branch}","tags_url":"https://api.github.com/repos/isidzukuri/e_crawler/tags","blobs_url":"https://api.github.com/repos/isidzukuri/e_crawler/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/isidzukuri/e_crawler/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/isidzukuri/e_crawler/git/refs{/sha}","trees_url":"https://api.github.com/repos/isidzukuri/e_crawler/git/trees{/sha}","statuses_url":"https://api.github.com/repos/isidzukuri/e_crawler/statuses/{sha}","languages_url":"https://api.github.com/repos/isidzukuri/e_crawler/languages","stargazers_url":"https://api.github.com/repos/isidzukuri/e_crawler/stargazers","contributors_url":"https://api.github.com/repos/isidzukuri/e_crawler/contributors","subscribers_url":"https://api.github.com/repos/isidzukuri/e_crawler/subscribers","subscription_url":"https://api.github.com/repos/isidzukuri/e_crawler/subscription","commits_url":"https://api.github.com/repos/isidzukuri/e_crawler/commits{/sha}","git_commits_url":"https://api.github.com/repos/isidzukuri/e_crawler/git/commits{/sha}","comments_url":"https://api.github.com/repos/isidzukuri/e_crawler/comments{/number}","issue_comment_url":"https://api.github.com/repos/isidzukuri/e_crawler/issues/comments{/number}","contents_url":"https://api.github.com/repos/isidzukuri/e_crawler/contents/{+path}","compare_url":"https://api.github.com/repos/isidzukuri/e_crawler/compare/{base}...{head}","merges_url":"https://api.github.com/repos/isidzukuri/e_crawler/merges","archive_url":"https://api.github.com/repos/isidzukuri/e_crawler/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/isidzukuri/e_crawler/downloads","issues_url":"https://api.github.com/repos/isidzukuri/e_crawler/issues{/number}","pulls_url":"https://api.github.com/repos/isidzukuri/e_crawler/pulls{/number}","milestones_url":"https://api.github.com/repos/isidzukuri/e_crawler/milestones{/number}","notifications_url":"https://api.github.com/repos/isidzukuri/e_crawler/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/isidzukuri/e_crawler/labels{/name}","releases_url":"https://api.github.com/repos/isidzukuri/e_crawler/releases{/id}","deployments_url":"https://api.github.com/repos/isidzukuri/e_crawler/deployments","created_at":"2016-10-10T07:01:43Z","updated_at":"2016-10-10T07:19:18Z","pushed_at":"2016-10-12T14:42:31Z","git_url":"git://github.com/isidzukuri/e_crawler.git","ssh_url":"git@github.com:isidzukuri/e_crawler.git","clone_url":"https://github.com/isidzukuri/e_crawler.git","svn_url":"https://github.com/isidzukuri/e_crawler","homepage":null,"size":129,"stargazers_count":0,"watchers_count":0,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/isidzukuri/e_crawler/pulls/6"},"html":{"href":"https://github.com/isidzukuri/e_crawler/pull/6"},"issue":{"href":"https://api.github.com/repos/isidzukuri/e_crawler/issues/6"},"comments":{"href":"https://api.github.com/repos/isidzukuri/e_crawler/issues/6/comments"},"review_comments":{"href":"https://api.github.com/repos/isidzukuri/e_crawler/pulls/6/comments"},"review_comment":{"href":"https://api.github.com/repos/isidzukuri/e_crawler/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/isidzukuri/e_crawler/pulls/6/commits"},"statuses":{"href":"https://api.github.com/repos/isidzukuri/e_crawler/statuses/b4c6866f39db415505691dcdaa988d388f49bbfa"}}}}
{"actor":{"display_login":"AKovtunov"}}