{"action":"created","comment":{"url":"https://api.github.com/repos/eagletmt/ttcoder/pulls/comments/54345398","id":54345398,"diff_hunk":"@@ -0,0 +1,63 @@\n+require 'faraday'\n+require 'json'\n+\n+class CodeforcesCrawler\n+  def run(opts = {})\n+    params = {}\n+    params[:count] = opts[:count] || 1000\n+    from = Time.now\n+    crawl(params)\n+    to = Time.now\n+    Fluent::Logger.post('crawl_codeforces', params: params, duration: (to - from))\n+  end\n+\n+  def crawl(params)\n+    @conn ||= Faraday.new(url: 'http://codeforces.com/api/') do |faraday|\n+      faraday.use Faraday::Response::RaiseError\n+      faraday.adapter Faraday.default_adapter\n+    end\n+\n+    api_method = params[:method] || 'problemset.recentStatus'","path":"lib/codeforces_crawler.rb","position":20,"original_position":20,"commit_id":"6d2a12cafb284e94d8411f263e2f099f95b06e26","original_commit_id":"6d2a12cafb284e94d8411f263e2f099f95b06e26","user":{"login":"eagletmt","id":69755,"avatar_url":"https://avatars.githubusercontent.com/u/69755?v=3","gravatar_id":"","url":"https://api.github.com/users/eagletmt","html_url":"https://github.com/eagletmt","followers_url":"https://api.github.com/users/eagletmt/followers","following_url":"https://api.github.com/users/eagletmt/following{/other_user}","gists_url":"https://api.github.com/users/eagletmt/gists{/gist_id}","starred_url":"https://api.github.com/users/eagletmt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eagletmt/subscriptions","organizations_url":"https://api.github.com/users/eagletmt/orgs","repos_url":"https://api.github.com/users/eagletmt/repos","events_url":"https://api.github.com/users/eagletmt/events{/privacy}","received_events_url":"https://api.github.com/users/eagletmt/received_events","type":"User","site_admin":false},"body":"あ、descriptioin ちゃんと読んでなかった。とりあえずこれでいきましょう。","created_at":"2016-02-28T07:38:34Z","updated_at":"2016-02-28T07:38:34Z","html_url":"https://github.com/eagletmt/ttcoder/pull/10#discussion_r54345398","pull_request_url":"https://api.github.com/repos/eagletmt/ttcoder/pulls/10","_links":{"self":{"href":"https://api.github.com/repos/eagletmt/ttcoder/pulls/comments/54345398"},"html":{"href":"https://github.com/eagletmt/ttcoder/pull/10#discussion_r54345398"},"pull_request":{"href":"https://api.github.com/repos/eagletmt/ttcoder/pulls/10"}}},"pull_request":{"url":"https://api.github.com/repos/eagletmt/ttcoder/pulls/10","id":60952854,"html_url":"https://github.com/eagletmt/ttcoder/pull/10","diff_url":"https://github.com/eagletmt/ttcoder/pull/10.diff","patch_url":"https://github.com/eagletmt/ttcoder/pull/10.patch","issue_url":"https://api.github.com/repos/eagletmt/ttcoder/issues/10","number":10,"state":"open","locked":false,"title":"Add Codeforces","user":{"login":"hadrori","id":4272032,"avatar_url":"https://avatars.githubusercontent.com/u/4272032?v=3","gravatar_id":"","url":"https://api.github.com/users/hadrori","html_url":"https://github.com/hadrori","followers_url":"https://api.github.com/users/hadrori/followers","following_url":"https://api.github.com/users/hadrori/following{/other_user}","gists_url":"https://api.github.com/users/hadrori/gists{/gist_id}","starred_url":"https://api.github.com/users/hadrori/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hadrori/subscriptions","organizations_url":"https://api.github.com/users/hadrori/orgs","repos_url":"https://api.github.com/users/hadrori/repos","events_url":"https://api.github.com/users/hadrori/events{/privacy}","received_events_url":"https://api.github.com/users/hadrori/received_events","type":"User","site_admin":false},"body":"### 概要\r\n\r\nAOJだと大体誰かしら解いてる感じで，POJもあんまり使われてないっぽいのでCodeforcesを追加したいです．\r\n主な変更点は以下です．\r\n- `codeforces_submissions`table追加\r\n- `users`tableに`codeforces_user`column追加\r\n- `CodeforcesSubmission`, `CodeforcesController`, `CodeforcesCrawler`追加(AOJ, POJと大体同じ)\r\n- Userの`codeforces_user`が変わったらそのユーザーの過去のsubmissionを全件取得する\r\n\r\n最後の全件取得のとき，submissionがむちゃくちゃ多いと時間がかかっちゃうかもしれないです．\r\n手元だと僕のが700件ぐらいで604msでした．\r\n\r\n### CodeforcesのAPIについて\r\n\r\n- problemset.recentStatusは頭からしか取れない  \r\nので今回crawlerは1000件1回しかまわさないようになってます．これ何かいい解決方法ありますかね．\r\n\r\n### スクショ\r\n![screen shot 2016-02-27 at 7 00 01 pm](https://cloud.githubusercontent.com/assets/4272032/13372387/952780c4-dd84-11e5-9ea8-2f8f8ec8c777.png)\r\n\r\n![screen shot 2016-02-27 at 7 47 50 pm](https://cloud.githubusercontent.com/assets/4272032/13372555/2f7c4d3e-dd8b-11e5-80cc-803cf6498261.png)\r\n\r\n![screen shot 2016-02-27 at 11 59 26 pm](https://cloud.githubusercontent.com/assets/4272032/13373559/2f292f5a-ddae-11e5-8481-9bc34a96ea73.png)\r\n\r\n\r\n@eagletmt 暇なときにでも見てもらえませんか．宜しくお願いします","created_at":"2016-02-27T15:00:10Z","updated_at":"2016-02-28T07:38:34Z","closed_at":null,"merged_at":null,"merge_commit_sha":"278227ae1492b0042302a07c8b68dfd70e4ad522","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/eagletmt/ttcoder/pulls/10/commits","review_comments_url":"https://api.github.com/repos/eagletmt/ttcoder/pulls/10/comments","review_comment_url":"https://api.github.com/repos/eagletmt/ttcoder/pulls/comments{/number}","comments_url":"https://api.github.com/repos/eagletmt/ttcoder/issues/10/comments","statuses_url":"https://api.github.com/repos/eagletmt/ttcoder/statuses/6d2a12cafb284e94d8411f263e2f099f95b06e26","head":{"label":"hadrori:add_codeforces","ref":"add_codeforces","sha":"6d2a12cafb284e94d8411f263e2f099f95b06e26","user":{"login":"hadrori","id":4272032,"avatar_url":"https://avatars.githubusercontent.com/u/4272032?v=3","gravatar_id":"","url":"https://api.github.com/users/hadrori","html_url":"https://github.com/hadrori","followers_url":"https://api.github.com/users/hadrori/followers","following_url":"https://api.github.com/users/hadrori/following{/other_user}","gists_url":"https://api.github.com/users/hadrori/gists{/gist_id}","starred_url":"https://api.github.com/users/hadrori/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hadrori/subscriptions","organizations_url":"https://api.github.com/users/hadrori/orgs","repos_url":"https://api.github.com/users/hadrori/repos","events_url":"https://api.github.com/users/hadrori/events{/privacy}","received_events_url":"https://api.github.com/users/hadrori/received_events","type":"User","site_admin":false},"repo":{"id":49952172,"name":"ttcoder","full_name":"hadrori/ttcoder","owner":{"login":"hadrori","id":4272032,"avatar_url":"https://avatars.githubusercontent.com/u/4272032?v=3","gravatar_id":"","url":"https://api.github.com/users/hadrori","html_url":"https://github.com/hadrori","followers_url":"https://api.github.com/users/hadrori/followers","following_url":"https://api.github.com/users/hadrori/following{/other_user}","gists_url":"https://api.github.com/users/hadrori/gists{/gist_id}","starred_url":"https://api.github.com/users/hadrori/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hadrori/subscriptions","organizations_url":"https://api.github.com/users/hadrori/orgs","repos_url":"https://api.github.com/users/hadrori/repos","events_url":"https://api.github.com/users/hadrori/events{/privacy}","received_events_url":"https://api.github.com/users/hadrori/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/hadrori/ttcoder","description":"TokyoTechCoder","fork":true,"url":"https://api.github.com/repos/hadrori/ttcoder","forks_url":"https://api.github.com/repos/hadrori/ttcoder/forks","keys_url":"https://api.github.com/repos/hadrori/ttcoder/keys{/key_id}","collaborators_url":"https://api.github.com/repos/hadrori/ttcoder/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/hadrori/ttcoder/teams","hooks_url":"https://api.github.com/repos/hadrori/ttcoder/hooks","issue_events_url":"https://api.github.com/repos/hadrori/ttcoder/issues/events{/number}","events_url":"https://api.github.com/repos/hadrori/ttcoder/events","assignees_url":"https://api.github.com/repos/hadrori/ttcoder/assignees{/user}","branches_url":"https://api.github.com/repos/hadrori/ttcoder/branches{/branch}","tags_url":"https://api.github.com/repos/hadrori/ttcoder/tags","blobs_url":"https://api.github.com/repos/hadrori/ttcoder/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/hadrori/ttcoder/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/hadrori/ttcoder/git/refs{/sha}","trees_url":"https://api.github.com/repos/hadrori/ttcoder/git/trees{/sha}","statuses_url":"https://api.github.com/repos/hadrori/ttcoder/statuses/{sha}","languages_url":"https://api.github.com/repos/hadrori/ttcoder/languages","stargazers_url":"https://api.github.com/repos/hadrori/ttcoder/stargazers","contributors_url":"https://api.github.com/repos/hadrori/ttcoder/contributors","subscribers_url":"https://api.github.com/repos/hadrori/ttcoder/subscribers","subscription_url":"https://api.github.com/repos/hadrori/ttcoder/subscription","commits_url":"https://api.github.com/repos/hadrori/ttcoder/commits{/sha}","git_commits_url":"https://api.github.com/repos/hadrori/ttcoder/git/commits{/sha}","comments_url":"https://api.github.com/repos/hadrori/ttcoder/comments{/number}","issue_comment_url":"https://api.github.com/repos/hadrori/ttcoder/issues/comments{/number}","contents_url":"https://api.github.com/repos/hadrori/ttcoder/contents/{+path}","compare_url":"https://api.github.com/repos/hadrori/ttcoder/compare/{base}...{head}","merges_url":"https://api.github.com/repos/hadrori/ttcoder/merges","archive_url":"https://api.github.com/repos/hadrori/ttcoder/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/hadrori/ttcoder/downloads","issues_url":"https://api.github.com/repos/hadrori/ttcoder/issues{/number}","pulls_url":"https://api.github.com/repos/hadrori/ttcoder/pulls{/number}","milestones_url":"https://api.github.com/repos/hadrori/ttcoder/milestones{/number}","notifications_url":"https://api.github.com/repos/hadrori/ttcoder/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/hadrori/ttcoder/labels{/name}","releases_url":"https://api.github.com/repos/hadrori/ttcoder/releases{/id}","deployments_url":"https://api.github.com/repos/hadrori/ttcoder/deployments","created_at":"2016-01-19T12:47:16Z","updated_at":"2016-01-19T12:47:17Z","pushed_at":"2016-02-27T14:40:27Z","git_url":"git://github.com/hadrori/ttcoder.git","ssh_url":"git@github.com:hadrori/ttcoder.git","clone_url":"https://github.com/hadrori/ttcoder.git","svn_url":"https://github.com/hadrori/ttcoder","homepage":"https://ttc.wanko.cc/","size":457,"stargazers_count":0,"watchers_count":0,"language":"Ruby","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"eagletmt:master","ref":"master","sha":"878547f5b6bd26bd467f40c88f6984988cd87b1e","user":{"login":"eagletmt","id":69755,"avatar_url":"https://avatars.githubusercontent.com/u/69755?v=3","gravatar_id":"","url":"https://api.github.com/users/eagletmt","html_url":"https://github.com/eagletmt","followers_url":"https://api.github.com/users/eagletmt/followers","following_url":"https://api.github.com/users/eagletmt/following{/other_user}","gists_url":"https://api.github.com/users/eagletmt/gists{/gist_id}","starred_url":"https://api.github.com/users/eagletmt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eagletmt/subscriptions","organizations_url":"https://api.github.com/users/eagletmt/orgs","repos_url":"https://api.github.com/users/eagletmt/repos","events_url":"https://api.github.com/users/eagletmt/events{/privacy}","received_events_url":"https://api.github.com/users/eagletmt/received_events","type":"User","site_admin":false},"repo":{"id":16878353,"name":"ttcoder","full_name":"eagletmt/ttcoder","owner":{"login":"eagletmt","id":69755,"avatar_url":"https://avatars.githubusercontent.com/u/69755?v=3","gravatar_id":"","url":"https://api.github.com/users/eagletmt","html_url":"https://github.com/eagletmt","followers_url":"https://api.github.com/users/eagletmt/followers","following_url":"https://api.github.com/users/eagletmt/following{/other_user}","gists_url":"https://api.github.com/users/eagletmt/gists{/gist_id}","starred_url":"https://api.github.com/users/eagletmt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eagletmt/subscriptions","organizations_url":"https://api.github.com/users/eagletmt/orgs","repos_url":"https://api.github.com/users/eagletmt/repos","events_url":"https://api.github.com/users/eagletmt/events{/privacy}","received_events_url":"https://api.github.com/users/eagletmt/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/eagletmt/ttcoder","description":"TokyoTechCoder","fork":false,"url":"https://api.github.com/repos/eagletmt/ttcoder","forks_url":"https://api.github.com/repos/eagletmt/ttcoder/forks","keys_url":"https://api.github.com/repos/eagletmt/ttcoder/keys{/key_id}","collaborators_url":"https://api.github.com/repos/eagletmt/ttcoder/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/eagletmt/ttcoder/teams","hooks_url":"https://api.github.com/repos/eagletmt/ttcoder/hooks","issue_events_url":"https://api.github.com/repos/eagletmt/ttcoder/issues/events{/number}","events_url":"https://api.github.com/repos/eagletmt/ttcoder/events","assignees_url":"https://api.github.com/repos/eagletmt/ttcoder/assignees{/user}","branches_url":"https://api.github.com/repos/eagletmt/ttcoder/branches{/branch}","tags_url":"https://api.github.com/repos/eagletmt/ttcoder/tags","blobs_url":"https://api.github.com/repos/eagletmt/ttcoder/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/eagletmt/ttcoder/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/eagletmt/ttcoder/git/refs{/sha}","trees_url":"https://api.github.com/repos/eagletmt/ttcoder/git/trees{/sha}","statuses_url":"https://api.github.com/repos/eagletmt/ttcoder/statuses/{sha}","languages_url":"https://api.github.com/repos/eagletmt/ttcoder/languages","stargazers_url":"https://api.github.com/repos/eagletmt/ttcoder/stargazers","contributors_url":"https://api.github.com/repos/eagletmt/ttcoder/contributors","subscribers_url":"https://api.github.com/repos/eagletmt/ttcoder/subscribers","subscription_url":"https://api.github.com/repos/eagletmt/ttcoder/subscription","commits_url":"https://api.github.com/repos/eagletmt/ttcoder/commits{/sha}","git_commits_url":"https://api.github.com/repos/eagletmt/ttcoder/git/commits{/sha}","comments_url":"https://api.github.com/repos/eagletmt/ttcoder/comments{/number}","issue_comment_url":"https://api.github.com/repos/eagletmt/ttcoder/issues/comments{/number}","contents_url":"https://api.github.com/repos/eagletmt/ttcoder/contents/{+path}","compare_url":"https://api.github.com/repos/eagletmt/ttcoder/compare/{base}...{head}","merges_url":"https://api.github.com/repos/eagletmt/ttcoder/merges","archive_url":"https://api.github.com/repos/eagletmt/ttcoder/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/eagletmt/ttcoder/downloads","issues_url":"https://api.github.com/repos/eagletmt/ttcoder/issues{/number}","pulls_url":"https://api.github.com/repos/eagletmt/ttcoder/pulls{/number}","milestones_url":"https://api.github.com/repos/eagletmt/ttcoder/milestones{/number}","notifications_url":"https://api.github.com/repos/eagletmt/ttcoder/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/eagletmt/ttcoder/labels{/name}","releases_url":"https://api.github.com/repos/eagletmt/ttcoder/releases{/id}","deployments_url":"https://api.github.com/repos/eagletmt/ttcoder/deployments","created_at":"2014-02-16T04:34:21Z","updated_at":"2016-01-15T19:10:01Z","pushed_at":"2016-02-27T15:00:10Z","git_url":"git://github.com/eagletmt/ttcoder.git","ssh_url":"git@github.com:eagletmt/ttcoder.git","clone_url":"https://github.com/eagletmt/ttcoder.git","svn_url":"https://github.com/eagletmt/ttcoder","homepage":"https://ttc.wanko.cc/","size":432,"stargazers_count":3,"watchers_count":3,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":3,"forks":1,"open_issues":3,"watchers":3,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/eagletmt/ttcoder/pulls/10"},"html":{"href":"https://github.com/eagletmt/ttcoder/pull/10"},"issue":{"href":"https://api.github.com/repos/eagletmt/ttcoder/issues/10"},"comments":{"href":"https://api.github.com/repos/eagletmt/ttcoder/issues/10/comments"},"review_comments":{"href":"https://api.github.com/repos/eagletmt/ttcoder/pulls/10/comments"},"review_comment":{"href":"https://api.github.com/repos/eagletmt/ttcoder/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/eagletmt/ttcoder/pulls/10/commits"},"statuses":{"href":"https://api.github.com/repos/eagletmt/ttcoder/statuses/6d2a12cafb284e94d8411f263e2f099f95b06e26"}}}}