{"action":"created","comment":{"url":"https://api.github.com/repos/topicai/weakand/pulls/comments/52970689","id":52970689,"diff_hunk":"@@ -0,0 +1,106 @@\n+#Search multiple shards\n+## Overview\n+The idea behind multiple shards is to support search through large number of documents and utilize multiple machines to process one query parallely.\n+\n+## Requirements\n+1. The system should be able to split reverted index into multiple shards\n+2. The system should be able to handle node failure\n+3. The system should be able to add more nodes while keeping online\n+4. The system should be able to balance size of reverted index among all nodes\n+\n+## Architecture\n+![Alt text](http://g.gravizo.com/g?\n+  digraph G {\n+    aize =\"4,4\";\n+    Root -> Leaf1;\n+    Root -> Leaf2;\n+    Root -> Leaf3;\n+    PersistentStorage [shape=box]\n+    Root -> PersistentStorage\n+    Leaf1 -> PersistentStorage\n+    Leaf2 -> PersistentStorage\n+    Leaf3 -> PersistentStorage\n+    Etcd [shape=box]\n+    Root -> Etcd\n+    Leaf1 -> Etcd\n+    Leaf2 -> Etcd\n+    Leaf3 -> Etcd\n+  }\n+)\n+\n+## Specifications\n+### Sharding strategy\n+We plan to use Document based sharding. A comparison between doc based sharding and term based sharding([stolen from Jeff Dean](http://web.stanford.edu/class/cs276/Jeff-Dean-Stanford-CS276-April-2015.pdf)):","path":"multishard_design.md","position":33,"original_position":33,"commit_id":"9adebf25fba2a0844cd61ca4d76ba69b592b3672","original_commit_id":"9adebf25fba2a0844cd61ca4d76ba69b592b3672","user":{"login":"yitopic","id":16951881,"avatar_url":"https://avatars.githubusercontent.com/u/16951881?v=3","gravatar_id":"","url":"https://api.github.com/users/yitopic","html_url":"https://github.com/yitopic","followers_url":"https://api.github.com/users/yitopic/followers","following_url":"https://api.github.com/users/yitopic/following{/other_user}","gists_url":"https://api.github.com/users/yitopic/gists{/gist_id}","starred_url":"https://api.github.com/users/yitopic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yitopic/subscriptions","organizations_url":"https://api.github.com/users/yitopic/orgs","repos_url":"https://api.github.com/users/yitopic/repos","events_url":"https://api.github.com/users/yitopic/events{/privacy}","received_events_url":"https://api.github.com/users/yitopic/received_events","type":"User","site_admin":false},"body":"Awesome you found this comparison!","created_at":"2016-02-16T05:42:14Z","updated_at":"2016-02-16T05:42:14Z","html_url":"https://github.com/topicai/weakand/pull/9#discussion_r52970689","pull_request_url":"https://api.github.com/repos/topicai/weakand/pulls/9","_links":{"self":{"href":"https://api.github.com/repos/topicai/weakand/pulls/comments/52970689"},"html":{"href":"https://github.com/topicai/weakand/pull/9#discussion_r52970689"},"pull_request":{"href":"https://api.github.com/repos/topicai/weakand/pulls/9"}}},"pull_request":{"url":"https://api.github.com/repos/topicai/weakand/pulls/9","id":59415512,"html_url":"https://github.com/topicai/weakand/pull/9","diff_url":"https://github.com/topicai/weakand/pull/9.diff","patch_url":"https://github.com/topicai/weakand/pull/9.patch","issue_url":"https://api.github.com/repos/topicai/weakand/issues/9","number":9,"state":"open","locked":false,"title":"design doc(to be continued)","user":{"login":"chenkaitopic","id":17173065,"avatar_url":"https://avatars.githubusercontent.com/u/17173065?v=3","gravatar_id":"","url":"https://api.github.com/users/chenkaitopic","html_url":"https://github.com/chenkaitopic","followers_url":"https://api.github.com/users/chenkaitopic/followers","following_url":"https://api.github.com/users/chenkaitopic/following{/other_user}","gists_url":"https://api.github.com/users/chenkaitopic/gists{/gist_id}","starred_url":"https://api.github.com/users/chenkaitopic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chenkaitopic/subscriptions","organizations_url":"https://api.github.com/users/chenkaitopic/orgs","repos_url":"https://api.github.com/users/chenkaitopic/repos","events_url":"https://api.github.com/users/chenkaitopic/events{/privacy}","received_events_url":"https://api.github.com/users/chenkaitopic/received_events","type":"User","site_admin":false},"body":"Click View button to see the fully rendered markdown text.","created_at":"2016-02-15T21:29:12Z","updated_at":"2016-02-16T05:42:14Z","closed_at":null,"merged_at":null,"merge_commit_sha":"410888d2ef43a0c253298642af9b1dd502b453b0","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/topicai/weakand/pulls/9/commits","review_comments_url":"https://api.github.com/repos/topicai/weakand/pulls/9/comments","review_comment_url":"https://api.github.com/repos/topicai/weakand/pulls/comments{/number}","comments_url":"https://api.github.com/repos/topicai/weakand/issues/9/comments","statuses_url":"https://api.github.com/repos/topicai/weakand/statuses/9adebf25fba2a0844cd61ca4d76ba69b592b3672","head":{"label":"topicai:multishard","ref":"multishard","sha":"9adebf25fba2a0844cd61ca4d76ba69b592b3672","user":{"login":"topicai","id":16812557,"avatar_url":"https://avatars.githubusercontent.com/u/16812557?v=3","gravatar_id":"","url":"https://api.github.com/users/topicai","html_url":"https://github.com/topicai","followers_url":"https://api.github.com/users/topicai/followers","following_url":"https://api.github.com/users/topicai/following{/other_user}","gists_url":"https://api.github.com/users/topicai/gists{/gist_id}","starred_url":"https://api.github.com/users/topicai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/topicai/subscriptions","organizations_url":"https://api.github.com/users/topicai/orgs","repos_url":"https://api.github.com/users/topicai/repos","events_url":"https://api.github.com/users/topicai/events{/privacy}","received_events_url":"https://api.github.com/users/topicai/received_events","type":"Organization","site_admin":false},"repo":{"id":50558299,"name":"weakand","full_name":"topicai/weakand","owner":{"login":"topicai","id":16812557,"avatar_url":"https://avatars.githubusercontent.com/u/16812557?v=3","gravatar_id":"","url":"https://api.github.com/users/topicai","html_url":"https://github.com/topicai","followers_url":"https://api.github.com/users/topicai/followers","following_url":"https://api.github.com/users/topicai/following{/other_user}","gists_url":"https://api.github.com/users/topicai/gists{/gist_id}","starred_url":"https://api.github.com/users/topicai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/topicai/subscriptions","organizations_url":"https://api.github.com/users/topicai/orgs","repos_url":"https://api.github.com/users/topicai/repos","events_url":"https://api.github.com/users/topicai/events{/privacy}","received_events_url":"https://api.github.com/users/topicai/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/topicai/weakand","description":"","fork":false,"url":"https://api.github.com/repos/topicai/weakand","forks_url":"https://api.github.com/repos/topicai/weakand/forks","keys_url":"https://api.github.com/repos/topicai/weakand/keys{/key_id}","collaborators_url":"https://api.github.com/repos/topicai/weakand/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/topicai/weakand/teams","hooks_url":"https://api.github.com/repos/topicai/weakand/hooks","issue_events_url":"https://api.github.com/repos/topicai/weakand/issues/events{/number}","events_url":"https://api.github.com/repos/topicai/weakand/events","assignees_url":"https://api.github.com/repos/topicai/weakand/assignees{/user}","branches_url":"https://api.github.com/repos/topicai/weakand/branches{/branch}","tags_url":"https://api.github.com/repos/topicai/weakand/tags","blobs_url":"https://api.github.com/repos/topicai/weakand/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/topicai/weakand/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/topicai/weakand/git/refs{/sha}","trees_url":"https://api.github.com/repos/topicai/weakand/git/trees{/sha}","statuses_url":"https://api.github.com/repos/topicai/weakand/statuses/{sha}","languages_url":"https://api.github.com/repos/topicai/weakand/languages","stargazers_url":"https://api.github.com/repos/topicai/weakand/stargazers","contributors_url":"https://api.github.com/repos/topicai/weakand/contributors","subscribers_url":"https://api.github.com/repos/topicai/weakand/subscribers","subscription_url":"https://api.github.com/repos/topicai/weakand/subscription","commits_url":"https://api.github.com/repos/topicai/weakand/commits{/sha}","git_commits_url":"https://api.github.com/repos/topicai/weakand/git/commits{/sha}","comments_url":"https://api.github.com/repos/topicai/weakand/comments{/number}","issue_comment_url":"https://api.github.com/repos/topicai/weakand/issues/comments{/number}","contents_url":"https://api.github.com/repos/topicai/weakand/contents/{+path}","compare_url":"https://api.github.com/repos/topicai/weakand/compare/{base}...{head}","merges_url":"https://api.github.com/repos/topicai/weakand/merges","archive_url":"https://api.github.com/repos/topicai/weakand/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/topicai/weakand/downloads","issues_url":"https://api.github.com/repos/topicai/weakand/issues{/number}","pulls_url":"https://api.github.com/repos/topicai/weakand/pulls{/number}","milestones_url":"https://api.github.com/repos/topicai/weakand/milestones{/number}","notifications_url":"https://api.github.com/repos/topicai/weakand/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/topicai/weakand/labels{/name}","releases_url":"https://api.github.com/repos/topicai/weakand/releases{/id}","deployments_url":"https://api.github.com/repos/topicai/weakand/deployments","created_at":"2016-01-28T05:30:49Z","updated_at":"2016-02-11T07:08:27Z","pushed_at":"2016-02-15T21:29:13Z","git_url":"git://github.com/topicai/weakand.git","ssh_url":"git@github.com:topicai/weakand.git","clone_url":"https://github.com/topicai/weakand.git","svn_url":"https://github.com/topicai/weakand","homepage":null,"size":4957,"stargazers_count":1,"watchers_count":1,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":6,"forks":2,"open_issues":6,"watchers":1,"default_branch":"develop"}},"base":{"label":"topicai:develop","ref":"develop","sha":"2649836dc1aa11874ecb5717499fe6755db7ed80","user":{"login":"topicai","id":16812557,"avatar_url":"https://avatars.githubusercontent.com/u/16812557?v=3","gravatar_id":"","url":"https://api.github.com/users/topicai","html_url":"https://github.com/topicai","followers_url":"https://api.github.com/users/topicai/followers","following_url":"https://api.github.com/users/topicai/following{/other_user}","gists_url":"https://api.github.com/users/topicai/gists{/gist_id}","starred_url":"https://api.github.com/users/topicai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/topicai/subscriptions","organizations_url":"https://api.github.com/users/topicai/orgs","repos_url":"https://api.github.com/users/topicai/repos","events_url":"https://api.github.com/users/topicai/events{/privacy}","received_events_url":"https://api.github.com/users/topicai/received_events","type":"Organization","site_admin":false},"repo":{"id":50558299,"name":"weakand","full_name":"topicai/weakand","owner":{"login":"topicai","id":16812557,"avatar_url":"https://avatars.githubusercontent.com/u/16812557?v=3","gravatar_id":"","url":"https://api.github.com/users/topicai","html_url":"https://github.com/topicai","followers_url":"https://api.github.com/users/topicai/followers","following_url":"https://api.github.com/users/topicai/following{/other_user}","gists_url":"https://api.github.com/users/topicai/gists{/gist_id}","starred_url":"https://api.github.com/users/topicai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/topicai/subscriptions","organizations_url":"https://api.github.com/users/topicai/orgs","repos_url":"https://api.github.com/users/topicai/repos","events_url":"https://api.github.com/users/topicai/events{/privacy}","received_events_url":"https://api.github.com/users/topicai/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/topicai/weakand","description":"","fork":false,"url":"https://api.github.com/repos/topicai/weakand","forks_url":"https://api.github.com/repos/topicai/weakand/forks","keys_url":"https://api.github.com/repos/topicai/weakand/keys{/key_id}","collaborators_url":"https://api.github.com/repos/topicai/weakand/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/topicai/weakand/teams","hooks_url":"https://api.github.com/repos/topicai/weakand/hooks","issue_events_url":"https://api.github.com/repos/topicai/weakand/issues/events{/number}","events_url":"https://api.github.com/repos/topicai/weakand/events","assignees_url":"https://api.github.com/repos/topicai/weakand/assignees{/user}","branches_url":"https://api.github.com/repos/topicai/weakand/branches{/branch}","tags_url":"https://api.github.com/repos/topicai/weakand/tags","blobs_url":"https://api.github.com/repos/topicai/weakand/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/topicai/weakand/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/topicai/weakand/git/refs{/sha}","trees_url":"https://api.github.com/repos/topicai/weakand/git/trees{/sha}","statuses_url":"https://api.github.com/repos/topicai/weakand/statuses/{sha}","languages_url":"https://api.github.com/repos/topicai/weakand/languages","stargazers_url":"https://api.github.com/repos/topicai/weakand/stargazers","contributors_url":"https://api.github.com/repos/topicai/weakand/contributors","subscribers_url":"https://api.github.com/repos/topicai/weakand/subscribers","subscription_url":"https://api.github.com/repos/topicai/weakand/subscription","commits_url":"https://api.github.com/repos/topicai/weakand/commits{/sha}","git_commits_url":"https://api.github.com/repos/topicai/weakand/git/commits{/sha}","comments_url":"https://api.github.com/repos/topicai/weakand/comments{/number}","issue_comment_url":"https://api.github.com/repos/topicai/weakand/issues/comments{/number}","contents_url":"https://api.github.com/repos/topicai/weakand/contents/{+path}","compare_url":"https://api.github.com/repos/topicai/weakand/compare/{base}...{head}","merges_url":"https://api.github.com/repos/topicai/weakand/merges","archive_url":"https://api.github.com/repos/topicai/weakand/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/topicai/weakand/downloads","issues_url":"https://api.github.com/repos/topicai/weakand/issues{/number}","pulls_url":"https://api.github.com/repos/topicai/weakand/pulls{/number}","milestones_url":"https://api.github.com/repos/topicai/weakand/milestones{/number}","notifications_url":"https://api.github.com/repos/topicai/weakand/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/topicai/weakand/labels{/name}","releases_url":"https://api.github.com/repos/topicai/weakand/releases{/id}","deployments_url":"https://api.github.com/repos/topicai/weakand/deployments","created_at":"2016-01-28T05:30:49Z","updated_at":"2016-02-11T07:08:27Z","pushed_at":"2016-02-15T21:29:13Z","git_url":"git://github.com/topicai/weakand.git","ssh_url":"git@github.com:topicai/weakand.git","clone_url":"https://github.com/topicai/weakand.git","svn_url":"https://github.com/topicai/weakand","homepage":null,"size":4957,"stargazers_count":1,"watchers_count":1,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":6,"forks":2,"open_issues":6,"watchers":1,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/topicai/weakand/pulls/9"},"html":{"href":"https://github.com/topicai/weakand/pull/9"},"issue":{"href":"https://api.github.com/repos/topicai/weakand/issues/9"},"comments":{"href":"https://api.github.com/repos/topicai/weakand/issues/9/comments"},"review_comments":{"href":"https://api.github.com/repos/topicai/weakand/pulls/9/comments"},"review_comment":{"href":"https://api.github.com/repos/topicai/weakand/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/topicai/weakand/pulls/9/commits"},"statuses":{"href":"https://api.github.com/repos/topicai/weakand/statuses/9adebf25fba2a0844cd61ca4d76ba69b592b3672"}}}}