{"action":"created","comment":{"url":"https://api.github.com/repos/voldemort/voldemort/pulls/comments/73612114","id":73612114,"diff_hunk":"@@ -409,6 +380,69 @@ public void build() {\n     }\n \n     /**\n+     * This function configures a {@link JobConf} instance based on the {@param size} of the data\n+     * that Voldemort will need to serve. If the size provided is that of the input data, then\n+     * these settings will be calculated in a best-effort basis. It is not possible to predict in\n+     * advance the correct settings based on the input data, since compression factors may vary\n+     * a lot between data sets.\n+     *\n+     * On the other hand, if the size is that of the mappers' output, then it should be possible\n+     * to predict proper settings. There is still another edge case, however, which we cannot\n+     * avoid completely: if the data is heavily skewed towards a hot partition, then it may not\n+     * be possible at all to tweak these settings correctly.\n+     *\n+     * @param size of the dataset\n+     * @param conf MapReduce {@link JobConf} for which to set properties.\n+     * @return\n+     */\n+    private void setNumChunksAndNumReduceTasks(long size, JobConf conf) {\n+        logger.info(\"Data size = \" + size +\n+                \", replication factor = \" + storeDef.getReplicationFactor() +\n+                \", numNodes = \" + cluster.getNumberOfNodes() +\n+                \", numPartitions = \" + cluster.getNumberOfPartitions() +\n+                \", chunk size = \" + chunkSizeBytes);\n+\n+        // Base numbers of chunks and reducers, will get modified according to various settings\n+        int numChunks = (int) (size / cluster.getNumberOfPartitions() / chunkSizeBytes) + 1; /* +1 so we round up */","path":"contrib/hadoop-store-builder/src/java/voldemort/store/readonly/mr/HadoopStoreBuilder.java","position":163,"original_position":163,"commit_id":"110cfbe782f5159c31b65845c5bdbea4b9b51822","original_commit_id":"110cfbe782f5159c31b65845c5bdbea4b9b51822","user":{"login":"mattwisein","id":16215822,"avatar_url":"https://avatars.githubusercontent.com/u/16215822?v=3","gravatar_id":"","url":"https://api.github.com/users/mattwisein","html_url":"https://github.com/mattwisein","followers_url":"https://api.github.com/users/mattwisein/followers","following_url":"https://api.github.com/users/mattwisein/following{/other_user}","gists_url":"https://api.github.com/users/mattwisein/gists{/gist_id}","starred_url":"https://api.github.com/users/mattwisein/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattwisein/subscriptions","organizations_url":"https://api.github.com/users/mattwisein/orgs","repos_url":"https://api.github.com/users/mattwisein/repos","events_url":"https://api.github.com/users/mattwisein/events{/privacy}","received_events_url":"https://api.github.com/users/mattwisein/received_events","type":"User","site_admin":false},"body":"Note that the +1 is added on this line.  The refactoring of this method makes it harder to see that it is a change.","created_at":"2016-08-04T22:12:55Z","updated_at":"2016-08-04T22:12:55Z","html_url":"https://github.com/voldemort/voldemort/pull/434#discussion_r73612114","pull_request_url":"https://api.github.com/repos/voldemort/voldemort/pulls/434","_links":{"self":{"href":"https://api.github.com/repos/voldemort/voldemort/pulls/comments/73612114"},"html":{"href":"https://github.com/voldemort/voldemort/pull/434#discussion_r73612114"},"pull_request":{"href":"https://api.github.com/repos/voldemort/voldemort/pulls/434"}}},"pull_request":{"url":"https://api.github.com/repos/voldemort/voldemort/pulls/434","id":80158619,"html_url":"https://github.com/voldemort/voldemort/pull/434","diff_url":"https://github.com/voldemort/voldemort/pull/434.diff","patch_url":"https://github.com/voldemort/voldemort/pull/434.patch","issue_url":"https://api.github.com/repos/voldemort/voldemort/issues/434","number":434,"state":"open","locked":false,"title":"Provide chunk size suggestion for BnP jobs with chunk overflow except…","user":{"login":"mattwisein","id":16215822,"avatar_url":"https://avatars.githubusercontent.com/u/16215822?v=3","gravatar_id":"","url":"https://api.github.com/users/mattwisein","html_url":"https://github.com/mattwisein","followers_url":"https://api.github.com/users/mattwisein/followers","following_url":"https://api.github.com/users/mattwisein/following{/other_user}","gists_url":"https://api.github.com/users/mattwisein/gists{/gist_id}","starred_url":"https://api.github.com/users/mattwisein/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattwisein/subscriptions","organizations_url":"https://api.github.com/users/mattwisein/orgs","repos_url":"https://api.github.com/users/mattwisein/repos","events_url":"https://api.github.com/users/mattwisein/events{/privacy}","received_events_url":"https://api.github.com/users/mattwisein/received_events","type":"User","site_admin":false},"body":"…ions and fix num chunks algorith to round up\r\n\r\nI'm looking for feedback on this change, I'm still in the process of testing it, so it is not ready to merge.","created_at":"2016-08-04T22:11:46Z","updated_at":"2016-08-04T22:12:55Z","closed_at":null,"merged_at":null,"merge_commit_sha":"97259507620d47e87a20072780d42a05323175aa","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/voldemort/voldemort/pulls/434/commits","review_comments_url":"https://api.github.com/repos/voldemort/voldemort/pulls/434/comments","review_comment_url":"https://api.github.com/repos/voldemort/voldemort/pulls/comments{/number}","comments_url":"https://api.github.com/repos/voldemort/voldemort/issues/434/comments","statuses_url":"https://api.github.com/repos/voldemort/voldemort/statuses/110cfbe782f5159c31b65845c5bdbea4b9b51822","head":{"label":"mattwisein:retry_BnP_MR_job_if_bad_num_chunks","ref":"retry_BnP_MR_job_if_bad_num_chunks","sha":"110cfbe782f5159c31b65845c5bdbea4b9b51822","user":{"login":"mattwisein","id":16215822,"avatar_url":"https://avatars.githubusercontent.com/u/16215822?v=3","gravatar_id":"","url":"https://api.github.com/users/mattwisein","html_url":"https://github.com/mattwisein","followers_url":"https://api.github.com/users/mattwisein/followers","following_url":"https://api.github.com/users/mattwisein/following{/other_user}","gists_url":"https://api.github.com/users/mattwisein/gists{/gist_id}","starred_url":"https://api.github.com/users/mattwisein/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattwisein/subscriptions","organizations_url":"https://api.github.com/users/mattwisein/orgs","repos_url":"https://api.github.com/users/mattwisein/repos","events_url":"https://api.github.com/users/mattwisein/events{/privacy}","received_events_url":"https://api.github.com/users/mattwisein/received_events","type":"User","site_admin":false},"repo":{"id":61653658,"name":"voldemort","full_name":"mattwisein/voldemort","owner":{"login":"mattwisein","id":16215822,"avatar_url":"https://avatars.githubusercontent.com/u/16215822?v=3","gravatar_id":"","url":"https://api.github.com/users/mattwisein","html_url":"https://github.com/mattwisein","followers_url":"https://api.github.com/users/mattwisein/followers","following_url":"https://api.github.com/users/mattwisein/following{/other_user}","gists_url":"https://api.github.com/users/mattwisein/gists{/gist_id}","starred_url":"https://api.github.com/users/mattwisein/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattwisein/subscriptions","organizations_url":"https://api.github.com/users/mattwisein/orgs","repos_url":"https://api.github.com/users/mattwisein/repos","events_url":"https://api.github.com/users/mattwisein/events{/privacy}","received_events_url":"https://api.github.com/users/mattwisein/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mattwisein/voldemort","description":"An open source clone of Amazon's Dynamo.","fork":true,"url":"https://api.github.com/repos/mattwisein/voldemort","forks_url":"https://api.github.com/repos/mattwisein/voldemort/forks","keys_url":"https://api.github.com/repos/mattwisein/voldemort/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mattwisein/voldemort/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mattwisein/voldemort/teams","hooks_url":"https://api.github.com/repos/mattwisein/voldemort/hooks","issue_events_url":"https://api.github.com/repos/mattwisein/voldemort/issues/events{/number}","events_url":"https://api.github.com/repos/mattwisein/voldemort/events","assignees_url":"https://api.github.com/repos/mattwisein/voldemort/assignees{/user}","branches_url":"https://api.github.com/repos/mattwisein/voldemort/branches{/branch}","tags_url":"https://api.github.com/repos/mattwisein/voldemort/tags","blobs_url":"https://api.github.com/repos/mattwisein/voldemort/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mattwisein/voldemort/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mattwisein/voldemort/git/refs{/sha}","trees_url":"https://api.github.com/repos/mattwisein/voldemort/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mattwisein/voldemort/statuses/{sha}","languages_url":"https://api.github.com/repos/mattwisein/voldemort/languages","stargazers_url":"https://api.github.com/repos/mattwisein/voldemort/stargazers","contributors_url":"https://api.github.com/repos/mattwisein/voldemort/contributors","subscribers_url":"https://api.github.com/repos/mattwisein/voldemort/subscribers","subscription_url":"https://api.github.com/repos/mattwisein/voldemort/subscription","commits_url":"https://api.github.com/repos/mattwisein/voldemort/commits{/sha}","git_commits_url":"https://api.github.com/repos/mattwisein/voldemort/git/commits{/sha}","comments_url":"https://api.github.com/repos/mattwisein/voldemort/comments{/number}","issue_comment_url":"https://api.github.com/repos/mattwisein/voldemort/issues/comments{/number}","contents_url":"https://api.github.com/repos/mattwisein/voldemort/contents/{+path}","compare_url":"https://api.github.com/repos/mattwisein/voldemort/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mattwisein/voldemort/merges","archive_url":"https://api.github.com/repos/mattwisein/voldemort/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mattwisein/voldemort/downloads","issues_url":"https://api.github.com/repos/mattwisein/voldemort/issues{/number}","pulls_url":"https://api.github.com/repos/mattwisein/voldemort/pulls{/number}","milestones_url":"https://api.github.com/repos/mattwisein/voldemort/milestones{/number}","notifications_url":"https://api.github.com/repos/mattwisein/voldemort/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mattwisein/voldemort/labels{/name}","releases_url":"https://api.github.com/repos/mattwisein/voldemort/releases{/id}","deployments_url":"https://api.github.com/repos/mattwisein/voldemort/deployments","created_at":"2016-06-21T17:32:08Z","updated_at":"2016-06-21T17:32:13Z","pushed_at":"2016-08-04T20:25:56Z","git_url":"git://github.com/mattwisein/voldemort.git","ssh_url":"git@github.com:mattwisein/voldemort.git","clone_url":"https://github.com/mattwisein/voldemort.git","svn_url":"https://github.com/mattwisein/voldemort","homepage":"http://project-voldemort.com","size":210939,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"voldemort:master","ref":"master","sha":"6e5e96ecf66325fcae3fef9add0862036b08a089","user":{"login":"voldemort","id":90277,"avatar_url":"https://avatars.githubusercontent.com/u/90277?v=3","gravatar_id":"","url":"https://api.github.com/users/voldemort","html_url":"https://github.com/voldemort","followers_url":"https://api.github.com/users/voldemort/followers","following_url":"https://api.github.com/users/voldemort/following{/other_user}","gists_url":"https://api.github.com/users/voldemort/gists{/gist_id}","starred_url":"https://api.github.com/users/voldemort/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/voldemort/subscriptions","organizations_url":"https://api.github.com/users/voldemort/orgs","repos_url":"https://api.github.com/users/voldemort/repos","events_url":"https://api.github.com/users/voldemort/events{/privacy}","received_events_url":"https://api.github.com/users/voldemort/received_events","type":"User","site_admin":false},"repo":{"id":214605,"name":"voldemort","full_name":"voldemort/voldemort","owner":{"login":"voldemort","id":90277,"avatar_url":"https://avatars.githubusercontent.com/u/90277?v=3","gravatar_id":"","url":"https://api.github.com/users/voldemort","html_url":"https://github.com/voldemort","followers_url":"https://api.github.com/users/voldemort/followers","following_url":"https://api.github.com/users/voldemort/following{/other_user}","gists_url":"https://api.github.com/users/voldemort/gists{/gist_id}","starred_url":"https://api.github.com/users/voldemort/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/voldemort/subscriptions","organizations_url":"https://api.github.com/users/voldemort/orgs","repos_url":"https://api.github.com/users/voldemort/repos","events_url":"https://api.github.com/users/voldemort/events{/privacy}","received_events_url":"https://api.github.com/users/voldemort/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/voldemort/voldemort","description":"An open source clone of Amazon's Dynamo.","fork":false,"url":"https://api.github.com/repos/voldemort/voldemort","forks_url":"https://api.github.com/repos/voldemort/voldemort/forks","keys_url":"https://api.github.com/repos/voldemort/voldemort/keys{/key_id}","collaborators_url":"https://api.github.com/repos/voldemort/voldemort/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/voldemort/voldemort/teams","hooks_url":"https://api.github.com/repos/voldemort/voldemort/hooks","issue_events_url":"https://api.github.com/repos/voldemort/voldemort/issues/events{/number}","events_url":"https://api.github.com/repos/voldemort/voldemort/events","assignees_url":"https://api.github.com/repos/voldemort/voldemort/assignees{/user}","branches_url":"https://api.github.com/repos/voldemort/voldemort/branches{/branch}","tags_url":"https://api.github.com/repos/voldemort/voldemort/tags","blobs_url":"https://api.github.com/repos/voldemort/voldemort/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/voldemort/voldemort/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/voldemort/voldemort/git/refs{/sha}","trees_url":"https://api.github.com/repos/voldemort/voldemort/git/trees{/sha}","statuses_url":"https://api.github.com/repos/voldemort/voldemort/statuses/{sha}","languages_url":"https://api.github.com/repos/voldemort/voldemort/languages","stargazers_url":"https://api.github.com/repos/voldemort/voldemort/stargazers","contributors_url":"https://api.github.com/repos/voldemort/voldemort/contributors","subscribers_url":"https://api.github.com/repos/voldemort/voldemort/subscribers","subscription_url":"https://api.github.com/repos/voldemort/voldemort/subscription","commits_url":"https://api.github.com/repos/voldemort/voldemort/commits{/sha}","git_commits_url":"https://api.github.com/repos/voldemort/voldemort/git/commits{/sha}","comments_url":"https://api.github.com/repos/voldemort/voldemort/comments{/number}","issue_comment_url":"https://api.github.com/repos/voldemort/voldemort/issues/comments{/number}","contents_url":"https://api.github.com/repos/voldemort/voldemort/contents/{+path}","compare_url":"https://api.github.com/repos/voldemort/voldemort/compare/{base}...{head}","merges_url":"https://api.github.com/repos/voldemort/voldemort/merges","archive_url":"https://api.github.com/repos/voldemort/voldemort/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/voldemort/voldemort/downloads","issues_url":"https://api.github.com/repos/voldemort/voldemort/issues{/number}","pulls_url":"https://api.github.com/repos/voldemort/voldemort/pulls{/number}","milestones_url":"https://api.github.com/repos/voldemort/voldemort/milestones{/number}","notifications_url":"https://api.github.com/repos/voldemort/voldemort/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/voldemort/voldemort/labels{/name}","releases_url":"https://api.github.com/repos/voldemort/voldemort/releases{/id}","deployments_url":"https://api.github.com/repos/voldemort/voldemort/deployments","created_at":"2009-05-31T03:27:10Z","updated_at":"2016-08-03T17:37:49Z","pushed_at":"2016-08-04T22:11:46Z","git_url":"git://github.com/voldemort/voldemort.git","ssh_url":"git@github.com:voldemort/voldemort.git","clone_url":"https://github.com/voldemort/voldemort.git","svn_url":"https://github.com/voldemort/voldemort","homepage":"http://project-voldemort.com","size":271977,"stargazers_count":1734,"watchers_count":1734,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":490,"mirror_url":null,"open_issues_count":67,"forks":490,"open_issues":67,"watchers":1734,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/voldemort/voldemort/pulls/434"},"html":{"href":"https://github.com/voldemort/voldemort/pull/434"},"issue":{"href":"https://api.github.com/repos/voldemort/voldemort/issues/434"},"comments":{"href":"https://api.github.com/repos/voldemort/voldemort/issues/434/comments"},"review_comments":{"href":"https://api.github.com/repos/voldemort/voldemort/pulls/434/comments"},"review_comment":{"href":"https://api.github.com/repos/voldemort/voldemort/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/voldemort/voldemort/pulls/434/commits"},"statuses":{"href":"https://api.github.com/repos/voldemort/voldemort/statuses/110cfbe782f5159c31b65845c5bdbea4b9b51822"}}}}
{"actor":{"display_login":"mattwisein"}}