{"action":"created","comment":{"url":"https://api.github.com/repos/druid-io/druid/pulls/comments/56218764","id":56218764,"diff_hunk":"@@ -653,27 +654,70 @@ private void stopBeingLeader()\n     if (config.isConvertSegments()) {\n       helpers.add(new DruidCoordinatorVersionConverter(indexingServiceClient, whitelistRef));\n     }\n-    if (config.isMergeSegments()) {\n-      helpers.add(new DruidCoordinatorSegmentMerger(indexingServiceClient, whitelistRef));\n-      helpers.add(\n-          new DruidCoordinatorHelper()\n-          {\n-            @Override\n-            public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)\n-            {\n-              CoordinatorStats stats = params.getCoordinatorStats();\n-              log.info(\"Issued merge requests for %s segments\", stats.getGlobalStats().get(\"mergedCount\").get());\n \n-              params.getEmitter().emit(\n-                  new ServiceMetricEvent.Builder().build(\n-                      \"coordinator/merge/count\", stats.getGlobalStats().get(\"mergedCount\")\n-                  )\n-              );\n+    final String mergeStrategy = config.getMergeStrategy();\n \n-              return params;\n-            }\n-          }\n-      );\n+    if (mergeStrategy != null) {\n+\n+      switch (mergeStrategy) {\n+        case \"append\": {","path":"server/src/main/java/io/druid/server/coordinator/DruidCoordinator.java","position":37,"original_position":37,"commit_id":"b9d62be7a90560d34926a90b78bc1f188d6fcd99","original_commit_id":"b9d62be7a90560d34926a90b78bc1f188d6fcd99","user":{"login":"drcrallen","id":8213081,"avatar_url":"https://avatars.githubusercontent.com/u/8213081?v=3","gravatar_id":"","url":"https://api.github.com/users/drcrallen","html_url":"https://github.com/drcrallen","followers_url":"https://api.github.com/users/drcrallen/followers","following_url":"https://api.github.com/users/drcrallen/following{/other_user}","gists_url":"https://api.github.com/users/drcrallen/gists{/gist_id}","starred_url":"https://api.github.com/users/drcrallen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drcrallen/subscriptions","organizations_url":"https://api.github.com/users/drcrallen/orgs","repos_url":"https://api.github.com/users/drcrallen/repos","events_url":"https://api.github.com/users/drcrallen/events{/privacy}","received_events_url":"https://api.github.com/users/drcrallen/received_events","type":"User","site_admin":false},"body":"Most other places have this kind of logic handled through jackson type bindings. Can that approach be done here?","created_at":"2016-03-15T18:44:36Z","updated_at":"2016-03-15T18:44:36Z","html_url":"https://github.com/druid-io/druid/pull/1998#discussion_r56218764","pull_request_url":"https://api.github.com/repos/druid-io/druid/pulls/1998","_links":{"self":{"href":"https://api.github.com/repos/druid-io/druid/pulls/comments/56218764"},"html":{"href":"https://github.com/druid-io/druid/pull/1998#discussion_r56218764"},"pull_request":{"href":"https://api.github.com/repos/druid-io/druid/pulls/1998"}}},"pull_request":{"url":"https://api.github.com/repos/druid-io/druid/pulls/1998","id":51412074,"html_url":"https://github.com/druid-io/druid/pull/1998","diff_url":"https://github.com/druid-io/druid/pull/1998.diff","patch_url":"https://github.com/druid-io/druid/pull/1998.patch","issue_url":"https://api.github.com/repos/druid-io/druid/issues/1998","number":1998,"state":"open","locked":false,"title":"Hadoop reindex task from Coordinator","user":{"login":"guobingkun","id":11789051,"avatar_url":"https://avatars.githubusercontent.com/u/11789051?v=3","gravatar_id":"","url":"https://api.github.com/users/guobingkun","html_url":"https://github.com/guobingkun","followers_url":"https://api.github.com/users/guobingkun/followers","following_url":"https://api.github.com/users/guobingkun/following{/other_user}","gists_url":"https://api.github.com/users/guobingkun/gists{/gist_id}","starred_url":"https://api.github.com/users/guobingkun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guobingkun/subscriptions","organizations_url":"https://api.github.com/users/guobingkun/orgs","repos_url":"https://api.github.com/users/guobingkun/repos","events_url":"https://api.github.com/users/guobingkun/events{/privacy}","received_events_url":"https://api.github.com/users/guobingkun/received_events","type":"User","site_admin":false},"body":"Fixes https://github.com/druid-io/druid/issues/1517\r\n\r\nDeprecates `druid.coordinator.merge.on`. Instead, user will use `druid.coordinator.merge.strategy` for segment merging.\r\n\r\nDepends on  ~~#1988~~\r\n\r\nIn this PR,\r\n ```DruidCoordinatorHadoopSegmentMerger``` will find small segments in the segment timeline, and submit HadoopIndexTask to reindex those imbalanced segments.\r\n\r\nFor example,\r\nHere is a timeline of segments for a specific dataSource (Note: The segment granularity for each segment could be arbitrary, there might also be a gap between two segments)\r\n```\r\nSegID:    1   2   3   4   5   6   7   8   9  10  11  12   13\r\nShard0: |_O_|_O_|_O_|_O_|_S_|_S_|_S_|_O_|_S_|_O_|_S_|_O_|_S_|__|__|__|__|__|__|__|\r\nShard1:         |_S_|                   |_S_|   |_S_|_S_|\r\nShard2:         |_S_|                   |_S_|   |_S_|\r\n\r\nS: the size of shard is below configured optimal size\r\nO: the size of shard >= configured optimal size\r\n```\r\nAs ```DruidCoordinatorHadoopSegmentMerger``` runs periodically, eventually there will be no shards in the timeline that are tagged with S.\r\n\r\nHere is the algorithm ```DruidCoordinatorHadoopSegmentMerger``` implements for finding imbalanced segments and submits the reindex task,\r\n\r\n```\r\nunbalancedIntervals = []\r\ncurrTotalSize = 0\r\nintervalToReindex = null\r\nshouldBeMerged = false  // whether we should reindex segments within intervalToReindex\r\n\r\n// the direction of scanning will alternate if we don't want to merge segments\r\n// that have gaps between them\r\nfor each segment \r\n  \r\n  expand intervalToReindex with the interval of current segment\r\n\r\n  for each shard of current segment\r\n    if shard.getSize() < optimal size\r\n      shouldBeMerged = true // found a small segment\r\n    currTotalSize += shard.getSize()\r\n\r\n  // this is an estimate that whether we can get a good-size segment by\r\n  // reindexing the segments within intervalToReindex\r\n  if currTotalSize >= optimal size\r\n    if shouldBeMerged \r\n      unbalancedIntervals.add(intervalToReindex)\r\n    currTotalSize = 0\r\n    intervalToReindex = null\r\n    shouldBeMerged = false\r\n  else {\r\n    // do nothing. Since we don't have enough total size to make a segment \r\n    // with size >= optimal size, we'll seek to merge it with the next segment in the timeline\r\n  }\r\n\r\nif unbalancedIntervals is not empty:\r\n  submit HadoopIndexTask with unbalancedIntervals \r\n\r\nif we want to keep the segment gap during merge\r\n  invert the scan direction so that in next Coordinator's run, we will scan the segments in the inverted direction.\r\nNote: The reason we need to invert the scan direction when keepGap is enabled \r\nis because it's possible to have a small segment that is at the boundary of gap left over\r\n and never gets merged.\r\n\r\n```\r\n\r\nBasically, the algorithm iterates each segment in the timeline, and for each shard in that segment, we check if its size is below optimal size, if yes, we should definitely try to merge that shard with others. However, it's possible that the total size of shards in that segment is not big enough to make an optimal size segment, so we should seek to merge the current segment with the next one in the timeline.\r\nNote: If we want to keep the gap between segments that don't abut, it's possible to have small segments left over, however, we can invert the scan direction in the next Coordinator's run to eliminate those small segments.\r\n\r\n~~After #1728, #1768 are merged, ```DruidCoordinatorHadoopSegmentMerger``` should be able to get Aggregator information by sending SegmentMetadataQuery. Currently, users have to provide aggregation information through runtime properties.~~","created_at":"2015-11-20T21:42:58Z","updated_at":"2016-03-15T18:44:36Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b38fead0a48d3027bd399ee2585509336fc3c372","assignee":null,"milestone":{"url":"https://api.github.com/repos/druid-io/druid/milestones/18","html_url":"https://github.com/druid-io/druid/milestones/0.9.1","labels_url":"https://api.github.com/repos/druid-io/druid/milestones/18/labels","id":1526819,"number":18,"title":"0.9.1","description":"","creator":{"login":"fjy","id":428325,"avatar_url":"https://avatars.githubusercontent.com/u/428325?v=3","gravatar_id":"","url":"https://api.github.com/users/fjy","html_url":"https://github.com/fjy","followers_url":"https://api.github.com/users/fjy/followers","following_url":"https://api.github.com/users/fjy/following{/other_user}","gists_url":"https://api.github.com/users/fjy/gists{/gist_id}","starred_url":"https://api.github.com/users/fjy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fjy/subscriptions","organizations_url":"https://api.github.com/users/fjy/orgs","repos_url":"https://api.github.com/users/fjy/repos","events_url":"https://api.github.com/users/fjy/events{/privacy}","received_events_url":"https://api.github.com/users/fjy/received_events","type":"User","site_admin":false},"open_issues":32,"closed_issues":70,"state":"open","created_at":"2016-01-20T19:04:22Z","updated_at":"2016-03-15T18:12:02Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/druid-io/druid/pulls/1998/commits","review_comments_url":"https://api.github.com/repos/druid-io/druid/pulls/1998/comments","review_comment_url":"https://api.github.com/repos/druid-io/druid/pulls/comments{/number}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/1998/comments","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/b9d62be7a90560d34926a90b78bc1f188d6fcd99","head":{"label":"guobingkun:hadoop_reindex_task","ref":"hadoop_reindex_task","sha":"b9d62be7a90560d34926a90b78bc1f188d6fcd99","user":{"login":"guobingkun","id":11789051,"avatar_url":"https://avatars.githubusercontent.com/u/11789051?v=3","gravatar_id":"","url":"https://api.github.com/users/guobingkun","html_url":"https://github.com/guobingkun","followers_url":"https://api.github.com/users/guobingkun/followers","following_url":"https://api.github.com/users/guobingkun/following{/other_user}","gists_url":"https://api.github.com/users/guobingkun/gists{/gist_id}","starred_url":"https://api.github.com/users/guobingkun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guobingkun/subscriptions","organizations_url":"https://api.github.com/users/guobingkun/orgs","repos_url":"https://api.github.com/users/guobingkun/repos","events_url":"https://api.github.com/users/guobingkun/events{/privacy}","received_events_url":"https://api.github.com/users/guobingkun/received_events","type":"User","site_admin":false},"repo":{"id":33376574,"name":"druid","full_name":"guobingkun/druid","owner":{"login":"guobingkun","id":11789051,"avatar_url":"https://avatars.githubusercontent.com/u/11789051?v=3","gravatar_id":"","url":"https://api.github.com/users/guobingkun","html_url":"https://github.com/guobingkun","followers_url":"https://api.github.com/users/guobingkun/followers","following_url":"https://api.github.com/users/guobingkun/following{/other_user}","gists_url":"https://api.github.com/users/guobingkun/gists{/gist_id}","starred_url":"https://api.github.com/users/guobingkun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guobingkun/subscriptions","organizations_url":"https://api.github.com/users/guobingkun/orgs","repos_url":"https://api.github.com/users/guobingkun/repos","events_url":"https://api.github.com/users/guobingkun/events{/privacy}","received_events_url":"https://api.github.com/users/guobingkun/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/guobingkun/druid","description":"Real²time Exploratory Analytics on Large Datasets","fork":true,"url":"https://api.github.com/repos/guobingkun/druid","forks_url":"https://api.github.com/repos/guobingkun/druid/forks","keys_url":"https://api.github.com/repos/guobingkun/druid/keys{/key_id}","collaborators_url":"https://api.github.com/repos/guobingkun/druid/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/guobingkun/druid/teams","hooks_url":"https://api.github.com/repos/guobingkun/druid/hooks","issue_events_url":"https://api.github.com/repos/guobingkun/druid/issues/events{/number}","events_url":"https://api.github.com/repos/guobingkun/druid/events","assignees_url":"https://api.github.com/repos/guobingkun/druid/assignees{/user}","branches_url":"https://api.github.com/repos/guobingkun/druid/branches{/branch}","tags_url":"https://api.github.com/repos/guobingkun/druid/tags","blobs_url":"https://api.github.com/repos/guobingkun/druid/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/guobingkun/druid/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/guobingkun/druid/git/refs{/sha}","trees_url":"https://api.github.com/repos/guobingkun/druid/git/trees{/sha}","statuses_url":"https://api.github.com/repos/guobingkun/druid/statuses/{sha}","languages_url":"https://api.github.com/repos/guobingkun/druid/languages","stargazers_url":"https://api.github.com/repos/guobingkun/druid/stargazers","contributors_url":"https://api.github.com/repos/guobingkun/druid/contributors","subscribers_url":"https://api.github.com/repos/guobingkun/druid/subscribers","subscription_url":"https://api.github.com/repos/guobingkun/druid/subscription","commits_url":"https://api.github.com/repos/guobingkun/druid/commits{/sha}","git_commits_url":"https://api.github.com/repos/guobingkun/druid/git/commits{/sha}","comments_url":"https://api.github.com/repos/guobingkun/druid/comments{/number}","issue_comment_url":"https://api.github.com/repos/guobingkun/druid/issues/comments{/number}","contents_url":"https://api.github.com/repos/guobingkun/druid/contents/{+path}","compare_url":"https://api.github.com/repos/guobingkun/druid/compare/{base}...{head}","merges_url":"https://api.github.com/repos/guobingkun/druid/merges","archive_url":"https://api.github.com/repos/guobingkun/druid/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/guobingkun/druid/downloads","issues_url":"https://api.github.com/repos/guobingkun/druid/issues{/number}","pulls_url":"https://api.github.com/repos/guobingkun/druid/pulls{/number}","milestones_url":"https://api.github.com/repos/guobingkun/druid/milestones{/number}","notifications_url":"https://api.github.com/repos/guobingkun/druid/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/guobingkun/druid/labels{/name}","releases_url":"https://api.github.com/repos/guobingkun/druid/releases{/id}","deployments_url":"https://api.github.com/repos/guobingkun/druid/deployments","created_at":"2015-04-03T18:24:06Z","updated_at":"2016-01-11T21:33:50Z","pushed_at":"2016-03-15T04:27:57Z","git_url":"git://github.com/guobingkun/druid.git","ssh_url":"git@github.com:guobingkun/druid.git","clone_url":"https://github.com/guobingkun/druid.git","svn_url":"https://github.com/guobingkun/druid","homepage":"http://druid.io","size":105565,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"druid-io:master","ref":"master","sha":"ec949d76e36d5150d84b5d4b4cabe6843bd186dd","user":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars.githubusercontent.com/u/4203474?v=3","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"repo":{"id":6358188,"name":"druid","full_name":"druid-io/druid","owner":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars.githubusercontent.com/u/4203474?v=3","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/druid-io/druid","description":"Column oriented distributed data store ideal for powering interactive applications","fork":false,"url":"https://api.github.com/repos/druid-io/druid","forks_url":"https://api.github.com/repos/druid-io/druid/forks","keys_url":"https://api.github.com/repos/druid-io/druid/keys{/key_id}","collaborators_url":"https://api.github.com/repos/druid-io/druid/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/druid-io/druid/teams","hooks_url":"https://api.github.com/repos/druid-io/druid/hooks","issue_events_url":"https://api.github.com/repos/druid-io/druid/issues/events{/number}","events_url":"https://api.github.com/repos/druid-io/druid/events","assignees_url":"https://api.github.com/repos/druid-io/druid/assignees{/user}","branches_url":"https://api.github.com/repos/druid-io/druid/branches{/branch}","tags_url":"https://api.github.com/repos/druid-io/druid/tags","blobs_url":"https://api.github.com/repos/druid-io/druid/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/druid-io/druid/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/druid-io/druid/git/refs{/sha}","trees_url":"https://api.github.com/repos/druid-io/druid/git/trees{/sha}","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/{sha}","languages_url":"https://api.github.com/repos/druid-io/druid/languages","stargazers_url":"https://api.github.com/repos/druid-io/druid/stargazers","contributors_url":"https://api.github.com/repos/druid-io/druid/contributors","subscribers_url":"https://api.github.com/repos/druid-io/druid/subscribers","subscription_url":"https://api.github.com/repos/druid-io/druid/subscription","commits_url":"https://api.github.com/repos/druid-io/druid/commits{/sha}","git_commits_url":"https://api.github.com/repos/druid-io/druid/git/commits{/sha}","comments_url":"https://api.github.com/repos/druid-io/druid/comments{/number}","issue_comment_url":"https://api.github.com/repos/druid-io/druid/issues/comments{/number}","contents_url":"https://api.github.com/repos/druid-io/druid/contents/{+path}","compare_url":"https://api.github.com/repos/druid-io/druid/compare/{base}...{head}","merges_url":"https://api.github.com/repos/druid-io/druid/merges","archive_url":"https://api.github.com/repos/druid-io/druid/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/druid-io/druid/downloads","issues_url":"https://api.github.com/repos/druid-io/druid/issues{/number}","pulls_url":"https://api.github.com/repos/druid-io/druid/pulls{/number}","milestones_url":"https://api.github.com/repos/druid-io/druid/milestones{/number}","notifications_url":"https://api.github.com/repos/druid-io/druid/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/druid-io/druid/labels{/name}","releases_url":"https://api.github.com/repos/druid-io/druid/releases{/id}","deployments_url":"https://api.github.com/repos/druid-io/druid/deployments","created_at":"2012-10-23T19:08:07Z","updated_at":"2016-03-15T16:39:57Z","pushed_at":"2016-03-15T18:25:57Z","git_url":"git://github.com/druid-io/druid.git","ssh_url":"git@github.com:druid-io/druid.git","clone_url":"https://github.com/druid-io/druid.git","svn_url":"https://github.com/druid-io/druid","homepage":"http://druid.io","size":127507,"stargazers_count":2543,"watchers_count":2543,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":608,"mirror_url":null,"open_issues_count":362,"forks":608,"open_issues":362,"watchers":2543,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/druid-io/druid/pulls/1998"},"html":{"href":"https://github.com/druid-io/druid/pull/1998"},"issue":{"href":"https://api.github.com/repos/druid-io/druid/issues/1998"},"comments":{"href":"https://api.github.com/repos/druid-io/druid/issues/1998/comments"},"review_comments":{"href":"https://api.github.com/repos/druid-io/druid/pulls/1998/comments"},"review_comment":{"href":"https://api.github.com/repos/druid-io/druid/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/druid-io/druid/pulls/1998/commits"},"statuses":{"href":"https://api.github.com/repos/druid-io/druid/statuses/b9d62be7a90560d34926a90b78bc1f188d6fcd99"}}}}