{"action":"created","comment":{"url":"https://api.github.com/repos/apache/kafka/pulls/comments/83919358","pull_request_review_id":4730922,"id":83919358,"diff_hunk":"@@ -346,6 +346,127 @@\n \tTopic: foo\tPartition: 0\tLeader: 5\tReplicas: 5,6,7\tIsr: 5,6,7\n </pre>\n \n+<h4><a id=\"rep-throttle\" href=\"#rep-throttle\">Limiting Bandwidth Usage during Data Migration</a></h4>\n+Kafka lets you apply a throttle to replication traffic, setting an upper bound on the bandwidth used to move replicas from machine to machine. This is useful when rebalancing a cluster, bootstrapping a new broker or adding or removing brokers, as it limits the impact these data-intensive operations will have on users.\n+<p></p>\n+There are two interfaces that can be used to engage a throttle. The simplest, and safest, is to apply a throttle when invoking the kafka-reassign-partitions.sh, but kaka-configs.sh can also be used to view and alter the throttle values directly.\n+<p></p>\n+So for example, if you were to execute a rebalance, with the below command, it would move partitions at no more than 50MB/s.\n+<pre>$ bin/kafka-reassign-partitions.sh --zookeeper myhost:2181--execute --reassignment-json-file bigger-cluster.json —throttle 50000000</pre>\n+When you execute this script you will see the throttle engage:\n+<pre>\n+The throttle limit was set to 50000000 B/s\n+Successfully started reassignment of partitions.</pre>\n+<p>Should you wish to alter the throttle, during a rebalance, say to increase the throughput so it completes quicker, you can do this by re-running the execute command passing the same reassignment-json-file:</p>\n+<pre>$ bin/kafka-reassign-partitions.sh --zookeeper localhost:2181  --execute --reassignment-json-file bigger-cluster.json --throttle 700000000\n+There is an existing assignment running.\n+The throttle limit was set to 700000000 B/s</pre>\n+\n+<p>Once the rebalance completes the administrator can check the status of the rebalance using the --verify option.\n+    If the rebalance has completed, the throttle will be removed via the --verify command. It is important that\n+    administrators remove the throttle in a timely manner once rebalancing completes by running the command with\n+    the --verify option. Failure to do so could cause regular replication traffic to be throttled. </p>\n+<p>When the --verify option is executed, and the reassignment has completed, the script will confirm that the throttle was removed:</p>\n+\n+<pre>$ bin/kafka-reassign-partitions.sh --zookeeper localhost:2181  --verify --reassignment-json-file bigger-cluster.json\n+Status of partition reassignment:\n+Reassignment of partition [my-topic,1] completed successfully\n+Reassignment of partition [mytopic,0] completed successfully\n+Throttle was removed.</pre>\n+\n+<p>The administrator can also validate the assigned configs using the kafka-configs.sh. There are two pairs of throttle\n+    configuration used to manage the throttling process. The throttle value itself. This is configured, at a broker\n+    level, using the dynamic properties: </p>\n+\n+<pre>leader.replication.throttled.rate\n+follower.replication.throttled.rate</pre>\n+\n+<p>There is also an enumerated set of throttled replicas: </p>\n+\n+<pre>leader.replication.throttled.replicas\n+follower.replication.throttled.replicas</pre>\n+\n+<p>Which are configured per topic. All four config values are automatically assigned by kafka-reassign-partitions.sh\n+    (discussed below). </p>\n+<p>To view the throttle limit configuration:</p>\n+\n+<pre>$ bin/kafka-configs.sh --describe --zookeeper localhost:2181 --entity-type brokers\n+Configs for brokers '2' are leader.replication.throttled.rate=1000000,follower.replication.throttled.rate=1000000\n+Configs for brokers '1' are leader.replication.throttled.rate=1000000,follower.replication.throttled.rate=1000000</pre>\n+\n+<p>This shows the throttle applied to both leader and follower side of the replication protocol. By default both sides\n+    are assigned the same throttled throughput value. </p>\n+\n+<p>To view the list of throttled replicas:</p>\n+\n+<pre>$ bin/kafka-configs.sh --describe --zookeeper localhost:2181 --entity-type topics\n+Configs for topic 'my-topic' are leader.replication.throttled.replicas=1:102,0:101,\n+    follower.replication.throttled.replicas=1:101,0:102</pre>\n+\n+<p>Here we see the leader throttle is applied to partition 1 on broker 102 and partition 0 on broker 101. Likewise the\n+    follower throttle is applied to partition 1 on\n+    broker 101 and partition 0 on broker 102. </p>\n+\n+<p>By default kaka-reassign-partitions.sh will apply the leader throttle to all replicas that exist before the\n+    rebalance, any one of which might be leader.\n+    It will apply the follower throttle to all move destinations. So if there is a partition with replicas on brokers\n+    101,102, being reassigned to 102,103, a leader throttle,\n+    for that partition, would be applied to 101,102 and a follower throttle would be applied to 103 only. </p>\n+\n+\n+<p>If required, you can also use the --alter switch on kafka-configs.sh to alter the throttle configurations manually.\n+</p>\n+\n+<h5>Safe usage of throttled replication</h5>\n+\n+<p>Some care should be taken when using throttled replication. In particular:</p>\n+\n+<p><i>(1) Throttle Removal:</i></p>\n+The throttle should be removed in a timely manner once reassignment completes (by running kaka-reassign-partitions\n+—verify).\n+\n+<p><i>(2) Ensuring Progress:</i></p>\n+<p>If the throttle is set too low, in comparison to the incoming write rate, it is possible for replication to not\n+    make progress. This occurs when:</p>\n+<pre>max(BytesInPerSec) > throttle</pre>\n+<p>\n+    Where BytesInPerSec is the metric that monitors the write throughput of producers into each broker. </p>\n+<p>The administrator can monitor whether replication is making progress, during the rebalance, using the metric:</p>\n+\n+<pre>kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=([-.\\w]+),topic=([-.\\w]+),partition=([0-9]+)</pre>\n+\n+<p>The lag should constantly decrease during replication.  If the metric does not decrease the administrator should\n+    increase the\n+    throttle throughput as described above. </p>\n+\n+<p><i>(3) Avoiding long delays during replication:</i></p>\n+<p>The throttled throughput should be large enough that replicas cannot be starved for extended periods. A good,\n+    conservative rule of thumb is\n+    to keep throttle above #brokers MB/s where #brokers is the number of brokers in your cluster.  </p>\n+<p>Administrators wishing to use lower throttle values can tune the response size used for replication based on\n+    the relation:</p>\n+<pre></pre>Worst-Case-Delay = replica.fetch.response.max.bytes x #brokers / throttle</pre>\n+\n+<p>Here, the admin should tune the throttle  and/or replica.fetch.response.max.bytes  appropriately to ensure the\n+    delay is never larger than replica.lag.time.max.ms (as it is possible for some partitions, particularly smaller\n+    ones, to enter the ISR before the rebalance completes) or the outer throttle window:","path":"docs/ops.html","position":107,"original_position":107,"commit_id":"becce0b29a3db3d70ecf1ddc45138d3403088de4","original_commit_id":"becce0b29a3db3d70ecf1ddc45138d3403088de4","user":{"login":"junrao","id":132330,"avatar_url":"https://avatars.githubusercontent.com/u/132330?v=3","gravatar_id":"","url":"https://api.github.com/users/junrao","html_url":"https://github.com/junrao","followers_url":"https://api.github.com/users/junrao/followers","following_url":"https://api.github.com/users/junrao/following{/other_user}","gists_url":"https://api.github.com/users/junrao/gists{/gist_id}","starred_url":"https://api.github.com/users/junrao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/junrao/subscriptions","organizations_url":"https://api.github.com/users/junrao/orgs","repos_url":"https://api.github.com/users/junrao/repos","events_url":"https://api.github.com/users/junrao/events{/privacy}","received_events_url":"https://api.github.com/users/junrao/received_events","type":"User","site_admin":false},"body":"outer throttle window? Perhaps full throttle window?","created_at":"2016-10-18T18:25:45Z","updated_at":"2016-10-18T18:25:52Z","html_url":"https://github.com/apache/kafka/pull/2033#discussion_r83919358","pull_request_url":"https://api.github.com/repos/apache/kafka/pulls/2033","_links":{"self":{"href":"https://api.github.com/repos/apache/kafka/pulls/comments/83919358"},"html":{"href":"https://github.com/apache/kafka/pull/2033#discussion_r83919358"},"pull_request":{"href":"https://api.github.com/repos/apache/kafka/pulls/2033"}}},"pull_request":{"url":"https://api.github.com/repos/apache/kafka/pulls/2033","id":89702895,"html_url":"https://github.com/apache/kafka/pull/2033","diff_url":"https://github.com/apache/kafka/pull/2033.diff","patch_url":"https://github.com/apache/kafka/pull/2033.patch","issue_url":"https://api.github.com/repos/apache/kafka/issues/2033","number":2033,"state":"open","locked":false,"title":"Documentation for Throttled Replication","user":{"login":"benstopford","id":1297498,"avatar_url":"https://avatars.githubusercontent.com/u/1297498?v=3","gravatar_id":"","url":"https://api.github.com/users/benstopford","html_url":"https://github.com/benstopford","followers_url":"https://api.github.com/users/benstopford/followers","following_url":"https://api.github.com/users/benstopford/following{/other_user}","gists_url":"https://api.github.com/users/benstopford/gists{/gist_id}","starred_url":"https://api.github.com/users/benstopford/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benstopford/subscriptions","organizations_url":"https://api.github.com/users/benstopford/orgs","repos_url":"https://api.github.com/users/benstopford/repos","events_url":"https://api.github.com/users/benstopford/events{/privacy}","received_events_url":"https://api.github.com/users/benstopford/received_events","type":"User","site_admin":false},"body":"","created_at":"2016-10-17T21:04:45Z","updated_at":"2016-10-18T18:25:52Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6d378ad7c46bed34491fff93dceff7d8356d8b73","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/kafka/pulls/2033/commits","review_comments_url":"https://api.github.com/repos/apache/kafka/pulls/2033/comments","review_comment_url":"https://api.github.com/repos/apache/kafka/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/kafka/issues/2033/comments","statuses_url":"https://api.github.com/repos/apache/kafka/statuses/becce0b29a3db3d70ecf1ddc45138d3403088de4","head":{"label":"benstopford:throttling-docs","ref":"throttling-docs","sha":"becce0b29a3db3d70ecf1ddc45138d3403088de4","user":{"login":"benstopford","id":1297498,"avatar_url":"https://avatars.githubusercontent.com/u/1297498?v=3","gravatar_id":"","url":"https://api.github.com/users/benstopford","html_url":"https://github.com/benstopford","followers_url":"https://api.github.com/users/benstopford/followers","following_url":"https://api.github.com/users/benstopford/following{/other_user}","gists_url":"https://api.github.com/users/benstopford/gists{/gist_id}","starred_url":"https://api.github.com/users/benstopford/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benstopford/subscriptions","organizations_url":"https://api.github.com/users/benstopford/orgs","repos_url":"https://api.github.com/users/benstopford/repos","events_url":"https://api.github.com/users/benstopford/events{/privacy}","received_events_url":"https://api.github.com/users/benstopford/received_events","type":"User","site_admin":false},"repo":{"id":40359697,"name":"kafka","full_name":"benstopford/kafka","owner":{"login":"benstopford","id":1297498,"avatar_url":"https://avatars.githubusercontent.com/u/1297498?v=3","gravatar_id":"","url":"https://api.github.com/users/benstopford","html_url":"https://github.com/benstopford","followers_url":"https://api.github.com/users/benstopford/followers","following_url":"https://api.github.com/users/benstopford/following{/other_user}","gists_url":"https://api.github.com/users/benstopford/gists{/gist_id}","starred_url":"https://api.github.com/users/benstopford/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benstopford/subscriptions","organizations_url":"https://api.github.com/users/benstopford/orgs","repos_url":"https://api.github.com/users/benstopford/repos","events_url":"https://api.github.com/users/benstopford/events{/privacy}","received_events_url":"https://api.github.com/users/benstopford/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/benstopford/kafka","description":"Mirror of Apache Kafka","fork":true,"url":"https://api.github.com/repos/benstopford/kafka","forks_url":"https://api.github.com/repos/benstopford/kafka/forks","keys_url":"https://api.github.com/repos/benstopford/kafka/keys{/key_id}","collaborators_url":"https://api.github.com/repos/benstopford/kafka/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/benstopford/kafka/teams","hooks_url":"https://api.github.com/repos/benstopford/kafka/hooks","issue_events_url":"https://api.github.com/repos/benstopford/kafka/issues/events{/number}","events_url":"https://api.github.com/repos/benstopford/kafka/events","assignees_url":"https://api.github.com/repos/benstopford/kafka/assignees{/user}","branches_url":"https://api.github.com/repos/benstopford/kafka/branches{/branch}","tags_url":"https://api.github.com/repos/benstopford/kafka/tags","blobs_url":"https://api.github.com/repos/benstopford/kafka/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/benstopford/kafka/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/benstopford/kafka/git/refs{/sha}","trees_url":"https://api.github.com/repos/benstopford/kafka/git/trees{/sha}","statuses_url":"https://api.github.com/repos/benstopford/kafka/statuses/{sha}","languages_url":"https://api.github.com/repos/benstopford/kafka/languages","stargazers_url":"https://api.github.com/repos/benstopford/kafka/stargazers","contributors_url":"https://api.github.com/repos/benstopford/kafka/contributors","subscribers_url":"https://api.github.com/repos/benstopford/kafka/subscribers","subscription_url":"https://api.github.com/repos/benstopford/kafka/subscription","commits_url":"https://api.github.com/repos/benstopford/kafka/commits{/sha}","git_commits_url":"https://api.github.com/repos/benstopford/kafka/git/commits{/sha}","comments_url":"https://api.github.com/repos/benstopford/kafka/comments{/number}","issue_comment_url":"https://api.github.com/repos/benstopford/kafka/issues/comments{/number}","contents_url":"https://api.github.com/repos/benstopford/kafka/contents/{+path}","compare_url":"https://api.github.com/repos/benstopford/kafka/compare/{base}...{head}","merges_url":"https://api.github.com/repos/benstopford/kafka/merges","archive_url":"https://api.github.com/repos/benstopford/kafka/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/benstopford/kafka/downloads","issues_url":"https://api.github.com/repos/benstopford/kafka/issues{/number}","pulls_url":"https://api.github.com/repos/benstopford/kafka/pulls{/number}","milestones_url":"https://api.github.com/repos/benstopford/kafka/milestones{/number}","notifications_url":"https://api.github.com/repos/benstopford/kafka/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/benstopford/kafka/labels{/name}","releases_url":"https://api.github.com/repos/benstopford/kafka/releases{/id}","deployments_url":"https://api.github.com/repos/benstopford/kafka/deployments","created_at":"2015-08-07T12:36:48Z","updated_at":"2016-01-25T13:54:01Z","pushed_at":"2016-10-17T21:25:46Z","git_url":"git://github.com/benstopford/kafka.git","ssh_url":"git@github.com:benstopford/kafka.git","clone_url":"https://github.com/benstopford/kafka.git","svn_url":"https://github.com/benstopford/kafka","homepage":null,"size":34690,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"trunk"}},"base":{"label":"apache:trunk","ref":"trunk","sha":"6199c62776bf3ce9467703ca651a0119b261e60e","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":2211243,"name":"kafka","full_name":"apache/kafka","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/kafka","description":"Mirror of Apache Kafka","fork":false,"url":"https://api.github.com/repos/apache/kafka","forks_url":"https://api.github.com/repos/apache/kafka/forks","keys_url":"https://api.github.com/repos/apache/kafka/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/kafka/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/kafka/teams","hooks_url":"https://api.github.com/repos/apache/kafka/hooks","issue_events_url":"https://api.github.com/repos/apache/kafka/issues/events{/number}","events_url":"https://api.github.com/repos/apache/kafka/events","assignees_url":"https://api.github.com/repos/apache/kafka/assignees{/user}","branches_url":"https://api.github.com/repos/apache/kafka/branches{/branch}","tags_url":"https://api.github.com/repos/apache/kafka/tags","blobs_url":"https://api.github.com/repos/apache/kafka/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/kafka/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/kafka/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/kafka/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/kafka/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/kafka/languages","stargazers_url":"https://api.github.com/repos/apache/kafka/stargazers","contributors_url":"https://api.github.com/repos/apache/kafka/contributors","subscribers_url":"https://api.github.com/repos/apache/kafka/subscribers","subscription_url":"https://api.github.com/repos/apache/kafka/subscription","commits_url":"https://api.github.com/repos/apache/kafka/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/kafka/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/kafka/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/kafka/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/kafka/contents/{+path}","compare_url":"https://api.github.com/repos/apache/kafka/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/kafka/merges","archive_url":"https://api.github.com/repos/apache/kafka/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/kafka/downloads","issues_url":"https://api.github.com/repos/apache/kafka/issues{/number}","pulls_url":"https://api.github.com/repos/apache/kafka/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/kafka/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/kafka/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/kafka/labels{/name}","releases_url":"https://api.github.com/repos/apache/kafka/releases{/id}","deployments_url":"https://api.github.com/repos/apache/kafka/deployments","created_at":"2011-08-15T18:06:16Z","updated_at":"2016-10-18T16:21:48Z","pushed_at":"2016-10-18T18:08:36Z","git_url":"git://github.com/apache/kafka.git","ssh_url":"git@github.com:apache/kafka.git","clone_url":"https://github.com/apache/kafka.git","svn_url":"https://github.com/apache/kafka","homepage":null,"size":51304,"stargazers_count":3933,"watchers_count":3933,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2433,"mirror_url":"git://git.apache.org/kafka.git","open_issues_count":279,"forks":2433,"open_issues":279,"watchers":3933,"default_branch":"trunk"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/kafka/pulls/2033"},"html":{"href":"https://github.com/apache/kafka/pull/2033"},"issue":{"href":"https://api.github.com/repos/apache/kafka/issues/2033"},"comments":{"href":"https://api.github.com/repos/apache/kafka/issues/2033/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/kafka/pulls/2033/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/kafka/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/kafka/pulls/2033/commits"},"statuses":{"href":"https://api.github.com/repos/apache/kafka/statuses/becce0b29a3db3d70ecf1ddc45138d3403088de4"}}}}
{"actor":{"display_login":"junrao"}}