{"action":"created","comment":{"url":"https://api.github.com/repos/apache/kafka/pulls/comments/237355006","pull_request_review_id":179630938,"id":237355006,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIzNzM1NTAwNg==","diff_hunk":"@@ -637,7 +640,11 @@ class KafkaController(val config: KafkaConfig, zkClient: KafkaZkClient, time: Ti\n \n   private def initializeControllerContext() {\n     // update controller cache with delete topic information\n-    controllerContext.liveBrokers = zkClient.getAllBrokersInCluster.toSet\n+    val curBrokerAndEpochs = zkClient.getAllBrokerAndEpochsInCluster\n+    controllerContext.liveBrokers = curBrokerAndEpochs.map(_._1).toSet\n+    // update broker epoch cache for all live brokers\n+    curBrokerAndEpochs.foreach(e => controllerContext.brokerEpochs(e._1.id) = e._2)\n+    info(s\"Initialized broker epochs cache: ${controllerContext.brokerEpochs}\")","path":"core/src/main/scala/kafka/controller/KafkaController.scala","position":null,"original_position":56,"commit_id":"0479f14a1f60098b8a748f4652676f337820ac1e","original_commit_id":"88e4eb606fc72761a17378ee6bdf6732765808ef","user":{"login":"hzxa21","id":5518566,"node_id":"MDQ6VXNlcjU1MTg1NjY=","avatar_url":"https://avatars1.githubusercontent.com/u/5518566?v=4","gravatar_id":"","url":"https://api.github.com/users/hzxa21","html_url":"https://github.com/hzxa21","followers_url":"https://api.github.com/users/hzxa21/followers","following_url":"https://api.github.com/users/hzxa21/following{/other_user}","gists_url":"https://api.github.com/users/hzxa21/gists{/gist_id}","starred_url":"https://api.github.com/users/hzxa21/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hzxa21/subscriptions","organizations_url":"https://api.github.com/users/hzxa21/orgs","repos_url":"https://api.github.com/users/hzxa21/repos","events_url":"https://api.github.com/users/hzxa21/events{/privacy}","received_events_url":"https://api.github.com/users/hzxa21/received_events","type":"User","site_admin":false},"body":"I think it is better to keep it info because the broker epoch information is as informative as the broker ids. We als log the broker ids in info log so we should keep it consistent.","created_at":"2018-11-29T05:12:19Z","updated_at":"2018-11-29T05:12:19Z","html_url":"https://github.com/apache/kafka/pull/5821#discussion_r237355006","pull_request_url":"https://api.github.com/repos/apache/kafka/pulls/5821","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/apache/kafka/pulls/comments/237355006"},"html":{"href":"https://github.com/apache/kafka/pull/5821#discussion_r237355006"},"pull_request":{"href":"https://api.github.com/repos/apache/kafka/pulls/5821"}},"in_reply_to_id":234024398},"pull_request":{"url":"https://api.github.com/repos/apache/kafka/pulls/5821","id":224458749,"node_id":"MDExOlB1bGxSZXF1ZXN0MjI0NDU4NzQ5","html_url":"https://github.com/apache/kafka/pull/5821","diff_url":"https://github.com/apache/kafka/pull/5821.diff","patch_url":"https://github.com/apache/kafka/pull/5821.patch","issue_url":"https://api.github.com/repos/apache/kafka/issues/5821","number":5821,"state":"open","locked":false,"title":"KAFKA-7235: Detect outdated control requests and bounced brokers using broker generation","user":{"login":"hzxa21","id":5518566,"node_id":"MDQ6VXNlcjU1MTg1NjY=","avatar_url":"https://avatars1.githubusercontent.com/u/5518566?v=4","gravatar_id":"","url":"https://api.github.com/users/hzxa21","html_url":"https://github.com/hzxa21","followers_url":"https://api.github.com/users/hzxa21/followers","following_url":"https://api.github.com/users/hzxa21/following{/other_user}","gists_url":"https://api.github.com/users/hzxa21/gists{/gist_id}","starred_url":"https://api.github.com/users/hzxa21/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hzxa21/subscriptions","organizations_url":"https://api.github.com/users/hzxa21/orgs","repos_url":"https://api.github.com/users/hzxa21/repos","events_url":"https://api.github.com/users/hzxa21/events{/privacy}","received_events_url":"https://api.github.com/users/hzxa21/received_events","type":"User","site_admin":false},"body":"This PR introduces the broker generation concept and leverage it to allow controller to detect fast bounced brokers and allow broker to reject outdated control requests.\r\n\r\nIt has the changes required to implement KIP-380:\r\n\r\n[Common]\r\n- Refactor ZookeeperClient to expose the zookeeper `multi` request directly\r\n- Refactor KafkaZkClient to use MultiRequest instead of raw zookeeper transaciton\r\n- Atomically get creation transaction id (czxid) with broker znode creation and use it as broker epoch to identify broker gerneration across bounces\r\n- Introduce LeaderAndIsrRequest V2, UpdateMetadataRequest V5 and StopReplicaRequest V1 to include broker epoch in control requests and normalize their schemas to make it more memory efficient\r\n- Add STALE_BROKER_EPOCH error\r\n\r\n[Broker]\r\n- Cache the current broker epoch after broker znode registration\r\n- Reject LeaderAndIsrRequest, UpdateMetadataRequest and StopReplicaRequest if the request's broker epoch < current broker epoch, and respond back with STALE_BROKER_EPOCH error\r\n\r\n[Controller]\r\n- Cache/update broker epochs in `controllerContext.brokerEpochsCache` after reading from zk when processing `BrokerChange` event and `onControllerFailover`\r\n- Detect bounced brokers in `BrokerChange` event by comparing the broker epochs get from zk and cached broker epochs and trigger necessary state changes\r\n- Avoid sending out requests to dead brokers\r\n\r\n[Test]\r\n- Add `BrokerEpochIntegrationTest` to test broker processing new versions of the control requests and rejecting requests with stale broker epoch\r\n- Add a test case in `ControllerIntegrationTest` to test controller detecting bounced brokers\r\n- Add test cases in `RequestResponseTest` to test seralization and de-seralization for new versions of the control requests\r\n- Add `ControlRequstTest` unit test to test control requests schemas normalization\r\n\r\n### Committer Checklist (excluded from commit message)\r\n- [ ] Verify design and implementation \r\n- [ ] Verify test coverage and CI build status\r\n- [ ] Verify documentation (including upgrade notes)\r\n","created_at":"2018-10-20T07:38:39Z","updated_at":"2018-11-29T05:12:19Z","closed_at":null,"merged_at":null,"merge_commit_sha":"8f1a5a0e7785ea4e020ad9d6d32c58e3b1fdd8e6","assignee":{"login":"lindong28","id":5873403,"node_id":"MDQ6VXNlcjU4NzM0MDM=","avatar_url":"https://avatars0.githubusercontent.com/u/5873403?v=4","gravatar_id":"","url":"https://api.github.com/users/lindong28","html_url":"https://github.com/lindong28","followers_url":"https://api.github.com/users/lindong28/followers","following_url":"https://api.github.com/users/lindong28/following{/other_user}","gists_url":"https://api.github.com/users/lindong28/gists{/gist_id}","starred_url":"https://api.github.com/users/lindong28/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lindong28/subscriptions","organizations_url":"https://api.github.com/users/lindong28/orgs","repos_url":"https://api.github.com/users/lindong28/repos","events_url":"https://api.github.com/users/lindong28/events{/privacy}","received_events_url":"https://api.github.com/users/lindong28/received_events","type":"User","site_admin":false},"assignees":[{"login":"lindong28","id":5873403,"node_id":"MDQ6VXNlcjU4NzM0MDM=","avatar_url":"https://avatars0.githubusercontent.com/u/5873403?v=4","gravatar_id":"","url":"https://api.github.com/users/lindong28","html_url":"https://github.com/lindong28","followers_url":"https://api.github.com/users/lindong28/followers","following_url":"https://api.github.com/users/lindong28/following{/other_user}","gists_url":"https://api.github.com/users/lindong28/gists{/gist_id}","starred_url":"https://api.github.com/users/lindong28/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lindong28/subscriptions","organizations_url":"https://api.github.com/users/lindong28/orgs","repos_url":"https://api.github.com/users/lindong28/repos","events_url":"https://api.github.com/users/lindong28/events{/privacy}","received_events_url":"https://api.github.com/users/lindong28/received_events","type":"User","site_admin":false}],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/kafka/pulls/5821/commits","review_comments_url":"https://api.github.com/repos/apache/kafka/pulls/5821/comments","review_comment_url":"https://api.github.com/repos/apache/kafka/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/kafka/issues/5821/comments","statuses_url":"https://api.github.com/repos/apache/kafka/statuses/0479f14a1f60098b8a748f4652676f337820ac1e","head":{"label":"hzxa21:KIP-380-PR","ref":"KIP-380-PR","sha":"0479f14a1f60098b8a748f4652676f337820ac1e","user":{"login":"hzxa21","id":5518566,"node_id":"MDQ6VXNlcjU1MTg1NjY=","avatar_url":"https://avatars1.githubusercontent.com/u/5518566?v=4","gravatar_id":"","url":"https://api.github.com/users/hzxa21","html_url":"https://github.com/hzxa21","followers_url":"https://api.github.com/users/hzxa21/followers","following_url":"https://api.github.com/users/hzxa21/following{/other_user}","gists_url":"https://api.github.com/users/hzxa21/gists{/gist_id}","starred_url":"https://api.github.com/users/hzxa21/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hzxa21/subscriptions","organizations_url":"https://api.github.com/users/hzxa21/orgs","repos_url":"https://api.github.com/users/hzxa21/repos","events_url":"https://api.github.com/users/hzxa21/events{/privacy}","received_events_url":"https://api.github.com/users/hzxa21/received_events","type":"User","site_admin":false},"repo":{"id":131780470,"node_id":"MDEwOlJlcG9zaXRvcnkxMzE3ODA0NzA=","name":"kafka","full_name":"hzxa21/kafka","private":false,"owner":{"login":"hzxa21","id":5518566,"node_id":"MDQ6VXNlcjU1MTg1NjY=","avatar_url":"https://avatars1.githubusercontent.com/u/5518566?v=4","gravatar_id":"","url":"https://api.github.com/users/hzxa21","html_url":"https://github.com/hzxa21","followers_url":"https://api.github.com/users/hzxa21/followers","following_url":"https://api.github.com/users/hzxa21/following{/other_user}","gists_url":"https://api.github.com/users/hzxa21/gists{/gist_id}","starred_url":"https://api.github.com/users/hzxa21/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hzxa21/subscriptions","organizations_url":"https://api.github.com/users/hzxa21/orgs","repos_url":"https://api.github.com/users/hzxa21/repos","events_url":"https://api.github.com/users/hzxa21/events{/privacy}","received_events_url":"https://api.github.com/users/hzxa21/received_events","type":"User","site_admin":false},"html_url":"https://github.com/hzxa21/kafka","description":"Mirror of Apache Kafka","fork":true,"url":"https://api.github.com/repos/hzxa21/kafka","forks_url":"https://api.github.com/repos/hzxa21/kafka/forks","keys_url":"https://api.github.com/repos/hzxa21/kafka/keys{/key_id}","collaborators_url":"https://api.github.com/repos/hzxa21/kafka/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/hzxa21/kafka/teams","hooks_url":"https://api.github.com/repos/hzxa21/kafka/hooks","issue_events_url":"https://api.github.com/repos/hzxa21/kafka/issues/events{/number}","events_url":"https://api.github.com/repos/hzxa21/kafka/events","assignees_url":"https://api.github.com/repos/hzxa21/kafka/assignees{/user}","branches_url":"https://api.github.com/repos/hzxa21/kafka/branches{/branch}","tags_url":"https://api.github.com/repos/hzxa21/kafka/tags","blobs_url":"https://api.github.com/repos/hzxa21/kafka/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/hzxa21/kafka/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/hzxa21/kafka/git/refs{/sha}","trees_url":"https://api.github.com/repos/hzxa21/kafka/git/trees{/sha}","statuses_url":"https://api.github.com/repos/hzxa21/kafka/statuses/{sha}","languages_url":"https://api.github.com/repos/hzxa21/kafka/languages","stargazers_url":"https://api.github.com/repos/hzxa21/kafka/stargazers","contributors_url":"https://api.github.com/repos/hzxa21/kafka/contributors","subscribers_url":"https://api.github.com/repos/hzxa21/kafka/subscribers","subscription_url":"https://api.github.com/repos/hzxa21/kafka/subscription","commits_url":"https://api.github.com/repos/hzxa21/kafka/commits{/sha}","git_commits_url":"https://api.github.com/repos/hzxa21/kafka/git/commits{/sha}","comments_url":"https://api.github.com/repos/hzxa21/kafka/comments{/number}","issue_comment_url":"https://api.github.com/repos/hzxa21/kafka/issues/comments{/number}","contents_url":"https://api.github.com/repos/hzxa21/kafka/contents/{+path}","compare_url":"https://api.github.com/repos/hzxa21/kafka/compare/{base}...{head}","merges_url":"https://api.github.com/repos/hzxa21/kafka/merges","archive_url":"https://api.github.com/repos/hzxa21/kafka/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/hzxa21/kafka/downloads","issues_url":"https://api.github.com/repos/hzxa21/kafka/issues{/number}","pulls_url":"https://api.github.com/repos/hzxa21/kafka/pulls{/number}","milestones_url":"https://api.github.com/repos/hzxa21/kafka/milestones{/number}","notifications_url":"https://api.github.com/repos/hzxa21/kafka/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/hzxa21/kafka/labels{/name}","releases_url":"https://api.github.com/repos/hzxa21/kafka/releases{/id}","deployments_url":"https://api.github.com/repos/hzxa21/kafka/deployments","created_at":"2018-05-02T01:11:44Z","updated_at":"2018-06-28T00:44:30Z","pushed_at":"2018-11-29T02:37:07Z","git_url":"git://github.com/hzxa21/kafka.git","ssh_url":"git@github.com:hzxa21/kafka.git","clone_url":"https://github.com/hzxa21/kafka.git","svn_url":"https://github.com/hzxa21/kafka","homepage":null,"size":74989,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"trunk"}},"base":{"label":"apache:trunk","ref":"trunk","sha":"f52de5f0adf1d3d3557aa38421d9212a1f768b19","user":{"login":"apache","id":47359,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ3MzU5","avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":2211243,"node_id":"MDEwOlJlcG9zaXRvcnkyMjExMjQz","name":"kafka","full_name":"apache/kafka","private":false,"owner":{"login":"apache","id":47359,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ3MzU5","avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/apache/kafka","description":"Mirror of Apache Kafka","fork":false,"url":"https://api.github.com/repos/apache/kafka","forks_url":"https://api.github.com/repos/apache/kafka/forks","keys_url":"https://api.github.com/repos/apache/kafka/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/kafka/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/kafka/teams","hooks_url":"https://api.github.com/repos/apache/kafka/hooks","issue_events_url":"https://api.github.com/repos/apache/kafka/issues/events{/number}","events_url":"https://api.github.com/repos/apache/kafka/events","assignees_url":"https://api.github.com/repos/apache/kafka/assignees{/user}","branches_url":"https://api.github.com/repos/apache/kafka/branches{/branch}","tags_url":"https://api.github.com/repos/apache/kafka/tags","blobs_url":"https://api.github.com/repos/apache/kafka/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/kafka/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/kafka/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/kafka/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/kafka/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/kafka/languages","stargazers_url":"https://api.github.com/repos/apache/kafka/stargazers","contributors_url":"https://api.github.com/repos/apache/kafka/contributors","subscribers_url":"https://api.github.com/repos/apache/kafka/subscribers","subscription_url":"https://api.github.com/repos/apache/kafka/subscription","commits_url":"https://api.github.com/repos/apache/kafka/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/kafka/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/kafka/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/kafka/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/kafka/contents/{+path}","compare_url":"https://api.github.com/repos/apache/kafka/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/kafka/merges","archive_url":"https://api.github.com/repos/apache/kafka/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/kafka/downloads","issues_url":"https://api.github.com/repos/apache/kafka/issues{/number}","pulls_url":"https://api.github.com/repos/apache/kafka/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/kafka/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/kafka/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/kafka/labels{/name}","releases_url":"https://api.github.com/repos/apache/kafka/releases{/id}","deployments_url":"https://api.github.com/repos/apache/kafka/deployments","created_at":"2011-08-15T18:06:16Z","updated_at":"2018-11-29T02:55:39Z","pushed_at":"2018-11-29T02:56:52Z","git_url":"git://github.com/apache/kafka.git","ssh_url":"git@github.com:apache/kafka.git","clone_url":"https://github.com/apache/kafka.git","svn_url":"https://github.com/apache/kafka","homepage":null,"size":76685,"stargazers_count":10296,"watchers_count":10296,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":5744,"mirror_url":null,"archived":false,"open_issues_count":535,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":5744,"open_issues":535,"watchers":10296,"default_branch":"trunk"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/kafka/pulls/5821"},"html":{"href":"https://github.com/apache/kafka/pull/5821"},"issue":{"href":"https://api.github.com/repos/apache/kafka/issues/5821"},"comments":{"href":"https://api.github.com/repos/apache/kafka/issues/5821/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/kafka/pulls/5821/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/kafka/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/kafka/pulls/5821/commits"},"statuses":{"href":"https://api.github.com/repos/apache/kafka/statuses/0479f14a1f60098b8a748f4652676f337820ac1e"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"hzxa21"}}