{"action":"created","issue":{"url":"https://api.github.com/repos/Parsely/pykafka/issues/457","repository_url":"https://api.github.com/repos/Parsely/pykafka","labels_url":"https://api.github.com/repos/Parsely/pykafka/issues/457/labels{/name}","comments_url":"https://api.github.com/repos/Parsely/pykafka/issues/457/comments","events_url":"https://api.github.com/repos/Parsely/pykafka/issues/457/events","html_url":"https://github.com/Parsely/pykafka/issues/457","id":132273657,"number":457,"title":"Produced messages getting lost when broker is down","user":{"login":"jonsource","id":3274187,"avatar_url":"https://avatars.githubusercontent.com/u/3274187?v=3","gravatar_id":"","url":"https://api.github.com/users/jonsource","html_url":"https://github.com/jonsource","followers_url":"https://api.github.com/users/jonsource/followers","following_url":"https://api.github.com/users/jonsource/following{/other_user}","gists_url":"https://api.github.com/users/jonsource/gists{/gist_id}","starred_url":"https://api.github.com/users/jonsource/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonsource/subscriptions","organizations_url":"https://api.github.com/users/jonsource/orgs","repos_url":"https://api.github.com/users/jonsource/repos","events_url":"https://api.github.com/users/jonsource/events{/privacy}","received_events_url":"https://api.github.com/users/jonsource/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Parsely/pykafka/labels/needs%20response","name":"needs response","color":"fef2c0"},{"url":"https://api.github.com/repos/Parsely/pykafka/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/Parsely/pykafka/milestones/17","html_url":"https://github.com/Parsely/pykafka/milestones/2.2.1","labels_url":"https://api.github.com/repos/Parsely/pykafka/milestones/17/labels","id":1567376,"number":17,"title":"2.2.1","description":"","creator":{"login":"emmett9001","id":723615,"avatar_url":"https://avatars.githubusercontent.com/u/723615?v=3","gravatar_id":"","url":"https://api.github.com/users/emmett9001","html_url":"https://github.com/emmett9001","followers_url":"https://api.github.com/users/emmett9001/followers","following_url":"https://api.github.com/users/emmett9001/following{/other_user}","gists_url":"https://api.github.com/users/emmett9001/gists{/gist_id}","starred_url":"https://api.github.com/users/emmett9001/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emmett9001/subscriptions","organizations_url":"https://api.github.com/users/emmett9001/orgs","repos_url":"https://api.github.com/users/emmett9001/repos","events_url":"https://api.github.com/users/emmett9001/events{/privacy}","received_events_url":"https://api.github.com/users/emmett9001/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":4,"state":"open","created_at":"2016-02-05T22:58:08Z","updated_at":"2016-02-11T17:19:35Z","due_on":null,"closed_at":null},"comments":12,"created_at":"2016-02-08T22:23:42Z","updated_at":"2016-02-12T18:41:28Z","closed_at":null,"body":"Hello, I have the following issue with pykafka, in this setting:\r\n\r\nRunning 3 kafka brokers and 1 zookeeper in docker on one machine in docker\r\nzookeeper 172.17.0.2:2181\r\nkafka[1-3] 172.17.0.[3-5]:9092\r\nall default configs except kafka `config/server.properties` has \r\n`leader.imbalance.check.interval.seconds = 30` added, otherwise the topic leader rebalancing interval is too long - 5 minutes.\r\n_(actually I noticed first in a real high availability cluster with 3 kafka brokers and 3 zookeepers, but that's not important I think - I reduced the problem into a local setting)_\r\nusing pykafka latest commit 0891c81 or 2.2.0, python 2.7.6, kafka 0.9.0.0, zookeeper 3.4.6, docker 1.9.1, running on Ubuntu 14.04\r\n\r\nI send a few messages to be produced, then stop one of the kafka containers (simulating a failure) and send a few more - _(source added in gist)_. Messages for the broker that has stopped get lost.\r\n\r\nThe strange thing is pykafka reconnects all-right after a few time-outs, starts the proper worker threads, but doesn't send the messages. But during the time-out pykafka attempts to resend the messages, it seems it forgets them when the reconnect occurs _(logfile in the gist)_.\r\nI reduced the problem to a gist here: https://gist.github.com/jonsource/4f5ec5b345f75f21b16b\r\nwith a log of a typical run, and testPykafka.py source at the bottom.\r\n\r\nOne more strange thing I noticed - the messages get lost when pausing kafka1 or kafka2. When kafka3 is paused, they get sent OK after the reconnect.\r\n\r\nBest regards."},"comment":{"url":"https://api.github.com/repos/Parsely/pykafka/issues/comments/183445011","html_url":"https://github.com/Parsely/pykafka/issues/457#issuecomment-183445011","issue_url":"https://api.github.com/repos/Parsely/pykafka/issues/457","id":183445011,"user":{"login":"emmett9001","id":723615,"avatar_url":"https://avatars.githubusercontent.com/u/723615?v=3","gravatar_id":"","url":"https://api.github.com/users/emmett9001","html_url":"https://github.com/emmett9001","followers_url":"https://api.github.com/users/emmett9001/followers","following_url":"https://api.github.com/users/emmett9001/following{/other_user}","gists_url":"https://api.github.com/users/emmett9001/gists{/gist_id}","starred_url":"https://api.github.com/users/emmett9001/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emmett9001/subscriptions","organizations_url":"https://api.github.com/users/emmett9001/orgs","repos_url":"https://api.github.com/users/emmett9001/repos","events_url":"https://api.github.com/users/emmett9001/events{/privacy}","received_events_url":"https://api.github.com/users/emmett9001/received_events","type":"User","site_admin":false},"created_at":"2016-02-12T18:41:28Z","updated_at":"2016-02-12T18:41:28Z","body":"Thanks for the information, @cscheffler - It's really helpful to have this practical information written down."}}