{"action":"created","issue":{"url":"https://api.github.com/repos/confluentinc/confluent-kafka-python/issues/18","repository_url":"https://api.github.com/repos/confluentinc/confluent-kafka-python","labels_url":"https://api.github.com/repos/confluentinc/confluent-kafka-python/issues/18/labels{/name}","comments_url":"https://api.github.com/repos/confluentinc/confluent-kafka-python/issues/18/comments","events_url":"https://api.github.com/repos/confluentinc/confluent-kafka-python/issues/18/events","html_url":"https://github.com/confluentinc/confluent-kafka-python/issues/18","id":165187320,"number":18,"title":"Consumer poll() does not always return","user":{"login":"mikesparr","id":5553105,"avatar_url":"https://avatars.githubusercontent.com/u/5553105?v=3","gravatar_id":"","url":"https://api.github.com/users/mikesparr","html_url":"https://github.com/mikesparr","followers_url":"https://api.github.com/users/mikesparr/followers","following_url":"https://api.github.com/users/mikesparr/following{/other_user}","gists_url":"https://api.github.com/users/mikesparr/gists{/gist_id}","starred_url":"https://api.github.com/users/mikesparr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikesparr/subscriptions","organizations_url":"https://api.github.com/users/mikesparr/orgs","repos_url":"https://api.github.com/users/mikesparr/repos","events_url":"https://api.github.com/users/mikesparr/events{/privacy}","received_events_url":"https://api.github.com/users/mikesparr/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/confluentinc/confluent-kafka-python/labels/librdkafka","name":"librdkafka","color":"f9d0c4"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-07-12T21:37:16Z","updated_at":"2016-08-04T16:10:37Z","closed_at":null,"body":"I'm running into an odd scenario with Consumer class implementation where sometimes the `poll()` return is null or empty. I would expect this to be one of the object with message, or object with error (sometimes just EOF for partition), but never empty or null?\r\n\r\nHere is my code and out of 25,000 messages in topic, just under 2,000 warnings in log that `poll()` returned no msg object. What conditions would this occur or did I implement incorrectly below?\r\n\r\n~~~\r\n                while running:\r\n                    try:\r\n                        msg = c.poll()\r\n                        if msg:\r\n                            if not msg.error():\r\n                                processor(msg)\r\n                            elif msg.error().code() == kafka.KafkaError._PARTITION_EOF:\r\n                                # End of partition event\r\n                                logger.debug( \"{} [{}] reached end at offset {}\".format(\r\n                                                msg.topic(), msg.partition(), msg.offset()) )\r\n                                logger.debug( \"Consumed [{}] records so far (reached end of offset)\".format(self.consumed) )\r\n                            else:\r\n                                logger.debug( \"Unknown error [{}] during polling.\".format(msg.error()) )\r\n                                raise kafka.KafkaException(msg.error())\r\n                            \r\n                            self.consumed += 1\r\n                            if (self.consumed % 100) == 0:\r\n                                logger.debug( \"Consumed [{}] records since startup.\".format(self.consumed) )\r\n                        else:\r\n                            logger.warn( \"No message object returned on poll() [{}]\".format(msg) )\r\n                            \r\n                    except Exception, e:\r\n                        logger.warn( \"Error polling messages [{}]\".format(str(e)) )\r\n                        logger.debug( traceback.format_exc() )\r\n                        # we do not stop loop because some polling errors anticipated\r\n                        # https://github.com/confluentinc/confluent-kafka-python/issues/12\r\n                # end while running loop\r\n~~~\r\n\r\nIn logs:\r\n\r\n~~~\r\n2016-07-12 15:09:25,331 - WARNING - stream_consumer.pyc - No message object returned on poll()\r\n2016-07-12 15:09:25,332 - WARNING - stream_consumer.pyc - No message object returned on poll()\r\n2016-07-12 15:09:25,342 - WARNING - stream_consumer.pyc - No message object returned on poll()\r\n2016-07-12 15:09:25,343 - WARNING - stream_consumer.pyc - No message object returned on poll()\r\n2016-07-12 15:09:25,343 - WARNING - stream_consumer.pyc - No message object returned on poll()\r\n2016-07-12 15:09:25,343 - WARNING - stream_consumer.pyc - No message object returned on poll()\r\n2016-07-12 15:09:25,343 - WARNING - stream_consumer.pyc - No message object returned on poll()\r\n2016-07-12 15:09:25,343 - WARNING - stream_consumer.pyc - No message object returned on poll()\r\n2016-07-12 15:09:25,343 - WARNING - stream_consumer.pyc - No message object returned on poll()\r\n2016-07-12 15:09:25,343 - WARNING - stream_consumer.pyc - No message object returned on poll()\r\n2016-07-12 15:09:25,343 - WARNING - stream_consumer.pyc - No message object returned on poll()\r\n2016-07-12 15:09:25,344 - WARNING - stream_consumer.pyc - No message object returned on poll()\r\n2016-07-12 15:09:25,344 - WARNING - stream_consumer.pyc - No message object returned on poll()\r\n2016-07-12 15:09:25,344 - WARNING - stream_consumer.pyc - No message object returned on poll()\r\n2016-07-12 15:09:25,344 - DEBUG - stream_consumer.pyc - Consumed [24800] records since startup.\r\n2016-07-12 15:09:25,344 - WARNING - stream_consumer.pyc - No message object returned on poll()\r\n2016-07-12 15:09:25,345 - WARNING - stream_consumer.pyc - No message object returned on poll()\r\n~~~\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/confluentinc/confluent-kafka-python/issues/comments/237602207","html_url":"https://github.com/confluentinc/confluent-kafka-python/issues/18#issuecomment-237602207","issue_url":"https://api.github.com/repos/confluentinc/confluent-kafka-python/issues/18","id":237602207,"user":{"login":"mikesparr","id":5553105,"avatar_url":"https://avatars.githubusercontent.com/u/5553105?v=3","gravatar_id":"","url":"https://api.github.com/users/mikesparr","html_url":"https://github.com/mikesparr","followers_url":"https://api.github.com/users/mikesparr/followers","following_url":"https://api.github.com/users/mikesparr/following{/other_user}","gists_url":"https://api.github.com/users/mikesparr/gists{/gist_id}","starred_url":"https://api.github.com/users/mikesparr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikesparr/subscriptions","organizations_url":"https://api.github.com/users/mikesparr/orgs","repos_url":"https://api.github.com/users/mikesparr/repos","events_url":"https://api.github.com/users/mikesparr/events{/privacy}","received_events_url":"https://api.github.com/users/mikesparr/received_events","type":"User","site_admin":false},"created_at":"2016-08-04T16:10:37Z","updated_at":"2016-08-04T16:10:37Z","body":"@edenhill  I haven't checked offset consistency but could output those to log to figure out if non-sequential. I'm wrapping up something else at the moment. I did have to switch to single partition to fix the data \"loss\" issue reported with that librdkafka bug you confirmed when consumer returns no msg object sometimes. I've played around now with the consumer config settings and queues. I have no errors in logs and with single partition I have 1:1 record count as expected - adding partitions I get loss (with single consumer across many partitions)."}}
{"actor":{"display_login":"mikesparr"}}