title: Python Kafka consumer missing to poll some messages
body: <p>The code for my Kafka consumer looks like this</p>
  
  <pre><code>def read_messages_from_kafka():
      topic = 'my-topic'
      consumer = KafkaConsumer(
          bootstrap_servers=['my-host1', 'my-host2'],
          client_id='my-client',
          group_id='my-group',
          auto_offset_reset='earliest',
          enable_auto_commit=False,
          api_version=(0, 8, 2)
      )
      consumer.assign([TopicPartition(topic, 0), TopicPartition(topic, 1)])
  
      messages = consumer.poll(timeout_ms=kafka_config.poll_timeout_ms, max_records=kafka_config.poll_max_records)
  
      for partition in messages.values():
          for message in partition:
              log.info("read {}".format(message))
  
      if messages:
          consumer.commit()
  
      next_offset0, next_offset1 = consumer.position(TopicPartition(topic, 0)), consumer.position(TopicPartition(topic, 1))
      log.info("next offset0={} and offset1={}".format(next_offset0, next_offset1))
  
  while True:
      read_messages_from_kafka()
      sleep(kafka_config.poll_sleep_ms / 1000.0)
  </code></pre>
  
  <p>I have realised that this setup of consumer is not able to read all the messages. And I am not able to reproduce this as it's intermittent issue.</p>
  
  <p>When I compare last 100 messages using <code>kafka-cat</code> to this consumer, I found that my consumer intermittently misses few messages randomly. What's wrong with my consumer?</p>
  
  <pre><code>kafkacat -C -b my-host1 -X broker.version.fallback=0.8.2.1 -t my-topic -o -100
  </code></pre>
  
  <p>There are just <a href="https://www.programcreek.com/python/example/98440/kafka.KafkaConsumer" rel="nofollow noreferrer">too many ways to consume messages in python</a>. There should be one and preferably only one obvious way to do it.</p>