{"action":"created","issue":{"url":"https://api.github.com/repos/edenhill/librdkafka/issues/2096","repository_url":"https://api.github.com/repos/edenhill/librdkafka","labels_url":"https://api.github.com/repos/edenhill/librdkafka/issues/2096/labels{/name}","comments_url":"https://api.github.com/repos/edenhill/librdkafka/issues/2096/comments","events_url":"https://api.github.com/repos/edenhill/librdkafka/issues/2096/events","html_url":"https://github.com/edenhill/librdkafka/issues/2096","id":378813132,"node_id":"MDU6SXNzdWUzNzg4MTMxMzI=","number":2096,"title":"Consuming same message twice","user":{"login":"amircodota","id":25413470,"node_id":"MDQ6VXNlcjI1NDEzNDcw","avatar_url":"https://avatars0.githubusercontent.com/u/25413470?v=4","gravatar_id":"","url":"https://api.github.com/users/amircodota","html_url":"https://github.com/amircodota","followers_url":"https://api.github.com/users/amircodota/followers","following_url":"https://api.github.com/users/amircodota/following{/other_user}","gists_url":"https://api.github.com/users/amircodota/gists{/gist_id}","starred_url":"https://api.github.com/users/amircodota/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amircodota/subscriptions","organizations_url":"https://api.github.com/users/amircodota/orgs","repos_url":"https://api.github.com/users/amircodota/repos","events_url":"https://api.github.com/users/amircodota/events{/privacy}","received_events_url":"https://api.github.com/users/amircodota/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-08T16:27:28Z","updated_at":"2018-11-15T08:45:37Z","closed_at":null,"author_association":"NONE","body":"Description\r\n===========\r\nI have a situation where I am consuming the same message twice. This happens during the same process run and with the same consumer object.\r\n\r\nI'm doing manual commits. I'm processing messages asynchronously, and I want to commit only messages that have completed processing.\r\nPer https://github.com/edenhill/librdkafka/wiki/Consumer-offset-management, I've set 'enable.auto.offset.store' to false, and I am manually calling commit.\r\nFor each partition, I manage a list of offsets in flight, and I am only committing when I've completed all offsets up to a certain offset.\r\n\r\nSo my question is: Is it possible that when enable.auto.offset.store is false, the next fetch request will fetch from the last commit and not from where we actually last consumed? \r\n\r\nChecklist\r\n=========\r\n\r\nPlease provide the following information:\r\n\r\n - [x] librdkafka version (release number or git tag): 0.11.5\r\n - [x ] Apache Kafka version: 2.0.0\r\n - [x] librdkafka client configuration: 'auto.offset.reset': 'earliest', 'enable.auto.offset.store': false\r\n - [x] Operating system: Ubuntu 18 & 16\r\n - [ ] Provide logs (with `debug=..` as necessary) from librdkafka\r\n - [ ] Provide broker log excerpts\r\n - [ ] Critical issue\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/edenhill/librdkafka/issues/comments/438962481","html_url":"https://github.com/edenhill/librdkafka/issues/2096#issuecomment-438962481","issue_url":"https://api.github.com/repos/edenhill/librdkafka/issues/2096","id":438962481,"node_id":"MDEyOklzc3VlQ29tbWVudDQzODk2MjQ4MQ==","user":{"login":"edenhill","id":524990,"node_id":"MDQ6VXNlcjUyNDk5MA==","avatar_url":"https://avatars0.githubusercontent.com/u/524990?v=4","gravatar_id":"","url":"https://api.github.com/users/edenhill","html_url":"https://github.com/edenhill","followers_url":"https://api.github.com/users/edenhill/followers","following_url":"https://api.github.com/users/edenhill/following{/other_user}","gists_url":"https://api.github.com/users/edenhill/gists{/gist_id}","starred_url":"https://api.github.com/users/edenhill/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edenhill/subscriptions","organizations_url":"https://api.github.com/users/edenhill/orgs","repos_url":"https://api.github.com/users/edenhill/repos","events_url":"https://api.github.com/users/edenhill/events{/privacy}","received_events_url":"https://api.github.com/users/edenhill/received_events","type":"User","site_admin":false},"created_at":"2018-11-15T08:45:37Z","updated_at":"2018-11-15T08:45:37Z","author_association":"OWNER","body":"You will need to call offsets_store() (after processing) to let the client know you've processed a message, otherwise the internal fetcher might rewind itself upon leader failover, consumer rebalance, etc."}}
{"actor":{"display_login":"edenhill"}}