{"action":"created","issue":{"url":"https://api.github.com/repos/confluentinc/confluent-kafka-dotnet/issues/527","repository_url":"https://api.github.com/repos/confluentinc/confluent-kafka-dotnet","labels_url":"https://api.github.com/repos/confluentinc/confluent-kafka-dotnet/issues/527/labels{/name}","comments_url":"https://api.github.com/repos/confluentinc/confluent-kafka-dotnet/issues/527/comments","events_url":"https://api.github.com/repos/confluentinc/confluent-kafka-dotnet/issues/527/events","html_url":"https://github.com/confluentinc/confluent-kafka-dotnet/issues/527","id":319993970,"number":527,"title":"Partition assignment and offset semantic","user":{"login":"vchekan","id":106474,"avatar_url":"https://avatars1.githubusercontent.com/u/106474?v=4","gravatar_id":"","url":"https://api.github.com/users/vchekan","html_url":"https://github.com/vchekan","followers_url":"https://api.github.com/users/vchekan/followers","following_url":"https://api.github.com/users/vchekan/following{/other_user}","gists_url":"https://api.github.com/users/vchekan/gists{/gist_id}","starred_url":"https://api.github.com/users/vchekan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vchekan/subscriptions","organizations_url":"https://api.github.com/users/vchekan/orgs","repos_url":"https://api.github.com/users/vchekan/repos","events_url":"https://api.github.com/users/vchekan/events{/privacy}","received_events_url":"https://api.github.com/users/vchekan/received_events","type":"User","site_admin":false},"labels":[{"id":473160567,"url":"https://api.github.com/repos/confluentinc/confluent-kafka-dotnet/labels/question","name":"question","color":"cc317c","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-05-03T16:32:52Z","updated_at":"2018-05-18T23:05:56Z","closed_at":"2018-05-16T18:05:23Z","author_association":"NONE","body":"Description\r\n===========\r\n\r\nI am trying to understand how to structure my program in regard to offsets management.\r\n\r\n1. First thing to notice is that current implementation of confluent kafka client is using librdkafka function rd_kafka_consume0, which save offset first and then return message to the consumer:\r\nhttps://github.com/edenhill/librdkafka/blob/72b4e4169d7e1e2229595083bf31149fa01277cb/src/rdkafka.c#L2053\r\nAs result, offset is saved before processing is complete, which means that message loss is possible. \r\nIs it correct conclusion that \"enable.auto.commit\" can cause lost messages?\r\n\r\n2. My message handling is asynchronous, so I have to commit offsets manually. If I turn \"enable.auto.commit\" off, and will call `StoreOffsets` from async handler and will call `Commit()` from a timer, then I would replicate autocommit in my async handler and will achieve \"at least once\" guarantee?\r\n\r\nMy question though is, are `StoreOffset` and `Commit()` thread safe? There is no notion about tread safety of those functions in documentation.\r\n\r\n3. I am having hard times understanding concept of Consumer.OnPartitionsAssigned. Documentation says \"You should typically call the Consumer.Assign\", but why? In English this API reads as \"Upon partition asiignment you need to assign partitions\" which is nonsense. If partitions have been assigned, why do I have to assign them? My only possible explanation is that OnPartitionsAssigned is in fact \"OnOffsetDiscovered\" and driver would not assign discovered partitions automatically.\r\n\r\nBottom line is, i do not understand what is going on, nor reasoning behind this API.\r\n\r\nChecklist\r\n=========\r\nPlease provide the following information:\r\n\r\n - [x] Confluent.Kafka nuget version: 0.11.4\r\n"},"comment":{"url":"https://api.github.com/repos/confluentinc/confluent-kafka-dotnet/issues/comments/390354548","html_url":"https://github.com/confluentinc/confluent-kafka-dotnet/issues/527#issuecomment-390354548","issue_url":"https://api.github.com/repos/confluentinc/confluent-kafka-dotnet/issues/527","id":390354548,"user":{"login":"mhowlett","id":19174,"avatar_url":"https://avatars2.githubusercontent.com/u/19174?v=4","gravatar_id":"","url":"https://api.github.com/users/mhowlett","html_url":"https://github.com/mhowlett","followers_url":"https://api.github.com/users/mhowlett/followers","following_url":"https://api.github.com/users/mhowlett/following{/other_user}","gists_url":"https://api.github.com/users/mhowlett/gists{/gist_id}","starred_url":"https://api.github.com/users/mhowlett/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mhowlett/subscriptions","organizations_url":"https://api.github.com/users/mhowlett/orgs","repos_url":"https://api.github.com/users/mhowlett/repos","events_url":"https://api.github.com/users/mhowlett/events{/privacy}","received_events_url":"https://api.github.com/users/mhowlett/received_events","type":"User","site_admin":false},"created_at":"2018-05-18T23:05:56Z","updated_at":"2018-05-18T23:05:56Z","author_association":"COLLABORATOR","body":"yes, you'd need some sort of ordered list/map thing to keep track of each message individually.\r\n\r\nin some scenarios, it may work to increase the number of partitions (and have one consumer per partition).\r\n\r\nthis is one thing traditional messaging systems give you that Kafka doesn't. but traditional messaging systems are also much slower of course."}}
{"actor":{"display_login":"mhowlett"}}