{"action":"created","issue":{"url":"https://api.github.com/repos/ah-/rdkafka-dotnet/issues/87","repository_url":"https://api.github.com/repos/ah-/rdkafka-dotnet","labels_url":"https://api.github.com/repos/ah-/rdkafka-dotnet/issues/87/labels{/name}","comments_url":"https://api.github.com/repos/ah-/rdkafka-dotnet/issues/87/comments","events_url":"https://api.github.com/repos/ah-/rdkafka-dotnet/issues/87/events","html_url":"https://github.com/ah-/rdkafka-dotnet/issues/87","id":192447740,"number":87,"title":"Getting _PARTITION_EOF error when using consume","user":{"login":"bkim54","id":11249298,"avatar_url":"https://avatars.githubusercontent.com/u/11249298?v=3","gravatar_id":"","url":"https://api.github.com/users/bkim54","html_url":"https://github.com/bkim54","followers_url":"https://api.github.com/users/bkim54/followers","following_url":"https://api.github.com/users/bkim54/following{/other_user}","gists_url":"https://api.github.com/users/bkim54/gists{/gist_id}","starred_url":"https://api.github.com/users/bkim54/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bkim54/subscriptions","organizations_url":"https://api.github.com/users/bkim54/orgs","repos_url":"https://api.github.com/users/bkim54/repos","events_url":"https://api.github.com/users/bkim54/events{/privacy}","received_events_url":"https://api.github.com/users/bkim54/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-11-30T01:07:55Z","updated_at":"2016-11-30T16:41:40Z","closed_at":null,"body":"I am using the Consumer.Consume method with a 30 sec timeout (because I want to receive just 1 message) however randomly (about 1/10 times) the returned MessageAndError contains the error \"_PARTITION_EOF\"\r\n\r\nthis is how I set up my consumer\r\n```\r\nConfig kafkaConfig = new Config() \r\n {\r\n      GroupId = replyInboxString, // this is a random guid-string\r\n       DefaultTopicConfig = new TopicConfig { [\"auto.offset.reset\"] = \"smallest\" },\r\n       EnableAutoCommit = true\r\n};\r\n\r\n```\r\nThen I use assign and set each offset for each partition for my one topic to be Offset.Beginning\r\n\r\nMy cluster is setup with 3 partitions.\r\n\r\nam I doing something wrong? why do I get this error randomly?"},"comment":{"url":"https://api.github.com/repos/ah-/rdkafka-dotnet/issues/comments/263924532","html_url":"https://github.com/ah-/rdkafka-dotnet/issues/87#issuecomment-263924532","issue_url":"https://api.github.com/repos/ah-/rdkafka-dotnet/issues/87","id":263924532,"user":{"login":"treziac","id":2729252,"avatar_url":"https://avatars.githubusercontent.com/u/2729252?v=3","gravatar_id":"","url":"https://api.github.com/users/treziac","html_url":"https://github.com/treziac","followers_url":"https://api.github.com/users/treziac/followers","following_url":"https://api.github.com/users/treziac/following{/other_user}","gists_url":"https://api.github.com/users/treziac/gists{/gist_id}","starred_url":"https://api.github.com/users/treziac/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/treziac/subscriptions","organizations_url":"https://api.github.com/users/treziac/orgs","repos_url":"https://api.github.com/users/treziac/repos","events_url":"https://api.github.com/users/treziac/events{/privacy}","received_events_url":"https://api.github.com/users/treziac/received_events","type":"User","site_admin":false},"created_at":"2016-11-30T16:41:40Z","updated_at":"2016-11-30T16:41:40Z","body":"Nope, it will come from the partition from which you just consumed the last message. Or at start, from empty partition or with no new message\r\n\r\nExample: let's assume you are consuming and there is no new message in any partition.\r\nYou are then trying to consume, and timeout every 30s, which means a null message is return from Consume (and typically just ignore it, look at eventConsumer for example)\r\n\r\nYou then push 3 messages, 2 to P0 and 1 to P1 (assuming 3 partitions)\r\nYour loop will then:\r\n- A1 : Consume and return immediately message 1 from P0\r\n- B1 : Consume and return immediately message 1 from P1\r\n- A2 : Consume and return immediately message 2 from P0\r\n- A3 : Consume and return immediately EOF from P0 (as there is no more available message in the internal queue)\r\n- B2 : Consume and return immediately EOF from P1\r\n- Consume and wait for timeout (no more message available)\r\n\r\nNote that A and B (A for P0 and B for P1) may be mixed in whatever order (A1/A2/A3/B1/B2 or B1/A1/A2/B2/B3 are other possible flow, there is one thread for each partition internally)\r\n\r\nIn an application where you receive lot of messages but can easily keep up with the throughput, you will receive lot of EOF. It just means you are not cowling under request.\r\nIn a topic with lot of partition which are partitioned, when you process lot of past data from kafka, it allow you to know when all data from a partition have been received so you can process them immediately.\r\n\r\nIn your use case, it seems useless :) If you are not interested by eof notifications, you can either ignore them or desactivate them via conf (conf[\"enable.partition.eof\"] = \"false\")\r\n\r\nBy the way ( @ah- correct me if i'm wrong) you should stick with a 1s timeout as consume also triggers errors (transport error for example) so you may want to get them immediately and not wait for a timeout (i'm not entirely sure with this)"}}
{"actor":{"display_login":"treziac"}}