{"action":"created","issue":{"url":"https://api.github.com/repos/edenhill/librdkafka/issues/794","repository_url":"https://api.github.com/repos/edenhill/librdkafka","labels_url":"https://api.github.com/repos/edenhill/librdkafka/issues/794/labels{/name}","comments_url":"https://api.github.com/repos/edenhill/librdkafka/issues/794/comments","events_url":"https://api.github.com/repos/edenhill/librdkafka/issues/794/events","html_url":"https://github.com/edenhill/librdkafka/issues/794","id":177346304,"number":794,"title":"Regarding KafkaConsumer","user":{"login":"abhit011","id":10906885,"avatar_url":"https://avatars.githubusercontent.com/u/10906885?v=3","gravatar_id":"","url":"https://api.github.com/users/abhit011","html_url":"https://github.com/abhit011","followers_url":"https://api.github.com/users/abhit011/followers","following_url":"https://api.github.com/users/abhit011/following{/other_user}","gists_url":"https://api.github.com/users/abhit011/gists{/gist_id}","starred_url":"https://api.github.com/users/abhit011/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abhit011/subscriptions","organizations_url":"https://api.github.com/users/abhit011/orgs","repos_url":"https://api.github.com/users/abhit011/repos","events_url":"https://api.github.com/users/abhit011/events{/privacy}","received_events_url":"https://api.github.com/users/abhit011/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/edenhill/librdkafka/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-09-16T05:09:38Z","updated_at":"2016-09-16T08:13:06Z","closed_at":null,"body":"Description\r\n===========\r\n\r\n@edenhill \r\n\r\nI have queries with respect to KafkaConsumer since I am producing from multiple threads into \r\n(3 Kafka Brokers with 10 topics ,  3 partitions and replication factor of 3) \r\n\r\n1. Now I am coding the Consumer part, a little confused, in my main thread i am creating the KafkaConsumerHandler, registering callbacks, and subscribing to all the 10 topics.  Should I create \r\nN Consume threads to consume messages ? or Just one thread to consume message from those 10 topics ? \r\ne.g \r\nthread code for consuming \r\nwhile (!CThread::currentThread().isInterrupted()){\r\n\t\t\tRdKafka::Message *msg = m_Ptr->getKafkaConsumerInstance()->consume(1000);\r\n\t\t\tkafkaMsgConsume(msg, NULL);\r\n\t\t\tpassItToDatabase(msg); // I am doing the deallocation of msg in this passItToDB  //after    succcesful dump into DB\r\n                       }   \r\n\r\n2. Trust me I could not able to understand what rebalancing callback does and why its required too ?\r\n\r\n3. Whats the maximum size of kafka logs directory for each broker?\r\n\r\n4. Is it possible that messages from producer and after successful reading each message from Kafka Consumer, those messages should be deleted from the logs ?\r\n\r\nAbhi\r\n\r\n\r\nChecklist\r\n=========\r\nPlease provide the following information:\r\n\r\n - [ ] librdkafka version (release number or git tag):\r\n - [ X] Apache Kafka version:kafka_2.11-0.10.0.1 \r\n - [ ] librdkafka client configuration:\r\n - [X ] Operating system: Windows 10\r\n - [ ] Using the legacy Consumer\r\n - [X] Using the high-level KafkaConsumer: yes intended to use that only\r\n - [ ] Provide logs (with `debug=..` as necessary) from librdkafka\r\n - [ ] Provide broker log excerpts\r\n - [ ] Critical issue\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/edenhill/librdkafka/issues/comments/247543560","html_url":"https://github.com/edenhill/librdkafka/issues/794#issuecomment-247543560","issue_url":"https://api.github.com/repos/edenhill/librdkafka/issues/794","id":247543560,"user":{"login":"abhit011","id":10906885,"avatar_url":"https://avatars.githubusercontent.com/u/10906885?v=3","gravatar_id":"","url":"https://api.github.com/users/abhit011","html_url":"https://github.com/abhit011","followers_url":"https://api.github.com/users/abhit011/followers","following_url":"https://api.github.com/users/abhit011/following{/other_user}","gists_url":"https://api.github.com/users/abhit011/gists{/gist_id}","starred_url":"https://api.github.com/users/abhit011/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abhit011/subscriptions","organizations_url":"https://api.github.com/users/abhit011/orgs","repos_url":"https://api.github.com/users/abhit011/repos","events_url":"https://api.github.com/users/abhit011/events{/privacy}","received_events_url":"https://api.github.com/users/abhit011/received_events","type":"User","site_admin":false},"created_at":"2016-09-16T08:13:06Z","updated_at":"2016-09-16T08:13:06Z","body":"@edenhill \r\n\r\nPlease correct me if I am wrong . \r\n\r\n**As per kafka documentation**\r\nThe Kafka cluster retains all published messages—whether or not they have been consumed—for a configurable period of time. For example if the log retention is set to two days, then for the two days after a message is published it is available for consumption, after which it will be discarded to free up space. **Kafka's performance is effectively constant with respect to data size so retaining lots of data is not a problem.**\r\nIn fact the only metadata retained on a per-consumer basis is the position of the consumer in  the log, called the \"offset\". This offset is controlled by the consumer: normally a consumer will advance its offset linearly as it reads messages, but in fact the position is controlled by the consumer and it can consume messages in any order it likes. For example a consumer can reset to an older offset to reprocess.\r\n\r\ne.g something like this t_KafkaConf->set(\"auto.offset.reset\", \"smallest\", errstr); will always start from the beginning ? \r\n\r\nNow there is confusion like \r\n\r\nLast offset read by Consumer for topic **00** was 52, topic **01** was 70, topic **02** was 10. and now I shut down the Consumer and lets say producer is keep on dumping the data into Kafka logs and current offset of topic **00** is 68, topic **01** is 100, topic **02** is 59 etc.. \r\n\r\nNow when i restart the consumer, from where the consumer will start consuming the data, e.g from where it let its last/ topic offset or like start from 0 ? \r\n\r\nAbhi"}}
{"actor":{"display_login":"abhit011"}}