{"action":"created","issue":{"url":"https://api.github.com/repos/edenhill/librdkafka/issues/1666","repository_url":"https://api.github.com/repos/edenhill/librdkafka","labels_url":"https://api.github.com/repos/edenhill/librdkafka/issues/1666/labels{/name}","comments_url":"https://api.github.com/repos/edenhill/librdkafka/issues/1666/comments","events_url":"https://api.github.com/repos/edenhill/librdkafka/issues/1666/events","html_url":"https://github.com/edenhill/librdkafka/issues/1666","id":293571586,"number":1666,"title":"Produce queue latency delay in librdkafka","user":{"login":"keyankay","id":26476356,"avatar_url":"https://avatars2.githubusercontent.com/u/26476356?v=4","gravatar_id":"","url":"https://api.github.com/users/keyankay","html_url":"https://github.com/keyankay","followers_url":"https://api.github.com/users/keyankay/followers","following_url":"https://api.github.com/users/keyankay/following{/other_user}","gists_url":"https://api.github.com/users/keyankay/gists{/gist_id}","starred_url":"https://api.github.com/users/keyankay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keyankay/subscriptions","organizations_url":"https://api.github.com/users/keyankay/orgs","repos_url":"https://api.github.com/users/keyankay/repos","events_url":"https://api.github.com/users/keyankay/events{/privacy}","received_events_url":"https://api.github.com/users/keyankay/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-02-01T15:26:52Z","updated_at":"2018-02-05T11:57:49Z","closed_at":null,"author_association":"NONE","body":"Read the FAQ first: https://github.com/edenhill/librdkafka/wiki/FAQ\r\n\r\n\r\n\r\nDescription\r\n===========\r\nWe are building a system which needs to have a very low latency.\r\nFor Producer we are using librdkafka c++ wrappers and on the consumer side we are using java based kafka processor API.\r\n\r\nConfiguration is mentioned below\r\n\r\nWe expect that the kafka producer will have to pump atleast 120 messages in a millisecond of length 300 bytes each.\r\n\r\n\r\nWe observe sometimes when more messages are being pumped, the producer is able to send the message within 1 ms to the broker and sometimes it takes 40 to 50 ms \r\n\r\n However our configuration currently is just 1 broker and 1 topic partition.\r\n\r\n\r\n\r\n\r\nHow to reproduce\r\n================\r\nUsed v0.11.3\r\n\r\n\r\nChecklist\r\n=========\r\n\r\n**IMPORTANT**: We will close issues where the checklist has not been completed.\r\n\r\nPlease provide the following information:\r\n\r\n - [x] librdkafka version (release number or git tag): v0.11.3\r\n - [ ] Apache Kafka version:0.11\r\n - [ ] librdkafka client configuration: \r\nqueue.buffering.max.ms=0\r\nbatch.num.messages=100\r\nconsumertimeout=5\r\nfetch.wait.max.ms=2\r\nfetch.error.backoff.ms=2\r\n\r\n\r\n - [ ] Operating system: `Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64\r\n[rdkafka_debuglog.txt](https://github.com/edenhill/librdkafka/files/1686042/rdkafka_debuglog.txt)\r\n\r\n - [ ] Provide logs (with `debug=..` as necessary) from librdkafka\r\n - [ ] Provide broker log excerpts\r\n - [ ] Critical issue\r\n\r\n[rdkafka_debuglog.txt](https://github.com/edenhill/librdkafka/files/1686045/rdkafka_debuglog.txt)\r\n\r\n**Producer log: sending message at 59:848**\r\n03-01-2018 02:59:59:848 INFO  - ******** Received 313 data of type GN_DATA ********\r\n03-01-2018 02:59:59:848 INFO  - Successfully sent the message to Router :\r\n\r\n**Consumer receiving at 59:936.**\r\n\r\n2018-01-03 02:59:59.742 INFO  com.nokia.router.distributor.DistributorUtil.forwardUnicastToAll(DistributorUtil.java:129) - Distributing the message to session id: 15\r\n-- look at the delay here\r\n2018-01-03 02:59:59.936 DEBUG com.nokia.router.stream.StreamProcess.process(StreamProcess.java:42) - New message received Current Time : 1514928599936\r\n\r\nWe recieve a delay of 50 ms in produce queue\r\n"},"comment":{"url":"https://api.github.com/repos/edenhill/librdkafka/issues/comments/363064182","html_url":"https://github.com/edenhill/librdkafka/issues/1666#issuecomment-363064182","issue_url":"https://api.github.com/repos/edenhill/librdkafka/issues/1666","id":363064182,"user":{"login":"edenhill","id":524990,"avatar_url":"https://avatars0.githubusercontent.com/u/524990?v=4","gravatar_id":"","url":"https://api.github.com/users/edenhill","html_url":"https://github.com/edenhill","followers_url":"https://api.github.com/users/edenhill/followers","following_url":"https://api.github.com/users/edenhill/following{/other_user}","gists_url":"https://api.github.com/users/edenhill/gists{/gist_id}","starred_url":"https://api.github.com/users/edenhill/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edenhill/subscriptions","organizations_url":"https://api.github.com/users/edenhill/orgs","repos_url":"https://api.github.com/users/edenhill/repos","events_url":"https://api.github.com/users/edenhill/events{/privacy}","received_events_url":"https://api.github.com/users/edenhill/received_events","type":"User","site_admin":false},"created_at":"2018-02-05T11:57:49Z","updated_at":"2018-02-05T11:57:49Z","author_association":"OWNER","body":"Btw: you can transition to master branch, the nonatomic_msgq branch has been merged"}}
{"actor":{"display_login":"edenhill"}}