{"action":"created","issue":{"url":"https://api.github.com/repos/vert-x3/vertx-kafka-client/issues/112","repository_url":"https://api.github.com/repos/vert-x3/vertx-kafka-client","labels_url":"https://api.github.com/repos/vert-x3/vertx-kafka-client/issues/112/labels{/name}","comments_url":"https://api.github.com/repos/vert-x3/vertx-kafka-client/issues/112/comments","events_url":"https://api.github.com/repos/vert-x3/vertx-kafka-client/issues/112/events","html_url":"https://github.com/vert-x3/vertx-kafka-client/issues/112","id":356710259,"node_id":"MDU6SXNzdWUzNTY3MTAyNTk=","number":112,"title":"Lost messages while consuming multiple partitions with one consumer","user":{"login":"emasab","id":2727141,"node_id":"MDQ6VXNlcjI3MjcxNDE=","avatar_url":"https://avatars2.githubusercontent.com/u/2727141?v=4","gravatar_id":"","url":"https://api.github.com/users/emasab","html_url":"https://github.com/emasab","followers_url":"https://api.github.com/users/emasab/followers","following_url":"https://api.github.com/users/emasab/following{/other_user}","gists_url":"https://api.github.com/users/emasab/gists{/gist_id}","starred_url":"https://api.github.com/users/emasab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emasab/subscriptions","organizations_url":"https://api.github.com/users/emasab/orgs","repos_url":"https://api.github.com/users/emasab/repos","events_url":"https://api.github.com/users/emasab/events{/privacy}","received_events_url":"https://api.github.com/users/emasab/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-09-04T08:46:26Z","updated_at":"2018-09-11T06:48:08Z","closed_at":null,"author_association":"NONE","body":"I'm experiencing lost messages while using a consumer subscribed to multiple partitions . The problem happens with batches of messages of more than 10 messages. \r\n\r\nWhat happens is that for some partitions only a multiple of 10 messages is received instead of receiving the complete batch for that one.\r\n\r\nI've seen that putting a conditional breakpoint in class KafkaReadStreamImpl at this line \r\n\r\n```java\r\nthis.current = records.iterator();\r\n```\r\n\r\nwith condition\r\n\r\n```java\r\nthis.current.count()>0\r\n```\r\ncauses the breakpoint to be triggered when the problem happens.\r\n\r\nIt seems that the iterator is overwritten while still full.\r\n\r\nI've tried to fix it putting a limit of 1 to the number of polls active at a specific time and after this fix the problem never happens even after consuming hundreds of messages, while without the fix the problem happen often with less than one hundred messages.\r\n\r\nMy fix is in this pull request \r\n\r\nhttps://github.com/vert-x3/vertx-kafka-client/pull/111\r\n\r\nPlease let me know if you can reproduce the issue\r\nThanks"},"comment":{"url":"https://api.github.com/repos/vert-x3/vertx-kafka-client/issues/comments/420166083","html_url":"https://github.com/vert-x3/vertx-kafka-client/issues/112#issuecomment-420166083","issue_url":"https://api.github.com/repos/vert-x3/vertx-kafka-client/issues/112","id":420166083,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMDE2NjA4Mw==","user":{"login":"emasab","id":2727141,"node_id":"MDQ6VXNlcjI3MjcxNDE=","avatar_url":"https://avatars2.githubusercontent.com/u/2727141?v=4","gravatar_id":"","url":"https://api.github.com/users/emasab","html_url":"https://github.com/emasab","followers_url":"https://api.github.com/users/emasab/followers","following_url":"https://api.github.com/users/emasab/following{/other_user}","gists_url":"https://api.github.com/users/emasab/gists{/gist_id}","starred_url":"https://api.github.com/users/emasab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emasab/subscriptions","organizations_url":"https://api.github.com/users/emasab/orgs","repos_url":"https://api.github.com/users/emasab/repos","events_url":"https://api.github.com/users/emasab/events{/privacy}","received_events_url":"https://api.github.com/users/emasab/received_events","type":"User","site_admin":false},"created_at":"2018-09-11T06:48:08Z","updated_at":"2018-09-11T06:48:08Z","author_association":"NONE","body":"The issue happens with a single partition as well as multiple, when more batches are received one after one, but it usually happens when using a pause - commit - resume cycle.\r\n"}}
{"actor":{"display_login":"emasab"}}