{"action":"created","issue":{"url":"https://api.github.com/repos/microdevs/missy/issues/29","repository_url":"https://api.github.com/repos/microdevs/missy","labels_url":"https://api.github.com/repos/microdevs/missy/issues/29/labels{/name}","comments_url":"https://api.github.com/repos/microdevs/missy/issues/29/comments","events_url":"https://api.github.com/repos/microdevs/missy/issues/29/events","html_url":"https://github.com/microdevs/missy/pull/29","id":306510110,"number":29,"title":"proposition of dead letter queue implementation:","user":{"login":"michalkarolik","id":36033247,"avatar_url":"https://avatars2.githubusercontent.com/u/36033247?v=4","gravatar_id":"","url":"https://api.github.com/users/michalkarolik","html_url":"https://github.com/michalkarolik","followers_url":"https://api.github.com/users/michalkarolik/followers","following_url":"https://api.github.com/users/michalkarolik/following{/other_user}","gists_url":"https://api.github.com/users/michalkarolik/gists{/gist_id}","starred_url":"https://api.github.com/users/michalkarolik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michalkarolik/subscriptions","organizations_url":"https://api.github.com/users/michalkarolik/orgs","repos_url":"https://api.github.com/users/michalkarolik/repos","events_url":"https://api.github.com/users/michalkarolik/events{/privacy}","received_events_url":"https://api.github.com/users/michalkarolik/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-03-19T15:25:01Z","updated_at":"2018-03-21T15:49:42Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/microdevs/missy/pulls/29","html_url":"https://github.com/microdevs/missy/pull/29","diff_url":"https://github.com/microdevs/missy/pull/29.diff","patch_url":"https://github.com/microdevs/missy/pull/29.patch"},"body":"I've created this PR to open discussion, ignore the code for now.\r\n\r\nSo.. consumer groups + commiting offsets does not really work the way we thought. Imagine situation when we have 3 messages (m1, m2, m3) and a consumer from consumer group starts to consume them. Processing m1 went fine, processing m2 failed.. what happens next? Even though consumer did not commit offset for m2 it will proceed to m3 on next poll. It **will not read** not commited message again. BOOM, message is lost. It would be read again only if this consumer crash and other one would take its place in consumer group (kafka rebalance). \r\nIf consuming/processing succeeds with m3 it will commit offset for m3 which effectively will also commit offset for m2.\r\n\r\nI can't see 100% correct and convenient solution for this problem, but what we can do is add Dead Letter Queues (DLQ) concept to our kafka topics. This way we can at least trace and react manually if something is really bad with particular message. \r\n\r\nlets say - try 5 times to process message, if it fails then move it to DLQ and pray :) (We could also send some notification on email / slack in a future )\r\n\r\n1.If consuming message returns error then retry configured number of times ( resend same message to same topic but with retryCounter incremented).\r\n2. If all tries failed and counter reached configured limit  - send message to DLQ so we can later do something with it. Topic name convention is topicName+\".dlq\"."},"comment":{"url":"https://api.github.com/repos/microdevs/missy/issues/comments/374988060","html_url":"https://github.com/microdevs/missy/pull/29#issuecomment-374988060","issue_url":"https://api.github.com/repos/microdevs/missy/issues/29","id":374988060,"user":{"login":"GitCop","id":7943652,"avatar_url":"https://avatars0.githubusercontent.com/u/7943652?v=4","gravatar_id":"","url":"https://api.github.com/users/GitCop","html_url":"https://github.com/GitCop","followers_url":"https://api.github.com/users/GitCop/followers","following_url":"https://api.github.com/users/GitCop/following{/other_user}","gists_url":"https://api.github.com/users/GitCop/gists{/gist_id}","starred_url":"https://api.github.com/users/GitCop/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GitCop/subscriptions","organizations_url":"https://api.github.com/users/GitCop/orgs","repos_url":"https://api.github.com/users/GitCop/repos","events_url":"https://api.github.com/users/GitCop/events{/privacy}","received_events_url":"https://api.github.com/users/GitCop/received_events","type":"User","site_admin":false},"created_at":"2018-03-21T15:49:42Z","updated_at":"2018-03-21T15:49:42Z","author_association":"NONE","body":"There were the following issues with your Pull Request\n\n * Commit: 3fd827365428574a06c482843f33f3251de97e18\n  * Commits must be in the following format: **%{type}(%{scope}): %{description}**\n\n * Commit: 605f19cd7e24ed3e91288e3050fc3602805c1761\n  * Commits must be in the following format: **%{type}(%{scope}): %{description}**\n\n * Commit: e3c7368783761a6f5d22a72cdcd497276d61594a\n  * Commits must be in the following format: **%{type}(%{scope}): %{description}**\n  * Your subject line is longer than 72 characters\n  * Invalid type. Valid types are **feat, fix, docs, style, refactor, perf, test, chore, revert, merge**\n\nGuidelines are available at https://github.com/microdevs/missy\n\n---\nThis message was auto-generated by https://gitcop.com\n"}}
{"actor":{"display_login":"GitCop"}}