{"action":"created","issue":{"url":"https://api.github.com/repos/Blizzard/node-rdkafka/issues/404","repository_url":"https://api.github.com/repos/Blizzard/node-rdkafka","labels_url":"https://api.github.com/repos/Blizzard/node-rdkafka/issues/404/labels{/name}","comments_url":"https://api.github.com/repos/Blizzard/node-rdkafka/issues/404/comments","events_url":"https://api.github.com/repos/Blizzard/node-rdkafka/issues/404/events","html_url":"https://github.com/Blizzard/node-rdkafka/pull/404","id":314858900,"number":404,"title":"Back off ConsumerStream's consumption exponentially upon receiving time outs","user":{"login":"JaapRood","id":857549,"avatar_url":"https://avatars1.githubusercontent.com/u/857549?v=4","gravatar_id":"","url":"https://api.github.com/users/JaapRood","html_url":"https://github.com/JaapRood","followers_url":"https://api.github.com/users/JaapRood/followers","following_url":"https://api.github.com/users/JaapRood/following{/other_user}","gists_url":"https://api.github.com/users/JaapRood/gists{/gist_id}","starred_url":"https://api.github.com/users/JaapRood/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JaapRood/subscriptions","organizations_url":"https://api.github.com/users/JaapRood/orgs","repos_url":"https://api.github.com/users/JaapRood/repos","events_url":"https://api.github.com/users/JaapRood/events{/privacy}","received_events_url":"https://api.github.com/users/JaapRood/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-17T00:01:20Z","updated_at":"2018-04-17T00:02:42Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/Blizzard/node-rdkafka/pulls/404","html_url":"https://github.com/Blizzard/node-rdkafka/pull/404","diff_url":"https://github.com/Blizzard/node-rdkafka/pull/404.diff","patch_url":"https://github.com/Blizzard/node-rdkafka/pull/404.patch"},"body":"This is a proposal for a fix to #401. It forwards consumption time-out errors to the Node.js `KafkaConsumer`, which lets `KafkaConsumerStream` handle it and back off consumption. A couple of notes:\r\n\r\n- Before `ERR__TIMED_OUT` and `ERR__TIMED_OUT_QUEUE` were caught and never made it to Node, meaning `KafkaConsumer.consume` could now return an error for code where it didn't before. This could be interpreted as a breaking change.\r\n- To implement the exponential back-off I've added [`retry`](https://www.npmjs.com/package/retry) as a dependency. That was mostly to focus on the effect of the back-off, rather than implementing an effective back-off mechanism (which, [as this article describes well](http://dthain.blogspot.com.au/2009/02/exponential-backoff-in-distributed.html), isn't totally trivial). \r\n- While it's solved the issue for which I created #401, this back-off is rather \"blind\". It doesn't use the [`throttleTime` ](https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas#KIP-13-Quotas-PublicInterfaces) field as returned by the Broker. If anyone has some good concrete ideas about how to incorporate that, I'm all ears, because it would be a lot more effective."},"comment":{"url":"https://api.github.com/repos/Blizzard/node-rdkafka/issues/comments/381787186","html_url":"https://github.com/Blizzard/node-rdkafka/pull/404#issuecomment-381787186","issue_url":"https://api.github.com/repos/Blizzard/node-rdkafka/issues/404","id":381787186,"user":{"login":"JaapRood","id":857549,"avatar_url":"https://avatars1.githubusercontent.com/u/857549?v=4","gravatar_id":"","url":"https://api.github.com/users/JaapRood","html_url":"https://github.com/JaapRood","followers_url":"https://api.github.com/users/JaapRood/followers","following_url":"https://api.github.com/users/JaapRood/following{/other_user}","gists_url":"https://api.github.com/users/JaapRood/gists{/gist_id}","starred_url":"https://api.github.com/users/JaapRood/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JaapRood/subscriptions","organizations_url":"https://api.github.com/users/JaapRood/orgs","repos_url":"https://api.github.com/users/JaapRood/repos","events_url":"https://api.github.com/users/JaapRood/events{/privacy}","received_events_url":"https://api.github.com/users/JaapRood/received_events","type":"User","site_admin":false},"created_at":"2018-04-17T00:02:41Z","updated_at":"2018-04-17T00:02:41Z","author_association":"CONTRIBUTOR","body":"@webmakersteve I'd really appreciate it if you could shed your light on this. I'm not familiar with the codebase back to front, so it's very possible this approach is not a great one. I'll hold off on making more progress on it in the meantime."}}
{"actor":{"display_login":"JaapRood"}}