{"action":"created","issue":{"url":"https://api.github.com/repos/olivere/elastic/issues/698","repository_url":"https://api.github.com/repos/olivere/elastic","labels_url":"https://api.github.com/repos/olivere/elastic/issues/698/labels{/name}","comments_url":"https://api.github.com/repos/olivere/elastic/issues/698/comments","events_url":"https://api.github.com/repos/olivere/elastic/issues/698/events","html_url":"https://github.com/olivere/elastic/pull/698","id":293033244,"number":698,"title":"add backpressure when all connections are dead","user":{"login":"rwynn","id":260672,"avatar_url":"https://avatars2.githubusercontent.com/u/260672?v=4","gravatar_id":"","url":"https://api.github.com/users/rwynn","html_url":"https://github.com/rwynn","followers_url":"https://api.github.com/users/rwynn/followers","following_url":"https://api.github.com/users/rwynn/following{/other_user}","gists_url":"https://api.github.com/users/rwynn/gists{/gist_id}","starred_url":"https://api.github.com/users/rwynn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rwynn/subscriptions","organizations_url":"https://api.github.com/users/rwynn/orgs","repos_url":"https://api.github.com/users/rwynn/repos","events_url":"https://api.github.com/users/rwynn/events{/privacy}","received_events_url":"https://api.github.com/users/rwynn/received_events","type":"User","site_admin":false},"labels":[{"id":21403610,"url":"https://api.github.com/repos/olivere/elastic/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-01-31T05:11:04Z","updated_at":"2018-02-10T14:27:00Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/olivere/elastic/pulls/698","html_url":"https://github.com/olivere/elastic/pull/698","diff_url":"https://github.com/olivere/elastic/pull/698.diff","patch_url":"https://github.com/olivere/elastic/pull/698.patch"},"body":"I've noticed that if I have a BulkProcessor with a short `FlushInterval` and all connections to Elasticsearch become dead, bad things happen.  Since the BulkService `requests` slice is only reset after a successful commit, the short flush interval causes the Processor to bang the same set of requests over and over again against dead connections.  Additionally, without any back pressure, producers can potentially keep adding fuel to the fire by adding to the `requests` slice of the BulkService.  \r\n\r\nThis PR is an attempt to add some relief in this scenario.  I think it can possibly be enhanced to only trigger liveness checks after errors of \"type connection\".  \r\n\r\nOriginally, I was looking into why requests were being repeated so often on failure when I pulled the plug on Elasticsearch.  My retrier was set to a StopRetrier, so something was fishy.  Then I realized that the request state doesn't get cleared on failure and my FlushInterval is less than a second.\r\n\r\nThis PR can be tested starting a BulkProcessor with a sub second FlushInterval, adding some BulkableRequests, then stopping Elasticsearch, and finally adding a few more BulkableRequests.   "},"comment":{"url":"https://api.github.com/repos/olivere/elastic/issues/comments/364658706","html_url":"https://github.com/olivere/elastic/pull/698#issuecomment-364658706","issue_url":"https://api.github.com/repos/olivere/elastic/issues/698","id":364658706,"user":{"login":"olivere","id":134848,"avatar_url":"https://avatars2.githubusercontent.com/u/134848?v=4","gravatar_id":"","url":"https://api.github.com/users/olivere","html_url":"https://github.com/olivere","followers_url":"https://api.github.com/users/olivere/followers","following_url":"https://api.github.com/users/olivere/following{/other_user}","gists_url":"https://api.github.com/users/olivere/gists{/gist_id}","starred_url":"https://api.github.com/users/olivere/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olivere/subscriptions","organizations_url":"https://api.github.com/users/olivere/orgs","repos_url":"https://api.github.com/users/olivere/repos","events_url":"https://api.github.com/users/olivere/events{/privacy}","received_events_url":"https://api.github.com/users/olivere/received_events","type":"User","site_admin":false},"created_at":"2018-02-10T14:27:00Z","updated_at":"2018-02-10T14:27:00Z","author_association":"OWNER","body":"Hmm... there's something wrong after applying your patch. Simply run the bulk_processor recipe, then stop ES after ~10s. It starts well, then starts to enqueue again, behaving just like before:\r\n\r\n```\r\n$ go run main.go\r\nQueued=    0 Written=    1358 Succeeded=       0 Failed=       0 Comitted=     0 Flushed=     0\r\nQueued=    0 Written=    2839 Succeeded=    1723 Failed=       0 Comitted=     4 Flushed=     1\r\nQueued=  385 Written=    4374 Succeeded=    2957 Failed=       0 Comitted=     8 Flushed=     3\r\nQueued=    0 Written=    5788 Succeeded=    4487 Failed=       0 Comitted=    12 Flushed=     3\r\nQueued=    0 Written=    7209 Succeeded=    5906 Failed=       0 Comitted=    16 Flushed=     4\r\nQueued=    0 Written=    8606 Succeeded=    7337 Failed=       0 Comitted=    20 Flushed=     5\r\nQueued=    0 Written=   10000 Succeeded=    8734 Failed=       0 Comitted=    24 Flushed=     7\r\nQueued=    0 Written=   11407 Succeeded=   10111 Failed=       0 Comitted=    28 Flushed=     7\r\nQueued=    0 Written=   12845 Succeeded=   11517 Failed=       0 Comitted=    32 Flushed=     8\r\nQueued=    0 Written=   14208 Succeeded=   12969 Failed=       0 Comitted=    36 Flushed=     9\r\nQueued=    0 Written=   15765 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    11\r\nQueued= 1449 Written=   15767 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    11\r\nQueued= 1449 Written=   15767 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    11\r\nQueued= 1449 Written=   15767 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    11\r\nQueued= 1449 Written=   15767 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    11\r\n...\r\nQueued= 1449 Written=   15767 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    11\r\nQueued= 1449 Written=   15767 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    11\r\nQueued= 1449 Written=   15767 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    11\r\nQueued= 1449 Written=   15767 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    11\r\nQueued= 1449 Written=   15767 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    11\r\nQueued= 1453 Written=   16897 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    12\r\nQueued= 2916 Written=   18243 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    14\r\nQueued= 4246 Written=   18564 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    14\r\nQueued= 4496 Written=   18814 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    16\r\nQueued= 4500 Written=   18818 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    16\r\nQueued= 4742 Written=   19060 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    17\r\nQueued= 4745 Written=   19063 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    18\r\nQueued= 4988 Written=   19306 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    19\r\nQueued= 4991 Written=   19309 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    20\r\nQueued= 5234 Written=   19552 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    21\r\nQueued= 5237 Written=   19555 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    22\r\nQueued= 5480 Written=   19798 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    24\r\nQueued= 5483 Written=   19801 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    24\r\nQueued= 5726 Written=   20044 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    26\r\nQueued= 5729 Written=   20047 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    26\r\nQueued= 5976 Written=   20294 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    27\r\nQueued= 5979 Written=   20297 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    28\r\nQueued= 6222 Written=   20540 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    29\r\nQueued= 6225 Written=   20543 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    30\r\nQueued= 6468 Written=   20786 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    31\r\nQueued= 6471 Written=   20789 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    32\r\nQueued= 6718 Written=   21036 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    33\r\nQueued= 6721 Written=   21039 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    34\r\nQueued= 6964 Written=   21282 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    36\r\nQueued= 6967 Written=   21285 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    36\r\nQueued= 7210 Written=   21528 Succeeded=   14317 Failed=       0 Comitted=    40 Flushed=    38\r\n```\r\n\r\nNotice that I didn't start ES again. Can you please test this with your PR?\r\n"}}
{"actor":{"display_login":"olivere"}}