{"action":"created","issue":{"url":"https://api.github.com/repos/balabit/syslog-ng/issues/2063","repository_url":"https://api.github.com/repos/balabit/syslog-ng","labels_url":"https://api.github.com/repos/balabit/syslog-ng/issues/2063/labels{/name}","comments_url":"https://api.github.com/repos/balabit/syslog-ng/issues/2063/comments","events_url":"https://api.github.com/repos/balabit/syslog-ng/issues/2063/events","html_url":"https://github.com/balabit/syslog-ng/pull/2063","id":327124216,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkxMDAxNTE5","number":2063,"title":"LogThreadedDestDriver batching","user":{"login":"bazsi","id":714102,"node_id":"MDQ6VXNlcjcxNDEwMg==","avatar_url":"https://avatars3.githubusercontent.com/u/714102?v=4","gravatar_id":"","url":"https://api.github.com/users/bazsi","html_url":"https://github.com/bazsi","followers_url":"https://api.github.com/users/bazsi/followers","following_url":"https://api.github.com/users/bazsi/following{/other_user}","gists_url":"https://api.github.com/users/bazsi/gists{/gist_id}","starred_url":"https://api.github.com/users/bazsi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bazsi/subscriptions","organizations_url":"https://api.github.com/users/bazsi/orgs","repos_url":"https://api.github.com/users/bazsi/repos","events_url":"https://api.github.com/users/bazsi/events{/privacy}","received_events_url":"https://api.github.com/users/bazsi/received_events","type":"User","site_admin":false},"labels":[{"id":104723147,"node_id":"MDU6TGFiZWwxMDQ3MjMxNDc=","url":"https://api.github.com/repos/balabit/syslog-ng/labels/in%20progress","name":"in progress","color":"207de5","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":66,"created_at":"2018-05-28T22:01:48Z","updated_at":"2018-07-16T18:27:19Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/balabit/syslog-ng/pulls/2063","html_url":"https://github.com/balabit/syslog-ng/pull/2063","diff_url":"https://github.com/balabit/syslog-ng/pull/2063.diff","patch_url":"https://github.com/balabit/syslog-ng/pull/2063.patch"},"body":"This is still a work in progress, however the basic function did work and I wanted to solicit feedback on it.\r\n\r\nSo this basically adds a batched API to LogThreadedDestDriver and enables it in the HTTP destination,\r\nwhich would send batches of 10 messages in a single HTTP transaction.\r\n\r\nThe bulk of the patch is a largish cleanup of LogThreadedDestDriver with lots of non-functional changes, renaming functions, moving them around, making stuff easier to navigate there.\r\n\r\nThe infrastructure to support batching is pretty basic: you can now return WORKER_INSERT_RESULT_QUEUED to add the current message into a batch, whose size is tracked by LogThreadedDestDriver. Any time the driver returns SUCCESS, the entire batch is considered to be sent. If the driver returns an error, the entire batch is retried and dropped after 3 retries. I have also started to add an explicit acknowledgement model, but that's the most foggy at this point.\r\n\r\nNOTE: this is not final yet, some of the stuff is not configurable and a number of corner cases not yet thought through. Not to mention that I did the final changes, rebased and writing this note after roughly being awake for ~24 hours.\r\n"},"comment":{"url":"https://api.github.com/repos/balabit/syslog-ng/issues/comments/405336282","html_url":"https://github.com/balabit/syslog-ng/pull/2063#issuecomment-405336282","issue_url":"https://api.github.com/repos/balabit/syslog-ng/issues/2063","id":405336282,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNTMzNjI4Mg==","user":{"login":"bazsi","id":714102,"node_id":"MDQ6VXNlcjcxNDEwMg==","avatar_url":"https://avatars3.githubusercontent.com/u/714102?v=4","gravatar_id":"","url":"https://api.github.com/users/bazsi","html_url":"https://github.com/bazsi","followers_url":"https://api.github.com/users/bazsi/followers","following_url":"https://api.github.com/users/bazsi/following{/other_user}","gists_url":"https://api.github.com/users/bazsi/gists{/gist_id}","starred_url":"https://api.github.com/users/bazsi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bazsi/subscriptions","organizations_url":"https://api.github.com/users/bazsi/orgs","repos_url":"https://api.github.com/users/bazsi/repos","events_url":"https://api.github.com/users/bazsi/events{/privacy}","received_events_url":"https://api.github.com/users/bazsi/received_events","type":"User","site_admin":false},"created_at":"2018-07-16T18:27:19Z","updated_at":"2018-07-16T18:27:19Z","author_association":"MEMBER","body":"thanks a lot for helping out with @kira-syslogng "}}
{"actor":{"display_login":"bazsi"}}