{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/logstash/issues/4726","repository_url":"https://api.github.com/repos/elastic/logstash","labels_url":"https://api.github.com/repos/elastic/logstash/issues/4726/labels{/name}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/4726/comments","events_url":"https://api.github.com/repos/elastic/logstash/issues/4726/events","html_url":"https://github.com/elastic/logstash/issues/4726","id":136881028,"number":4726,"title":"Rate limit filter","user":{"login":"bobrik","id":89186,"avatar_url":"https://avatars.githubusercontent.com/u/89186?v=3","gravatar_id":"","url":"https://api.github.com/users/bobrik","html_url":"https://github.com/bobrik","followers_url":"https://api.github.com/users/bobrik/followers","following_url":"https://api.github.com/users/bobrik/following{/other_user}","gists_url":"https://api.github.com/users/bobrik/gists{/gist_id}","starred_url":"https://api.github.com/users/bobrik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bobrik/subscriptions","organizations_url":"https://api.github.com/users/bobrik/orgs","repos_url":"https://api.github.com/users/bobrik/repos","events_url":"https://api.github.com/users/bobrik/events{/privacy}","received_events_url":"https://api.github.com/users/bobrik/received_events","type":"User","site_admin":false},"labels":[{"id":90997545,"url":"https://api.github.com/repos/elastic/logstash/labels/enhancement","name":"enhancement","color":"009800","default":true},{"id":234923969,"url":"https://api.github.com/repos/elastic/logstash/labels/high%20hanging%20fruit","name":"high hanging fruit","color":"009800","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2016-02-27T05:25:46Z","updated_at":"2016-12-14T21:01:28Z","closed_at":null,"body":"Often times than not you don't really care if some hosts have thousands or millions of errors per second, the fact that they have more than 10 per second is enough to care about. Unfortunately, with logstash you cannot selectively drop messages when rate approaches levels when you don't care anymore.\n\nI propose to add \"ratelimit\" filter with the following settings:\n- `key` — key template for rate limit, could be something like `%{host}` or `%{host}/%{service}`.\n- `interval` — time interval to limit messages in, could be 1 minute or 10 minutes for example.\n- `limit` — max number of messages with given key per interval.\n\nIt can optionally add the number of dropped messages for key to the first message after `interval` reset.\n\nWhile I have high doubts that logstash is really suitable for the cases when your load exceeds capacity of what jruby can comfortably do in terms of requests per second, I am also lazy and want to reuse existing stuff that I don't have to write and/or support.\n\nThe plugin itself should be pretty trivial to implement from what I can tell.\n\nLet me know what you think.\n"},"comment":{"url":"https://api.github.com/repos/elastic/logstash/issues/comments/267155397","html_url":"https://github.com/elastic/logstash/issues/4726#issuecomment-267155397","issue_url":"https://api.github.com/repos/elastic/logstash/issues/4726","id":267155397,"user":{"login":"jordansissel","id":131818,"avatar_url":"https://avatars.githubusercontent.com/u/131818?v=3","gravatar_id":"","url":"https://api.github.com/users/jordansissel","html_url":"https://github.com/jordansissel","followers_url":"https://api.github.com/users/jordansissel/followers","following_url":"https://api.github.com/users/jordansissel/following{/other_user}","gists_url":"https://api.github.com/users/jordansissel/gists{/gist_id}","starred_url":"https://api.github.com/users/jordansissel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jordansissel/subscriptions","organizations_url":"https://api.github.com/users/jordansissel/orgs","repos_url":"https://api.github.com/users/jordansissel/repos","events_url":"https://api.github.com/users/jordansissel/events{/privacy}","received_events_url":"https://api.github.com/users/jordansissel/received_events","type":"User","site_admin":false},"created_at":"2016-12-14T21:01:28Z","updated_at":"2016-12-14T21:01:28Z","body":"@sherry-ger I still believe this can be solved with the throttle filter.\r\n\r\nThe throttle filter can tag events that are occurring more than a given capacity (1000 in 30 seconds) and you can combine that with the sleep filter (to slow down) or the drop filter (to drop) to achieve whatever you might need.\r\n\r\nOn the other hand, if throttle+drop+sleep filters are insufficient, please let me know."}}
{"actor":{"display_login":"jordansissel"}}