{"action":"created","comment":{"url":"https://api.github.com/repos/elastic/logstash/pulls/comments/85544969","pull_request_review_id":6256482,"id":85544969,"diff_hunk":"@@ -0,0 +1,46 @@\n+require_relative '../framework/fixture'\n+require_relative '../framework/settings'\n+require_relative '../services/logstash_service'\n+require_relative '../framework/helpers'\n+require \"logstash/devutils/rspec/spec_helper\"\n+require \"yaml\"\n+\n+describe \"Test Logstash Slowlog\" do\n+  before(:all) {\n+    @fixture = Fixture.new(__FILE__)\n+    # used in multiple LS tests\n+    @ls = @fixture.get_service(\"logstash\")\n+  }\n+\n+  after(:all) {\n+    @fixture.teardown\n+  }\n+\n+  before(:each) {\n+    # backup the application settings file -- logstash.yml\n+    FileUtils.cp(@ls.application_settings_file, \"#{@ls.application_settings_file}.original\")","path":"qa/integration/specs/slowlog_spec.rb","position":21,"original_position":21,"commit_id":"fbf11f49ad17fafab53d040c74b81cb2fe07d60f","original_commit_id":"fbf11f49ad17fafab53d040c74b81cb2fe07d60f","user":{"login":"suyograo","id":1595958,"avatar_url":"https://avatars.githubusercontent.com/u/1595958?v=3","gravatar_id":"","url":"https://api.github.com/users/suyograo","html_url":"https://github.com/suyograo","followers_url":"https://api.github.com/users/suyograo/followers","following_url":"https://api.github.com/users/suyograo/following{/other_user}","gists_url":"https://api.github.com/users/suyograo/gists{/gist_id}","starred_url":"https://api.github.com/users/suyograo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suyograo/subscriptions","organizations_url":"https://api.github.com/users/suyograo/orgs","repos_url":"https://api.github.com/users/suyograo/repos","events_url":"https://api.github.com/users/suyograo/events{/privacy}","received_events_url":"https://api.github.com/users/suyograo/received_events","type":"User","site_admin":false},"body":"Don't need to do this in this particular test","created_at":"2016-10-28T14:47:12Z","updated_at":"2016-10-28T14:47:12Z","html_url":"https://github.com/elastic/logstash/pull/6128#discussion_r85544969","pull_request_url":"https://api.github.com/repos/elastic/logstash/pulls/6128","_links":{"self":{"href":"https://api.github.com/repos/elastic/logstash/pulls/comments/85544969"},"html":{"href":"https://github.com/elastic/logstash/pull/6128#discussion_r85544969"},"pull_request":{"href":"https://api.github.com/repos/elastic/logstash/pulls/6128"}}},"pull_request":{"url":"https://api.github.com/repos/elastic/logstash/pulls/6128","id":91100405,"html_url":"https://github.com/elastic/logstash/pull/6128","diff_url":"https://github.com/elastic/logstash/pull/6128.diff","patch_url":"https://github.com/elastic/logstash/pull/6128.patch","issue_url":"https://api.github.com/repos/elastic/logstash/issues/6128","number":6128,"state":"open","locked":false,"title":"Slowlog For Filters in Logstash","user":{"login":"talevy","id":388837,"avatar_url":"https://avatars.githubusercontent.com/u/388837?v=3","gravatar_id":"","url":"https://api.github.com/users/talevy","html_url":"https://github.com/talevy","followers_url":"https://api.github.com/users/talevy/followers","following_url":"https://api.github.com/users/talevy/following{/other_user}","gists_url":"https://api.github.com/users/talevy/gists{/gist_id}","starred_url":"https://api.github.com/users/talevy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talevy/subscriptions","organizations_url":"https://api.github.com/users/talevy/orgs","repos_url":"https://api.github.com/users/talevy/repos","events_url":"https://api.github.com/users/talevy/events{/privacy}","received_events_url":"https://api.github.com/users/talevy/received_events","type":"User","site_admin":false},"body":"Ever wish you can find out what is taking a long time in your Logstash Pipeline?\r\n\r\nWith the new addition of the [Monitoring API](https://www.elastic.co/guide/en/logstash/5.0/monitoring.html), you can introspect Logstash at the thread level, but there is no easy way to see what actions are taking long. Slowlog in Logstash is the first step in catching some important plugin actions that are slowing the pipeline down.\r\n\r\nFor demonstration, here is a sample pipeline with a test filter that will `sleep` for varying durations that, at times, will trigger a slowlog event to be logged.\r\n\r\n```\r\ninput {\r\n  generator { count => 1 }\r\n}\r\n\r\nfilter {\r\n  sleep {\r\n    time => \"2\"\r\n  }\r\n}\r\noutput {\r\n  null {}\r\n}\r\n```\r\n\r\nNow you can configure a slowlog to log to `/path/to/logs/logstash-slowlog-plain-YYYY-MM-dd.log`\r\n\r\nSlowlog is currently only configured on filters. If any filter action on an event exceeds a pre-defined threshold, the instance will be logged.\r\nThe thresholds are configured in `logstash.yml` as the following configuration keys:\r\n\r\n```\r\nslowlog.threshold.warn (default: -1)\r\nslowlog.threshold.info (default: -1)\r\nslowlog.threshold.debug (default: -1)\r\nslowlog.threshold.trace (default: -1)\r\n```\r\n\r\nThese options take in a time value similar to [Elasticsearch's Slowlog](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-slowlog.html).  By default, these values are set to `-1` to represent an infinite threshold where no slowlog will be invoked.\r\n\r\nHere is a sample:\r\n\r\n```\r\nslowlog.threshold.warn: 2s\r\nslowlog.threshold.info 1s\r\nslowlog.threshold.debug 500ms\r\nslowlog.threshold.trace 100ms\r\n```\r\n\r\n### Contents of Slowlog\r\nAssuming the above pipeline and the example configuration were run in tandem, we may find our slowlog filled with the following content in the case of a filter taking a little over 2 seconds. The log line will include the offending event and the time it took to process within the filter\r\n\r\n```\r\n...\r\n[2016-10-27T13:48:19,308][WARN ][slowlog.logstash.filters.sleep] event processing time {:took_in_nanos=>1000614130, :event=>\"{\\\"sequence\\\":1,\\\"@timestamp\\\":\\\"2016-10-27T20:48:18.150Z\\\",\\\"@version\\\":\\\"1\\\",\\\"host\\\":\\\"elasticbox-2.local\\\",\\\"mes\r\nsage\\\":\\\"Hello world!\\\"}\"}\r\n```\r\n\r\n### Old efforts forgotten?\r\n\r\nThere were some previous efforts to introduce this feature to Logstash. This is picking up that torch and moving forward with the task.\r\n\r\nfor reference to older discussions, please reference:\r\n\r\n| Feature Issue | https://github.com/elastic/logstash/issues/5733 |\r\n|----------|------|\r\n| The PR | https://github.com/elastic/logstash/pull/5886 |\r\n\r\nI hope that many of the discussion from there will continue here. One major difference between\r\nthat PR and this is that this only sets up the slowlog functionality itself with no consideration of the API for tracking \"top-K\" style metrics of slow operations. That can continue in follow-up commits.","created_at":"2016-10-26T22:22:50Z","updated_at":"2016-10-28T14:47:12Z","closed_at":null,"merged_at":null,"merge_commit_sha":"f6ab8d418363355c2a8967da1e133c765e8f287e","assignee":{"login":"suyograo","id":1595958,"avatar_url":"https://avatars.githubusercontent.com/u/1595958?v=3","gravatar_id":"","url":"https://api.github.com/users/suyograo","html_url":"https://github.com/suyograo","followers_url":"https://api.github.com/users/suyograo/followers","following_url":"https://api.github.com/users/suyograo/following{/other_user}","gists_url":"https://api.github.com/users/suyograo/gists{/gist_id}","starred_url":"https://api.github.com/users/suyograo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suyograo/subscriptions","organizations_url":"https://api.github.com/users/suyograo/orgs","repos_url":"https://api.github.com/users/suyograo/repos","events_url":"https://api.github.com/users/suyograo/events{/privacy}","received_events_url":"https://api.github.com/users/suyograo/received_events","type":"User","site_admin":false},"assignees":[{"login":"suyograo","id":1595958,"avatar_url":"https://avatars.githubusercontent.com/u/1595958?v=3","gravatar_id":"","url":"https://api.github.com/users/suyograo","html_url":"https://github.com/suyograo","followers_url":"https://api.github.com/users/suyograo/followers","following_url":"https://api.github.com/users/suyograo/following{/other_user}","gists_url":"https://api.github.com/users/suyograo/gists{/gist_id}","starred_url":"https://api.github.com/users/suyograo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suyograo/subscriptions","organizations_url":"https://api.github.com/users/suyograo/orgs","repos_url":"https://api.github.com/users/suyograo/repos","events_url":"https://api.github.com/users/suyograo/events{/privacy}","received_events_url":"https://api.github.com/users/suyograo/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/elastic/logstash/pulls/6128/commits","review_comments_url":"https://api.github.com/repos/elastic/logstash/pulls/6128/comments","review_comment_url":"https://api.github.com/repos/elastic/logstash/pulls/comments{/number}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/6128/comments","statuses_url":"https://api.github.com/repos/elastic/logstash/statuses/fbf11f49ad17fafab53d040c74b81cb2fe07d60f","head":{"label":"talevy:slowlog","ref":"slowlog","sha":"fbf11f49ad17fafab53d040c74b81cb2fe07d60f","user":{"login":"talevy","id":388837,"avatar_url":"https://avatars.githubusercontent.com/u/388837?v=3","gravatar_id":"","url":"https://api.github.com/users/talevy","html_url":"https://github.com/talevy","followers_url":"https://api.github.com/users/talevy/followers","following_url":"https://api.github.com/users/talevy/following{/other_user}","gists_url":"https://api.github.com/users/talevy/gists{/gist_id}","starred_url":"https://api.github.com/users/talevy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talevy/subscriptions","organizations_url":"https://api.github.com/users/talevy/orgs","repos_url":"https://api.github.com/users/talevy/repos","events_url":"https://api.github.com/users/talevy/events{/privacy}","received_events_url":"https://api.github.com/users/talevy/received_events","type":"User","site_admin":false},"repo":{"id":25496528,"name":"logstash","full_name":"talevy/logstash","owner":{"login":"talevy","id":388837,"avatar_url":"https://avatars.githubusercontent.com/u/388837?v=3","gravatar_id":"","url":"https://api.github.com/users/talevy","html_url":"https://github.com/talevy","followers_url":"https://api.github.com/users/talevy/followers","following_url":"https://api.github.com/users/talevy/following{/other_user}","gists_url":"https://api.github.com/users/talevy/gists{/gist_id}","starred_url":"https://api.github.com/users/talevy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talevy/subscriptions","organizations_url":"https://api.github.com/users/talevy/orgs","repos_url":"https://api.github.com/users/talevy/repos","events_url":"https://api.github.com/users/talevy/events{/privacy}","received_events_url":"https://api.github.com/users/talevy/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/talevy/logstash","description":"logstash - logs/event transport, processing, management, search.","fork":true,"url":"https://api.github.com/repos/talevy/logstash","forks_url":"https://api.github.com/repos/talevy/logstash/forks","keys_url":"https://api.github.com/repos/talevy/logstash/keys{/key_id}","collaborators_url":"https://api.github.com/repos/talevy/logstash/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/talevy/logstash/teams","hooks_url":"https://api.github.com/repos/talevy/logstash/hooks","issue_events_url":"https://api.github.com/repos/talevy/logstash/issues/events{/number}","events_url":"https://api.github.com/repos/talevy/logstash/events","assignees_url":"https://api.github.com/repos/talevy/logstash/assignees{/user}","branches_url":"https://api.github.com/repos/talevy/logstash/branches{/branch}","tags_url":"https://api.github.com/repos/talevy/logstash/tags","blobs_url":"https://api.github.com/repos/talevy/logstash/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/talevy/logstash/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/talevy/logstash/git/refs{/sha}","trees_url":"https://api.github.com/repos/talevy/logstash/git/trees{/sha}","statuses_url":"https://api.github.com/repos/talevy/logstash/statuses/{sha}","languages_url":"https://api.github.com/repos/talevy/logstash/languages","stargazers_url":"https://api.github.com/repos/talevy/logstash/stargazers","contributors_url":"https://api.github.com/repos/talevy/logstash/contributors","subscribers_url":"https://api.github.com/repos/talevy/logstash/subscribers","subscription_url":"https://api.github.com/repos/talevy/logstash/subscription","commits_url":"https://api.github.com/repos/talevy/logstash/commits{/sha}","git_commits_url":"https://api.github.com/repos/talevy/logstash/git/commits{/sha}","comments_url":"https://api.github.com/repos/talevy/logstash/comments{/number}","issue_comment_url":"https://api.github.com/repos/talevy/logstash/issues/comments{/number}","contents_url":"https://api.github.com/repos/talevy/logstash/contents/{+path}","compare_url":"https://api.github.com/repos/talevy/logstash/compare/{base}...{head}","merges_url":"https://api.github.com/repos/talevy/logstash/merges","archive_url":"https://api.github.com/repos/talevy/logstash/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/talevy/logstash/downloads","issues_url":"https://api.github.com/repos/talevy/logstash/issues{/number}","pulls_url":"https://api.github.com/repos/talevy/logstash/pulls{/number}","milestones_url":"https://api.github.com/repos/talevy/logstash/milestones{/number}","notifications_url":"https://api.github.com/repos/talevy/logstash/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/talevy/logstash/labels{/name}","releases_url":"https://api.github.com/repos/talevy/logstash/releases{/id}","deployments_url":"https://api.github.com/repos/talevy/logstash/deployments","created_at":"2014-10-21T01:56:42Z","updated_at":"2016-10-24T21:39:18Z","pushed_at":"2016-10-27T20:51:58Z","git_url":"git://github.com/talevy/logstash.git","ssh_url":"git@github.com:talevy/logstash.git","clone_url":"https://github.com/talevy/logstash.git","svn_url":"https://github.com/talevy/logstash","homepage":"http://logstash.net/","size":75844,"stargazers_count":1,"watchers_count":1,"language":"Ruby","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":0,"forks":1,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"elastic:master","ref":"master","sha":"b43a9cf11da53f7589eef87fc4e438556c4f4cc8","user":{"login":"elastic","id":6764390,"avatar_url":"https://avatars.githubusercontent.com/u/6764390?v=3","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"repo":{"id":1090311,"name":"logstash","full_name":"elastic/logstash","owner":{"login":"elastic","id":6764390,"avatar_url":"https://avatars.githubusercontent.com/u/6764390?v=3","gravatar_id":"","url":"https://api.github.com/users/elastic","html_url":"https://github.com/elastic","followers_url":"https://api.github.com/users/elastic/followers","following_url":"https://api.github.com/users/elastic/following{/other_user}","gists_url":"https://api.github.com/users/elastic/gists{/gist_id}","starred_url":"https://api.github.com/users/elastic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elastic/subscriptions","organizations_url":"https://api.github.com/users/elastic/orgs","repos_url":"https://api.github.com/users/elastic/repos","events_url":"https://api.github.com/users/elastic/events{/privacy}","received_events_url":"https://api.github.com/users/elastic/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/elastic/logstash","description":"logstash - transport and process your logs, events, or other data","fork":false,"url":"https://api.github.com/repos/elastic/logstash","forks_url":"https://api.github.com/repos/elastic/logstash/forks","keys_url":"https://api.github.com/repos/elastic/logstash/keys{/key_id}","collaborators_url":"https://api.github.com/repos/elastic/logstash/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/elastic/logstash/teams","hooks_url":"https://api.github.com/repos/elastic/logstash/hooks","issue_events_url":"https://api.github.com/repos/elastic/logstash/issues/events{/number}","events_url":"https://api.github.com/repos/elastic/logstash/events","assignees_url":"https://api.github.com/repos/elastic/logstash/assignees{/user}","branches_url":"https://api.github.com/repos/elastic/logstash/branches{/branch}","tags_url":"https://api.github.com/repos/elastic/logstash/tags","blobs_url":"https://api.github.com/repos/elastic/logstash/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/elastic/logstash/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/elastic/logstash/git/refs{/sha}","trees_url":"https://api.github.com/repos/elastic/logstash/git/trees{/sha}","statuses_url":"https://api.github.com/repos/elastic/logstash/statuses/{sha}","languages_url":"https://api.github.com/repos/elastic/logstash/languages","stargazers_url":"https://api.github.com/repos/elastic/logstash/stargazers","contributors_url":"https://api.github.com/repos/elastic/logstash/contributors","subscribers_url":"https://api.github.com/repos/elastic/logstash/subscribers","subscription_url":"https://api.github.com/repos/elastic/logstash/subscription","commits_url":"https://api.github.com/repos/elastic/logstash/commits{/sha}","git_commits_url":"https://api.github.com/repos/elastic/logstash/git/commits{/sha}","comments_url":"https://api.github.com/repos/elastic/logstash/comments{/number}","issue_comment_url":"https://api.github.com/repos/elastic/logstash/issues/comments{/number}","contents_url":"https://api.github.com/repos/elastic/logstash/contents/{+path}","compare_url":"https://api.github.com/repos/elastic/logstash/compare/{base}...{head}","merges_url":"https://api.github.com/repos/elastic/logstash/merges","archive_url":"https://api.github.com/repos/elastic/logstash/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/elastic/logstash/downloads","issues_url":"https://api.github.com/repos/elastic/logstash/issues{/number}","pulls_url":"https://api.github.com/repos/elastic/logstash/pulls{/number}","milestones_url":"https://api.github.com/repos/elastic/logstash/milestones{/number}","notifications_url":"https://api.github.com/repos/elastic/logstash/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/elastic/logstash/labels{/name}","releases_url":"https://api.github.com/repos/elastic/logstash/releases{/id}","deployments_url":"https://api.github.com/repos/elastic/logstash/deployments","created_at":"2010-11-18T01:31:50Z","updated_at":"2016-10-28T14:24:41Z","pushed_at":"2016-10-28T13:55:43Z","git_url":"git://github.com/elastic/logstash.git","ssh_url":"git@github.com:elastic/logstash.git","clone_url":"https://github.com/elastic/logstash.git","svn_url":"https://github.com/elastic/logstash","homepage":"https://www.elastic.co/products/logstash","size":90924,"stargazers_count":6609,"watchers_count":6609,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1933,"mirror_url":null,"open_issues_count":857,"forks":1933,"open_issues":857,"watchers":6609,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/elastic/logstash/pulls/6128"},"html":{"href":"https://github.com/elastic/logstash/pull/6128"},"issue":{"href":"https://api.github.com/repos/elastic/logstash/issues/6128"},"comments":{"href":"https://api.github.com/repos/elastic/logstash/issues/6128/comments"},"review_comments":{"href":"https://api.github.com/repos/elastic/logstash/pulls/6128/comments"},"review_comment":{"href":"https://api.github.com/repos/elastic/logstash/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/elastic/logstash/pulls/6128/commits"},"statuses":{"href":"https://api.github.com/repos/elastic/logstash/statuses/fbf11f49ad17fafab53d040c74b81cb2fe07d60f"}}}}
{"actor":{"display_login":"suyograo"}}