{"action":"opened","issue":{"url":"https://api.github.com/repos/logstash-plugins/logstash-filter-multiline/issues/29","repository_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-multiline","labels_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-multiline/issues/29/labels{/name}","comments_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-multiline/issues/29/comments","events_url":"https://api.github.com/repos/logstash-plugins/logstash-filter-multiline/issues/29/events","html_url":"https://github.com/logstash-plugins/logstash-filter-multiline/issues/29","id":158195669,"number":29,"title":"The last multiline event doesn't get flushed","user":{"login":"mark-ritchie","id":4214183,"avatar_url":"https://avatars.githubusercontent.com/u/4214183?v=3","gravatar_id":"","url":"https://api.github.com/users/mark-ritchie","html_url":"https://github.com/mark-ritchie","followers_url":"https://api.github.com/users/mark-ritchie/followers","following_url":"https://api.github.com/users/mark-ritchie/following{/other_user}","gists_url":"https://api.github.com/users/mark-ritchie/gists{/gist_id}","starred_url":"https://api.github.com/users/mark-ritchie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mark-ritchie/subscriptions","organizations_url":"https://api.github.com/users/mark-ritchie/orgs","repos_url":"https://api.github.com/users/mark-ritchie/repos","events_url":"https://api.github.com/users/mark-ritchie/events{/privacy}","received_events_url":"https://api.github.com/users/mark-ritchie/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-06-02T17:26:34Z","updated_at":"2016-06-02T17:26:34Z","closed_at":null,"body":"I'm using logstash to collect events which are key values pairs with one pair per line and blank line between events.  I've used the multiline codec to collect the lines together into one event and the kv filter to parse the pairs.  This all seems to work fine except that the last event is not output from logstash.  Input one event, get none out.  Input two and get one.  Input 45 and get 44. :-(\r\n\r\nSo far I've been able to work around the problem by appending a dummy event to the input stream however there must be a better way.\r\n\r\nFor all general issues, please provide the following details for fast resolution:\r\n- Version: logstash-2.3.2\r\n- Operating System: OSX 10.11.2\r\n- Config File (if you have sensitive info, please remove it): inline below\r\n- Sample Data: inline below\r\n- Steps to Reproduce: below\r\n\r\nFor example, the first event is parsed and output however the second one is not:\r\n$ echo -en \"foo=some\\nbar=thing\\n\\nblat=other\\nspam=thang\\n\\n\" | logstash-2.3.2/bin/logstash agent -e 'input { stdin { codec => multiline { pattern => \"^$\" negate => true what => \"next\" } } } filter { kv { field_split => \"\\n\" } }'\r\nSettings: Default pipeline workers: 8\r\nPipeline main started\r\n{\r\n    \"@timestamp\" => \"2016-06-02T17:10:57.934Z\",\r\n       \"message\" => \"foo=some\\nbar=thing\\n\",\r\n      \"@version\" => \"1\",\r\n          \"tags\" => [\r\n        [0] \"multiline\"\r\n    ],\r\n          \"host\" => \"stroke.local\",\r\n           \"foo\" => \"some\",\r\n           \"bar\" => \"thing\"\r\n}\r\nPipeline main has been shutdown\r\nstopping pipeline {:id=>\"main\"}\r\n$ \r\n\r\nI can work around the issue by appending a dummy event to the input stream with awk:\r\n$ echo -en \"foo=some\\nbar=thing\\n\\nblat=other\\nspam=thang\\n\\n\" | awk '{ print } END { print \"ThIs=EmPtYEvEnT\\n\" }' | logstash-2.3.2/bin/logstash agent -e 'input { stdin { codec => multiline { pattern => \"^$\" negate => true what => \"next\" } } } filter { kv { field_split => \"\\n\" } }'\r\nSettings: Default pipeline workers: 8\r\nPipeline main started\r\n{\r\n    \"@timestamp\" => \"2016-06-02T17:19:04.768Z\",\r\n       \"message\" => \"foo=some\\nbar=thing\\n\",\r\n      \"@version\" => \"1\",\r\n          \"tags\" => [\r\n        [0] \"multiline\"\r\n    ],\r\n          \"host\" => \"stroke.local\",\r\n           \"foo\" => \"some\",\r\n           \"bar\" => \"thing\"\r\n}\r\n{\r\n    \"@timestamp\" => \"2016-06-02T17:19:04.788Z\",\r\n       \"message\" => \"blat=other\\nspam=thang\\n\",\r\n      \"@version\" => \"1\",\r\n          \"tags\" => [\r\n        [0] \"multiline\"\r\n    ],\r\n          \"host\" => \"stroke.local\",\r\n          \"blat\" => \"other\",\r\n          \"spam\" => \"thang\"\r\n}\r\nPipeline main has been shutdown\r\nstopping pipeline {:id=>\"main\"}\r\n$ \r\n\r\nI'm thinking that this is an issue with the multiline codec because when I process the same input without the codec, I get all the lines processed, all be it as different events and that's not what I need.\r\n$ echo -en \"foo=some\\nbar=thing\\n\\nblat=other\\nspam=thang\\n\\n\" | logstash-2.3.2/bin/logstash agent | grep message\r\n{:timestamp=>\"2016-06-02T13:23:35.425000-0400\", :message=>\"Pipeline main started\"}\r\n       \"message\" => \"foo=some\",\r\n       \"message\" => \"bar=thing\",\r\n       \"message\" => \"\",\r\n       \"message\" => \"blat=other\",\r\n       \"message\" => \"spam=thang\",\r\n       \"message\" => \"\",\r\n{:timestamp=>\"2016-06-02T13:23:35.540000-0400\", :message=>\"Pipeline main has been shutdown\"}\r\n{:timestamp=>\"2016-06-02T13:23:38.440000-0400\", :message=>\"stopping pipeline\", :id=>\"main\"}\r\n$ \r\n\r\nThanks!"}}
{"actor":{"display_login":"mark-ritchie"}}