title: logstash multiline issue with java logs
body: <p>I am installing logstash with tomcat and having issue to grab java stack trace call, following is my config</p>
  
  <pre><code>input   {
   udp {
     type =&gt; "tomcat"
     port =&gt; "514"
     format =&gt; "plain"
  
   }
  
  }
  
  filter{
  multiline {
              pattern =&gt; "(^.+Exception.*)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
              what =&gt; "previous"
  }
  }
  </code></pre>
  
  <p>Here is sample logs of tomcat</p>
  
  <pre><code>2014-03-24 19:08:53,246 [thread-pool8] ERROR org.apache.catalina.core.ContainerBase.[engine].[localhost] - Exception Processing ErrorPage[errorCode=500, location=/error/error500.jsp]
  org.apache.jasper.JasperException: java.lang.NullPointerException
          at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:549)
          at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:470)
          at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
          at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
          at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
          at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:489)
          at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:412)
          at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
          at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:467)
          at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:338)
          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:203)
          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
          at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
          at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
          at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1686)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:744)
  Caused by: java.lang.NullPointerException
          at org.apache.jsp.error.error500_jsp._jspService(error500_jsp.java:266)
          at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
          at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
          ... 21 more
  </code></pre>
  
  <p>I have tired all combination and none of work :( i don't know how it works for other folks.. </p>
  
  <h1>EDIT:</h1>
  
  <p>I have tried following and it didn't work too</p>
  
  <pre><code>pattern =&gt; "^%{TIMESTAMP_ISO8601}"
         negate =&gt; true
  </code></pre>
  
  <p>following is result</p>
  
  <pre><code>filter received {:event=&gt;{"message"=&gt;"&lt;139&gt;2014-03-24 21:07:58,908 [] [] [thread-pool4] ERROR org.apache.catalina.core.ContainerBase.[engine].[localhost] - Exception Processing ErrorPage[errorCode=500, location=/error/error500.jsp]\n", "@version"=&gt;"1", "@timestamp"=&gt;"2014-03-25T01:07:59.128Z", "type"=&gt;"tomcat", "host"=&gt;"10.3.68.22"}, :level=&gt;:debug, :file=&gt;"(eval)", :line=&gt;"15"}
  &lt;139&gt;2014-03-24 21:07:58,908 [] [] [thread-pool4] ERROR org.apache.catalina.core.ContainerBase.[engine].[localhost] - Exception Processing ErrorPage[errorCode=500, location=/error/error500.jsp]
   {:pattern=&gt;"^%{TIMESTAMP_ISO8601} ", :match=&gt;false, :negate=&gt;true, :level=&gt;:debug, :file=&gt;"logstash/filters/multiline.rb", :line=&gt;"160"}
  filter received {:event=&gt;{"message"=&gt;"&lt;139&gt;org.apache.jasper.JasperException: java.lang.NullPointerException", "@version"=&gt;"1", "@timestamp"=&gt;"2014-03-25T01:07:59.131Z", "type"=&gt;"tomcat", "host"=&gt;"10.3.68.22"}, :level=&gt;:debug, :file=&gt;"(eval)", :line=&gt;"15"}
  &lt;139&gt;org.apache.jasper.JasperException: java.lang.NullPointerException {:pattern=&gt;"^%{TIMESTAMP_ISO8601} ", :match=&gt;false, :negate=&gt;true, :level=&gt;:debug, :file=&gt;"logstash/filters/multiline.rb", :line=&gt;"160"}
  filter received {:event=&gt;{"message"=&gt;"&lt;139&gt;    at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:549)", "@version"=&gt;"1", "@timestamp"=&gt;"2014-03-25T01:07:59.134Z", "type"=&gt;"tomcat", "host"=&gt;"10.3.68.22"}, :level=&gt;:debug, :file=&gt;"(eval)", :line=&gt;"15"}
  </code></pre>
  
  <h1>UPDATE:</h1>
  
  <p>I ran logstash in debug mode with udp protocol and some strange number <code>&lt;139&gt;</code> coming in <code>@messages</code> see following output of debug, if i use <code>nc</code> command to send sample logs it works but somehow with tomcat syslog its not working </p>
  
  <pre><code>{
         "message" =&gt; "&lt;139&gt;2014-03-28 13:52:25,548 [] [] [thread-pool2] ERROR org.apache.catalina.core.ContainerBase.[engine].[localhost] - Exception Processing ErrorPage[errorCode=500, location=/error/error500.jsp]\n",
        "@version" =&gt; "1",
      "@timestamp" =&gt; "2014-03-28T17:52:26.116Z",
            "host" =&gt; "10.3.68.22"
  }
  {
         "message" =&gt; "&lt;139&gt;org.apache.jasper.JasperException: java.lang.NullPointerException",
        "@version" =&gt; "1",
      "@timestamp" =&gt; "2014-03-28T17:52:26.134Z",
            "host" =&gt; "10.3.68.22"
  }
  {
         "message" =&gt; "&lt;139&gt;    at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:549)",
        "@version" =&gt; "1",
      "@timestamp" =&gt; "2014-03-28T17:52:26.151Z",
            "host" =&gt; "10.3.68.22"
  }
  {
         "message" =&gt; "&lt;139&gt;    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:470)",
        "@version" =&gt; "1",
      "@timestamp" =&gt; "2014-03-28T17:52:26.166Z",
            "host" =&gt; "10.3.68.22"
  }
  {
         "message" =&gt; "&lt;139&gt;    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)",
        "@version" =&gt; "1",
      "@timestamp" =&gt; "2014-03-28T17:52:26.183Z",
            "host" =&gt; "10.3.68.22"
  }
  </code></pre>