title: Docker syslog driver with multiline parsing in logstash
body: <p>I am forwarding my docker logs via the syslog drivers to logstash.  This works great for normal log lines, but having issues with multilines.  The issue I am running into is that the docker log forwarding adds the syslog message format to each log line.  If I use the logstash filter multiline (which logstash does not recommend), I can successfully combine the multilines and remove the syslog messages on the additional lines...however, this is not thread safe. I cannot get the logic to work via an input codec which is what logstash recommends.</p>
  
  <p>So for example:</p>
  
  <p><strong>Docker command:</strong></p>
  
  <pre><code>docker run --rm -it \
        --log-driver syslog \
        --log-opt syslog-address=tcp://localhost:15008 \
        helloWorld:latest
  </code></pre>
  
  <p><strong>Logs in docker container</strong>:</p>
  
  <pre><code>Log message A
  &lt;&lt;ML&gt;&gt; Log message B
    more B1
    more B2
    more B3
  Log message C
  </code></pre>
  
  <p><strong>Logs as received into logstash</strong></p>
  
  <pre><code>&lt;30&gt;Jul 13 16:04:36 [1290]: Log message A
  &lt;30&gt;Jul 13 16:04:37 [1290]: &lt;&lt;ML&gt;&gt; Log message B
  &lt;30&gt;Jul 13 16:04:38 [1290]:  more B1
  &lt;30&gt;Jul 13 16:04:39 [1290]:  more B2
  &lt;30&gt;Jul 13 16:04:40 [1290]:  more B3
  &lt;30&gt;Jul 13 16:04:41 [1290]:Log message C
  </code></pre>
  
  <p>Now I can get everything to parse as I want using the following filter:</p>
  
  <p><strong>logstash filter multiline</strong></p>
  
  <pre><code>input { 
    tcp {
     port =&gt; 15008
     type =&gt; "multiline"
   }
  }
  
  filter {
    if ( [type] == "multiline") {
      grok {
        match =&gt; { "message" =&gt; [
          "^&lt;(?&lt;ignore&gt;\d*)&gt;(?&lt;syslogDateTime&gt;[\S]*)\s\[(?&lt;pid&gt;\d*)\]:.(?&lt;newMessage&gt;[\s\S]*)"
        ]}
      }
  
      multiline {
        pattern =&gt; "^[\s\S]*\&lt;\&lt;[M][L]\&gt;\&gt;"
        negate =&gt; true
        what =&gt; "previous"
        source =&gt; "newMessage"
        stream_identity =&gt; "%{host}.%{pid}"
      }
  }
  </code></pre>
  
  <p>This is exactly what I want in my logstash messages</p>
  
  <p><strong>output</strong></p>
  
  <pre><code>message: Log message A
  message: &lt;&lt;ML&gt;&gt; Log message B more B1 more B2 more B3
  message: Log message C
  </code></pre>
  
  <p>However, that runs for a few minutes...but then hangs and stops processing</p>
  
  <p>Trying to get it to work via the codec multiline which is logstash recommendation</p>
  
  <p><strong>logstash codec multiline</strong></p>
  
  <pre><code> input { 
        tcp {
         port =&gt; 15008
         type =&gt; "multiline"
         codec =&gt; multiline {
           pattern =&gt; "^[\s\S]*\&lt;\&lt;[M][L]\&gt;\&gt;"
           negate =&gt; true
           what =&gt; "previous"
         }
       }
      }
  
      filter {
        if ( [type] == "multiline") {
          grok {
            match =&gt; { "message" =&gt; [
              "^&lt;(?&lt;ignore&gt;\d*)&gt;(?&lt;syslogDateTime&gt;[\S]*)\s\[(?&lt;pid&gt;\d*)\]:.(?&lt;newMessage&gt;[\s\S]*)"
            ]}
          }
      }
  </code></pre>
  
  <p>It combines the multilines correctly, but I now get those syslog messages mixed into my multiline messages</p>
  
  <p><strong>output</strong></p>
  
  <pre><code>message: Log message A
  message: &lt;&lt;ML&gt;&gt; Log message B &lt;30&gt;Jul 13 16:04:38 [1290]: more B1 &lt;30&gt;Jul 13 16:04:39 [1290]: more B2 &lt;30&gt;Jul 13 16:04:40 [1290]: more B3
  message: Log message C
  </code></pre>
  
  <p>How to get the codec processing to output like the filter one?</p>