title: input json to logstash - config issues?
body: <p>i have the following json input that i want to dump to logstash (and eventually search/dashboard in elasticsearch/kibana).</p>
  
  <pre><code>{"vulnerabilities":[
      {"ip":"10.1.1.1","dns":"z.acme.com","vid":"12345"},
      {"ip":"10.1.1.2","dns":"y.acme.com","vid":"12345"},
      {"ip":"10.1.1.3","dns":"x.acme.com","vid":"12345"}
  ]}
  </code></pre>
  
  <p>i'm using the following logstash configuration</p>
  
  <pre><code>input {
    file {
      path =&gt; "/tmp/logdump/*"
      type =&gt; "assets"
      codec =&gt; "json"
    }
  }
  output {
    stdout { codec =&gt; rubydebug }
    elasticsearch { host =&gt; localhost }
  }
  </code></pre>
  
  <p>output</p>
  
  <pre><code>{
         "message" =&gt; "{\"vulnerabilities\":[\r",
        "@version" =&gt; "1",
      "@timestamp" =&gt; "2014-10-30T23:41:19.788Z",
            "type" =&gt; "assets",
            "host" =&gt; "av12612sn00-pn9",
            "path" =&gt; "/tmp/logdump/stack3.json"
  }
  {
         "message" =&gt; "{\"ip\":\"10.1.1.30\",\"dns\":\"z.acme.com\",\"vid\":\"12345\"},\r",
        "@version" =&gt; "1",
      "@timestamp" =&gt; "2014-10-30T23:41:19.838Z",
            "type" =&gt; "assets",
            "host" =&gt; "av12612sn00-pn9",
            "path" =&gt; "/tmp/logdump/stack3.json"
  }
  {
         "message" =&gt; "{\"ip\":\"10.1.1.31\",\"dns\":\"y.acme.com\",\"vid\":\"12345\"},\r",
        "@version" =&gt; "1",
      "@timestamp" =&gt; "2014-10-30T23:41:19.870Z",
            "type" =&gt; "shellshock",
            "host" =&gt; "av1261wag2sn00-pn9",
            "path" =&gt; "/tmp/logdump/stack3.json"
  }
  {
              "ip" =&gt; "10.1.1.32",
             "dns" =&gt; "x.acme.com",
             "vid" =&gt; "12345",
        "@version" =&gt; "1",
      "@timestamp" =&gt; "2014-10-30T23:41:19.884Z",
            "type" =&gt; "assets",
            "host" =&gt; "av12612sn00-pn9",
            "path" =&gt; "/tmp/logdump/stack3.json"
  }
  </code></pre>
  
  <p>obviously logstash is treating each line as an event and it thinks <code>{"vulnerabilities":[</code> is an event and i'm guessing the trailing commas on the 2 subsequent nodes mess up the parsing, and the last node appears coorrect. how do i tell logstash to parse the events inside the vulnerabilities array and to ignore the commas at the end of the line?</p>
  
  <p><strong>Updated: 2014-11-05</strong>
  Following Magnus' recommendations, I added the json filter and it's working perfectly. However, it would not parse the last line of the json correctly without specifying <code>start_position =&gt; "beginning"</code> in the file input block. Any ideas why not? I know it parses bottom up by default but would anticipate the mutate/gsub would handle this smoothly?</p>
  
  <pre><code>file {
      path =&gt; "/tmp/logdump/*"
      type =&gt; "assets"
      start_position =&gt; "beginning"
    }
  }
  filter {
    if [message] =~ /^\[?{"ip":/ {
      mutate {
        gsub =&gt; [
          "message", "^\[{", "{",
          "message", "},?\]?$", "}"
        ]
      }
      json {
        source =&gt; "message"
        remove_field =&gt; ["message"]
      }
    }
  }
  output {
    stdout { codec =&gt; rubydebug }
    elasticsearch { host =&gt; localhost }
  }
  </code></pre>