title: Setting up Logstash & ElasticSearch for custom log structure
body: <p>I'm trying to setup logstash forwarding to redis/logstash/elasticsearch using <a href="https://github.com/jlecren/logserver" rel="nofollow">https://github.com/jlecren/logserver</a> (note -- this is my first attempt using these, so I may have some things way wrong). </p>
  
  <p>However, the logs I want to send into elastic-search not a standard log format (it a custom application).  They are already in JSON format, so I'm simply trying to forward these up to redis.  </p>
  
  <p>I'm forwarding the JSON up to redis using logstash and this config:</p>
  
  <pre><code>input {
     file {
        path =&gt; "/tmp/*.json"
        tags =&gt; "restserver"
        codec =&gt; "json"
     }
  }
  
  output {
    stdout { codec =&gt; rubydebug }
    redis { host =&gt; "192.168.33.101" data_type =&gt; "list" key =&gt; "logstash" }
  }
  </code></pre>
  
  <p>I'm trying to understand how to format the JSON output so I can search with kibana after they are in ES.  However, I think I'm missing how to create the ES template, and how to properly format the fields.</p>
  
  <p>As a first pass, I've started with this JSON format (note -- this is just a first pass, more fields need to be added):</p>
  
  <pre><code>{
    "@timestamp": "2014-02-06T15:23:10-0000",
    "tags": {
      "serial": "XX2013111111",
      "command": "Thumbnail"
    },
    "ip": "10.6.1.21",
    "source": "XX2013111111",
    "fields": {
      "type": "Thumbnail",
      "controller": "Image",
      "thumbnail": {
        "datetaken": [
          "2014-02-06T13:22:35"
        ],
        "name": [
          "DCIM\\IMG_008736.JPG"
        ]
      }
    },
    "message": "....."
  }
  </code></pre>
  
  <p>However, I'm confused about tag, fields, and how to create a template in ES (I think I need to do this, right?).</p>
  
  <p>Is there a guide somewhere about new log formats and how to integrate into logstash/kibana/ES?</p>