title: Pushing structured log data directly to elastic search with filebeat
body: <p>I have configured filebeat to harvest my structured log output (greenfield project so each log entry is a JSON document in a pre-defined format) and publish it directly to ELS.  </p>
  
  <p>Example log file excerpt (note that <code>additional</code> is free form, all other properties are fixed.  Pretty formatted for this post but each top level object is on a single line in the file):</p>
  
  <pre><code>{
      "TimeUtc": "2016-09-23T14:13:02.217520245Z",
      "ServiceKey": "MAAS_SVC",
      "Title": "Get All Campaigns - Start",
      "Additional": {
          "HTTPRequest": {
              "Method": "GET",
              "URL": {
                  "Scheme": "",
                  "Opaque": "",
                  "User": null,
                  "Host": "",
                  "Path": "/admin/campaigns",
                  "RawPath": "",
                  "ForceQuery": false,
                  "RawQuery": "",
                  "Fragment": ""
              },
              "Proto": "HTTP/1.1",
              "ProtoMajor": 1,
              "ProtoMinor": 1,
              "Header": {
                  "Accept": ["*/*"],
                  "Accept-Encoding": ["gzip, deflate"],
                  "Connection": ["keep-alive"],
                  "Requestkey": ["78478050-47f0-4d0d-44e8-615d0599574a"],
                  "User-Agent": ["python-requests/2.7.0 CPython/2.7.12 Linux/3.13.0-74-generic"]
              },
              "Body": {
                  "Closer": {
                      "Reader": null
                  }
              },
              "ContentLength": 0,
              "TransferEncoding": null,
              "Close": false,
              "Host": "xxxxxxxxx",
              "Form": null,
              "PostForm": null,
              "MultipartForm": null,
              "Trailer": null,
              "RemoteAddr": "xxx.xxx.xxx.xxx",
              "RequestURI": "/admin/campaigns",
              "TLS": null,
              "Cancel": ,
              "Response": null
          }
      },
      "RequestKey": "78478050-47f0-4d0d-44e8-615d0599574a",
      "HostAddress": "xxxxxxxxx"
  } 
  </code></pre>
  
  <p>This results in filebeat making the following request to ELS:</p>
  
  <pre><code>{
      "@timestamp": "2016-10-12T13:53:21.597Z",
      "beat": {
          "hostname": "7bca0e28e69e",
          "name": "7bca0e28e69e"
      },
      "count": 1,
      "fields": null,
      "input_type": "log",
      "message": "{\"TimeUtc\":\"2016-09-23T14:13:02.217520245Z\",\"ServiceKey\":\"MAAS_SVC\",\"Title\":\"Get All Campaigns - Start\",\"Additional\":{\"HTTPRequest\":{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/admin/campaigns\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Connection\":[\"keep-alive\"],\"Requestkey\":[\"78478050-47f0-4d0d-44e8-615d0599574a\"],\"User-Agent\":[\"python-requests/2.7.0 CPython/2.7.12 Linux/3.13.0-74-generic\"]},\"Body\":{\"Closer\":{\"Reader\":null}},\"ContentLength\":0,\"TransferEncoding\":null,\"Close\":false,\"Host\":\"bistromath.marathon.mesos:40072\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.20.1.70:42854\",\"RequestURI\":\"/admin/campaigns\",\"TLS\":null,\"Cancel\":,\"Response\":null}},\"RequestKey\":\"78478050-47f0-4d0d-44e8-615d0599574a\",\"HostAddress\":\"ba47316c9c45\"}",
      "offset": 0,
      "source": "/filebeat/log-harvest/maas-service-single.log",
      "type": "log"
  }
  </code></pre>
  
  <p>Can I prevent filebeat from escaping my log JSON so that it's becomes a nested object rather than a string, or do I need to patch filebeat?</p>