title: Logstash: Handling of large messages
body: <p>I'm trying to parse a large message with Logstash using a file input, a json filter, and an elasticsearch output.  99% of the time this works fine, but when one of my log messages is too large, I get JSON parse errors, as the initial message is broken up into two partial invalid JSON streams.  The size of such messages is about 40,000+ characters long.  I've looked to see if there is any information on the size of the buffer, or some max length that I should try to stay under, but haven't had any luck.  The only answers I found related to the udp input, and being able to change the buffer size.</p>
  
  <p><a href="https://stackoverflow.com/questions/24606059/does-logstash-has-a-limit-size-for-each-event-message">Does Logstash has a limit size for each event-message?</a>
  <a href="https://github.com/elastic/logstash/issues/1505" rel="nofollow noreferrer">https://github.com/elastic/logstash/issues/1505</a></p>
  
  <p>This could also be similar to this question, but there were never any replies or suggestions: <a href="https://stackoverflow.com/questions/20577500/logstash-json-filter-behaving-unexpectedly-for-large-nested-jsons">Logstash Json filter behaving unexpectedly for large nested JSONs</a></p>
  
  <p>As a workaround, I wanted to split my message up into multiple messages, but I'm unable to do this, as I need all the information to be in the same record in Elasticsearch.  I don't believe there is a way to call the Update API from logstash.  Additionally, most of the data is in an array, so while I can update an Elasticsearch record's array using a script (<a href="https://stackoverflow.com/questions/18028280/elasticsearch-upserting-and-appending-to-array">Elasticsearch upserting and appending to array</a>), I can't do that from Logstash.</p>
  
  <p>The data records look something like this:</p>
  
  <pre><code>{ "variable1":"value1", 
   ......, 
   "variable30": "value30", 
   "attachements": [ {5500 charcters of JSON}, 
                     {5500 charcters of JSON}, 
                     {5500 charcters of JSON}.. 
                     ...
                     {8th dictionary of JSON}]
   }
  </code></pre>
  
  <p>Does anyone know of a way to have Logstash process these large JSON messages, or a way that I can split them up and have them end up in the same Elasticsearch record (using Logstash)?</p>
  
  <p>Any help is appreciated, and I'm happy to add any information needed!</p>