title: Logstash s3 output wrote format logstash s3 input doesn't understand
body: <p>I'm moving data from two ES clusters which are seperated.  I've added s3 as a common area and have two logstash instances, one that writes to s3 from Elasticsearch and another that reads S3 and loads Elasticsearch.</p>
  
  <p>The problem is that only one document from each index is loaded.  The output file written by s3 output plugin is a single long line, with many json documents all run together without commas or opening or closing square brackets for the array.  For example, instead of <code>[{"id":1},{"id":2},{"id":3}]</code> the output is writing files which read <code>{"id":1}{"id":2}{"id":3}</code>. In which case only <code>{"id":1}</code> is read by logstash using s3 as an input.</p>
  
  <p>The configuration to go to s3 is:</p>
  
  <pre><code>input {
    elasticsearch {
      hosts =&gt; ["${ES_HOST}:${ES_PORT}"]
      index =&gt; "${ES_INDEX}"
      password =&gt; "${ES_PASS}"
      ssl =&gt; "true"
      user =&gt; "${ES_USER}"
  }
    }
  output {
    s3 {
        bucket =&gt; "${S3_BUCKET}"
        encoding =&gt; "gzip"
        codec =&gt; "json"
        prefix =&gt; "${S3_PREFIX}/${ES_INDEX}"
        region =&gt; "ap-southeast-2"
      }
  }
  </code></pre>
  
  <p>The configuration reading S3 is:</p>
  
  <pre><code>input {
    s3 {
        bucket =&gt; "${S3_BUCKET}"
        codec =&gt; "json"
        prefix =&gt; "${S3_PREFIX}/${ES_INDEX}/"
        region =&gt; "ap-southeast-2"
        watch_for_new_files =&gt; false
      }
    }
  output {
    stdout { }
  }
  </code></pre>
  
  <p>In both cases the <code>${}</code> variables are set in the environment (bash shell).</p>
  
  <p>Both servers are running logstash 7.6.0</p>
  
  <p>PS: I don't think they are important, but the stdout log from logstash says:</p>
  
  <pre><code>OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
  WARNING: An illegal reflective access operation has occurred
  WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/home/ec2-user/logstash-7.6.0/logstash-core/lib/jars/jruby-complete-9.2.9.0.jar) to method sun.nio.ch.NativeThread.signal(long)
  WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
  WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
  WARNING: All illegal access operations will be denied in a future release
  Sending Logstash logs to /home/ec2-user/logstash-7.6.0/logs which is now configured via log4j2.properties
  [2020-03-09T01:10:35,168][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
  [2020-03-09T01:10:35,353][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=&gt;"7.6.0"}
  [2020-03-09T01:10:37,813][INFO ][org.reflections.Reflections] Reflections took 48 ms to scan 1 urls, producing 20 keys and 40 values 
  [2020-03-09T01:10:53,476][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
  [2020-03-09T01:10:53,515][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=&gt;"main", "pipeline.workers"=&gt;2, "pipeline.batch.size"=&gt;125, "pipeline.batch.delay"=&gt;50, "pipeline.max_inflight"=&gt;250, "pipeline.sources"=&gt;["/home/ec2-user/kibana/from_s3.conf"], :thread=&gt;"#&lt;Thread:0x1364485f run&gt;"}
  [2020-03-09T01:10:54,561][INFO ][logstash.inputs.s3       ][main] Registering s3 input {:bucket=&gt;"my-bucket-here", :region=&gt;"ap-southeast-2"}
  [2020-03-09T01:10:55,334][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=&gt;"main"}
  [2020-03-09T01:10:55,435][INFO ][logstash.agent           ] Pipelines running {:count=&gt;1, :running_pipelines=&gt;[:main], :non_running_pipelines=&gt;[]}
  [2020-03-09T01:10:55,833][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=&gt;9600}
  [2020-03-09T01:10:57,507][INFO ][logstash.inputs.s3       ][main] Using default generated file for the sincedb {:filename=&gt;"/home/ec2-user/logstash-7.6.0/data/plugins/inputs/s3/sincedb_1906e463a09b003733b719c08277c793"}
  /home/ec2-user/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
  {
  my-document-here
  }
  [2020-03-09T01:10:59,587][INFO ][logstash.runner          ] Logstash shut down.
  </code></pre>
  
  <p>PPS: deleting the since DB allows the one row to load, the file is not changing.</p>