title: using logstash to get data out of elasticsearch to a csv file
body: <p>I am new to using logstash and am struggling to get data out of elasticsearch using logstash as a csv.</p>
  
  <p>To create some sample data, we can first add a basic csv into elasticsearch... the head of the sample csv can be seen below</p>
  
  <pre><code>$ head uu.csv
  "hh","hh1","hh3","id"
  -0.979646332669359,1.65186132910743,"L",1
  -0.283939374784435,-0.44785377794233,"X",2
  0.922659898930901,-1.11689020559612,"F",3
  0.348918777124474,1.95766948269957,"U",4
  0.52667811182958,0.0168862169880919,"Y",5
  -0.804765331279075,-0.186456470768865,"I",6
  0.11411203100637,-0.149340801708981,"Q",7
  -0.952836952412902,-1.68807271639322,"Q",8
  -0.373528919496876,0.750994450392907,"F",9
  </code></pre>
  
  <p>I then put that into logstash using the following...</p>
  
  <pre><code>$ cat uu.conf 
  input {
    stdin {}
  }
  
  filter {
    csv {
        columns =&gt; [
          "hh","hh1","hh3","id"
        ]
    }
  
    if [hh1] == "hh1" {
        drop { }
    } else {
        mutate {
            remove_field =&gt; [ "message", "host", "@timestamp", "@version" ]
        }
  
        mutate {
            convert =&gt; { "hh" =&gt; "float" }
            convert =&gt; { "hh1" =&gt; "float" }
            convert =&gt; { "hh3" =&gt; "string" }
            convert =&gt; { "id" =&gt; "integer" }
        }
    }
  }
  
  output {
    stdout { codec =&gt; dots }
    elasticsearch {
        index =&gt; "temp_index"
        document_type =&gt; "temp_doc"
        document_id =&gt; "%{id}"
    }
  }
  </code></pre>
  
  <p>This is put into logstash with the following command....</p>
  
  <pre><code>$ cat uu.csv | logstash-2.1.3/bin/logstash -f uu.conf 
  Settings: Default filter workers: 16
  Logstash startup completed
   ....................................................................................................Logstash shutdown completed
  </code></pre>
  
  <p>So far so good, but I would like to get some of the data out in particular the hh and hh3 fields in the temp_index.</p>
  
  <p>I wrote the following to extract the data out of elasticsearch into a csv.</p>
  
  <pre><code>$ cat yy.conf
  input {
    elasticsearch {
      hosts =&gt; "localhost:9200"
      index =&gt; "temp_index"
      query =&gt; "*"
    }
  }
  
  filter {
      elasticsearch{
          add_field =&gt; {"hh" =&gt; "%{hh}"}
          add_field =&gt; {"hh3" =&gt; "%{hh3}"}
      }
  }
  
  
  output {
    stdout { codec =&gt; dots }
    csv {
        fields =&gt; ['hh','hh3']
        path =&gt; '/home/username/yy.csv'
    }
  }
  </code></pre>
  
  <p>But get the following error when trying to run logstash...</p>
  
  <pre><code>$ logstash-2.1.3/bin/logstash -f yy.conf
  The error reported is: 
    Couldn't find any filter plugin named 'elasticsearch'. Are you sure this is correct? Trying to load the elasticsearch filter plugin resulted in this error: no such file to load -- logstash/filters/elasticsearch
  </code></pre>
  
  <p>What do I need to change to yy.conf so that a logstash command will extract the data out of elasticsearch and input into a new csv called yy.csv.</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>changing <code>yy.conf</code> to be the following...</p>
  
  <pre><code>$ cat yy.conf
  input {
    elasticsearch {
      hosts =&gt; "localhost:9200"
      index =&gt; "temp_index"
      query =&gt; "*"
    }
  }
  
  filter {}
  
  output {
    stdout { codec =&gt; dots }
    csv {
        fields =&gt; ['hh','hh3']
        path =&gt; '/home/username/yy.csv'
    }
  }
  </code></pre>
  
  <p>I got the following error...</p>
  
  <pre><code>$ logstash-2.1.3/bin/logstash -f yy.conf
  Settings: Default filter workers: 16
  Logstash startup completed
  A plugin had an unrecoverable error. Will restart this plugin.
    Plugin: &lt;LogStash::Inputs::Elasticsearch hosts=&gt;["localhost:9200"], index=&gt;"temp_index", query=&gt;"*", codec=&gt;&lt;LogStash::Codecs::JSON charset=&gt;"UTF-8"&gt;, scan=&gt;true, size=&gt;1000, scroll=&gt;"1m", docinfo=&gt;false, docinfo_target=&gt;"@metadata", docinfo_fields=&gt;["_index", "_type", "_id"], ssl=&gt;false&gt;
    Error: [400] {"error":{"root_cause":[{"type":"parse_exception","reason":"Failed to derive xcontent"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"init_scan","grouped":true,"failed_shards":[{"shard":0,"index":"temp_index","node":"zu3E6F7kQRWnDPY5L9zF-w","reason":{"type":"parse_exception","reason":"Failed to derive xcontent"}}]},"status":400} {:level=&gt;:error}
  A plugin had an unrecoverable error. Will restart this plugin.
    Plugin: &lt;LogStash::Inputs::Elasticsearch hosts=&gt;["localhost:9200"], index=&gt;"temp_index", query=&gt;"*", codec=&gt;&lt;LogStash::Codecs::JSON charset=&gt;"UTF-8"&gt;, scan=&gt;true, size=&gt;1000, scroll=&gt;"1m", docinfo=&gt;false, docinfo_target=&gt;"@metadata", docinfo_fields=&gt;["_index", "_type", "_id"], ssl=&gt;false&gt;
    Error: [400] {"error":{"root_cause":[{"type":"parse_exception","reason":"Failed to derive xcontent"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"init_scan","grouped":true,"failed_shards":[{"shard":0,"index":"temp_index","node":"zu3E6F7kQRWnDPY5L9zF-w","reason":{"type":"parse_exception","reason":"Failed to derive xcontent"}}]},"status":400} {:level=&gt;:error}
  A plugin had an unrecoverable error. Will restart this plugin.
  </code></pre>
  
  <p>Interestingly...if i change <code>yy.conf</code> to remove <code>elasticsearch{}</code> to look like...</p>
  
  <pre><code>$ cat yy.conf
  input {
    elasticsearch {
      hosts =&gt; "localhost:9200"
      index =&gt; "temp_index"
      query =&gt; "*"
    }
  }
  
  filter {
          add_field =&gt; {"hh" =&gt; "%{hh}"}
          add_field =&gt; {"hh3" =&gt; "%{hh3}"}
  }
  
  
  output {
    stdout { codec =&gt; dots }
    csv {
        fields =&gt; ['hh','hh3']
        path =&gt; '/home/username/yy.csv'
    }
  }
  </code></pre>
  
  <p>I get the following error...</p>
  
  <pre><code>$ logstash-2.1.3/bin/logstash -f yy.conf
  Error: Expected one of #, { at line 10, column 19 (byte 134) after filter {
          add_field 
  You may be interested in the '--configtest' flag which you can
  use to validate logstash's configuration before you choose
  to restart a running system.
  </code></pre>
  
  <p>Also when changing yy.conf to be something similar to take into account the error message</p>
  
  <pre><code>$ cat yy.conf
  input {
    elasticsearch {
      hosts =&gt; "localhost:9200"
      index =&gt; "temp_index"
      query =&gt; "*"
    }
  }
  
  filter {
          add_field {"hh" =&gt; "%{hh}"}
          add_field {"hh3" =&gt; "%{hh3}"}
  }
  
  
  output {
    stdout { codec =&gt; dots }
    csv {
        fields =&gt; ['hh','hh3']
        path =&gt; '/home/username/yy.csv'
    }
  }
  </code></pre>
  
  <p>I get the following error...</p>
  
  <pre><code>$ logstash-2.1.3/bin/logstash -f yy.conf
  The error reported is: 
    Couldn't find any filter plugin named 'add_field'. Are you sure this is correct? Trying to load the add_field filter plugin resulted in this error: no such file to load -- logstash/filters/add_field
  </code></pre>
  
  <p><strong>* UPDATE 2 *</strong></p>
  
  <p>Thanks to Val I have made some progress and started to get output. But they don't seem correct. I get the following outputs when running the following commands...</p>
  
  <pre><code>$ cat uu.csv | logstash-2.1.3/bin/logstash -f uu.conf 
  Settings: Default filter workers: 16
  Logstash startup completed
  ....................................................................................................Logstash shutdown completed
  
  $ logstash-2.1.3/bin/logstash -f yy.conf
  Settings: Default filter workers: 16
  Logstash startup completed
  ....................................................................................................Logstash shutdown completed
  
  $ head uu.csv
  "hh","hh1","hh3","id"
  -0.979646332669359,1.65186132910743,"L",1
  -0.283939374784435,-0.44785377794233,"X",2
  0.922659898930901,-1.11689020559612,"F",3
  0.348918777124474,1.95766948269957,"U",4
  0.52667811182958,0.0168862169880919,"Y",5
  -0.804765331279075,-0.186456470768865,"I",6
  0.11411203100637,-0.149340801708981,"Q",7
  -0.952836952412902,-1.68807271639322,"Q",8
  -0.373528919496876,0.750994450392907,"F",9
  
  $ head yy.csv
  -0.106007607975644E1,F
  0.385395589205671E0,S
  0.722392598488791E-1,Q
  0.119773830827963E1,Q
  -0.151090510772458E1,W
  -0.74978830916084E0,G
  -0.98888121700762E-1,M
  0.965827615823707E0,S
  -0.165311094671424E1,F
  0.523818819076447E0,R
  </code></pre>
  
  <p>Any help would be much appreciated...</p>