title: logstash cloudfront codec plugin: Error: Object: #Version: 1.0 is not a legal argument to this wrapper, cause it doesn't respond to "read"
body: <p>Logstash version 1.5.0.1</p>
  
  <p>I am trying to use the logstash <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-s3.html" rel="nofollow noreferrer">s3 input plugin</a> to download cloudfront logs and the <a href="https://www.elastic.co/guide/en/logstash/master/plugins-codecs-cloudfront.html" rel="nofollow noreferrer">cloudfront codec plugin</a> to filter the stream.</p>
  
  <p>I installed the cloudfront codec with <code>bin/plugin install logstash-codec-cloudfront</code>.</p>
  
  <p>I am getting the following: <strong>Error: Object: #Version: 1.0 is not a legal argument to this wrapper, cause it doesn't respond to "read".</strong></p>
  
  <p>Here is the full error message from /var/logs/logstash/logstash.log</p>
  
  <pre><code> {:timestamp=&gt;"2015-08-05T13:35:20.809000-0400", :message=&gt;"A plugin had an unrecoverable error. Will restart this plugin.\n  Plugin: &lt;LogStash::Inputs::S3 bucket=&gt;\"[BUCKETNAME]\", prefix=&gt;\"cloudfront/\", region=&gt;\"us-east-1\", type=&gt;\"cloudfront\", secret_access_key=&gt;\"[SECRETKEY]/1\", access_key_id=&gt;\"[KEYID]\", sincedb_path=&gt;\"/opt/logstash_input/s3/cloudfront/sincedb\", backup_to_dir=&gt;\"/opt/logstash_input/s3/cloudfront/backup\", temporary_directory=&gt;\"/var/lib/logstash/logstash\"&gt;\n  Error: Object: #Version: 1.0\n is not a legal argument to this wrapper, cause it doesn't respond to \"read\".", :level=&gt;:error}
  </code></pre>
  
  <p>My logstash config file: /etc/logstash/conf.d/cloudfront.conf</p>
  
  <pre><code>input {
    s3 {
      bucket =&gt; "[BUCKETNAME]"
      delete =&gt; false
      interval =&gt; 60 # seconds
      prefix =&gt; "cloudfront/"
      region =&gt; "us-east-1"
      type =&gt; "cloudfront"
      codec =&gt; "cloudfront"
      secret_access_key =&gt; "[SECRETKEY]"
      access_key_id =&gt; "[KEYID]"
      sincedb_path =&gt; "/opt/logstash_input/s3/cloudfront/sincedb"
      backup_to_dir =&gt; "/opt/logstash_input/s3/cloudfront/backup"
      use_ssl =&gt; true
    }
  }
  </code></pre>
  
  <p>I'm using a similar s3 input stream successfully to get my cloudtrail logs into logstash that is based on the <a href="https://stackoverflow.com/questions/29819970/getting-cloudtrail-logs-into-logstash">Answer </a> from a stackoverflow post.</p>
  
  <p>CloudFront logfile from s3 (I only included the header from the file):</p>
  
  <pre><code> #Version: 1.0
   #Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type
  </code></pre>
  
  <p>The header looks like it is basically the correct format based on lines 26-29 from the cloudfront plugin github repo <a href="https://github.com/logstash-plugins/logstash-codec-cloudfront/blob/master/spec/codecs/cloudfront_spec.rb" rel="nofollow noreferrer">cloudfront_spec.rb
  </a> and the official AWS CloudFront <a href="http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html" rel="nofollow noreferrer">Access Logs</a> docs.</p>
  
  <p>Any ideas?  Thanks!</p>
  
  <p>[UPDATE 9/23/2015]</p>
  
  <p>Based on this <a href="https://logstash.jira.com/browse/LOGSTASH-1550" rel="nofollow noreferrer">post</a> I tried using the <a href="https://www.elastic.co/guide/en/logstash/current/plugins-codecs-gzip_lines.html" rel="nofollow noreferrer">gzip_lines</a> codec plugin, installed with <code>bin/plugin install logstash-codec-gzip_lines</code> and parse the file with a filter, unfortunately I am getting the exact same error.  It looks like it is an issue with the first character of the log file having <code>#</code>.  </p>
  
  <p>For the record, here is the new attempt, including an updated pattern for parsing the cloudfront logfile due to four new fields:</p>
  
  <p>/etc/logstash/conf.d/cloudfront.conf</p>
  
  <pre><code>input {
    s3 {
      bucket =&gt; "[BUCKETNAME]"
      delete =&gt; false
      interval =&gt; 60 # seconds
      prefix =&gt; "cloudfront/"
      region =&gt; "us-east-1"
      type =&gt; "cloudfront"
      codec =&gt; "gzip_lines"
      secret_access_key =&gt; "[SECRETKEY]"
      access_key_id =&gt; "[KEYID]"
      sincedb_path =&gt; "/opt/logstash_input/s3/cloudfront/sincedb"
      backup_to_dir =&gt; "/opt/logstash_input/s3/cloudfront/backup"
      use_ssl =&gt; true
    }
  }
  filter {
      grok {
      type =&gt; "cloudfront"
      pattern =&gt; "%{DATE_EU:date}\t%{TIME:time}\t%{WORD:x_edge_location}\t(?:%{NUMBER:sc_bytes}|-)\t%{IPORHOST:c_ip}\t%{WORD:cs_method}\t%{HOSTNAME:cs_host}\t%{NOTSPACE:cs_uri_stem}\t%{NUMBER:sc_status}\t%{GREEDYDATA:referrer}\t%{GREEDYDATA:User_Agent}\t%{GREEDYDATA:cs_uri_stem}\t%{GREEDYDATA:cookies}\t%{WORD:x_edge_result_type}\t%{NOTSPACE:x_edge_request_id}\t%{HOSTNAME:x_host_header}\t%{URIPROTO:cs_protocol}\t%{INT:cs_bytes}\t%{GREEDYDATA:time_taken}\t%{GREEDYDATA:x_forwarded_for}\t%{GREEDYDATA:ssl_protocol}\t%{GREEDYDATA:ssl_cipher}\t%{GREEDYDATA:x_edge_response_result_type}"
    }
  
  mutate {
      type =&gt; "cloudfront"
          add_field =&gt; [ "listener_timestamp", "%{date} %{time}" ]
      }
  
  date {
        type =&gt; "cloudfront"
        match =&gt; [ "listener_timestamp", "yy-MM-dd HH:mm:ss" ]
      }
  
  }
  </code></pre>