{"action":"created","issue":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-kinesis/issues/3","repository_url":"https://api.github.com/repos/logstash-plugins/logstash-input-kinesis","labels_url":"https://api.github.com/repos/logstash-plugins/logstash-input-kinesis/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/logstash-plugins/logstash-input-kinesis/issues/3/comments","events_url":"https://api.github.com/repos/logstash-plugins/logstash-input-kinesis/issues/3/events","html_url":"https://github.com/logstash-plugins/logstash-input-kinesis/pull/3","id":171264343,"number":3,"title":"Add CloudWatch Logs support","user":{"login":"threadwaste","id":1635478,"avatar_url":"https://avatars.githubusercontent.com/u/1635478?v=3","gravatar_id":"","url":"https://api.github.com/users/threadwaste","html_url":"https://github.com/threadwaste","followers_url":"https://api.github.com/users/threadwaste/followers","following_url":"https://api.github.com/users/threadwaste/following{/other_user}","gists_url":"https://api.github.com/users/threadwaste/gists{/gist_id}","starred_url":"https://api.github.com/users/threadwaste/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/threadwaste/subscriptions","organizations_url":"https://api.github.com/users/threadwaste/orgs","repos_url":"https://api.github.com/users/threadwaste/repos","events_url":"https://api.github.com/users/threadwaste/events{/privacy}","received_events_url":"https://api.github.com/users/threadwaste/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-08-15T21:18:50Z","updated_at":"2016-08-23T15:33:02Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-kinesis/pulls/3","html_url":"https://github.com/logstash-plugins/logstash-input-kinesis/pull/3","diff_url":"https://github.com/logstash-plugins/logstash-input-kinesis/pull/3.diff","patch_url":"https://github.com/logstash-plugins/logstash-input-kinesis/pull/3.patch"},"body":"This is an initial attempt at adding support for CloudWatch Logs subscription data and general gzip-compressed records per https://github.com/logstash-plugins/logstash-input-kinesis/issues/2. I'm looking for feedback before cleaning up what's been done. This has been tested against live CloudWatch Logs data.\r\n\r\nThis PR presents two new configuration options, `compression` and `cloudwatch_logs`. Their defaults maintain the plugin's current behavior.\r\n\r\nThe `compression` option accepts nil or \"gzip.\" When nil, the plugin's original behavior is retained. When set to \"gzip,\" the plugin decompresses each record with gzip. \r\n\r\nWhen `cloudwatch_logs` is set to true, each record is split on its logEvents field. The end result is n events containing the original record's logGroup, logStream, messageType, owner, and subscriptionFilters fields merged with each logEvents document. The original logEvents field is removed from each split event. When `cloudwatch_logs` is true, `compression` must be set to `gzip`."},"comment":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-kinesis/issues/comments/241772582","html_url":"https://github.com/logstash-plugins/logstash-input-kinesis/pull/3#issuecomment-241772582","issue_url":"https://api.github.com/repos/logstash-plugins/logstash-input-kinesis/issues/3","id":241772582,"user":{"login":"codekitchen","id":52172,"avatar_url":"https://avatars.githubusercontent.com/u/52172?v=3","gravatar_id":"","url":"https://api.github.com/users/codekitchen","html_url":"https://github.com/codekitchen","followers_url":"https://api.github.com/users/codekitchen/followers","following_url":"https://api.github.com/users/codekitchen/following{/other_user}","gists_url":"https://api.github.com/users/codekitchen/gists{/gist_id}","starred_url":"https://api.github.com/users/codekitchen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/codekitchen/subscriptions","organizations_url":"https://api.github.com/users/codekitchen/orgs","repos_url":"https://api.github.com/users/codekitchen/repos","events_url":"https://api.github.com/users/codekitchen/events{/privacy}","received_events_url":"https://api.github.com/users/codekitchen/received_events","type":"User","site_admin":false},"created_at":"2016-08-23T15:33:02Z","updated_at":"2016-08-23T15:33:02Z","body":"@threadwaste That doesn't seem like a problem with what I'm suggesting, though. You could use a codec in the simple case when you know that all the messages need the same processing, otherwise skip using a codec and use filters or some other mechanism.\r\n\r\nI definitely think we should explicitly decode the overall JSON message rather than depending on the codec being set to JSON, since that's a core property of cloudwatch logs, not something that should be configurable."}}
{"actor":{"display_login":"codekitchen"}}