{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/osquery/issues/2157","repository_url":"https://api.github.com/repos/facebook/osquery","labels_url":"https://api.github.com/repos/facebook/osquery/issues/2157/labels{/name}","comments_url":"https://api.github.com/repos/facebook/osquery/issues/2157/comments","events_url":"https://api.github.com/repos/facebook/osquery/issues/2157/events","html_url":"https://github.com/facebook/osquery/pull/2157","id":159673938,"number":2157,"title":"Optional randomization of shard ids for AWS Kinesis load balancing.","user":{"login":"hackgnar","id":4490430,"avatar_url":"https://avatars.githubusercontent.com/u/4490430?v=3","gravatar_id":"","url":"https://api.github.com/users/hackgnar","html_url":"https://github.com/hackgnar","followers_url":"https://api.github.com/users/hackgnar/followers","following_url":"https://api.github.com/users/hackgnar/following{/other_user}","gists_url":"https://api.github.com/users/hackgnar/gists{/gist_id}","starred_url":"https://api.github.com/users/hackgnar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hackgnar/subscriptions","organizations_url":"https://api.github.com/users/hackgnar/orgs","repos_url":"https://api.github.com/users/hackgnar/repos","events_url":"https://api.github.com/users/hackgnar/events{/privacy}","received_events_url":"https://api.github.com/users/hackgnar/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/facebook/osquery/labels/CLA%20Signed","name":"CLA Signed","color":"009900"},{"url":"https://api.github.com/repos/facebook/osquery/labels/logging","name":"logging","color":"1d76db"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2016-06-10T16:32:32Z","updated_at":"2016-06-15T16:42:47Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/facebook/osquery/pulls/2157","html_url":"https://github.com/facebook/osquery/pull/2157","diff_url":"https://github.com/facebook/osquery/pull/2157.diff","patch_url":"https://github.com/facebook/osquery/pull/2157.patch"},"body":"The AWS logging plugin sends shard_id_ as a partition key to AWS kinesis when pushing data.  The current implementation uses a static string as the shard id.  Kinesis uses these partition keys (i.e. shard ids) to load balance between shards in a stream.  By randomizing this value, you will get better performance on larger deployments using many shards in a Kinesis stream.  For more background on this, please read the following:\r\n\r\nhttp://sdk.amazonaws.com/cpp/api/0.11.0-7-g6be9f8d/d5/d84/class_aws_1_1_kinesis_1_1_model_1_1_put_record_request.html#a1f230751b741869bb8a3d0618d055cb6\r\n\r\nThis feature can be enabled by setting the following config variable otherwise the functionality will remain the same:\r\n\r\n````\r\n\"aws_kinesis_random_shardid\": \"true\",\r\n````"},"comment":{"url":"https://api.github.com/repos/facebook/osquery/issues/comments/226246548","html_url":"https://github.com/facebook/osquery/pull/2157#issuecomment-226246548","issue_url":"https://api.github.com/repos/facebook/osquery/issues/2157","id":226246548,"user":{"login":"zwass","id":575602,"avatar_url":"https://avatars.githubusercontent.com/u/575602?v=3","gravatar_id":"","url":"https://api.github.com/users/zwass","html_url":"https://github.com/zwass","followers_url":"https://api.github.com/users/zwass/followers","following_url":"https://api.github.com/users/zwass/following{/other_user}","gists_url":"https://api.github.com/users/zwass/gists{/gist_id}","starred_url":"https://api.github.com/users/zwass/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zwass/subscriptions","organizations_url":"https://api.github.com/users/zwass/orgs","repos_url":"https://api.github.com/users/zwass/repos","events_url":"https://api.github.com/users/zwass/events{/privacy}","received_events_url":"https://api.github.com/users/zwass/received_events","type":"User","site_admin":false},"created_at":"2016-06-15T16:42:47Z","updated_at":"2016-06-15T16:42:47Z","body":"I'm good with the changes. Perhaps we can add something to the docs along the lines of \"Note that using this setting will result in the logs of each host distributed across shards, so do not use it if you need logs from each host to be processed by a consistent shard.\""}}
{"actor":{"display_login":"zwass"}}