{"action":"opened","number":2703,"pull_request":{"url":"https://api.github.com/repos/facebook/osquery/pulls/2703","id":91859281,"html_url":"https://github.com/facebook/osquery/pull/2703","diff_url":"https://github.com/facebook/osquery/pull/2703.diff","patch_url":"https://github.com/facebook/osquery/pull/2703.patch","issue_url":"https://api.github.com/repos/facebook/osquery/issues/2703","number":2703,"state":"open","locked":false,"title":"Fix for sending only failed data to kinesis on full or partial failed uploads","user":{"login":"hackgnar","id":4490430,"avatar_url":"https://avatars.githubusercontent.com/u/4490430?v=3","gravatar_id":"","url":"https://api.github.com/users/hackgnar","html_url":"https://github.com/hackgnar","followers_url":"https://api.github.com/users/hackgnar/followers","following_url":"https://api.github.com/users/hackgnar/following{/other_user}","gists_url":"https://api.github.com/users/hackgnar/gists{/gist_id}","starred_url":"https://api.github.com/users/hackgnar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hackgnar/subscriptions","organizations_url":"https://api.github.com/users/hackgnar/orgs","repos_url":"https://api.github.com/users/hackgnar/repos","events_url":"https://api.github.com/users/hackgnar/events{/privacy}","received_events_url":"https://api.github.com/users/hackgnar/received_events","type":"User","site_admin":false},"body":"@zwass could I get your opinion/assistance on this PR?  In short, there is an issue in the kinesis logger plugin where duplicate data gets sent to kinesis if there is an upload which partially fails.  \r\n\r\nIt may not seem too dire, but I have seen single endpoints send over 100k dup events over a 24h time period from just ~3k source events.  This typically happens in environments where kinesis is under heavy load and rate limit errors pop up.\r\n\r\nWith the current code path, when you receive errors like the following, osquery will resend all data (even the ones that successfully uploaded) until all data in the batch is successfully uploaded.\r\n\r\n```\r\nE1101 15:20:31.287334 4284416 aws_kinesis.cpp:111] Kinesis write for 5 of 140 records failed with error Rate exceeded for shard....REDACTED...\r\n```\r\n\r\nSo in the above example, osquery tried to send 140 records.  135 of them were successfully inserted into kinesis but 5 failed. Osquery will resend all 140 records over and over again until all 140 are successfully upload.  When kinesis is under heavy load and rate limits are the cause of the failure, the next uploads will typically fail as well.  This can cause a large amount of dup data. \r\n\r\nIm not a huge fan of my PR, but I wanted to get something submitted sooner than later so we can agree on a proper fix.  It should also help you get an idea of the issue.\r\n\r\nDo you think we should:\r\n\r\n1. clean up this PR and go with it?\r\n2. drop the flags and come up with sane defaults?\r\n3. do something completely different?","created_at":"2016-11-01T20:41:12Z","updated_at":"2016-11-01T20:41:12Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/facebook/osquery/pulls/2703/commits","review_comments_url":"https://api.github.com/repos/facebook/osquery/pulls/2703/comments","review_comment_url":"https://api.github.com/repos/facebook/osquery/pulls/comments{/number}","comments_url":"https://api.github.com/repos/facebook/osquery/issues/2703/comments","statuses_url":"https://api.github.com/repos/facebook/osquery/statuses/9638d26c35b737d8a3fcfb1f64a75d94d94ca1a1","head":{"label":"hackgnar:pr_aws_kinesis_failed_uploads","ref":"pr_aws_kinesis_failed_uploads","sha":"9638d26c35b737d8a3fcfb1f64a75d94d94ca1a1","user":{"login":"hackgnar","id":4490430,"avatar_url":"https://avatars.githubusercontent.com/u/4490430?v=3","gravatar_id":"","url":"https://api.github.com/users/hackgnar","html_url":"https://github.com/hackgnar","followers_url":"https://api.github.com/users/hackgnar/followers","following_url":"https://api.github.com/users/hackgnar/following{/other_user}","gists_url":"https://api.github.com/users/hackgnar/gists{/gist_id}","starred_url":"https://api.github.com/users/hackgnar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hackgnar/subscriptions","organizations_url":"https://api.github.com/users/hackgnar/orgs","repos_url":"https://api.github.com/users/hackgnar/repos","events_url":"https://api.github.com/users/hackgnar/events{/privacy}","received_events_url":"https://api.github.com/users/hackgnar/received_events","type":"User","site_admin":false},"repo":{"id":59076445,"name":"osquery","full_name":"hackgnar/osquery","owner":{"login":"hackgnar","id":4490430,"avatar_url":"https://avatars.githubusercontent.com/u/4490430?v=3","gravatar_id":"","url":"https://api.github.com/users/hackgnar","html_url":"https://github.com/hackgnar","followers_url":"https://api.github.com/users/hackgnar/followers","following_url":"https://api.github.com/users/hackgnar/following{/other_user}","gists_url":"https://api.github.com/users/hackgnar/gists{/gist_id}","starred_url":"https://api.github.com/users/hackgnar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hackgnar/subscriptions","organizations_url":"https://api.github.com/users/hackgnar/orgs","repos_url":"https://api.github.com/users/hackgnar/repos","events_url":"https://api.github.com/users/hackgnar/events{/privacy}","received_events_url":"https://api.github.com/users/hackgnar/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/hackgnar/osquery","description":"SQL powered operating system instrumentation, monitoring, and analytics.","fork":true,"url":"https://api.github.com/repos/hackgnar/osquery","forks_url":"https://api.github.com/repos/hackgnar/osquery/forks","keys_url":"https://api.github.com/repos/hackgnar/osquery/keys{/key_id}","collaborators_url":"https://api.github.com/repos/hackgnar/osquery/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/hackgnar/osquery/teams","hooks_url":"https://api.github.com/repos/hackgnar/osquery/hooks","issue_events_url":"https://api.github.com/repos/hackgnar/osquery/issues/events{/number}","events_url":"https://api.github.com/repos/hackgnar/osquery/events","assignees_url":"https://api.github.com/repos/hackgnar/osquery/assignees{/user}","branches_url":"https://api.github.com/repos/hackgnar/osquery/branches{/branch}","tags_url":"https://api.github.com/repos/hackgnar/osquery/tags","blobs_url":"https://api.github.com/repos/hackgnar/osquery/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/hackgnar/osquery/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/hackgnar/osquery/git/refs{/sha}","trees_url":"https://api.github.com/repos/hackgnar/osquery/git/trees{/sha}","statuses_url":"https://api.github.com/repos/hackgnar/osquery/statuses/{sha}","languages_url":"https://api.github.com/repos/hackgnar/osquery/languages","stargazers_url":"https://api.github.com/repos/hackgnar/osquery/stargazers","contributors_url":"https://api.github.com/repos/hackgnar/osquery/contributors","subscribers_url":"https://api.github.com/repos/hackgnar/osquery/subscribers","subscription_url":"https://api.github.com/repos/hackgnar/osquery/subscription","commits_url":"https://api.github.com/repos/hackgnar/osquery/commits{/sha}","git_commits_url":"https://api.github.com/repos/hackgnar/osquery/git/commits{/sha}","comments_url":"https://api.github.com/repos/hackgnar/osquery/comments{/number}","issue_comment_url":"https://api.github.com/repos/hackgnar/osquery/issues/comments{/number}","contents_url":"https://api.github.com/repos/hackgnar/osquery/contents/{+path}","compare_url":"https://api.github.com/repos/hackgnar/osquery/compare/{base}...{head}","merges_url":"https://api.github.com/repos/hackgnar/osquery/merges","archive_url":"https://api.github.com/repos/hackgnar/osquery/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/hackgnar/osquery/downloads","issues_url":"https://api.github.com/repos/hackgnar/osquery/issues{/number}","pulls_url":"https://api.github.com/repos/hackgnar/osquery/pulls{/number}","milestones_url":"https://api.github.com/repos/hackgnar/osquery/milestones{/number}","notifications_url":"https://api.github.com/repos/hackgnar/osquery/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/hackgnar/osquery/labels{/name}","releases_url":"https://api.github.com/repos/hackgnar/osquery/releases{/id}","deployments_url":"https://api.github.com/repos/hackgnar/osquery/deployments","created_at":"2016-05-18T02:48:48Z","updated_at":"2016-10-31T20:36:48Z","pushed_at":"2016-11-01T19:52:43Z","git_url":"git://github.com/hackgnar/osquery.git","ssh_url":"git@github.com:hackgnar/osquery.git","clone_url":"https://github.com/hackgnar/osquery.git","svn_url":"https://github.com/hackgnar/osquery","homepage":"https://osquery.io","size":9424,"stargazers_count":1,"watchers_count":1,"language":"C++","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"facebook:master","ref":"master","sha":"3495b3cafe08ef6059d4d0b32c9fdc6106e39fd7","user":{"login":"facebook","id":69631,"avatar_url":"https://avatars.githubusercontent.com/u/69631?v=3","gravatar_id":"","url":"https://api.github.com/users/facebook","html_url":"https://github.com/facebook","followers_url":"https://api.github.com/users/facebook/followers","following_url":"https://api.github.com/users/facebook/following{/other_user}","gists_url":"https://api.github.com/users/facebook/gists{/gist_id}","starred_url":"https://api.github.com/users/facebook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/facebook/subscriptions","organizations_url":"https://api.github.com/users/facebook/orgs","repos_url":"https://api.github.com/users/facebook/repos","events_url":"https://api.github.com/users/facebook/events{/privacy}","received_events_url":"https://api.github.com/users/facebook/received_events","type":"Organization","site_admin":false},"repo":{"id":22394357,"name":"osquery","full_name":"facebook/osquery","owner":{"login":"facebook","id":69631,"avatar_url":"https://avatars.githubusercontent.com/u/69631?v=3","gravatar_id":"","url":"https://api.github.com/users/facebook","html_url":"https://github.com/facebook","followers_url":"https://api.github.com/users/facebook/followers","following_url":"https://api.github.com/users/facebook/following{/other_user}","gists_url":"https://api.github.com/users/facebook/gists{/gist_id}","starred_url":"https://api.github.com/users/facebook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/facebook/subscriptions","organizations_url":"https://api.github.com/users/facebook/orgs","repos_url":"https://api.github.com/users/facebook/repos","events_url":"https://api.github.com/users/facebook/events{/privacy}","received_events_url":"https://api.github.com/users/facebook/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/facebook/osquery","description":"SQL powered operating system instrumentation, monitoring, and analytics.","fork":false,"url":"https://api.github.com/repos/facebook/osquery","forks_url":"https://api.github.com/repos/facebook/osquery/forks","keys_url":"https://api.github.com/repos/facebook/osquery/keys{/key_id}","collaborators_url":"https://api.github.com/repos/facebook/osquery/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/facebook/osquery/teams","hooks_url":"https://api.github.com/repos/facebook/osquery/hooks","issue_events_url":"https://api.github.com/repos/facebook/osquery/issues/events{/number}","events_url":"https://api.github.com/repos/facebook/osquery/events","assignees_url":"https://api.github.com/repos/facebook/osquery/assignees{/user}","branches_url":"https://api.github.com/repos/facebook/osquery/branches{/branch}","tags_url":"https://api.github.com/repos/facebook/osquery/tags","blobs_url":"https://api.github.com/repos/facebook/osquery/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/facebook/osquery/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/facebook/osquery/git/refs{/sha}","trees_url":"https://api.github.com/repos/facebook/osquery/git/trees{/sha}","statuses_url":"https://api.github.com/repos/facebook/osquery/statuses/{sha}","languages_url":"https://api.github.com/repos/facebook/osquery/languages","stargazers_url":"https://api.github.com/repos/facebook/osquery/stargazers","contributors_url":"https://api.github.com/repos/facebook/osquery/contributors","subscribers_url":"https://api.github.com/repos/facebook/osquery/subscribers","subscription_url":"https://api.github.com/repos/facebook/osquery/subscription","commits_url":"https://api.github.com/repos/facebook/osquery/commits{/sha}","git_commits_url":"https://api.github.com/repos/facebook/osquery/git/commits{/sha}","comments_url":"https://api.github.com/repos/facebook/osquery/comments{/number}","issue_comment_url":"https://api.github.com/repos/facebook/osquery/issues/comments{/number}","contents_url":"https://api.github.com/repos/facebook/osquery/contents/{+path}","compare_url":"https://api.github.com/repos/facebook/osquery/compare/{base}...{head}","merges_url":"https://api.github.com/repos/facebook/osquery/merges","archive_url":"https://api.github.com/repos/facebook/osquery/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/facebook/osquery/downloads","issues_url":"https://api.github.com/repos/facebook/osquery/issues{/number}","pulls_url":"https://api.github.com/repos/facebook/osquery/pulls{/number}","milestones_url":"https://api.github.com/repos/facebook/osquery/milestones{/number}","notifications_url":"https://api.github.com/repos/facebook/osquery/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/facebook/osquery/labels{/name}","releases_url":"https://api.github.com/repos/facebook/osquery/releases{/id}","deployments_url":"https://api.github.com/repos/facebook/osquery/deployments","created_at":"2014-07-29T20:27:33Z","updated_at":"2016-11-01T19:29:31Z","pushed_at":"2016-11-01T20:41:12Z","git_url":"git://github.com/facebook/osquery.git","ssh_url":"git@github.com:facebook/osquery.git","clone_url":"https://github.com/facebook/osquery.git","svn_url":"https://github.com/facebook/osquery","homepage":"https://osquery.io","size":13553,"stargazers_count":7339,"watchers_count":7339,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":831,"mirror_url":null,"open_issues_count":88,"forks":831,"open_issues":88,"watchers":7339,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/facebook/osquery/pulls/2703"},"html":{"href":"https://github.com/facebook/osquery/pull/2703"},"issue":{"href":"https://api.github.com/repos/facebook/osquery/issues/2703"},"comments":{"href":"https://api.github.com/repos/facebook/osquery/issues/2703/comments"},"review_comments":{"href":"https://api.github.com/repos/facebook/osquery/pulls/2703/comments"},"review_comment":{"href":"https://api.github.com/repos/facebook/osquery/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/facebook/osquery/pulls/2703/commits"},"statuses":{"href":"https://api.github.com/repos/facebook/osquery/statuses/9638d26c35b737d8a3fcfb1f64a75d94d94ca1a1"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":1,"additions":33,"deletions":6,"changed_files":2}}
{"actor":{"display_login":"hackgnar"}}