{"action":"created","issue":{"url":"https://api.github.com/repos/markitx/dynamo-backup-to-s3/issues/20","repository_url":"https://api.github.com/repos/markitx/dynamo-backup-to-s3","labels_url":"https://api.github.com/repos/markitx/dynamo-backup-to-s3/issues/20/labels{/name}","comments_url":"https://api.github.com/repos/markitx/dynamo-backup-to-s3/issues/20/comments","events_url":"https://api.github.com/repos/markitx/dynamo-backup-to-s3/issues/20/events","html_url":"https://github.com/markitx/dynamo-backup-to-s3/issues/20","id":162719768,"number":20,"title":"Missing items in backup","user":{"login":"nathanwelch","id":498595,"avatar_url":"https://avatars.githubusercontent.com/u/498595?v=3","gravatar_id":"","url":"https://api.github.com/users/nathanwelch","html_url":"https://github.com/nathanwelch","followers_url":"https://api.github.com/users/nathanwelch/followers","following_url":"https://api.github.com/users/nathanwelch/following{/other_user}","gists_url":"https://api.github.com/users/nathanwelch/gists{/gist_id}","starred_url":"https://api.github.com/users/nathanwelch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nathanwelch/subscriptions","organizations_url":"https://api.github.com/users/nathanwelch/orgs","repos_url":"https://api.github.com/users/nathanwelch/repos","events_url":"https://api.github.com/users/nathanwelch/events{/privacy}","received_events_url":"https://api.github.com/users/nathanwelch/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"dylanlingelbach","id":213594,"avatar_url":"https://avatars.githubusercontent.com/u/213594?v=3","gravatar_id":"","url":"https://api.github.com/users/dylanlingelbach","html_url":"https://github.com/dylanlingelbach","followers_url":"https://api.github.com/users/dylanlingelbach/followers","following_url":"https://api.github.com/users/dylanlingelbach/following{/other_user}","gists_url":"https://api.github.com/users/dylanlingelbach/gists{/gist_id}","starred_url":"https://api.github.com/users/dylanlingelbach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dylanlingelbach/subscriptions","organizations_url":"https://api.github.com/users/dylanlingelbach/orgs","repos_url":"https://api.github.com/users/dylanlingelbach/repos","events_url":"https://api.github.com/users/dylanlingelbach/events{/privacy}","received_events_url":"https://api.github.com/users/dylanlingelbach/received_events","type":"User","site_admin":false},"assignees":[{"login":"dylanlingelbach","id":213594,"avatar_url":"https://avatars.githubusercontent.com/u/213594?v=3","gravatar_id":"","url":"https://api.github.com/users/dylanlingelbach","html_url":"https://github.com/dylanlingelbach","followers_url":"https://api.github.com/users/dylanlingelbach/followers","following_url":"https://api.github.com/users/dylanlingelbach/following{/other_user}","gists_url":"https://api.github.com/users/dylanlingelbach/gists{/gist_id}","starred_url":"https://api.github.com/users/dylanlingelbach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dylanlingelbach/subscriptions","organizations_url":"https://api.github.com/users/dylanlingelbach/orgs","repos_url":"https://api.github.com/users/dylanlingelbach/repos","events_url":"https://api.github.com/users/dylanlingelbach/events{/privacy}","received_events_url":"https://api.github.com/users/dylanlingelbach/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2016-06-28T15:47:17Z","updated_at":"2016-07-12T01:06:11Z","closed_at":null,"body":"Hi I have a table with ~1200 items and when I back it up I used to only get about half of them on version 0.1.9. I updated to the latest version and now I get more but still not all. Interestingly, the percentage of records backed up goes **down** as I increase the read throughput. For example, I get ~850 when I have read percentage at 75% but ~1100 at 25%. \r\n\r\nI did some initial debugging by modifying the code to emit an event for every item that's processed and I can confirm that all items are processed. Additionally if I modify dynamo-backup.js to save all records to a file and then upload that file, I can confirm they are all processed and uploaded. To me this points to some bug either in the readable stream or in what is handling the S3 upload stream. Given that the percentage of items backed up goes up as the read rate goes down, I'm guessing it's reading from dynamo faster than it can write to S3 and that the readable stream cannot handle this.\r\n\r\nIs anyone aware of any issues that would cause this? My temporary fix in my fork (https://github.com/markitx/dynamo-backup-to-s3/compare/master...smashgg:backup-to-file) is to add an option to backup to file and then upload that file after the dynamo backup has completed. \r\n\r\nNode version: Tried in 0.10.x, 0.12.x, and 6.2.2 with fresh installs between each version change\r\ndynamo-backup-to-s3 version: 0.4.x\r\n"},"comment":{"url":"https://api.github.com/repos/markitx/dynamo-backup-to-s3/issues/comments/231910275","html_url":"https://github.com/markitx/dynamo-backup-to-s3/issues/20#issuecomment-231910275","issue_url":"https://api.github.com/repos/markitx/dynamo-backup-to-s3/issues/20","id":231910275,"user":{"login":"dylanlingelbach","id":213594,"avatar_url":"https://avatars.githubusercontent.com/u/213594?v=3","gravatar_id":"","url":"https://api.github.com/users/dylanlingelbach","html_url":"https://github.com/dylanlingelbach","followers_url":"https://api.github.com/users/dylanlingelbach/followers","following_url":"https://api.github.com/users/dylanlingelbach/following{/other_user}","gists_url":"https://api.github.com/users/dylanlingelbach/gists{/gist_id}","starred_url":"https://api.github.com/users/dylanlingelbach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dylanlingelbach/subscriptions","organizations_url":"https://api.github.com/users/dylanlingelbach/orgs","repos_url":"https://api.github.com/users/dylanlingelbach/repos","events_url":"https://api.github.com/users/dylanlingelbach/events{/privacy}","received_events_url":"https://api.github.com/users/dylanlingelbach/received_events","type":"User","site_admin":false},"created_at":"2016-07-12T01:06:11Z","updated_at":"2016-07-12T01:06:11Z","body":"This is very odd. I'll try to test some tomorrow. I haven't been using this project lately so difficult for me to test. Let me know if anyone gets more data here"}}
{"actor":{"display_login":"dylanlingelbach"}}