{"action":"opened","number":42,"pull_request":{"url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/pulls/42","id":208423383,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA4NDIzMzgz","html_url":"https://github.com/drone-plugins/drone-s3-sync/pull/42","diff_url":"https://github.com/drone-plugins/drone-s3-sync/pull/42.diff","patch_url":"https://github.com/drone-plugins/drone-s3-sync/pull/42.patch","issue_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/issues/42","number":42,"state":"open","locked":false,"title":"add flag to allow user set dynamic concurrency instead of using hardcoded number","user":{"login":"stephensxu","id":6464677,"node_id":"MDQ6VXNlcjY0NjQ2Nzc=","avatar_url":"https://avatars0.githubusercontent.com/u/6464677?v=4","gravatar_id":"","url":"https://api.github.com/users/stephensxu","html_url":"https://github.com/stephensxu","followers_url":"https://api.github.com/users/stephensxu/followers","following_url":"https://api.github.com/users/stephensxu/following{/other_user}","gists_url":"https://api.github.com/users/stephensxu/gists{/gist_id}","starred_url":"https://api.github.com/users/stephensxu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stephensxu/subscriptions","organizations_url":"https://api.github.com/users/stephensxu/orgs","repos_url":"https://api.github.com/users/stephensxu/repos","events_url":"https://api.github.com/users/stephensxu/events{/privacy}","received_events_url":"https://api.github.com/users/stephensxu/received_events","type":"User","site_admin":false},"body":"We would like to add a feature that allow user to customize number of concurrency. Status quo remains 100 if user does not specify.\r\n\r\nWhy do we want this?\r\n\r\nCurrently, the concurrent number of files being processes is a hard-coded `100`. This would fail at times when there are not enough resources available on the machine.\r\n\r\nAt Skyscanner, we run the `drone-s3-sync` in a docker process that is deployed to a shared AWS EC2 cluster. This docker container gets deployed to a different machine every time build is triggered. Sometime when it got deployed to a busy machine with little resources left, and if it's a big project with thousands of files to sync with S3 bucket(3000 files in our case), the entire process would error out with `too many open files`. \r\n\r\nIn practice this is causing almost 50% of our deployment processes fail due to this error. \r\n\r\nI'm also able to duplicate this problem on my local machine as well, see screenshots below:\r\n\r\n<img width=\"1143\" alt=\"drone-s3-sync-error-cloud\" src=\"https://user-images.githubusercontent.com/6464677/44120826-84fd38a2-9fd2-11e8-8185-98f8c5919902.png\">\r\n\r\n<img width=\"1438\" alt=\"drone-s3-sync-error-local\" src=\"https://user-images.githubusercontent.com/6464677/44120839-8ad0c05a-9fd2-11e8-9b95-4926d86efdc3.png\">\r\n\r\nWith this change, users can set a dynamic concurrency number, allowing us to set a relative conservative concurrency(such as 20 or even 10). \r\n\r\nNote this change DOES NOT change status quo behavior, if user does not have resource limits and does not specify concurrency, it will still use the default `100` value.. ","created_at":"2018-08-14T22:02:54Z","updated_at":"2018-08-14T22:02:54Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/pulls/42/commits","review_comments_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/pulls/42/comments","review_comment_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/pulls/comments{/number}","comments_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/issues/42/comments","statuses_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/statuses/c0496f588c554a55ce3afe2b3e98176f40ebc4b5","head":{"label":"stephensxu:feature/allow_dynamic_concurrency","ref":"feature/allow_dynamic_concurrency","sha":"c0496f588c554a55ce3afe2b3e98176f40ebc4b5","user":{"login":"stephensxu","id":6464677,"node_id":"MDQ6VXNlcjY0NjQ2Nzc=","avatar_url":"https://avatars0.githubusercontent.com/u/6464677?v=4","gravatar_id":"","url":"https://api.github.com/users/stephensxu","html_url":"https://github.com/stephensxu","followers_url":"https://api.github.com/users/stephensxu/followers","following_url":"https://api.github.com/users/stephensxu/following{/other_user}","gists_url":"https://api.github.com/users/stephensxu/gists{/gist_id}","starred_url":"https://api.github.com/users/stephensxu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stephensxu/subscriptions","organizations_url":"https://api.github.com/users/stephensxu/orgs","repos_url":"https://api.github.com/users/stephensxu/repos","events_url":"https://api.github.com/users/stephensxu/events{/privacy}","received_events_url":"https://api.github.com/users/stephensxu/received_events","type":"User","site_admin":false},"repo":{"id":144191938,"node_id":"MDEwOlJlcG9zaXRvcnkxNDQxOTE5Mzg=","name":"drone-s3-sync","full_name":"stephensxu/drone-s3-sync","owner":{"login":"stephensxu","id":6464677,"node_id":"MDQ6VXNlcjY0NjQ2Nzc=","avatar_url":"https://avatars0.githubusercontent.com/u/6464677?v=4","gravatar_id":"","url":"https://api.github.com/users/stephensxu","html_url":"https://github.com/stephensxu","followers_url":"https://api.github.com/users/stephensxu/followers","following_url":"https://api.github.com/users/stephensxu/following{/other_user}","gists_url":"https://api.github.com/users/stephensxu/gists{/gist_id}","starred_url":"https://api.github.com/users/stephensxu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stephensxu/subscriptions","organizations_url":"https://api.github.com/users/stephensxu/orgs","repos_url":"https://api.github.com/users/stephensxu/repos","events_url":"https://api.github.com/users/stephensxu/events{/privacy}","received_events_url":"https://api.github.com/users/stephensxu/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/stephensxu/drone-s3-sync","description":"Drone plugin for syncing files with an Amazon S3 Bucket","fork":true,"url":"https://api.github.com/repos/stephensxu/drone-s3-sync","forks_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/forks","keys_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/keys{/key_id}","collaborators_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/teams","hooks_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/hooks","issue_events_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/issues/events{/number}","events_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/events","assignees_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/assignees{/user}","branches_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/branches{/branch}","tags_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/tags","blobs_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/git/refs{/sha}","trees_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/git/trees{/sha}","statuses_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/statuses/{sha}","languages_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/languages","stargazers_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/stargazers","contributors_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/contributors","subscribers_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/subscribers","subscription_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/subscription","commits_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/commits{/sha}","git_commits_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/git/commits{/sha}","comments_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/comments{/number}","issue_comment_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/issues/comments{/number}","contents_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/contents/{+path}","compare_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/compare/{base}...{head}","merges_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/merges","archive_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/downloads","issues_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/issues{/number}","pulls_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/pulls{/number}","milestones_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/milestones{/number}","notifications_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/labels{/name}","releases_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/releases{/id}","deployments_url":"https://api.github.com/repos/stephensxu/drone-s3-sync/deployments","created_at":"2018-08-09T18:54:22Z","updated_at":"2018-08-09T18:54:24Z","pushed_at":"2018-08-14T21:50:17Z","git_url":"git://github.com/stephensxu/drone-s3-sync.git","ssh_url":"git@github.com:stephensxu/drone-s3-sync.git","clone_url":"https://github.com/stephensxu/drone-s3-sync.git","svn_url":"https://github.com/stephensxu/drone-s3-sync","homepage":"","size":880,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"drone-plugins:master","ref":"master","sha":"d98a437ca38369e3bc3c2660f853d63b23db927b","user":{"login":"drone-plugins","id":11234644,"node_id":"MDEyOk9yZ2FuaXphdGlvbjExMjM0NjQ0","avatar_url":"https://avatars1.githubusercontent.com/u/11234644?v=4","gravatar_id":"","url":"https://api.github.com/users/drone-plugins","html_url":"https://github.com/drone-plugins","followers_url":"https://api.github.com/users/drone-plugins/followers","following_url":"https://api.github.com/users/drone-plugins/following{/other_user}","gists_url":"https://api.github.com/users/drone-plugins/gists{/gist_id}","starred_url":"https://api.github.com/users/drone-plugins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drone-plugins/subscriptions","organizations_url":"https://api.github.com/users/drone-plugins/orgs","repos_url":"https://api.github.com/users/drone-plugins/repos","events_url":"https://api.github.com/users/drone-plugins/events{/privacy}","received_events_url":"https://api.github.com/users/drone-plugins/received_events","type":"Organization","site_admin":false},"repo":{"id":44406904,"node_id":"MDEwOlJlcG9zaXRvcnk0NDQwNjkwNA==","name":"drone-s3-sync","full_name":"drone-plugins/drone-s3-sync","owner":{"login":"drone-plugins","id":11234644,"node_id":"MDEyOk9yZ2FuaXphdGlvbjExMjM0NjQ0","avatar_url":"https://avatars1.githubusercontent.com/u/11234644?v=4","gravatar_id":"","url":"https://api.github.com/users/drone-plugins","html_url":"https://github.com/drone-plugins","followers_url":"https://api.github.com/users/drone-plugins/followers","following_url":"https://api.github.com/users/drone-plugins/following{/other_user}","gists_url":"https://api.github.com/users/drone-plugins/gists{/gist_id}","starred_url":"https://api.github.com/users/drone-plugins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drone-plugins/subscriptions","organizations_url":"https://api.github.com/users/drone-plugins/orgs","repos_url":"https://api.github.com/users/drone-plugins/repos","events_url":"https://api.github.com/users/drone-plugins/events{/privacy}","received_events_url":"https://api.github.com/users/drone-plugins/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/drone-plugins/drone-s3-sync","description":"Drone plugin for syncing files with an Amazon S3 Bucket","fork":false,"url":"https://api.github.com/repos/drone-plugins/drone-s3-sync","forks_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/forks","keys_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/keys{/key_id}","collaborators_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/teams","hooks_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/hooks","issue_events_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/issues/events{/number}","events_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/events","assignees_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/assignees{/user}","branches_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/branches{/branch}","tags_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/tags","blobs_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/git/refs{/sha}","trees_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/git/trees{/sha}","statuses_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/statuses/{sha}","languages_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/languages","stargazers_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/stargazers","contributors_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/contributors","subscribers_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/subscribers","subscription_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/subscription","commits_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/commits{/sha}","git_commits_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/git/commits{/sha}","comments_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/comments{/number}","issue_comment_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/issues/comments{/number}","contents_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/contents/{+path}","compare_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/compare/{base}...{head}","merges_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/merges","archive_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/downloads","issues_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/issues{/number}","pulls_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/pulls{/number}","milestones_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/milestones{/number}","notifications_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/labels{/name}","releases_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/releases{/id}","deployments_url":"https://api.github.com/repos/drone-plugins/drone-s3-sync/deployments","created_at":"2015-10-16T19:42:35Z","updated_at":"2018-07-23T10:59:27Z","pushed_at":"2018-07-01T00:48:03Z","git_url":"git://github.com/drone-plugins/drone-s3-sync.git","ssh_url":"git@github.com:drone-plugins/drone-s3-sync.git","clone_url":"https://github.com/drone-plugins/drone-s3-sync.git","svn_url":"https://github.com/drone-plugins/drone-s3-sync","homepage":"","size":881,"stargazers_count":13,"watchers_count":13,"language":"Go","has_issues":true,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":16,"mirror_url":null,"archived":false,"open_issues_count":7,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":16,"open_issues":7,"watchers":13,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/drone-plugins/drone-s3-sync/pulls/42"},"html":{"href":"https://github.com/drone-plugins/drone-s3-sync/pull/42"},"issue":{"href":"https://api.github.com/repos/drone-plugins/drone-s3-sync/issues/42"},"comments":{"href":"https://api.github.com/repos/drone-plugins/drone-s3-sync/issues/42/comments"},"review_comments":{"href":"https://api.github.com/repos/drone-plugins/drone-s3-sync/pulls/42/comments"},"review_comment":{"href":"https://api.github.com/repos/drone-plugins/drone-s3-sync/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/drone-plugins/drone-s3-sync/pulls/42/commits"},"statuses":{"href":"https://api.github.com/repos/drone-plugins/drone-s3-sync/statuses/c0496f588c554a55ce3afe2b3e98176f40ebc4b5"}},"author_association":"NONE","merged":false,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"maintainer_can_modify":true,"commits":1,"additions":17,"deletions":3,"changed_files":2}}
{"actor":{"display_login":"stephensxu"}}