{"action":"created","comment":{"url":"https://api.github.com/repos/aws/aws-parallelcluster/pulls/comments/238190529","pull_request_review_id":180659880,"id":238190529,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIzODE5MDUyOQ==","diff_hunk":"@@ -249,6 +242,153 @@ def _upload_and_get_command(boto3_factory, args, job_name, region, s3_bucket, lo\n     return command\n \n \n+def _get_stdin_and_upload(boto3_factory, s3_bucket, job_folder, job_script, timeout):\n+    \"\"\"\n+    Create file from STDIN and upload to S3.\n+\n+    :param boto3_factory: initialized Boto3ClientFactory object\n+    :param s3_bucket: S3 bucket\n+    :param job_folder: S3 bucket subfolder\n+    :param job_script: job script name\n+    :param timeout: S3 expiration time in seconds\n+    \"\"\"\n+    try:\n+        # copy stdin to temporary file\n+        with os.fdopen(sys.stdin.fileno(), \"rb\") as src:\n+            with open(job_script, \"wb\") as dst:\n+                shutil.copyfileobj(src, dst)\n+\n+        _upload_to_s3(boto3_factory, s3_bucket, job_script, job_folder + job_script, timeout)","path":"cli/awsbatch/awsbsub.py","position":249,"original_position":249,"commit_id":"0ebfb42d5abe40d32eb6e7d49210108ad2ed3e05","original_commit_id":"0ebfb42d5abe40d32eb6e7d49210108ad2ed3e05","user":{"login":"demartinofra","id":12721611,"node_id":"MDQ6VXNlcjEyNzIxNjEx","avatar_url":"https://avatars0.githubusercontent.com/u/12721611?v=4","gravatar_id":"","url":"https://api.github.com/users/demartinofra","html_url":"https://github.com/demartinofra","followers_url":"https://api.github.com/users/demartinofra/followers","following_url":"https://api.github.com/users/demartinofra/following{/other_user}","gists_url":"https://api.github.com/users/demartinofra/gists{/gist_id}","starred_url":"https://api.github.com/users/demartinofra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/demartinofra/subscriptions","organizations_url":"https://api.github.com/users/demartinofra/orgs","repos_url":"https://api.github.com/users/demartinofra/repos","events_url":"https://api.github.com/users/demartinofra/events{/privacy}","received_events_url":"https://api.github.com/users/demartinofra/received_events","type":"User","site_admin":false},"body":"I'm totally fine with leaving it as it is but this function could become part of an S3Uploader class that holds the factory and the bucket as part of its attributes so that these don't need to be passed around at every invocation.","created_at":"2018-12-03T09:20:20Z","updated_at":"2018-12-03T09:41:53Z","html_url":"https://github.com/aws/aws-parallelcluster/pull/708#discussion_r238190529","pull_request_url":"https://api.github.com/repos/aws/aws-parallelcluster/pulls/708","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/aws/aws-parallelcluster/pulls/comments/238190529"},"html":{"href":"https://github.com/aws/aws-parallelcluster/pull/708#discussion_r238190529"},"pull_request":{"href":"https://api.github.com/repos/aws/aws-parallelcluster/pulls/708"}}},"pull_request":{"url":"https://api.github.com/repos/aws/aws-parallelcluster/pulls/708","id":232980509,"node_id":"MDExOlB1bGxSZXF1ZXN0MjMyOTgwNTA5","html_url":"https://github.com/aws/aws-parallelcluster/pull/708","diff_url":"https://github.com/aws/aws-parallelcluster/pull/708.diff","patch_url":"https://github.com/aws/aws-parallelcluster/pull/708.patch","issue_url":"https://api.github.com/repos/aws/aws-parallelcluster/issues/708","number":708,"state":"open","locked":false,"title":"Add possibility to export environment variables in the job environment","user":{"login":"enrico-usai","id":10634438,"node_id":"MDQ6VXNlcjEwNjM0NDM4","avatar_url":"https://avatars3.githubusercontent.com/u/10634438?v=4","gravatar_id":"","url":"https://api.github.com/users/enrico-usai","html_url":"https://github.com/enrico-usai","followers_url":"https://api.github.com/users/enrico-usai/followers","following_url":"https://api.github.com/users/enrico-usai/following{/other_user}","gists_url":"https://api.github.com/users/enrico-usai/gists{/gist_id}","starred_url":"https://api.github.com/users/enrico-usai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enrico-usai/subscriptions","organizations_url":"https://api.github.com/users/enrico-usai/orgs","repos_url":"https://api.github.com/users/enrico-usai/repos","events_url":"https://api.github.com/users/enrico-usai/events{/privacy}","received_events_url":"https://api.github.com/users/enrico-usai/received_events","type":"User","site_admin":false},"body":"Usage:\r\n\r\n```\r\n  $ echo \"env | sort\" | awsbsub -jn test-env -e CUSTOM1,CUSTOM2\r\n  $ echo \"env | sort\" | awsbsub -jn test-all-env -e all\r\n  $ echo \"env | sort\" | awsbsub -jn test-blacklisted-env -e all -eb CUSTOM1,CUSTOM2\r\n```\r\n\r\nI also changed the s3 folder for the uploaded script from: `s3_bucket/batch/` to `s3_bucket/batch/jobid/`\r\nThis because now we have two files, one for the exported environment and the user script.","created_at":"2018-11-22T13:31:00Z","updated_at":"2018-12-03T09:41:53Z","closed_at":null,"merged_at":null,"merge_commit_sha":"cb6821f46de2b4276d5904067b86603ef3d20e6f","assignee":null,"assignees":[],"requested_reviewers":[{"login":"mauri-melato","id":1615209,"node_id":"MDQ6VXNlcjE2MTUyMDk=","avatar_url":"https://avatars3.githubusercontent.com/u/1615209?v=4","gravatar_id":"","url":"https://api.github.com/users/mauri-melato","html_url":"https://github.com/mauri-melato","followers_url":"https://api.github.com/users/mauri-melato/followers","following_url":"https://api.github.com/users/mauri-melato/following{/other_user}","gists_url":"https://api.github.com/users/mauri-melato/gists{/gist_id}","starred_url":"https://api.github.com/users/mauri-melato/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mauri-melato/subscriptions","organizations_url":"https://api.github.com/users/mauri-melato/orgs","repos_url":"https://api.github.com/users/mauri-melato/repos","events_url":"https://api.github.com/users/mauri-melato/events{/privacy}","received_events_url":"https://api.github.com/users/mauri-melato/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/aws/aws-parallelcluster/pulls/708/commits","review_comments_url":"https://api.github.com/repos/aws/aws-parallelcluster/pulls/708/comments","review_comment_url":"https://api.github.com/repos/aws/aws-parallelcluster/pulls/comments{/number}","comments_url":"https://api.github.com/repos/aws/aws-parallelcluster/issues/708/comments","statuses_url":"https://api.github.com/repos/aws/aws-parallelcluster/statuses/0ebfb42d5abe40d32eb6e7d49210108ad2ed3e05","head":{"label":"enrico-usai:wip/env","ref":"wip/env","sha":"0ebfb42d5abe40d32eb6e7d49210108ad2ed3e05","user":{"login":"enrico-usai","id":10634438,"node_id":"MDQ6VXNlcjEwNjM0NDM4","avatar_url":"https://avatars3.githubusercontent.com/u/10634438?v=4","gravatar_id":"","url":"https://api.github.com/users/enrico-usai","html_url":"https://github.com/enrico-usai","followers_url":"https://api.github.com/users/enrico-usai/followers","following_url":"https://api.github.com/users/enrico-usai/following{/other_user}","gists_url":"https://api.github.com/users/enrico-usai/gists{/gist_id}","starred_url":"https://api.github.com/users/enrico-usai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enrico-usai/subscriptions","organizations_url":"https://api.github.com/users/enrico-usai/orgs","repos_url":"https://api.github.com/users/enrico-usai/repos","events_url":"https://api.github.com/users/enrico-usai/events{/privacy}","received_events_url":"https://api.github.com/users/enrico-usai/received_events","type":"User","site_admin":false},"repo":{"id":131268556,"node_id":"MDEwOlJlcG9zaXRvcnkxMzEyNjg1NTY=","name":"aws-parallelcluster","full_name":"enrico-usai/aws-parallelcluster","private":false,"owner":{"login":"enrico-usai","id":10634438,"node_id":"MDQ6VXNlcjEwNjM0NDM4","avatar_url":"https://avatars3.githubusercontent.com/u/10634438?v=4","gravatar_id":"","url":"https://api.github.com/users/enrico-usai","html_url":"https://github.com/enrico-usai","followers_url":"https://api.github.com/users/enrico-usai/followers","following_url":"https://api.github.com/users/enrico-usai/following{/other_user}","gists_url":"https://api.github.com/users/enrico-usai/gists{/gist_id}","starred_url":"https://api.github.com/users/enrico-usai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enrico-usai/subscriptions","organizations_url":"https://api.github.com/users/enrico-usai/orgs","repos_url":"https://api.github.com/users/enrico-usai/repos","events_url":"https://api.github.com/users/enrico-usai/events{/privacy}","received_events_url":"https://api.github.com/users/enrico-usai/received_events","type":"User","site_admin":false},"html_url":"https://github.com/enrico-usai/aws-parallelcluster","description":"CfnCluster is a framework that deploys and maintains HPC clusters on AWS.","fork":true,"url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster","forks_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/forks","keys_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/keys{/key_id}","collaborators_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/teams","hooks_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/hooks","issue_events_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/issues/events{/number}","events_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/events","assignees_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/assignees{/user}","branches_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/branches{/branch}","tags_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/tags","blobs_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/git/refs{/sha}","trees_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/git/trees{/sha}","statuses_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/statuses/{sha}","languages_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/languages","stargazers_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/stargazers","contributors_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/contributors","subscribers_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/subscribers","subscription_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/subscription","commits_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/commits{/sha}","git_commits_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/git/commits{/sha}","comments_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/comments{/number}","issue_comment_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/issues/comments{/number}","contents_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/contents/{+path}","compare_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/compare/{base}...{head}","merges_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/merges","archive_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/downloads","issues_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/issues{/number}","pulls_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/pulls{/number}","milestones_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/milestones{/number}","notifications_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/labels{/name}","releases_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/releases{/id}","deployments_url":"https://api.github.com/repos/enrico-usai/aws-parallelcluster/deployments","created_at":"2018-04-27T08:28:08Z","updated_at":"2018-11-30T16:23:00Z","pushed_at":"2018-12-03T09:01:13Z","git_url":"git://github.com/enrico-usai/aws-parallelcluster.git","ssh_url":"git@github.com:enrico-usai/aws-parallelcluster.git","clone_url":"https://github.com/enrico-usai/aws-parallelcluster.git","svn_url":"https://github.com/enrico-usai/aws-parallelcluster","homepage":"http://aws.amazon.com/hpc/cfncluster","size":2891,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"develop"}},"base":{"label":"aws:develop","ref":"develop","sha":"cf925014be6936ae8d5a05fce60ed0757b484447","user":{"login":"aws","id":2232217,"node_id":"MDEyOk9yZ2FuaXphdGlvbjIyMzIyMTc=","avatar_url":"https://avatars3.githubusercontent.com/u/2232217?v=4","gravatar_id":"","url":"https://api.github.com/users/aws","html_url":"https://github.com/aws","followers_url":"https://api.github.com/users/aws/followers","following_url":"https://api.github.com/users/aws/following{/other_user}","gists_url":"https://api.github.com/users/aws/gists{/gist_id}","starred_url":"https://api.github.com/users/aws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aws/subscriptions","organizations_url":"https://api.github.com/users/aws/orgs","repos_url":"https://api.github.com/users/aws/repos","events_url":"https://api.github.com/users/aws/events{/privacy}","received_events_url":"https://api.github.com/users/aws/received_events","type":"Organization","site_admin":false},"repo":{"id":19718034,"node_id":"MDEwOlJlcG9zaXRvcnkxOTcxODAzNA==","name":"aws-parallelcluster","full_name":"aws/aws-parallelcluster","private":false,"owner":{"login":"aws","id":2232217,"node_id":"MDEyOk9yZ2FuaXphdGlvbjIyMzIyMTc=","avatar_url":"https://avatars3.githubusercontent.com/u/2232217?v=4","gravatar_id":"","url":"https://api.github.com/users/aws","html_url":"https://github.com/aws","followers_url":"https://api.github.com/users/aws/followers","following_url":"https://api.github.com/users/aws/following{/other_user}","gists_url":"https://api.github.com/users/aws/gists{/gist_id}","starred_url":"https://api.github.com/users/aws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aws/subscriptions","organizations_url":"https://api.github.com/users/aws/orgs","repos_url":"https://api.github.com/users/aws/repos","events_url":"https://api.github.com/users/aws/events{/privacy}","received_events_url":"https://api.github.com/users/aws/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/aws/aws-parallelcluster","description":"AWS ParallelCluster is an AWS supported Open Source cluster management tool to deploy and manage HPC clusters in the AWS cloud.","fork":false,"url":"https://api.github.com/repos/aws/aws-parallelcluster","forks_url":"https://api.github.com/repos/aws/aws-parallelcluster/forks","keys_url":"https://api.github.com/repos/aws/aws-parallelcluster/keys{/key_id}","collaborators_url":"https://api.github.com/repos/aws/aws-parallelcluster/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/aws/aws-parallelcluster/teams","hooks_url":"https://api.github.com/repos/aws/aws-parallelcluster/hooks","issue_events_url":"https://api.github.com/repos/aws/aws-parallelcluster/issues/events{/number}","events_url":"https://api.github.com/repos/aws/aws-parallelcluster/events","assignees_url":"https://api.github.com/repos/aws/aws-parallelcluster/assignees{/user}","branches_url":"https://api.github.com/repos/aws/aws-parallelcluster/branches{/branch}","tags_url":"https://api.github.com/repos/aws/aws-parallelcluster/tags","blobs_url":"https://api.github.com/repos/aws/aws-parallelcluster/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/aws/aws-parallelcluster/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/aws/aws-parallelcluster/git/refs{/sha}","trees_url":"https://api.github.com/repos/aws/aws-parallelcluster/git/trees{/sha}","statuses_url":"https://api.github.com/repos/aws/aws-parallelcluster/statuses/{sha}","languages_url":"https://api.github.com/repos/aws/aws-parallelcluster/languages","stargazers_url":"https://api.github.com/repos/aws/aws-parallelcluster/stargazers","contributors_url":"https://api.github.com/repos/aws/aws-parallelcluster/contributors","subscribers_url":"https://api.github.com/repos/aws/aws-parallelcluster/subscribers","subscription_url":"https://api.github.com/repos/aws/aws-parallelcluster/subscription","commits_url":"https://api.github.com/repos/aws/aws-parallelcluster/commits{/sha}","git_commits_url":"https://api.github.com/repos/aws/aws-parallelcluster/git/commits{/sha}","comments_url":"https://api.github.com/repos/aws/aws-parallelcluster/comments{/number}","issue_comment_url":"https://api.github.com/repos/aws/aws-parallelcluster/issues/comments{/number}","contents_url":"https://api.github.com/repos/aws/aws-parallelcluster/contents/{+path}","compare_url":"https://api.github.com/repos/aws/aws-parallelcluster/compare/{base}...{head}","merges_url":"https://api.github.com/repos/aws/aws-parallelcluster/merges","archive_url":"https://api.github.com/repos/aws/aws-parallelcluster/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/aws/aws-parallelcluster/downloads","issues_url":"https://api.github.com/repos/aws/aws-parallelcluster/issues{/number}","pulls_url":"https://api.github.com/repos/aws/aws-parallelcluster/pulls{/number}","milestones_url":"https://api.github.com/repos/aws/aws-parallelcluster/milestones{/number}","notifications_url":"https://api.github.com/repos/aws/aws-parallelcluster/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/aws/aws-parallelcluster/labels{/name}","releases_url":"https://api.github.com/repos/aws/aws-parallelcluster/releases{/id}","deployments_url":"https://api.github.com/repos/aws/aws-parallelcluster/deployments","created_at":"2014-05-12T22:42:19Z","updated_at":"2018-12-03T08:58:15Z","pushed_at":"2018-12-03T09:01:15Z","git_url":"git://github.com/aws/aws-parallelcluster.git","ssh_url":"git@github.com:aws/aws-parallelcluster.git","clone_url":"https://github.com/aws/aws-parallelcluster.git","svn_url":"https://github.com/aws/aws-parallelcluster","homepage":"https://github.com/aws/aws-parallelcluster","size":2923,"stargazers_count":329,"watchers_count":329,"language":"Python","has_issues":true,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":127,"mirror_url":null,"archived":false,"open_issues_count":86,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":127,"open_issues":86,"watchers":329,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/aws/aws-parallelcluster/pulls/708"},"html":{"href":"https://github.com/aws/aws-parallelcluster/pull/708"},"issue":{"href":"https://api.github.com/repos/aws/aws-parallelcluster/issues/708"},"comments":{"href":"https://api.github.com/repos/aws/aws-parallelcluster/issues/708/comments"},"review_comments":{"href":"https://api.github.com/repos/aws/aws-parallelcluster/pulls/708/comments"},"review_comment":{"href":"https://api.github.com/repos/aws/aws-parallelcluster/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/aws/aws-parallelcluster/pulls/708/commits"},"statuses":{"href":"https://api.github.com/repos/aws/aws-parallelcluster/statuses/0ebfb42d5abe40d32eb6e7d49210108ad2ed3e05"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"demartinofra"}}