{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/167062542","pull_request_review_id":95225640,"id":167062542,"diff_hunk":"@@ -17,16 +17,21 @@\n \n package org.apache.spark.scheduler.cluster.mesos\n \n+import java.io.{File, FileNotFoundException}\n+\n import scala.collection.JavaConverters._\n+import scala.collection.immutable.Map","path":"resource-managers/mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtilsSuite.scala","position":7,"original_position":7,"commit_id":"8903ba641b4713799e6739ce71c9baf81183127f","original_commit_id":"8903ba641b4713799e6739ce71c9baf81183127f","user":{"login":"vanzin","id":1694083,"avatar_url":"https://avatars0.githubusercontent.com/u/1694083?v=4","gravatar_id":"","url":"https://api.github.com/users/vanzin","html_url":"https://github.com/vanzin","followers_url":"https://api.github.com/users/vanzin/followers","following_url":"https://api.github.com/users/vanzin/following{/other_user}","gists_url":"https://api.github.com/users/vanzin/gists{/gist_id}","starred_url":"https://api.github.com/users/vanzin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanzin/subscriptions","organizations_url":"https://api.github.com/users/vanzin/orgs","repos_url":"https://api.github.com/users/vanzin/repos","events_url":"https://api.github.com/users/vanzin/events{/privacy}","received_events_url":"https://api.github.com/users/vanzin/received_events","type":"User","site_admin":false},"body":"Why do you need this explicit import?","created_at":"2018-02-08T20:48:08Z","updated_at":"2018-02-08T20:50:12Z","html_url":"https://github.com/apache/spark/pull/20167#discussion_r167062542","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/20167","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/167062542"},"html":{"href":"https://github.com/apache/spark/pull/20167#discussion_r167062542"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/20167"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/20167","id":161389606,"html_url":"https://github.com/apache/spark/pull/20167","diff_url":"https://github.com/apache/spark/pull/20167.diff","patch_url":"https://github.com/apache/spark/pull/20167.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/20167","number":20167,"state":"open","locked":false,"title":"[SPARK-16501] [MESOS] Allow providing Mesos principal & secret via files ","user":{"login":"rvesse","id":2104864,"avatar_url":"https://avatars0.githubusercontent.com/u/2104864?v=4","gravatar_id":"","url":"https://api.github.com/users/rvesse","html_url":"https://github.com/rvesse","followers_url":"https://api.github.com/users/rvesse/followers","following_url":"https://api.github.com/users/rvesse/following{/other_user}","gists_url":"https://api.github.com/users/rvesse/gists{/gist_id}","starred_url":"https://api.github.com/users/rvesse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rvesse/subscriptions","organizations_url":"https://api.github.com/users/rvesse/orgs","repos_url":"https://api.github.com/users/rvesse/repos","events_url":"https://api.github.com/users/rvesse/events{/privacy}","received_events_url":"https://api.github.com/users/rvesse/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nThis commit modifies the Mesos submission client to allow the principal\r\nand secret to be provided indirectly via files.  The path to these files\r\ncan be specified either via Spark configuration or via environment\r\nvariable.\r\n\r\nAssuming these files are appropriately protected by FS/OS permissions\r\nthis means we don't ever leak the actual values in process info like ps\r\n\r\nEnvironment variable specification is useful because it allows you to\r\ninterpolate the location of this file when using per-user Mesos\r\ncredentials.\r\n\r\nFor some background as to why we have taken this approach I will briefly describe our set up.  On our systems we provide each authorised user account with their own Mesos credentials to provide certain security and audit guarantees to our customers. These credentials are managed by a central Secret management service. In our `spark-env.sh` we determine the appropriate secret and principal files to use depending on the user who is invoking Spark hence the need to inject these via environment variables as well as by configuration properties. So we set these environment variables appropriately and our Spark read in the contents of those files to authenticate itself with Mesos.\r\n\r\n## How was this patch tested?\r\n\r\nThis is functionality we have been using it in production across multiple customer sites for some time. This has been in the field for around 18 months with no reported issues. These changes have been sufficient to meet our customer security and audit requirements.\r\n\r\nWe have been building and deploying custom builds of Apache Spark with various minor tweaks like this which we are now looking to contribute back into the community in order that we can rely upon stock Apache Spark builds and stop maintaining our own internal fork. ","created_at":"2018-01-05T17:33:14Z","updated_at":"2018-02-08T20:50:13Z","closed_at":null,"merged_at":null,"merge_commit_sha":"1b99e16d87ffea1267117e69fea30c34759a0963","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/20167/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/20167/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/20167/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/8903ba641b4713799e6739ce71c9baf81183127f","head":{"label":"rvesse:SPARK-16501","ref":"SPARK-16501","sha":"8903ba641b4713799e6739ce71c9baf81183127f","user":{"login":"rvesse","id":2104864,"avatar_url":"https://avatars0.githubusercontent.com/u/2104864?v=4","gravatar_id":"","url":"https://api.github.com/users/rvesse","html_url":"https://github.com/rvesse","followers_url":"https://api.github.com/users/rvesse/followers","following_url":"https://api.github.com/users/rvesse/following{/other_user}","gists_url":"https://api.github.com/users/rvesse/gists{/gist_id}","starred_url":"https://api.github.com/users/rvesse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rvesse/subscriptions","organizations_url":"https://api.github.com/users/rvesse/orgs","repos_url":"https://api.github.com/users/rvesse/repos","events_url":"https://api.github.com/users/rvesse/events{/privacy}","received_events_url":"https://api.github.com/users/rvesse/received_events","type":"User","site_admin":false},"repo":{"id":29888493,"name":"spark","full_name":"rvesse/spark","owner":{"login":"rvesse","id":2104864,"avatar_url":"https://avatars0.githubusercontent.com/u/2104864?v=4","gravatar_id":"","url":"https://api.github.com/users/rvesse","html_url":"https://github.com/rvesse","followers_url":"https://api.github.com/users/rvesse/followers","following_url":"https://api.github.com/users/rvesse/following{/other_user}","gists_url":"https://api.github.com/users/rvesse/gists{/gist_id}","starred_url":"https://api.github.com/users/rvesse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rvesse/subscriptions","organizations_url":"https://api.github.com/users/rvesse/orgs","repos_url":"https://api.github.com/users/rvesse/repos","events_url":"https://api.github.com/users/rvesse/events{/privacy}","received_events_url":"https://api.github.com/users/rvesse/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/rvesse/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/rvesse/spark","forks_url":"https://api.github.com/repos/rvesse/spark/forks","keys_url":"https://api.github.com/repos/rvesse/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/rvesse/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/rvesse/spark/teams","hooks_url":"https://api.github.com/repos/rvesse/spark/hooks","issue_events_url":"https://api.github.com/repos/rvesse/spark/issues/events{/number}","events_url":"https://api.github.com/repos/rvesse/spark/events","assignees_url":"https://api.github.com/repos/rvesse/spark/assignees{/user}","branches_url":"https://api.github.com/repos/rvesse/spark/branches{/branch}","tags_url":"https://api.github.com/repos/rvesse/spark/tags","blobs_url":"https://api.github.com/repos/rvesse/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/rvesse/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/rvesse/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/rvesse/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/rvesse/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/rvesse/spark/languages","stargazers_url":"https://api.github.com/repos/rvesse/spark/stargazers","contributors_url":"https://api.github.com/repos/rvesse/spark/contributors","subscribers_url":"https://api.github.com/repos/rvesse/spark/subscribers","subscription_url":"https://api.github.com/repos/rvesse/spark/subscription","commits_url":"https://api.github.com/repos/rvesse/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/rvesse/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/rvesse/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/rvesse/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/rvesse/spark/contents/{+path}","compare_url":"https://api.github.com/repos/rvesse/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/rvesse/spark/merges","archive_url":"https://api.github.com/repos/rvesse/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/rvesse/spark/downloads","issues_url":"https://api.github.com/repos/rvesse/spark/issues{/number}","pulls_url":"https://api.github.com/repos/rvesse/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/rvesse/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/rvesse/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/rvesse/spark/labels{/name}","releases_url":"https://api.github.com/repos/rvesse/spark/releases{/id}","deployments_url":"https://api.github.com/repos/rvesse/spark/deployments","created_at":"2015-01-26T23:59:15Z","updated_at":"2018-01-05T16:43:34Z","pushed_at":"2018-02-06T10:44:26Z","git_url":"git://github.com/rvesse/spark.git","ssh_url":"git@github.com:rvesse/spark.git","clone_url":"https://github.com/rvesse/spark.git","svn_url":"https://github.com/rvesse/spark","homepage":null,"size":244419,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"51c33bd0d402af9e0284c6cbc0111f926446bfba","user":{"login":"apache","id":47359,"avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2018-02-08T16:58:22Z","pushed_at":"2018-02-08T20:48:54Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":267350,"stargazers_count":16107,"watchers_count":16107,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":14905,"mirror_url":null,"archived":false,"open_issues_count":425,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":14905,"open_issues":425,"watchers":16107,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/20167"},"html":{"href":"https://github.com/apache/spark/pull/20167"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/20167"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/20167/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/20167/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/20167/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/8903ba641b4713799e6739ce71c9baf81183127f"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"vanzin"}}