{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/55550641","id":55550641,"diff_hunk":"@@ -441,3 +441,91 @@ If you need a reference to the proper location to put log files in the YARN so t\n - In `cluster` mode, the local directories used by the Spark executors and the Spark driver will be the local directories configured for YARN (Hadoop YARN config `yarn.nodemanager.local-dirs`). If the user specifies `spark.local.dir`, it will be ignored. In `client` mode, the Spark executors will use the local directories configured for YARN while the Spark driver will use those defined in `spark.local.dir`. This is because the Spark driver does not run on the YARN cluster in `client` mode, only the Spark executors do.\n - The `--files` and `--archives` options support specifying file names with the # similar to Hadoop. For example you can specify: `--files localtest.txt#appSees.txt` and this will upload the file you have locally named `localtest.txt` into HDFS but this will be linked to by the name `appSees.txt`, and your application should use the name as `appSees.txt` to reference it when running on YARN.\n - The `--jars` option allows the `SparkContext.addJar` function to work if you are using it with local files and running in `cluster` mode. It does not need to be used if you are using it with HDFS, HTTP, HTTPS, or FTP files.\n+\n+# Running in a secure YARN cluster\n+\n+As covered in [security](security.html), Kerberos is used in a secure YARN cluster to\n+authenticate principals associated with services and clients. This allows clients to\n+make requests of these authenticated services; the services to grant rights\n+to the authenticated principals.\n+\n+Hadoop services issue *hadoop tokens* to grant access to the services and data,\n+tokens which the client must supply over Hadoop IPC and REST/Web APIs as proof of access rights.\n+For Spark applications launched in a YARN cluster to interact with HDFS, HBase and Hive,\n+the application must acquire the relevant tokens\n+using the Kerberos credentials of the user launching the application —that is, the principal whose\n+identity will become that of the launched Spark application.\n+\n+This is normally done at launch time: in a secure cluster Spark will automatically obtain a\n+token for the cluster's HDFS filesystem, and, if required, HBase and Hive.\n+\n+If an application needs to interact with other secure HDFS clusters, then\n+the tokens needed to access these clusters must be explicitly requested at\n+launch time. This is done by listing them in the `spark.yarn.access.namenodes` property.\n+\n+```\n+spark.yarn.access.namenodes hdfs://ireland.emea.example.org:8020/,hdfs://frankfurt.emea.example.org:8020/\n+```\n+\n+Hadoop tokens expire. They can be renewed \"for a while\";\n+However, eventually, they will stop being renewable —after which all attempts to\n+access secure data will fail. The only way to avoid that is for the application to be launched\n+with the secrets needed to log in to Kerberos directly: a \"keytab\". Consult\n+the [Spark Property](#Spark Properties) `spark.yarn.keytab` for the specifics.\n+\n+## Launching your application with Apache Oozie\n+\n+Apache Oozie can launch Spark.\n+In a secure cluster, such an application will need the relevant tokens to access the cluster's\n+services. If Spark is launched with a keytab, this is automatic.\n+However, if Spark is to be launched without a keytab, the responsibility for setting up security\n+must be handed over to Oozie.\n+\n+The details of configuring Oozie for secure clusters and obtaining\n+credentials for a job can be found on the [Oozie web site](http://oozie.apache.org/)\n+in the \"Authentication\" section of the specific release's documentation.\n+ \n+For Spark applications, the Oozie workflow must be set up for Oozie to request all tokens, including:","path":"docs/running-on-yarn.md","position":48,"original_position":48,"commit_id":"3dbcac0f6960c6707edad355d5f134074f8ab122","original_commit_id":"3dbcac0f6960c6707edad355d5f134074f8ab122","user":{"login":"tgravescs","id":4563792,"avatar_url":"https://avatars.githubusercontent.com/u/4563792?v=3","gravatar_id":"","url":"https://api.github.com/users/tgravescs","html_url":"https://github.com/tgravescs","followers_url":"https://api.github.com/users/tgravescs/followers","following_url":"https://api.github.com/users/tgravescs/following{/other_user}","gists_url":"https://api.github.com/users/tgravescs/gists{/gist_id}","starred_url":"https://api.github.com/users/tgravescs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgravescs/subscriptions","organizations_url":"https://api.github.com/users/tgravescs/orgs","repos_url":"https://api.github.com/users/tgravescs/repos","events_url":"https://api.github.com/users/tgravescs/events{/privacy}","received_events_url":"https://api.github.com/users/tgravescs/received_events","type":"User","site_admin":false},"body":"This isn't completely true as if I'm not doing hive or hbase I don't need those.  Can we change to clarify that those are optional.","created_at":"2016-03-09T16:59:05Z","updated_at":"2016-03-09T16:59:05Z","html_url":"https://github.com/apache/spark/pull/11033#discussion_r55550641","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/11033","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/55550641"},"html":{"href":"https://github.com/apache/spark/pull/11033#discussion_r55550641"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/11033"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11033","id":58059938,"html_url":"https://github.com/apache/spark/pull/11033","diff_url":"https://github.com/apache/spark/pull/11033.diff","patch_url":"https://github.com/apache/spark/pull/11033.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/11033","number":11033,"state":"open","locked":false,"title":"[SPARK-13148] [YARN] zero-keytab Oozie application launch via HADOOP_TOKEN_FILE_LOCATION env var","user":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"body":"This patch looks for the env var `HADOOP_TOKEN_FILE_LOCATION`, and if set skips trying to collect new delegation tokens. Instead it relies on the environment (i.e. oozie) passing them down...it does list them all to identify when the oozie configuration forgets something.\r\n\r\nIt's tagged as a WiP as I haven't tested it yet; I'm currently (elsewhere) writing a bit of code to collect HDFS and RM tokens and save them to a file, ready to set up the env var. I could add this code into spark, where it would be able to use the HBase and Hive reflection logic to add these. This would just be for testing, to isolate Spark operation from Oozie.","created_at":"2016-02-02T22:21:15Z","updated_at":"2016-03-09T16:59:05Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b38fe053c557169546eae1151b2a138778436c6d","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/11033/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/11033/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/11033/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/3dbcac0f6960c6707edad355d5f134074f8ab122","head":{"label":"steveloughran:stevel/feature/SPARK-13148-oozie","ref":"stevel/feature/SPARK-13148-oozie","sha":"3dbcac0f6960c6707edad355d5f134074f8ab122","user":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"repo":{"id":32387210,"name":"spark","full_name":"steveloughran/spark","owner":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/steveloughran/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/steveloughran/spark","forks_url":"https://api.github.com/repos/steveloughran/spark/forks","keys_url":"https://api.github.com/repos/steveloughran/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/steveloughran/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/steveloughran/spark/teams","hooks_url":"https://api.github.com/repos/steveloughran/spark/hooks","issue_events_url":"https://api.github.com/repos/steveloughran/spark/issues/events{/number}","events_url":"https://api.github.com/repos/steveloughran/spark/events","assignees_url":"https://api.github.com/repos/steveloughran/spark/assignees{/user}","branches_url":"https://api.github.com/repos/steveloughran/spark/branches{/branch}","tags_url":"https://api.github.com/repos/steveloughran/spark/tags","blobs_url":"https://api.github.com/repos/steveloughran/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/steveloughran/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/steveloughran/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/steveloughran/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/steveloughran/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/steveloughran/spark/languages","stargazers_url":"https://api.github.com/repos/steveloughran/spark/stargazers","contributors_url":"https://api.github.com/repos/steveloughran/spark/contributors","subscribers_url":"https://api.github.com/repos/steveloughran/spark/subscribers","subscription_url":"https://api.github.com/repos/steveloughran/spark/subscription","commits_url":"https://api.github.com/repos/steveloughran/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/steveloughran/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/steveloughran/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/steveloughran/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/steveloughran/spark/contents/{+path}","compare_url":"https://api.github.com/repos/steveloughran/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/steveloughran/spark/merges","archive_url":"https://api.github.com/repos/steveloughran/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/steveloughran/spark/downloads","issues_url":"https://api.github.com/repos/steveloughran/spark/issues{/number}","pulls_url":"https://api.github.com/repos/steveloughran/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/steveloughran/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/steveloughran/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/steveloughran/spark/labels{/name}","releases_url":"https://api.github.com/repos/steveloughran/spark/releases{/id}","deployments_url":"https://api.github.com/repos/steveloughran/spark/deployments","created_at":"2015-03-17T10:36:33Z","updated_at":"2015-03-17T10:36:52Z","pushed_at":"2016-03-09T11:52:23Z","git_url":"git://github.com/steveloughran/spark.git","ssh_url":"git@github.com:steveloughran/spark.git","clone_url":"https://github.com/steveloughran/spark.git","svn_url":"https://github.com/steveloughran/spark","homepage":null,"size":146836,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"c3689bc24e03a9471cd6e8169da61963c4528252","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2016-03-09T16:00:12Z","pushed_at":"2016-03-09T16:34:39Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":439589,"stargazers_count":7729,"watchers_count":7729,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":7053,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":403,"forks":7053,"open_issues":403,"watchers":7729,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/11033"},"html":{"href":"https://github.com/apache/spark/pull/11033"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/11033"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/11033/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/11033/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/11033/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/3dbcac0f6960c6707edad355d5f134074f8ab122"}}}}