{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/175171592","pull_request_review_id":104664050,"id":175171592,"diff_hunk":"@@ -3,47 +3,291 @@ layout: global\n displayTitle: Spark Security\n title: Security\n ---\n+* This will become a table of contents (this text will be scraped).\n+{:toc}\n \n-Spark currently supports authentication via a shared secret. Authentication can be configured to be on via the `spark.authenticate` configuration parameter. This parameter controls whether the Spark communication protocols do authentication using the shared secret. This authentication is a basic handshake to make sure both sides have the same shared secret and are allowed to communicate. If the shared secret is not identical they will not be allowed to communicate. The shared secret is created as follows:\n+# Spark RPC\n \n-* For Spark on [YARN](running-on-yarn.html) and local deployments, configuring `spark.authenticate` to `true` will automatically handle generating and distributing the shared secret. Each application will use a unique shared secret.\n-* For other types of Spark deployments, the Spark parameter `spark.authenticate.secret` should be configured on each of the nodes. This secret will be used by all the Master/Workers and applications.\n+## Authentication\n \n-## Web UI\n+Spark currently supports authentication for RPC channels using a shared secret. Authentication can\n+be turned on by setting the `spark.authenticate` configuration parameter.\n \n-The Spark UI can be secured by using [javax servlet filters](http://docs.oracle.com/javaee/6/api/javax/servlet/Filter.html) via the `spark.ui.filters` setting\n-and by using [https/SSL](http://en.wikipedia.org/wiki/HTTPS) via [SSL settings](security.html#ssl-configuration).\n+The exact mechanism used to generate and distribute the shared secret is deployment-specific.\n \n-### Authentication\n+For Spark on [YARN](running-on-yarn.html) and local deployments, Spark will automatically handle\n+generating and distributing the shared secret. Each application will use a unique shared secret. In\n+the case of YARN, this feature relies on YARN RPC encryption being enabled for the distribution of\n+secrets to be secure.\n \n-A user may want to secure the UI if it has data that other users should not be allowed to see. The javax servlet filter specified by the user can authenticate the user and then once the user is logged in, Spark can compare that user versus the view ACLs to make sure they are authorized to view the UI. The configs `spark.acls.enable`, `spark.ui.view.acls` and `spark.ui.view.acls.groups` control the behavior of the ACLs. Note that the user who started the application always has view access to the UI.  On YARN, the Spark UI uses the standard YARN web application proxy mechanism and will authenticate via any installed Hadoop filters.\n+For other resource managers, `spark.authenticate.secret` must be configured on each of the nodes.\n+This secret will be shared by all the daemons and applications, so this deployment configuration is\n+not as secure as the above, especially when considering multi-tenant clusters.\n \n-Spark also supports modify ACLs to control who has access to modify a running Spark application. This includes things like killing the application or a task. This is controlled by the configs `spark.acls.enable`, `spark.modify.acls` and `spark.modify.acls.groups`. Note that if you are authenticating the web UI, in order to use the kill button on the web UI it might be necessary to add the users in the modify acls to the view acls also. On YARN, the modify acls are passed in and control who has modify access via YARN interfaces.\n-Spark allows for a set of administrators to be specified in the acls who always have view and modify permissions to all the applications. is controlled by the configs `spark.admin.acls` and `spark.admin.acls.groups`. This is useful on a shared cluster where you might have administrators or support staff who help users debug applications.\n+<table class=\"table\">\n+<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>\n+<tr>\n+  <td><code>spark.authenticate</code></td>\n+  <td>false</td>\n+  <td>Whether Spark authenticates its internal connections.</td>\n+</tr>\n+<tr>\n+  <td><code>spark.authenticate.secret</code></td>\n+  <td>None</td>\n+  <td>\n+    The secret key used authentication. See above for when this configuration should be set.\n+  </td>\n+</tr>\n+</table>\n \n-## Event Logging\n+## Encryption\n \n-If your applications are using event logging, the directory where the event logs go (`spark.eventLog.dir`) should be manually created and have the proper permissions set on it. If you want those log files secured, the permissions should be set to `drwxrwxrwxt` for that directory. The owner of the directory should be the super user who is running the history server and the group permissions should be restricted to super user group. This will allow all users to write to the directory but will prevent unprivileged users from removing or renaming a file unless they own the file or directory. The event log files will be created by Spark with permissions such that only the user and group have read and write access.\n+Spark supports AES-based encryption for RPC connections. For encryption to be enabled, RPC\n+authentication must also be enabled and properly configured. AES encryption uses the\n+[Apache Commons Crypto](http://commons.apache.org/proper/commons-crypto/) library, and Spark's\n+configuration system allows access to that library's configuration for advanced users.\n+\n+There is also support for SASL-based encryption, although it should be considered deprectated. It\n+is still required when talking to shuffle services from Spark versions older than 2.2.0.\n+\n+The following table describes the different options available for configuring this feature.\n+\n+<table class=\"table\">\n+<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>\n+<tr>\n+  <td><code>spark.network.crypto.enabled</code></td>\n+  <td>false</td>\n+  <td>\n+    Enable AES-based RPC encryption, including the new authentication protocol added in 2.2.0.\n+  </td>\n+</tr>\n+<tr>\n+  <td><code>spark.network.crypto.keyLength</code></td>\n+  <td>128</td>\n+  <td>\n+    The length in bits of the encryption key to generate. Valid values are 128, 192 and 256.\n+  </td>\n+</tr>\n+<tr>\n+  <td><code>spark.network.crypto.keyFactoryAlgorithm</code></td>\n+  <td>PBKDF2WithHmacSHA1</td>\n+  <td>\n+    The key factory algorithm to use when generating encryption keys. Should be one of the\n+    algorithms supported by the javax.crypto.SecretKeyFactory class in the JRE being used.\n+  </td>\n+</tr>\n+<tr>\n+  <td><code>spark.network.crypto.config.*</code></td>\n+  <td>None</td>\n+  <td>\n+    Configuration values for the commons-crypto library, such as which cipher implementations to\n+    use. The config name should be the name of commons-crypto configuration without the\n+    <code>commons.crypto</code> prefix.\n+  </td>\n+</tr>\n+<tr>\n+  <td><code>spark.network.crypto.saslFallback</code></td>\n+  <td>true</td>\n+  <td>\n+    Whether to fall back to SASL authentication if authentication fails using Spark's internal\n+    mechanism. This is useful when the application is connecting to old shuffle services that\n+    do not support the internal Spark authentication protocol. On the shuffle service side, enabling","path":"docs/security.md","position":104,"original_position":104,"commit_id":"832d87130ede866e4d877ca407e7a621282d4612","original_commit_id":"832d87130ede866e4d877ca407e7a621282d4612","user":{"login":"squito","id":71240,"avatar_url":"https://avatars2.githubusercontent.com/u/71240?v=4","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"body":"is this part right -- enabling the option (not disabling) makes the shuffle service refuse to fallback to sasl?  from a quick scan I expected the opposite:\r\n\r\nhttps://github.com/apache/spark/blob/master/common/network-common/src/main/java/org/apache/spark/network/crypto/AuthRpcHandler.java#L97","created_at":"2018-03-16T18:02:35Z","updated_at":"2018-03-16T18:11:19Z","html_url":"https://github.com/apache/spark/pull/20742#discussion_r175171592","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/20742","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/175171592"},"html":{"href":"https://github.com/apache/spark/pull/20742#discussion_r175171592"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/20742"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/20742","id":173014068,"html_url":"https://github.com/apache/spark/pull/20742","diff_url":"https://github.com/apache/spark/pull/20742.diff","patch_url":"https://github.com/apache/spark/pull/20742.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/20742","number":20742,"state":"open","locked":false,"title":"[SPARK-23572][docs] Bring \"security.md\" up to date.","user":{"login":"vanzin","id":1694083,"avatar_url":"https://avatars0.githubusercontent.com/u/1694083?v=4","gravatar_id":"","url":"https://api.github.com/users/vanzin","html_url":"https://github.com/vanzin","followers_url":"https://api.github.com/users/vanzin/followers","following_url":"https://api.github.com/users/vanzin/following{/other_user}","gists_url":"https://api.github.com/users/vanzin/gists{/gist_id}","starred_url":"https://api.github.com/users/vanzin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanzin/subscriptions","organizations_url":"https://api.github.com/users/vanzin/orgs","repos_url":"https://api.github.com/users/vanzin/repos","events_url":"https://api.github.com/users/vanzin/events{/privacy}","received_events_url":"https://api.github.com/users/vanzin/received_events","type":"User","site_admin":false},"body":"This change basically rewrites the security documentation so that it's\r\nup to date with new features, more correct, and more complete.\r\n\r\nBecause security is such an important feature, I chose to move all the\r\nrelevant configuration documentation to the security page, instead of\r\nhaving them peppered all over the place in the configuration page. This\r\nallows an almost one-stop shop for security configuration in Spark. The\r\nonly exceptions are some YARN-specific minor features which I left in\r\nthe YARN page.\r\n\r\nI also re-organized the page's topics, since they didn't make a lot of\r\nsense. You had kerberos features described inside paragraphs talking\r\nabout UI access control, and other oddities. It should be easier now\r\nto find information about specific Spark security features. I also\r\nenabled TOCs for both the Security and YARN pages, since that makes it\r\neasier to see what is covered.\r\n\r\nI removed most of the comments from the SecurityManager javadoc since\r\nthey just replicated information in the security doc, with different\r\nlevels of out-of-dateness.\r\n","created_at":"2018-03-05T23:11:39Z","updated_at":"2018-03-16T18:11:19Z","closed_at":null,"merged_at":null,"merge_commit_sha":"54efe656a01b8269caf3fcbc57e78ac138342123","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/20742/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/20742/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/20742/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/832d87130ede866e4d877ca407e7a621282d4612","head":{"label":"vanzin:SPARK-23572","ref":"SPARK-23572","sha":"832d87130ede866e4d877ca407e7a621282d4612","user":{"login":"vanzin","id":1694083,"avatar_url":"https://avatars0.githubusercontent.com/u/1694083?v=4","gravatar_id":"","url":"https://api.github.com/users/vanzin","html_url":"https://github.com/vanzin","followers_url":"https://api.github.com/users/vanzin/followers","following_url":"https://api.github.com/users/vanzin/following{/other_user}","gists_url":"https://api.github.com/users/vanzin/gists{/gist_id}","starred_url":"https://api.github.com/users/vanzin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanzin/subscriptions","organizations_url":"https://api.github.com/users/vanzin/orgs","repos_url":"https://api.github.com/users/vanzin/repos","events_url":"https://api.github.com/users/vanzin/events{/privacy}","received_events_url":"https://api.github.com/users/vanzin/received_events","type":"User","site_admin":false},"repo":{"id":18302272,"name":"spark","full_name":"vanzin/spark","owner":{"login":"vanzin","id":1694083,"avatar_url":"https://avatars0.githubusercontent.com/u/1694083?v=4","gravatar_id":"","url":"https://api.github.com/users/vanzin","html_url":"https://github.com/vanzin","followers_url":"https://api.github.com/users/vanzin/followers","following_url":"https://api.github.com/users/vanzin/following{/other_user}","gists_url":"https://api.github.com/users/vanzin/gists{/gist_id}","starred_url":"https://api.github.com/users/vanzin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanzin/subscriptions","organizations_url":"https://api.github.com/users/vanzin/orgs","repos_url":"https://api.github.com/users/vanzin/repos","events_url":"https://api.github.com/users/vanzin/events{/privacy}","received_events_url":"https://api.github.com/users/vanzin/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/vanzin/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/vanzin/spark","forks_url":"https://api.github.com/repos/vanzin/spark/forks","keys_url":"https://api.github.com/repos/vanzin/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/vanzin/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/vanzin/spark/teams","hooks_url":"https://api.github.com/repos/vanzin/spark/hooks","issue_events_url":"https://api.github.com/repos/vanzin/spark/issues/events{/number}","events_url":"https://api.github.com/repos/vanzin/spark/events","assignees_url":"https://api.github.com/repos/vanzin/spark/assignees{/user}","branches_url":"https://api.github.com/repos/vanzin/spark/branches{/branch}","tags_url":"https://api.github.com/repos/vanzin/spark/tags","blobs_url":"https://api.github.com/repos/vanzin/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/vanzin/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/vanzin/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/vanzin/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/vanzin/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/vanzin/spark/languages","stargazers_url":"https://api.github.com/repos/vanzin/spark/stargazers","contributors_url":"https://api.github.com/repos/vanzin/spark/contributors","subscribers_url":"https://api.github.com/repos/vanzin/spark/subscribers","subscription_url":"https://api.github.com/repos/vanzin/spark/subscription","commits_url":"https://api.github.com/repos/vanzin/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/vanzin/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/vanzin/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/vanzin/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/vanzin/spark/contents/{+path}","compare_url":"https://api.github.com/repos/vanzin/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/vanzin/spark/merges","archive_url":"https://api.github.com/repos/vanzin/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/vanzin/spark/downloads","issues_url":"https://api.github.com/repos/vanzin/spark/issues{/number}","pulls_url":"https://api.github.com/repos/vanzin/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/vanzin/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/vanzin/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/vanzin/spark/labels{/name}","releases_url":"https://api.github.com/repos/vanzin/spark/releases{/id}","deployments_url":"https://api.github.com/repos/vanzin/spark/deployments","created_at":"2014-03-31T17:49:25Z","updated_at":"2018-01-06T03:57:54Z","pushed_at":"2018-03-13T18:05:28Z","git_url":"git://github.com/vanzin/spark.git","ssh_url":"git@github.com:vanzin/spark.git","clone_url":"https://github.com/vanzin/spark.git","svn_url":"https://github.com/vanzin/spark","homepage":null,"size":245287,"stargazers_count":1,"watchers_count":1,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"508573958dc9b6402e684cd6dd37202deaaa97f6","user":{"login":"apache","id":47359,"avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2018-03-16T17:42:30Z","pushed_at":"2018-03-16T17:39:11Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":269531,"stargazers_count":16519,"watchers_count":16519,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":15221,"mirror_url":null,"archived":false,"open_issues_count":480,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":15221,"open_issues":480,"watchers":16519,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/20742"},"html":{"href":"https://github.com/apache/spark/pull/20742"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/20742"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/20742/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/20742/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/20742/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/832d87130ede866e4d877ca407e7a621282d4612"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"squito"}}