{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15971","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15971/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15971/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15971/events","html_url":"https://github.com/apache/spark/pull/15971","id":190875371,"number":15971,"title":"[SPARK-18535][UI][YARN] Redact sensitive information from Spark logs and UI","user":{"login":"markgrover","id":1709451,"avatar_url":"https://avatars.githubusercontent.com/u/1709451?v=3","gravatar_id":"","url":"https://api.github.com/users/markgrover","html_url":"https://github.com/markgrover","followers_url":"https://api.github.com/users/markgrover/followers","following_url":"https://api.github.com/users/markgrover/following{/other_user}","gists_url":"https://api.github.com/users/markgrover/gists{/gist_id}","starred_url":"https://api.github.com/users/markgrover/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markgrover/subscriptions","organizations_url":"https://api.github.com/users/markgrover/orgs","repos_url":"https://api.github.com/users/markgrover/repos","events_url":"https://api.github.com/users/markgrover/events{/privacy}","received_events_url":"https://api.github.com/users/markgrover/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2016-11-22T00:28:11Z","updated_at":"2016-11-24T01:53:32Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15971","html_url":"https://github.com/apache/spark/pull/15971","diff_url":"https://github.com/apache/spark/pull/15971.diff","patch_url":"https://github.com/apache/spark/pull/15971.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis patch adds a new property called `spark.secret.redactionPattern` that\r\nallows users to specify a scala regex to decide which Spark configuration\r\nproperties and environment variables in driver and executor environments\r\ncontain sensitive information. When this regex matches the property or\r\nenvironment variable name, its value is redacted from the environment UI and\r\nvarious logs like YARN and event logs.\r\n\r\nThis change uses this property to redact information from event logs and YARN\r\nlogs. It also, updates the UI code to adhere to this property instead of\r\nhardcoding the logic to decipher which properties are sensitive.\r\n\r\nHere's an image of the UI post-redaction:\r\n![image](https://cloud.githubusercontent.com/assets/1709451/20506215/4cc30654-b007-11e6-8aee-4cde253fba2f.png)\r\n\r\nHere's the text in the YARN logs, post-redaction:\r\n``HADOOP_CREDSTORE_PASSWORD -> *********(redacted)``\r\n\r\nHere's the text in the event logs, post-redaction:\r\n``...,\"spark.executorEnv.HADOOP_CREDSTORE_PASSWORD\":\"*********(redacted)\",\"spark.yarn.appMasterEnv.HADOOP_CREDSTORE_PASSWORD\":\"*********(redacted)\",...``\r\n\r\n## How was this patch tested?\r\n1. Unit tests are added to ensure that redaction works.\r\n2. A YARN job reading data off of S3 with confidential information\r\n(hadoop credential provider password) being provided in the environment\r\nvariables of driver and executor. And, afterwards, logs were grepped to make\r\nsure that no mention of secret password was present. It was also ensure that\r\nthe job was able to read the data off of S3 correctly, thereby ensuring that\r\nthe sensitive information was being trickled down to the right places to read\r\nthe data.\r\n3. The event logs were checked to make sure no mention of secret password was\r\npresent.\r\n4. UI environment tab was checked to make sure there was no secret information\r\nbeing displayed."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/262672330","html_url":"https://github.com/apache/spark/pull/15971#issuecomment-262672330","issue_url":"https://api.github.com/repos/apache/spark/issues/15971","id":262672330,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-11-24T01:53:32Z","updated_at":"2016-11-24T01:53:32Z","body":"Merged build finished. Test PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}