{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14659","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14659/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14659/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14659/events","html_url":"https://github.com/apache/spark/pull/14659","id":171323912,"number":14659,"title":"[SPARK-16757] Set up Spark caller context to HDFS","user":{"login":"Sherry302","id":8546874,"avatar_url":"https://avatars.githubusercontent.com/u/8546874?v=3","gravatar_id":"","url":"https://api.github.com/users/Sherry302","html_url":"https://github.com/Sherry302","followers_url":"https://api.github.com/users/Sherry302/followers","following_url":"https://api.github.com/users/Sherry302/following{/other_user}","gists_url":"https://api.github.com/users/Sherry302/gists{/gist_id}","starred_url":"https://api.github.com/users/Sherry302/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Sherry302/subscriptions","organizations_url":"https://api.github.com/users/Sherry302/orgs","repos_url":"https://api.github.com/users/Sherry302/repos","events_url":"https://api.github.com/users/Sherry302/events{/privacy}","received_events_url":"https://api.github.com/users/Sherry302/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":19,"created_at":"2016-08-16T04:44:59Z","updated_at":"2016-08-25T00:49:39Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14659","html_url":"https://github.com/apache/spark/pull/14659","diff_url":"https://github.com/apache/spark/pull/14659.diff","patch_url":"https://github.com/apache/spark/pull/14659.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\n1. Pass `jobId` to Task.\r\n2. Invoke Hadoop APIs. \r\n    * A new function `setCallerContext` is added in `Utils`. `setCallerContext` function invokes APIs of   `org.apache.hadoop.ipc.CallerContext` to set up spark caller contexts, which will be written into `hdfs-audit.log`.\r\n    * For applications in Yarn client mode, `org.apache.hadoop.ipc.CallerContext` are called in `Task` and Yarn `Client`. For applications in Yarn cluster mode, `org.apache.hadoop.ipc.CallerContext` are be called in `Task` and `ApplicationMaster`.\r\n    * The Spark caller contexts written into `hdfs-audit.log` are applications' name ` {spark.app.name}` and `JobID_stageID_stageAttemptId_taskID_attemptNumbe`.\r\n\r\n## How was this patch tested?\r\nManual Tests against some Spark applications in Yarn client mode and Yarn cluster mode. Need to check if spark caller contexts are written into HDFS hdfs-audit.log successfully.\r\n\r\nFor example, run SparkKmeans in Yarn client mode: \r\n```\r\n./bin/spark-submit  --master yarn --deploy-mode client --class org.apache.spark.examples.SparkKMeans examples/target/original-spark-examples_2.11-2.1.0-SNAPSHOT.jar hdfs://localhost:9000/lr_big.txt 2 5\r\n```\r\n\r\n**Before**:\r\nThere will be no Spark caller context in records of `hdfs-audit.log`.\r\n\r\n**After**:\r\nSpark caller contexts will be written in records of `hdfs-audit.log`.\r\n\r\n(_Spark caller context below is in the log as Hadoop caller context API was invoked in Yarn Client_)\r\n```\r\n2016-07-21 13:52:30,802 INFO FSNamesystem.audit: allowed=true        ugi=wyang (auth:SIMPLE)        ip=/127.0.0.1        cmd=getfileinfo        src=/lr_big.txt        dst=null        perm=null        proto=rpc        callerContext=SparkKMeans running on Spark \r\n```\r\n(_Spark caller context below is in the log as Hadoop caller context API was invoked in Task_)\r\n```\r\n2016-07-21 13:52:35,584 INFO FSNamesystem.audit: allowed=true        ugi=wyang (auth:SIMPLE)        ip=/127.0.0.1        cmd=open        src=/lr_big.txt        dst=null        perm=null        proto=rpc        callerContext=JobId_0_StageID_0_stageAttemptId_0_taskID_0_attemptNumber_0 on Spark\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/242251784","html_url":"https://github.com/apache/spark/pull/14659#issuecomment-242251784","issue_url":"https://api.github.com/repos/apache/spark/issues/14659","id":242251784,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-08-25T00:49:39Z","updated_at":"2016-08-25T00:49:39Z","body":"**[Test build #64380 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64380/consoleFull)** for PR 14659 at commit [`1512775`](https://github.com/apache/spark/commit/1512775a3faddb9de9299662a6f3bfec3f6fe205).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}