{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/71111198","id":71111198,"diff_hunk":"@@ -193,8 +200,14 @@ private[yarn] class AMDelegationTokenRenewer(\n       }\n     }\n     val nextSuffix = lastCredentialsFileSuffix + 1\n+    // Time of next update should be late than renewal,\n+    // which is around new token issue date + 0.8 * renew interval\n+    val timeOfNextUpdate =","path":"yarn/src/main/scala/org/apache/spark/deploy/yarn/security/AMDelegationTokenRenewer.scala","position":81,"original_position":81,"commit_id":"a8141a54dd87f77c3ce6eb0bccfabacd64841b4b","original_commit_id":"a8141a54dd87f77c3ce6eb0bccfabacd64841b4b","user":{"login":"jerryshao","id":850797,"avatar_url":"https://avatars.githubusercontent.com/u/850797?v=3","gravatar_id":"","url":"https://api.github.com/users/jerryshao","html_url":"https://github.com/jerryshao","followers_url":"https://api.github.com/users/jerryshao/followers","following_url":"https://api.github.com/users/jerryshao/following{/other_user}","gists_url":"https://api.github.com/users/jerryshao/gists{/gist_id}","starred_url":"https://api.github.com/users/jerryshao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerryshao/subscriptions","organizations_url":"https://api.github.com/users/jerryshao/orgs","repos_url":"https://api.github.com/users/jerryshao/repos","events_url":"https://api.github.com/users/jerryshao/events{/privacy}","received_events_url":"https://api.github.com/users/jerryshao/received_events","type":"User","site_admin":false},"body":"Not sure how this part could be consolidate into `obtainCredentials` method, the return value of `obtainCredentials` is time of next renewal, here it is time of next updating, which should be slightly later than next renewal. \r\n\r\nSo in shortly IIUC if we also want to consolidate this code, so we should change the return value `obtainCredentials` to `(timeOfNextRenewal, timeOfNextUpdate)`, what do you think?","created_at":"2016-07-18T08:08:29Z","updated_at":"2016-07-18T08:08:29Z","html_url":"https://github.com/apache/spark/pull/14065#discussion_r71111198","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/14065","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/71111198"},"html":{"href":"https://github.com/apache/spark/pull/14065#discussion_r71111198"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/14065"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14065","id":76395633,"html_url":"https://github.com/apache/spark/pull/14065","diff_url":"https://github.com/apache/spark/pull/14065.diff","patch_url":"https://github.com/apache/spark/pull/14065.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/14065","number":14065,"state":"open","locked":false,"title":"[SPARK-14743][YARN] Add a configurable token manager for Spark running on YARN","user":{"login":"jerryshao","id":850797,"avatar_url":"https://avatars.githubusercontent.com/u/850797?v=3","gravatar_id":"","url":"https://api.github.com/users/jerryshao","html_url":"https://github.com/jerryshao","followers_url":"https://api.github.com/users/jerryshao/followers","following_url":"https://api.github.com/users/jerryshao/following{/other_user}","gists_url":"https://api.github.com/users/jerryshao/gists{/gist_id}","starred_url":"https://api.github.com/users/jerryshao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerryshao/subscriptions","organizations_url":"https://api.github.com/users/jerryshao/orgs","repos_url":"https://api.github.com/users/jerryshao/repos","events_url":"https://api.github.com/users/jerryshao/events{/privacy}","received_events_url":"https://api.github.com/users/jerryshao/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nAdd a configurable token manager for Spark on running on yarn.\r\n\r\n### Current Problems ###\r\n\r\n1. Supported token provider is hard-coded, currently only hdfs, hbase and hive are supported and it is impossible for user to add new token provider without code changes.\r\n2. Also this problem exits in timely token renewer and updater.\r\n\r\n### Changes In This Proposal ###\r\n\r\nIn this proposal, to address the problems mentioned above and make the current code more cleaner and easier to understand, mainly has 3 changes:\r\n\r\n1. Abstract a `ServiceTokenProvider` as well as `ServiceTokenRenewable` interface for token provider. Each service wants to communicate with Spark through token way needs to implement this interface. \r\n2. Provide a `ConfigurableTokenManager` to manage all the register token providers, also token renewer and updater. Also this class offers the API for other modules to obtain tokens, get renewal interval and so on.\r\n3. Implement 3 built-in token providers `HDFSTokenProvider`, `HiveTokenProvider` and `HBaseTokenProvider` to keep the same semantics as supported today. Whether to load in these built-in token providers is controlled by configuration \"spark.yarn.security.tokens.${service}.enabled\", by default for all the built-in token providers are loaded.\r\n\r\n### Behavior Changes ###\r\n\r\nFor the end user there's no behavior change, we still use the same configuration `spark.yarn.security.tokens.${service}.enabled` to decide which token provider is enabled (hbase or hive).\r\n\r\nFor user implemented token provider (assume the name of token provider is \"test\") needs to add into this class should have two configurations:\r\n\r\n1. `spark.yarn.security.tokens.test.enabled` to true\r\n2. `spark.yarn.security.tokens.test.class` to the full qualified class name.\r\n\r\nSo we still keep the same semantics as current code while add one new configuration.\r\n\r\n### Current Status ###\r\n\r\n- [x] token provider interface and management framework.\r\n- [x] implement built-in token providers (hdfs, hbase, hive).\r\n- [x] Coverage of unit test.\r\n- [x] Integrated test with security cluster.\r\n\r\n## How was this patch tested?\r\n\r\nUnit test and integrated test.\r\n\r\nPlease suggest and review, any comment is greatly appreciated.\r\n","created_at":"2016-07-06T08:31:52Z","updated_at":"2016-07-18T08:08:29Z","closed_at":null,"merged_at":null,"merge_commit_sha":"8817f72721f70bf915d7dda4992b3df9e4644940","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/14065/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/14065/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/14065/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/a8141a54dd87f77c3ce6eb0bccfabacd64841b4b","head":{"label":"jerryshao:SPARK-16342","ref":"SPARK-16342","sha":"a8141a54dd87f77c3ce6eb0bccfabacd64841b4b","user":{"login":"jerryshao","id":850797,"avatar_url":"https://avatars.githubusercontent.com/u/850797?v=3","gravatar_id":"","url":"https://api.github.com/users/jerryshao","html_url":"https://github.com/jerryshao","followers_url":"https://api.github.com/users/jerryshao/followers","following_url":"https://api.github.com/users/jerryshao/following{/other_user}","gists_url":"https://api.github.com/users/jerryshao/gists{/gist_id}","starred_url":"https://api.github.com/users/jerryshao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerryshao/subscriptions","organizations_url":"https://api.github.com/users/jerryshao/orgs","repos_url":"https://api.github.com/users/jerryshao/repos","events_url":"https://api.github.com/users/jerryshao/events{/privacy}","received_events_url":"https://api.github.com/users/jerryshao/received_events","type":"User","site_admin":false},"repo":{"id":18259371,"name":"apache-spark","full_name":"jerryshao/apache-spark","owner":{"login":"jerryshao","id":850797,"avatar_url":"https://avatars.githubusercontent.com/u/850797?v=3","gravatar_id":"","url":"https://api.github.com/users/jerryshao","html_url":"https://github.com/jerryshao","followers_url":"https://api.github.com/users/jerryshao/followers","following_url":"https://api.github.com/users/jerryshao/following{/other_user}","gists_url":"https://api.github.com/users/jerryshao/gists{/gist_id}","starred_url":"https://api.github.com/users/jerryshao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerryshao/subscriptions","organizations_url":"https://api.github.com/users/jerryshao/orgs","repos_url":"https://api.github.com/users/jerryshao/repos","events_url":"https://api.github.com/users/jerryshao/events{/privacy}","received_events_url":"https://api.github.com/users/jerryshao/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jerryshao/apache-spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/jerryshao/apache-spark","forks_url":"https://api.github.com/repos/jerryshao/apache-spark/forks","keys_url":"https://api.github.com/repos/jerryshao/apache-spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jerryshao/apache-spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jerryshao/apache-spark/teams","hooks_url":"https://api.github.com/repos/jerryshao/apache-spark/hooks","issue_events_url":"https://api.github.com/repos/jerryshao/apache-spark/issues/events{/number}","events_url":"https://api.github.com/repos/jerryshao/apache-spark/events","assignees_url":"https://api.github.com/repos/jerryshao/apache-spark/assignees{/user}","branches_url":"https://api.github.com/repos/jerryshao/apache-spark/branches{/branch}","tags_url":"https://api.github.com/repos/jerryshao/apache-spark/tags","blobs_url":"https://api.github.com/repos/jerryshao/apache-spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jerryshao/apache-spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jerryshao/apache-spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/jerryshao/apache-spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jerryshao/apache-spark/statuses/{sha}","languages_url":"https://api.github.com/repos/jerryshao/apache-spark/languages","stargazers_url":"https://api.github.com/repos/jerryshao/apache-spark/stargazers","contributors_url":"https://api.github.com/repos/jerryshao/apache-spark/contributors","subscribers_url":"https://api.github.com/repos/jerryshao/apache-spark/subscribers","subscription_url":"https://api.github.com/repos/jerryshao/apache-spark/subscription","commits_url":"https://api.github.com/repos/jerryshao/apache-spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/jerryshao/apache-spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/jerryshao/apache-spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/jerryshao/apache-spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/jerryshao/apache-spark/contents/{+path}","compare_url":"https://api.github.com/repos/jerryshao/apache-spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jerryshao/apache-spark/merges","archive_url":"https://api.github.com/repos/jerryshao/apache-spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jerryshao/apache-spark/downloads","issues_url":"https://api.github.com/repos/jerryshao/apache-spark/issues{/number}","pulls_url":"https://api.github.com/repos/jerryshao/apache-spark/pulls{/number}","milestones_url":"https://api.github.com/repos/jerryshao/apache-spark/milestones{/number}","notifications_url":"https://api.github.com/repos/jerryshao/apache-spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jerryshao/apache-spark/labels{/name}","releases_url":"https://api.github.com/repos/jerryshao/apache-spark/releases{/id}","deployments_url":"https://api.github.com/repos/jerryshao/apache-spark/deployments","created_at":"2014-03-30T07:04:45Z","updated_at":"2014-07-09T01:42:23Z","pushed_at":"2016-07-14T08:08:46Z","git_url":"git://github.com/jerryshao/apache-spark.git","ssh_url":"git@github.com:jerryshao/apache-spark.git","clone_url":"https://github.com/jerryshao/apache-spark.git","svn_url":"https://github.com/jerryshao/apache-spark","homepage":null,"size":172738,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":1,"forks":1,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"c9a6762150cfd62691a6361e05d2839b110fe8d0","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2016-07-18T07:34:27Z","pushed_at":"2016-07-18T08:00:23Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":501799,"stargazers_count":9323,"watchers_count":9323,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":8643,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":443,"forks":8643,"open_issues":443,"watchers":9323,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/14065"},"html":{"href":"https://github.com/apache/spark/pull/14065"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/14065"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/14065/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/14065/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/14065/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/a8141a54dd87f77c3ce6eb0bccfabacd64841b4b"}}}}
{"actor":{"display_login":"jerryshao"}}