{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/22598","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/22598/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/22598/comments","events_url":"https://api.github.com/repos/apache/spark/issues/22598/events","html_url":"https://github.com/apache/spark/pull/22598","id":365395498,"node_id":"MDExOlB1bGxSZXF1ZXN0MjE5MzI0NjU3","number":22598,"title":"[SPARK-25501][SS] Add kafka delegation token support.","user":{"login":"gaborgsomogyi","id":18561820,"node_id":"MDQ6VXNlcjE4NTYxODIw","avatar_url":"https://avatars2.githubusercontent.com/u/18561820?v=4","gravatar_id":"","url":"https://api.github.com/users/gaborgsomogyi","html_url":"https://github.com/gaborgsomogyi","followers_url":"https://api.github.com/users/gaborgsomogyi/followers","following_url":"https://api.github.com/users/gaborgsomogyi/following{/other_user}","gists_url":"https://api.github.com/users/gaborgsomogyi/gists{/gist_id}","starred_url":"https://api.github.com/users/gaborgsomogyi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gaborgsomogyi/subscriptions","organizations_url":"https://api.github.com/users/gaborgsomogyi/orgs","repos_url":"https://api.github.com/users/gaborgsomogyi/repos","events_url":"https://api.github.com/users/gaborgsomogyi/events{/privacy}","received_events_url":"https://api.github.com/users/gaborgsomogyi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":20,"created_at":"2018-10-01T10:17:55Z","updated_at":"2018-11-09T00:38:29Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/22598","html_url":"https://github.com/apache/spark/pull/22598","diff_url":"https://github.com/apache/spark/pull/22598.diff","patch_url":"https://github.com/apache/spark/pull/22598.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nIt adds kafka delegation token support for structured streaming. Please see the relevant [SPIP](https://docs.google.com/document/d/1ouRayzaJf_N5VQtGhVq9FURXVmRpXzEEWYHob0ne3NY/edit?usp=sharing)\r\n\r\nWhat this PR contains:\r\n* Configuration parameters for the feature\r\n* Delegation token fetching from broker\r\n* Usage of token through dynamic JAAS configuration\r\n* Minor refactoring in the existing code\r\n\r\nWhat this PR doesn't contain:\r\n* Documentation changes because design can change\r\n\r\n## How was this patch tested?\r\n\r\nExisting tests + added small amount of additional unit tests.\r\n\r\nBecause it's an external service integration mainly tested on cluster.\r\n* 4 node cluster\r\n* Kafka broker version 1.1.0\r\n* Topic with 4 partitions\r\n* security.protocol = SASL_SSL\r\n* sasl.mechanism = SCRAM-SHA-256\r\n\r\nAn example of obtaining a token:\r\n```\r\n18/10/01 01:07:49 INFO kafka010.TokenUtil: TOKENID         HMAC                           OWNER           RENEWERS                  ISSUEDATE       EXPIRYDATE      MAXDATE\r\n18/10/01 01:07:49 INFO kafka010.TokenUtil: D1-v__Q5T_uHx55rW16Jwg [hidden] User:user    []                        2018-10-01T01:07 2018-10-02T01:07 2018-10-08T01:07\r\n18/10/01 01:07:49 INFO security.KafkaDelegationTokenProvider: Get token from Kafka: Kind: KAFKA_DELEGATION_TOKEN, Service: kafka.server.delegation.token, Ident: 44 31 2d 76 5f 5f 51 35 54 5f 75 48 78 35 35 72 57 31 36 4a 77 67\r\n```\r\n\r\nAn example token usage:\r\n```\r\n18/10/01 01:08:07 INFO kafka010.KafkaSecurityHelper: Scram JAAS params: org.apache.kafka.common.security.scram.ScramLoginModule required tokenauth=true serviceName=\"kafka\" username=\"D1-v__Q5T_uHx55rW16Jwg\" password=\"[hidden]\";\r\n18/10/01 01:08:07 INFO kafka010.KafkaSourceProvider: Delegation token detected, using it for login.\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/437208258","html_url":"https://github.com/apache/spark/pull/22598#issuecomment-437208258","issue_url":"https://api.github.com/repos/apache/spark/issues/22598","id":437208258,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNzIwODI1OA==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-11-09T00:38:29Z","updated_at":"2018-11-09T00:38:29Z","author_association":"NONE","body":"Merged build finished. Test PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}