{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16113","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16113/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16113/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16113/events","html_url":"https://github.com/apache/spark/pull/16113","id":193028713,"number":16113,"title":"[SPARK-18657][SPARK-18668] Make StreamingQuery.id persists across restart and not auto-generate StreamingQuery.name","user":{"login":"tdas","id":663212,"avatar_url":"https://avatars.githubusercontent.com/u/663212?v=3","gravatar_id":"","url":"https://api.github.com/users/tdas","html_url":"https://github.com/tdas","followers_url":"https://api.github.com/users/tdas/followers","following_url":"https://api.github.com/users/tdas/following{/other_user}","gists_url":"https://api.github.com/users/tdas/gists{/gist_id}","starred_url":"https://api.github.com/users/tdas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdas/subscriptions","organizations_url":"https://api.github.com/users/tdas/orgs","repos_url":"https://api.github.com/users/tdas/repos","events_url":"https://api.github.com/users/tdas/events{/privacy}","received_events_url":"https://api.github.com/users/tdas/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-12-02T03:57:11Z","updated_at":"2016-12-02T04:46:39Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16113","html_url":"https://github.com/apache/spark/pull/16113","diff_url":"https://github.com/apache/spark/pull/16113.diff","patch_url":"https://github.com/apache/spark/pull/16113.patch"},"body":"## What changes were proposed in this pull request?\r\nHere are the major changes in this PR. \r\n- Added the ability to recover `StreamingQuery.id` from checkpoint location, by writing the id to `checkpointLoc/metadata`.\r\n- Added `StreamingQuery.runId` which is unique for every query started and does not persist across restarts. This is to identify each restart of a query separately (same as earlier behavior of `id`).\r\n- Removed auto-generation of `StreamingQuery.name`. The purpose of name was to have the ability to define an identifier across restarts, but since id is precisely that, there is no need for a auto-generated name. This means name becomes purely cosmetic, and is null by default.\r\n- Added runId to `StreamingQueryListener` events and `StreamingQueryProgress`.\r\n\r\nImplementation details\r\n- Renamed existing `StreamExecutionMetadata` to `OffsetSeqMetadata`, and moved it to the file `OffsetSeq.scala`, because that is what this metadata is tied to. Also did some refactoring to make the code cleaner (got rid of a lot of `.json` and `.getOrElse(\"{}\")`).\r\n- Added the id as the new `StreamMetadata`. \r\n- When a StreamingQuery is created it gets or writes the `StreamMetadata` from `checkpointLoc/metadata`.\r\n- All internal logging in `StreamExecution` uses `(name, id, runId)` instead of just `name`\r\n\r\nTODO\r\n- [x] Test handling of name=null in json generation of StreamingQueryProgress\r\n- [ ] Test handling of name=null in json generation of StreamingQueryListener events\r\n- [ ] Test python API of runId\r\n\r\n## How was this patch tested?\r\nUpdate unit tests and new unit tests"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/264372503","html_url":"https://github.com/apache/spark/pull/16113#issuecomment-264372503","issue_url":"https://api.github.com/repos/apache/spark/issues/16113","id":264372503,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-12-02T04:46:39Z","updated_at":"2016-12-02T04:46:39Z","body":"Merged build finished. Test FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}