{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15954","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15954/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15954/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15954/events","html_url":"https://github.com/apache/spark/pull/15954","id":190621759,"number":15954,"title":"[WIP][SPARK-18516][SQL] Split state and progress in streaming","user":{"login":"marmbrus","id":527,"avatar_url":"https://avatars.githubusercontent.com/u/527?v=3","gravatar_id":"","url":"https://api.github.com/users/marmbrus","html_url":"https://github.com/marmbrus","followers_url":"https://api.github.com/users/marmbrus/followers","following_url":"https://api.github.com/users/marmbrus/following{/other_user}","gists_url":"https://api.github.com/users/marmbrus/gists{/gist_id}","starred_url":"https://api.github.com/users/marmbrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmbrus/subscriptions","organizations_url":"https://api.github.com/users/marmbrus/orgs","repos_url":"https://api.github.com/users/marmbrus/repos","events_url":"https://api.github.com/users/marmbrus/events{/privacy}","received_events_url":"https://api.github.com/users/marmbrus/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-11-21T05:20:42Z","updated_at":"2016-11-21T07:29:41Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15954","html_url":"https://github.com/apache/spark/pull/15954","diff_url":"https://github.com/apache/spark/pull/15954.diff","patch_url":"https://github.com/apache/spark/pull/15954.patch"},"body":"This PR separates the status of a `StreamingQuery` into two separate APIs:\r\n - `status` - describes the status of a `StreamingQuery` at this moment, including what phase of processing is currently happening and if data is available.\r\n - `recentProgress` - an array of statistics about the most recent microbatches that have executed.\r\n\r\nA recent progress contains the following information:\r\n```\r\n{\r\n  \"id\" : \"2be8670a-fce1-4859-a530-748f29553bb6\",\r\n  \"name\" : \"query-29\",\r\n  \"timestamp\" : 1479705392724,\r\n  \"inputRecordsPerSecond\" : 230.76923076923077,\r\n  \"processedRecordsPerSecond\" : 10.869565217391303,\r\n  \"durationMs\" : {\r\n    \"triggerExecution\" : 276,\r\n    \"queryPlanning\" : 3,\r\n    \"getBatch\" : 5,\r\n    \"getOffset\" : 3,\r\n    \"addBatch\" : 234,\r\n    \"walCommit\" : 30\r\n  },\r\n  \"currentWatermark\" : 0,\r\n  \"stateOperators\" : [ ],\r\n  \"sources\" : [ {\r\n    \"description\" : \"KafkaSource[Subscribe[topic-14]]\",\r\n    \"startOffset\" : {\r\n      \"topic-14\" : {\r\n        \"2\" : 0,\r\n        \"4\" : 1,\r\n        \"1\" : 0,\r\n        \"3\" : 0,\r\n        \"0\" : 0\r\n      }\r\n    },\r\n    \"endOffset\" : {\r\n      \"topic-14\" : {\r\n        \"2\" : 1,\r\n        \"4\" : 2,\r\n        \"1\" : 0,\r\n        \"3\" : 0,\r\n        \"0\" : 1\r\n      }\r\n    },\r\n    \"numRecords\" : 3,\r\n    \"inputRecordsPerSecond\" : 230.76923076923077,\r\n    \"processedRecordsPerSecond\" : 10.869565217391303\r\n  } ]\r\n}\r\n```\r\n\r\nAdditionally, in order to make it possible to correlate progress updates across restarts, we change the `id` field from an integer that is unique with in the JVM to a `UUID` that is globally unique.\r\n\r\nThis is a work in progress until we finish the following:\r\n - [ ] Port the remaining tests to the new API\r\n - [ ] Store the query UUID in the checkpoint directory"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/261863271","html_url":"https://github.com/apache/spark/pull/15954#issuecomment-261863271","issue_url":"https://api.github.com/repos/apache/spark/issues/15954","id":261863271,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-11-21T07:29:41Z","updated_at":"2016-11-21T07:29:41Z","body":"**[Test build #68923 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68923/consoleFull)** for PR 15954 at commit [`f4357d1`](https://github.com/apache/spark/commit/f4357d1376ae933564dc344d7ac142acec044283).\n * This patch **fails to build**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}