{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/11645","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/11645/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/11645/comments","events_url":"https://api.github.com/repos/apache/spark/issues/11645/events","html_url":"https://github.com/apache/spark/pull/11645","id":140082592,"number":11645,"title":"[SPARK-13809][SQL] State store for streaming aggregations","user":{"login":"tdas","id":663212,"avatar_url":"https://avatars.githubusercontent.com/u/663212?v=3","gravatar_id":"","url":"https://api.github.com/users/tdas","html_url":"https://github.com/tdas","followers_url":"https://api.github.com/users/tdas/followers","following_url":"https://api.github.com/users/tdas/following{/other_user}","gists_url":"https://api.github.com/users/tdas/gists{/gist_id}","starred_url":"https://api.github.com/users/tdas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdas/subscriptions","organizations_url":"https://api.github.com/users/tdas/orgs","repos_url":"https://api.github.com/users/tdas/repos","events_url":"https://api.github.com/users/tdas/events{/privacy}","received_events_url":"https://api.github.com/users/tdas/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":27,"created_at":"2016-03-11T04:28:57Z","updated_at":"2016-03-21T22:12:38Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11645","html_url":"https://github.com/apache/spark/pull/11645","diff_url":"https://github.com/apache/spark/pull/11645.diff","patch_url":"https://github.com/apache/spark/pull/11645.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nIn this PR, I am implementing a new abstraction for management of streaming state data - State Store. It is a key-value store for persisting running aggregates for aggregate operations in streaming dataframes. The motivation and design is discussed here. \r\n\r\nhttps://docs.google.com/document/d/1-ncawFx8JS5Zyfq1HAEGBx56RDet9wfVp_hDM8ZL254/edit#\r\n\r\n## How was this patch tested?\r\n- [x] Unit tests\r\n- [x] Cluster tests\r\n\r\n**Coverage from unit tests**\r\n\r\n<img width=\"952\" alt=\"screen shot 2016-03-21 at 3 09 40 pm\" src=\"https://cloud.githubusercontent.com/assets/663212/13935872/fdc8ba86-ef76-11e5-93e8-9fa310472c7b.png\">\r\n\r\n\r\n## TODO\r\n- [x] Fix updates() iterator to avoid duplicate updates for same key\r\n- [x] Use Coordinator in ContinuousQueryManager\r\n- [x] Plugging in hadoop conf and other confs\r\n- [x] Unit tests\r\n  - [x] StateStore object lifecycle and methods\r\n  - [x] StateStoreCoordinator communication and logic\r\n  - [x] StateStoreRDD fault-tolerance\r\n  - [x] StateStoreRDD preferred location using StateStoreCoordinator\r\n- [ ] Cluster tests\r\n  - [ ] Whether preferred locations are set correctly\r\n  - [ ] Whether recovery works correctly with distributed storage\r\n  - [x] Basic performance tests\r\n- [x] Docs\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/199511917","html_url":"https://github.com/apache/spark/pull/11645#issuecomment-199511917","issue_url":"https://api.github.com/repos/apache/spark/issues/11645","id":199511917,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-03-21T22:12:38Z","updated_at":"2016-03-21T22:12:38Z","body":"**[Test build #53718 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53718/consoleFull)** for PR 11645 at commit [`3824053`](https://github.com/apache/spark/commit/3824053324aae89956601db060bfb77b013a2303)."}}