{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/11534","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/11534/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/11534/comments","events_url":"https://api.github.com/repos/apache/spark/issues/11534/events","html_url":"https://github.com/apache/spark/pull/11534","id":138647278,"number":11534,"title":"[SPARK-13696][WIP] Remove BlockStore class & simplify interfaces of mem. & disk stores","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2016-03-05T05:11:51Z","updated_at":"2016-03-08T23:37:35Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11534","html_url":"https://github.com/apache/spark/pull/11534","diff_url":"https://github.com/apache/spark/pull/11534.diff","patch_url":"https://github.com/apache/spark/pull/11534.patch"},"body":"Today, both the MemoryStore and DiskStore implement a common `BlockStore` API, but I feel that this API is inappropriate because it abstracts away important distinctions between the behavior of these two stores.\r\n\r\nFor instance, the disk store doesn't have a notion of storing deserialized objects, so it's confusing for it to expose object-based APIs like putIterator() and getValues() instead of only exposing binary APIs and pushing the responsibilities of serialization and deserialization to the client. Similarly, the DiskStore put() methods accepted a `StorageLevel` parameter even though the disk store can only store blocks in one form.\r\n\r\nAs part of a larger BlockManager interface cleanup, this patch remove the BlockStore interface and refines the MemoryStore and DiskStore interfaces to reflect more narrow sets of responsibilities for those components. Some of the benefits of this interface cleanup are reflected in simplifications to several unit tests to eliminate now-unnecessary mocking, significant simplification of the BlockManager's `getLocal()` method, and a narrower API between the MemoryStore and DiskStore.\r\n\r\nNext week I'll update this PR description to have a more comprehensive list of improvements. In the meantime, though, the commit messages provide lots of detail into my rationale for the intermediate steps in this refactoring."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/194025752","html_url":"https://github.com/apache/spark/pull/11534#issuecomment-194025752","issue_url":"https://api.github.com/repos/apache/spark/issues/11534","id":194025752,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-03-08T23:37:35Z","updated_at":"2016-03-08T23:37:35Z","body":"**[Test build #52702 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52702/consoleFull)** for PR 11534 at commit [`92c5125`](https://github.com/apache/spark/commit/92c5125f2f4736335971e779fc39e9fa74f8c310)."}}