{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/11942","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/11942/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/11942/comments","events_url":"https://api.github.com/repos/apache/spark/issues/11942/events","html_url":"https://github.com/apache/spark/pull/11942","id":143340527,"number":11942,"title":"[SPARK-14135] Add off-heap storage memory bookkeeping support to MemoryManager","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2016-03-24T19:42:21Z","updated_at":"2016-03-25T23:22:36Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11942","html_url":"https://github.com/apache/spark/pull/11942","diff_url":"https://github.com/apache/spark/pull/11942.diff","patch_url":"https://github.com/apache/spark/pull/11942.patch"},"body":"This patch extends Spark's `UnifiedMemoryManager` to add bookkeeping support for off-heap storage memory, an requirement for enabling off-heap caching (which will be done by #11805). The `MemoryManager`'s `storageMemoryPool` has been split into separate on- and off-heap pools and the storage and unroll memory allocation methods have been updated to accept a `memoryMode` parameter to specify whether allocations should be performed on- or off-heap.\r\n\r\nIn order to reduce the testing surface, the `StaticMemoryManager` does not support off-heap caching (we plan to eventually remove the `StaticMemoryManager`, so this isn't a significant limitation)."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/201579941","html_url":"https://github.com/apache/spark/pull/11942#issuecomment-201579941","issue_url":"https://api.github.com/repos/apache/spark/issues/11942","id":201579941,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-03-25T23:22:35Z","updated_at":"2016-03-25T23:22:35Z","body":"**[Test build #54204 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54204/consoleFull)** for PR 11942 at commit [`c3d0f61`](https://github.com/apache/spark/commit/c3d0f61245764eb194070952561ec1fb6e7ede9a).\n * This patch **fails from timeout after a configured wait of \\`250m\\`**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}