{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/23424","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/23424/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/23424/comments","events_url":"https://api.github.com/repos/apache/spark/issues/23424/events","html_url":"https://github.com/apache/spark/pull/23424","id":395146896,"node_id":"MDExOlB1bGxSZXF1ZXN0MjQxNjcwNjI4","number":23424,"title":"[SPARK-24421][CORE][FOLLOWUP] Use normal direct ByteBuffer allocation if Cleaner can't be set","user":{"login":"srowen","id":822522,"node_id":"MDQ6VXNlcjgyMjUyMg==","avatar_url":"https://avatars0.githubusercontent.com/u/822522?v=4","gravatar_id":"","url":"https://api.github.com/users/srowen","html_url":"https://github.com/srowen","followers_url":"https://api.github.com/users/srowen/followers","following_url":"https://api.github.com/users/srowen/following{/other_user}","gists_url":"https://api.github.com/users/srowen/gists{/gist_id}","starred_url":"https://api.github.com/users/srowen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/srowen/subscriptions","organizations_url":"https://api.github.com/users/srowen/orgs","repos_url":"https://api.github.com/users/srowen/repos","events_url":"https://api.github.com/users/srowen/events{/privacy}","received_events_url":"https://api.github.com/users/srowen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2019-01-02T04:27:19Z","updated_at":"2019-01-02T08:05:01Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/23424","html_url":"https://github.com/apache/spark/pull/23424","diff_url":"https://github.com/apache/spark/pull/23424.diff","patch_url":"https://github.com/apache/spark/pull/23424.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nIn Java 9+ we can't use sun.misc.Cleaner by default anymore, and this was largely handled in https://github.com/apache/spark/pull/22993 However I think the change there left a significant problem.\r\n\r\nIf a DirectByteBuffer is allocated using the reflective hack in Platform, now, we by default can't set a Cleaner. But I believe this means the memory isn't freed promptly or possibly at all. If a Cleaner can't be set, I think we need to use normal APIs to allocate the direct ByteBuffer.\r\n\r\nAccording to comments in the code, the downside is simply that the normal APIs will check and impose limits on how much off-heap memory can be allocated. Per the original review on https://github.com/apache/spark/pull/22993 this much seems fine, as either way in this case the user would have to add a JVM setting (increase max, or allow the reflective access).\r\n\r\n## How was this patch tested?\r\n\r\nExisting tests. This resolved an OutOfMemoryError in Java 11 from TimSort tests without increasing test heap size. (See https://github.com/apache/spark/pull/23419#issuecomment-450772125 ) This suggests there is a problem and that this resolves it."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/450804608","html_url":"https://github.com/apache/spark/pull/23424#issuecomment-450804608","issue_url":"https://api.github.com/repos/apache/spark/issues/23424","id":450804608,"node_id":"MDEyOklzc3VlQ29tbWVudDQ1MDgwNDYwOA==","user":{"login":"SparkQA","id":8108735,"node_id":"MDQ6VXNlcjgxMDg3MzU=","avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2019-01-02T08:05:01Z","updated_at":"2019-01-02T08:05:01Z","author_association":"NONE","body":"**[Test build #100633 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/100633/testReport)** for PR 23424 at commit [`2f267a5`](https://github.com/apache/spark/commit/2f267a5d63e10d3d1e986a346a4385a93a27ce7c).\n * This patch **fails due to an unknown error code, -9**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}