{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21502","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21502/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21502/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21502/events","html_url":"https://github.com/apache/spark/pull/21502","id":329896577,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkzMDM1NjYw","number":21502,"title":"[SPARK-22575][SQL] Add destroy to Dataset","user":{"login":"mgaido91","id":8821783,"node_id":"MDQ6VXNlcjg4MjE3ODM=","avatar_url":"https://avatars1.githubusercontent.com/u/8821783?v=4","gravatar_id":"","url":"https://api.github.com/users/mgaido91","html_url":"https://github.com/mgaido91","followers_url":"https://api.github.com/users/mgaido91/followers","following_url":"https://api.github.com/users/mgaido91/following{/other_user}","gists_url":"https://api.github.com/users/mgaido91/gists{/gist_id}","starred_url":"https://api.github.com/users/mgaido91/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgaido91/subscriptions","organizations_url":"https://api.github.com/users/mgaido91/orgs","repos_url":"https://api.github.com/users/mgaido91/repos","events_url":"https://api.github.com/users/mgaido91/events{/privacy}","received_events_url":"https://api.github.com/users/mgaido91/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-06-06T14:36:44Z","updated_at":"2018-06-07T16:18:07Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21502","html_url":"https://github.com/apache/spark/pull/21502","diff_url":"https://github.com/apache/spark/pull/21502.diff","patch_url":"https://github.com/apache/spark/pull/21502.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nIn the Dataset API we may acquire resources which we cannot deallocate. This happens for broadcast joins. The broadcasted object is never destroyed and we rely on the garbage collection of broadcasted object to free it. In a general use case, this is a safe assumption, but when dynamic allocation is enabled, the current approach can lead to resource leakage.\r\n\r\nIn particular, when a Spark application is submitted on YARN with dynamic allocation enabled, we may leak disk space. Indeed, in such a scenario, when query with a broadcast join is executed, it is likely that we ask for new containers. These containers are used for the execution of the query and then killed. They may be killed before the broadcast object is GCed. In this case, the files which have been written are never removed (as the container is not alive anymore to remove them and YARN removes them only when the application ends).\r\n\r\nIn order to solve the above-mentioned issue, the PR proposes to add a `destroy` method to the `Dataset` class, which can be used to free all the resources which have been acquired in the plan execution. Eagerly destroying the acquired resources, they are freed before the containers are killed, avoiding (or at least reducing considerably) the problem.\r\n\r\n## How was this patch tested?\r\n\r\nadded UT\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/395480070","html_url":"https://github.com/apache/spark/pull/21502#issuecomment-395480070","issue_url":"https://api.github.com/repos/apache/spark/issues/21502","id":395480070,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NTQ4MDA3MA==","user":{"login":"SparkQA","id":8108735,"node_id":"MDQ6VXNlcjgxMDg3MzU=","avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2018-06-07T16:18:07Z","updated_at":"2018-06-07T16:18:07Z","author_association":"NONE","body":"**[Test build #91528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91528/testReport)** for PR 21502 at commit [`4d080cf`](https://github.com/apache/spark/commit/4d080cff795457c6a02b255acc691157afc94e81)."}}
{"actor":{"display_login":"SparkQA"}}