{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/10534","labels_url":"https://api.github.com/repos/apache/spark/issues/10534/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/10534/comments","events_url":"https://api.github.com/repos/apache/spark/issues/10534/events","html_url":"https://github.com/apache/spark/pull/10534","id":124419039,"number":10534,"title":"[SPARK-7689] Remove TTL-based metadata cleaning in Spark 2.0","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2015-12-31T00:54:56Z","updated_at":"2016-01-06T23:29:11Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/10534","html_url":"https://github.com/apache/spark/pull/10534","diff_url":"https://github.com/apache/spark/pull/10534.diff","patch_url":"https://github.com/apache/spark/pull/10534.patch"},"body":"This PR removes `spark.cleaner.ttl` and the associated TTL-based metadata cleaning code.\r\n\r\nNow that we have the `ContextCleaner` and a timer to trigger periodic GCs, I don't think that `spark.cleaner.ttl` is necessary anymore. The TTL-based cleaning isn't enabled by default, isn't included in our end-to-end tests, and has been a source of user confusion when it is misconfigured. If the TTL is set too low, data which is still being used may be evicted / deleted, leading to hard to diagnose bugs.\r\n\r\nFor all of these reasons, I think that we should remove this functionality in Spark 2.0. Additional benefits of doing this include marginally reduced memory usage, since we no longer need to store timetsamps in hashmaps, and a handful fewer threads."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/169496060","html_url":"https://github.com/apache/spark/pull/10534#issuecomment-169496060","issue_url":"https://api.github.com/repos/apache/spark/issues/10534","id":169496060,"user":{"login":"tdas","id":663212,"avatar_url":"https://avatars.githubusercontent.com/u/663212?v=3","gravatar_id":"","url":"https://api.github.com/users/tdas","html_url":"https://github.com/tdas","followers_url":"https://api.github.com/users/tdas/followers","following_url":"https://api.github.com/users/tdas/following{/other_user}","gists_url":"https://api.github.com/users/tdas/gists{/gist_id}","starred_url":"https://api.github.com/users/tdas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdas/subscriptions","organizations_url":"https://api.github.com/users/tdas/orgs","repos_url":"https://api.github.com/users/tdas/repos","events_url":"https://api.github.com/users/tdas/events{/privacy}","received_events_url":"https://api.github.com/users/tdas/received_events","type":"User","site_admin":false},"created_at":"2016-01-06T23:29:11Z","updated_at":"2016-01-06T23:29:11Z","body":"LGTM, pending tests."}}