{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/10705","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/10705/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/10705/comments","events_url":"https://api.github.com/repos/apache/spark/issues/10705/events","html_url":"https://github.com/apache/spark/pull/10705","id":126031609,"number":10705,"title":"[SPARK-12757][WIP] Use reference counting to prevent blocks from being evicted during reads","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":32,"created_at":"2016-01-11T20:39:08Z","updated_at":"2016-02-15T02:32:28Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/10705","html_url":"https://github.com/apache/spark/pull/10705","diff_url":"https://github.com/apache/spark/pull/10705.diff","patch_url":"https://github.com/apache/spark/pull/10705.patch"},"body":"**Note: description is out of date and will be updated later**.\r\n\r\n**This is blocked by #11193**\r\n\r\nAs a pre-requisite to off-heap caching of blocks, we need a mechanism to prevent pages / blocks from being evicted while they are being read. With on-heap objects, evicting a block while it is being read merely leads to memory-accounting problems (because we assume that an evicted block is a candidate for garbage-collection, which will not be true during a read), but with off-heap memory this will lead to either data corruption or segmentation faults.\r\n\r\nTo address this, we should add a pin-counting mechanism to track which blocks/pages are being read in order to prevent them from being evicted prematurely. I propose to do this in two phases: first, add a safe, conservative approach in which all `BlockManager.get*()` calls implicitly increment the pin count of blocks and where tasks' pins are automatically freed upon task completion (this PR). This will be correct but may have adverse performance impacts because it will prevent legitimate block evictions. In phase two, we should incrementally add `unpin()` calls in order to fix the eviction of unreferenced blocks. The latter change may need to touch many different components, which is why I propose to do it separately in order to make the changes easier to reason about and review.\r\n\r\nThis PR is currently WIP, pending test fixes and a few additional improvements:\r\n\r\n- [ ] Add significantly more debug logging statements. From my experience in working on other memory-management-related things in Spark, I've realized that it's extremely useful to have a set of verbose logging statements that we can enable with a Log4J conf.\r\n- [x] Add call to free all pins at end of task.\r\n- [ ] Guard the \"non-zero pin count prevents eviction\" check behind a debugging feature-flag to let us disable this feature for testing. This will be a useful debugging aid in phase 2.\r\n- [ ] Get the existing tests to pass.\r\n  - [x] Fix BlockManagerReplicationSuite failures.\r\n- [ ] Add new unit tests for the eviction behavior:\r\n  - [x] Add test to ensure that pinned blocks cannot be evicted.\r\n- [ ] Write API documentation for the `unpin()` methods."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/184033065","html_url":"https://github.com/apache/spark/pull/10705#issuecomment-184033065","issue_url":"https://api.github.com/repos/apache/spark/issues/10705","id":184033065,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-02-15T02:32:27Z","updated_at":"2016-02-15T02:32:27Z","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51289/\nTest FAILed."}}