{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/19373","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/19373/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/19373/comments","events_url":"https://api.github.com/repos/apache/spark/issues/19373/events","html_url":"https://github.com/apache/spark/pull/19373","id":261130310,"node_id":"MDExOlB1bGxSZXF1ZXN0MTQzNTA5ODIy","number":19373,"title":"[SPARK-22150][CORE] PeriodicCheckpointer fails in case of dependent RDDs","user":{"login":"szhem","id":1523889,"node_id":"MDQ6VXNlcjE1MjM4ODk=","avatar_url":"https://avatars1.githubusercontent.com/u/1523889?v=4","gravatar_id":"","url":"https://api.github.com/users/szhem","html_url":"https://github.com/szhem","followers_url":"https://api.github.com/users/szhem/followers","following_url":"https://api.github.com/users/szhem/following{/other_user}","gists_url":"https://api.github.com/users/szhem/gists{/gist_id}","starred_url":"https://api.github.com/users/szhem/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/szhem/subscriptions","organizations_url":"https://api.github.com/users/szhem/orgs","repos_url":"https://api.github.com/users/szhem/repos","events_url":"https://api.github.com/users/szhem/events{/privacy}","received_events_url":"https://api.github.com/users/szhem/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":18,"created_at":"2017-09-27T21:54:57Z","updated_at":"2018-10-22T10:45:40Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/19373","html_url":"https://github.com/apache/spark/pull/19373","diff_url":"https://github.com/apache/spark/pull/19373.diff","patch_url":"https://github.com/apache/spark/pull/19373.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nFix for [SPARK-22150](https://issues.apache.org/jira/browse/SPARK-22150) JIRA issue.\r\n\r\nIn case of checkpointing RDDs which depend on previously checkpointed RDDs (for example in iterative algorithms) PeriodicCheckpointer removes already checkpointed materialized RDDs too early leading to FileNotFoundExceptions.\r\n\r\nConsider the following snippet\r\n\r\n    // create a periodic checkpointer with interval of 2\r\n    val checkpointer = new PeriodicRDDCheckpointer[Double](2, sc)\r\n    \r\n    val rdd1 = createRDD(sc)\r\n    checkpointer.update(rdd1)\r\n    // on the second update rdd1 is checkpointed\r\n    checkpointer.update(rdd1)\r\n    // on action checkpointed rdd is materialized and its lineage is truncated\r\n    rdd1.count() \r\n    \r\n    // rdd2 depends on rdd1\r\n    val rdd2 = rdd1.filter(_ => true)\r\n    checkpointer.update(rdd2)\r\n    // on the second update rdd2 is checkpointed and checkpoint files of rdd1 are deleted\r\n    checkpointer.update(rdd2)\r\n    // on action it's necessary to read already removed checkpoint files of rdd1\r\n    rdd2.count()\r\n\r\nThis PR proposes to preserve all the checkpoints the last one depends on to be able to evaluate the final RDD even if the last checkpoint (the final RDD depends on) is not yet materialized.\r\n\r\n## How was this patch tested?\r\n\r\nUnit tests as well as manually in production jobs which previously were failing until this PR applied."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/431800275","html_url":"https://github.com/apache/spark/pull/19373#issuecomment-431800275","issue_url":"https://api.github.com/repos/apache/spark/issues/19373","id":431800275,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMTgwMDI3NQ==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-10-22T10:45:40Z","updated_at":"2018-10-22T10:45:40Z","author_association":"NONE","body":"Can one of the admins verify this patch?"}}
{"actor":{"display_login":"AmplabJenkins"}}