{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/19410","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/19410/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/19410/comments","events_url":"https://api.github.com/repos/apache/spark/issues/19410/events","html_url":"https://github.com/apache/spark/pull/19410","id":262118056,"node_id":"MDExOlB1bGxSZXF1ZXN0MTQ0MjA3ODM0","number":19410,"title":"[SPARK-22184][CORE][GRAPHX] GraphX fails in case of insufficient memory and checkpoints enabled","user":{"login":"szhem","id":1523889,"node_id":"MDQ6VXNlcjE1MjM4ODk=","avatar_url":"https://avatars1.githubusercontent.com/u/1523889?v=4","gravatar_id":"","url":"https://api.github.com/users/szhem","html_url":"https://github.com/szhem","followers_url":"https://api.github.com/users/szhem/followers","following_url":"https://api.github.com/users/szhem/following{/other_user}","gists_url":"https://api.github.com/users/szhem/gists{/gist_id}","starred_url":"https://api.github.com/users/szhem/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/szhem/subscriptions","organizations_url":"https://api.github.com/users/szhem/orgs","repos_url":"https://api.github.com/users/szhem/repos","events_url":"https://api.github.com/users/szhem/events{/privacy}","received_events_url":"https://api.github.com/users/szhem/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2017-10-02T15:08:27Z","updated_at":"2018-10-22T12:55:51Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/19410","html_url":"https://github.com/apache/spark/pull/19410","diff_url":"https://github.com/apache/spark/pull/19410.diff","patch_url":"https://github.com/apache/spark/pull/19410.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nFix for [SPARK-22184](https://issues.apache.org/jira/browse/SPARK-22184) JIRA issue (and also includes the related #19373).\r\n\r\nIn case of GraphX jobs, when checkpoints are enabled, GraphX can fail with `FileNotFoundException`.\r\n\r\nThe failure can happen during Pregel iterations or when Pregel completes only in cases of insufficient memory when checkpointed RDDs are evicted from memory and have to be read from disk (but already removed from there).\r\n\r\nThis PR proposes to preserve all the checkpoints the last one (checkpoint) of `messages` and `graph` depends on during the iterations, and all the checkpoints of `messages` and `graph` the resulting `graph` depends at the end of Pregel iterations.\r\n\r\n## How was this patch tested?\r\n\r\nUnit tests as well as manually in production jobs which previously were failing until this PR applied."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/431827339","html_url":"https://github.com/apache/spark/pull/19410#issuecomment-431827339","issue_url":"https://api.github.com/repos/apache/spark/issues/19410","id":431827339,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMTgyNzMzOQ==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-10-22T12:55:51Z","updated_at":"2018-10-22T12:55:51Z","author_association":"NONE","body":"Can one of the admins verify this patch?"}}
{"actor":{"display_login":"AmplabJenkins"}}