{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/22423","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/22423/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/22423/comments","events_url":"https://api.github.com/repos/apache/spark/issues/22423/events","html_url":"https://github.com/apache/spark/pull/22423","id":360484093,"node_id":"MDExOlB1bGxSZXF1ZXN0MjE1NzMzOTcx","number":22423,"title":"[SPARK-25302][STREAMING] Checkpoint the reducedStream in ReducedWindo…","user":{"login":"nikunjb","id":5704596,"node_id":"MDQ6VXNlcjU3MDQ1OTY=","avatar_url":"https://avatars1.githubusercontent.com/u/5704596?v=4","gravatar_id":"","url":"https://api.github.com/users/nikunjb","html_url":"https://github.com/nikunjb","followers_url":"https://api.github.com/users/nikunjb/followers","following_url":"https://api.github.com/users/nikunjb/following{/other_user}","gists_url":"https://api.github.com/users/nikunjb/gists{/gist_id}","starred_url":"https://api.github.com/users/nikunjb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nikunjb/subscriptions","organizations_url":"https://api.github.com/users/nikunjb/orgs","repos_url":"https://api.github.com/users/nikunjb/repos","events_url":"https://api.github.com/users/nikunjb/events{/privacy}","received_events_url":"https://api.github.com/users/nikunjb/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-09-14T23:37:11Z","updated_at":"2018-10-22T10:44:37Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/22423","html_url":"https://github.com/apache/spark/pull/22423","diff_url":"https://github.com/apache/spark/pull/22423.diff","patch_url":"https://github.com/apache/spark/pull/22423.patch"},"body":"…wDStream so as to cut the lineage completely to parent\r\n\r\n  ## What changes were proposed in this pull request?\r\n\r\n  Dstream.reduceByKeyAndWindow() with inverse reduce functions eventually creates\r\n  a ReducedWindowDStream but did not checkpoint it.\r\n  When combined with the issue described in SPARK-25303,\r\n  it results in the problem described in the post here:\r\n\r\n  http://apache-spark-user-list.1001560.n3.nabble.com/DStream-reduceByKeyAndWindow-not-using-checkpointed-data-for-inverse-reducing-old-data-td33332.html\r\n\r\n  This change will checkpoint reducedStream inside to cut the lineage.\r\n  A separate patch is being created for the issue in SPARK-25303\r\n\r\n  ## How was this patch tested?\r\n\r\n  Running the existing Unit tests. A couple of existing unit test classes were\r\n  written assuming they will not be used in serialization. Had to declare\r\n  SparkContext as transient like other tests in order to make them work\r\n  due to new checkpointing causing serialization to occur.\r\n\r\n  ## What improvement does this patch make?\r\n\r\n  Cuts the lineage to the parent DStream. When combined with the fix\r\n  in SPARK-25303, unpersits the intermediate RDDs and Input DStreams that\r\n  were being remembered for too long, resulting in much lower memory usage\r\n  on the Executors after the fixes. Observed from the DAG chart differences and\r\n  data from the Storage tab on the Driver UI.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/431800082","html_url":"https://github.com/apache/spark/pull/22423#issuecomment-431800082","issue_url":"https://api.github.com/repos/apache/spark/issues/22423","id":431800082,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMTgwMDA4Mg==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-10-22T10:44:35Z","updated_at":"2018-10-22T10:44:35Z","author_association":"NONE","body":"Can one of the admins verify this patch?"}}
{"actor":{"display_login":"AmplabJenkins"}}