{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16312","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16312/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16312/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16312/events","html_url":"https://github.com/apache/spark/pull/16312","id":196098620,"number":16312,"title":"[SPARK-18862][SPARKR][ML] Split SparkR mllib.R into multiple files","user":{"login":"yanboliang","id":1962026,"avatar_url":"https://avatars.githubusercontent.com/u/1962026?v=3","gravatar_id":"","url":"https://api.github.com/users/yanboliang","html_url":"https://github.com/yanboliang","followers_url":"https://api.github.com/users/yanboliang/followers","following_url":"https://api.github.com/users/yanboliang/following{/other_user}","gists_url":"https://api.github.com/users/yanboliang/gists{/gist_id}","starred_url":"https://api.github.com/users/yanboliang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yanboliang/subscriptions","organizations_url":"https://api.github.com/users/yanboliang/orgs","repos_url":"https://api.github.com/users/yanboliang/repos","events_url":"https://api.github.com/users/yanboliang/events{/privacy}","received_events_url":"https://api.github.com/users/yanboliang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2016-12-16T16:30:13Z","updated_at":"2016-12-23T05:48:25Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16312","html_url":"https://github.com/apache/spark/pull/16312","diff_url":"https://github.com/apache/spark/pull/16312.diff","patch_url":"https://github.com/apache/spark/pull/16312.patch"},"body":"## What changes were proposed in this pull request?\r\nSparkR ```mllib.R``` is getting bigger as we add more ML wrappers, I'd like to split it into multiple files to make us easy to maintain:\r\n* mllib_classification.R\r\n* mllib_clustering.R\r\n* mllib_recommendation.R\r\n* mllib_regression.R\r\n* mllib_stat.R\r\n* mllib_tree.R\r\n* mllib_utils.R\r\n\r\nNote: Only reorg, no actual code change.\r\n\r\n## How was this patch tested?\r\nExisting tests.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/268943837","html_url":"https://github.com/apache/spark/pull/16312#issuecomment-268943837","issue_url":"https://api.github.com/repos/apache/spark/issues/16312","id":268943837,"user":{"login":"shivaram","id":143893,"avatar_url":"https://avatars.githubusercontent.com/u/143893?v=3","gravatar_id":"","url":"https://api.github.com/users/shivaram","html_url":"https://github.com/shivaram","followers_url":"https://api.github.com/users/shivaram/followers","following_url":"https://api.github.com/users/shivaram/following{/other_user}","gists_url":"https://api.github.com/users/shivaram/gists{/gist_id}","starred_url":"https://api.github.com/users/shivaram/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shivaram/subscriptions","organizations_url":"https://api.github.com/users/shivaram/orgs","repos_url":"https://api.github.com/users/shivaram/repos","events_url":"https://api.github.com/users/shivaram/events{/privacy}","received_events_url":"https://api.github.com/users/shivaram/received_events","type":"User","site_admin":false},"created_at":"2016-12-23T05:48:25Z","updated_at":"2016-12-23T05:48:25Z","body":"I looked at this more closely and I think I found the problem - Not sure its easy to fix though.\r\nWhat I traced here is:\r\n- When we call sparkR.session.stop and sparkR.session the same JVM backend is reused and only the SparkContext is stopped / recreated\r\n- Now the problem happens when we call `read.ml` to read a model after creating a new SparkSession. This in turn calls into RWrappers[1] which has an `sc` member variable\r\n- My understanding is that the `sc` member variable is bound the first time we create a SparkSession and when we stop, restart it has a handle to the stale SparkContext\r\n- Thus we see errors where it says `Cannot call methods on a stopped SparkContext`\r\n\r\nI think the right fix here is to pass along a SparkContext into RWrappers and not rely on a prior initialization. However I'm not sure why that design decision was made before, so maybe I'm missing something.\r\n\r\n[1] https://github.com/apache/spark/blob/f252cb5d161e064d39cc1ed1d9299307a0636174/mllib/src/main/scala/org/apache/spark/ml/r/RWrappers.scala#L36\r\n"}}
{"actor":{"display_login":"shivaram"}}