{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/12604","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/12604/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/12604/comments","events_url":"https://api.github.com/repos/apache/spark/issues/12604/events","html_url":"https://github.com/apache/spark/pull/12604","id":150288109,"number":12604,"title":"[SPARK-14706][ML][PySpark] Python ML persistence integration test","user":{"login":"yinxusen","id":2637239,"avatar_url":"https://avatars.githubusercontent.com/u/2637239?v=3","gravatar_id":"","url":"https://api.github.com/users/yinxusen","html_url":"https://github.com/yinxusen","followers_url":"https://api.github.com/users/yinxusen/followers","following_url":"https://api.github.com/users/yinxusen/following{/other_user}","gists_url":"https://api.github.com/users/yinxusen/gists{/gist_id}","starred_url":"https://api.github.com/users/yinxusen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yinxusen/subscriptions","organizations_url":"https://api.github.com/users/yinxusen/orgs","repos_url":"https://api.github.com/users/yinxusen/repos","events_url":"https://api.github.com/users/yinxusen/events{/privacy}","received_events_url":"https://api.github.com/users/yinxusen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-04-22T07:04:05Z","updated_at":"2016-04-26T01:25:58Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/12604","html_url":"https://github.com/apache/spark/pull/12604","diff_url":"https://github.com/apache/spark/pull/12604.diff","patch_url":"https://github.com/apache/spark/pull/12604.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis patch tests Python ML persistence integration. \r\n\r\n- Add persistency tests of CrossValidator(TrainValidationSplit(LogisticRegression)) and TrainValidationSplit(CrossValidator(LogisticRegression)).\r\n\r\n- Enhance `_compare_pipelines` with checking of CrossValidator, CrossValidatorModel, TrainValidationSplit, TrainValidationSplitModel, OneVsRest, OneVsRestModel.\r\n\r\n**Bugs found and fixed in this PR:**\r\n\r\n- OneVsRest, CrossValidator and TrainValidationSplit should have `_transfer_param_map_to_java` and `_transfer_param_map_from_java`, otherwise they can't be used as estimators in tuning.\r\n\r\n- \r\n```scala\r\nlr = LogisticRegression()\r\nlr.getThresholds()\r\n```\r\nproduces `keyNotFoundError` because thresholds neither be set nor in `_defaultParamMap`, which leads the previous JavaParams parameter equality check error.\r\n\r\n- `trainRatio` in `TrainValidationSplit` should have float type converter.\r\n\r\n- `OneVsRest` with `classifier` in `estimatorParamMaps` of tuning fail to persistence. I.e.\r\n```scala\r\novr = OneVsRest()\r\nepms = [{ovr.classifier: xxxx}, {ovr.classifier: xxx}]\r\ncv = CrossValidator(estimator=ovr, estimatorParamMaps=epms, ...)\r\ncv.load()\r\n```\r\nfails because classifier cannot be serialized via JSON.\r\n\r\nThe last one is not trivial, so I left it unsolved in this PR. \r\n\r\n## How was this patch tested?\r\n\r\nThe patch tests with Python unit tests.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/214580036","html_url":"https://github.com/apache/spark/pull/12604#issuecomment-214580036","issue_url":"https://api.github.com/repos/apache/spark/issues/12604","id":214580036,"user":{"login":"jkbradley","id":5084283,"avatar_url":"https://avatars.githubusercontent.com/u/5084283?v=3","gravatar_id":"","url":"https://api.github.com/users/jkbradley","html_url":"https://github.com/jkbradley","followers_url":"https://api.github.com/users/jkbradley/followers","following_url":"https://api.github.com/users/jkbradley/following{/other_user}","gists_url":"https://api.github.com/users/jkbradley/gists{/gist_id}","starred_url":"https://api.github.com/users/jkbradley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkbradley/subscriptions","organizations_url":"https://api.github.com/users/jkbradley/orgs","repos_url":"https://api.github.com/users/jkbradley/repos","events_url":"https://api.github.com/users/jkbradley/events{/privacy}","received_events_url":"https://api.github.com/users/jkbradley/received_events","type":"User","site_admin":false},"created_at":"2016-04-26T01:25:58Z","updated_at":"2016-04-26T01:25:58Z","body":"I added some comments.  I think the tuning.py changes could be simplified somewhat, but I'll need to return to this later."}}