{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13584","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13584/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13584/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13584/events","html_url":"https://github.com/apache/spark/pull/13584","id":159539116,"number":13584,"title":"[SPARK-15509][ML][SparkR] R MLlib algorithms should support input columns \"features\" and \"label\"","user":{"login":"keypointt","id":3925641,"avatar_url":"https://avatars.githubusercontent.com/u/3925641?v=3","gravatar_id":"","url":"https://api.github.com/users/keypointt","html_url":"https://github.com/keypointt","followers_url":"https://api.github.com/users/keypointt/followers","following_url":"https://api.github.com/users/keypointt/following{/other_user}","gists_url":"https://api.github.com/users/keypointt/gists{/gist_id}","starred_url":"https://api.github.com/users/keypointt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keypointt/subscriptions","organizations_url":"https://api.github.com/users/keypointt/orgs","repos_url":"https://api.github.com/users/keypointt/repos","events_url":"https://api.github.com/users/keypointt/events{/privacy}","received_events_url":"https://api.github.com/users/keypointt/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2016-06-10T00:50:09Z","updated_at":"2016-09-01T06:22:39Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13584","html_url":"https://github.com/apache/spark/pull/13584","diff_url":"https://github.com/apache/spark/pull/13584.diff","patch_url":"https://github.com/apache/spark/pull/13584.patch"},"body":"https://issues.apache.org/jira/browse/SPARK-15509\r\n\r\n## What changes were proposed in this pull request?\r\n\r\nCurrently in SparkR, when you load a LibSVM dataset using the sqlContext and then pass it to an MLlib algorithm, the ML wrappers will fail since they will try to create a \"features\" column, which conflicts with the existing \"features\" column from the LibSVM loader. E.g., using the \"mnist\" dataset from LibSVM:\r\n`training <- loadDF(sqlContext, \".../mnist\", \"libsvm\")`\r\n`model <- naiveBayes(label ~ features, training)`\r\nThis fails with:\r\n```\r\n16/05/24 11:52:41 ERROR RBackendHandler: fit on org.apache.spark.ml.r.NaiveBayesWrapper failed\r\nError in invokeJava(isStatic = TRUE, className, methodName, ...) : \r\n  java.lang.IllegalArgumentException: Output column features already exists.\r\n\tat org.apache.spark.ml.feature.VectorAssembler.transformSchema(VectorAssembler.scala:120)\r\n\tat org.apache.spark.ml.Pipeline$$anonfun$transformSchema$4.apply(Pipeline.scala:179)\r\n\tat org.apache.spark.ml.Pipeline$$anonfun$transformSchema$4.apply(Pipeline.scala:179)\r\n\tat scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57)\r\n\tat scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66)\r\n\tat scala.collection.mutable.ArrayOps$ofRef.foldLeft(ArrayOps.scala:186)\r\n\tat org.apache.spark.ml.Pipeline.transformSchema(Pipeline.scala:179)\r\n\tat org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:67)\r\n\tat org.apache.spark.ml.Pipeline.fit(Pipeline.scala:131)\r\n\tat org.apache.spark.ml.feature.RFormula.fit(RFormula.scala:169)\r\n\tat org.apache.spark.ml.r.NaiveBayesWrapper$.fit(NaiveBayesWrapper.scala:62)\r\n\tat org.apache.spark.ml.r.NaiveBayesWrapper.fit(NaiveBayesWrapper.sca\r\nThe same issue appears for the \"label\" column once you rename the \"features\" column.\r\n```\r\nThe cause is, when using `loadDF()` to generate dataframes, sometimes it’s with default column name `“label”` and `“features”`, and these two name will conflict with default column names `setDefault(labelCol, \"label\")` and ` setDefault(featuresCol, \"features\")` of `SharedParams.scala`\r\n\r\n\r\n\r\n\r\n## How was this patch tested?\r\n\r\nTest on my local machine.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/243986942","html_url":"https://github.com/apache/spark/pull/13584#issuecomment-243986942","issue_url":"https://api.github.com/repos/apache/spark/issues/13584","id":243986942,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-09-01T06:22:38Z","updated_at":"2016-09-01T06:22:38Z","body":"**[Test build #64764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64764/consoleFull)** for PR 13584 at commit [`caa4183`](https://github.com/apache/spark/commit/caa41833c5d6dcfac046e5a804b1498258121d94)."}}
{"actor":{"display_login":"SparkQA"}}