{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/11443","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/11443/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/11443/comments","events_url":"https://api.github.com/repos/apache/spark/issues/11443/events","html_url":"https://github.com/apache/spark/pull/11443","id":137544436,"number":11443,"title":"[SPARK-13244][SQL] Migrates DataFrame to Dataset","user":{"login":"liancheng","id":230655,"avatar_url":"https://avatars.githubusercontent.com/u/230655?v=3","gravatar_id":"","url":"https://api.github.com/users/liancheng","html_url":"https://github.com/liancheng","followers_url":"https://api.github.com/users/liancheng/followers","following_url":"https://api.github.com/users/liancheng/following{/other_user}","gists_url":"https://api.github.com/users/liancheng/gists{/gist_id}","starred_url":"https://api.github.com/users/liancheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liancheng/subscriptions","organizations_url":"https://api.github.com/users/liancheng/orgs","repos_url":"https://api.github.com/users/liancheng/repos","events_url":"https://api.github.com/users/liancheng/events{/privacy}","received_events_url":"https://api.github.com/users/liancheng/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":33,"created_at":"2016-03-01T11:16:08Z","updated_at":"2016-03-09T13:26:18Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11443","html_url":"https://github.com/apache/spark/pull/11443","diff_url":"https://github.com/apache/spark/pull/11443.diff","patch_url":"https://github.com/apache/spark/pull/11443.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR unifies DataFrame and Dataset by migrating existing DataFrame operations to Dataset and make `DataFrame` a type alias of `Dataset[Row]`.\r\n\r\nMost Scala code changes are source compatible, but Java API is broken as Java knows nothing about Scala type alias (mostly replacing `DataFrame` with `Dataset<Row>`).\r\n\r\nThere are several noticeable API changes related to those returning arrays:\r\n\r\n1.  `collect`/`take`\r\n\r\n    -   Old APIs in class `DataFrame`:\r\n\r\n        ```scala\r\n        def collect(): Array[Row]\r\n        def take(n: Int): Array[Row]\r\n        ```\r\n\r\n    -   New APIs in class `Dataset[T]`:\r\n\r\n        ```scala\r\n        def collect(): Array[T]\r\n        def take(n: Int): Array[T]\r\n\r\n        def collectRows(): Array[Row]\r\n        def takeRows(n: Int): Array[Row]\r\n        ```\r\n\r\n    Two specialized methods `collectRows` and `takeRows` are added because Java doesn't support returning generic arrays. Thus, for example, `DataFrame.collect(): Array[T]` actually returns `Object` instead of `Array<T>` from Java side.\r\n\r\n    Normally, Java users may fall back to `collectAsList` and `takeAsList`.  The two new specialized versions are added to avoid performance regression in ML related code (but maybe I'm wrong and they are not necessary here).\r\n\r\n1.  `randomSplit`\r\n\r\n    -   Old APIs in class `DataFrame`:\r\n\r\n        ```scala\r\n        def randomSplit(weights: Array[Double], seed: Long): Array[DataFrame]\r\n        def randomSplit(weights: Array[Double]): Array[DataFrame]\r\n        ```\r\n\r\n    -   New APIs in class `Dataset[T]`:\r\n\r\n        ```scala\r\n        def randomSplit(weights: Array[Double], seed: Long): Array[Dataset[T]]\r\n        def randomSplit(weights: Array[Double]): Array[Dataset[T]]\r\n        ```\r\n\r\n    Similar problem as above, but hasn't been addressed for Java API yet.  We can probably add `randomSplitAsList` to fix this one.\r\n\r\n1.  `groupBy`\r\n\r\n    Some original `DataFrame.groupBy` methods have conflicting signature with original `Dataset.groupBy` methods.  To distinguish these two, typed `Dataset.groupBy` methods are renamed to `groupByKey`.\r\n\r\nOther noticeable changes:\r\n\r\n1.  Dataset always do eager analysis now\r\n\r\n    We used to support disabling DataFrame eager analysis to help reporting partially analyzed malformed logical plan on analysis failure.  However, Dataset encoders requires eager analysi during Dataset construction.  To preserve the error reporting feature, `AnalysisException` now takes an extra `Option[LogicalPlan]` argument to hold the partially analyzed plan, so that we can check the plan tree when reporting test failures.  This plan is passed by `QueryExecution.assertAnalyzed`.\r\n\r\n## How was this patch tested?\r\n\r\nExisting tests do the work.\r\n\r\n## TODO\r\n\r\n- [ ] Fix all tests\r\n- [ ] Re-enable MiMA check\r\n- [ ] Update ScalaDoc (`@since`, `@group`, and example code)\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/194295417","html_url":"https://github.com/apache/spark/pull/11443#issuecomment-194295417","issue_url":"https://api.github.com/repos/apache/spark/issues/11443","id":194295417,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-03-09T13:26:18Z","updated_at":"2016-03-09T13:26:18Z","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52747/\nTest FAILed."}}