{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/62399743","id":62399743,"diff_hunk":"@@ -1214,6 +1214,77 @@ setMethod(\"dapply\",\n             dataFrame(sdf)\n           })\n \n+#' gapply\n+#'\n+#' Apply a function to each group of a DataFrame. The group is defined by an input\n+#' grouping column(s).\n+#'\n+#' @param x A SparkDataFrame\n+#' @param func A function to be applied to each group partition specified by grouping\n+#'             column(s) of the SparkDataFrame.\n+#'             The output of func is a local R data.frame.\n+#' @param schema The schema of the resulting SparkDataFrame after the function is applied.\n+#'               It must match the output of func.\n+#' @family SparkDataFrame functions\n+#' @rdname gapply\n+#' @name gapply\n+#' @export\n+#' @examples\n+#'\n+#' \\dontrun{\n+#'\n+#' Computes the arithmetic mean of `Sepal_Width` by grouping\n+#' on `Species`. Output the grouping value and the average.\n+#'\n+#' df <- createDataFrame (sqlContext, iris)\n+#' schema <-  structType(structField(\"Species\", \"string\"), structField(\"Avg\", \"double\"))\n+#' df1 <- gapply(\n+#'   df,\n+#'   function(x) {\n+#'     data.frame(x$Species[1], mean(x$Sepal_Width), stringsAsFactors = FALSE)\n+#'   },\n+#'   schema, col=df$\"Species\")\n+#' collect(df1)\n+#'\n+#' Species      Avg\n+#' -----------------\n+#' virginica   2.974\n+#' versicolor  2.770\n+#' setosa      3.428\n+#'\n+#' Fits linear models on iris dataset by grouping on the `Species` column and\n+#' using `Sepal_Length` as a target variable, `Sepal_Width`, `Petal_Length`\n+#' and `Petal_Width` as training features.\n+#'\n+#' df <- createDataFrame (sqlContext, iris)\n+#' schema <- structType(structField(\"(Intercept)\", \"double\"),\n+#'   structField(\"Sepal_Width\", \"double\"), structField(\"Petal_Length\", \"double\"),\n+#'   structField(\"Petal_Width\", \"double\"))\n+#' df1 <- gapply(\n+#'   df,\n+#'   function(x) {\n+#'     model <- suppressWarnings(lm(Sepal_Length ~\n+#'     Sepal_Width + Petal_Length + Petal_Width, x))\n+#'     data.frame(t(coef(model)))\n+#'   }, schema, df$\"Species\")\n+#' collect(df1)\n+#'\n+#'Result\n+#'---------\n+#' Model  (Intercept)  Sepal_Width  Petal_Length  Petal_Width\n+#' 1        0.699883    0.3303370    0.9455356    -0.1697527\n+#' 2        1.895540    0.3868576    0.9083370    -0.6792238\n+#' 3        2.351890    0.6548350    0.2375602     0.2521257\n+#'\n+#'}\n+setMethod(\"gapply\",\n+          signature(x = \"SparkDataFrame\", func = \"function\", schema = \"structType\",\n+                    col = \"Column\"),","path":"R/pkg/R/DataFrame.R","position":69,"original_position":69,"commit_id":"30693c2b40ab459a9fe252a2d00595c8190f2094","original_commit_id":"30693c2b40ab459a9fe252a2d00595c8190f2094","user":{"login":"shivaram","id":143893,"avatar_url":"https://avatars.githubusercontent.com/u/143893?v=3","gravatar_id":"","url":"https://api.github.com/users/shivaram","html_url":"https://github.com/shivaram","followers_url":"https://api.github.com/users/shivaram/followers","following_url":"https://api.github.com/users/shivaram/following{/other_user}","gists_url":"https://api.github.com/users/shivaram/gists{/gist_id}","starred_url":"https://api.github.com/users/shivaram/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shivaram/subscriptions","organizations_url":"https://api.github.com/users/shivaram/orgs","repos_url":"https://api.github.com/users/shivaram/repos","events_url":"https://api.github.com/users/shivaram/events{/privacy}","received_events_url":"https://api.github.com/users/shivaram/received_events","type":"User","site_admin":false},"body":"can we handle multiple columns now that we are using repartition ?","created_at":"2016-05-06T22:31:06Z","updated_at":"2016-05-06T22:31:06Z","html_url":"https://github.com/apache/spark/pull/12966#discussion_r62399743","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/12966","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/62399743"},"html":{"href":"https://github.com/apache/spark/pull/12966#discussion_r62399743"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/12966"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/12966","id":69217300,"html_url":"https://github.com/apache/spark/pull/12966","diff_url":"https://github.com/apache/spark/pull/12966.diff","patch_url":"https://github.com/apache/spark/pull/12966.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/12966","number":12966,"state":"open","locked":false,"title":"[SPARK-15196][SparkR] Add a wrapper for dapply(repartition(col,...), ... )","user":{"login":"NarineK","id":12010601,"avatar_url":"https://avatars.githubusercontent.com/u/12010601?v=3","gravatar_id":"","url":"https://api.github.com/users/NarineK","html_url":"https://github.com/NarineK","followers_url":"https://api.github.com/users/NarineK/followers","following_url":"https://api.github.com/users/NarineK/following{/other_user}","gists_url":"https://api.github.com/users/NarineK/gists{/gist_id}","starred_url":"https://api.github.com/users/NarineK/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/NarineK/subscriptions","organizations_url":"https://api.github.com/users/NarineK/orgs","repos_url":"https://api.github.com/users/NarineK/repos","events_url":"https://api.github.com/users/NarineK/events{/privacy}","received_events_url":"https://api.github.com/users/NarineK/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nAs mentioned in :\r\nhttps://github.com/apache/spark/pull/12836#issuecomment-217338855\r\nWe would like to create a wrapper for: dapply(repartiition(col,...), ... )\r\nThis will allow to run aggregate functions on groups which are identified by a list of grouping columns.\r\n\r\nI called the wrapper method gapply. We can rename it if we want to call it differently.\r\nWe could also have:\r\n` setMethod(\"dapply\", signature(x = \"SparkDataFrame\", func = \"function\", schema = \"structType\", col = \"Column\"),` , however, dapply already has many examples in the doc and if we add new examples for aggregate functions that will make the documentation longer and less clear.\r\n\r\n## How was this patch tested?\r\nUnit tests\r\n1. Group by a column and compute mean\r\n2.  Group by a column and train linear model\r\n\r\n\r\n\r\n","created_at":"2016-05-06T22:09:15Z","updated_at":"2016-05-06T22:31:06Z","closed_at":null,"merged_at":null,"merge_commit_sha":"29300483a0875415021396f6e2456e4aa1e8ecfc","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/12966/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/12966/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/12966/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/30693c2b40ab459a9fe252a2d00595c8190f2094","head":{"label":"NarineK:repartitionWithDapply","ref":"repartitionWithDapply","sha":"30693c2b40ab459a9fe252a2d00595c8190f2094","user":{"login":"NarineK","id":12010601,"avatar_url":"https://avatars.githubusercontent.com/u/12010601?v=3","gravatar_id":"","url":"https://api.github.com/users/NarineK","html_url":"https://github.com/NarineK","followers_url":"https://api.github.com/users/NarineK/followers","following_url":"https://api.github.com/users/NarineK/following{/other_user}","gists_url":"https://api.github.com/users/NarineK/gists{/gist_id}","starred_url":"https://api.github.com/users/NarineK/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/NarineK/subscriptions","organizations_url":"https://api.github.com/users/NarineK/orgs","repos_url":"https://api.github.com/users/NarineK/repos","events_url":"https://api.github.com/users/NarineK/events{/privacy}","received_events_url":"https://api.github.com/users/NarineK/received_events","type":"User","site_admin":false},"repo":{"id":42900480,"name":"spark","full_name":"NarineK/spark","owner":{"login":"NarineK","id":12010601,"avatar_url":"https://avatars.githubusercontent.com/u/12010601?v=3","gravatar_id":"","url":"https://api.github.com/users/NarineK","html_url":"https://github.com/NarineK","followers_url":"https://api.github.com/users/NarineK/followers","following_url":"https://api.github.com/users/NarineK/following{/other_user}","gists_url":"https://api.github.com/users/NarineK/gists{/gist_id}","starred_url":"https://api.github.com/users/NarineK/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/NarineK/subscriptions","organizations_url":"https://api.github.com/users/NarineK/orgs","repos_url":"https://api.github.com/users/NarineK/repos","events_url":"https://api.github.com/users/NarineK/events{/privacy}","received_events_url":"https://api.github.com/users/NarineK/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/NarineK/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/NarineK/spark","forks_url":"https://api.github.com/repos/NarineK/spark/forks","keys_url":"https://api.github.com/repos/NarineK/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/NarineK/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/NarineK/spark/teams","hooks_url":"https://api.github.com/repos/NarineK/spark/hooks","issue_events_url":"https://api.github.com/repos/NarineK/spark/issues/events{/number}","events_url":"https://api.github.com/repos/NarineK/spark/events","assignees_url":"https://api.github.com/repos/NarineK/spark/assignees{/user}","branches_url":"https://api.github.com/repos/NarineK/spark/branches{/branch}","tags_url":"https://api.github.com/repos/NarineK/spark/tags","blobs_url":"https://api.github.com/repos/NarineK/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/NarineK/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/NarineK/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/NarineK/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/NarineK/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/NarineK/spark/languages","stargazers_url":"https://api.github.com/repos/NarineK/spark/stargazers","contributors_url":"https://api.github.com/repos/NarineK/spark/contributors","subscribers_url":"https://api.github.com/repos/NarineK/spark/subscribers","subscription_url":"https://api.github.com/repos/NarineK/spark/subscription","commits_url":"https://api.github.com/repos/NarineK/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/NarineK/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/NarineK/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/NarineK/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/NarineK/spark/contents/{+path}","compare_url":"https://api.github.com/repos/NarineK/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/NarineK/spark/merges","archive_url":"https://api.github.com/repos/NarineK/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/NarineK/spark/downloads","issues_url":"https://api.github.com/repos/NarineK/spark/issues{/number}","pulls_url":"https://api.github.com/repos/NarineK/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/NarineK/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/NarineK/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/NarineK/spark/labels{/name}","releases_url":"https://api.github.com/repos/NarineK/spark/releases{/id}","deployments_url":"https://api.github.com/repos/NarineK/spark/deployments","created_at":"2015-09-22T00:03:17Z","updated_at":"2015-09-22T00:03:47Z","pushed_at":"2016-05-06T22:18:37Z","git_url":"git://github.com/NarineK/spark.git","ssh_url":"git@github.com:NarineK/spark.git","clone_url":"https://github.com/NarineK/spark.git","svn_url":"https://github.com/NarineK/spark","homepage":null,"size":167191,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"f7b7ef41662d7d02fc4f834f3c6c4ee8802e949c","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2016-05-06T20:29:35Z","pushed_at":"2016-05-06T22:21:42Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":478762,"stargazers_count":8417,"watchers_count":8417,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":7815,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":428,"forks":7815,"open_issues":428,"watchers":8417,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/12966"},"html":{"href":"https://github.com/apache/spark/pull/12966"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/12966"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/12966/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/12966/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/12966/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/30693c2b40ab459a9fe252a2d00595c8190f2094"}}}}