{"action":"created","comment":{"url":"https://api.github.com/repos/JuliaData/DataFrames.jl/pulls/comments/238077726","pull_request_review_id":180539607,"id":238077726,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIzODA3NzcyNg==","diff_hunk":"@@ -2,7 +2,20 @@\n \n Many data analysis tasks involve splitting a data set into groups, applying some functions to each of the groups and then combining the results. A standardized framework for handling this sort of computation is described in the paper \"[The Split-Apply-Combine Strategy for Data Analysis](http://www.jstatsoft.org/v40/i01)\", written by Hadley Wickham.\n \n-The DataFrames package supports the Split-Apply-Combine strategy through the `by` function, which takes in three arguments: (1) a DataFrame, (2) one or more columns to split the DataFrame on, and (3) a function or expression to apply to each subset of the DataFrame.\n+The DataFrames package supports the Split-Apply-Combine strategy through the `by` function, which takes in three arguments: (1) a `DataFrame`, (2) one or more columns to split the `DataFrame` on, and (3) a specification of one or more functions to apply to each subset of the `DataFrame`. This specification can be of the following forms:\n+- a `col => function` pair indicating that `function` should be called with the vector of values for column `col`, which can be a column name or index","path":"docs/src/man/split_apply_combine.md","position":6,"original_position":6,"commit_id":"4781ecb85747f2a9209fbea6dca7d7e1eb9789f2","original_commit_id":"4781ecb85747f2a9209fbea6dca7d7e1eb9789f2","user":{"login":"bkamins","id":6187170,"node_id":"MDQ6VXNlcjYxODcxNzA=","avatar_url":"https://avatars3.githubusercontent.com/u/6187170?v=4","gravatar_id":"","url":"https://api.github.com/users/bkamins","html_url":"https://github.com/bkamins","followers_url":"https://api.github.com/users/bkamins/followers","following_url":"https://api.github.com/users/bkamins/following{/other_user}","gists_url":"https://api.github.com/users/bkamins/gists{/gist_id}","starred_url":"https://api.github.com/users/bkamins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bkamins/subscriptions","organizations_url":"https://api.github.com/users/bkamins/orgs","repos_url":"https://api.github.com/users/bkamins/repos","events_url":"https://api.github.com/users/bkamins/events{/privacy}","received_events_url":"https://api.github.com/users/bkamins/received_events","type":"User","site_admin":false},"body":"maybe a numbered list as later you reference to the number of the form.","created_at":"2018-12-01T21:53:47Z","updated_at":"2018-12-01T21:53:47Z","html_url":"https://github.com/JuliaData/DataFrames.jl/pull/1601#discussion_r238077726","pull_request_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/pulls/1601","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/JuliaData/DataFrames.jl/pulls/comments/238077726"},"html":{"href":"https://github.com/JuliaData/DataFrames.jl/pull/1601#discussion_r238077726"},"pull_request":{"href":"https://api.github.com/repos/JuliaData/DataFrames.jl/pulls/1601"}}},"pull_request":{"url":"https://api.github.com/repos/JuliaData/DataFrames.jl/pulls/1601","id":231737696,"node_id":"MDExOlB1bGxSZXF1ZXN0MjMxNzM3Njk2","html_url":"https://github.com/JuliaData/DataFrames.jl/pull/1601","diff_url":"https://github.com/JuliaData/DataFrames.jl/pull/1601.diff","patch_url":"https://github.com/JuliaData/DataFrames.jl/pull/1601.patch","issue_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/issues/1601","number":1601,"state":"open","locked":false,"title":"Support type-stable map and combine on GroupedDataFrame","user":{"login":"nalimilan","id":1120448,"node_id":"MDQ6VXNlcjExMjA0NDg=","avatar_url":"https://avatars3.githubusercontent.com/u/1120448?v=4","gravatar_id":"","url":"https://api.github.com/users/nalimilan","html_url":"https://github.com/nalimilan","followers_url":"https://api.github.com/users/nalimilan/followers","following_url":"https://api.github.com/users/nalimilan/following{/other_user}","gists_url":"https://api.github.com/users/nalimilan/gists{/gist_id}","starred_url":"https://api.github.com/users/nalimilan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nalimilan/subscriptions","organizations_url":"https://api.github.com/users/nalimilan/orgs","repos_url":"https://api.github.com/users/nalimilan/repos","events_url":"https://api.github.com/users/nalimilan/events{/privacy}","received_events_url":"https://api.github.com/users/nalimilan/received_events","type":"User","site_admin":false},"body":"This is inspired by the [JuliaDB API](http://juliadb.org/latest/api/aggregation.html#IndexedTables.groupby). When columns are specified via `select`, the user-provided function is passed column vectors (as `SubArray`s) rather than a `SubDataFrame`. This gives fully type-stable code and dramatically improves performance.\r\n\r\nThis relatively limited change allows taking full advantage of the recent refactoring (https://github.com/JuliaData/DataFrames.jl/pull/1520). The performance gain is really incredible: about 200× for a simple sum after compilation.\r\n```julia\r\nusing DataFrames, BenchmarkTools\r\ndf = DataFrame(a = repeat(1:40000, outer=[20]),\r\n               b = randn(800000))\r\ngd = groupby(df, :a)\r\n\r\njulia> @time combine(d -> sum(d.b), gd);\r\n  4.657727 seconds (7.39 M allocations: 640.692 MiB, 2.92% gc time)\r\n\r\njulia> @btime combine(d -> sum(d.b), gd);\r\n  3.129 s (4399077 allocations: 494.68 MiB)\r\n\r\njulia> @time combine(:b => sum, gd);\r\n  0.545264 seconds (1.65 M allocations: 80.211 MiB, 3.91% gc time)\r\n\r\njulia> @btime combine(:b => sum, gd);\r\n  10.395 ms (399620 allocations: 16.78 MiB)\r\n```\r\n\r\nI still need to write tests, but otherwise the main question is the API. The discussion has already been started [on Discourse](https://discourse.julialang.org/t/dataframes-obtaining-the-subset-of-rows-by-a-set-of-values/15923/24?u=nalimilan). To sum up: while the `select` API illustrated above is reasonable, the slow variant is a bit shorter and more intuitive, in particular because it avoids repeating the column names in two different places. Yet we *really* don't want people to use the slow variant unless they really need to (for complex operations or when the number of columns isn't known in advance), since its terribly slow.\r\n\r\nThe solution I proposed to that problem is to detect whether the user-provided function expects a `SubDataFrame` or columns, assuming the argument names are those of columns. This would allow writing the example above as simply `combine(b -> sum(b), gd)`. Implementation-wise, it is actually quite easy to extract the argument names and match them to column names. The main problem is that when the function takes a single argument, we don't know whether it's supposed to be a `SubDataFrame` or a column. So far I haven't found a good solution to that, except writing `combine((b, args...) -> sum(b), gd)` or something silly like this.\r\n\r\nAnyway, we don't necessarily need to decide this right now, as the `select` approach could be complementary to another more convenient approach: `select` can be useful to choose programmatically which column(s) to operate on, and a convenience syntax could be based on that for its implementation.\r\n\r\nCc: @bkamins, @piever, @pdeffebach ","created_at":"2018-11-17T16:42:27Z","updated_at":"2018-12-01T21:53:47Z","closed_at":null,"merged_at":null,"merge_commit_sha":"e5f5c73e4bdba342fa08b9c74d485d6bc9f21207","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/pulls/1601/commits","review_comments_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/pulls/1601/comments","review_comment_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/pulls/comments{/number}","comments_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/issues/1601/comments","statuses_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/statuses/4781ecb85747f2a9209fbea6dca7d7e1eb9789f2","head":{"label":"JuliaData:nl/select","ref":"nl/select","sha":"4781ecb85747f2a9209fbea6dca7d7e1eb9789f2","user":{"login":"JuliaData","id":20142970,"node_id":"MDEyOk9yZ2FuaXphdGlvbjIwMTQyOTcw","avatar_url":"https://avatars3.githubusercontent.com/u/20142970?v=4","gravatar_id":"","url":"https://api.github.com/users/JuliaData","html_url":"https://github.com/JuliaData","followers_url":"https://api.github.com/users/JuliaData/followers","following_url":"https://api.github.com/users/JuliaData/following{/other_user}","gists_url":"https://api.github.com/users/JuliaData/gists{/gist_id}","starred_url":"https://api.github.com/users/JuliaData/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JuliaData/subscriptions","organizations_url":"https://api.github.com/users/JuliaData/orgs","repos_url":"https://api.github.com/users/JuliaData/repos","events_url":"https://api.github.com/users/JuliaData/events{/privacy}","received_events_url":"https://api.github.com/users/JuliaData/received_events","type":"Organization","site_admin":false},"repo":{"id":4992873,"node_id":"MDEwOlJlcG9zaXRvcnk0OTkyODcz","name":"DataFrames.jl","full_name":"JuliaData/DataFrames.jl","private":false,"owner":{"login":"JuliaData","id":20142970,"node_id":"MDEyOk9yZ2FuaXphdGlvbjIwMTQyOTcw","avatar_url":"https://avatars3.githubusercontent.com/u/20142970?v=4","gravatar_id":"","url":"https://api.github.com/users/JuliaData","html_url":"https://github.com/JuliaData","followers_url":"https://api.github.com/users/JuliaData/followers","following_url":"https://api.github.com/users/JuliaData/following{/other_user}","gists_url":"https://api.github.com/users/JuliaData/gists{/gist_id}","starred_url":"https://api.github.com/users/JuliaData/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JuliaData/subscriptions","organizations_url":"https://api.github.com/users/JuliaData/orgs","repos_url":"https://api.github.com/users/JuliaData/repos","events_url":"https://api.github.com/users/JuliaData/events{/privacy}","received_events_url":"https://api.github.com/users/JuliaData/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/JuliaData/DataFrames.jl","description":"Library for working with tabular data in Julia","fork":false,"url":"https://api.github.com/repos/JuliaData/DataFrames.jl","forks_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/forks","keys_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/keys{/key_id}","collaborators_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/teams","hooks_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/hooks","issue_events_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/issues/events{/number}","events_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/events","assignees_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/assignees{/user}","branches_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/branches{/branch}","tags_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/tags","blobs_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/git/refs{/sha}","trees_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/git/trees{/sha}","statuses_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/statuses/{sha}","languages_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/languages","stargazers_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/stargazers","contributors_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/contributors","subscribers_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/subscribers","subscription_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/subscription","commits_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/commits{/sha}","git_commits_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/git/commits{/sha}","comments_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/comments{/number}","issue_comment_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/issues/comments{/number}","contents_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/contents/{+path}","compare_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/compare/{base}...{head}","merges_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/merges","archive_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/downloads","issues_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/issues{/number}","pulls_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/pulls{/number}","milestones_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/milestones{/number}","notifications_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/labels{/name}","releases_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/releases{/id}","deployments_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/deployments","created_at":"2012-07-11T18:00:09Z","updated_at":"2018-12-01T12:24:55Z","pushed_at":"2018-12-01T20:52:20Z","git_url":"git://github.com/JuliaData/DataFrames.jl.git","ssh_url":"git@github.com:JuliaData/DataFrames.jl.git","clone_url":"https://github.com/JuliaData/DataFrames.jl.git","svn_url":"https://github.com/JuliaData/DataFrames.jl","homepage":"","size":9907,"stargazers_count":551,"watchers_count":551,"language":"Julia","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":215,"mirror_url":null,"archived":false,"open_issues_count":93,"license":{"key":"other","name":"Other","spdx_id":"NOASSERTION","url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":215,"open_issues":93,"watchers":551,"default_branch":"master"}},"base":{"label":"JuliaData:master","ref":"master","sha":"84a3f254dfe0fc5f22899a248c099e89f68542e9","user":{"login":"JuliaData","id":20142970,"node_id":"MDEyOk9yZ2FuaXphdGlvbjIwMTQyOTcw","avatar_url":"https://avatars3.githubusercontent.com/u/20142970?v=4","gravatar_id":"","url":"https://api.github.com/users/JuliaData","html_url":"https://github.com/JuliaData","followers_url":"https://api.github.com/users/JuliaData/followers","following_url":"https://api.github.com/users/JuliaData/following{/other_user}","gists_url":"https://api.github.com/users/JuliaData/gists{/gist_id}","starred_url":"https://api.github.com/users/JuliaData/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JuliaData/subscriptions","organizations_url":"https://api.github.com/users/JuliaData/orgs","repos_url":"https://api.github.com/users/JuliaData/repos","events_url":"https://api.github.com/users/JuliaData/events{/privacy}","received_events_url":"https://api.github.com/users/JuliaData/received_events","type":"Organization","site_admin":false},"repo":{"id":4992873,"node_id":"MDEwOlJlcG9zaXRvcnk0OTkyODcz","name":"DataFrames.jl","full_name":"JuliaData/DataFrames.jl","private":false,"owner":{"login":"JuliaData","id":20142970,"node_id":"MDEyOk9yZ2FuaXphdGlvbjIwMTQyOTcw","avatar_url":"https://avatars3.githubusercontent.com/u/20142970?v=4","gravatar_id":"","url":"https://api.github.com/users/JuliaData","html_url":"https://github.com/JuliaData","followers_url":"https://api.github.com/users/JuliaData/followers","following_url":"https://api.github.com/users/JuliaData/following{/other_user}","gists_url":"https://api.github.com/users/JuliaData/gists{/gist_id}","starred_url":"https://api.github.com/users/JuliaData/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JuliaData/subscriptions","organizations_url":"https://api.github.com/users/JuliaData/orgs","repos_url":"https://api.github.com/users/JuliaData/repos","events_url":"https://api.github.com/users/JuliaData/events{/privacy}","received_events_url":"https://api.github.com/users/JuliaData/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/JuliaData/DataFrames.jl","description":"Library for working with tabular data in Julia","fork":false,"url":"https://api.github.com/repos/JuliaData/DataFrames.jl","forks_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/forks","keys_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/keys{/key_id}","collaborators_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/teams","hooks_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/hooks","issue_events_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/issues/events{/number}","events_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/events","assignees_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/assignees{/user}","branches_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/branches{/branch}","tags_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/tags","blobs_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/git/refs{/sha}","trees_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/git/trees{/sha}","statuses_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/statuses/{sha}","languages_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/languages","stargazers_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/stargazers","contributors_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/contributors","subscribers_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/subscribers","subscription_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/subscription","commits_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/commits{/sha}","git_commits_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/git/commits{/sha}","comments_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/comments{/number}","issue_comment_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/issues/comments{/number}","contents_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/contents/{+path}","compare_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/compare/{base}...{head}","merges_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/merges","archive_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/downloads","issues_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/issues{/number}","pulls_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/pulls{/number}","milestones_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/milestones{/number}","notifications_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/labels{/name}","releases_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/releases{/id}","deployments_url":"https://api.github.com/repos/JuliaData/DataFrames.jl/deployments","created_at":"2012-07-11T18:00:09Z","updated_at":"2018-12-01T12:24:55Z","pushed_at":"2018-12-01T20:52:20Z","git_url":"git://github.com/JuliaData/DataFrames.jl.git","ssh_url":"git@github.com:JuliaData/DataFrames.jl.git","clone_url":"https://github.com/JuliaData/DataFrames.jl.git","svn_url":"https://github.com/JuliaData/DataFrames.jl","homepage":"","size":9907,"stargazers_count":551,"watchers_count":551,"language":"Julia","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":215,"mirror_url":null,"archived":false,"open_issues_count":93,"license":{"key":"other","name":"Other","spdx_id":"NOASSERTION","url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":215,"open_issues":93,"watchers":551,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/JuliaData/DataFrames.jl/pulls/1601"},"html":{"href":"https://github.com/JuliaData/DataFrames.jl/pull/1601"},"issue":{"href":"https://api.github.com/repos/JuliaData/DataFrames.jl/issues/1601"},"comments":{"href":"https://api.github.com/repos/JuliaData/DataFrames.jl/issues/1601/comments"},"review_comments":{"href":"https://api.github.com/repos/JuliaData/DataFrames.jl/pulls/1601/comments"},"review_comment":{"href":"https://api.github.com/repos/JuliaData/DataFrames.jl/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/JuliaData/DataFrames.jl/pulls/1601/commits"},"statuses":{"href":"https://api.github.com/repos/JuliaData/DataFrames.jl/statuses/4781ecb85747f2a9209fbea6dca7d7e1eb9789f2"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"bkamins"}}