{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21082","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21082/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21082/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21082/events","html_url":"https://github.com/apache/spark/pull/21082","id":314835650,"node_id":"MDExOlB1bGxSZXF1ZXN0MTgyMDAxNjY4","number":21082,"title":"[SPARK-22239][SQL][Python] Enable grouped aggregate pandas UDFs as window functions with unbounded window frames","user":{"login":"icexelloss","id":793516,"node_id":"MDQ6VXNlcjc5MzUxNg==","avatar_url":"https://avatars0.githubusercontent.com/u/793516?v=4","gravatar_id":"","url":"https://api.github.com/users/icexelloss","html_url":"https://github.com/icexelloss","followers_url":"https://api.github.com/users/icexelloss/followers","following_url":"https://api.github.com/users/icexelloss/following{/other_user}","gists_url":"https://api.github.com/users/icexelloss/gists{/gist_id}","starred_url":"https://api.github.com/users/icexelloss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/icexelloss/subscriptions","organizations_url":"https://api.github.com/users/icexelloss/orgs","repos_url":"https://api.github.com/users/icexelloss/repos","events_url":"https://api.github.com/users/icexelloss/events{/privacy}","received_events_url":"https://api.github.com/users/icexelloss/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":41,"created_at":"2018-04-16T22:07:52Z","updated_at":"2018-06-06T07:09:52Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21082","html_url":"https://github.com/apache/spark/pull/21082","diff_url":"https://github.com/apache/spark/pull/21082.diff","patch_url":"https://github.com/apache/spark/pull/21082.patch"},"body":"## What changes were proposed in this pull request?\r\nThis PR enables using a grouped aggregate pandas UDFs as window functions. The semantics is the same as using SQL aggregation function as window functions.\r\n\r\n```\r\n       >>> from pyspark.sql.functions import pandas_udf, PandasUDFType\r\n       >>> from pyspark.sql import Window\r\n       >>> df = spark.createDataFrame(\r\n       ...     [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)],\r\n       ...     (\"id\", \"v\"))\r\n       >>> @pandas_udf(\"double\", PandasUDFType.GROUPED_AGG)\r\n       ... def mean_udf(v):\r\n       ...     return v.mean()\r\n       >>> w = Window.partitionBy('id')\r\n       >>> df.withColumn('mean_v', mean_udf(df['v']).over(w)).show()\r\n       +---+----+------+\r\n       | id|   v|mean_v|\r\n       +---+----+------+\r\n       |  1| 1.0|   1.5|\r\n       |  1| 2.0|   1.5|\r\n       |  2| 3.0|   6.0|\r\n       |  2| 5.0|   6.0|\r\n       |  2|10.0|   6.0|\r\n       +---+----+------+\r\n```\r\n\r\nThe scope of this PR is somewhat limited in terms of:\r\n(1) Only supports unbounded window, which acts essentially as group by.\r\n(2) Only supports aggregation functions, not \"transform\" like window functions (n -> n mapping)\r\n\r\nBoth of these are left as future work. Especially, (1) needs careful thinking w.r.t. how to pass rolling window data to python efficiently. (2) is a bit easier but does require more changes therefore I think it's better to leave it as a separate PR.\r\n\r\n## How was this patch tested?\r\n\r\nWindowPandasUDFTests\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/394964307","html_url":"https://github.com/apache/spark/pull/21082#issuecomment-394964307","issue_url":"https://api.github.com/repos/apache/spark/issues/21082","id":394964307,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NDk2NDMwNw==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-06-06T07:09:52Z","updated_at":"2018-06-06T07:09:52Z","author_association":"NONE","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3822/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}