{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/12008","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/12008/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/12008/comments","events_url":"https://api.github.com/repos/apache/spark/issues/12008/events","html_url":"https://github.com/apache/spark/pull/12008","id":144055075,"number":12008,"title":"[SPARK-14160] Time Windowing functions for Datasets","user":{"login":"brkyvz","id":5243515,"avatar_url":"https://avatars.githubusercontent.com/u/5243515?v=3","gravatar_id":"","url":"https://api.github.com/users/brkyvz","html_url":"https://github.com/brkyvz","followers_url":"https://api.github.com/users/brkyvz/followers","following_url":"https://api.github.com/users/brkyvz/following{/other_user}","gists_url":"https://api.github.com/users/brkyvz/gists{/gist_id}","starred_url":"https://api.github.com/users/brkyvz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brkyvz/subscriptions","organizations_url":"https://api.github.com/users/brkyvz/orgs","repos_url":"https://api.github.com/users/brkyvz/repos","events_url":"https://api.github.com/users/brkyvz/events{/privacy}","received_events_url":"https://api.github.com/users/brkyvz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2016-03-28T19:06:55Z","updated_at":"2016-03-29T00:05:23Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/12008","html_url":"https://github.com/apache/spark/pull/12008","diff_url":"https://github.com/apache/spark/pull/12008.diff","patch_url":"https://github.com/apache/spark/pull/12008.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR adds the function `window` as a column expression.\r\n\r\n`window` can be used to bucket rows into time windows given a time column. With this expression, performing time series analysis on batch data, as well as streaming data should become much more simpler.\r\n\r\n### Usage\r\n\r\nAssume the following schema:\r\n\r\n`sensor_id, measurement, timestamp`\r\n\r\nTo average 5 minute data every 1 minute (window length of 5 minutes, slide duration of 1 minute), we will use:\r\n```scala\r\ndf.groupBy(window(\"timestamp\", “5 minutes”, “1 minute”), \"sensor_id\")\r\n  .agg(mean(\"measurement\").as(\"avg_meas\"))\r\n```\r\n\r\nThis will generate windows such as:\r\n```\r\n09:00:00-09:05:00\r\n09:01:00-09:06:00\r\n09:02:00-09:07:00 ...\r\n```\r\n\r\nIntervals will start at every `slideDuration` starting at the unix epoch (1970-01-01 00:00:00 UTC).\r\nTo start intervals at a different point of time, e.g. 30 seconds after a minute, the `startTime` parameter can be used.\r\n\r\n```scala\r\ndf.groupBy(window(\"timestamp\", “5 minutes”, “1 minute”, \"30 second\"), \"sensor_id\")\r\n  .agg(mean(\"measurement\").as(\"avg_meas\"))\r\n```\r\n\r\nThis will generate windows such as:\r\n```\r\n09:00:30-09:05:30\r\n09:01:30-09:06:30\r\n09:02:30-09:07:30 ...\r\n```\r\n\r\nSupport for Python will be made in a follow up PR after this.\r\n\r\n## How was this patch tested?\r\n\r\nThis patch has some basic unit tests for the `TimeWindow` expression testing that the parameters pass validation, and it also has some unit/integration tests testing the correctness of the windowing and usability in complex operations (multi-column grouping, multi-column projections, joins).\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/202633875","html_url":"https://github.com/apache/spark/pull/12008#issuecomment-202633875","issue_url":"https://api.github.com/repos/apache/spark/issues/12008","id":202633875,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-03-29T00:05:22Z","updated_at":"2016-03-29T00:05:22Z","body":"Merged build finished. Test FAILed."}}