{"action":"created","comment":{"url":"https://api.github.com/repos/influxdata/ifql/pulls/comments/176246087","pull_request_review_id":105916678,"id":176246087,"diff_hunk":"@@ -0,0 +1,282 @@\n+\n+IFQL has an unanswered design question:\n+\n+> Should aggregate/selection operations be arguments to the window function? Or should they be their own functions?\n+\n+\n+This document attempts to capture the existing conversations around this question and to clearly state the implications that each option holds.\n+\n+### Disclaimer:\n+\n+I am heavily in favor of the current model as opposed to changing the way the window function works.\n+I have tried to be objective in my analysis but I am declaring my bias.\n+\n+If you are in a hurry, there is a one sentence TL;DR at the end of this document.\n+\n+## The Problem\n+\n+I postulate that whether window functions accept an aggregate function or whether aggregate functions standalone, is actually a question of which underlying types should be used to represent the data.\n+\n+As such this question really boils down to the type system we use to model the data on which IFQL operates.\n+The type system exposes itself in IFQL the language and the query engine.\n+If we can clearly state what type is at play in any expression in an IFQL script and declare how a function operates on that type then I believe we will have solved this design question.\n+\n+The current confusion around how the window function operates in relation with aggregate functions stems from the underlying type system not being explicitly defined anywhere, or clearly expressed to users.\n+\n+I will explain the two type systems at play and how that arises to the two different manifestations in IFQL syntax.\n+Then we can make a decision on which type system we like best for expressing the operations. \n+\n+## The vector type system\n+\n+The vector type system treats everything as a vector of data. There are two kinds of vectors:\n+\n+* Column vectors - Column vectors contain a list of values, the values all represent the same point in time and each value comes from a different series.\n+* Row vectors - Row vectors contain a list of values, the values all represent data from the same series and each value comes from a different time.\n+\n+A table is either a set of row vectors or a set of column vectors. This means that a table has series along is row dimension and time along its column dimension.\n+\n+The window function accepts three arguments: \n+\n+* a table,\n+* an aggregate function,\n+* an interval on which to window the data (this could be a set of arguments defining how data should be windowed, but for simplicity let's assume its a single argument).\n+\n+The window function returns a table as a set of column vectors.\n+\n+ The window function does the following:\n+\n+1. Split each row vector into multiple row vectors based on the window interval\n+2. Apply the aggregate function to each row vector producing a scalar.\n+3. Combine the scalars from all series into a column vector for a specific instant in time.\n+4. Returns a table which is the set of column vectors.\n+\n+### Example IFQL syntax:\n+\n+```\n+from(db:”foo”)\n+    |> range(start:-1h)\n+    |> window(every: 1m, fn: (row) => mean(row))\n+```\n+\n+Interestingly this model would apply to the group function as well, expect the group function operates on columns:\n+\n+ \n+```\n+from(db:”foo”)\n+    |> range(start:-1h)\n+    |> group(by: [“host”, “region”], fn: (col) => mean(col))\n+```\n+\n+In this type system the aggregate functions would not operate on tables but only on either row or column vectors.\n+\n+### Some questions:\n+\n+> How do you perform both a group by aggregate and window aggregate in the same step?\n+\n+What if I want to compute the mean RAM usage across all hosts by region over the last 5m.\n+I could compute the mean by each host first over the last 5m, and then compute the mean of the host means.\n+But that is not numerically the same result.\n+\n+If the answer is that group by should not accept an aggregate function then why? Why is window different?\n+\n+> How are series with multiple values modeled?\n+\n+I am not sure.\n+\n+## The block type system\n+\n+The current implementation of IFQL uses a type system I am calling the block type system because the fundamental type is a block.\n+\n+A block is a table of data.\n+The columns are named and represent the time, tags and fields of the data.\n+The rows represent each record in the dataset.\n+A block closely resembles a traditional SQL relation (aka table).\n+\n+A stream is a set of blocks.\n+A stream may be unbounded meaning that there may be potentially an infinite number of blocks in the set.\n+The blocks within a stream have not ordering.\n+NOTE: We currently call a stream a `table`, which is probably contributing to the confusion in the data model.\n+\n+A stream is data in motion, a block is data at rest.\n+\n+Every function in IFQL consumes a stream and produces a stream.\n+There is no relation between the number of blocks that enter a function with the number of blocks that leave a function.\n+\n+Specifically the window function may consume any number of blocks from the input stream and output any number of blocks on the output stream.\n+\n+Since the window function is consuming an arbitrary number of blocks and producing an arbitrary number of blocks, the question arises: When do we know a block is complete and can be emitted on the output stream? The answer is to use triggers.\n+Triggers indicate when a block should be materialized and emitted on the output stream.\n+The conditions that a trigger can use vary, but for simplicity I’ll explain only the “lagged trigger”.\n+A lagged trigger “fires” when the current processing time + some fixed amount of lag has elapsed past the event time of the window.\n+Each outgoing block being produced by the window function has its own trigger.\n+\n+The window function does the following:\n+\n+1. Receive an incoming block\n+2. Assign each row from the incoming block to any number of outgoing blocks.\n+3. Check the triggers for all outgoing blocks.\n+4. Emit blocks who’s trigger has “fired”.\n+\n+\n+The group by function actually does the same thing as the window function except that step 2 uses a different method of assigning input rows to output blocks.\n+The window function assigns output blocks by the time of the row, the group by function assigns outgoing blocks by the tags of the row.\n+\n+In fact all functions in this model perform steps 1, 3 and 4.\n+This is not specific to windowing or grouping. A function need only define its behavior in step 2.\n+\n+An aggregate/selection function consumes any number of input blocks and always outputs exactly one output block per group represented in the input blocks.\n+Each incoming block produces a single row in the output block.\n+The output block time bounds represent the entire time range of the query.\n+In contrast the grouping bounds of the output block remain unchanged. \n+\n+Using the same example as the vector system, this is the current IFQL syntax:\n+\n+```\n+from(db:”foo”)\n+    |> range(start:-1h)\n+    |> window(every: 1m)\n+    |> mean()\n+```\n+\n+Similarly the group by becomes\n+\n+ \n+```\n+from(db:”foo”)\n+    |> range(start:-1h)\n+    |> group(by: [“host”, “region”])\n+    |> mean()\n+```\n+\n+And finally we can now express computing the mean of RAM usage for all hosts by region over the last 5m.\n+\n+```\n+from(db:”foo”)\n+    |> range(start:-1h)\n+    |> group(by: [“region”])\n+    |> window(every:5m)\n+    |> mean()\n+```\n+\n+\n+### Some questions:\n+\n+I think the use of the parameter name `table` has possibly caused some confusion with how the underlying model is understood.\n+Maybe we should use `stream` since in reality each function is accepting a stream of tables and outputting a stream of tables.\n+In addition we would rename block to `table` to make it clear.\n+\n+> Why do aggregate/selection functions produce only a single block per group instead of having a one-to-one relation with input blocks?\n+\n+I think we need to discuss this more and possibly explore what it would mean to change this so that aggregate functions produce a block for every input block.\n+\n+\n+## Window/Aggregate with the Block type system\n+\n+I want to directly address why the design of having the window function accept an aggregate function as an argument is incompatible with the block type system.\n+\n+First, since the aggregate functions can be expressed in the type system independent of the window function, why would we want to force coupling the concepts.\n+\n+In addition, if we were to force the coupling then the window function would need to implement internal to itself the logic to handle internal block boundaries and aggregation instead of letting the existing framework handle the logic.\n+\n+\n+## Concerns with the current model\n+\n+There have been some great feedback about the current model. I want to address that feedback directly.\n+The sources of the feedback have been documented below in the NOTES section.\n+\n+> How do I know when an IFQL function is getting a stream of blocks instead of just a single block?\n+\n+IFQL functions always receives a stream of blocks, that stream may have many blocks or just one but can always be thought of as a stream.","path":"docs/window.md","position":189,"original_position":189,"commit_id":"cfd6a3b6e2ffe75729010ed59035ffe31c33fc9e","original_commit_id":"cfd6a3b6e2ffe75729010ed59035ffe31c33fc9e","user":{"login":"nathanielc","id":3771906,"avatar_url":"https://avatars3.githubusercontent.com/u/3771906?v=4","gravatar_id":"","url":"https://api.github.com/users/nathanielc","html_url":"https://github.com/nathanielc","followers_url":"https://api.github.com/users/nathanielc/followers","following_url":"https://api.github.com/users/nathanielc/following{/other_user}","gists_url":"https://api.github.com/users/nathanielc/gists{/gist_id}","starred_url":"https://api.github.com/users/nathanielc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nathanielc/subscriptions","organizations_url":"https://api.github.com/users/nathanielc/orgs","repos_url":"https://api.github.com/users/nathanielc/repos","events_url":"https://api.github.com/users/nathanielc/events{/privacy}","received_events_url":"https://api.github.com/users/nathanielc/received_events","type":"User","site_admin":false},"body":"Yeah, I need to think about this some more.","created_at":"2018-03-21T21:35:34Z","updated_at":"2018-03-21T21:35:34Z","html_url":"https://github.com/influxdata/ifql/pull/271#discussion_r176246087","pull_request_url":"https://api.github.com/repos/influxdata/ifql/pulls/271","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/influxdata/ifql/pulls/comments/176246087"},"html":{"href":"https://github.com/influxdata/ifql/pull/271#discussion_r176246087"},"pull_request":{"href":"https://api.github.com/repos/influxdata/ifql/pulls/271"}},"in_reply_to_id":176219530},"pull_request":{"url":"https://api.github.com/repos/influxdata/ifql/pulls/271","id":176567604,"html_url":"https://github.com/influxdata/ifql/pull/271","diff_url":"https://github.com/influxdata/ifql/pull/271.diff","patch_url":"https://github.com/influxdata/ifql/pull/271.patch","issue_url":"https://api.github.com/repos/influxdata/ifql/issues/271","number":271,"state":"open","locked":false,"title":"Add document discussing window design","user":{"login":"nathanielc","id":3771906,"avatar_url":"https://avatars3.githubusercontent.com/u/3771906?v=4","gravatar_id":"","url":"https://api.github.com/users/nathanielc","html_url":"https://github.com/nathanielc","followers_url":"https://api.github.com/users/nathanielc/followers","following_url":"https://api.github.com/users/nathanielc/following{/other_user}","gists_url":"https://api.github.com/users/nathanielc/gists{/gist_id}","starred_url":"https://api.github.com/users/nathanielc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nathanielc/subscriptions","organizations_url":"https://api.github.com/users/nathanielc/orgs","repos_url":"https://api.github.com/users/nathanielc/repos","events_url":"https://api.github.com/users/nathanielc/events{/privacy}","received_events_url":"https://api.github.com/users/nathanielc/received_events","type":"User","site_admin":false},"body":"Please comment on this PR about the proposed design any questions.","created_at":"2018-03-21T18:17:51Z","updated_at":"2018-03-21T21:35:34Z","closed_at":null,"merged_at":null,"merge_commit_sha":"3f3f343e8511732cd4ebdd5dee9e66ad337a74dc","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/influxdata/ifql/pulls/271/commits","review_comments_url":"https://api.github.com/repos/influxdata/ifql/pulls/271/comments","review_comment_url":"https://api.github.com/repos/influxdata/ifql/pulls/comments{/number}","comments_url":"https://api.github.com/repos/influxdata/ifql/issues/271/comments","statuses_url":"https://api.github.com/repos/influxdata/ifql/statuses/cfd6a3b6e2ffe75729010ed59035ffe31c33fc9e","head":{"label":"influxdata:nc-window-design","ref":"nc-window-design","sha":"cfd6a3b6e2ffe75729010ed59035ffe31c33fc9e","user":{"login":"influxdata","id":5713248,"avatar_url":"https://avatars2.githubusercontent.com/u/5713248?v=4","gravatar_id":"","url":"https://api.github.com/users/influxdata","html_url":"https://github.com/influxdata","followers_url":"https://api.github.com/users/influxdata/followers","following_url":"https://api.github.com/users/influxdata/following{/other_user}","gists_url":"https://api.github.com/users/influxdata/gists{/gist_id}","starred_url":"https://api.github.com/users/influxdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/influxdata/subscriptions","organizations_url":"https://api.github.com/users/influxdata/orgs","repos_url":"https://api.github.com/users/influxdata/repos","events_url":"https://api.github.com/users/influxdata/events{/privacy}","received_events_url":"https://api.github.com/users/influxdata/received_events","type":"Organization","site_admin":false},"repo":{"id":98210239,"name":"ifql","full_name":"influxdata/ifql","owner":{"login":"influxdata","id":5713248,"avatar_url":"https://avatars2.githubusercontent.com/u/5713248?v=4","gravatar_id":"","url":"https://api.github.com/users/influxdata","html_url":"https://github.com/influxdata","followers_url":"https://api.github.com/users/influxdata/followers","following_url":"https://api.github.com/users/influxdata/following{/other_user}","gists_url":"https://api.github.com/users/influxdata/gists{/gist_id}","starred_url":"https://api.github.com/users/influxdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/influxdata/subscriptions","organizations_url":"https://api.github.com/users/influxdata/orgs","repos_url":"https://api.github.com/users/influxdata/repos","events_url":"https://api.github.com/users/influxdata/events{/privacy}","received_events_url":"https://api.github.com/users/influxdata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/influxdata/ifql","description":null,"fork":false,"url":"https://api.github.com/repos/influxdata/ifql","forks_url":"https://api.github.com/repos/influxdata/ifql/forks","keys_url":"https://api.github.com/repos/influxdata/ifql/keys{/key_id}","collaborators_url":"https://api.github.com/repos/influxdata/ifql/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/influxdata/ifql/teams","hooks_url":"https://api.github.com/repos/influxdata/ifql/hooks","issue_events_url":"https://api.github.com/repos/influxdata/ifql/issues/events{/number}","events_url":"https://api.github.com/repos/influxdata/ifql/events","assignees_url":"https://api.github.com/repos/influxdata/ifql/assignees{/user}","branches_url":"https://api.github.com/repos/influxdata/ifql/branches{/branch}","tags_url":"https://api.github.com/repos/influxdata/ifql/tags","blobs_url":"https://api.github.com/repos/influxdata/ifql/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/influxdata/ifql/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/influxdata/ifql/git/refs{/sha}","trees_url":"https://api.github.com/repos/influxdata/ifql/git/trees{/sha}","statuses_url":"https://api.github.com/repos/influxdata/ifql/statuses/{sha}","languages_url":"https://api.github.com/repos/influxdata/ifql/languages","stargazers_url":"https://api.github.com/repos/influxdata/ifql/stargazers","contributors_url":"https://api.github.com/repos/influxdata/ifql/contributors","subscribers_url":"https://api.github.com/repos/influxdata/ifql/subscribers","subscription_url":"https://api.github.com/repos/influxdata/ifql/subscription","commits_url":"https://api.github.com/repos/influxdata/ifql/commits{/sha}","git_commits_url":"https://api.github.com/repos/influxdata/ifql/git/commits{/sha}","comments_url":"https://api.github.com/repos/influxdata/ifql/comments{/number}","issue_comment_url":"https://api.github.com/repos/influxdata/ifql/issues/comments{/number}","contents_url":"https://api.github.com/repos/influxdata/ifql/contents/{+path}","compare_url":"https://api.github.com/repos/influxdata/ifql/compare/{base}...{head}","merges_url":"https://api.github.com/repos/influxdata/ifql/merges","archive_url":"https://api.github.com/repos/influxdata/ifql/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/influxdata/ifql/downloads","issues_url":"https://api.github.com/repos/influxdata/ifql/issues{/number}","pulls_url":"https://api.github.com/repos/influxdata/ifql/pulls{/number}","milestones_url":"https://api.github.com/repos/influxdata/ifql/milestones{/number}","notifications_url":"https://api.github.com/repos/influxdata/ifql/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/influxdata/ifql/labels{/name}","releases_url":"https://api.github.com/repos/influxdata/ifql/releases{/id}","deployments_url":"https://api.github.com/repos/influxdata/ifql/deployments","created_at":"2017-07-24T16:14:20Z","updated_at":"2018-03-21T21:14:04Z","pushed_at":"2018-03-21T21:14:37Z","git_url":"git://github.com/influxdata/ifql.git","ssh_url":"git@github.com:influxdata/ifql.git","clone_url":"https://github.com/influxdata/ifql.git","svn_url":"https://github.com/influxdata/ifql","homepage":null,"size":1597,"stargazers_count":57,"watchers_count":57,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3,"mirror_url":null,"archived":false,"open_issues_count":82,"license":{"key":"other","name":"Other","spdx_id":null,"url":null},"forks":3,"open_issues":82,"watchers":57,"default_branch":"master"}},"base":{"label":"influxdata:master","ref":"master","sha":"7b4459c115ce8971f9a6f637155709102e870f08","user":{"login":"influxdata","id":5713248,"avatar_url":"https://avatars2.githubusercontent.com/u/5713248?v=4","gravatar_id":"","url":"https://api.github.com/users/influxdata","html_url":"https://github.com/influxdata","followers_url":"https://api.github.com/users/influxdata/followers","following_url":"https://api.github.com/users/influxdata/following{/other_user}","gists_url":"https://api.github.com/users/influxdata/gists{/gist_id}","starred_url":"https://api.github.com/users/influxdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/influxdata/subscriptions","organizations_url":"https://api.github.com/users/influxdata/orgs","repos_url":"https://api.github.com/users/influxdata/repos","events_url":"https://api.github.com/users/influxdata/events{/privacy}","received_events_url":"https://api.github.com/users/influxdata/received_events","type":"Organization","site_admin":false},"repo":{"id":98210239,"name":"ifql","full_name":"influxdata/ifql","owner":{"login":"influxdata","id":5713248,"avatar_url":"https://avatars2.githubusercontent.com/u/5713248?v=4","gravatar_id":"","url":"https://api.github.com/users/influxdata","html_url":"https://github.com/influxdata","followers_url":"https://api.github.com/users/influxdata/followers","following_url":"https://api.github.com/users/influxdata/following{/other_user}","gists_url":"https://api.github.com/users/influxdata/gists{/gist_id}","starred_url":"https://api.github.com/users/influxdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/influxdata/subscriptions","organizations_url":"https://api.github.com/users/influxdata/orgs","repos_url":"https://api.github.com/users/influxdata/repos","events_url":"https://api.github.com/users/influxdata/events{/privacy}","received_events_url":"https://api.github.com/users/influxdata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/influxdata/ifql","description":null,"fork":false,"url":"https://api.github.com/repos/influxdata/ifql","forks_url":"https://api.github.com/repos/influxdata/ifql/forks","keys_url":"https://api.github.com/repos/influxdata/ifql/keys{/key_id}","collaborators_url":"https://api.github.com/repos/influxdata/ifql/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/influxdata/ifql/teams","hooks_url":"https://api.github.com/repos/influxdata/ifql/hooks","issue_events_url":"https://api.github.com/repos/influxdata/ifql/issues/events{/number}","events_url":"https://api.github.com/repos/influxdata/ifql/events","assignees_url":"https://api.github.com/repos/influxdata/ifql/assignees{/user}","branches_url":"https://api.github.com/repos/influxdata/ifql/branches{/branch}","tags_url":"https://api.github.com/repos/influxdata/ifql/tags","blobs_url":"https://api.github.com/repos/influxdata/ifql/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/influxdata/ifql/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/influxdata/ifql/git/refs{/sha}","trees_url":"https://api.github.com/repos/influxdata/ifql/git/trees{/sha}","statuses_url":"https://api.github.com/repos/influxdata/ifql/statuses/{sha}","languages_url":"https://api.github.com/repos/influxdata/ifql/languages","stargazers_url":"https://api.github.com/repos/influxdata/ifql/stargazers","contributors_url":"https://api.github.com/repos/influxdata/ifql/contributors","subscribers_url":"https://api.github.com/repos/influxdata/ifql/subscribers","subscription_url":"https://api.github.com/repos/influxdata/ifql/subscription","commits_url":"https://api.github.com/repos/influxdata/ifql/commits{/sha}","git_commits_url":"https://api.github.com/repos/influxdata/ifql/git/commits{/sha}","comments_url":"https://api.github.com/repos/influxdata/ifql/comments{/number}","issue_comment_url":"https://api.github.com/repos/influxdata/ifql/issues/comments{/number}","contents_url":"https://api.github.com/repos/influxdata/ifql/contents/{+path}","compare_url":"https://api.github.com/repos/influxdata/ifql/compare/{base}...{head}","merges_url":"https://api.github.com/repos/influxdata/ifql/merges","archive_url":"https://api.github.com/repos/influxdata/ifql/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/influxdata/ifql/downloads","issues_url":"https://api.github.com/repos/influxdata/ifql/issues{/number}","pulls_url":"https://api.github.com/repos/influxdata/ifql/pulls{/number}","milestones_url":"https://api.github.com/repos/influxdata/ifql/milestones{/number}","notifications_url":"https://api.github.com/repos/influxdata/ifql/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/influxdata/ifql/labels{/name}","releases_url":"https://api.github.com/repos/influxdata/ifql/releases{/id}","deployments_url":"https://api.github.com/repos/influxdata/ifql/deployments","created_at":"2017-07-24T16:14:20Z","updated_at":"2018-03-21T21:14:04Z","pushed_at":"2018-03-21T21:14:37Z","git_url":"git://github.com/influxdata/ifql.git","ssh_url":"git@github.com:influxdata/ifql.git","clone_url":"https://github.com/influxdata/ifql.git","svn_url":"https://github.com/influxdata/ifql","homepage":null,"size":1597,"stargazers_count":57,"watchers_count":57,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3,"mirror_url":null,"archived":false,"open_issues_count":82,"license":{"key":"other","name":"Other","spdx_id":null,"url":null},"forks":3,"open_issues":82,"watchers":57,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/influxdata/ifql/pulls/271"},"html":{"href":"https://github.com/influxdata/ifql/pull/271"},"issue":{"href":"https://api.github.com/repos/influxdata/ifql/issues/271"},"comments":{"href":"https://api.github.com/repos/influxdata/ifql/issues/271/comments"},"review_comments":{"href":"https://api.github.com/repos/influxdata/ifql/pulls/271/comments"},"review_comment":{"href":"https://api.github.com/repos/influxdata/ifql/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/influxdata/ifql/pulls/271/commits"},"statuses":{"href":"https://api.github.com/repos/influxdata/ifql/statuses/cfd6a3b6e2ffe75729010ed59035ffe31c33fc9e"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"nathanielc"}}