{"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-systemml/issues/32","labels_url":"https://api.github.com/repos/apache/incubator-systemml/issues/32/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-systemml/issues/32/comments","events_url":"https://api.github.com/repos/apache/incubator-systemml/issues/32/events","html_url":"https://github.com/apache/incubator-systemml/pull/32","id":124859725,"number":32,"title":"[SYSTEMML-217] Add New Unary Aggregates: Variance & Standard Deviation","user":{"login":"dusenberrymw","id":5431337,"avatar_url":"https://avatars.githubusercontent.com/u/5431337?v=3","gravatar_id":"","url":"https://api.github.com/users/dusenberrymw","html_url":"https://github.com/dusenberrymw","followers_url":"https://api.github.com/users/dusenberrymw/followers","following_url":"https://api.github.com/users/dusenberrymw/following{/other_user}","gists_url":"https://api.github.com/users/dusenberrymw/gists{/gist_id}","starred_url":"https://api.github.com/users/dusenberrymw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dusenberrymw/subscriptions","organizations_url":"https://api.github.com/users/dusenberrymw/orgs","repos_url":"https://api.github.com/users/dusenberrymw/repos","events_url":"https://api.github.com/users/dusenberrymw/events{/privacy}","received_events_url":"https://api.github.com/users/dusenberrymw/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2016-01-04T23:06:43Z","updated_at":"2016-01-12T00:57:53Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/incubator-systemml/pulls/32","html_url":"https://github.com/apache/incubator-systemml/pull/32","diff_url":"https://github.com/apache/incubator-systemml/pull/32.diff","patch_url":"https://github.com/apache/incubator-systemml/pull/32.patch"},"body":"This introduces new \"***variance***\" and \"***standard deviation***\" unary operators to the DML language, including row and column variants for each: `var(X)`, `rowVars(X)`, `colVars(X)`, `sd(X)`, `rowSds(X)`, `colSds(X)`.  The implementation includes parser, HOP, LOP, runtime, and optimizer components.\r\n\r\nThe following provides an slightly more-detailed overview, and a full discussion can be found in the design document <sup>**[1]**</sup>:\r\n\r\n* Adds `sd`, `colSds`, and `rowSds` built-in functions for standard deviation to the DML language.\r\n* Adds `var`, `colVars`, and `rowVars` built-in functions for variance to the DML language.\r\n* Creates a new `VAR` operation type for the existing `AggUnaryOp` HOP.\r\n* Compiles the `sd`, `colSds`, and `rowSds` built-in DML functions to the square root of the variance via the `VAR` HOP type wrapped in a `OpOp1.SQRT` HOP type.\r\n* Creates a new `Var` operation type for the existing `Aggregate` and `PartialAggregate` LOPs.\r\n* Compiles the `VAR` HOP type to the `Var` LOP type.\r\n* Creates new runtime opcodes: `\"avar\"`, `\"uavar\"`, `\"uarvar\"`, `\"uacvar\"`.\r\n* Compiles the `Var` LOP type to new opcodes.\r\n* Parses the new opcodes into runtime aggregate operators that use the `CM` runtime function with `Variance` type.\r\n* Creates a new `VAR` aggregate type in the core unary aggregate library.\r\n* Matches the runtime aggregate operators using the `CM` function of `Variance` type to the `VAR` case.\r\n* Extends the core unary aggregate library with new variance functions for the `VAR` case.\r\n* Adds logic for incremental aggregation of variances computed for multiple partitions. <sup>**[2]**</sup>\r\n* Extends existing optimizer rewrites for column-wise and row-wise aggregation to cover row and column variances.\r\n* Adds new optimizer rewrites to optimize row and column variance operations under certain conditions.\r\n\r\n**[1]**: ~~*To be posted later once the JIRA project is set up.*~~  This has been posted to [SYSTEMML-217](https://issues.apache.org/jira/browse/SYSTEMML-217).\r\n**[2]**: The current implementations of `incrementalAggregate(...)` in `MatrixBlock` still need to be refactored, however, I've elected to postpone that to a future JIRA in order to limit the scope of this one.\r\n\r\ncc @mboehm7 "},"comment":{"url":"https://api.github.com/repos/apache/incubator-systemml/issues/comments/170746586","html_url":"https://github.com/apache/incubator-systemml/pull/32#issuecomment-170746586","issue_url":"https://api.github.com/repos/apache/incubator-systemml/issues/32","id":170746586,"user":{"login":"dusenberrymw","id":5431337,"avatar_url":"https://avatars.githubusercontent.com/u/5431337?v=3","gravatar_id":"","url":"https://api.github.com/users/dusenberrymw","html_url":"https://github.com/dusenberrymw","followers_url":"https://api.github.com/users/dusenberrymw/followers","following_url":"https://api.github.com/users/dusenberrymw/following{/other_user}","gists_url":"https://api.github.com/users/dusenberrymw/gists{/gist_id}","starred_url":"https://api.github.com/users/dusenberrymw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dusenberrymw/subscriptions","organizations_url":"https://api.github.com/users/dusenberrymw/orgs","repos_url":"https://api.github.com/users/dusenberrymw/repos","events_url":"https://api.github.com/users/dusenberrymw/events{/privacy}","received_events_url":"https://api.github.com/users/dusenberrymw/received_events","type":"User","site_admin":false},"created_at":"2016-01-12T00:57:53Z","updated_at":"2016-01-12T00:57:53Z","body":"Actually, I read (1) wrong.  No, the Jenkins test builds are placed in separate base directories, so they shouldn't interfere with each other. "}}