{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/16817","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/16817/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/16817/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/16817/events","html_url":"https://github.com/elastic/elasticsearch/issues/16817","id":136601947,"number":16817,"title":"Correlation Metric Aggregator","user":{"login":"nknize","id":830187,"avatar_url":"https://avatars.githubusercontent.com/u/830187?v=3","gravatar_id":"","url":"https://api.github.com/users/nknize","html_url":"https://github.com/nknize","followers_url":"https://api.github.com/users/nknize/followers","following_url":"https://api.github.com/users/nknize/following{/other_user}","gists_url":"https://api.github.com/users/nknize/gists{/gist_id}","starred_url":"https://api.github.com/users/nknize/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nknize/subscriptions","organizations_url":"https://api.github.com/users/nknize/orgs","repos_url":"https://api.github.com/users/nknize/repos","events_url":"https://api.github.com/users/nknize/events{/privacy}","received_events_url":"https://api.github.com/users/nknize/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Aggregations","name":":Aggregations","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/feature","name":"feature","color":"328a58"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v2.3.0","name":"v2.3.0","color":"dddddd"}],"state":"open","locked":false,"assignee":{"login":"nknize","id":830187,"avatar_url":"https://avatars.githubusercontent.com/u/830187?v=3","gravatar_id":"","url":"https://api.github.com/users/nknize","html_url":"https://github.com/nknize","followers_url":"https://api.github.com/users/nknize/followers","following_url":"https://api.github.com/users/nknize/following{/other_user}","gists_url":"https://api.github.com/users/nknize/gists{/gist_id}","starred_url":"https://api.github.com/users/nknize/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nknize/subscriptions","organizations_url":"https://api.github.com/users/nknize/orgs","repos_url":"https://api.github.com/users/nknize/repos","events_url":"https://api.github.com/users/nknize/events{/privacy}","received_events_url":"https://api.github.com/users/nknize/received_events","type":"User","site_admin":false},"milestone":null,"comments":7,"created_at":"2016-02-26T05:33:23Z","updated_at":"2016-02-26T17:51:34Z","closed_at":null,"body":"This first Multi-Field metric aggregation will compute the [Pearson product-moment correlation coefficient](https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient) for a given list of numeric fields. \r\n\r\nExample usage (correlating stock value against S&P 500):\r\n\r\n```javascript\r\n\"pearson_correlation\" : {\r\n    \"correlation\" : {\r\n        \"field\" : [ \"FOO_Corp_Stock_Value\", \"AAPL\", \"INDEXSP\" ],\r\n    }\r\n}\r\n```\r\n\r\nExample result:\r\n\r\nThe result provides just the upper triangle (excluding the auto-correlated diagonal which would just be 1.0)\r\n```javascript\r\n\"pearson_correlation\" : {\r\n    \"correlation\" : {\r\n        \"FOO_Corp_Stock_Value\" : {\r\n            \"AAPL\" : 0.015411381889249296,\r\n            \"INDEXSP\" : 0.028740529828528586\r\n        },\r\n        \"AAPL\" : {\r\n            \"INDEXSP\" : 0.429228885527728843\r\n        }\r\n    }\r\n}\r\n```\r\n\r\nThis hypothetical example shows a small positive correlation (0.0154) between Apple and FOO Corp. stock value. A small positive correlation (0.0287) between FOO Corp. stock and the S&P 500, and a strong (fake) positive correlation (0.4292) between Apple Stock and the S&P 500.\r\n\r\nWhile a made up example with contrived data, this correlation provides a useful statistical tool for drawing correlations between all kinds of data (medical, finance, log trends) and provides a foundation for much more advanced statistics (e.g., spatial statistics and econometrics)."},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/189392073","html_url":"https://github.com/elastic/elasticsearch/issues/16817#issuecomment-189392073","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/16817","id":189392073,"user":{"login":"polyfractal","id":1224228,"avatar_url":"https://avatars.githubusercontent.com/u/1224228?v=3","gravatar_id":"","url":"https://api.github.com/users/polyfractal","html_url":"https://github.com/polyfractal","followers_url":"https://api.github.com/users/polyfractal/followers","following_url":"https://api.github.com/users/polyfractal/following{/other_user}","gists_url":"https://api.github.com/users/polyfractal/gists{/gist_id}","starred_url":"https://api.github.com/users/polyfractal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/polyfractal/subscriptions","organizations_url":"https://api.github.com/users/polyfractal/orgs","repos_url":"https://api.github.com/users/polyfractal/repos","events_url":"https://api.github.com/users/polyfractal/events{/privacy}","received_events_url":"https://api.github.com/users/polyfractal/received_events","type":"User","site_admin":false},"created_at":"2016-02-26T17:51:34Z","updated_at":"2016-02-26T17:51:34Z","body":"If your data is \"well distributed\" and i.i.d. I think a single-pass over the data would give a good estimate of correlation (I think?).  If we assume that each shard has a reasonable sampling of your data, we can pretend each shard can be correlated in isolation and merge the results into a pseudo-approximate answer.\r\n\r\nIt's effectively taking the same set of tradeoffs that search makes.  We can't search everything together, so just pretend the data is distributed the same across all shards and run the calculations in isolation.  If your data is distributed wildly differently between shards, all bets are off (but they are with search ranking too).\r\n\r\nThat said, I was thinking about this yesterday and we'd probably need a pipeline version too.  Time-series data *isn't* i.i.d. (current value is usually related to last value in some fashion), so you'd likely need other pipeline aggs to first remove inter-series dependence before testing for correlation.  Didn't occur to me until I started playing around with TS correlations yesterday =/"}}