{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15990","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15990/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15990/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15990/events","html_url":"https://github.com/apache/spark/pull/15990","id":191212870,"number":15990,"title":"[SPARK-18559] [SQL] Fix HLL++ with small relative error","user":{"login":"wzhfy","id":10878553,"avatar_url":"https://avatars.githubusercontent.com/u/10878553?v=3","gravatar_id":"","url":"https://api.github.com/users/wzhfy","html_url":"https://github.com/wzhfy","followers_url":"https://api.github.com/users/wzhfy/followers","following_url":"https://api.github.com/users/wzhfy/following{/other_user}","gists_url":"https://api.github.com/users/wzhfy/gists{/gist_id}","starred_url":"https://api.github.com/users/wzhfy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wzhfy/subscriptions","organizations_url":"https://api.github.com/users/wzhfy/orgs","repos_url":"https://api.github.com/users/wzhfy/repos","events_url":"https://api.github.com/users/wzhfy/events{/privacy}","received_events_url":"https://api.github.com/users/wzhfy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2016-11-23T08:36:56Z","updated_at":"2016-11-23T15:24:38Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15990","html_url":"https://github.com/apache/spark/pull/15990","diff_url":"https://github.com/apache/spark/pull/15990.diff","patch_url":"https://github.com/apache/spark/pull/15990.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nIn `HyperLogLogPlusPlus`, if the relative error is so small that p >= 19, it will cause ArrayIndexOutOfBoundsException in `THRESHOLDS(p-4)` . We should check `p` and when p >= 19, regress to the original HLL result and use the small range correction they use.\r\n\r\nThe pr also fixes the upper bound in the log info in `require()`.\r\nThe upper bound is computed by:\r\n```\r\nval relativeSD = 1.106d / Math.pow(Math.E, p * Math.log(2.0d) / 2.0d)\r\n```\r\nwhich is derived from the equation for computing `p`:\r\n```\r\nval p = 2.0d * Math.log(1.106d / relativeSD) / Math.log(2.0d)\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nadd test case for:\r\n1. checking validity of parameter relatvieSD\r\n2. estimation with smaller relative error so that p >= 19"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/262544972","html_url":"https://github.com/apache/spark/pull/15990#issuecomment-262544972","issue_url":"https://api.github.com/repos/apache/spark/issues/15990","id":262544972,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-11-23T15:24:38Z","updated_at":"2016-11-23T15:24:38Z","body":"Merged build finished. Test PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}