{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/11232","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/11232/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/11232/comments","events_url":"https://api.github.com/repos/apache/spark/issues/11232/events","html_url":"https://github.com/apache/spark/pull/11232","id":134176890,"number":11232,"title":"[SPARK-13333] [SQL] Added Rand and Randn Functions Generating Deterministic Results","user":{"login":"gatorsmile","id":11567269,"avatar_url":"https://avatars.githubusercontent.com/u/11567269?v=3","gravatar_id":"","url":"https://api.github.com/users/gatorsmile","html_url":"https://github.com/gatorsmile","followers_url":"https://api.github.com/users/gatorsmile/followers","following_url":"https://api.github.com/users/gatorsmile/following{/other_user}","gists_url":"https://api.github.com/users/gatorsmile/gists{/gist_id}","starred_url":"https://api.github.com/users/gatorsmile/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gatorsmile/subscriptions","organizations_url":"https://api.github.com/users/gatorsmile/orgs","repos_url":"https://api.github.com/users/gatorsmile/repos","events_url":"https://api.github.com/users/gatorsmile/events{/privacy}","received_events_url":"https://api.github.com/users/gatorsmile/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-02-17T04:42:08Z","updated_at":"2016-02-19T03:06:52Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11232","html_url":"https://github.com/apache/spark/pull/11232","diff_url":"https://github.com/apache/spark/pull/11232.diff","patch_url":"https://github.com/apache/spark/pull/11232.patch"},"body":"`rand` and `randn` functions with a `seed` argument are commonly used. Based on the common sense, the results of `rand` and `randn` should be deterministic if the `seed` parameter value is provided. For example, in MS SQL Server, it also has a function `rand`. Regarding the parameter `seed`, the description is like: ```Seed is an integer expression (tinyint, smallint, or int) that gives the seed value. If seed is not specified, the SQL Server Database Engine assigns a seed value at random. For a specified seed value, the result returned is always the same.```\r\n\r\nHowever, the current implementation is unable to generate deterministic results. It depends on data partitioning and task scheduling.\r\n\r\n@jkbradley hit an issue and provided an example in the following JIRA: https://issues.apache.org/jira/browse/SPARK-13333\r\n\r\nThis PR is to introduce a new parameter `deterministic` for `Rand` and `Randn` functions. When users set it true, the results will be deterministic. That means, if `rand` and `randn` are called with the same seed, they will generate and return identical sequences of numbers.\r\n\r\n**Question:** should we introduce a new parameter `deterministic` for `Rand` and `Randn` functions? Or just make the results deterministic when users input the parameter value of `seed`? \r\n\r\ncc component maintainers: @rxin @marmbrus because this PR made external API/behavior changes\r\ncc @cloud-fan who did most recent changes in `rand` and `randn`\r\ncc @jkbradley  who opened the JIRA and hit a problem"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/186026463","html_url":"https://github.com/apache/spark/pull/11232#issuecomment-186026463","issue_url":"https://api.github.com/repos/apache/spark/issues/11232","id":186026463,"user":{"login":"rxin","id":323388,"avatar_url":"https://avatars.githubusercontent.com/u/323388?v=3","gravatar_id":"","url":"https://api.github.com/users/rxin","html_url":"https://github.com/rxin","followers_url":"https://api.github.com/users/rxin/followers","following_url":"https://api.github.com/users/rxin/following{/other_user}","gists_url":"https://api.github.com/users/rxin/gists{/gist_id}","starred_url":"https://api.github.com/users/rxin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rxin/subscriptions","organizations_url":"https://api.github.com/users/rxin/orgs","repos_url":"https://api.github.com/users/rxin/repos","events_url":"https://api.github.com/users/rxin/events{/privacy}","received_events_url":"https://api.github.com/users/rxin/received_events","type":"User","site_admin":false},"created_at":"2016-02-19T03:06:52Z","updated_at":"2016-02-19T03:06:52Z","body":"Well when partitions are fixed, this does give you deterministic result.\r\n"}}