{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14389","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14389/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14389/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14389/events","html_url":"https://github.com/apache/spark/pull/14389","id":168032281,"number":14389,"title":"[SPARK-16714][SQL] Refactor type widening for consistency","user":{"login":"petermaxlee","id":20197098,"avatar_url":"https://avatars.githubusercontent.com/u/20197098?v=3","gravatar_id":"","url":"https://api.github.com/users/petermaxlee","html_url":"https://github.com/petermaxlee","followers_url":"https://api.github.com/users/petermaxlee/followers","following_url":"https://api.github.com/users/petermaxlee/following{/other_user}","gists_url":"https://api.github.com/users/petermaxlee/gists{/gist_id}","starred_url":"https://api.github.com/users/petermaxlee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermaxlee/subscriptions","organizations_url":"https://api.github.com/users/petermaxlee/orgs","repos_url":"https://api.github.com/users/petermaxlee/repos","events_url":"https://api.github.com/users/petermaxlee/events{/privacy}","received_events_url":"https://api.github.com/users/petermaxlee/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":20,"created_at":"2016-07-28T07:37:29Z","updated_at":"2016-07-29T12:56:01Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14389","html_url":"https://github.com/apache/spark/pull/14389","diff_url":"https://github.com/apache/spark/pull/14389.diff","patch_url":"https://github.com/apache/spark/pull/14389.patch"},"body":"## What changes were proposed in this pull request?\r\nThis patch refactors type widening and makes the usage of them in expressions more consistent.\r\n\r\nBefore this patch, we have the following 6 functions for type widening (and their usage):\r\n\r\n```\r\nfindTightestCommonTypeOfTwo (binary version)\r\n- BinaryOperator\r\n- IfNull\r\n- NullIf\r\n- Nvl2\r\n- JSON schema inference\r\n\r\nfindTightestCommonTypeToString (binary version)\r\n- Nvl\r\n\r\nfindTightestCommonTypeAndPromoteToString (n-ary version)\r\n- CreateArray\r\n- CreateMap\r\n\r\nfindTightestCommonType (n-ary version)\r\n- Greatest\r\n- Least\r\n\r\nfindWiderTypeForTwo (binary version)\r\n- IfCoercion\r\n\r\nfindWiderCommonType (n-ary version)\r\n- WidenSetOperationTypes\r\n- InConversion\r\n- Coalesce\r\n- CaseWhenCoercion\r\n```\r\n\r\nAfter this patch, we have only 3 functions for type widening (and their usage):\r\n\r\n```\r\nfindTightestCommonTypeOfTwo (binary version)\r\n- BinaryOperator\r\n- JSON schema inference\r\n\r\nfindWiderTypeForTwo (binary version)\r\n- IfCoercion\r\n- Nvl\r\n- IfNull\r\n- NullIf\r\n- Nvl2\r\n\r\nfindWiderCommonType (n-ary version)\r\n- WidenSetOperationTypes\r\n- InConversion\r\n- Coalesce\r\n- CaseWhenCoercion\r\n- Greatest\r\n- Least\r\n- CreateArray\r\n- CreateMap\r\n```\r\n\r\nAs a result, this patch changes the type coercion rule for the aforementioned functions so they can accept decimals of different precision/scale. This is not a regression from Spark 1.x, but it is a much bigger problem in Spark 2.0 because floating point literals are parsed as decimals. Queries below would fail in Spark 2.0:\r\n\r\n```\r\nscala> sql(\"select array(0.001, 0.02)\")\r\norg.apache.spark.sql.AnalysisException: cannot resolve `array(CAST(0.001 AS DECIMAL(3,3)), CAST(0.02 AS DECIMAL(2,2)))` due to data type mismatch: input to function array should all be the same type, but it's [decimal(3,3), decimal(2,2)]; line 1 pos 7\r\n```\r\n\r\n## How was this patch tested?\r\nCreated a new end-to-end test suite SQLTypeCoercionSuite. In the future we can move all other type checking tests here. I first tried adding a test in SQLQuerySuite but the suite was clearly already too large.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/236172997","html_url":"https://github.com/apache/spark/pull/14389#issuecomment-236172997","issue_url":"https://api.github.com/repos/apache/spark/issues/14389","id":236172997,"user":{"login":"cloud-fan","id":3182036,"avatar_url":"https://avatars.githubusercontent.com/u/3182036?v=3","gravatar_id":"","url":"https://api.github.com/users/cloud-fan","html_url":"https://github.com/cloud-fan","followers_url":"https://api.github.com/users/cloud-fan/followers","following_url":"https://api.github.com/users/cloud-fan/following{/other_user}","gists_url":"https://api.github.com/users/cloud-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/cloud-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloud-fan/subscriptions","organizations_url":"https://api.github.com/users/cloud-fan/orgs","repos_url":"https://api.github.com/users/cloud-fan/repos","events_url":"https://api.github.com/users/cloud-fan/events{/privacy}","received_events_url":"https://api.github.com/users/cloud-fan/received_events","type":"User","site_admin":false},"created_at":"2016-07-29T12:56:01Z","updated_at":"2016-07-29T12:56:01Z","body":"and can you make sure that it's safe to use `findWiderTypeForTwo` for `Nvl` and `Nvl2`?"}}
{"actor":{"display_login":"cloud-fan"}}