{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13773","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13773/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13773/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13773/events","html_url":"https://github.com/apache/spark/pull/13773","id":161082658,"number":13773,"title":"[SPARK-16056] [SPARK-16057] [SPARK-16058] [SQL] Fix Multiple Bugs in Column Partitioning in JDBC Source ","user":{"login":"gatorsmile","id":11567269,"avatar_url":"https://avatars.githubusercontent.com/u/11567269?v=3","gravatar_id":"","url":"https://api.github.com/users/gatorsmile","html_url":"https://github.com/gatorsmile","followers_url":"https://api.github.com/users/gatorsmile/followers","following_url":"https://api.github.com/users/gatorsmile/following{/other_user}","gists_url":"https://api.github.com/users/gatorsmile/gists{/gist_id}","starred_url":"https://api.github.com/users/gatorsmile/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gatorsmile/subscriptions","organizations_url":"https://api.github.com/users/gatorsmile/orgs","repos_url":"https://api.github.com/users/gatorsmile/repos","events_url":"https://api.github.com/users/gatorsmile/events{/privacy}","received_events_url":"https://api.github.com/users/gatorsmile/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2016-06-19T17:22:03Z","updated_at":"2016-06-21T03:24:10Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13773","html_url":"https://github.com/apache/spark/pull/13773","diff_url":"https://github.com/apache/spark/pull/13773.diff","patch_url":"https://github.com/apache/spark/pull/13773.patch"},"body":"#### What changes were proposed in this pull request?\r\nThis PR is to fix the following bugs:\r\n\r\n**Issue 1: Wrong Results when lowerBound is larger than upperBound in Column Partitioning**\r\n```scala\r\nspark.read.jdbc(\r\n  url = urlWithUserAndPass,\r\n  table = \"TEST.seq\",\r\n  columnName = \"id\",\r\n  lowerBound = 4,\r\n  upperBound = 0,\r\n  numPartitions = 3,\r\n  connectionProperties = new Properties)\r\n```\r\n**Before code changes:**\r\nThe returned results are wrong and the generated partitions are wrong:\r\n```\r\n  Part 0 id < 3 or id is null\r\n  Part 1 id >= 3 AND id < 2\r\n  Part 2 id >= 2\r\n```\r\n**After code changes:**\r\nIssue an `IllegalArgumentException` exception:\r\n```\r\nOperation not allowed: the lower bound of partitioning column is larger than the upper bound. lowerBound: 5; higherBound: 1\r\n```\r\n**Issue 2: numPartitions is more than the number of key values between upper and lower bounds**\r\n```scala\r\nspark.read.jdbc(\r\n  url = urlWithUserAndPass,\r\n  table = \"TEST.seq\",\r\n  columnName = \"id\",\r\n  lowerBound = 1,\r\n  upperBound = 5,\r\n  numPartitions = 10,\r\n  connectionProperties = new Properties)\r\n```\r\n**Before code changes:**\r\nReturned correct results but the generated partitions are very inefficient, like:\r\n```\r\nPartition 0: id < 1 or id is null\r\nPartition 1: id >= 1 AND id < 1\r\nPartition 2: id >= 1 AND id < 1\r\nPartition 3: id >= 1 AND id < 1\r\nPartition 4: id >= 1 AND id < 1\r\nPartition 5: id >= 1 AND id < 1\r\nPartition 6: id >= 1 AND id < 1\r\nPartition 7: id >= 1 AND id < 1\r\nPartition 8: id >= 1 AND id < 1\r\nPartition 9: id >= 1\r\n```\r\n**After code changes:**\r\nAdjust `numPartitions` and can return the correct answers:\r\n```\r\nPartition 0: id < 2 or id is null\r\nPartition 1: id >= 2 AND id < 3\r\nPartition 2: id >= 3 AND id < 4\r\nPartition 3: id >= 4\r\n```\r\n**Issue 3: java.lang.ArithmeticException when numPartitions is zero**\r\n```Scala\r\nspark.read.jdbc(\r\n  url = urlWithUserAndPass,\r\n  table = \"TEST.seq\",\r\n  columnName = \"id\",\r\n  lowerBound = 0,\r\n  upperBound = 4,\r\n  numPartitions = 0,\r\n  connectionProperties = new Properties)\r\n```\r\n**Before code changes:**\r\nGot the following exception:\r\n```\r\n  java.lang.ArithmeticException: / by zero\r\n```\r\n**After code changes:**\r\nAble to return a correct answer by disabling column partitioning when numPartitions is equal to or less than zero\r\n\r\n#### How was this patch tested?\r\nAdded test cases to verify the results"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/227330892","html_url":"https://github.com/apache/spark/pull/13773#issuecomment-227330892","issue_url":"https://api.github.com/repos/apache/spark/issues/13773","id":227330892,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-06-21T03:24:10Z","updated_at":"2016-06-21T03:24:10Z","body":"**[Test build #60895 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60895/consoleFull)** for PR 13773 at commit [`7d2ae9a`](https://github.com/apache/spark/commit/7d2ae9a0ea061577402aece4c2b49458009d7029).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}