{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/23215","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/23215/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/23215/comments","events_url":"https://api.github.com/repos/apache/spark/issues/23215/events","html_url":"https://github.com/apache/spark/pull/23215","id":387199780,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM1NzQwNDk4","number":23215,"title":"[SPARK-26263][SQL] Validate partition values with user provided schema","user":{"login":"gengliangwang","id":1097932,"node_id":"MDQ6VXNlcjEwOTc5MzI=","avatar_url":"https://avatars0.githubusercontent.com/u/1097932?v=4","gravatar_id":"","url":"https://api.github.com/users/gengliangwang","html_url":"https://github.com/gengliangwang","followers_url":"https://api.github.com/users/gengliangwang/followers","following_url":"https://api.github.com/users/gengliangwang/following{/other_user}","gists_url":"https://api.github.com/users/gengliangwang/gists{/gist_id}","starred_url":"https://api.github.com/users/gengliangwang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gengliangwang/subscriptions","organizations_url":"https://api.github.com/users/gengliangwang/orgs","repos_url":"https://api.github.com/users/gengliangwang/repos","events_url":"https://api.github.com/users/gengliangwang/events{/privacy}","received_events_url":"https://api.github.com/users/gengliangwang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2018-12-04T09:46:28Z","updated_at":"2018-12-06T15:40:04Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/23215","html_url":"https://github.com/apache/spark/pull/23215","diff_url":"https://github.com/apache/spark/pull/23215.diff","patch_url":"https://github.com/apache/spark/pull/23215.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nCurrently if user provides data schema, partition column values are converted as per it. But if the conversion failed, e.g. converting string to int, the column value is null. \r\n\r\nThis PR proposes to throw exception in such case, instead of converting into null value silently:\r\n1. These null partition column values doesn't make sense to users in most cases. It is better to show the conversion failure, and then users can adjust the schema or ETL jobs to fix it.\r\n2. There are always exceptions on such conversion failure for non-partition data columns. Partition columns should have the same behavior.\r\n\r\n\r\n\r\nWe can reproduce the case above as following:\r\n```\r\n/tmp/testDir\r\n├── p=bar\r\n└── p=foo\r\n```\r\nIf we run:\r\n```\r\nval schema = StructType(Seq(StructField(\"p\", IntegerType, false)))\r\nspark.read.schema(schema).csv(\"/tmp/testDir/\").show()\r\n```\r\nWe will get:\r\n```\r\n+----+\r\n|   p|\r\n+----+\r\n|null|\r\n|null|\r\n+----+\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nUnit test\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/444915152","html_url":"https://github.com/apache/spark/pull/23215#issuecomment-444915152","issue_url":"https://api.github.com/repos/apache/spark/issues/23215","id":444915152,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NDkxNTE1Mg==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-12-06T15:40:04Z","updated_at":"2018-12-06T15:40:04Z","author_association":"NONE","body":"Merged build finished. Test FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}