{"action":"created","issue":{"url":"https://api.github.com/repos/databricks/spark-csv/issues/290","repository_url":"https://api.github.com/repos/databricks/spark-csv","labels_url":"https://api.github.com/repos/databricks/spark-csv/issues/290/labels{/name}","comments_url":"https://api.github.com/repos/databricks/spark-csv/issues/290/comments","events_url":"https://api.github.com/repos/databricks/spark-csv/issues/290/events","html_url":"https://github.com/databricks/spark-csv/issues/290","id":139634243,"number":290,"title":"issue with reading null values","user":{"login":"amitmohanty","id":1369847,"avatar_url":"https://avatars.githubusercontent.com/u/1369847?v=3","gravatar_id":"","url":"https://api.github.com/users/amitmohanty","html_url":"https://github.com/amitmohanty","followers_url":"https://api.github.com/users/amitmohanty/followers","following_url":"https://api.github.com/users/amitmohanty/following{/other_user}","gists_url":"https://api.github.com/users/amitmohanty/gists{/gist_id}","starred_url":"https://api.github.com/users/amitmohanty/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amitmohanty/subscriptions","organizations_url":"https://api.github.com/users/amitmohanty/orgs","repos_url":"https://api.github.com/users/amitmohanty/repos","events_url":"https://api.github.com/users/amitmohanty/events{/privacy}","received_events_url":"https://api.github.com/users/amitmohanty/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-03-09T16:48:03Z","updated_at":"2016-03-10T00:30:03Z","closed_at":null,"body":"I know there has been some discussion around reading null values in spark-csv, but I could not find any concrete answer. So, here I go.\r\n\r\nI have a csv file whose contents are like this\r\n\r\n![image](https://cloud.githubusercontent.com/assets/1369847/13642395/56100ff0-e5ea-11e5-99ae-b7f7eee9311c.png)\r\n\r\n\r\nI am starting my spark-shell like this \r\n\r\n`$ /s/god/spark-1.6.0-bin-hadoop2.6/bin/spark-shell --jars /s/god/spark-1.6.0-bin-hadoop2.6/lib/spark-csv_2.10-1.3.0.jar,/s/god/spark-1.6.0-bin-hadoop2.6/lib/commons-csv-1.2.jar --master yarn-client --num-executors 20 --executor-cores 8 --executor-memory 24g --driver-memory 8g`\r\n\r\nAnd do the following\r\n\r\n`scala> val trainSchema = StructType(Array(StructField(\"ID\", IntegerType, true), StructField(\"ID\", DoubleType, true)))`\r\n\r\n`scala> val a = sqlContext.read.format(\"com.databricks.spark.csv\").option(\"header\", \"true\").option(\"nullValue\", \"null\").schema(trainSchema).load(\"/user/god/test_data.csv\")\r\n`\r\nand\r\n\r\n`scala> a.show()\r\n`\r\ncauses the following error:\r\n\r\n`java.text.ParseException: Unparseable number: \"null\"`\r\n"},"comment":{"url":"https://api.github.com/repos/databricks/spark-csv/issues/comments/194587951","html_url":"https://github.com/databricks/spark-csv/issues/290#issuecomment-194587951","issue_url":"https://api.github.com/repos/databricks/spark-csv/issues/290","id":194587951,"user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"created_at":"2016-03-10T00:30:03Z","updated_at":"2016-03-10T00:30:03Z","body":"It looks the problem is that the fields in the schema is not set currently.\r\n\r\nit looks working fine. Would you try the codes below?\r\n\r\n- codes\r\n\r\n```scala\r\nval schema = StructType(\r\n  Array(\r\n    StructField(\"id\", IntegerType, true),\r\n    StructField(\"value\", DoubleType, true)))\r\nval results = sqlContext\r\n  .read\r\n  .format(\"com.databricks.spark.csv\")\r\n  .option(\"header\", \"true\")\r\n  .option(\"nullValue\", \"null\")\r\n  .schema(schema)\r\n  .load(\"/user/god/test_data.csv\")\r\nresults.show()\r\n```\r\n\r\n- output\r\n\r\n```\r\n+---+-----+\r\n| id|value|\r\n+---+-----+\r\n|  1| null|\r\n|  1| null|\r\n|  1| null|\r\n|  1| null|\r\n|  1| null|\r\n|  2|  2.0|\r\n|  2|  2.1|\r\n|  2|  3.1|\r\n|  2|  3.8|\r\n|  3| null|\r\n|  3| null|\r\n|  3| null|\r\n|  3|  4.0|\r\n|  3| null|\r\n|  3|  2.0|\r\n|  3|  2.0|\r\n|  3| null|\r\n+---+-----+\r\n```"}}