{"action":"created","issue":{"url":"https://api.github.com/repos/databricks/spark-csv/issues/142","repository_url":"https://api.github.com/repos/databricks/spark-csv","labels_url":"https://api.github.com/repos/databricks/spark-csv/issues/142/labels{/name}","comments_url":"https://api.github.com/repos/databricks/spark-csv/issues/142/comments","events_url":"https://api.github.com/repos/databricks/spark-csv/issues/142/events","html_url":"https://github.com/databricks/spark-csv/issues/142","id":104787497,"number":142,"title":"UTF-8 BOM not properly handled","user":{"login":"sfelsheim","id":12235090,"avatar_url":"https://avatars.githubusercontent.com/u/12235090?v=3","gravatar_id":"","url":"https://api.github.com/users/sfelsheim","html_url":"https://github.com/sfelsheim","followers_url":"https://api.github.com/users/sfelsheim/followers","following_url":"https://api.github.com/users/sfelsheim/following{/other_user}","gists_url":"https://api.github.com/users/sfelsheim/gists{/gist_id}","starred_url":"https://api.github.com/users/sfelsheim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sfelsheim/subscriptions","organizations_url":"https://api.github.com/users/sfelsheim/orgs","repos_url":"https://api.github.com/users/sfelsheim/repos","events_url":"https://api.github.com/users/sfelsheim/events{/privacy}","received_events_url":"https://api.github.com/users/sfelsheim/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2015-09-03T21:43:05Z","updated_at":"2016-08-18T02:14:09Z","closed_at":null,"body":"**Using Spark 1.4.1 & spark-csv_2.11 version 1.2.0** \r\n      \r\nI have two simple CSV files, address and people, with the following contents.\r\n\r\naddrId,city,state,zip\r\n0,sometown,wi,4444\r\n2,bedrock,il,1111\r\n\r\nfirst,last,addrId\r\nyour,mom,1\r\nfred,flintstone,2\r\n\r\nI load them into DataFrames using:\r\n\r\n    DataFrame pDf = sqlContext.read().format(\"com.databricks.spark.csv\")\r\n                .option(\"header\", \"true\")\r\n                .option(\"inferSchema\", \"false\")\r\n                .option(\"delimiter\", \",\")\r\n                .load(\"/Users/sfelsheim/data/people.csv\");\r\n\r\n        DataFrame aDf = sqlContext.read().format(\"com.databricks.spark.csv\")\r\n                .option(\"header\", \"true\")\r\n                .option(\"inferSchema\", \"false\")\r\n                .option(\"delimiter\", \",\")\r\n                .load(\"/Users/sfelsheim/data/address.csv\");\r\n\r\nWhen I try to join pDf and aDf like this:\r\n\r\n        DataFrame joined = aDf.join(pDf,\r\n                aDf.col(\"addrId\").equalTo(pDf.col(\"addrId\")));\r\n\r\nif fails with this exception:\r\n\r\n    Exception in thread \"main\" org.apache.spark.sql.AnalysisException: Cannot resolve column name     \"addrId\" among (﻿addrId, city, state, zip);\r\n\tat org.apache.spark.sql.DataFrame$$anonfun$resolve$1.apply(DataFrame.scala:159)\r\n\tat org.apache.spark.sql.DataFrame$$anonfun$resolve$1.apply(DataFrame.scala:159)\r\n\tat scala.Option.getOrElse(Option.scala:121)\r\n\tat org.apache.spark.sql.DataFrame.resolve(DataFrame.scala:158)\r\n\tat org.apache.spark.sql.DataFrame.col(DataFrame.scala:558)\r\n\tat dw.dataflow.TestJoin.main(TestJoin.java:81)\r\n\r\nIf I create DataFrames using the exact same two CSV files like this:\r\n\r\n    JavaRDD<Person> people = context.textFile(\"/Users/sfelsheim/data/people.csv\").map(\r\n                new Function<String, Person>() {\r\n                    public Person call(String line) throws Exception {\r\n                        String[] parts = line.split(\",\");\r\n\r\n                        Person person = new Person();\r\n                        person.setFirst(parts[0]);\r\n                        person.setLast(parts[1]);\r\n                        person.setAddrId(parts[2]);\r\n\r\n                        return person;\r\n                    }\r\n                });\r\n    DataFrame pDf = sqlContext.createDataFrame(people, Person.class);\r\nand\r\n\r\n      JavaRDD<Address> address = context.textFile(\"/Users/sfelsheim/data/address.csv\").map(\r\n                new Function<String, Address>() {\r\n                    public Address call(String line) throws Exception {\r\n                        String[] parts = line.split(\",\");\r\n\r\n                        Address addr = new Address();\r\n                        addr.setAddrId(parts[0]);\r\n                        addr.setCity(parts[1]);\r\n                        addr.setState(parts[2]);\r\n                        addr.setZip(parts[3]);\r\n\r\n                        return addr;\r\n                    }\r\n                });\r\n     DataFrame aDf = sqlContext.createDataFrame(address, Address.class);\r\nThe join I mentioned above works as I would expect.\r\n\r\nAny ideas?\r\n\r\nThanks\r\n\r\nSteve"},"comment":{"url":"https://api.github.com/repos/databricks/spark-csv/issues/comments/240604923","html_url":"https://github.com/databricks/spark-csv/issues/142#issuecomment-240604923","issue_url":"https://api.github.com/repos/databricks/spark-csv/issues/142","id":240604923,"user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"created_at":"2016-08-18T02:14:09Z","updated_at":"2016-08-18T02:14:09Z","body":"Not clean and perfect and not even tested but I think this might be a workaround.\r\n\r\n```scala\r\nval UTF8_BOM = \"...\"\r\nval path = \"...\"\r\nval csvRdd = sparkContext.textFile(path)\r\nval droppedCsvRDD = csvRdd.mapPartitionsWithIndex { (idx, iter) =>\r\n  if (idx == 0) {\r\n    var shouldStripBom = true\r\n    iter.map { line =>\r\n      if (bomSkipped) {\r\n        if (line.startsWith(UTF8_BOM)) {\r\n          // Here, strip BOM.\r\n          line.substring(1);\r\n          bomSkipped = false\r\n        }\r\n      } else {\r\n        line\r\n      }\r\n    }\r\n  } else {\r\n    iter\r\n  }\r\n}\r\nval csvDf = new CsvParser().csvRdd(sqlContext, droppedCsvRDD)\r\n```"}}
{"actor":{"display_login":"HyukjinKwon"}}