title: Read a bytes column in spark
body: <p>I have a data set which contains an ID field that is in an unknown (and not friendly) encoding. I can read the single column using plain python and verify that the values are distinct and consistent across multiple data sets (i.e. it can be used as a primary key for joining).</p>
  
  <p>When loading the file using <code>spark.read.csv</code>, it seems that spark is converting the column to <code>utf-8</code>. However, some of the multibyte sequences are converted to the Unicode character <code>U+FFFD REPLACEMENT CHARACTER.</code> (<code>EF BF BD</code> in hex).</p>
  
  <p>Is there a way to force Spark to read the column as bytes and not as a string?</p>
  
  <p>Here is some code that can be used to recreate my issue (let column <code>a</code> be the ID field):</p>
  
  <p><strong>Create a File with Sample Data</strong></p>
  
  <pre class="lang-python prettyprint-override"><code>data = [
      (bytes(b'\xba\xed\x85\x8e\x91\xd4\xc7\xb0'), '1', 'a'),
      (bytes(b'\xba\xed\x85\x8e\x91\xd4\xc7\xb1'), '2', 'b'),
      (bytes(b'\xba\xed\x85\x8e\x91\xd4\xc7\xb2'), '3', 'c')
  ]
  
  with open('sample.csv', 'wb') as f:
      header = ["a", "b", "c"]
      f.write(",".join(header)+"\n")
      for d in data:
          f.write(",".join(d) + "\n")
  </code></pre>
  
  <p><strong>Read using Pandas</strong></p>
  
  <pre class="lang-python prettyprint-override"><code>import pandas as pd
  df = pd.read_csv("sample.csv", converters={"a": lambda x: x.encode('hex')})
  print(df)
  #                  a  b  c
  #0  baed858e91d4c7b0  1  a
  #1  baed858e91d4c7b1  2  b
  #2  baed858e91d4c7b2  3  c
  </code></pre>
  
  <p><strong>Try reading the same file using Spark</strong></p>
  
  <pre class="lang-python prettyprint-override"><code>spark_df = spark.read.csv("sample.csv", header=True)
  spark_df.show()
  #+-----+---+---+
  #|a    |b  |c  |
  #+-----+---+---+
  #|�텎��ǰ|1  |a  |
  #|�텎��Ǳ|2  |b  |
  #|�텎��ǲ|3  |c  |
  #+-----+---+---+
  </code></pre>
  
  <p>Yikes! OK, so how about converting to <code>hex</code>?</p>
  
  <pre class="lang-python prettyprint-override"><code>import pyspark.sql.functions as f
  spark_df.withColumn("a", f.hex("a")).show(truncate=False)
  #+----------------------------+---+---+
  #|a                           |b  |c  |
  #+----------------------------+---+---+
  #|EFBFBDED858EEFBFBDEFBFBDC7B0|1  |a  |
  #|EFBFBDED858EEFBFBDEFBFBDC7B1|2  |b  |
  #|EFBFBDED858EEFBFBDEFBFBDC7B2|3  |c  |
  #+----------------------------+---+---+
  </code></pre>
  
  <p>(In this example the values are distinct, but that's not true in my larger file)</p>
  
  <p>As you can see, the values are <em>close</em>, but some of the bytes have been replaced by <code>EFBFBD</code></p>
  
  <p>Is there any way to read the file in Spark (maybe using <code>rdd</code>?) so that my output looks like the pandas version:</p>
  
  <pre class="lang-python prettyprint-override"><code>#+----------------+---+---+
  #|a               |b  |c  |
  #+----------------+---+---+
  #|baed858e91d4c7b0|1  |a  |
  #|baed858e91d4c7b1|2  |b  |
  #|baed858e91d4c7b2|3  |c  |
  #+----------------+---+---+
  </code></pre>
  
  <p>I've tried casting to <code>byte</code> and specifying the schema so that this column is <code>ByteType()</code> but that didn't work.</p>
  
  <p><strong>Edit</strong></p>
  
  <p>I am using Spark v 2.1.</p>