title: Apache Spark: Read umlauts with SparkContext.textFile
body: <p>I wrote a Spark Application which reads, filters and prints data from a CSV file, saved in HDFS. When I download the CSV to my pc and open it, everything is ok, all umlauts are displayed correctly. BUT: When I run my application, the umlauts are not shown correctly, they all are displayed as <code>ï¿½</code>.</p>
  
  <p>Is there a possibility to set an encoding for loading the csv via .textFile transaction in Spark or to "convert" the input?</p>
  
  <p><strong>This seems to be a Won't Fix:</strong> <a href="https://issues.apache.org/jira/browse/SPARK-1849" rel="nofollow">https://issues.apache.org/jira/browse/SPARK-1849</a></p>
  
  <p>The problem occurs due to the UTF-8 encoding of the file. Seems, that there's a workaround by using <code>.hadoopFile</code>, but I don't know how to use this method. Can someone help?</p>
  
  <p><strong>Tried the following code, without success:</strong></p>
  
  <p>test.txt file:</p>
  
  <pre><code>ÄPPLE
  APPLE
  ÖPPLE
  ÜPPLE
  </code></pre>
  
  <p>Spark program:</p>
  
  <pre><code>public static void main(String[] args) throws ClassNotFoundException {
      String inputFile = args[0];
  
      SparkConf sparkConf = new SparkConf().setAppName("EdekaOwgCheckUmlauts");
      sparkConf.registerKryoClasses(new Class&lt;?&gt;[]{
          Class.forName("org.apache.hadoop.io.LongWritable"),
          Class.forName("org.apache.hadoop.io.Text")
      });
  
      JavaSparkContext sc = new JavaSparkContext(sparkConf);
  
      JavaRDD&lt;Text&gt; input = sc.hadoopFile(inputFile, TextInputFormat.class, LongWritable.class, Text.class).map(new Function&lt;Tuple2&lt;LongWritable,Text&gt;, Text&gt;() {
          @Override
          public Text call(Tuple2&lt;LongWritable, Text&gt; v1) throws Exception {
              return v1._2();
          }
      });
  
      JavaRDD&lt;Text&gt; filtered = input.filter(new Function&lt;Text, Boolean&gt;() {
          @Override
          public Boolean call(Text v1) throws Exception {
              return v1.toString().toLowerCase().contains("pple");        // LOOK FOR ÄPPLE, APPLE etc.
          }
      });
  
      List&lt;Text&gt; collected = filtered.collect();
      System.out.println("=== PRINT ===");
      for(Text row : collected) {
          try {
              System.out.println(row.decode(row.getBytes()));
          } catch (CharacterCodingException e) {
              e.printStackTrace();
          }
      }
  }
  </code></pre>
  
  <p>Console Output:</p>
  
  <pre><code>ï¿½PPLE
  APPLE
  ï¿½PPLE
  ï¿½PPLE
  </code></pre>
  
  <p><strong>The file is encoded with the charset iso-8859-1</strong></p>
  
  <p>Calling <code>file -i myfile.csv</code> shows the following information:</p>
  
  <pre><code>myfile.csv: text/plain; charset=iso-8859-1
  </code></pre>
  
  <p>I tried to convert this charset into utf-8 using this code:</p>
  
  <pre><code>JavaRDD&lt;String&gt; inputRDD = sc.textFile(inputFile);
  List&lt;String&gt; asList = inputRDD.collect();
  for(String a : asList) {
  
      try {
          byte[] isoBytes = a.getBytes("ISO-8859-1");
          System.out.println(new String(isoBytes, "UTF-8"));
      } catch (Exception e) {
          e.printStackTrace();
      }
  }
  </code></pre>
  
  <p>without success, the console prints <code>?</code> instead ä, ö, ü...</p>