title: How to set 'charset' for DatumWriter || write avro that contains arabic characters to HDFS
body: <p>Some of the data contains value in Arabic format, and when the data is written, reader code/<code>hadoop fs -text</code> command shows <code>??</code> instead of Arabic characters.</p>
  <p><strong>1) Writer</strong></p>
  <pre><code>// avro object is provided as SpecificRecordBase
  Path path = new Path(pathStr);
  DatumWriter&lt;SpecificRecord&gt; datumWriter = new SpecificDatumWriter&lt;&gt;();
  FileSystem fs = FileSystem.get(URI.create(hdfsUri), conf); // HDFS File System
  
  FSDataOutputStream outputStream = fs.create(path);
  DataFileWriter&lt;SpecificRecord&gt; dataFileWriter = new DataFileWriter&lt;&gt;(datumWriter);
  
  Schema schema = getSchema(); // method to get schema
  dataFileWriter.setCodec(CodecFactory.snappyCodec());
  dataFileWriter.create(schema, outputStream);
  dataFileWriter.append(avroObject);
  </code></pre>
  <p><strong>2) Reader</strong></p>
  <pre><code>Configuration conf = new Configuration();
  FsInput in = new FsInput(new Path(hdfsFilePathStr), conf);
  DatumReader&lt;Row&gt; datumReader = new GenericDatumReader&lt;&gt;();
  DataFileReader&lt;Row&gt; dataFileReader = new DataFileReader&lt;&gt;(in, datumReader);
  GenericRecord outputData = (GenericRecord) dataFileReader.iterator.next();
  </code></pre>
  <p>I've tried <code>hadoop fs -text {filePath}</code> command, there also the values in Arabic appear as <code>??</code>.</p>
  <p>It will be really difficult to change the format in which data is written because there are numerous consumers of the same file.</p>
  <p>Tried reading through <code>SpecificRecordBase</code>, still getting <code>??</code>.</p>
  <p><strong>Edit</strong></p>
  <p>Also tried these (in both reader and writer):</p>
  <pre><code>Configuration conf = new Configuration();
  conf.set(&quot;file.encoding&quot;, StandardCharsets.UTF_16.displayName());
  </code></pre>
  <p>AND</p>
  <pre><code>System.setProperty(&quot;file.encoding&quot;, StandardCharsets.UTF_16.displayName());
  </code></pre>
  <p>Doesn't help.</p>