title: How to serialize the data to AVRO schema in Spark (with Java)?
body: <p>I have defined an AVRO schema, and generated some classes with avro-tools for the schemes. Now, I want to serialize the data to disk. I found some answers about scala for this, but not for Java. The class <code>Article</code> is generated with avro-tools, and is made from a schema defined by me.</p>
  
  <p>Here's a simplified version of the code of how I try to do it:</p>
  
  <pre><code>JavaPairRDD&lt;String, String&gt; filesRDD = context.wholeTextFiles(inputDataPath);
  JavaRDD&lt;Article&gt; processingFiles = filesRDD.map(fileNameContent -&gt; {
      // The name of the file
      String fileName = fileNameContent._1();
      // The content of the file
      String fileContent = fileNameContent._2();
  
      // An object from my avro schema
      Article a = new Article(fileContent);
  
      Processing processing = new Processing();
      // .... some processing of the content here ... //
  
      processing.serializeArticleToDisk(avroFileName);
  
      return a;
  });
  </code></pre>
  
  <p>where <code>serializeArticleToDisk(avroFileName)</code> is defined as follows:</p>
  
  <pre><code>public void serializeArticleToDisk(String filename) throws IOException{
      // Serialize article to disk
      DatumWriter&lt;Article&gt; articleDatumWriter = new SpecificDatumWriter&lt;Article&gt;(Article.class);
      DataFileWriter&lt;Article&gt; dataFileWriter = new DataFileWriter&lt;Article&gt;(articleDatumWriter);
      dataFileWriter.create(this.article.getSchema(), new File(filename));
      dataFileWriter.append(this.article);
      dataFileWriter.close();
  }
  </code></pre>
  
  <p>where <code>Article</code> is my avro schema.</p>
  
  <p>Now, the mapper throws me the error:</p>
  
  <pre><code>java.io.FileNotFoundException: hdfs:/...path.../avroFileName.avro (No such file or directory)   
  at java.io.FileOutputStream.open0(Native Method)    
  at java.io.FileOutputStream.open(FileOutputStream.java:270)     
  at java.io.FileOutputStream.&lt;init&gt;(FileOutputStream.java:213)   
  at java.io.FileOutputStream.&lt;init&gt;(FileOutputStream.java:162)   
  at org.apache.avro.file.SyncableFileOutputStream.&lt;init&gt;(SyncableFileOutputStream.java:60)   
  at org.apache.avro.file.DataFileWriter.create(DataFileWriter.java:129)
  at org.apache.avro.file.DataFileWriter.create(DataFileWriter.java:129)
  at sentences.ProcessXML.serializeArticleToDisk(ProcessXML.java:207)     
  . . . rest of the stacktrace ... 
  </code></pre>
  
  <p>although the file path is correct. </p>
  
  <p>I use a <code>collect()</code> method afterwards, so everything else within the <code>map</code> function works fine (except for the serialization part). </p>
  
  <p>I am quite new with Spark, so I am not sure if this might be something trivial actually. I suspect that I need to use some writing functions, not to do the writing in the mapper (not sure if this is true, though). Any ideas how to tackle this?</p>
  
  <p><strong>EDIT:</strong></p>
  
  <p>The last line of the error stack-trace I showed, is actually on this part:</p>
  
  <p><code>dataFileWriter.create(this.article.getSchema(), new File(filename));</code></p>
  
  <p>This is the part that throws the actual error. I am assuming the <code>dataFileWriter</code> needs to be replaced with something else. Any ideas?</p>