title: Sax: XMLReader: Streaming and converting CSV to Xml and validating the result at the same time
body: <p>I have to transform some CSV files (each > 600 MB) to XML while validating the end result with an XSD on the fly.</p>
  
  <p>Because of the size of each file I'm using InputStreams to read the content and OutpuStreams to stream the result back to my clients as XML.</p>
  
  <p>Let's start then with the easy part... (pseudocode)</p>
  
  <pre><code>void transform(final InputStream CSVCustomerStream, final OutputStream outputStream) {
         outputStream.write("&lt;customers&gt;")
         foreach csvCustomerRow in CSVCustomerStream {
              String xmlCustomerRow = csvCustomerRow.toXML();
              outputStream.write(xmlCustomerRow.getBytes();
         }
         outputStream.write("&lt;/customers&gt;")
  
         **MISSING_XMLVALIDATOR.parse(outputStream);**
  }
  </code></pre>
  
  <p>So far, each row of my source CSV file is transformed to xml and then written to the outputstream. </p>
  
  <p>Easy enough.</p>
  
  <p>The part though, where the xml is actually validated is still missing.</p>
  
  <p>For this I looked into <a href="http://docs.oracle.com/javase/7/docs/api/org/xml/sax/XMLReader.html" rel="nofollow">XMLReader</a>'s parse() method. The only problem with that, is the fact that parse() only accepts InputSources, while at the same time I'm streaming the content I'd like to be validated to an OutputStream.</p>
  
  <p>Of course, after reading the whole CSV content, I could turn the OutputStream into an InputStream via</p>
  
  <pre><code> new ByteArrayInputStream((outputstream).toByteArray())
  </code></pre>
  
  <p>but that would immediately bring 600 MB worth of XML back into memory, defying the whole purpose of streaming.</p>
  
  <p>PS: I have no control over the exact implementation of the OutputStream since my code runs as a REST-webservice</p>
  
  <pre><code>    return Response.ok(new StreamingOutput() {
          @Override
          public void write(OutputStream output) throws Exception {
              ....     loading and transforming csv ...
  
          }
      }).build();
  </code></pre>