title: Processing several files one-by-one separately by Spark
body: <p>I need help with implementation one workflow with Apache Spark. My task is in next:</p>
  
  <ol>
  <li>I have several CSV files as source data. Note: these files could has different layout</li>
  <li>I have metadata with info how I need parse each file (this is not problem)</li>
  <li>Main goal: result is source file with several additional columns. I have to update each source file without joining to one output range. For example: source 10 files -> 10 result files and each result file have data only from corresponding source file.</li>
  </ol>
  
  <p>As I know Spark can open many files by mask: </p>
  
  <pre><code>var source = sc.textFile("/source/data*.gz");
  </code></pre>
  
  <p>But in this case I can't recognize which line of a file. If I get list of source files and try to process by following scenario:</p>
  
  <pre><code>JavaSparkContext sc = new JavaSparkContext(...);
  List&lt;String&gt; files = new ArrayList() //list of source files full name's
  for(String f : files)
  {
     JavaRDD&lt;String&gt; data = sc.textFile(f);
     //process this file with Spark
     outRdd.coalesce(1, true).saveAsTextFile(f + "_out"); 
  }
  </code></pre>
  
  <p>But in this case I will processed all files in sequential mode.</p>
  
  <p><strong>My question is next</strong>: how I can processed many files in parallel mode?. For example: one file - one executor?</p>
  
  <p>Thank very much for help me!</p>