title: Camel file component throughput when readLock=changed
body: <p>I'm using Apache Camel to transfer files from an input directory to a messagebroker. The files are written via SFTP. To avoid consuming incomplete files that are still in transit, I've set <code>readLock=changed</code> and <code>readLockCheckInterval=3000</code>.</p>
  
  <p>As an example, this is how one of my tests looks:</p>
  
  <pre><code>    &lt;route&gt;
          &lt;from uri="file:inbox?readLock=changed&amp;amp;readLockCheckInterval=3000"/&gt;
          &lt;log message="copying ${file:name}"/&gt;
          &lt;to uri="file:outbox"/&gt;
      &lt;/route&gt;
  </code></pre>
  
  <p>I test this with <code>(echo line 1; sleep 2; echo line 2) &gt; inbox/test</code> and the file gets copied faithfully when <code>readLockCheckInterval=3000</code>. However, this doesn't scale, because the <code>file</code> component will wait three seconds before processing each file. So when I test with </p>
  
  <pre><code>for n in $(seq 1 100); do (echo line 1; sleep 2; echo line 2) &gt; inbox/$n &amp; done
  </code></pre>
  
  <p>it takes camel five minutes to move the files from <code>inbox</code> to <code>outbox</code>.</p>
  
  <p>I've read the chapter on parallel processing in the Camel in Action book. But the examples focus on parallelizing processing of lines in a single consumed file. I couldn't find a way to parallelize the consumer itself.</p>
  
  <p>A throughput of around one file per second would be fine in my use-case. I just don't like the idea of being forced to risk incomplete data to achieve it. The <code>readLock=changed</code> setting seems like a hack anyway, but we can't tell the customer to copy then move, so there doesn't seem to be another option.</p>
  
  <p>How can I improve throughput without sacrificing integrity in the face of network delays?</p>