title: Concurrent text file reads by numerous VMs
body: <p>I have a java app I am porting as a proof-of-concept to a cloud architecture. I want to process a very large text file by running the same processing program on chunks of the file on separate VMs. </p>
  
  <p>Worker nodes = n</p>
  
  <p>Head node running Master and one Worker, with n-1 Worker nodes</p>
  
  <p>I have two ideas in mind:</p>
  
  <ol>
  <li><p>Master reads file line-by-line, sends first line to first worker node, second to second worker node and so on, repeating every n lines.</p></li>
  <li><p>Master reads number of lines in file. Worker nodes then instructed to read <code>no_of_lines</code>/n concurrently from the file.</p></li>
  </ol>
  
  <p>I am considering using an RMI or sockets based approach for transfer of data. Could anyone tell me which of the above methods would be most efficient? If this question cannot be answered without specifying which java constructs I would be using, I would appreciate suggestions on those.</p>
  
  <p>Also, would locking be an issue with concurrent file access if I each node knows which lines it is supposed to read?</p>
  
  <p>Thanks for any suggestions</p>
  
  <p>Ian</p>